Speaking at KubeCon 2017 - Part 4 - The Talk

Wed, 03 Jan 2018 13:57:07 +0000

My final corrections and preparation, how it went, and future thoughts as I wrap up this four post series of my experience leading up to and speaking at KubeCon 2017.

More Visuals Needed!

While I felt like my dry-run slide deck was visually appealing in color scheme, used varying font styles for emphasis, and had decent clip-art style photos, it lacked a visual foundation. I needed to convey a picture of a Kubernetes cluster that was simple enough to understand by the audience yet had enough detail to encompass all the attacks/hardening steps.

I spent well over 20 hrs coming up with the visual of Slide 22 using Cloudcraft. I was happy that I now had a visual anchor of the playing field that I could leverage when explaining each attack type.

Preparation

I knew that I needed to spend less time on the setup, less time doing the demos, and more time explaining how to harden things. And shave about 20 mins off my run time.

As I practiced, I found ways to set up each slide with fewer words. I consolidated several slides and threw out a handful altogether. I even went so far as to hand edit the asciinema JSON such that the typing delays more closely matched my prepared words. So when I clicked “start” on the demo, what I said was concisely summarizing the text going across the screen and calling the audience’s attention to the exact spot I wanted them to see as it appeared.

I practiced maybe a dozen times before arriving in Austin. And I probably doubled that number while I was there in my hotel room before falling asleep each night. I suppose it helped me burn off the nervous energy since I was speaking after lunch on the last day.

The Day Of

I went to the keynotes in the morning, but I was so nervous, I skipped the two morning sessions. My talk was at 2pm, the first after lunch, so I grabbed a sandwich and went to an empty side of the conference to practice out loud. The problem with this approach, you see, is that your voice is a perishable commodity. Having recently gotten over a cough and having spoken out loud all week to myself in practicing, a bit of a hoarseness was creeping in.

The 10 mins waiting for the talk to start were when I was most nervous. Is my battery still good? Is the screen still displaying or did it sleep? Is my slide advancer still working? My microphone level good?

The moment I started speaking, I knew it was going to be just fine. Yes, I had a bit of that cough pop up, but the room was attentive and I got a lot of head nods. I thought I set up the demos decently, executed them well, and gotten their attention on the right moments. What threw me off, though, was the lack of questions. I think to some degree, most folks were a combination of conference-fatiguted and still somewhat firehosed with the info. I had a few really good conversations and congratulatory remarks from folks in the hallway afterward. Finally, I felt relief. I remember saying to myself, “Ok, THAT’s done”.

Parting Thoughts

I appreciate all the help I’ve received from my family, close friends, and the Kuberentes community members (see the last slide on my deck) for the patience, support, and guidance in helping me complete this journey. And thank you to the CNCF/KubeCon Committee members for giving me the opportunity to present.

This blog series is a diary of sorts to help me remember all the hard work, why I did it, and to allow myself to feel proud about the past 5 months. I hope that chronicling it all might help would-be presenters see my end-to-end process, see the final result, and have a better perspective of the overall experience. Of course, not every talk requires or should be as technically dense or try to cover so much ground. I firmly believe that everyone has something of value to offer the community if they look hard enough into their work or ask the right questions of themselves. If not, be patient and keep asking. It’ll come.

Speaking at KubeCon 2017 - Part 3 - First Run

Wed, 03 Jan 2018 13:56:07 +0000

The opportunity to present a “dry run” of this talk was the most important lesson learned of the level of preparation required to really pull off a solid performance at KubeCon.

The Dry Run

About the time I submitted my CFP, a the NOVA Kubernetes Meetup was just getting started and had its first meetup. I offered and was gratiously accepted to speak at the next meetup in October. I know myself enough to know that I work best with a deadline, so setting one would sharpen the focus.

I practiced a few times, and every iteration was better than the last, but also became longer. I became so driven to prove my attacks worked and were really simple that I added lots of asciinema demos. I figured I could chop it down a bit later.

Going into the talk, I was looking for a couple things.

Did I establish myself with enough credibility?
Did the demos present well?
Did I cover everything?

Here’s the original version of the talk given at the meetup: http://goo.gl/p42Shd

Initial Reception

During the talk, I quickly realized that despite the majority of the audience having used Kubernetes, I was only getting about 3 or 4 folks nodding their heads throughout each walkthrough. And when I glanced down at my watch, I realized that I was cracking 45 mins and I didn’t even get to the hardening bits yet.

I had overshot the audience, firehosed them with information that was far too dense, and went way over the time. The audience clapped and a few thanked me personally for the talk, but I knew I swung big and missed.

The slides were shared via Twitter, and I received a warm response from some folks, but a comment from one person made me realize that I had inadvertently made it seem like I was picking on a specific project. While I thought I was just using it as an example that covered the most use cases, it was received very differently. I responded to that person and vowed to be more careful about impartiality.

So, about those goals:

Did I establish myself with enough credibility? - Yep, but just being the speaker. I didn’t need to do anything more than that.
Did the demos present well? - Practically speaking, yes. They worked. I just had too many of them and went way too quickly.
Did I cover everything? - Yes, at the expense of overwhelming the audience and going way long on time.

Speaking at KubeCon 2017 - Part 2 - Research

Wed, 03 Jan 2018 13:55:07 +0000

Having performed a good portion of the research and validation prior to being accepted as a form of passing the time, I’ll admit I let the scope creep a bit too far.

How I Approached the Problem

My search for a Kubernetes installer or managed service that shared my desired set of default security capabilities and settings hardened out-of-the-box took much longer than I had anticipated. It was a very time consuming journey, but that arduous process fraught with tedium brought the perspective and expertise over time that I needed to feel confident during the talk.

After manually discovering, installing, configuring, testing/prodding, and destroying a dozen or so tools/services, I was able to come up with a set of possible attacks stemming from configuration issues that were common to most of them.

It was then that I realized that many of the installers had released updates in that short window of time, forever locking those findings to those versions.

I needed a tool to help me solve the problem of being able to easily acquire and install newer versions of an installer’s release while also being able to go back and re-assess an older release several months later reliably.

KubeATF was hacked together late at night using the tools I knew best: bash and Ansible. It attempts to be a simple CRUD interface for spinning up, validating, and destroying clusters with repeatable results. While it may not be useful outside this context, I think it was worth sharing in the interest of being completely transparent about my assessment approach in case anyone disagreed with it.

Reaching Out to the Community

When I reached out directly to the folks at Heptio, kops, and kube-aws, I got very positive responses to what I was highlighting. Instead of ignoring, downplaying, or outright dismissing me, I was asked to help them ensure they accurately captured the issues. This provided all the validation I needed to know that this work was not only useful – it was openly welcomed.

Disclosing to a couple of the bigger named providers was much more formal and drawn out, but it certainly wasn’t a negative experience. Thankfully, all of my issues identified were already on their radar and on track for resolution.

Scope Creep

The months between acceptance and actually presenting were longer than I expected, so I became somewhat of a connoisseur of installers. It was a pretty standard process to take one and make it work inside kubeatf, so I could assess them against my criteria from start to finish in under 3 hrs. In hindsight, I went overboard a bit, but it did make for a more comprehensive looking results graph.

I also spent some time making a plugin for Heptio’s Sonobuoy tool called bulkhead in the attempt to perform some level of assessment against the CIS Kubernetes Benchmark by dropping in Aqua Security’s kube-bench tool. This really could have been an entire talk topic itself, so I released it in its initial format with plans on working on it again in the future.

Speaking at KubeCon 2017 - Part 1 - The CFP Submission

Wed, 03 Jan 2018 13:54:07 +0000

In this four part blog series on my journey to speaking at KubeCon 2017, I want to share the knowledge that I have learned in the hopes that it encourages others in the community to muster the courage and confidence to bring their own unique experiences to light.

The Desire to Speak

I’ve been in the InfoSec space for some time, and I’ve attended a decent range of conferences as an attendee or vendor in all of them. In some cases, I’ve given presentations to decent sized audiences as a Sales Engineer, but it wasn’t a topic of passion. The preparation process and execution was excellent practice, though, and it definitely removed my fear of public speaking.

So, it seems I was always prepared mechanically to do it. I just needed the right mix of topic, expertise, and timing. I knew I could execute on it if I just found the right “thing” to speak about.

Choosing the Topic

When the project at work I was working on hit a major milestone, I had a chance to think about what we accomplished leveraging Kubernetes. I reflected on the fact that in the middle of our project, KubeCon 2016 videos were released. I distinctly remember saying to myself, “Wow, what a treasure trove!” and I immediately linked several videos to colleagues with a must watch tag. In the back of my mind, I sort of had this notion that I owed the community something in return.

That reflection led to thinking about a deep dive technical overview of what we built, how we built it, and what we learned as something to potentially submit. Everybody loves deep dives, right? All the hype gets peeled back, and the harsh realities and trade-offs of production come to light. What better way to give back to the system administrators and implementors?

While I was listing some of the challenges that we had to solve or work around, I realized that most of them were related to hardening our Kubernetes cluster for our use case. The defaults were pretty wide open, and we spent a lot of time “closing the door and locking it” so to speak.

As I made some notes for each item on the list of security hardening steps we took, I decided to look at some of the other projects in the community that installed Kubernetes to see if they already had these covered. As in, “maybe the installer we picked just had these as the defaults” and we needed to use another installer next time around. I began a quest of sorts to find “the one” Kubernetes installation or managed service that embodied and enforced the “internally secure components and isolated workload” opinions that I shared. However, after looking at about 9 or 10, the pattern had emerged.

The community had to know what I saw. I felt morally and ethically compelled to share this information, because I’d wager a good percentage of production clusters in the wild were essentially one small vulnerability from cluster takeover or cloud account compromise.

I set out to contact every single project and/or service I looked at to let them know what I was finding, and in that process, I met an uncharacteristically welcoming and friendly community. Every single one of them responded to my writeups. Every single one thanked me. Most kept in touch over the months leading up to KubeCon. To say that it was a positive culture difference from the InfoSec world would be an understatement.

Writing the CFP Submission

I suppose I was fortunate that I had this knowledge before writing the CFP submission. It certainly made it easier to write. But CFP submissions aren’t judged solely on technical merit. Having never written one before, I decided that mimicking the tone and delivery of prior successful abstract and bios was my best bet.

So, I hit up the CNCF Youtube page, sorted by “most popular”, and pulled the abstracts and bios of about two dozen of the best technical videos and tried to find a few that sounded like something I’d write, but better. After surgically creating my franken-bio-stract, I had a decent start.

Looking at the CFP submission form, though, there was an extra field to the effect of “How would your talk benefit the community?”. I’ll admit, this fantastic question threw me for a loop. “Yeah, how would I maximize this talk and the supporting data in terms of the contribution to others?” I remember saying to a good friend.

Peer Review

It’s my firm belief that the 4 conversations I had prior to submitting were what shaped it into something worthy of selection. Answering for questions like “who is your audience, and why would they care?” and “what are you doing to fix these issues?” made me realize my talk was missing a key goal: “What can I show the audience that is better today than 4 months ago when I found and disclosed these issues?”

Once I asked the right question of myself, I had the right answer to the community benefit section. My goals became:

Demonstrate what is feasible and possible in terms of potential attack vectors
Disclose those issues to the community in a responsible way early
Determine the best methods of prevention for each attack
Work really hard to have fixes already in place or clear guidance on how to prevent them before the talk
Present the issues with those fixes to the public/audience in clear and objective manner

Hacking and Hardening Kubernetes Clusters by Example wasn’t the greatest title, but it got to the point of the talk decently enough. After writing and re-writing it about 6 times, I finally submitted it once I was convinced “I simply can’t make it any better without making it worse”.

The Waiting Game

And then, the wait. Submissions were due on August 21st, but the overwhelming number of submissions led to a form of DoS attack on the committee that made the selection take a few weeks longer than anticipated.

The only way to take my mind off of the waiting was to decide that I was going to do the talk regardless of acceptance to KubeCon, so just set out to do the work as if I was.

When the acceptance email came, I was awash with relief. I had spent a lot of personal time and racked up some decent cloud bills doing the legwork, and I was happy to know the plan was coming together. The next day, though, I was overcome with deep, deep feelings of Imposter Syndrome. I felt this sudden weight of pressure to live up to some imaginary bar of the excellence of all previous KubeCon talks, the desire not to let the committee regret their choice because so many good folks had theirs rejected, and the thought that maybe I wasn’t “expert” enough to deliver this topic properly.

While I can’t say I completely overcame those feelings, sharing that I had them with several members of the community helped me achieve some perspective. I’ll admit they were the constant driver to keep going and to produce the best possible body of work I could on the topic.

Installing Kubernetes on Baremetal via CoreOS Tectonic with Grub Booting

Wed, 02 Aug 2017 08:54:07 +0000

At work, we use kube-aws to deploy our Kubernetes clusters running on top of Container Linux inside of AWS. I wanted to be able try new things with Kubernetes in my personal lab without having to rack up huge AWS bills, so that means figuring out a good way to deploy Kubernetes to baremetal in the least painful way. I wanted to try Tectonic because it offers a simplified graphical installer, and that means I also need to install Matchbox to support the baremetal provisioning aspects.

After reading through the entire Tectonic Installation Guide, I realized that it doesn’t cover some of the underlying OS provisioning components as they vary per environment. Since I don’t have that infrastructure as I’m starting from scratch, I had to get that going before continuing the installation. Here’s my summary of steps taken (including a custom dnsmasq container) to round out a full working guide.

Architecture Overview

My personal lab is a mixture of PCs, 1U servers, and Mac hardware. So, for this lab, I’m going to pick one of each to ensure the entire process would work on all my hardware.

As per the installation guide, Tectonic needs three systems at a minimum. Here are my systems per role:

Deployment/Provisioning system
- deploy.lab - 172.22.10.2/24
Kubernetes Controller
- mp.lab -172.22.10.50/24
Kubernetes Worker
- smpc.lab -172.22.10.54/24

Network Diagram

The simplistic network environment is as follows:

Detailed PXE/Netboot to CoreOS Installation Flow

The following describes the complete baremetal provisioning process from DHCP request to CoreOS installation. Configuring the provisioning system to support this workflow is described in the subsequent section.

Using IPMI or a keyboard during booting, tell the system to “boot from network”. This will cause the NIC to perform a DHCP request and look for TFTP related settings to boot from. In my case, it was F12 for my server and holding n during the boot chime for the Mac.
The DHCP server responds with an IP and subnet along with information pointing to the TFTP server and a filename of what to download/run from that TFTP server.
The system attempts to connect to the TFTP server and download/run that initial file. In the case of grub2, that initial boot file runs and then also tries to contact the same TFTP server looking for a grub boot configuration.
If a grub boot configuration file is found, it follows that configuration. In the case of matchbox, it should be a pointing to its web port and passing the NIC’s MAC address:

 1 set default="0"
 2 insmod video_bochs
 3 insmod video_cirrus
 4 insmod all_video
 5 set gfxpayload=keep
 6 insmod gzio
 7 insmod part_gpt
 8 insmod ext2
 9 set timeout=10
10 menuentry 'CoreOS Install' --class os {
11      insmod net
12      insmod http
13      echo 'Loading Linux ...'
14      set root=http,deploy.lab:8080
15      configfile /grub?mac=$net_default_mac
16 }

Since the initial grub configuration does nothing but load an HTTP module and defer to a web address for the rest of the grub configuration, it provides a convenient way to grab a system-specific boot configuration without having to change your TFTP provided configuration file. In this case, Matchbox answers to web requests at the /grub?mac=XX:XX:XX:XX:XX:XX URL with a tailored boot configuration, like so:

[admin@deploy ~]$ curl http://deploy.lab:8080/grub?mac=00:23:32:2f:40:3c
default=0
fallback=1
timeout=1
menuentry "CoreOS (EFI)" {
  echo "Loading kernel"
  linuxefi "/assets/coreos/1409.5.0/coreos_production_pxe.vmlinuz" coreos.config.url=http://deploy.lab:8080/ignition?mac=$net_efinet0_dhcp_mac coreos.first_boot=yes console=tty0 console=ttyS0
  echo "Loading initrd"
  initrdefi  "/assets/coreos/1409.5.0/coreos_production_pxe_image.cpio.gz"
}
menuentry "CoreOS (BIOS)" {
  echo "Loading kernel"
  linux "/assets/coreos/1409.5.0/coreos_production_pxe.vmlinuz" coreos.config.url=http://deploy.lab:8080/ignition?mac=$net_efinet0_dhcp_mac coreos.first_boot=yes console=tty0 console=ttyS0
  echo "Loading initrd"
  initrd  "/assets/coreos/1409.5.0/coreos_production_pxe_image.cpio.gz"
}

Now that grub knows what to boot, where to get it, and extra kernel parameters for the ignition configuration for what to install inside the OS and how, the system can begin and complete the installation. Here is the ignition configuration that the CoreOS kernel pulls from the coreos.config.url URL which basically says to install CoreOS Container Linux and reboot:

[admin@deploy ~]$ curl http://deploy.lab:8080/ignition?mac=00:23:32:2f:40:3c
{
  "ignition": {
    "version": "2.0.0",
    "config": {
    }
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "path": "\/opt\/installer",
        "contents": {
          "source": "data:,%23!%2Fbin%2Fbash%20-ex%0Acurl%20%22http%3A%2F%2Fdeploy.lab%3A8080%2Fignition%3Fmac%3D00%3A23%3A32%3A2f%3A40%3A3c%26os%3Dinstalled%22%20-o%20ignition.json%0Acoreos-install%20-d%20%2Fdev%2Fsda%20-C%20stable%20-V%201409.5.0%20-i%20ignition.json%20-b%20http%3A%2F%2Fdeploy.lab%3A8080%2Fassets%2Fcoreos%0Audevadm%20settle%0Asystemctl%20reboot%0A",
          "verification": {  
          }
        },
        "mode": 320,
        "user": {
        },
        "group": {
          
        }
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "name": "installer.service",
        "enable": true,
        "contents": "[Unit]\nRequires=network-online.target\nAfter=network-online.target\n[Service]\nType=simple\nExecStart=\/opt\/installer\n[Install]\nWantedBy=multi-user.target\n"
      }
    ]
  },
  "networkd": {
  },
  "passwd": {
    "users": [
      {
        "name": "debug",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCugz\/eu\/a6U5ZXtYVp9ufCghVSP0Lux\/nP6BgbranKE6r3h3xgqo8yR2LUG9VwH6Vo+BtGgToeww+jbgr3oq9g8\/mQmNvQEefvWyzhYxrpv6q36fdYS0KhQxA6vnpOZZ1M9cZ1q6iPaUryxDUFU3HULDKM4g\/6XBzqBJZ2illyuw=="
        ],
        "create": {
          "groups": [
            "sudo",
            "docker"
          ]
        }
      }
    ]
  }
}

Here are the URL-decoded contents of /opt/installer from inside the ignition configuration above. Notice how it uses the os=installed parameter to pull a “normal” boot configuration specific to this machine for future booting after being installed to disk /dev/sda:

1 #!/bin/bash -ex
2 curl "http://deploy.lab:8080/ignition?mac=00:23:32:2f:40:3c&os=installed" -o ignition.json
3 coreos-install -d /dev/sda -C stable -V 1409.5.0 -i ignition.json -b http://deploy.lab:8080/assets/coreos
4 udevadm settle
5 systemctl reboot

Also, if you notice the name of the user available via SSH during that first boot/installation is debug and uses the same SSH key as what will be available after the installation completes and is rebooted for the permanent user core. This is really handy if you are troubleshooting why the installation is failing or want to watch that process as it goes. Note that it’s really only available for a few minutes on quick systems since the installation completes so quickly.

At this point, CoreOS/Container Linux has been installed to /dev/sda, a user named core with an SSH key set, and has an ignition configuration that configured its systemd units. This is where Matchbox/Ignition stop and normal SSH-based administration can take over.

Provisioning Infrastructure Configuration

There are several components that run on the deploy.lab system that all need to work in concert for the above process to be successful:

Matchbox - The CoreOS provided container that handles the web serving of grub, CoreOS, and Igntition templates provided by the Tectonic installer.
DHCP, DNS, TFTP, Grub Network Boot Images and Configuration - This is handled by a custom container built using dnsmasq.

Setting up `deploy.lab`

I hand installed Centos 7.3 with the minimal install and ensured that it had a recent version of Docker with SSH key authentication as the admin user in the docker group:

~$ ssh deploy
[admin@deploy ~]$ groups 
admin wheel docker
[admin@deploy ~]$ id
uid=1000(admin) gid=1000(admin) groups=1000(admin),10(wheel),993(docker)
[admin@deploy ~]$ docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:25 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:25 2017
 OS/Arch:      linux/amd64
 Experimental: false

Installing and Configuring the Matchbox Container on `deploy.lab`

The matchbox documentation for running via docker is a bit misleading as it actually requires several things to be completed before actually running the container.

First, create the matchbox user and create/own the /var/lib/matchbox directory where it will keep all the assets and profiles.

[admin@deploy ~]$ sudo useradd -U matchbox
[admin@deploy ~]$ sudo mkdir -p /var/lib/matchbox/assets
[admin@deploy ~]$ sudo chown -R matchbox:matchbox /var/lib/matchbox

Download the matchbox package, verify its signature, and untar it:

[admin@deploy ~]$ wget https://github.com/coreos/matchbox/releases/download/v0.6.1/matchbox-v0.6.1-linux-amd64.tar.gz
[admin@deploy ~]$ wget https://github.com/coreos/matchbox/releases/download/v0.6.1/matchbox-v0.6.1-linux-amd64.tar.gz.asc
[admin@deploy ~]$ gpg --keyserver pgp.mit.edu --recv-key 18AD5014C99EF7E3BA5F6CE950BDD3E0FC8A365E
[admin@deploy ~]$ gpg --verify matchbox-v0.6.1-linux-amd64.tar.gz.asc matchbox-v0.6.1-linux-amd64.tar.gz

[admin@deploy ~]$ tar xzvf matchbox-v0.6.1-linux-amd64.tar.gz
[admin@deploy ~]$ cd matchbox-v0.6.1-linux-amd64

Run a script to grab the version(s) of CoreOS/Container Linux to the current directory and then copy them to the assets directory to be available via matchbox on port 8080:

[admin@deploy ~]$ ./scripts/get-coreos stable 1409.5.0 .
[admin@deploy ~]$ sudo cp -r coreos /var/lib/matchbox/assets

Drop out of the matchbox installation directory and run it via docker:

[admin@deploy ~]$ cd ~
[admin@deploy ~]$ docker run --net=host --rm \
    -v /var/lib/matchbox:/var/lib/matchbox:Z \
    -v /etc/matchbox:/etc/matchbox:Z,ro \
    quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 \
    -rpc-address=0.0.0.0:8081 -log-level=debug

Finally, verify that matchbox is running and able to serve up your downloaded CoreOS image(s). If you see this, you should be good to go:

[admin@deploy ~]$ curl http://deploy.lab:8080/assets/coreos/1409.5.0/
<pre>
<a href="CoreOS_Image_Signing_Key.asc">CoreOS_Image_Signing_Key.asc</a>
<a href="coreos_production_image.bin.bz2">coreos_production_image.bin.bz2</a>
<a href="coreos_production_image.bin.bz2.sig">coreos_production_image.bin.bz2.sig</a>
<a href="coreos_production_pxe.vmlinuz">coreos_production_pxe.vmlinuz</a>
<a href="coreos_production_pxe.vmlinuz.sig">coreos_production_pxe.vmlinuz.sig</a>
<a href="coreos_production_pxe_image.cpio.gz">coreos_production_pxe_image.cpio.gz</a>
<a href="coreos_production_pxe_image.cpio.gz.sig">coreos_production_pxe_image.cpio.gz.sig</a>
</pre>

Installing and Configuring the DNSMasq (DNS, DHCP, TFTP, Grub) Container on `deploy.lab`

It’s easiest to grab a copy of the repo and build your own docker image locally.

[admin@deploy ~]$ git clone https://github.com/bgeesaman/maas
[admin@deploy ~]$ cd maas

Edit the files in files directory as needed. Most changes are IP addresses, MAC addresses, and hostnames for your environment:

[admin@deploy maas]$ vi files/dnsmasq.conf
[admin@deploy maas]$ vi files/grub/*

Finally, build and run the image:

[admin@deploy maas]$ ./buildandrun.sh

Running the Tectonic Installer

With the above in place and a free license from CoreOS for Tectonic, you can now follow the Tectonic Baremetal with Graphical Installer guide having satisfied the pre-requisites–with one exception.

Grub-specific Gotcha

The Tectonic installer will run you through several steps of supplying configuration and will arrive at a point where it instructs you to “power on your systems” that are to be baremetal booted from the network, but it won’t work out-of-the-box. The details are in this github issue for what is happening to prevent grub from working by default. The good news is that there is a simple workaround. On the deploy.lab system, this is the default profile that the Tectonic GUI installer places into your /var/lib/matchbox/profiles folder:

[admin@deploy ~]$ cat /var/lib/matchbox/profiles/coreos-install.json 
{
	"id": "coreos-install",
	"ignition_id": "coreos-install.yaml.tmpl",
	"boot": {
		"kernel": "/assets/coreos/1409.5.0/coreos_production_pxe.vmlinuz",
		"initrd": [
			"/assets/coreos/1409.5.0/coreos_production_pxe_image.cpio.gz"
		],
		"args": [
			"coreos.config.url=http://deploy.lab:8080/ignition?uuid=${uuid}\u0026mac=${mac:hexhyp}",
			"coreos.first_boot=yes",
			"console=tty0",
			"console=ttyS0"
		]
	}
}

Notice the args section. When the system is booting from the network and pulls the /grub?mac=XX:XX:XX:XX:XX:XX configuration, this args list is directly dropped into the kernel line. However, the uuid and mac:hexhyp variables are ipxe boot environment specific. For grub2, it actually varies slightly. To fix this, we need to make some customizations to the matchbox groups configuration files. I chose to make one for each system based on the mac address selector. Notice that I now reference the coreos-install-mp or coreos-install-smpc profiles:

[admin@deploy ~]$ cat /var/lib/matchbox/groups/coreos-install-mp.json 
{
	"id": "coreos-install-mp",
	"profile": "coreos-install-mp",
	"selector": {
		"mac": "00:23:32:2f:40:3c"
	},
	"metadata": {
		"baseurl": "http://deploy.lab:8080/assets/coreos",
		"coreos_channel": "stable",
		"coreos_version": "1409.5.0",
		"ignition_endpoint": "http://deploy.lab:8080/ignition",
		"ssh_authorized_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCugz/eu/a6U5ZXtYVp9ufCghVSP0Lux/nP6BgbranKE6r3h3xgqo8yR2LUG9VwH6Vo+BtGgToeww+jbgr3oq9g8/mQmNvQEefvWyzhYxrpv6q36fdYS0KhQxA6vnpOZZ1M9cZ1q6iPaUryxDUFU3HULDKM4g/6XBzqBJZ2illyuw=="
	}
}
[admin@deploy ~]$ cat /var/lib/matchbox/groups/coreos-install-smpc.json 
{
	"id": "coreos-install-smpc",
	"profile": "coreos-install-smpc",
	"selector": {
		"mac": "00:30:48:fb:e2:44"
	},
	"metadata": {
		"baseurl": "http://deploy.lab:8080/assets/coreos",
		"coreos_channel": "stable",
		"coreos_version": "1409.5.0",
		"ignition_endpoint": "http://deploy.lab:8080/ignition",
		"ssh_authorized_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCugz/eu/a6U5ZXtYVp9ufCghVSP0Lux/nP6BgbranKE6r3h3xgqo8yR2LUG9VwH6Vo+BtGgToeww+jbgr3oq9g8/mQmNvQEefvWyzhYxrpv6q36fdYS0KhQxA6vnpOZZ1M9cZ1q6iPaUryxDUFU3HULDKM4g/6XBzqBJZ2illyuw=="
	}
}

We also need to make those renamed profiles. Notice the variable $net_efinet0_dhcp_mac for UEFI/Mac hardware and the $net_default_mac variable for BIOS booting hardware. Also notice that I made another unique ignition template to install to /dev/sdb instead of /dev/sda for the smpc.lab system:

[admin@deploy ~]$ cat /var/lib/matchbox/profiles/coreos-install-mp.json 
{
	"id": "coreos-install",
	"ignition_id": "coreos-install.yaml.tmpl",
	"boot": {
		"kernel": "/assets/coreos/1409.5.0/coreos_production_pxe.vmlinuz",
		"initrd": [
			"/assets/coreos/1409.5.0/coreos_production_pxe_image.cpio.gz"
		],
		"args": [
			"coreos.config.url=http://deploy.lab:8080/ignition?mac=$net_efinet0_dhcp_mac",
			"coreos.first_boot=yes",
			"console=tty0",
			"console=ttyS0"
		]
	}
}
[admin@deploy ~]$ cat /var/lib/matchbox/profiles/coreos-install-smpc.json 
{
	"id": "coreos-install",
	"ignition_id": "coreos-install-sdb.yaml.tmpl",
	"boot": {
		"kernel": "/assets/coreos/1409.5.0/coreos_production_pxe.vmlinuz",
		"initrd": [
			"/assets/coreos/1409.5.0/coreos_production_pxe_image.cpio.gz"
		],
		"args": [
			"coreos.config.url=http://deploy.lab:8080/ignition?mac=$net_default_mac",
			"coreos.first_boot=yes",
			"console=tty0",
			"console=ttyS0"
		]
	}
}

Once the above changes have been made, you should be able to successfully PXE/Netboot the systems and continue on with the final portion of the Tectonic installer. If you run into issues, double-check your formatting of the profiles and groups as well as hitting the /grub and /ignition endpoints with the proper parameters to see what configurations are being provided to your systems.

Congratulations! You should now be able to hit the web UI of the Tectonic Console, use kubectl, and ssh into the systems using the core user and the SSH key you supplied. I hope this helps you understand what’s going on behind a fairly sophisticated and easily customizable baremetal Container Linux and Kubernetes installation system.

CaaS Part 1 - Kubernetes on Centos 7.x Bare Metal

Tue, 15 Nov 2016 00:00:00 +0000

I’ve found these talks really helpful in understanding what Kubernetes does and how it works from an architectural level:

A Technical Overview of Kubernetes - by Brendan Burns, formerly of Google.
Kubernetes: Changing the Way That we Think and Talk About Computing - by Brian Dorsey at Google.

“Hello World” with Kubernetes

I’d highly recommend starting your first cluster in Google Container Engine (which is Kubernetes under the hood) by following this guide Quickstart for Kubernetes on GCE and then moving on to deploying a sample app such as a simple PHP/Redis guestbook to get a feel for how to use and interact with Kubernetes at an introductory level. New accounts currently get $300 in credits, so this shouldn’t cost you anything for a good while.

Deploying with Kubespray

So now that we know what Kubernetes does and a few things about how apps and services get deployed, exposed, and scaled in Kubernetes, it’s time to build our own cluster. For that task, I’m choosing the Kubespray project. It follows a similar pattern to that of the Kolla project: clone a repo, edit your inventory and some vars in a file, then say “go”. Out pops a working cluster.

Requirements

What’s needed:

Some spare hardware on the same subnet (e.g. 172.22.10.x/24)
All systems running a recent Linux and Docker (e.g. Centos 7.x updated and Docker 1.12.3)
Kubespray cloned to a deployment host
Internet Access on all nodes
Firewall disabled on all nodes
SSH Key on a sudo-enabled account on all nodes
Ansible 2.x and python-netaddr installed on the deploy system
Centos 7 specific: TTY support for sudo in /etc/sudoers on all nodes

Deploy System

On my OSX system, I did the following:

git clone https://github.com/kubernetes-incubator/kargo
cd kargo
pip install -r requirements.txt
cp inventory/inventory.example inventory/hosts
Edited inventory/hosts to be:

[nodes]
mm1 ansible_ssh_host=172.22.10.51 ip=172.22.10.51 ansible_user=admin
mm2 ansible_ssh_host=172.22.10.52 ip=172.22.10.52 ansible_user=admin
mm3 ansible_ssh_host=172.22.10.53 ip=172.22.10.53 ansible_user=admin
smpc ansible_ssh_host=172.22.10.54 ip=172.22.10.54 ansible_user=admin
sm1 ansible_ssh_host=172.22.10.55 ip=172.22.10.55 ansible_user=admin
sm2 ansible_ssh_host=172.22.10.56 ip=172.22.10.56 ansible_user=admin
d1 ansible_ssh_host=172.22.10.57 ip=172.22.10.57 ansible_user=admin
d2 ansible_ssh_host=172.22.10.58 ip=172.22.10.58 ansible_user=admin

[kube-master]
mm1
mm2

[etcd]
mm1
mm2
mm3

[kube-node]
mm1
mm2
mm3
smpc
sm1
sm2
d1
d2

[k8s-cluster:children]
kube-node
kube-master

I decided to make all my nodes available for scheduling as “minions”, have two “masters” on mm1 and mm2, and have three etcd nodes on all three mac minis. Added up, this provides over 80 vCPUs and 300+GB of RAM all running on top of local SSDs.

Node Preparation

On each of the nodes, I ensured that

Defaults    !requiretty

was in my /etc/sudoers file.

I then ran:

systemctl stop firewalld
systemctl disable firewalld

to disable the firewall for now.

Kubespray

Finally, on the deployment host, I ran:

$ ansible-playbook -i inventory/hosts -b --become-user=root cluster.yml

About 15-20 mins later, I had a working Kubernetes v1.4.3 cluster on all my nodes. To verify, I SSHed into mm1 using ssh admin@172.22.10.51and ran:

$ kubectl get nodes
NAME      STATUS    AGE
d1        Ready     1d
d2        Ready     1d
mm1       Ready     1d
mm2       Ready     1d
mm3       Ready     1d
sm1       Ready     1d
sm2       Ready     1d
smpc      Ready     1d

and

$ kubectl cluster-info 
Kubernetes master is running at http://localhost:8080
dnsmasq is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/dnsmasq

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

and

$ kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}

Next Steps

At this point, I now have a working Kubernetes cluster with a 3-node etcd cluster (fully TLS enabled), 2 master nodes, and 8 total worker/minion nodes with almost zero additional configuration. This should hopefully set myself up for following the Multi-Node Kolla-Kubernetes guide for getting Kolla/Openstack going on a cluster. That, however, is for another day.

Back to Index

MaaS Part 4 - A Revised Approach

Thu, 28 Jul 2016 00:00:00 +0000

In part 3, I identified several weaknesses to my approach to this project as shown in part 1 and part 2. To summarize:

Lack of central inventory of systems and asset attributes
Hardcoded paths and configuration files/settings in Docker containers
Hardcoded DHCP Leases
Individually crafted Kickstart configuration files
SELinux is disabled on deploy
One version of one operating system supported
Logging from the containers

However, I kept finding things to add to that list as I looked back on it:

It wasn’t very “DRY”

The containers shared a lot of similarities in terms of starting base image and what I was doing. I repeated a bunch of shared code between the containers. Yuck. Seemed better that these could be running as a single container and save some overhead. That violates the light vs heavy containers rule, but my scaling needs are limited right now. I’m valuing simplicity and speed of development above the complexity and possibility to scale.
There was no testing of the container contents

Everything I was doing was manually tested. Hacker Dockerfile, run build, exec into it, validate by hand, rinse, repeat. It just felt hacky. Also, using purpose-built containers like php5.6-apache meant having to go in and break up the entrypoint/cmd. I had several issues overriding those and adding in my own bits successfully.
Inspecting the containers as I built it was tedious

See point 2 above. It was incredibly hard to know if what I was doing was achieving the desired result. The build process happens and you get an image. Hopefully you didn’t make a mistake and it now no longer runs. I did that quite a bit.
I was building on my deploy system directly

There was again a problem keeping track of what’s working and what’s not. I had to frequently clear the system of images and start the build over to validate the process.

Revised Approach with Ansible-Container

I started a new repo called https://github.com/bgeesaman/netpxemirror-ac and implemented a simple helper shell script, orc to help me with using Ansible-Container. I edited the ansible/container.yml and the ansible/main.yml to create a single container called maas from the phusion/baseimage:

container.yml

 1 version: "1"
 2 services:
 3   maas:
 4     image: phusion/baseimage:latest
 5     command: ['/sbin/my_init']
 6     ports:
 7       - "80:80"
 8       - "67:67/udp"
 9       - "69:69/udp"
10     environment:
11       TERM: vt100
12 registries: {}

In the main.yml, I found it necessary to keep the hosts: all section intact from the example and just edit the per-“host” steps. The name maas from container.yml becomes the “host” in main.yml. Taking a page from previous work with packer and Ansible before, I separated things into a role called maas. Note that the name has no relation to the “host”, but it needs to exist under the ansible directory in a roles folder.

main.yml

1 - hosts: all
2   gather_facts: false
3   tasks:
4     - raw: which python || apt-get update
5     - raw: (which python && which aptitude) || apt-get install -y python python-apt aptitude
6 - hosts: maas
7   tasks:
8   roles:
9     - { role: "maas" }

I performed an ansible-galaxy init inside the ansible/roles/maas directory and began editing my tasks/main.yml and my vars/main.yml.

Stepping Through the Role

When ansible uses a role, it uses the variables, files, templates, and plays contained in that role when running the tasks/main.yml play. This gives a nice structure to hold the files needed to build this container.

CAVEAT

I have to comment out the service isc-dhcp-server start entry in /etc/my_init.d/30_dhcp on my workstation in order to run/test locally. This is because the subnets in the dhcpd configuration file don’t match any local interfaces. Also, this container will give out leases to systems talking on the same subnet as my workstation when running/testing. This might not be what you want.

tasks/main.yml

 1 ---
 2 - name: Update Apt
 3   apt: upgrade=yes
 4 
 5 - name: Install packages into container
 6   apt: name="" state=installed
 7   with_items: ""
 8 
 9 - name: Configure Web Mirror
10   include: "tasks/mirror.yml"
11 
12 - name: Configure Mirror Sync
13   include: "tasks/sync.yml"
14 
15 - name: Configure TFTP
16   include: "tasks/tftp.yml"
17 
18 - name: Configure DHCP
19   include: "tasks/dhcp.yml"
20 
21 - name: Add final script on init
22   template:
23     src: start.sh.j2
24     dest: /etc/my_init.d/99_start
25     owner: root
26     group: root
27     mode: 0744

Because this is a Debian based image, updating apt is a common first step. Next, it installs some packages listed in vars/main.yml. Then, it walks through separate plays for installing the yum mirror pieces, the syncing pieces, the TFTP server, and the DHCP server. Finally, I drop in the last “init” script with some basic shell stuff that looks a lot like what I was doing in my entry scripts in my Dockerfiles.

Easing the Build/Test Process

I wrote a really quick shell script in the root of the repo called orc which is very rough right now and specific to my needs. However, it means I can do a ./orc build followed by an ./orc test followed by a ./orc deploy as needed. Here’s what actions orc provides at a glance:

orc build

Runs ansible-container build

$ ./orc build
No DOCKER_HOST environment variable found. Assuming UNIX socket at /var/run/docker.sock
Starting Docker Compose engine to build your images...
Attaching to ansible_ansible-container_1
Cleaning up Ansible Container builder...
Attaching to ansible_ansible-container_1, ansible_maas_1
ansible-container_1  | 
ansible-container_1  | PLAY [all] *********************************************************************
ansible-container_1  | 
ansible-container_1  | TASK [raw] *********************************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [raw] *********************************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | PLAY [maas] ********************************************************************
ansible-container_1  | 
ansible-container_1  | TASK [setup] *******************************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Update Apt] *******************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install packages into container] **********************************
ansible-container_1  | ok: [maas] => (item=[u'ca-certificates', u'wget', u'net-tools'])
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure Web Mirror] *********************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/mirror.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install Apache2] **************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install php7] *****************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable mod_rewrite] ***********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure Apache2] ************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable Apache2 at start] ******************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure Mirror Sync] ********************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/sync.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install repo packages] ********************************************
ansible-container_1  | ok: [maas] => (item=[u'createrepo', u'rsync'])
ansible-container_1  | 
ansible-container_1  | TASK [maas : Copy Mirror Crontab] **********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Copy Mirror Sync Script] ******************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure TFTP] ***************************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/tftp.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install TFTP] *****************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure TFTPd] **************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable TFTPd at start] ********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Template grub.cfg and ks.php files into place] ********************
ansible-container_1  | ok: [maas] => (item={u'src': u'grub.cfg-net.j2', u'dst': u'/root/grub.cfg-net'})
ansible-container_1  | ok: [maas] => (item={u'src': u'grub.cfg-i386-pc.j2', u'dst': u'/root/grub.cfg-i386-pc'})
ansible-container_1  | ok: [maas] => (item={u'src': u'grub.cfg-x86_64-efi.j2', u'dst': u'/root/grub.cfg-x86_64-efi'})
ansible-container_1  | ok: [maas] => (item={u'src': u'ks.php.j2', u'dst': u'/root/ks.php'})
ansible-container_1  | 
ansible-container_1  | TASK [maas : Make kickstart directory] *****************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Generate kickstarts] **********************************************
ansible-container_1  | changed: [maas] => (item={u'name': u'mp', u'mgmtip': u'172.22.10.50', u'ram': u'24', u'mgmtif': u'enp7s0f0', u'platform': u'mac', u'mgmtmac': u'00:23:32:2f:40:3c', u'cores': u'8', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'mm1', u'mgmtip': u'172.22.10.51', u'ram': u'16', u'mgmtif': u'enp1s0f0', u'platform': u'mac', u'mgmtmac': u'a8:20:66:34:ff:e9', u'cores': u'4', u'disk': u'250', u'arch': u'x86_64', u'mbr': u'sdb'})
ansible-container_1  | changed: [maas] => (item={u'name': u'mm2', u'mgmtip': u'172.22.10.52', u'ram': u'16', u'mgmtif': u'enp1s0f0', u'platform': u'mac', u'mgmtmac': u'a8:20:66:4a:ce:46', u'cores': u'4', u'disk': u'250', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'mm3', u'mgmtip': u'172.22.10.53', u'ram': u'16', u'mgmtif': u'enp1s0f0', u'platform': u'mac', u'mgmtmac': u'a8:20:66:4a:d9:da', u'cores': u'4', u'disk': u'250', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'smpc', u'mgmtip': u'172.22.10.54', u'ram': u'12', u'mgmtif': u'enp6s0', u'platform': u'pc', u'mgmtmac': u'00:30:48:fb:e2:44', u'cores': u'4', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'sm1', u'mgmtip': u'172.22.10.55', u'ram': u'96', u'mgmtif': u'enp11s0', u'platform': u'pc', u'mgmtmac': u'00:25:90:96:c4:9a', u'cores': u'16', u'disk': u'512', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'sm2', u'mgmtip': u'172.22.10.56', u'ram': u'96', u'mgmtif': u'enp11s0', u'platform': u'pc', u'mgmtmac': u'00:25:90:96:c6:5a', u'cores': u'16', u'disk': u'512', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'd1', u'mgmtip': u'172.22.10.57', u'ram': u'32', u'mgmtif': u'em1', u'platform': u'pc', u'mgmtmac': u'bc:30:5b:e5:73:b7', u'cores': u'4', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'd2', u'mgmtip': u'172.22.10.58', u'ram': u'32', u'mgmtif': u'em1', u'platform': u'pc', u'mgmtmac': u'bc:30:5b:e5:75:28', u'cores': u'4', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure DHCP] ***************************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/dhcp.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install repo packages] ********************************************
ansible-container_1  | ok: [maas] => (item=[u'isc-dhcp-server', u'grub-common', u'grub2-common', u'grub-imageboot', u'grub-pc-bin', u'grub-efi'])
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure DHCPD] **************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable DHCP server] ***********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Add final script on init] *****************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | PLAY RECAP *********************************************************************
ansible-container_1  | maas                       : ok=27   changed=1    unreachable=0    failed=0   
ansible-container_1  | 
ansible_ansible-container_1 exited with code 0
Aborting on container exit...
Stopping ansible_maas_1 ... done
Exporting built containers as images...
Committing image...
Exported netpxemirror-ac-maas with image ID sha256:516ae0de22da34d87a9726dca0a6d92b9602b9df3c757b4bc7e71c9fc1e6ec60
Cleaning up maas build container...
Cleaning up Ansible Container builder...

orc buildclean

Runs ansible-container build --from-scratch to rebuild from a clean starting point. Useful if large changes to packages and scripts are made in the role.

orc run

Runs ansible-container run locally. This variation has a while loop for the init script instead of your “production” init script. In my case, it takes the place of /sbin/my_init from the phusion/baseimage.

orc test

Finds the id of the running container, runs the chef/inspec docker container and attaches to it to run the test suite. Thanks to the Chef folks for making inspec so easy to use/install.

$ docker pull chef/inspec
$ did="$(docker ps -q --filter='name=ansible_maas_1')"
$ docker run -it --rm -v $(pwd):/share -v /var/run/docker.sock:/var/run/docker.sock chef/inspec exec spec/test.rb -t docker://$did
$ No DOCKER_HOST environment variable found. Assuming UNIX socket at /var/run/docker.sock
Attaching to ansible_ansible-container_1
Cleaning up Ansible Container builder...
Attaching to ansible_maas_1
maas_1               | *** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
maas_1               | *** Running /etc/my_init.d/10_apache2...
maas_1               |  * Starting Apache httpd web server apache2
maas_1               | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
maas_1               |  * 
maas_1               | *** Running /etc/my_init.d/20_tftpd...
maas_1               |  * Starting HPA's tftpd in.tftpd
maas_1               |    ...done.
maas_1               | *** Running /etc/my_init.d/30_dhcp...
maas_1               | *** Running /etc/my_init.d/99_start...
maas_1               | Netboot directory for i386-pc created. Configure your DHCP server to point to /nbi/boot/grub/i386-pc/core.0
maas_1               | Netboot directory for x86_64-efi created. Configure your DHCP server to point to /nbi/boot/grub/x86_64-efi/core.efi
maas_1               | *** Running /etc/rc.local...
maas_1               | *** Booting runit daemon...
maas_1               | *** Runit started as PID 75
maas_1               | Jul 28 19:34:32 4ca404c631e7 syslog-ng[80]: syslog-ng starting up; version='3.5.6'
maas_1               | Jul 28 19:39:01 4ca404c631e7 CRON[137]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
maas_1               | Jul 28 20:09:01 4ca404c631e7 CRON[1264]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
maas_1               | Jul 28 20:17:01 4ca404c631e7 CRON[1306]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
maas_1               | Jul 28 20:39:01 4ca404c631e7 CRON[1309]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
...................

Finished in 3.65 seconds (files took 2.16 seconds to load)
19 examples, 0 failures

orc deploy

A bit of custom stuff to send over the latest image to my deploy system in docker-compose format and starts it there.

$ ./orc deploy
* Deploying...
 - Syncing the latest
 521MiB 0:00:43 [12.1MiB/s] [                                                    <=>                                   ]
 - Copying compose file
 - Restarting compose app
Stopping admin_maas_1 ... done
Removing admin_maas_1 ... done
* Done.

What’s Better?

Well, this approach pretty much provides the same functionality as before, but it’s much easier to adjust/tweak and maintain. Here’s where things stand after moving to this method:

Lack of central inventory of systems and asset attributes
- All of that data is now stored in the maas roles’ vars/main.yml in a format that Ansible can parse/loop through
Hardcoded paths and configuration files/settings in Docker containers
- All of the key variables and path names have been separated into variables–again stored in the maas roles’ vars/main.yml. Now, changing names of files/directories is a trivial exercises as needs change.
Hardcoded DHCP Leases
- These are generated during the templating run by Jinja2 filters in the dhcpd.conf.j2 template stored in the maas templates folder and called by the dhcp.yml task/play.
Individually crafted Kickstart configuration files
- Using the variable storage method in the roles’ vars/main.yml, the ks.cfg.j2 template is used to generate all the per-host kickstart files.
SELinux is disabled on deploy
- Not yet addressed, but it should be a simpler debugging exercise now that it’s a single docker container running on deploy.
One version of one operating system supported
- Not yet addressed, but it’s easier to add support now that all configuration files are templated with Ansible/Jinja2.
Logging from the containers
- The phusion/baseimage runs a syslog daemon out of the box, but it currently is not sent anywhere.
It wasn’t very “DRY”
- Instead of three separate containers from 2 different FROM base images, it’s now a single debian-based container running 3 key services.
There was no testing of the container contents
- Adding inspec testing means I can now confidently add test coverage and perform testing during the build process in a few seconds instead of manually validating functionality.
Inspecting the containers as I built it was tedious
- The additional insight gained by using debugging features in Ansible and the logging as the play is being run means I can more quickly diagnose where something went wrong. Ansible does a decent job of capturing the error message when things go sideways, and it spits it out immediately.
I was building on my deploy system directly
- Using a bit of docker and ssh glued together with some docker-compose, I have the build and test processes running on my workstation and the final container running on the deploy system.

Back to Index

MaaS Part 3 - Remaining Issues

Tue, 26 Jul 2016 00:00:00 +0000

In part 1 and part 2, I’ve got the basis for a simple network-based hardware provisioning system. However, it has some issues that I’d like to list here to drive future improvement efforts.

Update: part 4 addresses several of these issues plus some new ones using Ansible-Container, some shell scripting, and Inspec.

Lack of central inventory of systems and asset attributes

There is no central place that defines an asset and its attributes like hostname, mac address, system type, CPU, RAM, Disks, etc. As Ansible is typically my go-to tool of choice for lightweight orchestration and configuration management, it seems logical to define these in an inventory file and vars file.
Hardcoded paths and configuration files/settings in Docker containers

While Docker containers help make what’s needed explicit in terms of configuration settings, it’s difficult to avoid embedding paths that change per deployment. What if I don’t like my TFTP root as /nbi? Right now, it’d be very painful to change that path and re-test everything. I’m curious to see if Ansible-Container helps take the sting out of making templated configuration files inside Docker containers.
Hardcoded DHCP Leases

Because I have a fixed set of hardware, this is a one-time effort. However, it fails to scale at any size beyond where it is now. These static reservations should be generated from the asset system or leases be dynamic and have the IP updates tied into DNS directly. A quick looping Ansible playbook to generate these from the inventory/vars files could do the trick.
Individually crafted Kickstart configuration files

While 95% of each kickstart file is identical, templates with a couple variables from the asset system generating the final kickstart files would be better. Another quick looping Ansible playbook to generate these from the inventory/vars files would solve this.
SELinux is disabled on deploy

This was a matter of avoiding very difficult to debug issues during rapid iteration. It’s not “production ready” unless it has an SELinux policy that aligns with the needs of the app, and this one just needs some time and elbow grease.
One version of one operating system supported

I’m standardizing on Centos 7.x here, but an eventuality will be to support newer releases as time goes on. Assuming all the other issues are handled better with Ansible roles/playbooks, this problem most likely becomes much easier, too.
Logging from the containers

I want to know what’s happening in terms of DHCP giving out leases, TFTP files downloaded, Kickstart configs downloaded, and general errors when they happen. This will require more investigation into best practices.

Back to Index

MaaS Part 2 - Centos 7.x and Kickstart

Tue, 26 Jul 2016 00:00:00 +0000

In part 1, I walked through the initial network booting process to be able to start the Centos 7.x installation process. In this part, part 2, I’ll continue on to explain the Centos 7.x specific portions of things as well as break down the kickstart configuration file.

Centos 7 Specifics

In the grub.cfg file, the following kernel line:

  linux (tftp)/vmlinuz noipv6 inst.repo=http://deploy/repos/centos/7/os/x86_64/ inst.ks=http://deploy/ks.cfg inst.ks.sendmac

has three RedHat/Centos specific items under the inst prefix. Full reference

inst.repo

This option tells the installer (anaconda) where it can find its installation repository. In this case, the web root of our centos7mirror container.

inst.ks

This option tells anaconda where it can find the optional kickstart configuration file. Again in this case, the web root of our centos7mirror container.

inst.ks.sendmac

This option tells anaconda to send the NIC name and mac address of the interface used to request the kickstart inside as special HTTP header named X-RHN-Provisioning-MAC-0. e.g.

X-RHN-Provisioning-MAC-0: eth0 01:23:45:67:89:ab

Custom Kickstart Handler

So, in our /var/www/html/ks.php file, we parse the MAC header and serve up our custom kickstart file if it’s valid and exists as a configuration file. Remember, we use mod_rewrite rules to make any requests for ks.cfg be served by /var/www/html/ks.php:

 1 <?php
 2   # kickstart needs text/plain or it fails
 3   header("Content-Type: text/plain");
 4 
 5   # Cfg file path
 6   $cfgpath = "./cfg";
 7 
 8   # Did we get the inst.ks.sendmac header
 9   if (isset($_SERVER['HTTP_X_RHN_PROVISIONING_MAC_0'])) {
10     # Parse the header
11     $macstring = $_SERVER['HTTP_X_RHN_PROVISIONING_MAC_0'];
12     $macarray = explode(" ", $macstring);
13 
14     # Did the header contain a mac address?
15     if (isset($macarray[1])) {
16       $mac = $macarray[1];
17 
18       # Valid mac format?
19       if (preg_match('/([a-fA-F0-9]{2}:?){6}/', $mac) == 0) {
20         echo "Invalid Request\n";
21         exit;
22       }
23 
24       # Send its config if one exists
25       $configfile = "$cfgpath/$mac.cfg";
26       if (file_exists($configfile)) {
27         readfile($configfile);
28       } else {
29         echo "Invalid Request\n";
30       }
31     }
32   } else {
33     echo "Invalid Request\n";
34     exit;
35   }
36 ?>

And here is what’s in /var/www/html/cfg:

[admin@deploy cfg]$ pwd
/var/www/html/cfg
[admin@deploy cfg]$ ls -al
total 40
drwxr-xr-x 2 root root 4096 Jul 25 15:04 .
drwxr-xr-x 4 root root   58 Jul 26 01:00 ..
-rw-r--r-- 1 root root 1814 Jul 25 15:04 00:23:32:2f:40:3c.cfg
-rw-r--r-- 1 root root 1819 Jul 25 15:04 00:25:90:96:c4:9a.cfg
-rw-r--r-- 1 root root 1819 Jul 25 15:04 00:25:90:96:c6:5a.cfg
-rw-r--r-- 1 root root 1808 Jul 25 15:04 00:30:48:fb:e2:44.cfg
-rw-r--r-- 1 root root 1811 Jul 25 15:04 a8:20:66:34:ff:e9.cfg
-rw-r--r-- 1 root root 1811 Jul 25 15:04 a8:20:66:4a:ce:46.cfg
-rw-r--r-- 1 root root 1811 Jul 25 15:04 a8:20:66:4a:d9:da.cfg
-rw-r--r-- 1 root root 1810 Jul 25 15:04 bc:30:5b:e5:73:b7.cfg
-rw-r--r-- 1 root root 1810 Jul 25 15:04 bc:30:5b:e5:75:28.cfg

Custom Kickstart File

 1 #version=DEVEL
 2 # System authorization information
 3 auth --enableshadow --passalgo=sha512
 4 # Use CDROM installation media
 5 url --url http://deploy/repos/centos/7/os/x86_64/
 6 # Use graphical install
 7 #graphical
 8 text
 9 # Run the Setup Agent on first boot
10 firstboot --enable
11 ignoredisk --only-use=sdb
12 # Keyboard layouts
13 keyboard --vckeymap=us --xlayouts='us'
14 # System language
15 lang en_US.UTF-8
16 
17 # Network information
18 network  --bootproto=dhcp --device=enp2s0 --noipv6 --activate
19 network  --hostname=mm1
20 
21 # During Install
22 sshpw --user=root s3cr3t
23 
24 # Root password
25 rootpw --iscrypted $6$SANKK2GLJES9hSp9$PgaXUwpz2S4mRwg6kUQo.E3IyArxeA.MdIc77e1kkPRDD3BI8d3mxrwmoD.8BbZ2613XJoZNgWX0fwCsu1tih.
26 # System timezone
27 timezone America/New_York --isUtc
28 user --groups=wheel --name=admin --password=$6$0YFXS7zQLIQ99FCh$gqxId0ypSP16GCwiOfOcGUZK9AASxatDVF4g8PtTy5HDaZ5jXeIo9NeOU4ttK1KWjsI/6D6TVMfOh37XCMSG2/ --iscrypted --gecos="admin"
29 # System bootloader configuration
30 bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=sdb
31 autopart --type=lvm
32 # Partition clearing information
33 clearpart --all --initlabel --drives=sda,sdb
34 # Reboot after install finishes
35 reboot
36 
37 %packages
38 @^minimal
39 @core
40 kexec-tools
41 net-tools
42 tcpdump
43 wget
44 nc
45 %end
46 
47 %addon com_redhat_kdump --enable --reserve-mb='auto'
48 
49 %end
50 
51 %post --log=/root/postinstall.log
52 # SSH UseDNS no
53 sed -ri 's/#UseDNS yes/UseDNS no/'g /etc/ssh/sshd_config
54 # SSH Key
55 mkdir -p /home/admin/.ssh
56 chmod 700 /home/admin/.ssh
57 echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCugz/eu/a6U5ZXtYVp9ufCghVSP0Lux/nP6BgbranKE6r3h3xgqo8yR2LUG9VwH6Vo+BtGgToeww+jbgr3oq9g8/mQmNvQEefvWyzhYxrpv6q36fdYS0KhQxA6vnpOZZ1M9cZ1q6iPaUryxDUFU3HULDKM4g/6XBzqBJZ2illyuw==" > /home/admin/.ssh/authorized_keys
58 chmod 600 /home/admin/.ssh/authorized_keys
59 chown -R admin:admin /home/admin/.ssh
60 
61 yum -y update
62  
63 %end

A couple key notes:

The file was created by first performing a manual installation. This ends up as /root/anaconda-ks.cfg after installation completes. I then added a couple packages and then the contents of the post section.
All partitions are cleared and the default partitioning scheme is used
A default account of admin is made
A public SSH key for admin is added to enable access once booted
UseDNS no is changed to speed up SSH access by removing reverse DNS lookups on connection.

Back to Index

MaaS Part 1 - Netboot Macs and PXEBooting PCs from Linux

Tue, 26 Jul 2016 00:00:00 +0000

When building a “Metal as a Service” (MAAS) layer, the idea is to make it very easy to go from “box to rack to ready” for a large number of servers. To make this easier and to reduce complexity at this layer, most organizations purchase one or more sets of similar servers from a single manufacturer. Depending on the level of service, they may have the manufacturer/vendor prepare the systems a certain way before they show up at the door. A technician can just unbox them, inventory the asset, slide them into the rack, cable it up, and be done with their part.

The goal is that the provisioning system picks up from the power-on event from that point on. It gets a base operating system followed by some configuration management to put it into a production state. When that system is no longer being used in that production state, it goes through a similar re-provisioning process to become useful again for another application.

Naturally, this can get out of hand without proper tooling once you go past, say, 10 servers. I’ve been doing a fair bit of reading about this, and Digital Rebar looks like the best of breed for the power and customization aspects. It’s the latest from the folks that developed Crowbar.

So why am I not going to use Digital Rebar?

Well, two reasons. I know that learning comes best from experiencing the pain firsthand. So instead of just dropping in a very impressive toolset that does 500 things more than I need, I wanted to get familiar with the handful of things I do need in a focused way. I could spend a few days installing and configuring Digital Rebar and learning the way it solves the problem while relearning all that forgotten PXEBoot knowledge from 10 years ago, or I could choose to do one thing at a time and truly understand the simplest use case.

Secondly, I have a mix of systems, and I don’t believe my use case is baked into Digital Rebar. Namely, Mac Netboot support alongside PC PXEboot support. Until I got it working, I didn’t even know if it was possible. As far as I can google, it’s not been documented anywhere public. Maybe these posts will spark a discussion of working that support into a tool like Digital Rebar, but I’m not sure of the popularity of it. I mean, most folks want to Netboot OSX on a Mac or at least dual boot with Linux, not just Linux alone.

The Problem

So, how do I get Macs and PC servers to go from bare metal to a common base Centos 7.x latest state with a known IP configuration, hostname, and SSH access booting just Linux?

Create a deploy host.
Install and configure a DHCP server, a TFTP server, and a Webserver
Custom configuration
Walking Through the Process

Deploy Host

I am using a low-power Celeron system as my base Centos 7.x system from which to develop, test, and provide netboot and pxeboot support. I had downloaded the minimal ISO from here, ran dd to write it directly to a USB, and did a manual installation of the base packages.

Finally, I gave it a static IP of 172.22.10.2/24 on the management network, added a DNS entry for the hostname called deploy on my pfsense system (172.22.10.1), and generated a new SSH keypair.

Now, from my workstation, I can SSH into the deploy system on the management network that my servers have their primary NIC attached, build, and run the containers.

Supporting Services

I’ll admit I spent over a week getting Netboot to work from a Linux DHCP server trying several angles. I tried very hard to use ipxe in several ways, but never got very far. Oddly enough, I spent only 2 hrs adding PXEBoot support to that same DHCP/TFTP setup for the PCs.

I owe all I learned about this process from:

Thank you to those above who paved the way. I will take a smidge of credit for tying this all together in terms of grub2 booting over the network and figuring out the EFI stuff, but the heavy lifting and the DHCP Netboot trickery they did was pretty impressive.

Basic PXE/NetBoot Process

My hope is that understanding the boot process along with reading the comments in the handful of configuration files in the Docker containers will make things clear for anyone else attempting this.

Here’s the basic, high level process of PXE/Netbooting.

System is booted from a NIC that supports PXEbooting. This typically means changing the boot order to have the NIC first, hitting a key to network boot just this time, or configuring a BMC or ILOM card to boot the system from the network.
The system acquires an IP address via DHCP/Bootp. A properly configured DHCP server will respond with an IP, subnet, gateway, and file to boot from.
Optionally, the system also asks for a pointer to a file to boot from over the network.
If supplied by the DHCP server, the system uses the next-server and filename results to TFTP download/run that bootable file.
From there, the bootable file controls the remainder of the boot process (install an OS, run an OS from memory, etc). Often, this means hosting via a web server or FTP server the configuration for installation and even a repository of all the installation packages for the OS.

Back to Configuring all this stuff

I originally got this working with the version of isc-dhcp-server, tftpd, and httpd from the Centos7.x repos installed natively on deploy, but all the working and non-working config files, log files, and such were cluttering things up. It was very difficult to iterate knowing what files were being used or not.

So, of course, I decided to address that by dockerizing those services, wiping the deploy system, and redeploying the same services from pure docker containers. By recreating all my work and encapsulating it properly into Docker containers, I have a way to share my work that folks may be able to understand.

This also means my deploy system really only needs to run Docker and the knowledge to build and run the containers from scratch can come from a git repo.

On the deploy system, it’s installation was pretty straightforward. I followed the Centos 7.x guide for getting Docker up and running at boot. I use an account named admin with sudo privileges on my deploy host, so I added admin to the docker group to avoid having to type sudo on every docker command.

What needs to be dockerized:

DHCP - isc-dhcp-server - Provide IPs and pointers to TFTP boot files
TFTP - tftpd-hpa - Provide bootable grub2 images and grub.cfg files
Web - apache2 - Provide Kickstart configuration files and a local Centos 7.x mirror
Rsync - createrepo and rsync - Run on cron to keep the mirror updated each night

Right away, we know that this can get messy really quickly. Custom DHCP configs, TFTP roots, Web roots and configurations, and of course, the 28GB of stuff in the Centos 7.x Repo in that web root. Thankfully, the process of Dockerizing forces things to be explicitly handled properly.

Here is that repository with those services broken up into three separate containers: https://github.com/bgeesaman/netpxemirror

On my deploy system, after installing Docker, I ran:

$ cd ~
$ git clone https://github.com/bgeesaman/netpxemirror.git
Cloning into 'netpxemirror'...
remote: Counting objects: 50, done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 50 (delta 11), reused 47 (delta 11), pack-reused 0
Unpacking objects: 100% (50/50), done.
$ cd netpxemirror
### Configure the services to your environment ###
$ ./build.sh
$ ./run.sh

This built and ran netpxeboot, centos7mirror, and mirrorsync.

WARNING

Running the mirrorsync container for the first time will immediately begin rsyncing 28GB of data into /var/www/html/repos! I did this so that I could restore the state of a working deploy system from scratch assuming there was no local repo already. Edit the mirrorsync/files/start.sh to prevent this behavior before running build.sh.

Assuming the configuration is correct, a docker ps shows:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                NAMES
9bcdd7f44d38        netpxeboot          "/root/start.sh"       14 hours ago        Up 14 hours                              netpxeboot
6a1c28fdfc7a        mirrorsync          "/root/start.sh"       15 hours ago        Up 15 hours                              mirrorsync
eebd43890189        centos7mirror       "apache2-foreground"   23 hours ago        Up 23 hours         0.0.0.0:80->80/tcp   centos7mirror

Keen observers will notice that TCP port 80 is forwarded to the centos7mirror container running apache2, but UDP/67 and UDP/69 are not for netpxeboot. That’s because the netpxeboot container is started with --net=host in the docker run command so that it can bind to the host’s network stack and see broadcasts from the systems asking for IP addresses.

Containers at a Glance

centos7mirror

Based on the php:5.6-apache container, I add a custom docker-php.conf to apache and enable mod_rewrite. This makes this a PHP 5.6 webserver with the ability to serve up dynamic content based on the request data and to make the URLs cleaner.

mirrorsync

Based on phusion/baseimage, this Ubuntu based system provides a simple cron based functionality to keep the Yum repository in sync with the Centos 7.x mirror of my choice.

netpxeboot

Also based on phusion/baseimage, this Ubuntu based system provides the isc-dhcp-server, tftpd-hpa server, and the configuration files for grub booting over TFTP. A PHP script for serving custom kickstart files out of the webroot in centos7mirror based on mac address is also here.

Custom Configuration

Now, there’s probably a 100% chance that the containers didn’t work for you if you simply git cloned them and tried building them. So, I’ll point you to each of the places that you’ll want to edit for your needs. Namely, the IP addresses, paths, and URLs for things. I’ll start with the local mirror and mirror syncing process.

centos7mirror

Realistically, there is very little to configure here with the exception of the web root and the name of the file you want associated with serving kickstart configurations. I chose /var/www/html since it’s the default for the web root. Also, the ks.cfg mod_rewrite rule actually calls the /var/www/html/ks.php file for knowing what kickstart to serve up. ks.php is put into place by netpxeboot on its first run.

mirrorsync

If you edited the web root path, you’ll need to search/replace it in all the files in the files directory. Mostly, though, you’ll want to edit the cron.sh file to pull from another repo and crontab to adjust the schedule of when the sync happens.

11 1 * * * root /root/cron.sh >> /var/log/cron.log 2>&1

Means it runs on the eleventh minute of the first hour of every day. I tend to use slighty “off” times to avoid contention with other folks who run their jobs on the hour exactly.

netpxeboot

The files ADDed in the Dockerfile into this container are the primary points of configuration, but I’ll warn you that making one incorrect change here can easily break things.

ks.php serves up kickstart files in /var/www/html/cfg/ma:ca:dd:re:ss.cfg based on the mac address sent in the header as configured from the grub.cfg option called inst.ks.sendmac

tftpd-hpa specifies mostly the base path of the TFTP files. In my case, /nbi

dhcpd.conf is where the PXE/Netbooting magic happens. Reserved IPs, lease times and ranges, paths, and more go here. If you don’t have PCs, remove the pc class block. If you don’t have Macs, remove the entire netboot class block.

  1 # dhcpd.conf
  2 
  3 option domain-name "example.org";
  4 option domain-name-servers ns1.example.org, ns2.example.org;
  5 default-lease-time 3600;
  6 max-lease-time 7200;
  7 ddns-update-style none;
  8 ddns-updates off;
  9 ignore client-updates;
 10 allow booting;
 11 allow bootp;
 12 authoritative;
 13 log-facility local7;
 14 boot-unknown-clients on;
 15 ping-check off;
 16 allow-unknown-clients;
 17 allow-known-clients;
 18 
 19 subnet 172.22.10.0 netmask 255.255.255.0 {
 20   range 172.22.10.5 172.22.10.30;
 21   range dynamic-bootp 172.22.10.31 172.22.10.49;
 22   allow booting;
 23   allow bootp;
 24   option domain-name-servers 172.22.10.1; 
 25   option domain-name "lonimbus.com";
 26   option routers 172.22.10.1;
 27   option broadcast-address 172.22.10.255;
 28   default-lease-time 6000;
 29   max-lease-time 7200;
 30 }
 31 host mp {
 32   hardware ethernet 00:23:32:2f:40:3c;
 33   fixed-address 172.22.10.50;
 34 }
 35 host mm1 {
 36   hardware ethernet a8:20:66:34:ff:e9;
 37   fixed-address 172.22.10.51;
 38 }
 39 host mm2 {
 40   hardware ethernet a8:20:66:4a:ce:46;
 41   fixed-address 172.22.10.52;
 42 }
 43 host mm3 {
 44   hardware ethernet a8:20:66:4a:d9:da;
 45   fixed-address 172.22.10.53;
 46 }
 47 host smpc {
 48   hardware ethernet 00:30:48:fb:e2:44;
 49   fixed-address 172.22.10.54;
 50 }
 51 host sm1 {
 52   hardware ethernet 00:25:90:96:c4:9a;
 53   fixed-address 172.22.10.55;
 54 }
 55 host sm2 {
 56   hardware ethernet 00:25:90:96:c6:5a;
 57   fixed-address 172.22.10.56;
 58 }
 59 host d1 {
 60   hardware ethernet bc:30:5b:e5:73:b7;
 61   fixed-address 172.22.10.57;
 62 }
 63 host d2 {
 64   hardware ethernet bc:30:5b:e5:75:28;
 65   fixed-address 172.22.10.58;
 66 }
 67 
 68 class "pc" {
 69   match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
 70     filename "boot/grub/i386-pc/core.0";
 71     next-server 172.22.10.2;
 72 }
 73 class "netboot" {
 74     match if substring (option vendor-class-identifier, 0, 9) = "AAPLBSDPC";
 75     option dhcp-parameter-request-list 1,3,17,43,60;
 76 
 77     if (option dhcp-message-type = 1) {
 78         option vendor-class-identifier "AAPLBSDPC";
 79         option vendor-encapsulated-options
 80             08:04:81:00:00:89;    # bsdp option 8 (length 04) -- selected image id;
 81     } elsif (option dhcp-message-type = 8) {
 82         option vendor-class-identifier "AAPLBSDPC";
 83         if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
 84             log(debug, "bsdp_msgtype_list");
 85 
 86             # bsdp image list message:
 87             # one image, plus one default image (both are the same)
 88             option vendor-encapsulated-options 
 89                 01:01:01:                              # bsdp_msgtype_list
 90                 04:02:                                 # bsdp option code 4 (length 02) server priority
 91                     80:00:                             #  Priority (32768)
 92                 07:04:                                 # bsdp option code 7 (length 04) default image id
 93                     81:00:00:89:                       #  Image ID (137)
 94                 09:0e:                                 # bsdp option code 9 (length 0e) image list
 95                     81:00:00:89:                       #  Image ID (137)
 96                         09:54:68:65:2d:49:6d:61:67:65; #   Name (length 09) 'The-Image'
 97         } else {
 98             log(debug, "bspd_msgtype_select");
 99 
100             # details about the selected image
101             option vendor-encapsulated-options
102                 01:01:02:                       # bsdp_msgtype_select 
103                 08:04:                          # bsdptag_selected_boot_image
104                     81:00:00:89:                #  Image ID (137)
105                 82:09:                          # Machine Name (length 09)
106                     54:68:65:2d:49:6d:61:67:65; #  'The-Image'
107 
108             if (substring(option vendor-class-identifier, 10, 4) = "i386") {
109                 filename "mactel64.efi";
110                 next-server 172.22.10.2;
111             }
112         }
113     }
114 }

grub.cfg-net gets baked into the mactel64.efi file served up for macs

 1 insmod efi_gop
 2 insmod efi_uga
 3 insmod video_bochs
 4 insmod video_cirrus
 5 insmod all_video
 6 set gfxpayload=keep
 7 insmod gzio
 8 insmod part_gpt
 9 insmod ext2
10 
11 insmod net
12 insmod efinet
13 insmod tftp
14 
15 net_bootp efinet0
16 
17 set net_default_server=172.22.10.2
18 
19 configfile (tftp)/boot/grub/x86_64-efi/grub.cfg-01-$net_efinet0_mac

grub.cfg-* gets pulled via TFTP depending on architecture by the grub2 image first pulled via DHCP. Notice the URLs, IPs, and paths to ks.cfg here.

 1 set default="0"
 2 
 3 insmod video_bochs
 4 insmod video_cirrus
 5 insmod all_video
 6 set gfxpayload=keep
 7 insmod gzio
 8 insmod part_gpt
 9 insmod ext2
10 
11 set timeout=10
12 
13 menuentry 'Unattended Centos 7 Install' --class os {
14      insmod net
15      insmod tftp
16      # TFTP server
17      set net_default_server=172.22.10.2
18 
19      echo 'Network status: '
20      net_ls_cards
21      net_ls_addr
22      net_ls_routes
23 
24      echo 'Loading Linux ...'
25      #linux (tftp)/vmlinuz noipv6 inst.repo=http://centos.aol.com/7/os/x86_64/ inst.ks=http://172.22.10.2/ks.cfg inst.ks.sendmac
26      linux (tftp)/vmlinuz noipv6 inst.repo=http://deploy/repos/centos/7/os/x86_64/ inst.ks=http://deploy/ks.cfg inst.ks.sendmac
27      echo 'Loading initial ramdisk ...'
28      initrd (tftp)/initrd.img
29 }

start.sh the entry point of the container. Lots of one-time setup steps here necessary for successful operation. Change paths/URLs carefully here.

 1 #!/bin/bash
 2 
 3 set -euo pipefail
 4 
 5 TFTPPATH="/nbi"
 6 
 7 if [ ! -d /var/www/html/cfg ]; then
 8   mkdir -p /var/www/html/cfg
 9   cp -R /root/cfg/* /var/www/html/cfg/
10 fi
11 if [ ! -f /var/www/html/index.html ]; then
12   cp /root/index.html /var/www/html/index.html
13   chmod 644 /var/www/html/index.html
14 fi
15 if [ ! -f /var/www/html/ks.php ]; then
16   cp /root/ks.php /var/www/html/ks.php
17   chmod 644 /var/www/html/ks.php
18 fi
19 
20 if [ ! -f $TFTPPATH/grub.cfg-net ]; then
21   cp /root/grub.cfg-net $TFTPPATH/grub.cfg-net
22   chmod 644 $TFTPPATH/grub.cfg-net
23 fi
24 
25 # Generate mactel boot efi
26 if [ ! -f $TFTPPATH/mactel64.efi ]; then
27   grub-mkimage -d /usr/lib/grub/x86_64-efi/ -O x86_64-efi -o $TFTPPATH/mactel64.efi -p '(tftp)/boot/grub' -c $TFTPPATH/grub.cfg-net normal configfile net efinet tftp efi_gop efi_uga all_video gzio part_gpt ext2 echo linuxefi
28 fi
29 if [ ! -d $TFTPPATH/boot/grub ]; then
30   grub-mknetdir --net-directory=$TFTPPATH
31 fi
32 
33 cp /root/grub.cfg-i386-pc $TFTPPATH/boot/grub/i386-pc/grub.cfg
34 cp /root/grub.cfg-i386-pc $TFTPPATH/boot/grub/grub.cfg
35 cp /root/grub.cfg-x86_64-efi $TFTPPATH/boot/grub/x86_64-efi/grub.cfg
36 
37 if [ ! -f $TFTPPATH/initrd.img ]; then
38   wget --quiet -O $TFTPPATH/initrd.img http://centos.aol.com/7/os/x86_64/isolinux/initrd.img
39 fi
40 if [ ! -f $TFTPPATH/vmlinuz ]; then
41   wget --quiet -O $TFTPPATH/vmlinuz http://centos.aol.com/7/os/x86_64/isolinux/vmlinuz
42 fi
43 
44 # Start dhcp and tftpd
45 /etc/init.d/isc-dhcp-server start
46 /etc/init.d/tftpd-hpa start
47 
48 echo "Tailing logs..."
49 tail -f /var/log/syslog

Walking Through the Process

Step 1:

Boot the system from the network. My Dells and SuperMicros use f12 and the Macs have you hold n right at and slightly after the boot chime. Helps to have a keyboard and mouse on these systems as you iterate.

Step 2:

On the deploy system, you can run: tcpdump -ni eth0 not port 22 to watch the process unfold. This is my primary method of debugging along with watching the screen as it boots. You’ll see the system request an address via DHCP. If your DHCP server is configured and listening correctly, you’ll see it respond with a DHCP reply. On your PC system, you should see something indicating it got an IP. Macs just show the blinking globe by default unless you enable verbose booting.

Step 3:

Very shortly after the DHCP lease is acquired, you’ll see a TFTP RRQ for the boot/grub/i386-pc/core.0 or mactel64.efi binary. Both of them should initialize grub on the system to a point that it can look for its proper grub.cfg file on that same TFTP server. In that tcpdump output, you’ll see the file grabs over UDP/69 and the paths/names requested. You can narrow things down to only what’s being pulled via TFTP by running tcpdump -ni eth0 port 69 if you are having pathing or file name issues.

Step 4:

After it grabs the second grub.cfg via TFTP, it follows the instructions contained in it. In our case, it’s to load the vmlinuz and initrd.img from TFTP (again) and to boot from them with some additional options.

Step 5:

inst.repo and inst.ks are the custom boot options used here to specify where to get the base installation packages and the installation configuration files, respectively. Both are served via apache out of the centos7mirror container.

Step 6:

From this point, it’s a normal Kickstart and Centos 7.x installation process.

Back to Index

Speaking at KubeCon 2017 - Part 4 - The Talk

Post Series Navigation:

More Visuals Needed!

Preparation

The Day Of

Parting Thoughts

Post Series Navigation:

Speaking at KubeCon 2017 - Part 3 - First Run

Post Series Navigation:

The Dry Run

Initial Reception

Post Series Navigation:

Speaking at KubeCon 2017 - Part 2 - Research

Post Series Navigation:

How I Approached the Problem

Reaching Out to the Community

Scope Creep

Post Series Navigation:

Speaking at KubeCon 2017 - Part 1 - The CFP Submission

Post Series Navigation:

The Desire to Speak

Choosing the Topic

Writing the CFP Submission

Peer Review

The Waiting Game

Post Series Navigation:

Installing Kubernetes on Baremetal via CoreOS Tectonic with Grub Booting

Architecture Overview

Network Diagram

Detailed PXE/Netboot to CoreOS Installation Flow

Provisioning Infrastructure Configuration

Setting up deploy.lab

Installing and Configuring the Matchbox Container on deploy.lab

Installing and Configuring the DNSMasq (DNS, DHCP, TFTP, Grub) Container on deploy.lab

Running the Tectonic Installer

CaaS Part 1 - Kubernetes on Centos 7.x Bare Metal

“Hello World” with Kubernetes

Deploying with Kubespray

Requirements

Deploy System

Node Preparation

Kubespray

Next Steps

MaaS Part 4 - A Revised Approach

It wasn’t very “DRY”

There was no testing of the container contents

Inspecting the containers as I built it was tedious

I was building on my deploy system directly

Revised Approach with Ansible-Container

container.yml

main.yml

Stepping Through the Role

CAVEAT

tasks/main.yml

Easing the Build/Test Process

orc build

orc buildclean

orc run

orc test

orc deploy

What’s Better?

MaaS Part 3 - Remaining Issues

Lack of central inventory of systems and asset attributes

Hardcoded paths and configuration files/settings in Docker containers

Hardcoded DHCP Leases

Individually crafted Kickstart configuration files

SELinux is disabled on deploy

One version of one operating system supported

Logging from the containers

MaaS Part 2 - Centos 7.x and Kickstart

Centos 7 Specifics

inst.repo

inst.ks

inst.ks.sendmac

Custom Kickstart Handler

Custom Kickstart File

MaaS Part 1 - Netboot Macs and PXEBooting PCs from Linux

The Problem

Deploy Host

Supporting Services

Setting up `deploy.lab`

Installing and Configuring the Matchbox Container on `deploy.lab`

Installing and Configuring the DNSMasq (DNS, DHCP, TFTP, Grub) Container on `deploy.lab`

SELinux is disabled on `deploy`