MaaS Part 4 - A Revised Approach

In part 3, I identified several weaknesses to my approach to this project as shown in part 1 and part 2. To summarize:

  1. Lack of central inventory of systems and asset attributes
  2. Hardcoded paths and configuration files/settings in Docker containers
  3. Hardcoded DHCP Leases
  4. Individually crafted Kickstart configuration files
  5. SELinux is disabled on deploy
  6. One version of one operating system supported
  7. Logging from the containers

However, I kept finding things to add to that list as I looked back on it:

  1. It wasn’t very “DRY”

    The containers shared a lot of similarities in terms of starting base image and what I was doing. I repeated a bunch of shared code between the containers. Yuck. Seemed better that these could be running as a single container and save some overhead. That violates the light vs heavy containers rule, but my scaling needs are limited right now. I’m valuing simplicity and speed of development above the complexity and possibility to scale.

  2. There was no testing of the container contents

    Everything I was doing was manually tested. Hacker Dockerfile, run build, exec into it, validate by hand, rinse, repeat. It just felt hacky. Also, using purpose-built containers like php5.6-apache meant having to go in and break up the entrypoint/cmd. I had several issues overriding those and adding in my own bits successfully.

  3. Inspecting the containers as I built it was tedious

    See point 2 above. It was incredibly hard to know if what I was doing was achieving the desired result. The build process happens and you get an image. Hopefully you didn’t make a mistake and it now no longer runs. I did that quite a bit.

  4. I was building on my deploy system directly

    There was again a problem keeping track of what’s working and what’s not. I had to frequently clear the system of images and start the build over to validate the process.

Revised Approach with Ansible-Container

I started a new repo called https://github.com/bgeesaman/netpxemirror-ac and implemented a simple helper shell script, orc to help me with using Ansible-Container. I edited the ansible/container.yml and the ansible/main.yml to create a single container called maas from the phusion/baseimage:

container.yml

 1 version: "1"
 2 services:
 3   maas:
 4     image: phusion/baseimage:latest
 5     command: ['/sbin/my_init']
 6     ports:
 7       - "80:80"
 8       - "67:67/udp"
 9       - "69:69/udp"
10     environment:
11       TERM: vt100
12 registries: {}

In the main.yml, I found it necessary to keep the hosts: all section intact from the example and just edit the per-“host” steps. The name maas from container.yml becomes the “host” in main.yml. Taking a page from previous work with packer and Ansible before, I separated things into a role called maas. Note that the name has no relation to the “host”, but it needs to exist under the ansible directory in a roles folder.

main.yml

1 - hosts: all
2   gather_facts: false
3   tasks:
4     - raw: which python || apt-get update
5     - raw: (which python && which aptitude) || apt-get install -y python python-apt aptitude
6 - hosts: maas
7   tasks:
8   roles:
9     - { role: "maas" }

I performed an ansible-galaxy init inside the ansible/roles/maas directory and began editing my tasks/main.yml and my vars/main.yml.

Stepping Through the Role

When ansible uses a role, it uses the variables, files, templates, and plays contained in that role when running the tasks/main.yml play. This gives a nice structure to hold the files needed to build this container.

CAVEAT

I have to comment out the service isc-dhcp-server start entry in /etc/my_init.d/30_dhcp on my workstation in order to run/test locally. This is because the subnets in the dhcpd configuration file don’t match any local interfaces. Also, this container will give out leases to systems talking on the same subnet as my workstation when running/testing. This might not be what you want.

tasks/main.yml

 1 ---
 2 - name: Update Apt
 3   apt: upgrade=yes
 4 
 5 - name: Install packages into container
 6   apt: name="" state=installed
 7   with_items: ""
 8 
 9 - name: Configure Web Mirror
10   include: "tasks/mirror.yml"
11 
12 - name: Configure Mirror Sync
13   include: "tasks/sync.yml"
14 
15 - name: Configure TFTP
16   include: "tasks/tftp.yml"
17 
18 - name: Configure DHCP
19   include: "tasks/dhcp.yml"
20 
21 - name: Add final script on init
22   template:
23     src: start.sh.j2
24     dest: /etc/my_init.d/99_start
25     owner: root
26     group: root
27     mode: 0744

Because this is a Debian based image, updating apt is a common first step. Next, it installs some packages listed in vars/main.yml. Then, it walks through separate plays for installing the yum mirror pieces, the syncing pieces, the TFTP server, and the DHCP server. Finally, I drop in the last “init” script with some basic shell stuff that looks a lot like what I was doing in my entry scripts in my Dockerfiles.

Easing the Build/Test Process

I wrote a really quick shell script in the root of the repo called orc which is very rough right now and specific to my needs. However, it means I can do a ./orc build followed by an ./orc test followed by a ./orc deploy as needed. Here’s what actions orc provides at a glance:

orc build

Runs ansible-container build

$ ./orc build
No DOCKER_HOST environment variable found. Assuming UNIX socket at /var/run/docker.sock
Starting Docker Compose engine to build your images...
Attaching to ansible_ansible-container_1
Cleaning up Ansible Container builder...
Attaching to ansible_ansible-container_1, ansible_maas_1
ansible-container_1  | 
ansible-container_1  | PLAY [all] *********************************************************************
ansible-container_1  | 
ansible-container_1  | TASK [raw] *********************************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [raw] *********************************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | PLAY [maas] ********************************************************************
ansible-container_1  | 
ansible-container_1  | TASK [setup] *******************************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Update Apt] *******************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install packages into container] **********************************
ansible-container_1  | ok: [maas] => (item=[u'ca-certificates', u'wget', u'net-tools'])
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure Web Mirror] *********************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/mirror.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install Apache2] **************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install php7] *****************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable mod_rewrite] ***********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure Apache2] ************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable Apache2 at start] ******************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure Mirror Sync] ********************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/sync.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install repo packages] ********************************************
ansible-container_1  | ok: [maas] => (item=[u'createrepo', u'rsync'])
ansible-container_1  | 
ansible-container_1  | TASK [maas : Copy Mirror Crontab] **********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Copy Mirror Sync Script] ******************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure TFTP] ***************************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/tftp.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install TFTP] *****************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure TFTPd] **************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable TFTPd at start] ********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Template grub.cfg and ks.php files into place] ********************
ansible-container_1  | ok: [maas] => (item={u'src': u'grub.cfg-net.j2', u'dst': u'/root/grub.cfg-net'})
ansible-container_1  | ok: [maas] => (item={u'src': u'grub.cfg-i386-pc.j2', u'dst': u'/root/grub.cfg-i386-pc'})
ansible-container_1  | ok: [maas] => (item={u'src': u'grub.cfg-x86_64-efi.j2', u'dst': u'/root/grub.cfg-x86_64-efi'})
ansible-container_1  | ok: [maas] => (item={u'src': u'ks.php.j2', u'dst': u'/root/ks.php'})
ansible-container_1  | 
ansible-container_1  | TASK [maas : Make kickstart directory] *****************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Generate kickstarts] **********************************************
ansible-container_1  | changed: [maas] => (item={u'name': u'mp', u'mgmtip': u'172.22.10.50', u'ram': u'24', u'mgmtif': u'enp7s0f0', u'platform': u'mac', u'mgmtmac': u'00:23:32:2f:40:3c', u'cores': u'8', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'mm1', u'mgmtip': u'172.22.10.51', u'ram': u'16', u'mgmtif': u'enp1s0f0', u'platform': u'mac', u'mgmtmac': u'a8:20:66:34:ff:e9', u'cores': u'4', u'disk': u'250', u'arch': u'x86_64', u'mbr': u'sdb'})
ansible-container_1  | changed: [maas] => (item={u'name': u'mm2', u'mgmtip': u'172.22.10.52', u'ram': u'16', u'mgmtif': u'enp1s0f0', u'platform': u'mac', u'mgmtmac': u'a8:20:66:4a:ce:46', u'cores': u'4', u'disk': u'250', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'mm3', u'mgmtip': u'172.22.10.53', u'ram': u'16', u'mgmtif': u'enp1s0f0', u'platform': u'mac', u'mgmtmac': u'a8:20:66:4a:d9:da', u'cores': u'4', u'disk': u'250', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'smpc', u'mgmtip': u'172.22.10.54', u'ram': u'12', u'mgmtif': u'enp6s0', u'platform': u'pc', u'mgmtmac': u'00:30:48:fb:e2:44', u'cores': u'4', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'sm1', u'mgmtip': u'172.22.10.55', u'ram': u'96', u'mgmtif': u'enp11s0', u'platform': u'pc', u'mgmtmac': u'00:25:90:96:c4:9a', u'cores': u'16', u'disk': u'512', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'sm2', u'mgmtip': u'172.22.10.56', u'ram': u'96', u'mgmtif': u'enp11s0', u'platform': u'pc', u'mgmtmac': u'00:25:90:96:c6:5a', u'cores': u'16', u'disk': u'512', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'd1', u'mgmtip': u'172.22.10.57', u'ram': u'32', u'mgmtif': u'em1', u'platform': u'pc', u'mgmtmac': u'bc:30:5b:e5:73:b7', u'cores': u'4', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | changed: [maas] => (item={u'name': u'd2', u'mgmtip': u'172.22.10.58', u'ram': u'32', u'mgmtif': u'em1', u'platform': u'pc', u'mgmtmac': u'bc:30:5b:e5:75:28', u'cores': u'4', u'disk': u'240', u'arch': u'x86_64', u'mbr': u'sda'})
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure DHCP] ***************************************************
ansible-container_1  | included: /ansible-container/ansible/roles/maas/tasks/dhcp.yml for maas
ansible-container_1  | 
ansible-container_1  | TASK [maas : Install repo packages] ********************************************
ansible-container_1  | ok: [maas] => (item=[u'isc-dhcp-server', u'grub-common', u'grub2-common', u'grub-imageboot', u'grub-pc-bin', u'grub-efi'])
ansible-container_1  | 
ansible-container_1  | TASK [maas : Configure DHCPD] **************************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Enable DHCP server] ***********************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | TASK [maas : Add final script on init] *****************************************
ansible-container_1  | ok: [maas]
ansible-container_1  | 
ansible-container_1  | PLAY RECAP *********************************************************************
ansible-container_1  | maas                       : ok=27   changed=1    unreachable=0    failed=0   
ansible-container_1  | 
ansible_ansible-container_1 exited with code 0
Aborting on container exit...
Stopping ansible_maas_1 ... done
Exporting built containers as images...
Committing image...
Exported netpxemirror-ac-maas with image ID sha256:516ae0de22da34d87a9726dca0a6d92b9602b9df3c757b4bc7e71c9fc1e6ec60
Cleaning up maas build container...
Cleaning up Ansible Container builder...
orc buildclean

Runs ansible-container build --from-scratch to rebuild from a clean starting point. Useful if large changes to packages and scripts are made in the role.

orc run

Runs ansible-container run locally. This variation has a while loop for the init script instead of your “production” init script. In my case, it takes the place of /sbin/my_init from the phusion/baseimage.

orc test

Finds the id of the running container, runs the chef/inspec docker container and attaches to it to run the test suite. Thanks to the Chef folks for making inspec so easy to use/install.

$ docker pull chef/inspec
$ did="$(docker ps -q --filter='name=ansible_maas_1')"
$ docker run -it --rm -v $(pwd):/share -v /var/run/docker.sock:/var/run/docker.sock chef/inspec exec spec/test.rb -t docker://$did
$ No DOCKER_HOST environment variable found. Assuming UNIX socket at /var/run/docker.sock
Attaching to ansible_ansible-container_1
Cleaning up Ansible Container builder...
Attaching to ansible_maas_1
maas_1               | *** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
maas_1               | *** Running /etc/my_init.d/10_apache2...
maas_1               |  * Starting Apache httpd web server apache2
maas_1               | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
maas_1               |  * 
maas_1               | *** Running /etc/my_init.d/20_tftpd...
maas_1               |  * Starting HPA's tftpd in.tftpd
maas_1               |    ...done.
maas_1               | *** Running /etc/my_init.d/30_dhcp...
maas_1               | *** Running /etc/my_init.d/99_start...
maas_1               | Netboot directory for i386-pc created. Configure your DHCP server to point to /nbi/boot/grub/i386-pc/core.0
maas_1               | Netboot directory for x86_64-efi created. Configure your DHCP server to point to /nbi/boot/grub/x86_64-efi/core.efi
maas_1               | *** Running /etc/rc.local...
maas_1               | *** Booting runit daemon...
maas_1               | *** Runit started as PID 75
maas_1               | Jul 28 19:34:32 4ca404c631e7 syslog-ng[80]: syslog-ng starting up; version='3.5.6'
maas_1               | Jul 28 19:39:01 4ca404c631e7 CRON[137]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
maas_1               | Jul 28 20:09:01 4ca404c631e7 CRON[1264]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
maas_1               | Jul 28 20:17:01 4ca404c631e7 CRON[1306]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
maas_1               | Jul 28 20:39:01 4ca404c631e7 CRON[1309]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
...................

Finished in 3.65 seconds (files took 2.16 seconds to load)
19 examples, 0 failures

orc deploy

A bit of custom stuff to send over the latest image to my deploy system in docker-compose format and starts it there.

$ ./orc deploy
* Deploying...
 - Syncing the latest
 521MiB 0:00:43 [12.1MiB/s] [                                                    <=>                                   ]
 - Copying compose file
 - Restarting compose app
Stopping admin_maas_1 ... done
Removing admin_maas_1 ... done
* Done.

What’s Better?

Well, this approach pretty much provides the same functionality as before, but it’s much easier to adjust/tweak and maintain. Here’s where things stand after moving to this method:

  1. Lack of central inventory of systems and asset attributes
    • All of that data is now stored in the maas roles’ vars/main.yml in a format that Ansible can parse/loop through
  2. Hardcoded paths and configuration files/settings in Docker containers
    • All of the key variables and path names have been separated into variables–again stored in the maas roles’ vars/main.yml. Now, changing names of files/directories is a trivial exercises as needs change.
  3. Hardcoded DHCP Leases
    • These are generated during the templating run by Jinja2 filters in the dhcpd.conf.j2 template stored in the maas templates folder and called by the dhcp.yml task/play.
  4. Individually crafted Kickstart configuration files
    • Using the variable storage method in the roles’ vars/main.yml, the ks.cfg.j2 template is used to generate all the per-host kickstart files.
  5. SELinux is disabled on deploy
    • Not yet addressed, but it should be a simpler debugging exercise now that it’s a single docker container running on deploy.
  6. One version of one operating system supported
    • Not yet addressed, but it’s easier to add support now that all configuration files are templated with Ansible/Jinja2.
  7. Logging from the containers
    • The phusion/baseimage runs a syslog daemon out of the box, but it currently is not sent anywhere.
  8. It wasn’t very “DRY”
    • Instead of three separate containers from 2 different FROM base images, it’s now a single debian-based container running 3 key services.
  9. There was no testing of the container contents
    • Adding inspec testing means I can now confidently add test coverage and perform testing during the build process in a few seconds instead of manually validating functionality.
  10. Inspecting the containers as I built it was tedious
    • The additional insight gained by using debugging features in Ansible and the logging as the play is being run means I can more quickly diagnose where something went wrong. Ansible does a decent job of capturing the error message when things go sideways, and it spits it out immediately.
  11. I was building on my deploy system directly
    • Using a bit of docker and ssh glued together with some docker-compose, I have the build and test processes running on my workstation and the final container running on the deploy system.

Back to Index