The basic purpose of this write-up is to describe the process we underwent in trying to get a functional Ironic setup on POWER8, and document the various small hiccups we hit along the way. Deployment through Ironic has been found to be fully functional â€“ deploying from POWER8 to other POWER8 machines, as well as x86 machines, should work just fine.
For our testing, we used:
The Master branch of Disk Image Builder. (commit b833960c69645ccbb9c57af99704da6df1676fd2 at time of testing)
The version available in Ubuntuâ€™s repositories has a bug that prevents us from deploying to POWER8 correctly. This has been fixed in time for Ocata and is available in the master branch.
15.04 should also work, but we ran into issues with ramdisks built on 14.04
All the way back to 14.04 should work for user images, so all versions of Ubuntu currently supported by Canonical should work, based on our experiences.
Available at https://github.com/openstack-dev/devstack
We, of course, don’t recommend that you set up production services with DevStack, but this was a quick and easy way for us to set up our environment to validate Ironic. A DevStack guide can be found here: https://docs.openstack.org/developer/devstack/
For this walk-through, weâ€™re assuming that you have two basic requirements fulfilled:
You have the Keystone, Neutron, Nova, Swift, and Glance services set up. Again, if you would like to try to reproduce our steps, you can set up these adjacent services quickly with DevStack.
Your target machine will be connected to a flat provider network. A provider network is the network created by an OpenStack administrator, and maps directly to a network in a data center. Flat means that it is unsegmented (i.e. – by vlan). The Project Install Guide will use this by default – just make sure your machines can directly communicate over an interface, with no segmentation between them.
As of late 2016, Ironic has switched over to using a new install guide. The current one, as of March 10th, 2017, can be found here: https://docs.openstack.org/project-install-guide/baremetal/draft
If youâ€™ve somehow found your way to an older installation guide, it will cause many problems during installation, so please donâ€™t use it.
In general, walking through the guide should be fairly smooth. There were a few bumps that we encountered along the way, but most have since been fixed in the guide. Any issues that still persist will be mentioned in a relevant section of the Notes by Section portion of this write-up.
A couple of worthwhile points to note before beginning:
If you receive a "No Valid Hosts" error during deployment, this is a generic catch-all error.
Actual errors can be found within /var/log/syslog, and within the log files for each individual service.
If you’re having trouble with DHCP or TFTP, check the syslog. By default this is where dnsmasq and xinetd spit out errors.
xinetd is what the Ironic community recommends as a TFTP server. dnsmasq is what Neutron uses by default as a DHCP server.
screen -x, while Ironic will create its own log file, typically in /var/log/ironic.
glance_url), you must include the relevant protocol prefix. The user running the Ironic service (in most cases â€˜ironicâ€™ or â€˜ubuntuâ€™) should own the tftp images directory. ( /tftpboot )
To see what user is running ironic, run a
ps aux | grep -i ironic-api
pxe_append_params = nofb nomodeset vga=normal systemd.journald.forward_to_console=yes
Notes by Section
Here, we will step through each section of the Ironic set-up guide located at https://docs.openstack.org/project-install-guide/baremetal/draft and detail any issues we encountered that will need to be worked around, and disclose any tips that we believe to be helpful to getting a successful deployment.
Install and Configure for Ubuntu:
We had no real issues with this section, but it is worth mentioning that, while the guide recommends MySQL for the database back-end, this can cause many issues. There are two ways around this:
Use Mariadb instead. (recommended)
Alternatively, edit /etc/mysql/my.conf such that the max_connections value in the [mysqld] section is significantly higher. (1000 – 5000)
This may be located elsewhere on your distribution of choice.
Configure the Identity Service for Bare Metal Service
Once again, we had no real hitches here. Though, when we first began this process, the identity service step was not listed in the main guide list.
Configure Compute to use the Bare Metal Service
Initially, when we hit this section, we ran into resource allocation issues. Make sure your resource allocation is configured to look for a 1:1 memory ratio, and that you’re not asking for more hardware than any of your boxes can provide. The guide will describe where and how this can be accomplished.
Create Compute flavors for use with Bare Metal Service
Nothing of note.
Configure Networking to communicate with Bare Metal server
We used the network described in the guide, which is a flat provider network
Basically this means that the network is unsegmented so all machines on the same network can reach each other.
Note that when you add an interface to your port, the port will lose its IP and thus any connections on that port. (i.e. – your SSH session will break)
At time of writing, the guide does not describe how to add an address to the new bridge. For our Ubuntu environment, we configured our interfaces as follows:
On bring up, the Ethernet port and bridge are set to promiscuous mode.
This allows traffic that’s intended for the virtual bridge to not be thrown out by the NIC.
Static networking should work as well.
#eth0 set to manual, br0 set to dhcp
# The loopback network interface
iface lo inet loopback
# The primary network interface
iface eth0 inet manual
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down
iface br0 inet dhcp
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
These changes will not take effect until you’ve brought your interfaces down and back up. This can be accomplished by running the following: (once again, remember to substitute eth0)
ifdown br0; ifup br0
ifdown eth0; ifup eth0
Create and add images to the image service
This section has a few things worth noting:
Disk Image Builder Elements required: baremetal, pxe
We modified /usr/local/share/diskimage-builder/elements/ironic-agent/package-installs.yaml so that the last set of lines reads:
We do this because otherwise, console output will be very difficult to read.
Additionally, this will prevent any connection saturation when watching the deployment remotely via IPMI, which would slow down the deployment.
Configure the Bare Metal service for cleaning
We had no significant issues with this, but want to explicitly mention that you should not skip over this section â€“ the cleaning network will make it far easier to take down a node if it has a deployment error. Otherwise, you’ll have to set the node to maintenance mode and manually delete it each time.
We didn’t explicitly verify this section, as HTTPS isn’t necessary for internal deployment, but seeing as how all the required adjacent services are expected to work, it seems unlikely that there would be issues.
Setup the drivers for the Bare Metal service
For this section, just make sure /tftpboot and all sub-folders are owned by the user thatâ€™s running ironic. If you’ve been following along with the guide, this user is probably called ironic! See the General Notes section for more information. Also note that you will have to kill the tftpboot process, and restart xinetd, if the tftp server is already running if you change the folder’s owner.
At this point, your Ironic service should be configured enough to attempt deployment! This process is intended to be covered in a future blog post.