Intro

The basic purpose of this write-up is to describe the process we underwent in trying to get a functional Ironic setup on POWER8, and document the various small hiccups we hit along the way. Deployment through Ironic has been found to be fully functional – deploying from POWER8 to other POWER8 machines, as well as x86 machines, should work just fine.

For our testing, we used:

  • The Master branch of Disk Image Builder. (commit b833960c69645ccbb9c57af99704da6df1676fd2 at time of testing)
    • https://github.com/openstack/diskimage-builder
    • The version available in Ubuntu’s repositories has a bug that prevents us from deploying to POWER8 correctly. This has been fixed in time for Ocata and is available in the master branch.
  • Ubuntu 16.04 ramdisk.
    • 15.04 should also work, but we ran into issues with ramdisks built on 14.04
  • Ubuntu 16.04 as the user image.
    • All the way back to 14.04 should work for user images, so all versions of Ubuntu currently supported by Canonical should work, based on our experiences.
  • Ironic version from the Ubuntu repositories. (Currently Newton)
  • DevStack from Github to set up adjacent services. (but not Ironic!)

Preliminary Setup

For this walk-through, we’re assuming that you have two basic requirements fulfilled:

  • You have the Keystone, Neutron, Nova, Swift, and Glance services set up. Again, if you would like to try to reproduce our steps, you can set up these adjacent services quickly with DevStack.
  • Your target machine will be connected to a flat provider network. A provider network is the network created by an OpenStack administrator, and maps directly to a network in a data center. Flat means that it is unsegmented (i.e. – by vlan). The Project Install Guide will use this by default – just make sure your machines can directly communicate over an interface, with no segmentation between them.

As of late 2016, Ironic has switched over to using a new install guide. The current one, as of March 10th, 2017, can be found here: https://docs.openstack.org/project-install-guide/baremetal/draft

If you’ve somehow found your way to an older installation guide, it will cause many problems during installation, so please don’t use it.

General Notes

In general, walking through the guide should be fairly smooth. There were a few bumps that we encountered along the way, but most have since been fixed in the guide. Any issues that still persist will be mentioned in a relevant section of the Notes by Section portion of this write-up.

A couple of worthwhile points to note before beginning:

  • If you receive a "No Valid Hosts" error during deployment, this is a generic catch-all error.
    • Actual errors can be found within /var/log/syslog, and within the log files for each individual service.
      • If you’re having trouble with DHCP or TFTP, check the syslog. By default this is where dnsmasq and xinetd spit out errors.
      • xinetd is what the Ironic community recommends as a TFTP server. dnsmasq is what Neutron uses by default as a DHCP server.
    • In our experience, the ironic and nova logs were the most helpful.
    • If you are using DevStack, services that are not Ironic will report errors through a local Screen session, accessed readily via screen -x , while Ironic will create its own log file, typically in /var/log/ironic.
  • Whenever a change is made to the Ironic configuration file (ironic.conf), or the database it accesses, the Ironic service should be restarted. The process for this is detailed in the guide.
  • In any OpenStack service configuration file, if substituting or changing IP in a *_url field (e.g. glance_url), you must include the relevant protocol prefix.
  • The user running the Ironic service (in most cases ‘ironic’ or ‘ubuntu’) should own the tftp images directory. ( /tftpboot )
    • To see what user is running ironic, run a ps aux | grep -i ironic-api
  • If you want to see Ironic Python Agent’s console output, you should place this in the [pxe] section of ironic.conf:
    pxe_append_params = nofb nomodeset vga=normal systemd.journald.forward_to_console=yes

Notes by Section

Here, we will step through each section of the Ironic set-up guide located at https://docs.openstack.org/project-install-guide/baremetal/draft and detail any issues we encountered that will need to be worked around, and disclose any tips that we believe to be helpful to getting a successful deployment.

Install and Configure for Ubuntu:

We had no real issues with this section, but it is worth mentioning that, while the guide recommends MySQL for the database back-end, this can cause many issues. There are two ways around this:

  • Use Mariadb instead. (recommended)
  • Alternatively, edit /etc/mysql/my.conf such that the max_connections value in the [mysqld] section is significantly higher. (1000 – 5000)
    • This may be located elsewhere on your distribution of choice.

Configure the Identity Service for Bare Metal Service

Once again, we had no real hitches here. Though, when we first began this process, the identity service step was not listed in the main guide list.

Configure Compute to use the Bare Metal Service

Initially, when we hit this section, we ran into resource allocation issues. Make sure your resource allocation is configured to look for a 1:1 memory ratio, and that you’re not asking for more hardware than any of your boxes can provide. The guide will describe where and how this can be accomplished.

Create Compute flavors for use with Bare Metal Service

Nothing of note.

Configure Networking to communicate with Bare Metal server

  • We used the network described in the guide, which is a flat provider network
    • Basically this means that the network is unsegmented so all machines on the same network can reach each other.
    • Note that when you add an interface to your port, the port will lose its IP and thus any connections on that port. (i.e. – your SSH session will break)
    • At time of writing, the guide does not describe how to add an address to the new bridge. For our Ubuntu environment, we configured our interfaces as follows:
      • On bring up, the Ethernet port and bridge are set to promiscuous mode.
        • This allows traffic that’s intended for the virtual bridge to not be thrown out by the NIC.
      • The bridge is set to obtain IP via DHCP.
        • Static networking should work as well.
      • You can accomplish the configuration in Ubuntu with the following in /etc/network/interfaces: (be sure to replace all instances of eth0 with your desired interface!)

#eth0 set to manual, br0 set to dhcp
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down

auto br0
iface br0 inet dhcp
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off

  • These changes will not take effect until you’ve brought your interfaces down and back up. This can be accomplished by running the following: (once again, remember to substitute eth0)

ifdown br0; ifup br0
ifdown eth0; ifup eth0

Create and add images to the image service

This section has a few things worth noting:

  • Disk Image Builder Elements required: baremetal, pxe
  • We modified /usr/local/share/diskimage-builder/elements/ironic-agent/package-installs.yaml so that the last set of lines reads:

ureadahead:
    uninstall: True

  • We do this because otherwise, console output will be very difficult to read.
  • Additionally, this will prevent any connection saturation when watching the deployment remotely via IPMI, which would slow down the deployment.

Configure the Bare Metal service for cleaning

We had no significant issues with this, but want to explicitly mention that you should not skip over this section – the cleaning network will make it far easier to take down a node if it has a deployment error. Otherwise, you’ll have to set the node to maintenance mode and manually delete it each time.

Enabling HTTPS

We didn’t explicitly verify this section, as HTTPS isn’t necessary for internal deployment, but seeing as how all the required adjacent services are expected to work, it seems unlikely that there would be issues.

Setup the drivers for the Bare Metal service

For this section, just make sure /tftpboot and all sub-folders are owned by the user that’s running ironic. If you’ve been following along with the guide, this user is probably called ironic! See the General Notes section for more information. Also note that you will have to kill the tftpboot process, and restart xinetd, if the tftp server is already running if you change the folder’s owner.

At this point, your Ironic service should be configured enough to attempt deployment! This process is intended to be covered in a future blog post.

Join The Discussion

Your email address will not be published. Required fields are marked *