Implement high availability of Network Installation Management (NIM) for uninterrupted operations

Introduction

The Network Installation Management (NIM) server is crucial within any environment to oversee tasks such as a new system installation, backup creation, software deployment, and more. Incorporating a NIM server in your disaster recovery (DR) plan is essential, as it serves as the primary server for rebuilding failed systems. To enhance resilience, consider implementing high availability NIM, which involves deploying a backup NIM master on a separate system or logical partition (LPAR). This backup NIM master seamlessly takes over in the event of a primary NIM master failure, ensuring uninterrupted operations. Subsequently, when the primary NIM master is restored, a fallback operation transitions control back to it. These failover and fallback processes are initiated by the system administrator.

This tutorial explains how to set up both the NIM master and an alternate master, and also outlines the synchronization process for NIM databases and resources between them. Additionally, it offers guidance on performing takeover operations in the event of NIM master failure, covering both manual method and the utilization of IBM PowerHA.

Configuring NIM master and the alternate NIM master

This section explains how to configure the NIM master and the alternate NIM master.

Initial setup

Ensure that the required filesets are installed on both the NIM master and the alternate NIM master. Check for the availability of the following files:

• bos.sysmgt.nim.master
• bos.sysmgt.nim.spot
• bos.sysmgt.nim.client

It is essential that both the NIM master and the alternate NIM master run on the same version of AIX.

In this tutorial, the following LPARs are used for demonstration purposes:

• aix-install-zep01-lp016.aus.stglabs.ibm.com, which serves as the NIM master
• aix-install-zep01-lp017.aus.stglabs.ibm.com, which serves as the alternate NIM master

View larger image

View larger image

Configure the NIM master

Initialize the NIM master using the nimconfig command. You need to name the first network used by NIM. nimesis daemons will be started at this step.

Run the following command: \# smitty nim

Then, select the options in the following sequence:

Configure the NIM Environment 🡪 Advanced Configuration 🡪 Initialize the NIM Master Only.

As shown in the following screenshot, enter a name for the network and the interface. NIMSH (secure) connection enables cryptographic authentication.

View larger image

NIM resources such as spot and lpp_source can be created immediately. For the purpose of this tutorial, certain resources such as spot, lpp_source, mksysb, and network have already been created (and these will be replicated at a later stage).

Configure the alternate NIM master

On the backup NIM master server (lp017), you must set up the NIM environment in the same manner (as shown in the following screenshot) as set on the primary NIM master server.

View larger image

Set up the backup server to function as the alternate NIM master and the other way around

After completing the essential steps, you can designate the backup server as the alternate NIM master. This involves initializing the alternate NIM master from the backup NIM master, which in turn updates the NIM database on the primary NIM master and modifies the /etc/niminfo file on the backup NIM master.

In this example, lp017 serves as the backup server. Run the following command on the lp017 LPAR to initialize the system as an alternate master:

smitty nim_altmstr

After running the smitty nim_altmstr command, select the Initialize This Machine as an Alternate Master option as specify the options as shown in the following screenshot to initialize the system as an alternate master.

View larger image

lsnim output on the primary NIM master

View larger image

The /etc/niminfo file in the lp017 server is updated with the export field, NIM_ALTERNATE_MASTER=lp016, as shown in the following example.

View larger image

Note: At this point, the NIM database in the backup server (lp017) does not update lp016 as its alternate NIM master. This will be updated only after performing NIM synchronization.

Perform NIM synchronization

NIM synchronization should be done to synchronize NIM resources, such as spot, lpp_source, and others to the alternate NIM master.

On the primary NIM master server (lp016), run the smitty nim_altmstr command and select the Synchronize an Alternate Master's NIM database option. Then, specify the options as shown in the following screenshot for the backup NIM master server.

View larger image

After synchronization, you can see the resources on the alternative master.

View larger image

Takeover

If the NIM master goes offline, a takeover operation enables the alternate NIM master to assume the role of NIM master for the clients. This involves modifying the /etc/niminfo file on clients to update the NIM_MASTER_HOSTNAME and NIM_MASTER_HOSTNAME_LIST attributes.

View the output of lsnim and /etc/niminfo (as shown in the following screenshots) before the takeover operation.

View larger image

View larger image

Takeover operation is initiated from the alternate NIM master: nim $$-F$$ -o takeover -a Attribute=Value ... TargetName

Refer to the following screenshot for an example of the same.

View larger image

After takeover, you can see current master changes to alternative master.

View larger image

You can see the /etc/niminfo master hostname updated with the alternative NIM master.

View larger image

When the NIM master is running, initiate the takeover operation for the master using the following command: \# nim -o takeover -a show_progress=yes aix-install-zep01-lp017

Utilization of PowerHA for high availability of NIM

PowerHA enables the execution of the takeover command. This section describes the process to configure PowerHA between the NIM master and the alternative NIM master.

1. Install the following filesets.

   • cluster.adt.es
   • cluster.doc.en_US.es
   • cluster.es.client
   • cluster.es.cspoc
   • cluster.es.nfs
   • cluster.es.server
   • cluster.license
   • cluster.man.en_US.es

2. Set up network, files, and repository disk. This section explains the tasks and the commands to perform the task.

   • Configure boot IP: chdev -l enX -a netaddr=<> -a netmask=<> -a state=up

   • Set up /etc/hosts: Include all the necessary IPs to /etc/hosts For example:

      # cat /etc/hosts | grep aix-install-zep01-
      9.3.148.182 aix-install-zep01-lp016.aus.stglabs.ibm.com aix-install-    zep01-lp016
      9.3.148.186 aix-install-zep01-lp017.aus.stglabs.ibm.com aix-install-    zep01-lp017
      #
     

   • Set up /etc/cluster/rhosts: Include all the necessary IPs in /ect/cluster/rhosts. Notice that the syntax is different compared to how you did for the /etc/hosts file. Here, you need to add only the IPs).

      # cat /etc/cluster/rhosts
      9.3.148.182
      9.3.148.186
      #
     

   • Recycle the clcomd daemon: stopsrc -s clcomd; startsrc -s clcomd

   • Add a port VLAN ID (PVID) for the repository disk: chdev -l hdiskX -a pv=yes Note that the repository LUN should be shared, and the minimum value should be 512 MB.

3. Create a cluster (smitty hacmp).

   Run the smitty hacmp command and select the options in the given sequence for the following tasks:

   • Define a cluster: Select Cluster Nodes and Networks 🡪 Standard Cluster Deployment Setup a Cluster, Nodes and Networks. Then add a cluster name and a local node. View larger image

   • Add a second node to the cluster: Select Cluster Nodes and Networks 🡪Manage nodes 🡪 Add a Node. Then choose the other node and communication path to node.

   • Discover network interface and disks: Select Cluster Nodes and Networks 🡪 Discover Network Interfaces and Disks.

   • Remove unnecessary networks: Select Cluster Nodes and Networks 🡪 Manage Networks and Network Interfaces 🡪 Networks 🡪 Remove a Network.

   • Add repository disks: Select Cluster Nodes and Networks 🡪 Manage repository disks 🡪 Add a Repository Disk. View larger image

   • Invoke verify and sync: Select Cluster Nodes and Networks 🡪Verify and Synchronize Cluster Configuration.

   • Start the cluster: Run the smitty clstart command and then choose both the nodes. View larger image

4. Configure a resource and a resource group.

   Run the smitty hacmp command and select the options in the given sequence for the following tasks:

   • Add a resource: Select Cluster Applications and Resources 🡪 Resources 🡪 Configure User Applications (Scripts and Monitors) 🡪 Application Controller Scripts 🡪 Add Application Controller Scripts. On the Add Application Controller Scripts screen, specify name, script locations, foreground for the application controller script. View larger image

   • Add a resource group: Select Cluster Applications and Resources 🡪 Resource Groups 🡪 Add a Resource Group and specify the options as shown in the following screenshot. View larger image

   • Add resources to the resource group: Select Cluster Applications and Resources 🡪 Resource Groups 🡪 Change/Show Resources and Attributes for a Resource Group. View larger image

   • Invoke verify and sync operation: Select Cluster Applications and Resources 🡪 Verify and synchronize cluster. This action starts the resource group as well.

Illustrating NIM master failover or takeover in a PowerHA cluster

The startup script will be triggered, before launching the application during cluster initialization or in the event of a failover, when the primary NIM master is inaccessible. In the following script, we can include the takeover command. There is no need for stop or monitor scripts in this context.

# cat takeover
master=\`lsnim -t alternate_master | awk -F " " ' { print $1 } '\`
nim -o takeover -a show_progress=yes $master

View larger image

Before the NIM master's failure, the resource group had not been transferred to the alternative master.

View the output of lsnim and /etc/niminfo (as shown in the following screenshots) before the takeover operation.

View larger image

View larger image

To replicate the failure, I shut down aix-install-zep01-lp016.

Notice that the resource group (NIM_RG) has moved to alternative master (lp017). View larger image

After takeover, you can see that the current master changes to alternative master. View larger image

You can see the /etc/niminfo master hostname updated with alternative NIM master. View larger image

Summary

This tutorial provides a comprehensive guide to implement high availability NIM for NIM servers, ensuring uninterrupted operations even in the event of a primary NIM master failure.