IBM GPFS cluster installation and configuration in IBM AIX

Introduction to GPFS

The GPFS is a high-performance clustered file system that can be deployed in shared-disk or shared-nothing distributed parallel modes.

GPFS provides concurrent high-speed file access to the applications executing on multiple nodes of clusters which can be used with AIX 5L, Linux, and Windows platforms. GPFS provides file system storage capabilities and tools for managing GPFS clusters, which allows shared access to file systems from remote GPFS clusters.

Figure 1. Block diagram of GPFS cluster architecture

alt

Required packages and operating system level configuration

The following file sets are mandatory to configure GPFS cluster

  • gpfs.base
  • gpfs.crypto
  • gpfs.docs.data
  • gpfs.ext
  • gpfs.gskit
  • gpfs.msg.en_US

The base file sets can be ordered using an IBM Business Partner® ID. You can find more information at IBM Knowledge Centre.

Required GPFS fix packs are available at IBM Fix central.

Operating system level configuration

All nodes host names should be resolved from each and every node. For example, there were two nodes with the following details.

Node 1:

IP address: 192.168.1.10

Host name, fully qualified domain name (FQDN): node1.mycompany.com

Short host name: node1

Node 2:

IP address: 192.168.1.11

Host name (FQDN): node2.mycompany.com

Short host name: node2

All these entries must be available on the /etc/hosts file of all the nodes, and at least one raw disk which is shared between all nodes should be present.

Password-less authentication configuration

The best practice is to configure password-less authentication between all nodes. This section explains how to configure the password-less authentication with an example.You need to perform the following steps for password-less authentication for Secure Shell (SSH)

  1. Install the SSH server and client in the system.
  2. Generate the public-keys using the ssh-Keygen command.

    [root@localhost ~]# ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/root/.ssh/id_rsa):
    Created directory '/root/.ssh'.
    Enter passphrase (empty for no passphrase): (Just Enter Key for empty PWD)
    Enter same passphrase again:
    Your identification has been saved in /root/.ssh/id_rsa.
    Your public key has been saved in /root/.ssh/id_rsa.pub.
    The key fingerprint is:
    93:55:2d:d8:d7:a0:14:83:17:3f:d4:fc:46:a7:f6:90 root@localhost
    
  3. This key will be generated in /root/.ssh/id_rsa.pub. Perform the same steps across all nodes.

  4. Create a file, authorized_keys in the /root/.ssh/ directory, and change the file permissions to 644.
  5. Copy the content of id_rsa.pub to authorized_keys for enabling password-less authentication within the local system. Repeat step 4 and step 5 on all nodes for enabling local authentication.
  6. Copy the id_rsa.pub content on node1 to node2, node3, node4, and so on nodes to the authorization_Key file. This completes the configuration of password-less authentication on all the cluster nodes.
  7. Finally, test the command execution remotely from the local system. alt

If the given commands get executed successfully, then it indicates that our configuration is correct.

Installing the GPFS package

You need to perform the following steps to install the GPFS package.

  1. Run the smit installp command.
  2. On the Install and Update Software screen, select Install Software. alt
  3. Specify the location PATH, where the file sets are copied. alt
  4. Select the file sets, and accept the license agreements. alt
  5. Update the GPFS fix packs (if any).
  6. Run the smit update command to update the patches or fix packs. alt
  7. Provide the path to install the GPFS package and press Enter. Else, you can also use the following command:

    install_all_updates -c -d ./ -Y
    

After successful installation of the GPFS, the file sets will be available in the /usr/lpp/mmfs directory.

Note: Append /usr/lpp/mmfs/bin to the system path for ease of use of GPFS commands in /etc/profile.

alt

Configuration

Cluster configuration

Perform the following steps to configure a GPFS cluster.

  1. Log in to node1 as a root user and check the system to find whether it is a part of any other cluster using the mmlscluster command.

    # mmlscluster
    mmlscluster: This node does not belong to a GPFS cluster.
    mmlscluster: Command failed. Examine previous error messages to determine cause.
    
  2. Configure the cluster primary node using the mmcrcluster command.

    mmcrcluster -N node1.mycompany.com:manager-quorum -p node1.mycompany.com
    -r /usr/bin/ssh -R /usr/bin/scp
    Sat Jul 20 00:44:47 IST 2013 : mmcrcluster: Processing node node1
    mmcrcluster: Command successfully completed
    mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
        Use the mmchlicense command to designate licenses as needed.
    
  3. Run the mmlscluster command again to confirm that the cluster has been created.

    # /usr/lpp/mmfs/bin/mmlscluster
    GPFS cluster information
    ========================
      GPFS cluster name:         node1.mycompany.com
      GPFS cluster id:           7699862183884611001
      GPFS UID domain:           node1.in.ibm.com
      Remote shell command:      /usr/bin/ssh
      Remote file copy command:  /usr/bin/scp
    
    GPFS cluster configuration servers:
    -----------------------------------
      Primary server:    node1.mycompany.com
    Node  Daemon node name    IP address     Admin node name         Designation
    -----------------------------------------------------------------------------------------
     1    node1.mycompany.com 192.168.1.10   node1.mycompany.com    quorum-manager
    
  4. Apply the GPFS license on the primary node using the mmchlicense command.

    # mmchlicense server --accept -N node1
    The following nodes will be designated as possessing GPFS server licenses:
            node1.mycompany.com
    
    mmchlicense: Command successfully completed
    
  5. Start GPFS on one of the nodes in the GPFS cluster using the mmstartup command. If the Cluster configuration is successful, GPFS will start automatically on all the nodes.

    # mmstartup -a
    Sat Jul 20 01:13:26 IST 2013: mmstartup: Starting GPFS ...
    
  6. Check the status of the cluster using the mmgetstate command.

    # mmgetstate -a
    Node number  Node name        GPFS state
    ------------------------------------------------------------
         1       Node1             active
    
  7. Perform the GPFS package installation on the second node and compile the source.

  8. On node1, use the mmaddnode command to add node2 to the cluster

    mmaddnode -N node2
    
  9. Confirm that the node has been added to the cluster using the mmlscluster command.

    # mmlscluster
    GPFS cluster information
    ========================
      GPFS cluster name:         node1.in.ibm.com
      GPFS cluster id:           7699862183884611001
      GPFS UID domain:           node1.in.ibm.com
      Remote shell command:      /usr/bin/ssh
      Remote file copy command:  /usr/bin/scp
    
    GPFS cluster configuration servers:
    ------------------------------------------------
      Primary server:    node1.in.ibm.com
      Secondary server:  node2.in.ibm.com
    
     Node  Daemon node name  IP address   Admin node name   Designation
    --------------------------------------------------------------------
       1   node1.in.ibm.com  192.168.1.10  node1.in.ibm.com  quorum-manager
       2   node2.in.ibm.com  192.168.1.11  node2.in.ibm.com
    
  10. Use the mmchcluster command to set node2 as the secondary configuration server.

    # mmchcluster -s node2
    mmchcluster: GPFS cluster configuration servers:
    mmchcluster: Primary server:    node1.in.ibm.com
    mmchcluster: Secondary server:  node2.in.ibm.com
    mmchcluster: Command successfully completed
    
  11. Set the license mode for the node using the mmchlicense command. Use a server license for this node.

    # mmchlicense server --accept -N node2
    The following nodes will be designated as possessing GPFS server licenses:
            node2.in.ibm.com
    mmchlicense: Command successfully completed
    mmchlicense: Propagating the cluster configuration data to all
     affected nodes.  This is an asynchronous process.
    
  12. Start node2 using the mmstartup command.

    # mmstartup -N node2
    Sat Jul 20 01:49:44 IST 2013: mmstartup: Starting GPFS ...
    
  13. Use the mmgetstate command to verify that both nodes are in the active state.

    # mmgetstate -a
     Node number  Node name        GPFS state
    -------------------------------------------------------------
           1      node1            active
           2      node2            active
    

Cluster file system configuration

Perform the following steps to configure the cluster file system.

  1. Create a disk descriptor file / diskdesc.txt using the following format.

    DDiskName:ServerList::DiskUsage:FailureGroup:DesiredName:StoragePool
    hdiskX:::dataAndMetadata::nsd1:
    
  2. Create the NSDs using the mmcrnsd command.

    # mmcrnsd -F /diskdesc.txt
    

    Note: GPFS provides a block-level interface over TCP/IP networks called the NSD protocol. Whether using the NSD protocol or a direct attachment to the storage area network (SAN), the mounted file system looks the same to the users and the application (GPFS transparently handles I/O requests). A shared disk cluster is the most basic environment. In this configuration, the storage is directly attached to all the systems in the cluster. The direct connection means that each shared block device is available concurrently to all of the nodes in the GPFS cluster. Direct access means that the storage is accessible using a Small Computer System Interface (SCSI) or other block-level protocol using a SAN disk.

  3. Create a file system using the mmcrfs command.

    # mmcrfs /gpfs fs1 -F diskdesc.txt -B 64k
    
  4. Verify that the file system has been created correctly using the mmlsfs command.

    # mmlsfs fs1
    flag                value                    description
    ------------------- ------------------------ -----------------------------------
     -f                 2048                     Minimum fragment size in bytes
     -i                 512                      Inode size in bytes
     -I                 8192                     Indirect block size in bytes
     -m                 1                        Default number of metadata replicas
     -M                 2                        Maximum number of metadata replicas
     -r                 1                        Default number of data replicas
     -R                 2                        Maximum number of data replicas
     -j                 cluster                  Block allocation type
     -D                 nfs4                     File locking semantics in effect
     -k                 all                      ACL semantics in effect
     -n                 32                       Estimated number of nodes that will mount file system
     -B                 65536                    Block size
     -Q                 none                     Quotas enforced
                        none                     Default quotas enabled
     --filesetdf        no                       Fileset df enabled?
    -V                 13.23 (3.5.0.7)          File system version
    --create-time      Thu Jul 18 22:09:36 2013 File system creation time
    -u                 yes                      Support for large LUNs?
    -z                 no                       Is DMAPI enabled?
    -L                 4194304                  Logfile size
    -E                 yes                      Exact mtime mount option
    -S                 no                       Suppress atime mount option
    -K                 whenpossible             Strict replica allocation option
    --fastea           yes                      Fast external attributes enabled?
    --inode-limit      102528                   Maximum number of inodes
    -P                 system                   Disk storage pools in file system
    -d                 nsd1                     Disks in file system
    --perfileset-quota no                       Per-fileset quota enforcement
    -A                 yes                      Automatic mount option
    -o                 none                     Additional mount options
    -T                 /gpfs                    Default mount point
    --mount-priority   0                        Mount priority
    
  5. Mount the file system using the mmmount command.

    # mmmount all -a
    Sat Jul 20 02:22:45 IST 2013: mmmount: Mounting file systems ...
    
  6. Verify that the file system is mounted using the df command.

    # df -g
    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
    /dev/hd4           2.00      1.56   22%    10221     3% /
    /dev/hd2           5.00      2.08   59%    52098    10% /usr
    /dev/hd9var        2.00      1.70   16%     7250     2% /var
    /dev/hd3           5.00      4.76    5%      262     1% /tmp
    /dev/hd1           0.50      0.50    1%       79     1% /home
    /dev/hd11admin     0.12      0.12    1%        5     1% /admin
    /proc                 -         -    -         -     -  /proc
    /dev/hd10opt      25.00      5.72   78%   371887    21% /opt
    /dev/fs1              100G       282M       100G    1%     /gpfs  <-- This is the GPFS file
    System
    

    The file system will be available automatically on both the systems.

    node1:~ # df -g
    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
    /dev/hd4           2.00      1.56   22%    10221     3% /
    /dev/hd2           5.00      2.08   59%    52098    10% /usr
    /dev/hd9var        2.00      1.70   16%     7250     2% /var
    /dev/hd3           5.00      4.76    5%      262     1% /tmp
    /dev/hd1           0.50      0.50    1%       79     1% /home
    /dev/hd11admin     0.12      0.12    1%        5     1% /admin
    /proc                 -         -    -         -     -  /proc
    /dev/hd10opt      25.00      5.72   78%   371887    21% /opt
    /dev/fs1             100G        282M        100G     1%       /gpfs
    
    node1:~ # ssh node2 df -h
    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
    /dev/hd4           2.00      1.56   22%    10221     3% /
    /dev/hd2           5.00      2.08   59%    52098    10% /usr
    /dev/hd9var        2.00      1.70   16%     7250     2% /var
    /dev/hd3           5.00      4.76    5%      262     1% /tmp
    /dev/hd1           0.50      0.50    1%       79     1% /home
    /dev/hd11admin     0.12      0.12    1%        5     1% /admin
    /proc                 -         -    -         -     -  /proc
    /dev/hd10opt      25.00      5.72   78%   371887    21% /opt
    /dev/fs1              100G        282M        100G   1%       /gpfs
    
    node1:~ #
    
  7. Use the mmdf command to get information on the file system.

    # mmdf fs1
    disk                disk size   failure  holds     holds              free KB             free KB
    name                    in KB     group  metadata  data        in full blocks        in fragments
    --------------- -------------  --------  --------  ----- -------------------- -------------------
    Disks in storage pool: system (Maximum disk size allowed is 800 GB)
    nsd1                104857600       -1    yes        yes      104569408 (100%)            94 ( 0%)
                    -------------                             -------------------- -------------------
    (pool total)        104857600                                 104569408 (100%)            94 ( 0%)
    
                    =============                             ==================== ===================
    (total)             104857600                                 104569408 (100%)            94 ( 0%)
    
    Inode Information
    -----------------
    Number of used inodes:            4041
    Number of free inodes:           98487
    Number of allocated inodes:     102528
    Maximum number of inodes:       102528
    

Conclusion

GPFS cluster is useful in environments where we need high availability of file system. This paper provides guidelines to UNIX® administrators in installing and configuring GPFS cluster environment for AIX 5L cluster nodes, Linux cluster nodes, Microsoft Windows Server cluster nodes or heterogeneous cluster nodes for AIX, Linux, and Windows.

Narendra Babu Swarna