Article
IBM GPFS cluster installation and configuration in IBM AIX
IBM Spectrum Scale Installation and ConfigurationIntroduction to GPFS
The GPFS is a high-performance clustered file system that can be deployed in shared-disk or shared-nothing distributed parallel modes.
GPFS provides concurrent high-speed file access to the applications executing on multiple nodes of clusters which can be used with AIX 5L, Linux, and Windows platforms. GPFS provides file system storage capabilities and tools for managing GPFS clusters, which allows shared access to file systems from remote GPFS clusters.
Figure 1. Block diagram of GPFS cluster architecture
Required packages and operating system level configuration
The following file sets are mandatory to configure GPFS cluster
- gpfs.base
- gpfs.crypto
- gpfs.docs.data
- gpfs.ext
- gpfs.gskit
- gpfs.msg.en_US
The base file sets can be ordered using an IBM Business Partner® ID.
Required GPFS fix packs are available at IBM Fix central.
Operating system level configuration
All nodes host names should be resolved from each and every node. For example, there were two nodes with the following details.
Node 1:
IP address: 192.168.1.10
Host name, fully qualified domain name (FQDN): node1.mycompany.com
Short host name: node1
Node 2:
IP address: 192.168.1.11
Host name (FQDN): node2.mycompany.com
Short host name: node2
All these entries must be available on the /etc/hosts file of all the nodes, and at least one raw disk which is shared between all nodes should be present.
Password-less authentication configuration
The best practice is to configure password-less authentication between all nodes. This section explains how to configure the password-less authentication with an example.You need to perform the following steps for password-less authentication for Secure Shell (SSH)
- Install the SSH server and client in the system.
Generate the public-keys using the
ssh-Keygencommand.[root@localhost ~]# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): (Just Enter Key for empty PWD) Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 93:55:2d:d8:d7:a0:14:83:17:3f:d4:fc:46:a7:f6:90 root@localhostThis key will be generated in /root/.ssh/id_rsa.pub. Perform the same steps across all nodes.
- Create a file, authorized_keys in the /root/.ssh/ directory, and change the file permissions to 644.
- Copy the content of id_rsa.pub to authorized_keys for enabling password-less authentication within the local system. Repeat step 4 and step 5 on all nodes for enabling local authentication.
- Copy the id_rsa.pub content on node1 to node2, node3, node4, and so on nodes to the authorization_Key file. This completes the configuration of password-less authentication on all the cluster nodes.
- Finally, test the command execution remotely from the local system.
If the given commands get executed successfully, then it indicates that our configuration is correct.
Installing the GPFS package
You need to perform the following steps to install the GPFS package.
- Run the
smit installpcommand. - On the Install and Update Software screen, select Install Software.
- Specify the location PATH, where the file sets are copied.
- Select the file sets, and accept the license agreements.
- Update the GPFS fix packs (if any).
- Run the
smit updatecommand to update the patches or fix packs.
Provide the path to install the GPFS package and press Enter. Else, you can also use the following command:
install_all_updates -c -d ./ -Y
After successful installation of the GPFS, the file sets will be available in the /usr/lpp/mmfs directory.
Note: Append /usr/lpp/mmfs/bin to the system path for ease of use of GPFS commands in /etc/profile.
Configuration
Cluster configuration
Perform the following steps to configure a GPFS cluster.
Log in to node1 as a root user and check the system to find whether it is a part of any other cluster using the
mmlsclustercommand.# mmlscluster mmlscluster: This node does not belong to a GPFS cluster. mmlscluster: Command failed. Examine previous error messages to determine cause.Configure the cluster primary node using the
mmcrclustercommand.mmcrcluster -N node1.mycompany.com:manager-quorum -p node1.mycompany.com -r /usr/bin/ssh -R /usr/bin/scp Sat Jul 20 00:44:47 IST 2013 : mmcrcluster: Processing node node1 mmcrcluster: Command successfully completed mmcrcluster: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed.Run the
mmlsclustercommand again to confirm that the cluster has been created.# /usr/lpp/mmfs/bin/mmlscluster GPFS cluster information ======================== GPFS cluster name: node1.mycompany.com GPFS cluster id: 7699862183884611001 GPFS UID domain: node1.in.ibm.com Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp GPFS cluster configuration servers: ----------------------------------- Primary server: node1.mycompany.com Node Daemon node name IP address Admin node name Designation ----------------------------------------------------------------------------------------- 1 node1.mycompany.com 192.168.1.10 node1.mycompany.com quorum-managerApply the GPFS license on the primary node using the
mmchlicensecommand.# mmchlicense server --accept -N node1 The following nodes will be designated as possessing GPFS server licenses: node1.mycompany.com mmchlicense: Command successfully completedStart GPFS on one of the nodes in the GPFS cluster using the
mmstartupcommand. If the Cluster configuration is successful, GPFS will start automatically on all the nodes.# mmstartup -a Sat Jul 20 01:13:26 IST 2013: mmstartup: Starting GPFS ...Check the status of the cluster using the
mmgetstatecommand.# mmgetstate -a Node number Node name GPFS state ------------------------------------------------------------ 1 Node1 activePerform the GPFS package installation on the second node and compile the source.
On node1, use the
mmaddnodecommand to add node2 to the clustermmaddnode -N node2Confirm that the node has been added to the cluster using the
mmlsclustercommand.# mmlscluster GPFS cluster information ======================== GPFS cluster name: node1.in.ibm.com GPFS cluster id: 7699862183884611001 GPFS UID domain: node1.in.ibm.com Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp GPFS cluster configuration servers: ------------------------------------------------ Primary server: node1.in.ibm.com Secondary server: node2.in.ibm.com Node Daemon node name IP address Admin node name Designation -------------------------------------------------------------------- 1 node1.in.ibm.com 192.168.1.10 node1.in.ibm.com quorum-manager 2 node2.in.ibm.com 192.168.1.11 node2.in.ibm.comUse the
mmchclustercommand to set node2 as the secondary configuration server.# mmchcluster -s node2 mmchcluster: GPFS cluster configuration servers: mmchcluster: Primary server: node1.in.ibm.com mmchcluster: Secondary server: node2.in.ibm.com mmchcluster: Command successfully completedSet the license mode for the node using the
mmchlicensecommand. Use a server license for this node.# mmchlicense server --accept -N node2 The following nodes will be designated as possessing GPFS server licenses: node2.in.ibm.com mmchlicense: Command successfully completed mmchlicense: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.Start node2 using the
mmstartupcommand.# mmstartup -N node2 Sat Jul 20 01:49:44 IST 2013: mmstartup: Starting GPFS ...Use the
mmgetstatecommand to verify that both nodes are in the active state.# mmgetstate -a Node number Node name GPFS state ------------------------------------------------------------- 1 node1 active 2 node2 active
Cluster file system configuration
Perform the following steps to configure the cluster file system.
Create a disk descriptor file / diskdesc.txt using the following format.
DDiskName:ServerList::DiskUsage:FailureGroup:DesiredName:StoragePool hdiskX:::dataAndMetadata::nsd1:Create the NSDs using the
mmcrnsdcommand.# mmcrnsd -F /diskdesc.txtNote: GPFS provides a block-level interface over TCP/IP networks called the NSD protocol. Whether using the NSD protocol or a direct attachment to the storage area network (SAN), the mounted file system looks the same to the users and the application (GPFS transparently handles I/O requests). A shared disk cluster is the most basic environment. In this configuration, the storage is directly attached to all the systems in the cluster. The direct connection means that each shared block device is available concurrently to all of the nodes in the GPFS cluster. Direct access means that the storage is accessible using a Small Computer System Interface (SCSI) or other block-level protocol using a SAN disk.
Create a file system using the
mmcrfscommand.# mmcrfs /gpfs fs1 -F diskdesc.txt -B 64kVerify that the file system has been created correctly using the
mmlsfscommand.# mmlsfs fs1 flag value description ------------------- ------------------------ ----------------------------------- -f 2048 Minimum fragment size in bytes -i 512 Inode size in bytes -I 8192 Indirect block size in bytes -m 1 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j cluster Block allocation type -D nfs4 File locking semantics in effect -k all ACL semantics in effect -n 32 Estimated number of nodes that will mount file system -B 65536 Block size -Q none Quotas enforced none Default quotas enabled --filesetdf no Fileset df enabled? -V 13.23 (3.5.0.7) File system version --create-time Thu Jul 18 22:09:36 2013 File system creation time -u yes Support for large LUNs? -z no Is DMAPI enabled? -L 4194304 Logfile size -E yes Exact mtime mount option -S no Suppress atime mount option -K whenpossible Strict replica allocation option --fastea yes Fast external attributes enabled? --inode-limit 102528 Maximum number of inodes -P system Disk storage pools in file system -d nsd1 Disks in file system --perfileset-quota no Per-fileset quota enforcement -A yes Automatic mount option -o none Additional mount options -T /gpfs Default mount point --mount-priority 0 Mount priorityMount the file system using the
mmmountcommand.# mmmount all -a Sat Jul 20 02:22:45 IST 2013: mmmount: Mounting file systems ...Verify that the file system is mounted using the
dfcommand.# df -g Filesystem GB blocks Free %Used Iused %Iused Mounted on /dev/hd4 2.00 1.56 22% 10221 3% / /dev/hd2 5.00 2.08 59% 52098 10% /usr /dev/hd9var 2.00 1.70 16% 7250 2% /var /dev/hd3 5.00 4.76 5% 262 1% /tmp /dev/hd1 0.50 0.50 1% 79 1% /home /dev/hd11admin 0.12 0.12 1% 5 1% /admin /proc - - - - - /proc /dev/hd10opt 25.00 5.72 78% 371887 21% /opt /dev/fs1 100G 282M 100G 1% /gpfs <-- This is the GPFS file SystemThe file system will be available automatically on both the systems.
node1:~ # df -g Filesystem GB blocks Free %Used Iused %Iused Mounted on /dev/hd4 2.00 1.56 22% 10221 3% / /dev/hd2 5.00 2.08 59% 52098 10% /usr /dev/hd9var 2.00 1.70 16% 7250 2% /var /dev/hd3 5.00 4.76 5% 262 1% /tmp /dev/hd1 0.50 0.50 1% 79 1% /home /dev/hd11admin 0.12 0.12 1% 5 1% /admin /proc - - - - - /proc /dev/hd10opt 25.00 5.72 78% 371887 21% /opt /dev/fs1 100G 282M 100G 1% /gpfs node1:~ # ssh node2 df -h Filesystem GB blocks Free %Used Iused %Iused Mounted on /dev/hd4 2.00 1.56 22% 10221 3% / /dev/hd2 5.00 2.08 59% 52098 10% /usr /dev/hd9var 2.00 1.70 16% 7250 2% /var /dev/hd3 5.00 4.76 5% 262 1% /tmp /dev/hd1 0.50 0.50 1% 79 1% /home /dev/hd11admin 0.12 0.12 1% 5 1% /admin /proc - - - - - /proc /dev/hd10opt 25.00 5.72 78% 371887 21% /opt /dev/fs1 100G 282M 100G 1% /gpfs node1:~ #Use the
mmdfcommand to get information on the file system.# mmdf fs1 disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 800 GB) nsd1 104857600 -1 yes yes 104569408 (100%) 94 ( 0%) ------------- -------------------- ------------------- (pool total) 104857600 104569408 (100%) 94 ( 0%) ============= ==================== =================== (total) 104857600 104569408 (100%) 94 ( 0%) Inode Information ----------------- Number of used inodes: 4041 Number of free inodes: 98487 Number of allocated inodes: 102528 Maximum number of inodes: 102528
Conclusion
GPFS cluster is useful in environments where we need high availability of file system. This paper provides guidelines to UNIX® administrators in installing and configuring GPFS cluster environment for AIX 5L cluster nodes, Linux cluster nodes, Microsoft Windows Server cluster nodes or heterogeneous cluster nodes for AIX, Linux, and Windows.