Use case : In a GPFS stretch cluster / GPFS synchronous mirroring setup (which consist of 2 sites with 3rd site as tie breaker)  with shared storage deployment model ( block based storage at the backend) one is required to upgrade the setup due to EOL or other issues in a typical IBM PureApp based installation (which is based on server and not CCR ).
i) All the physical NSD servers on both side OR
ii) Entire Backend block based storage on both the sites OR
iii) both i) and ii)

Note: PureApp approach is that it “treats” spectrum scale systems as racks where upgrade requires to “replace” racks”.
Ideally, in Scale world, one would add the new servers and nsds to the cluster, and then either use mmrpldisk, or “add new disks–>suspend_old_ones–>restripe” – which will do the migration in “one round” instead of several rounds like needed to do with PureApp based setup explained below.
So, for non Pureapp based installation the process would be much easier -> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1adm_rpldisk.htm

Below are the guideline steps for up-gradation of such a setup.

In the below set of steps, there is a Spectrum Scale sync replication setup (also sometimes called as stretch cluster). The cluster has two sites , one called as Primary and other as Secondary. The problem definition is : How to migrate the Old/existing deployment (called as OldPrimary and OldMirror) with new Deployment (NewPrimary and NewMirror) which has an upgraded stack of Spectrum Scale nodes along with Block storage and operating systems). The upgrade steps below are planned in such a way that the OldMirror is upgraded to be NewPrimary and OldPrimary is upgraded to be NewMirror).

Representation is as follows:

   OldPrimary -> P1
   OldMirror  -> P2
   NewPrimary -> P3
   NewMirror  -> P4

Steps to reflect the fact that the OldMirror is being replaced by the NewPrimary and not the NewMirror. So this is the sequence –

   P1->prmary
   P2->mirror
   P2 replaced with P3
   P3->primary
   P1->mirror
   P1 replaced with P4
   P3->primary
   P4->mirror

1. Ensure file system operation with only one replica
   a. Remember value of strict replication:
         mmlsfs -K  
   b. Turn off strict replication, otherwise file creates, appends might get error:
         mmchfs -K no
   c. Verify
         mmlsfs -K

2. Remove P2 Disks
   a. Delete all disks in P2:
¬†¬†¬†¬†¬†¬†¬†¬† mmdeldisk ” -m
   b. Verify:
¬†¬†¬†¬†¬†¬†¬†¬† mmlsdisk -d “”¬†¬†
   c. Delete all nsds from the P2:
¬†¬†¬†¬†¬†¬†¬†¬† mmdelnsd “
   d. Verify:
¬†¬†¬†¬†¬†¬†¬†¬† mmlsnsd -d “

3. Remove P2 Nodes
   a. Move primary and secondary cluster configuration node to P1:
         mmchcluster -p -s
Note: If the cluster is using server-based repository which is kind of legacy and not CCR based) then they proceed with step 3a otherwise they
can skip to step 3c.
   b. Verify:
         mmlscluster
   c. Move cluster manager role to node in P1:
         mmchmgr -c
   d. Verify:
         mmlsmgr -c     
   e. Move fs manager role to node in P1:
         mmchmgr
   f. Verify:
         mmlsmgr
   g. Remove quorum and manager designations from P2 Nodes:
¬†¬†¬†¬†¬†¬†¬†¬† mmchnode –noquorum –nomanager -N
   h. Verify:
         mmlscluster
i. Stop any applications that are using file systems, then cleanly unmount the file system
mmumount all -N

   j. Shutdown gpfs on the P2 nodes: (This may return error if there are processes still using the file system; so retry until it returns successfully)
         mmshutdown -N

   k. Verify:
         mmgetstate -N
   l. Delete all nodes in P2:
         mmdelnode -N
   m. Verify:
         mmlsnode

4. Add P3 Nodes
   a. Add all nodes in P3:
         mmaddnode -N
Note: After mmaddnode new nodes, accept nodes licenses before proceeding.
   b. Verify:
         mmlsnode
   c. Start P3 Nodes:
          mmstartup -N
   d. Verify:
         mmgetstate -N

5. Add P3 Disks
   a. Create NSDs out of all disks in P3:
          mmcrnsd -F
   b. Verify
¬†¬†¬†¬†¬†¬†¬†¬† mmlsnsd -d “
   c. Add the NSDs in P3 making the P3 nodes as NSD servers for the disks:
         mmadddisk -F
   d. Verify:
¬†¬†¬†¬†¬†¬†¬†¬† mmlsdisk -d “”¬†¬†
   e. Restore replication:
         mmrestripefs -r -N       
   f. Mount the file system on the P3 nodes:
         mmmount -a
   g. Verify:
         mmlsmount

6. (Non-gpfs operation) Swap Primary and Mirror designations so that:
   P3 -> Primary
   P1 -> Mirror

7. Remove P1 Disks
   a. Delete all disks in P1:
¬†¬†¬†¬†¬†¬†¬†¬† mmdeldisk ” -m
   b. Verify:
¬†¬†¬†¬†¬†¬†¬†¬† mmlsdisk -d “”¬†¬†
   c. Delete all nsds from the P1:
¬†¬†¬†¬†¬†¬†¬†¬† mmdelnsd “
   d. Verify:
¬†¬†¬†¬†¬†¬†¬†¬† mmlsnsd -d “

8. Remove P1 Nodes
   a. Move primary and secondary cluster configuration node to P3:
         mmchcluster -p -s
   b. Verify:
         mmlscluster
   c. Move cluster manager role to node in P3:
         mmchmgr -c
   d. Verify:
         mmlsmgr -c     
   e. Move fs manager role to node in P3:
         mmchmgr
   f. Verify:
         mmlsmgr
   g. Remove quorum and manager designations from P1 Nodes:
¬†¬†¬†¬†¬†¬†¬†¬† mmchnode –noquorum –nomanager -N
   h. Verify:
         mmlscluster
   i. Shutdown gpfs on the P1 nodes: (loop until it completes successfully)
         mmshutdown -N
   j. Verify:
         mmgetstate -N
   k. Delete all nodes in P1:
         mmdelnode -N
   l. Verify:
         mmlsnode

9. Add P4 Nodes
   a. Add all nodes in P4:
         mmaddnode -N
   b. Verify:
         mmlsnode
   c. Start P4 Nodes:
         mmstartup -N
   d. Verify:
         mmgetstate -N

10. Add P4 Disks
   a. Create NSDs out of all disks in P4:
         mmcrnsd -F
   b. Verify
¬†¬†¬†¬†¬†¬†¬†¬† mmlsnsd -d “
   c. Add the NSDs in P4 making the P4 nodes as NSD servers for the disks:
         mmadddisk -F
   d. Verify:
¬†¬†¬†¬†¬†¬†¬†¬† mmlsdisk -d “”¬†¬†
   e. Restore replication:
         mmrestripefs -r -N        
   f. Mount the file system on the P4 nodes:
         mmmount -a
   g. Verify:
         mmlsmount

10. Restore state of strict replication:
   a. Restore state of strict replication:
         mmchfs -K
   b. Verify
         mmlsfs -K

Acknowledgement: Thanks to Asmahan A Ali, Tomer Perry, Ke KZ Zhu and Sandeep Patil for their review.

Join The Discussion

Your email address will not be published. Required fields are marked *