Data is the new oil
Like oil has developed the world into a better place data perhaps holds similar potential and is already responsible for most valuable brands in the IT industry. These companies are collecting tremendous data from billions of users and analyzing it to make gadgets and products even more smarter. But what if the data get leaked? According to a cyber-security research firm, recently more than 540 million records about Facebook users were publicly exposed on Amazon’s cloud computing service. Such scenarios governance the importance of Data Security.
(source: https://edition.cnn.com/2019/04/03/tech/facebook-records-exposed-amazon/index.html)

IBM Spectrum Scale, a high performance computing file system provides secure data at rest and secure data in motion (on wire).

Secure data at rest can be achieved through file system encryption provided by Spectrum Scale (in advanced edition). When file-system encryption is configured the data (not metadata) is secured at rest as well as in transit (i.e., data flow within the gpfs cluster).
If one wants to secure data in motion within a given spectrum scale cluster without configuring file-system encryption feature, one is required to make use of the cipherList parameter via mmchconfig (available in all Spectrum Scale editions).

This blog will demonstrate how to enable secure data in motion within Spectrum Scale cluster (inter-node communication) and how to verify that data is secured over wire using tcpdump. This is required by the security auditors.
Before we configure the cipherList parameter, let’s quickly understand the security mode of a cluster. The security mode of a cluster determines the level of security that the cluster provides for communications between nodes in the cluster and also for communications between clusters.

There are three security modes:

  1. EMPTY
  2. In this mode both receiving and the sending nodes do not authenticate each other, do not encrypt transmitted data and do not check the integrity of transmitted data.

  3. AUTHONLY
  4. This is the default security mode you will find on cluster (in IBM Spectrum Scale v4.2 or later). The sending and receiving nodes authenticate each other with a TLS handshake and then close the TLS connection. Communication continues in the readable format. The nodes do not encrypt transmitted data and do not check data integrity.

  5. Cipher
  6. This is the most secure mode among all. The sending and receiving nodes authenticate each other with a TLS handshake. A TLS connection is established. The transmitted data is encrypted with the specified cipher and is checked for data integrity.To set this mode, you must specify the name of a supported cipher, such as AES128-GCM-SHA256.

In nutshell,
Feature | EMPTY | AUTHONLY | CipherList
————————————————————-
Authentication | No | Yes | Yes
Data Integrity Check | No | No | Yes
Encryption | No | No | Yes
————————————————————-

To identify the current security mode of cluster run the following command
mmlsconfig cipherlist
You can change the security mode with the following command:
mmchconfig cipherlist=security_mode
Note: If you are changing the security mode from EMPTY to another mode, you can do so without stopping the GPFS daemon. However, if you are changing the security mode from another mode to EMPTY, you must stop the GPFS daemon on all the nodes in the cluster. Change the security mode to EMPTY and then restart the GPFS daemon.

Let’s start changing the security mode & sniffing the packets of Spectrum Scale communication in following 3 activities.

  1. Running Spectrum Scale command (mm commands like mmlscluster)
  2. Listing files on Spectrum Scale file system
  3. Write data to file on Spectrum Scale file system

Setup:
Prerequisite:

  • Spectrum Scale cluster (any edition)
  • Tcpdump (Network sniffer tool)
  • Wireshark (Optional, Packet Analyzer tool)

Steps:
1. Identify the interface of Spectrum Scale communication
[root@AaryaStark ~]# mmlscluster

Node Daemon node name IP address Admin node name Designation
——————————————————————-
1 nsd0 172.16.0.14 nsd0 quorum-manager-perfmon
2 nsd1 172.16.0.12 nsd1 quorum-manager-perfmon
3 nsd2 172.16.0.13 nsd2 quorum-manager-perfmon
4 ces0 172.16.0.10 ces0 perfmon
5 ces1 172.16.0.11 ces1 perfmon
[root@AaryaStark ~]# ip a

12: ib1: mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband 80:00:04:04:fe:80:00:00:00:00:00:00:00:02:c9:02:00:29:d2:41 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 172.16.0.10/16 brd 172.16.255.255 scope global ib1
valid_lft forever preferred_lft forever
inet6 fe80::ca27:c8e5:8f43:1ccc/64 scope link
valid_lft forever preferred_lft forever

2. Start tcpdump tool on interface through which Spectrum Scale communication is happening; Filter it to trace only Spectrum Scale communication (port 1191) in separate window/tab
Suntax: tcodump -i -w port [root@AaryaStark ~]# tcpdump -i ib1 -w tcpdumpFile.pcap port 1191

3. Complete all the below activities and them stop the tcpdump (it will automatically save the tcpdump file in local directory)

4. Open tcpdump file in wireshark (File > Open > tcpdumpFile.pcap)

5. To find data into tcpdump use Find Packet utility of wireshark (Edit > Find Packet); Search for string from the output you saw on command line

Demo 1: Cluster with AUTHONLY Security Mode
Task 0: Verify current Security Mode

[root@AaryaStark ~]# mmlsconfig
Configuration data for cluster elastic.scale:
———————————————
clusterName elastic.scale
clusterId 148294699616067838
autoload yes
profile gpfsProtocolDefaults
dmapiFileHandleSize 32
minReleaseLevel 5.0.3.0
ccrEnabled yes
cipherList AUTHONLY
maxblocksize 16M

Task 1: Running Spectrum Scale command (mmlscluster)

[root@AaryaStark ~]# mmlscluster
GPFS cluster information
========================
GPFS cluster name: elastic.scale
GPFS cluster id: 148294699616067838
GPFS UID domain: elastic.scale
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR
Node Daemon node name IP address Admin node name Designation
——————————————————————-
1 nsd0 172.16.0.14 nsd0 quorum-manager-perfmon
2 nsd1 172.16.0.12 nsd1 quorum-manager-perfmon
3 nsd2 172.16.0.13 nsd2 quorum-manager-perfmon
4 ces0 172.16.0.10 ces0 perfmon
5 ces1 172.16.0.11 ces1 perfmon

Screenshot from wireshark to verify the data is in human readable format

Task 2: Listing files on Spectrum Scale file system

[root@AaryaStark ~]# ls /ibm/cesSharedRoot/object/keystone/
Base object.lock pg_hba.conf pg_log pg_notify pg_snapshots pg_subtrans pg_twophase pg_xlog postmaster.opts global pg_clog pg_ident.conf pg_multixact pg_serial pg_stat_tmp pg_tblspc PG_VERSION postgresql.conf

Screenshot from wireshark to verify the data is in human readable format

Task 3: Read data to file on Spectrum Scale file system

[root@AaryaStark ~]# cat /ibm/fs1/test_file
This is the file open for writing on filesystem fs1

Screenshot from wireshark to verify the data is in human readable format

Changing Security mode from AUTHONLY to Cipher

[root@AaryaStark ~]# mmchconfig cipherList=AES256-SHA256
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

Demo 2: Cluster with Cipher Security Mode
Task 0: Verify current Security Mode

[root@AaryaStark ~]# mmlsconfig
Configuration data for cluster elastic.scale:
———————————————
clusterName elastic.scale
clusterId 148294699616067838
autoload yes
profile gpfsProtocolDefaults
dmapiFileHandleSize 32
minReleaseLevel 5.0.3.0
ccrEnabled yes
maxblocksize 16M
[common] cesSharedRoot /ibm/cesSharedRoot
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
cipherList AES256-SHA256
adminMode central

After enabling Encryption Security Mode wireshark shows all the communication (metadata as well as data) in encrypted format which is non human readable. To further prove that data in motion is secured (encrypted) we use the wireshark feature to explicitly search for the string which is part of the communication between the nodes inside tcpdump. (See task 3) .

Task 3: Write data to file on Spectrum Scale file system
[root@AaryaStark ~]# cat /ibm/fs1/test_file1
This is the file open for writing on encrypted filesystem fs1

Screenshot from wireshark to verify the data is NOT in human readable format

Wireshark was not able to identify a single string which was the part of communication which proves that data communicated over wore was encrypted

Recommendation

  • If you have need of ensuring secure data in transit within Spectrum Scale cluster then make use of security mode (cipherList=Encryption_type) provided by Spectrum Scale.
  • If you are using filesystem encryption then data (not metadata) in transit will alway be Encrypted format irrespective of Security mode.

Note: When one makes use of cipherList=Encyption_type, there is a performance impact which depends on your workload. If you are running scale with trusted network then AUTHONLY suffice the work.

PS: Thanks Felipe(knop@us.ibm.com) for your acknowledgement
(Views presented here are my own and not of my employer’s)

Join The Discussion

Your email address will not be published. Required fields are marked *