Tutorial

Seamlessly automate secure patch compliance for NIM clients in protected environments

Streamline patch management for isolated NIM clients in secure networks

By

Srikanth Thanneeru,

Akash Tripathi

Introduction

In a secure network environment, a Network Installation Management (NIM) server is designated as the central hub for network operations. Nim server is the only system connected to the internet, ensuring a controlled and secure point of access to external networks. Due to stringent security protocols, connected client systems are isolated from the internet to minimize exposure to potential threats.

When security patches or software updates are required in the connected NIM clients, the process is meticulously managed. The NIM server downloads the necessary files from trusted sources. These files are then transferred from the NIM server to the NIM client systems using any file transfer protocol or a physical medium if required. Once the files are on the client systems, the installation or updates are performed locally. This method ensures that the client systems receive the necessary updates while maintaining a high level of security by preventing direct internet access.

This setup is crucial for environments where data integrity and security are paramount, such as in financial institutions, governmental agencies, or other sensitive operations, where reducing the attack surface is a top priority.

This tutorial offers a comprehensive, step-by-step guide to programmatically obtaining NIM client information, retrieving necessary interim fixes, downloading them from a NIM server, and installing them on the NIM clients. Additionally, the tutorial demonstrates how to implement this process on a NIM server, enabling the automation of client machine management. By the end of this tutorial, you will have tools to automate the entire process, ensuring your NIM clients remain up-to-date with the updated software.

Retrieve and install suitable interim fixes for NIM client

In this section, you will learn how to retrieve NIM client information, find necessary interim fixes, and install them step-by-step.

  1. Retrieve the NIM client information from the NIM server.

    OSLEVEL=$(/usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT "/usr/bin/oslevel -s")
    output=$(/usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT /usr/sbin/prtconf)
    # Determine the System Model
    MTMVALUE=$(echo "$output" | grep "System Model" | awk -F': ' '{print $2}' | awk -F'[, ]' '{print $2}')
    # System firmware (booted) version
    SFIRMWAREVALUE=$(echo "$output" | grep "Platform Firmware level" | awk -F': ' '{print $2}')
    
  2. Create the POST request using the data collected in step 1 and save it as post_req.txt file.

    cat <<EOF > /post_req.txt
    POST /customercare/flrt/query?format=btext HTTP/1.0
    Host: esupport.ibm.com
    Accept: */*
    Content-Length: 178
    Content-Type: application/x-www-form-urlencoded
    
    reportname=FLRTReport 06 May 2024 03:19:54 AM&p0.mtm=$MTMVALUE&p0.fw=$SFIRMWAREVALUE&p0.parnm=$CLIENT&p0.os=aix&p0.aix=$OSLEVEL&format=btext
    EOF
    
  3. Send the POST request using the openssl command to esupport.ibm.com and save the response to an output file named FLRTReport.

    cat /post_req.txt | /usr/bin/openssl s_client -tls1_2 -quiet -crlf -connect esupport.ibm.com:443 -CApath /var/ssl_aix/certs > /FLRTReport
    

    Note: If the previous command fails to generate FLRTReport, one possible reason could be Content-Length: is different error. Please update Content-Length in the post_req.txt script.

    Extract the interim fixes and security.tar files from the FLRTReport.

    grep "aix/efixes/security" "/FLRTReport" | grep "tar" | tee "/SEC_IFIX" > /dev/null
    

    To determine the Common Vulnerability Scoring System (CVSS) base score and whether a restart is needed after installing a specific interim fix, download the detailed report from the esupport server and map the information using the respective authorized program analysis report (APAR) and advisory descriptions provided in the SEC_IFIX file.

    Sample script:

    cat <<EOF > /post_req2.txt
    POST /customercare/flrt/doc?page=aparCSVQ HTTP/1.0
    Host: esupport.ibm.com
    Accept: */*
    Content-Type: application/x-www-form-urlencoded
    
    reportname=FLRTReport format=btext
    EOF
    
    cat /post_req2.txt | /usr/bin/openssl s_client -tls1_2 -quiet -crlf -connect esupport.ibm.com:443 -CApath /var/ssl_aix/certs 1>/FLRTReportFULL 2>/dev/null
    
    echo "APAR\t|CVEID :CVSS Base Score\t|Reboot required\t|APAR Description\t| APAR download Link" > /details
    while read line
    do
       apar=$(echo $line | cut -f2 -d "~")
       adv_url=$(echo $line | cut -f4 -d "~")
       descr=$(echo $line | cut -f6 -d "~")
       dwld_url=$(echo $line | cut -f8 -d "~" |  sed 's/<br\/>//')
    
       echo "${dwld_url}" | grep -q ".tar" && echo "$dwld_url" >> "/recomd_ifix.lst" || echo "$adv_url" >> "/recomd_ifix.lst"
    
       CVSS_Score=$(cat /FLRTReportFULL | grep "$descr" | grep "$apar" | sed 's/"//g' | awk -F',' '{print $14}' | sort | uniq | tail -n 1 )
       Reboot_required=$(cat /FLRTReportFULL | grep "$descr" | grep "$apar" | sed 's/"//g' | awk -F',' '{print $15}' | sort | uniq | tail -n 1)
    
       echo "$apar\t|$CVSS_Score\t|$Reboot_required\t|$descr\t| $dwld_url" >> /details
    done < /SEC_IFIX
    
    cat /details
    

    Sample output:

    APAR    |CVEID :CVSS Base Score        |Reboot required        |APAR Description       | APAR download Link
    IJ26684 |CVE-2020-11868:5.9 / CVE-2020-13817:7.4 / CVE-2020-15025:4.4   |no     |There are vulnerabilities in NTPv4 that affect AIX.    |https://aix.software.ibm.com/aix/efixes/security/ntp_fix13.tar
    IJ48618 |CVE-2023-45171:6.2 / CVE-2023-45175:6.2        |yes    |AIX is vulnerable to denial of service vulnerabilities - kernel        | https://aix.software.ibm.com/aix/efixes/security/kernel_fix6.tar
    
  4. Verify if the interim fix is already installed on the NIM client using the APAR number from the previous step. If not installed, download the interim fix using the following command:

    /usr/sbin/emgr_download_ifix -L $(echo $line | awk '{print $NF}') -P $IFIX_FOLDER
    

    Parameters and variables:

    • -L: Specifies the link from which the interim fix tar file is downloaded.
    • IFIX_FOLDER: Specifies the folder where the interim fix is to be downloaded by the user.

    Sample command:

    /usr/sbin/emgr_download_ifix -L https://aix.software.ibm.com/aix/efixes/security/ntp_fix13.tar -P /test_folder
    
  5. Transfer the downloaded interim fixes from NIM server to client using the following commands:

    nim -o define -t file_res -a location=$IFIX_FOLDER -a dest_dir=$IFIX_FOLDER -a server=master secure_efix_folder
    nim -o allocate -a file_res=secure_efix_folder $CLIENT
    nim -o cust $CLIENT
    
  6. Run the emgr_sec_patch command on the client to install the interim fixes:

    /usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT "PATH=/usr/bin:$PATH /usr/sbin/emgr_sec_patch $IFIX_FOLDER/$file
    

    Parameters and variables:
    file: Specifies the name of the interim fix tar file

    Sample command:

    /usr/lpp/bos.sysmgt/nim/methods/c_rsh aix-install-zep01-lp009 PATH=/usr/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin /usr/sbin/emgr_sec_patch /POC/test_poc/./10355058/kernel_fix6.tar
    

Automate patch management with a custom script

This section demonstrates how to automate the entire process using a script that simplifies the retrieval, transfer, and installation of interim fixes on NIM clients.

Create a script named FLRT with the following content and then run the script.

Sample FLRT script:

cat ./FLRT
#!/bin/ksh
CLIENT=$1
IFIX_FOLDER=$2
mkdir -p $IFIX_FOLDER

# need temporary files to hold inventory info
EMGRL=$IFIX_FOLDER/emgrl.out
APAR_LIST=$IFIX_FOLDER/apar_list.out

OSLEVEL=$(/usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT "/usr/bin/oslevel -s")
# data collected, write to inventory file
output=$(/usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT /usr/sbin/prtconf)
# Determine the System Model
MTMVALUE=$(echo "$output" | grep "System Model" | awk -F': ' '{print $2}' | awk -F'[, ]' '{print $2}')
# System firmware (booted) version
SFIRMWAREVALUE=$(echo "$output" | grep "Platform Firmware level" | awk -F': ' '{print $2}')


/usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT '/usr/sbin/emgr -l' > $EMGRL
cat $EMGRL | grep -p "STATE LABEL" | tail -n +3 | sed '$d' | awk '{print $3}' > $APAR_LIST
i=0
while read line
do
   apar_list[$i]=$line
   let i=i+1
done < $APAR_LIST

cat <<EOF > $2/post_req.txt
POST /customercare/flrt/query?format=btext HTTP/1.0
Host: esupport.ibm.com
Accept: */*
Content-Length: 158
Content-Type: application/x-www-form-urlencoded

reportname=FLRTReport 06 May 2024 03:19:54 AM&p0.mtm=$MTMVALUE&p0.fw=$SFIRMWAREVALUE&p0.parnm=$CLIENT&p0.os=aix&p0.aix=$OSLEVEL&format=btext
EOF

cat $IFIX_FOLDER/post_req.txt | /usr/bin/openssl s_client -tls1_2 -quiet -crlf -connect esupport.ibm.com:443 -CApath /var/ssl_aix/certs 1>$IFIX_FOLDER/FLRTReport 2>/dev/null
# get the list of all Security ifix
grep "aix/efixes/security" "$IFIX_FOLDER/FLRTReport" | grep "tar" | tee "$IFIX_FOLDER/SEC_IFIX" > /dev/null


cat <<EOF > $2/post_req2.txt
POST /customercare/flrt/doc?page=aparCSVQ HTTP/1.0
Host: esupport.ibm.com
Accept: */*
Content-Type: application/x-www-form-urlencoded

reportname=FLRTReport format=btext
EOF
cat $IFIX_FOLDER/post_req2.txt | /usr/bin/openssl s_client -tls1_2 -quiet -crlf -connect esupport.ibm.com:443 -CApath /var/ssl_aix/certs 1>$IFIX_FOLDER/FLRTReportFULL 2>/dev/null


echo "APAR\t|CVEID :CVSS Base Score\t|Reboot required\t|APAR Description\t| APAR download Link" > $IFIX_FOLDER/details
while read line
do
   apar=$(echo $line | cut -f2 -d "~")
   adv_url=$(echo $line | cut -f4 -d "~")
   descr=$(echo $line | cut -f6 -d "~")
   dwld_url=$(echo $line | cut -f8 -d "~" |  sed 's/<br\/>//')

   # Check whether ifix is already installed or not
   echo "${apar_list[@]}" | grep $apar >>/dev/null
   if [ $? -eq 0 ]; then
      continue
   fi

   echo "${dwld_url}" | grep -q ".tar" && echo "$dwld_url" >> "$IFIX_FOLDER/recomd_ifix.lst" || echo "$adv_url" >> "$IFIX_FOLDER/recomd_ifix.lst"

   CVSS_Score=$(cat $IFIX_FOLDER/FLRTReportFULL | grep "$descr" | grep "$apar" | sed 's/"//g' | awk -F',' '{print $14}' | sort | uniq | tail -n 1 )
   Reboot_required=$(cat $IFIX_FOLDER/FLRTReportFULL | grep "$descr" | grep "$apar" | sed 's/"//g' | awk -F',' '{print $15}' | sort | uniq | tail -n 1)

   echo "$apar\t|$CVSS_Score\t|$Reboot_required\t|$descr\t| $dwld_url" >> $IFIX_FOLDER/details
done < $IFIX_FOLDER/SEC_IFIX

cat $IFIX_FOLDER/details

noOfIfix=$(wc -l $IFIX_FOLDER/recomd_ifix.lst | awk '{print $1}')
count=1
while read line
do
   echo "\nDownloading $count of $noOfIfix ..."

   # call emgr_download_ifix to download
   /usr/sbin/emgr_download_ifix -L $line -P $IFIX_FOLDER

   let count=count+1
done < "$IFIX_FOLDER/recomd_ifix.lst"

if [[ $IFIX_FOLDER = [/~]* ]] ; then
   echo "Absolute path: $IFIX_FOLDER"
else
   IFIX_FOLDER=$(echo "$(pwd)/${IFIX_FOLDER}" | sed 's:/\{2,\}:/:g')
fi

nim -o remove secure_efix_folder 2>/dev/null

#create a new file_res NIM resource, called secure_efix_folder
nim -o define -t file_res -a location=$IFIX_FOLDER -a dest_dir=$IFIX_FOLDER -a server=master secure_efix_folder

# allocate the new resource to the NIM client
nim -o allocate -a file_res=secure_efix_folder $CLIENT
#copy the ifixes to the client form server and give the required permission to the folder
nim -o cust $CLIENT
/usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT "/usr/bin/chmod -R 777 $IFIX_FOLDER"

tar_file=`/usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT "cd $IFIX_FOLDER ; /usr/bin/find . -type f -name \"*.tar\" "`

#install the ifixes into the NIM client
for file in $tar_file; do
   /usr/lpp/bos.sysmgt/nim/methods/c_rsh $CLIENT "PATH=/usr/bin:$PATH /usr/sbin/emgr_sec_patch $IFIX_FOLDER/$file"
done

exit 0

Now run the FLRT script using the following command:

./FLRT<NIM_client> <Efix_download_folder>

Sample script:

./FLRT aix-install-zep01-lp009 poc_test

Sample output:

APAR    |CVEID :CVSS Base Score        |Reboot required        |APAR Description       | APAR download Link
IJ26684 |CVE-2020-11868:5.9 / CVE-2020-13817:7.4 / CVE-2020-15025:4.4   |no     |There are vulnerabilities in NTPv4 that affect AIX.    | https://aix.software.ibm.com/aix/efixes/security/ntp_fix13.tar
IJ48618 |CVE-2023-45171:6.2 / CVE-2023-45175:6.2        |yes    |AIX is vulnerable to denial of service vulnerabilities - kernel        | https://aix.software.ibm.com/aix/efixes/security/kernel_fix6.tar

Downloading 1 of 2 ...
Downloading fix: https://aix.software.ibm.com/aix/efixes/security/ntp_fix13.tar
+-----------------------------------------------------------------------------+

Performing certificate verification ...
OpenSSL success!
Interim fix ntp_fix13.tar has been downloaded to poc_test directory.
+-----------------------------------------------------------------------------+

Downloading 2 of 2 ...
Downloading fix: https://aix.software.ibm.com/aix/efixes/security/kernel_fix6.tar
+-----------------------------------------------------------------------------+

Performing certificate verification ...
OpenSSL success!
Interim fix kernel_fix6.tar has been downloaded to poc_test directory.

Downloaded Tar file name is: /poc_test/./ntp_fix13.tar
+-----------------------------------------------------------------------------+
Verifying contents of /poc_test/./ntp_fix13.tar
+-----------------------------------------------------------------------------+
+-----------------------------------------------------------------------------+
Verifying integrity of Advisory.asc
+-----------------------------------------------------------------------------+
Advisory.asc integrity verification passed
+-----------------------------------------------------------------------------+
Checking System Level Prerequisites
+-----------------------------------------------------------------------------+
Skipping ifix

Downloaded Tar file name is: /poc_test/./ kernel_fix6.tar
+-----------------------------------------------------------------------------+
Verifying contents of /poc_test/./kernel _fix6.tar
+-----------------------------------------------------------------------------+
+-----------------------------------------------------------------------------+
Verifying integrity of Advisory.asc
+-----------------------------------------------------------------------------+
Advisory.asc integrity verification passed
+-----------------------------------------------------------------------------+
Checking System Level Prerequisites
+-----------------------------------------------------------------------------+
Skipping ifix

Summary

This tutorial outlined how to identify, download, and install the required interim fixes for the IBM AIX system when the NIM client is connected to the outside network only through the NIM server. Additionally, a script is provided to automate the entire process, streamlining patch management in secure environments.

References

To learn more about this topic, explore the following resources: