Perform live kernel updates with active Internet Protocol Security (IPsec) tunnels

Protecting data in flight (such as in network traffic) is essential in any regulated environment, and IPsec is one way to achieve this. IPsec provides data protection through built-in authentication, encryption, and integrity mechanisms. Hence, we must keep the IPsec running all the time including service window and system upgrades. IBM AIX Live Update is one of the mechanisms to help updating the AIX operating system without application downtime which includes operating system or its subsystem downtime. IPsec is one of the operating systems’ subsystem and it is essential to keep this subsystem running during a live update operation.

Introduction

Internet Protocol Security (IPsec) is an open, standard security technology developed by the Internet Engineering Task Force (IETF). IPsec provides cryptography-based protection of all data at the IP layer of the communications stack. No changes are needed for existing applications. IPsec is the industry-standard network-security framework chosen by the IETF for both IP version 4 (IPv4) and IP version 6 (IPv6) environments.

This blog explains how to perform live update or live kernel update with an active or running IPsec tunnels.

Note: In this blog, live update and live kernel updates are used interchangeably and can be consider synonymous.

IPSec protects your data traffic using the following cryptographic techniques:

• Authentication
• Encryption
• Integrity

IPsec and live update

Currently, IPsec is not live kernel update aware, meaning you cannot run live kernel update if there are IPsec tunnels or IPsec filters (static) configured on the LPAR. To make this work, users must manually bring down the IPsec tunnels, IPsec filters, and its respective components before starting live kernel update (using the geninstall –k command), otherwise the live update verification will fail with the following error statement:

1430-243 Unable to do the preview check for Check_active_ipsec.

IPsec has multiple kernel extensions and user space daemons, and all of these need to be brought down before running the live kernel update operation.

In order to keep IPsec alive, the entire IPsec subsystem which includes daemons and kernel extensions should be live kernel update aware. This is because, the subsystem maintains session data with respect to other end points, especially filter table and tunnel cache [which include source and destination Security Parameters Index (SPI), keys, tunnel lifetime, and so on]. Unless live kernel update migrates the state data from original to surrogate LPAR, we cannot retain the IPsec tunnels.   The following solution was implemented to run live kernel update with IPsec tunnels configured and active:

• Making IPsec as a live kernel update aware solution requires a reasonable amount effort and time. Therefore, as a short-term solution, a workaround or automation was introduced as part of live kernel update framework to handle the IPsec subsystem during live kernel update. This functionality was introduced as part of IBM AIX 7.3 TL3. A new user configurable attribute (ipsec_auto_migrate) is introduced as part of the live update file “/var/adm/ras/liveupdate/lvupdate.data”.
• If the user sets the ipsec_auto_migrate parameter to yes, then during the live kernel update prephase, IPsec will be disabled on the original logical partition (LPAR) which includes deleting IPsec tunnels and stopping the IPsec daemons. During the live kernel update completion phase, IPsec will be started on the surrogate LPAR which includes starting the daemons and activating the IPSec tunnels.
• Setting the ipsec_auto_migrate value to no, will fall back to the old behavior where live kernel update fails if there is a running IPSec subsystem.
• The default behavior is ipsec_auto_migrate=no.

Note: When the ipsec_auto_migrate option is set, instead of a user taking care of bringing down the IPsec subsystem before running live kernel update, the live kernel update itself will take care of tearing down IPsec and its subsystem, complete the live kernel update, and then bring back the exact IPsec configuration after live kernel update. As this is going to bring down the IPsec tunnel, there is a high chance that the traffic may go as plain text during live kernel update for a very short time, and this may not be acceptable in all environments. Hence, it is recommended to perform a careful study with IPsec and live kernel update before using this feature.

Change in live kernel update stages if ipsec_auto_migrate is set to yes

Live kernel update stages on source and surrogate LPARs

Source:

• ts1: User initiates a live kernel update operation at the source LPAR.
• ts2: Live kernel update infrastructure calls the CHECK phase. No preview check for IPSec at this stage.
• ts3: Live kernel update infrastructure calls the PRE phase. Incase IPsec is enabled, the live kernel update infrastructure brings down the tunnels and stops the IPsec daemons.
• ts4: Live kernel update freezes application and network.
• ts5: Live kernel update flushes the file system and makes them read only.

Surrogate:

• tt1: Data volume groups are varied on via live kernel update process.
• tt2: Live kernel update process mount the file system in the read-only mode.
• tt3: Live kernel update unfreezes applications, restarts the network, and mounts the file system in read/write mode.
• tt4: Live kernel update calls the POST phase of scripts.
• tt5: The live update operation is completed, and live kernel update calls the COMPLETE phase of scripts. Incase IPSec was running on source restore the IPSec tunnels.

Enable/Disable the ipsec_auto_migrate option

The “/var/adm/ras/liveupdate/lvupdate.template” file contains the latest description of all possible fields for the live update operation, and note that the file has been updated with the description for the new option, ipsec_auto_migrate, as shown below:

#      ipsec_auto_migrate =  <yes | no>  Blank defaults to no.  If yes, the
#      live update operation will be attempted with IPSEC enabled.
#      If no, preview check will be done for IPSEC, and if IPSEC is enabled,
#      lku will fail

To enable IPsec during live kernel update, users need to add the ipsec_auto_migrate attribute in the “/var/adm/ras/liveupdate/lvupdate.data” configuration file.

For example:

general:
kext_check = yes
ipsec_auto_migrate = yes
disks:
nhdisk  = hdisk1
mhdisk  = hdisk2
tohdisk = hdisk3
tshdisk =
hmc:
lpar_id  =
management_console = p9-hmc.aus.stglabs.ibm.com
user = hscroot

Limitations and risks of enabling the ipsec_auto_migrate option

• The Live Update supports Internet Protocol Security (IPsec) through the ipsec_auto_migrate option. This option is available for live kernel update for AIX 7.3 TL3 or later versions only.
• IPSec-enabled live kernel update feature is supported for IKEv2 (internet key exchange) version only.
• In between the live kernel update pre and complete phases, there will be a small window where the traffic may flow as a plain text.
• In case of on-demand tunnels, there is a small timing window where IPsec is getting shut down, and traffic from the remote end may try to bring up the tunnel again. In such situations, users may see encrypted traffic flowing from the remote end and plain text from the local end until fresh IPSec tunnels are re-established. 
• This feature provides an interim solution, where IPsec will be shut down on the original LPAR and started on the surrogate LPAR. As a result, the IPSec tunnel states are not going to be the same or intact, instead a fresh IPSec tunnel will be established. 
• There is a possibility that some tunnel which were inactive on the original LPAR may become active on the surrogate LPAR. Currently, the IPsec enabling script does not have the intelligence to not activate the tunnels on the surrogate LPAR which were inactive on the original LPAR. In future, this issue may get fixed, but for now it is a limitation. 
• During IPsec tunnels enablement in the live kernel update complete phase, tunnels will be activated from the LPAR where live kernel update is performed.
• In case of tunnel configuration such as IKE_Role or IPSec_Role set to Responder on the local LPAR or the remote end set as Initiator, tunnels will not be active post live kernel update. Users must manually initiate tunnels from the remote end point.
• In case NAT rules do not allow the live kernel update LPAR to initiate the tunnels, the tunnels will not be active post live kernel update. Users must manually initiate the tunnels from the remote end point.

Summary

When the ipsec_auto_migrate option is set in the /var/adm/ras/liveupdate/lvupdate.data file, the live kernel update operation will work with IPsec enabled, and the preview check for live kernel update will be successful. After the live kernel update operation is successful on the surrogate LPARs, the IPsec tunnels will be configured and become active.