The National Institute of Standards and Technology (NIST) defines a Bastion host as a specialized computer on a network that is specifically designed and configured to withstand attacks.

On IBM Cloud, the Privileged Access Gateway (PAG) is a managed service that provides a Bastion gateway server, serving as a single entry point to a set of customer servers and clusters. PAG is one of the solutions offered under IBM Cloud Security Network Access Guidance. All human interactions with servers on IBM Cloud must be routed through PAG. Each access session by a human operator to a Virtual Server or Kubernetes Cluster is recorded and stored in IBM Cloud Object Storage for auditing purposes.

In this tutorial, learn how to add Bastion support to your Kubernetes Clusters and Virtual Servers using the PAG Standard Plan in IBM Cloud.

Adding Bastion support to IBM Kubernetes (IKS) clusters

Before creating a Privileged Access Gateway (PAG), ensure you have completed the following prerequisites. Refer to the documentation for detailed instructions on these steps.

Set up a Virtual Private Cloud (VPC) with subnets enabled for a public gateway.
Configure a Virtual Private Network (VPN) Client-to-Server connection.
Create an IBM Cloud Object Storage (COS) bucket.
Configure the necessary Access Management roles.

The following architecture diagram illustrates how the VPN, PAG, and COS are interconnected.

alt

The state diagram follows:

State Diagram

Provisioning a virtual private network

To allow the VPN server to access the IBM Secrets Manager, follow the steps for Setting up client-to-site authentication for both VPN and Secrets Manager access. This involves creating an IAM service-to-service authorization to configure authentication settings between the VPN server and Secrets Manager. The VPN server and clients are authenticated using certificates managed through IBM Cloud® Secrets Manager.

Create the VPN server and client certificates as described in Managing VPN server and client certificates documentation. You can generate these certificates using OpenVPN Easy-RSA, Let's Encrypt, or by using the Secrets Manager Private Certificate feature.

Configuring Secrets Manager Private Certificate

To create a private certificate using IBM Secrets Manager, follow these steps:

Set up a root Root Certificate Authority (CA), an intermediate CA, and add a certificate template.
Go to the Secrets Manager and select the Certificate Authority and the certificate template created in the previous step.

After creating the VPN server and client certificates, grant permission to connect to the VPN server by following the appropriate instructions.

Next, create a Virtual Private Network (Client to Server) connection. Under the Authentication settings, select the VPN server and client certificates, using the Secrets Manager Private Certificate.

VPN Server

Modifying security group rules

Modify the security group created by the VPN. You can find the security group details in the VPN server configuration. Ensure the security group rules are set as required.

Security Group

You may add additional security group rules based on your specific needs. Refer to the relevant documentation for more details. Also, ensure that the Kubernetes endpoint is private-enabled and disable the public endpoint of the IKS cluster. For more details, refer to the Kubernetes API section.

Provisioning the Privileged Access Gateway

Log in to IBM Cloud and navigate to the Catalog. Search for Privileged Access Gateway and create a gateway with the Standard Plan in your desired region.
Choose the Virtual Private Cloud (VPC), Cloud Object Storage instance, and bucket within the same resource group as your Virtual Private Network.

Note: For high availability, create the Privileged Access Gateway (PAG) in two different zones (for example, us-south-1 and us-south-2).
Agree to the terms and conditions, and click Create. It may take a few minutes for the PAG to be provisioned. Once active, you'll find two endpoints for the selected zones.
Log in to cloud.ibm.com and go to the Resource List.
Under Security, find the Privileged Access Gateway. The endpoints for the two zones will be displayed under the Endpoint tab in the PAG service UI.
Add the VPN server routes for the PAG as shown below. Set the Action to Deliver and the Destination CIDR to the VPC subnets associated with the PAG.

Once the routes are established and healthy, you can start using the PAG with the endpoints provided in the PAG UI console.

Using OpenVPN to access Bastion-protected Kubernetes clusters

To access a Kubernetes cluster protected by a Bastion, you'll need to use an OpenVPN (OVPN) client from your local system. The OVPN client connects to the VPN server, which routes the request to the Privileged Access Gateway (PAG) endpoints and finally to the target IBM Kubernetes Service (IKS) cluster or Virtual Server Instance (VSI).

Prerequisites

OpenVPN Connect v3 (for macOS Catalina, and later)
IBM Cloud CLI and pag plugin installed

Steps

Set up the VPN Client:
1. Log in to IBM Cloud and navigate to the Resource List.
2. Go to Networking and find the VPN Client to Site Server.
3. Under the Clients tab, download the All client profiles zip file (for example, vpn-client-server-dcim-us-east.zip).
4. Unzip the file and open the vpn-ovpn-private-cert-us-east.ovpn profile with the OpenVPN client.
Connect the VPN client:
1. Open the OpenVPN client UI and import the .ovpn profile you downloaded.
2. Toggle the connection to activate the profile in the OpenVPN client.
  
  Note: The OVPN client may disconnect frequently.
After connecting the VPN client, follow the instructions to install the IBM Cloud CLI pag plugin.
Set the Privileged Access Gateway (PAG) endpoint:
1. Before running any Kubernetes commands, ensure the gateway is set to one of your instance's endpoints:
```
 ibmcloud login -a https://cloud.ibm.com -r us-east --sso
 # Follow the prompts to authenticate
 ibmcloud target -g <resource group>
 ibmcloud pag gateway set xxxx.us-east.pag.appdomain.cloud
```
2. You will see a message confirming that the gateway host is set.
Run the following command to connect to the PAG gateway and download the kubeconfig for the IKS cluster:

ibmcloud pag ks config <<IKS Cluster>>

Once successfully retrieved, you can now run kubectl commands to access the cluster through PAG.
You should now be able to use kubectl commands to manage your Kubernetes cluster:
```
 kubectl get pods -n dcim
 kubectl exec -it pod-name -n dcim -- /bin/bash
```
Note: These commands will work as long as the OpenVPN client is connected. Without the VPN connection, kubectl commands will fail.

Accessing PAG session recordings

To access and play operator session recordings stored in the Cloud Object Storage (COS) bucket configured in PAG, follow these steps:

Run the following commands inside a Kubernetes pod to generate a session recording.

kubectl get pods -n dcim

NAME                                            READY   STATUS    RESTARTS   AGE
kubernetesnodeapp-deployment-55c6865865-654lw   1/1     Running   0          75d

kubectl exec -it kubernetesnodeapp-deployment-55c6865865-654lw -n dcim -- /bin/bash
[root@kubernetesnodeapp-deployment-55c6865865-654lw app]# echo "Hello"
Hello
[root@kubernetesnodeapp-deployment-55c6865865-654lw app]# echo "World"
World
[root@kubernetesnodeapp-deployment-55c6865865-654lw app]# exit`

Access your COS bucket and look for a file starting with "kube…" matching the timestamp when the kubectl exec commands were executed.
Download and unzip the file.
In the command prompt, navigate to the folder containing the unzipped file.
Play the recording using the following command:

ibmcloud pag play kube_11cedd72-91a6-40cc-be4d-026f969ca0f8_IBMid-060000VYUA@dcim-dev-cluster

Adding Bastion support to IBM VPC Virtual Servers

To configure Bastion support for IBM Virtual Private Cloud (VPC) Virtual Servers (VSIs), follow these steps:

On the Client:

Set the Privileged Access Gateway (PAG) endpoint by running:

ibmcloud pag gateway set <gateway URL>

For example:

ibmcloud pag gateway set 522c171b-cf1c-4294-aafd-dbc88998b6a0-1.us-south.pag.appdomain.cloud
Run the following command to retrieve the public Certificate Authority (CA) used by PAG to log in to the VSIs:

ibmcloud pag ssh ca

On the Virtual Servers (VSIs):

Transfer the CA public key file to each VSI that you will access through the PAG gateway. For example, save it in /etc/ssh/pag-gateway-ca.pub.
Edit the SSH configuration file on each VSI: /etc/ssh/sshd_config and add the following line to reference the CA public key file:

TrustedUserCAKeys /etc/ssh/pag-gateway-ca.pub
Ensure that all users who need access to the VSI have an account. The user IDs should match their IBM Cloud Identity and Access Management (IAM) ID. You can obtain these from the IBM Cloud console under Manage > Access (IAM) > Users.
Add each user to the VSI using a command like:

useradd -m <username>
Restart the SSH service on each VSI to apply the changes.

This following figure illustrates the sequence of steps to execute:

alt

New user onboarding

Once the Privileged Access Gateway (PAG) is successfully installed and configured, onboarding a new user involves a single step: adding the user's IAM ID (the ID used by PAG to log in to the VSIs) to the Virtual Server Instances (VSIs).

To onboard a new user, execute the following command on each VSI as the root user or a user with administrative privileges:

useradd -m <IAM_ID>

If the IAM ID is not recognized, you may optionally restart the SSH service:

systemctl restart sshd

Refer to the following figure for the sequence of steps involved in the new user onboarding process.

alt

Process for administrators to add users

Obtain manager and group owner approval to grant operator login access.
Verify that the operator's ID exists in IBM Cloud.
Add the operator's ID to the designated VSI.

Transit gateway

For detailed documentation on Transit Gateway, refer to the IBM Cloud documentation.

Using a transit gateway, you can access Virtual Server Instances (VSIs) located in different Virtual Private Clouds (VPCs). To achieve this, connect the VPC where your Privileged Access Gateway (PAG) is deployed to the VPC where your resources reside. Refer to the following image for an example of a working transit gateway setup:

alt

Connecting to VSI through PAG

Download and install OpenVPN client:
- For Mac: Download OpenVPN Connect for macOS
- For Windows: Download OpenVPN Connect for Windows
Obtain the client profile file from your VPN server.
Open the VPN client, click on the + button in the Profiles section.
Choose UPLOAD FILE and click BROWSE. Select the client profile file downloaded in step 2.
Enter your IBM Cloud user ID in the USERNAME field and click CONNECT.
After connecting the OpenVPN client, log in to IBM Cloud by running the following command:

ibmcloud login -a https://cloud.ibm.com -sso

Follow the prompts to authenticate using a one-time code and select your desired account and region:

 API endpoint: https://cloud.ibm.com
 Get a one-time code from https://identity-3.uk-south.iam.cloud.ibm.com/identity/passcode to proceed.
 Open the URL in the default browser? [Y/n] > Y
 One-time code > [enter your code]
 Authenticating...
 OK
 Select an account:
 1. [Your account options...]
 Enter a number> 6
 Targeted account [Your account]
 Select a region (or press enter to skip):
 1. [Region options...]
 9. us-south
 Enter a number> 9
 Targeted region us-south

Ensure your API endpoint and region are correctly set to us-south or your desired region.

Start the SSH agent by running the following command:

eval $(ssh-agent)

Example:
```
pag-instance % eval $(ssh-agent)
 > Agent pid 17204
pag-instance %
```
Configure the PAG gateway to the endpoint deployed in Dallas (us-south):

ibmcloud pag gateway set 65880888-xxx-1.us-south.pag.appdomain.cloud

Use the following command to connect to the VSI's IP address:

ibmcloud pag ssh connect {ip_address}

Example for connecting to a VSI in Sydney:

ibmcloud pag ssh connect 192.168.32.4

Expected output:

 Retrieving a new SSH certificate
 OK
 Configuring SSH Agent
 OK
 Connecting to 192.***.32.4...
 By accessing this information system, users acknowledge and accept the following terms and conditions:
 - Users are accessing a U.S. Government or financial services information system;
 - Due to IBM security policies, information system usage will be monitored, recorded, and subject to audit in accordance with the applicable laws; and
 - Unauthorized use of the information system is prohibited and subject to criminal and civil penalties.
 Authorized uses only. All activity may be monitored and reported.
 Activate the web console with: systemctl enable --now cockpit.socket
 This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
 To register this system, run: insights-client --register
 Last login: Thu *** 13 09:22:55 2024 from 10.***.0.4
 [IBMid-********@*****-jh-001 ~]$ whoami
 IBMid-270007QTMH
 [IBMid-********@*****-jh-001 ~]$ cat /etc/os-release
 NAME="Red Hat Enterprise Linux"
 VERSION="8.3 (Ootpa)"
 ID="rhel"
 ID_LIKE="fedora"
 VERSION_ID="8.3"
 PLATFORM_ID="platform:el8"
 PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
 ANSI_COLOR="0;31"
 CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
 HOME_URL="https://www.redhat.com/"
 BUG_REPORT_URL="https://bugzilla.redhat.com/"
 REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
 REDHAT_BUGZILLA_PRODUCT_VERSION=8.3

Conclusion

You now have an understanding of the features provided by the Privileged Access Gateway (PAG). You have also learned how to configure the Standard Plan for PAG, enabling bastion support for Kubernetes Clusters and Virtual Servers running in IBM Cloud. Additionally, you learned how an operator can access a Kubernetes Cluster using kubectl commands and connect to Virtual Servers using the pag ssh command through PAG. You also learned how to play back operator session recordings stored in IBM Cloud Object Storage.

For more information about the Privileged Access Gateway, visit the IBM Cloud Documentation.

Configuring Privileged Access Gateway for secure access in IBM Cloud

Configuring Privileged Access Gateway for secure access in IBM Cloud