Introduction

As an IBM Cloud customer, you can deploy network and compute resources into your own IBM Cloud infrastructure account. The customer is then obliged to manage the network security of the environment provided to them. This document describes how you, an IBM Cloud customer, can automate the creation of a Secure Perimeter based on IBM Cloud best practices so that customer assets can be protected.

 

Secure Perimeter

The fundamental aspect of network isolation is the establishment of a Secure Perimeter. This Secure Perimeter controls traffic to and from the public internet and the customer assets hosted in IBM Cloud.  The Secure Perimeter will also enable direct connectivity from the customer enterprise by the use of Virtual Private Network (VPN) tunnels and IBM Cloud Direct Link.

A secure perimeter (SP) will protect assets within the environment, however it can also be useful to have network segregation between assets within a secure perimeter. This segregation (or segmentation) has several benefits, including access control and service traffic isolation between segments. The term Secure Perimeter Segment (SPS) is used here to define the network isolation behind the public perimeter.

An SPS is a set of VLANs that supports customer payloads. An SPS has two Virtual Local Area Networks (VLANs) – a front end VLAN and a backend VLAN.  These VLAN’s are connected to a Vyatta that manages the traffic into and out of the SPS.

Prerequisites for creating an SP and SPS using automation

Required IBM Cloud Permissions

In order to set up the Secure Perimeter you need to have (as a minimum) the following user permissions as configured by the account owner. Refer to IAM Permissions section in previously published blog describing how to manually set up a Secure Perimeter in IBM Cloud.

VPN Connection

You will also need to connect to your IBM Cloud infrastructure account using VPN in order to configure the Vyattas.  This connection is required for the duration of setting up the Secure Perimeter.

Once public access to the Vyattas is disabled, you will have to access the Vyattas via the IBM Cloud infrastructure VPN using the private IPs.

Software Prerequisites

The following software prerequisites must be in place. Listed with each prerequisite is a link to the documentation for installing/configuring the prerequisite.

Because the provisioning of a vyatta can take a number of hours it is also recommended but not required that you use a tool that will allow the program to run in the background after you have entered “yes”. For example “screen” can be used to facilitate this – Install screen

Install version 0.11.2 of ibm-cloud-provider:

mkdir -p /home/terraform_providers/terraform-provider-ibm

cd /home/terraform_providers/terraform-provider-ibm

 

Depending on whether you are using MAC or Linux download the correct binary

LINUX:

wget https://github.com/IBM-Cloud/terraform-provider-ibm/releases/download/v0.11.2/linux_amd64.zip

unzip linux_amd64.zip

MAC:

wget https://github.com/IBM-Cloud/terraform-provider-ibm/releases/download/v0.11.2/darwin_amd64.zip

unzip darwin_amd64.zip

 

vi $HOME/.terraformrc and add the following line:

providers {

ibm = "/home/terraform_providers/terraform-provider-ibm/terraform-provider-ibm"

}

 

Prerequisite Verification

Before the creation of a Secure Perimeter or a Secure Perimeter Segment using automation, the following prerequisites need to be verified.

IBM Cloud CLI Verification

To verify that the IBM Cloud CLI is installed, launch a terminal window and type:

bx login

 

This should prompt you to login to an API endpoint. Select the appropriate endpoint and login with your email address and password. If you need to login using an API key, use the following syntax:

bx login -a api.<region>.bluemix.net --apikey <api key>

kubectl Verification

To verify that kubectl is installed type:

kubectl version --short --client

 

If kubectl is installed then this command should return the version information for the client.

Client Version: v1.7.5

Container Registry Verification

To verify that the container registry plug-in is installed type:

bx plugin list

This should list the “container-registry” plug-in.

bx login -a api.<region>.ng.bluemix.net --apikey <api key>
bx cr region-set global
bx cr login
bx cr images --include-ibm |grep ibmcloud-secure-perimeter

This should return at least two entries for ibmcloud-secure-perimeter images:

  • registry.bluemix.net/ibm/ibmcloud-secure-perimeter-health
  • registry.bluemix.net/ibm/ibmcloud-secure-perimeter-network

 

Git Verification

To verify that git is installed type:

git --version

If the version of git is returned then git is installed correctly.

Docker Verification

To verify that docker is installed and running, type:

docker ps

This should return a list of running containers, which should be empty unless a container has been deployed.

Python Verification

To verify that python 2.7.14 is installed and running, type:

python --version

To check if required modules are installed, type:

pip list

Check that each of the following is listed

pyOpenSSL==17.5.0
requests==2.18.4
SoftLayer==5.4.2

Terraform Verification

To verify that terraform 0.11.0+  is installed and running, type:

terraform --version

ibm-cloud-provider Verification

Verify this file exists

ls -l /home/terraform_providers/terraform-provider-ibm/terraform-provider-ibm

Verify the contents of $HOME/.terraformrc

providers {

ibm = "/home/terraform_providers/terraform-provider-ibm/terraform-provider-ibm"

}

Create Secure Perimeter using automation

Terraform by HashiCorp  can be used to provision the Secure Perimeter in a customer’s IBM Cloud infrastructure account. Terraform is an “infrastructure as code” tool for building, changing, and versioning infrastructure safely and efficiently. A Terraform template is provided that will provision and manage infrastructure as a single unit.  You must first create a public and private VLAN.  The template will create a High Availability two member Vyatta. The template will associate and route the previously created VLANs to the Vyatta. Then the template will start a Docker container that uses an IBM provided docker image to configure the Vyatta.

This container will:

  • Create sync configuration for the primary and backup Vyatta
  • Create default inbound firewall rules on the Vyatta
  • Create the VLAN interface and gateway
  • Apply the outbound rules for Kubernetes

The template will also deploy a Kubernetes cluster  called the “monitoring cluster” using the initial pair of VLANs. Two pods are deployed into this cluster.

  • Network pod
    • Detects new subnets on already existing VLANs and applying configuration to vyatta to allow traffic  through this subnets.
    • Applies configuration for new VLANs and subnets created using “Create SPS” terraform template.
    • Exposes specific IPs/Subnets and ports  as requested by customer to Public Ingress.
  • Health pod
    •  Periodically checks for exposures between SPs and SPSs. It fetches each subnet pertaining to a SPS vlan in the customer’s  IBM Cloud infrastructure account and runs a scan to check for open ports outside of what the firewall has whitelisted. It gives a report at the end of each scan detailing the offending SP, the SPS VLAN in that SP and the IP addresses in that VLAN that have reachable ports.
    • Periodically scans the private network though it can be run manually to scan the public network from outside the customer’s IBM Cloud infrastructure account to check the public firewalls of each SP.

This guide assumes you have some familiarity with Terraform already.

It is recommended that you review the HashiCorp documentation for getting started to understand the basics of Terraform.

Steps to create Secure Perimeter

Login to IBM Cloud infrastructure VPN linked to Customer SL Acount

Before starting the automation the customer must create a front end and a back end vlan as described here. Note the vlan names as these will be used as input into the automation.

bx login -a api.<region>.ng.bluemix.net --apikey <api key>
bx cr region-set global
bx cr login

mkdir -p $HOME/SecurePerimeter
cd $HOME/SecurePerimeter
git clone  https://github.com/IBM/secure-perimeter.git
cd secure-perimeter/Create_Secure_Perimeter/
vi secure_perimeter/variables.tf

Fill in default values for:

• bluemix_api_key
• org
• space
• public vlan name (front end vlan name )
• private vlan name (back end vlan name )
• region ( for example us-south )
• datacenter ( for example dal13 )

and save the file

If you have chosen to use “screen” or some other tool to allow the program to run in the background then it should be started at this point.

./create_secure_perimeter.sh <Unique name for this secure perimeter - for example SP1>

Type yes and press Enter

Verification of Secure Perimeter

When the deployment is complete run

terraform state list |grep kubernetes_pod.network-pod

 

Check that the following line is returned.

module.monitoring_cluster.module.network-pod.kubernetes_pod.network-pod

Your Secure Perimeter is now set up.  At this point a two member HA Vyatta and one default SPS has been created. The Vyattas have been configured to make the perimeter secure. You can now create SPS’s specifically for your workload. The Vyattas will be automatically configured to allow clusters to run on your new SPS.

Note that the secure perimeter is also configured to support worker nodes within your cluster that only have a private interface.

Get the id of the Monitoring Cluster

When the deployment is complete run the following command to retrieve the name of the monitoring cluster. This will be required later when deploying your workload.

cd $HOME/SecurePerimeter/secure-perimeter/Create_Secure_Perimeter/
terraform output

and note the “name” of the cluster in the output. You will need this when deploying your Kubernetes workload on SPS later on.

monitoring_cluster_name = monitoring_cluster-<random-id>

Get the id or  name of the gateway – for when you are creating subsequent SPS

When the deployment is complete run the following command to retrieve the name or the id of the gateway. This will be needed when creating any subsequent SPSs

cd $HOME/SecurePerimeter/secure-perimeter/Create_Secure_Perimeter/
terraform output

Note either of the values for:

gateway_name = <gateway-name>
gateway_id = <gateway-id>

 

Get the ids and names of the public and private VLANs for the default SPS

When the deployment is complete run the following command to retrieve the ids of the VLANs for the default SPS.  You can start creating clusters on this SPS using the VLAN Ids.

cd $HOME/SecurePerimeter/secure-perimeter/Create_Secure_Perimeter/
terraform output

Note the values for:

sps_private_vlan_id = <private-vlan-id>
sps_private_vlan_num = <private-vlan-num>
sps_private_vlan_name = <private-vlan-name>
sps_public_vlan_id = <public-vlan-id>
sps_public_vlan_num = <public-vlan-num>
sps_public_vlan_name = <public-vlan-name>

Clusters can now be created on this default SPS. Use the above VLAN ids /names to create the cluster for your workload.

Create Secure Perimeter Segment using automation

A Secure Perimeter Segment is a network placed inside of the Secure Perimeter. The network is comprised of two VLAN’s that are previously created by the user. The automation will associate these VLANs with the Vyatta managing the Secure Perimeter. Customer workloads and IBM services (such as IBM Cloud Container Service) can be deployed on these VLAN’s.
Routing and firewall configuration is performed on the Vyatta to control access to and from these VLAN’s.
Multiple SPSs can exist within the same Secure Perimeter. This enables segmentation of workloads in each SPS and only allowed traffic can pass between them.

Note: Any user wishing to create a Secure Perimeter Segment must have access to the Secure Perimeter gateway devices in the relevant IBM Cloud infrastructure account.

 

The following steps are carried out when the SPS is created:

  • Before starting the automation the customer must create a front end and a back end vlan as described here. Note the vlan names as these will be used as input into the automation.
  • The automation will then
    • Associate VLAN’s with Vyatta
    • Configure Gateway Interface
    • Setup default Firewall rules for subnets
    • Enable IBM Cloud Container Service Kubernetes access

Note: An initial  pair of SPS VLANs have already been created and associated with the Vyatta.

Steps to Create Secure Perimeter Segment

cd $HOME/SecurePerimeter/secure-perimeter/Create_Secure_Perimeter_Segment
terraform workspace new SPS1  (Change to SPS2,SPS3 etc if creating subsequent SPSs within the same SP )
vi variables.tf

Fill in default values for:

• bluemix_api_key
• public vlan name
• private vlan name
• org
• space
• gateway_name_id
• region
• vlan_subnet_size

and save the file

terraform init
terraform apply 

Select yes and Enter

Verification of Secure Perimeter Segment

When the deployment is complete run

terraform state list

Verify that the following resources are listed

ibm_account.account
ibm_container_cluster_config.monitoring_cluster
ibm_network_vlan.private_vlan
ibm_network_vlan.public_vlan
ibm_network_vlan.sps_private_vlan
ibm_network_vlan.sps_public_vlan
ibm_org.org
ibm_space.space
module.ims_credentials.external.ims_credentials
module.network_gateway.external.network_gateway
module.sl_credentials.external.sl_credentials
null_resource.configure_private_vlan
null_resource.configure_public_vlan
null_resource.copy-config-from-pod
null_resource.copy-config-to-pod
null_resource.route_vlans
random_id.name

You are now ready to deploy your workload on the Secure Perimeter Segment.

Deploy your Kubernetes workload on your SPS

Now you can create your own Kubernetes cluster on the SPS and deploy your Kubernetes workload in that cluster.

When your Kubernetes Cluster is deployed in the SPS VLAN’s, the network pod has automatically enabled the primary and portable  subnets  on the Vyatta master and backup so that they can be used correctly.

If you want to expose payloads running on the Kubernetes cluster on the public internet, then the firewall needs to be modified to allow traffic to the subnet or specific IP addresses.  You must decide on the IP addresses and/or subnets to be exposed and the required ports.  The Subnets/IP’s are the ingress controller or load balancer of the Kube Cluster.

Steps to add IPs/Subnets and Ports to Public Ingress on the master and backup Vyattas

Retrieve the name of the monitoring cluster from the Create Secure Perimeter step.

bx login -a api.<region>.ng.bluemix.net --apikey <api key>

Download the cluster config file using the name of the monitoring cluster noted from earlier step “Get ID of Monitoring Cluster”

bx cs cluster-config monitoring_cluster-<id>
export KUBECONFIG=<output from bx cs cluster-config command>

Copy the rules.conf file  from the network pod

kubectl cp network-pod:/opt/secure-perimeter/rules.conf rules.conf

Update the rules.conf file with the subnet/IP and ports that should be exposed.
For example:

{
"external_subnets": [
"<your_subnet/cidr>",
"<your IP>"
],
"external_ports":[
"<your port number 1>",
"<your port number 2>"
]
}

Copy the updated file back onto the network pod.

kubectl cp rules.conf network-pod:/opt/secure-perimeter/rules.conf

 

Call the network pod to execute the rules on the vyatta.

kubectl exec network-pod python addrules.py

Your Subnets/IPs:port are now exposed from the secure-perimeter.

If you have any questions regarding the automation of the creation of the Secure Perimeter, the Secure Perimeter Segment or on how to expose IP’s and Subnets as well as ports from the Secure Perimeter please post a comment and I will respond as soon as possible.

Join The Discussion

Your email address will not be published. Required fields are marked *