Introduction

IBM Cloud offers customers network and compute resources in the IBM Cloud.  In the isolated offering it is the customer that has the obligation to manage network security of the environment provided to them.  This document describes how a customer of IBM Cloud can create an isolated network based on IBM Cloud best practices so that customer assets can be protected.

Secure Perimeter

The fundamental aspect of network isolation is the establishment of a secure perimeter.  This secure perimeter controls traffic to and from the public internet and the customer assets hosted in IBM Cloud.  The perimeter will also enable direct connectivity from the customer enterprise by the use of Virtual Private Network (VPN) tunnels and IBM Cloud Direct Link.

A secure perimeter will protect assets within the environment, however it can be useful to have network segregation between assets inside of a secure perimeter.  This segregation (or segmentation) has several benefits, including access control and service traffic isolation between segments.  The term Secure Perimeter Segment (SPS) is used here to define the network isolation behind the public perimeter.

An SPS is a set of VLANs that support customer payloads.  An SPS has two Virtual Local Area Networks (VLANs) – a front end VLAN and a back end VLAN.  These VLAN’s are connected to a Vyatta that manages the traffic into and out of the SPS.

Required IBM Cloud Permissions

It is expected that the customer has experience with Vyatta configurations.  Refer to Vyatta version 5.1 Configuration Guide for detailed configuration setup.

https://console.bluemix.net/docs/infrastructure/virtual-router-appliance/getting-started.html#getting-started

https://console.bluemix.net/docs/infrastructure/virtual-router-appliance/vra-docs.html#supplemental-vra-documentation

The customer setting up the isolated area should have familiarity with IBM Cloud and needs to have as a minimum the following user permissions as configured by the account owner.

Login into IBM Cloud using https://console.bluemix.net/

IAM Permissions

Navigate to IAM permissions

Select the user to modify and enable the permissions below.

Access policies

Area Setting
Services All Identity and Access enabled services
Region All regions
Assign platform access roles Administrator
Assign service access roles Manager

 

Cloud Foundry access

CF Access Role
Organization Manager
Space Developer

 

IAAS access

  • ‘View only’ access permissions

Note: Only relevant when creating a new user

 

Infrastructure Permissions

Navigate to Infrastructure

Select the User List under Account

IAAS user management, enable the following:

IAAS User Management

  • Generate API Key
  • Supply VPN SSL Access

Select the user and choose “Portal Permissions”

Set IAAS Portal permissions

Support Devices Network Service Account
  • View Ticket
  • Add ticket
  • Edit Ticket
  • View Hardware Details
  • View Virtual Server Details
  • Reboot server and view IPMI system information
  • Upgrade Server
  • Hardware Firewall
  • Issue OS Reloads and Initiate Rescue Kernel
  • Manage Load Balancers
  • Edit Hostname/Domain
  • Manage Port Control
  • Manage Network Subnet Routes
  • Manage Network VLAN Spanning
  • Manage IPSEC Network Tunnels
  • Manage Network Gateways
  • Add Compute with Public Network Port
  • Add IP Addresses
  • Manage SSH Keys
  • Manage Storage
  • Manage DNS, Reverse DNS, and WHOIS
  • View Certificates (SSL)
  • Manage Certificates (SSL)
  • View Account Summary
  • Add/Upgrade Cloud Instances
  • Cancel Server
  • Add/Upgrade Services
  • Cancel Services
  • Add Server
  • Add Storage

 

The customer will also need to have IBM Cloud infrastructure VPN access for their account. Once Public access to the Vyattas is disabled, the customer will have to access the Vyattas via the IBM Cloud infrastructure VPN using the private IPs.

Create Secure Perimeter

The first step is to create an outer boundary from the public internet that an SPS resides behind.  This outer boundary provides the security that isolates an SPS.  A Public VLAN provided by IBM Cloud infrastructure is a public network that is internet accessible and a gateway and firewall are needed to bridge this public VLAN and an SPS.  A Vyatta provides this gateway and firewall perimeter.

The configuration of the Vyatta secures the Secure Perimeter according to the configuration defined in this document.  For redundancy, two Vyatta instances are configured as a highly available pair and their configuration is synchronized.

An SPS is placed inside this perimeter by means of placing the SPS public and private VLANs under the control of the Vyatta.

The following upfront information should be obtained before creating the SPS:

Item Description
Datacenter The IBM Cloud Datacenter that the customer requires the secure perimeter to be placed. Regions can be determined from https://www.ibm.com/cloud-computing/bluemix/data-centers and within each Region there are individual Locations.
Hovering over a Region will provide a list of the Locations and an available capacity indicator.
Vyatta Hostname Friendly name to easily identify the purpose of a Vyatta instance.
Domain Customer domain name.
Public VLAN Name Friendly name to easily identify the purpose of the public VLAN.
Private VLAN Name Friendly name to easily identify the purpose of the private VLAN.

Create Front End VLAN

Order the VLAN from IBM Cloud infrastructure Web Portal (https://control.softlayer.com)

Navigate to the Order VLAN form:

Order -> Network -> Order VLAN -> +Order

Populate the order form:

  • Select “Order by Router”
  • Choose the frontend router from the “Select Router” pulldown e.g. fcrdal13

Select “8 Static Public IP Addresses” in Primary Subnet Size”

Enter the name of the VLAN (Public VLAN Name)

Press “Continue”

Select “4” for number of IP’s required in next 30 days and 12 months pulldowns

Complete the remaining entries in the form (contact details etc.)

Select the checkbox “Master Service Agreement” to agree to the terms therein.

Place the order by selecting “Place Order”

View VLAN in VLAN list (https://control.softlayer.com/network/vlans) and note the number.

Note: Use of this VLAN before it has been placed behind a Vyatta will expose that use to the public internet.  It is recommended that this VLAN not be used until the Vyatta has been configured to isolate and protect the VLAN.

Create Back End VLAN

Order the VLAN from IBM Cloud infrastructure Web Portal (https://control.softlayer.com)

Navigate to the Order VLAN form:  Order -> Network -> Order VLAN -> +Order

Populate the order form:

  • Select “Order by Router”
  • Choose the backend router from the “Select Router” pulldown e.g. bcrdal13

NOTE: The router selected should be same router used for the Gateway VLANs associated with the Vyatta created earlier (https://control.softlayer.com/network/gateways and click on the Vyatta)

Select “8 Static Public IP Addresses” in Primary Subnet Size”

Enter the name of the VLAN (Private VLAN Name)

Press “Continue”

Select “4” for number of IP’s required in next 30 days and 12 months pulldowns

Complete the remaining entries in the form (contact details etc.)

Select the checkbox “Master Service Agreement” to agree to the terms therein.

Place the order by selecting “Place Order”

View VLAN in VLAN list (https://control.softlayer.com/network/vlans) and note the number.

These VLAN’s will be used to host the Secure Perimeter Segment after the vyatta’s have been created and configured.

Create Vyatta Pair

Order the Vyatta pair from IBM Cloud infrastructure Web Portal (https://control.softlayer.com) and select the appropriate IBM Cloud infrastructure account as required.

Navigate to the Order Vyatta form:

Order -> Network -> Order Vyatta -> +Order

The Gateway Appliances order form opens

Choose Datacenter from Data Center from pulldown

Select “STARTING PRICE PER MONTH” for the desired server type to host the Vyatta – recommended to use at a minimum:

Dual Intel Xeon E5-2620 v4 (16 Cores, 2.10 GHz)

The Vyatta configuration form will open

Click the “High Availability Pair” checkbox

Choose the RAM configuration – recommended to use at a minimum: 64GB

Choose the Operating System (only one)

Ensure 4TB Sata disk is configured and added

Network configuration – 10G Dual Uplink

Select “ADD TO ORDER”

The CHECKOUT form opens

Navigate to VLAN Selection

  • Select Private VLAN Name in “Backend VLAN” pulldown
  • Select Public VLAN Name in “Frontend VLAN” pulldown

Add Hostname and Domain

[Optional] Select an ssh_key for the “SSH Keys pulldown” for each server

Note: An ssh key must be pre-configured by using the following instructions: https://knowledgelayer.softlayer.com/procedure/add-ssh-key

Select “Submit Order” to complete the Vyatta order request

Wait for notification that the Vyatta have been setup.  Monitor the creation at the following link: https://control.softlayer.com/network/gateways

Configure Vyatta

After the Vyatta has been created, the configuration can now start.  Initially the master and backup synchronization is disabled.  Identify the master Vyatta server from the following link https://control.softlayer.com/network/gateways.  Choose the Gateway instance to be configured to view the Vyatta details.

 

 

 

SSH into one of your Vyatta servers (public or private IP address) using vyatta as the username and enter the password from the figure above when prompted.  Note the password is that listed for root.

ssh vyatta@<Gateway 1 IP>

Retrieve the Virtual IP address (VIP) of the Vyatta

vyatta@vyatta:~$ show vrrp detail

vyatta@vyatta:~$ show vrrp detail
--------------------------------------------------
Interface: dp0bond0
--------------
  Group: 1
  ----------
  State:                        MASTER
…
  Source Address:               IP-ADDRESS
…
VIP count:                    1
<private vip>
Interface: dp0bond1
…
VIP count:                    1
<public vip>

 

Based on the IP-ADDRESS in the vrrp output, it is possible to identify the MASTER and BACKUP vyatta’s.  Also note the PRIORITY of the respective vyatta in the gateway member details above.  The PRIORITY is required when configuring VLAN’s on the vyatta’s.

 

Login to the master Vyatta server using a VIP (either public or private).

ssh vyatta@<vip>

Enter the following commands to configure sync:

Enter configure mode:

configure

Delete existing sync-map configuration:

delete system config-sync sync-map SYNC

Enter new config:

set system config-sync sync-map SYNC rule 1 action 'include'
set system config-sync sync-map SYNC rule 1 location 'service nat'
set system config-sync sync-map SYNC rule 2 action 'include'
set system config-sync sync-map SYNC rule 2 location 'resources group'
set system config-sync sync-map SYNC rule 3 action 'include'
set system config-sync sync-map SYNC rule 3 location 'security firewall'
set system config-sync sync-map SYNC rule 4 action 'include'
set system config-sync sync-map SYNC rule 4 location 'security vpn'
set system config-sync sync-map SYNC rule 5 action 'include'
set system config-sync sync-map SYNC rule 5 location 'interfaces loopback'
set system config-sync sync-map SYNC rule 6 action 'include'
set system config-sync sync-map SYNC rule 6 location 'policy'
set system config-sync sync-map SYNC rule 7 action 'include'
set system config-sync sync-map SYNC rule 7 location 'interfaces tunnel'
set system config-sync sync-map SYNC rule 8 action 'include'
set system config-sync sync-map SYNC rule 8 location 'interfaces vti'
set system config-sync sync-map SYNC rule 9 action 'include'
set system config-sync sync-map SYNC rule 9 location 'system login banner'
set system config-sync sync-map SYNC rule 16 action 'exclude'
set system config-sync sync-map SYNC rule 16 location 'protocols static route 0.0.0.0'
set system config-sync sync-map SYNC rule 17 action 'exclude'
set system config-sync sync-map SYNC rule 17 location 'protocols static route 10.0.0.0'
set system config-sync sync-map SYNC rule 18 action 'include'
set system config-sync sync-map SYNC rule 18 location 'protocols'

Apply and save the configuration:

commit
save

Login to the backup Vyatta server and repeat the above to update the sync-map configuration on the backup server.

View the sync status

show config-sync status

eg:

show config-sync status
remote-router:  <Private IP of MASTER>
version:              5.2R5S3
sync-map:             SYNC
last sync status:     succeeded
last sync time:       <date of last sync>
in-sync?:             yes
access-status:        connected
remote-router:  <Private IP of BACKUP>
version:              5.2R5S3
sync-map:             SYNC
last sync status:     succeeded
last sync time:       <date of last sync>
in-sync?:             yes
access-status:        connected

 

Verify the status

 

MASTER IP:                       correct

BACKUP IP:                       correct

2 x Sync Status:                succeeded

2 x In-sync:                        yes

Secure Perimeter Segment

A Secure Perimeter Segment is a network placed inside of the Isolated Area.  The network is comprised of two VLAN’s that are created and associated with the Vyatta managing the Isolated Area.  Customer workloads and IBM services (such as IBM Container Service) can be deployed on these VLAN’s.

Routing and firewall configuration is performed on the Vyatta to control access to and from these VLAN’s.

Multiple SPSs can exist within the same Core Isolated Area.  This enables segmentation of workloads in each SPS and only allowed traffic can pass between them.

The VLAN names should be determined before creating the SPS.  These names should be chosen to represent the workload that will be placed in the SPS.

Public VLAN Name & Private VLAN Name

Create SPS

The following steps are required:

  • Create VLAN’s
  • Associate VLAN’s with Vyatta
  • Configure Gateway Interface
  • Setup default Firewall rules for subnets
  • Enable IBM Cloud Container Service Kubernetes access

Note: If the initial SPS VLANs have already been created and associated with the Vyatta then those VLAN’s will be used in the SPS.  However, if a second SPS is being created within the secure perimeter, the Frontend and Backend VLANs will need to be created.

Create Front End VLAN

Order the VLAN from IBM Cloud infrastructure Web Portal (https://control.softlayer.com)

Navigate to the Order VLAN form:

Order -> Network -> Order VLAN -> +Order

Populate the order form:

  • Select “Order by Router”
  • Choose the frontend router from the “Select Router” pulldown e.g. fcrdal13

Select “8 Static Public IP Addresses” in Primary Subnet Size”

Enter the name of the VLAN (Public VLAN Name)

Press “Continue”

Select “4” for number of IP’s required in next 30 days and 12 months pulldowns

Complete the remaining entries in the form (contact details etc.)

Select the checkbox “Master Service Agreement” to agree to the terms therein.

Place the order by selecting “Place Order”

View VLAN in VLAN list (https://control.softlayer.com/network/vlans) and note the number.

Note: Use of this VLAN before it has been placed behind a Vyatta will expose that use to the public internet.  It is recommended that this VLAN not be used until the Vyatta has been configured to isolate and protect the VLAN.

Create Back End VLAN

Order the VLAN from IBM Cloud infrastructure Web Portal (https://control.softlayer.com)

Navigate to the Order VLAN form:  Order -> Network -> Order VLAN -> +Order

Populate the order form:

  • Select “Order by Router”
  • Choose the backend router from the “Select Router” pulldown e.g. bcrdal13

NOTE: The router selected should be same router used for the Gateway VLANs associated with the Vyatta created earlier (https://control.softlayer.com/network/gateways and click on the Vyatta)

Select “8 Static Public IP Addresses” in Primary Subnet Size”

Enter the name of the VLAN (Private VLAN Name)

Press “Continue”

Select “4” for number of IP’s required in next 30 days and 12 months pulldowns

Complete the remaining entries in the form (contact details etc.)

Select the checkbox “Master Service Agreement” to agree to the terms therein.

Place the order by selecting “Place Order”

View VLAN in VLAN list (https://control.softlayer.com/network/vlans) and note the number.

Associate the VLAN’s with the Gateways

The VLAN’s need to be associated with the Vyatta’s and routing enabled.

Associate VLAN’s to Vyatta’s from IBM Cloud infrastructure Web Portal (https://control.softlayer.com).

Navigate to the Gateway Appliances page:  Network -> Gateway Appliances

Select the Gateway instance

The Gateway details page opens

If the SPS Public VLAN has not been associated with the Vyatta, choose the SPS Public VLAN from “Associate a VLAN” pulldown

Select “Associate” to associate the Public VLAN to the Vyatta

Enable Routing for the SPS Public VLAN using Actions -> Route VLAN

If the SPS Private VLAN has not been associated with the Vyatta, choose the SPS Private VLAN from “Associate a VLAN” pulldown

Select “Associate” to associate the SPS Private VLAN to the Vyatta

Enable Routing for the SPS Private VLAN using Actions -> Route VLAN

Association and routing is now enabled for the SPS VLAN’s.

The Vyatta’s need to be configured to act as the gateways for all the subnets on the Public and Private VLAN’s.

Configure Gateway Interface for Public VLAN

Retrieve a list of all subnets on the SPS Public VLAN.

View VLAN’s in the IBM Cloud infrastructure Web Portal (https://control.softlayer.com)

Navigate to the VLAN page:  Network -> IP Management -> VLANs

Select and note the VLAN Number of the SPS Public VLAN instance to be added.

The VLAN details page opens.

Select the CIDR of the subnet(s) that need to be configured on the gateway (If this is the first-time subnets are added for this SPS then all subnets listed should be added by repeating this and the following steps)

The subnet page will open

Retrieve the Gateway IP address of the subnet

Navigate to Gateway Appliances page:  Network -> Gateway Appliances

Select the Gateway instance for this isolated network

Retrieve and note the Group Number value

Configure Gateways on Vyatta MASTER

Define and note the IP Address (IP) to be used within the Gateway (Vyatta’s) to route traffic to the SPS VLANs.

Use the following formula to create the IP address:
XX.REMAINDER = (VLAN Number * 4 + 16384) / 256

YY = (REMAINDER * 256) + 1

IP = 169.254.XX.YY

Login to the Vyatta master using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Enable the interface for the VLAN.

set interfaces bonding dp0bond1 vif <VLAN Number> address <IP>/30
set interfaces bonding dp0bond1 vif <VLAN Number> vrrp vrrp-group <GROUP Number> sync-group vgroup<GROUP Number>
set interfaces bonding dp0bond1 vif <VLAN Number> vrrp vrrp-group <GROUP Number> priority <MASTER PRIORITY>

 

Apply Gateway to Vyatta for this subnet (repeat for all subnets)

set interfaces bonding dp0bond1 vif <VLAN Number> vrrp vrrp-group <GROUP number> virtual-address <GATEWAY IP>/29

Commit and save the change

commit
save
exit

Configure Gateways on Vyatta BACKUP

Define the IP Address (IP) to be used within the Gateway (Vyatta’s) to route traffic to the SPS VLANs.

Use the following formula to create the IP address:
XX.REMAINDER = (VLAN Number * 4 + 16384) / 256

YY = (REMAINDER * 256) + 2

IP = 169.254.XX.YY

Login to the Vyatta backup using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Enable the interface for the VLAN.

set interfaces bonding dp0bond1 vif <VLAN Number> address <IP>
set interfaces bonding dp0bond1 vif <VLAN Number> vrrp vrrp-group <GROUP Number> sync-group vgroup<GROUP Number>
set interfaces bonding dp0bond1 vif <VLAN Number> vrrp vrrp-group <GROUP Number> priority <BACKUP PRIORITY>

Apply Gateway to Vyatta for this subnet (repeat for all subnets)

set interfaces bonding dp0bond1 vif <VLAN Number> vrrp vrrp-group <GROUP Number> virtual-address <GATEWAY IP>/29

Commit the change, save and exit

commit
save
exit

Verify connectivity to the subnet Gateway IP address (repeat for each subnet in VLAN). Ensure ping responds.

ping <GATEWAY IP>

Configure Gateway Interface for Private VLAN

Retrieve a list of all subnets on the SPS Private VLAN.

View VLAN’s in the IBM Cloud infrastructure Web Portal (https://control.softlayer.com)

Navigate to the VLAN page:  Network -> IP Management -> VLANs

Select the VLAN Number of the SPS Private VLAN instance to be added

The VLAN details page opens

Select the CIDR of the subnet(s) that need to be configured on the gateway (If this is the first-time subnets are added for this SPS then all subnets listed should be added by repeating this and the following steps)

The subnet page will open

Retrieve the Gateway IP address of the subnet

Navigate to Gateway Appliances page:  Network -> Gateway Appliances

Select the Gateway instance for this isolated network

Retrieve the Group Number value

Configure Gateways on Vyatta MASTER

Define the IP Address (IP) to be used within the Gateway (Vyatta’s) to route traffic to the SPS VLANs.

Use the following formula to create the IP address:
XX.REMAINDER = (VLAN Number * 4) / 256

YY = (REMAINDER * 256) + 1

IP = 169.254.XX.YY

Login to the Vyatta master using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Enable the interface for the VLAN.

set interfaces bonding dp0bond0 vif <VLAN Number> address <IP>/30
set interfaces bonding dp0bond0 vif <VLAN Number> vrrp vrrp-group <GROUP Number> sync-group vgroup<GROUP Number>
set interfaces bonding dp0bond0 vif <VLAN Number> vrrp vrrp-group <GROUP Number> priority <MASTER PRIORITY>

Apply Gateway to Vyatta for this subnet (repeat for all subnets)

set interfaces bonding dp0bond0 vif <VLAN Number> vrrp vrrp-group <GROUP Number> virtual-address <GATEWAY IP>/29

Commit the change, save and exit

commit
save
exit

Configure Gateways on Vyatta BACKUP

Define the IP Address (IP) to be used within the Gateway (Vyatta’s) to route traffic to the SPS VLANs.

Use the following formula to create the IP address:
XX.REMAINDER = (VLAN Number * 4) / 256
YY = (REMAINDER * 256) + 2
IP = 169.254.XX.YY

Login to the Vyatta backup using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Enable the interface for the VLAN.

set interfaces bonding dp0bond0 vif <VLAN Number> address <IP>
set interfaces bonding dp0bond0 vif <VLAN Number> vrrp vrrp-group <GROUP Number> sync-group vgroup<GROUP Number>
set interfaces bonding dp0bond0 vif <VLAN Number> vrrp vrrp-group <GROUP Number> priority <BACKUP PRIORITY>

Apply Gateway to Vyatta for this subnet (repeat for all subnets)

set interfaces bonding dp0bond0 vif <VLAN Number> vrrp vrrp-group <GROUP Number> virtual-address <GATEWAY IP>/29

Commit the change, save and exit

commit
save
exit

Verify connectivity to the subnet Gateway IP address (repeat for each subnet in VLAN). Ensure ping responds.

ping <GATEWAY IP>

Enable Network Tuning

Login to the Vyatta master using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Set the following in Vyatta.

set system session table-size '1048567'
set interfaces bonding dp0bond0 mtu '9000'

Commit the change and save

commit
save
exit

Login to the Vyatta backup using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Set the following in Vyatta.

set system session table-size '1048567'
set interfaces bonding dp0bond0 mtu '9000'

Commit the change and save

commit
save
exit

Set up Firewall Rules

By default, an SPS should block all inbound traffic with the exception of the whitelist IPs.

Login to the Vyatta master using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Optional step – Create a resource group for the whitelist inbound IP’s. Whitelist IP’s are IP addresses the customer chooses to allow ingress from.

set resources group address-group WHITE_LIST address
set resources group address-group WHITE_LIST address
set resources group address-group WHITE_LIST address

Create a resource group for the Gateway VLANs. These IP’s are the Public IP addresses and the VIP of the Vyatta pair.

set resources group address-group GATEWAY-VLANS address <gateway public vip>
set resources group address-group GATEWAY-VLANS address <master public ip>
set resources group address-group GATEWAY-VLANS address <backup public ip>

Create a resource group for the required IBM Cloud infrastructure IPs. This varies depending on the datacenter.  Refer to the public firewall table entries for the required datacenter from  https://knowledgelayer.softlayer.com/faqs/6 for the correct IP’s.  These are referenced as an example.

set resources group address-group SL_PUBLIC address '173.192.118.0/23'
set resources group address-group SL_PUBLIC address '184.172.118.0/23'
set resources group address-group SL_PUBLIC address '198.23.118.0/23'
set resources group address-group SL_PUBLIC address '169.46.118.0/23'
set resources group address-group SL_PUBLIC address '169.47.118.0/23'
set resources group address-group SL_PUBLIC address '169.48.118.0/24'

Set default to drop all inbound traffic.

set security firewall name FIREWALL-INBOUND-TRAFFIC default-action 'drop'

Allow IBM Cloud infrastructure public group traffic.

set security firewall name FIREWALL-INBOUND-TRAFFIC rule 20 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 20 description 'allow SL Public Networks'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 20 source address 'SL_PUBLIC'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 20 state 'enable'

Allow inbound icmp to Vyatta.

set security firewall name FIREWALL-INBOUND-TRAFFIC rule 30 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 30 description 'allow inbound icmp-ping'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 30 destination address GATEWAY-VLANS
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 30 icmp name 'echo-request'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 30 state 'enable'

Allow whitelist group traffic – apply if WHITE_LIST was configured above

set security firewall name FIREWALL-INBOUND-TRAFFIC rule 40 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 40 description 'allow Networks'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 40 source address 'WHITE_LIST'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 40 state 'enable'

Allow IPSec traffic (needed for VPN)

set security firewall name FIREWALL-INBOUND-TRAFFIC rule 100 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 100 description 'allow IPSEC ESP/50 protocols'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 100 protocol 'esp'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 100 state 'enable'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 110 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 110 description 'allow IPSEC AH/51 protocols'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 110 protocol 'ah'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 110 state 'enable'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 120 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 120 description 'allow IPSEC IKE/500 protocols'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 120 destination port '500'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 120 protocol 'udp'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 120 state 'enable'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 130 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 130 description 'allow negotiation of NAT-Traversal in IKE'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 130 destination port '4500'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 130 protocol 'udp'

Configure IP masquerade.

set service nat source rule 5 description 'Exclude Gateway IPs from SRC NAT'
set service nat source rule 5 'exclude'
set service nat source rule 5 outbound-interface 'dp0bond1'
set service nat source rule 5 source address 'GATEWAY-VLANS'
set service nat source rule 6 description 'Exclude IBM Cloud public IPs from SRC NAT'
set service nat source rule 6 'exclude'
set service nat source rule 6 outbound-interface 'dp0bond1'
set service nat source rule 6 destination address 'SL_PUBLIC'
set service nat source rule 200 description 'hide all IPs'
set service nat source rule 200 destination address '!10.0.0.0/8'
set service nat source rule 200 outbound-interface 'dp0bond1'
set service nat source rule 200 translation address 'masquerade'

Enable the firewall.

set interfaces bonding dp0bond1 firewall in 'FIREWALL-INBOUND-TRAFFIC'

NOTE: This command should also be run on the backup Vyatta server.

Commit the change, save and exit

commit
save
exit

Note: At this point, access to the vyatta via public IP is disabled.  From this point on, the customer will have to access the Vyattas via the IBM Cloud infrastructure VPN using the Private IPs.

Kubernetes Preparation

Communication must be allowed to the IBM Container Service Control Plane from the SPS.

Retrieve the IP addresses of the Kubernetes Control Plane and Container Service Registry for the Region that the Datacenter location is managed by.  The latest information can be obtained from: https://console.bluemix.net/docs/containers/cs_firewall.html#firewall

Important: You must allow outgoing traffic to port 443 for all of the locations within the region, to balance the load during the bootstrapping process. For example, if your cluster is in US South, you must allow traffic from port 443 to the IP addresses for all of the locations (dal10, dal12, and dal13).

Login to the Vyatta master using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Allow control plane destination addresses (format X.X.X.X)

set resources group address-group CLUSTER-SERVERS address <REGION IP 1>
set resources group address-group CLUSTER-SERVERS address <REGION IP 2>
set resources group address-group CLUSTER-SERVERS address <REGION IP 3>
set resources group address-group CLUSTER-SERVERS address <REGION IP 4>

Allow registry IP addresses (format X.X.X.X/X)

set resources group address-group CLUSTER-SERVERS address <REGISTRY IP 1>
set resources group address-group CLUSTER-SERVERS address <REGISTRY IP 2>
set resources group address-group CLUSTER-SERVERS address <REGISTRY IP 3>
set resources group address-group CLUSTER-SERVERS address <REGISTRY IP 4>

Drop all outbound traffic by default.

set security firewall name NETWORK-SPS-OUTBOUND  default-action 'drop'

Allow outbound icmp-ping.

set security firewall name NETWORK-SPS-OUTBOUND  rule 30 action 'accept'
set security firewall name NETWORK-SPS-OUTBOUND  rule 30 icmp name 'echo-request'
set security firewall name NETWORK-SPS-OUTBOUND  rule 30 state 'enable'

Allow outbound PMTUD (type 3 code 4).

set security firewall name NETWORK-SPS-OUTBOUND  rule 50 action 'accept'
set security firewall name NETWORK-SPS-OUTBOUND  rule 50 icmp name 'fragmentation-needed'
set security firewall name NETWORK-SPS-OUTBOUND  rule 50 protocol 'icmp'
set security firewall name NETWORK-SPS-OUTBOUND  rule 50 state 'enable'

Allow outbound port 443.

set security firewall name NETWORK-SPS-OUTBOUND  rule 60 action 'accept'
set security firewall name NETWORK-SPS-OUTBOUND  rule 60 description 'allow outgoing port 443'
set security firewall name NETWORK-SPS-OUTBOUND  rule 60 destination address CLUSTER-SERVERS
set security firewall name NETWORK-SPS-OUTBOUND  rule 60 destination port 443
set security firewall name NETWORK-SPS-OUTBOUND  rule 60 protocol 'tcp'
set security firewall name NETWORK-SPS-OUTBOUND  rule 60 state 'enable'

Allow outgoing udp port 20000-32767.

set security firewall name NETWORK-SPS-OUTBOUND  rule 70 action 'accept'
set security firewall name NETWORK-SPS-OUTBOUND  rule 70 description 'allow outgoing udp port 20000-32767'
set security firewall name NETWORK-SPS-OUTBOUND  rule 70 destination address CLUSTER-SERVERS
set security firewall name NETWORK-SPS-OUTBOUND  rule 70 destination port 20000-32767
set security firewall name NETWORK-SPS-OUTBOUND  rule 70 protocol 'udp'
set security firewall name NETWORK-SPS-OUTBOUND  rule 70 state 'enable'

Allow outgoing tcp [depends on customer rules].

set security firewall name NETWORK-SPS-OUTBOUND  rule 80 action 'accept'
set security firewall name NETWORK-SPS-OUTBOUND  rule 80 description 'allow outgoing tcp'
set security firewall name NETWORK-SPS-OUTBOUND  rule 80 protocol 'tcp'
set security firewall name NETWORK-SPS-OUTBOUND  rule 80 state 'enable'

Allow outgoing udp [depends on customer rules].

set security firewall name NETWORK-SPS-OUTBOUND  rule 90 action 'accept'
set security firewall name NETWORK-SPS-OUTBOUND  rule 90 description 'allow outgoing udp DNS'
set security firewall name NETWORK-SPS-OUTBOUND  rule 90 protocol 'udp'
set security firewall name NETWORK-SPS-OUTBOUND  rule 90 state 'enable'

Allow outgoing tcp.

set interfaces bonding dp0bond1 firewall out NETWORK-SPS-OUTBOUND

Commit changes, save and exit

commit
save
exit

Login to the Vyatta backup using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Allow outgoing tcp.

set interfaces bonding dp0bond1 firewall out NETWORK-SPS-OUTBOUND

Commit changes, save and exit

commit
save
exit

User IP

An SPS can support a customer provided subnet.  This subnet is applied as an overlay network on the SPS private VLAN.  The subnet is passed to the deployment configuration to be used by the Kubernetes cluster.

Using the provided subnet range, determine the following:

  • Enterprise Subnet CIDR
  • Enterprise Subnet Gateway IP
  • Enterprise Allow IP

Note that the Enterprise Allow IP is a CIDR to allow traffic from the enterprise be passed to the user subnet.

View VLAN’s in IBM Cloud infrastructure Web Portal (https://control.softlayer.com/)

Navigate to Gateway Appliances page:  Network -> Gateway Appliances

Select the Gateway instance for this isolated network

Retrieve the Group Number value

Navigate to the VLAN page:  Network -> IP Management -> VLANs

Select the VLAN Number of the SPS Private VLAN instance to be added

Login to the Vyatta master using ssh. Supply the username as vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Set the user subnet gateway ip on the interface.

set interfaces bonding dp0bond0 vif <VLAN Number> vrrp vrrp-group <GROUP Number> virtual-address ‘<ENTERPRISE SUBNET GATEWAY IP>/<CIDR mask>’

NOTE: This command should also be run on the backup Vyatta server.

Set the routing policy for the subnet.

set policy route prefix-list USERIP rule 2 action ‘permit’
set policy route prefix-list USERIP rule 2 prefix ‘<ENTERPRISE SUBNET CIDR>’
set policy route prefix-list SLPRIVATE rule 2 action ‘permit’
set policy route prefix-list SLPRIVATE rule 2 prefix ‘10.0.0.0/8’
set policy route route-map PRIVATEANDUSERIP rule 10 action ‘permit’
set policy route route-map PRIVATEANDUSERIP rule 10 match ip address prefix-list ‘SLPRIVATE’
set policy route route-map PRIVATEANDUSERIP rule 20 action ‘permit’
set policy route route-map PRIVATEANDUSERIP rule 20 match ip address prefix-list ‘USERIP’
set policy route route-map PRIVATEANDUSERIP rule 90 action ‘deny’

Commit the change, save and exit

commit
save
exit

Use the following guide to configure Kubernetes cluster with user IP range.

https://www.ibm.com/blogs/bluemix/2017/12/private-ip-addresses-in-cloud/

Kubernetes Deployment

When a Kubernetes Cluster is deployed in the environment (not covered here) primary subnets are created on the SPS VLAN’s the cluster is hosted on.  These new subnets must be enabled on the Vyatta master and backup before they can be used correctly.

Note the newly created subnets must have the subnet gateway to the Vyatta (see Configure Gateway Interface above) .

If the customer chooses to expose payloads running on the Kube cluster on the public internet, then the firewall needs to be modified to allow traffic to the subnet or specific IP addresses.  Decide on the IP addresses and/or subnets to be exposed and the required ports.  The IP’s are the ingress controller or loadbalancer of the Kube Cluster.

Login to the Vyatta master using ssh. Supply the username vyatta and the correct password when requested.

Set the Vyatta into configure mode.

configure

Create or update a resource group for the public inbound subnet and ports.

set resources group address-group PUBLIC_INGRESS address <INTERNAL_SUBNET>

Create or update a resource group for the public inbound IP’s and ports. This can be individual ports and or a range of ports as per the examples below:

set resources group port-group SERVICE-PORTS port 80
set resources group port-group SERVICE-PORTS port 443
set resources group port-group SERVICE-PORTS port 9000-9999
set resources group port-group SERVICE-PORTS port 30000-32767
set resources group port-group SERVICE-PORTS port <EXTERNAL PORT>

Allow ingress for the resource group.

set security firewall name FIREWALL-INBOUND-TRAFFIC rule 50 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 50 description 'allow Ingress access'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 50 destination address PUBLIC_INGRESS
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 50 destination port SERVICE-PORTS
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 50 protocol tcp
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 50 state 'enable'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 60 action 'accept'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 60 description 'allow ping Ingress'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 60 destination address PUBLIC_INGRESS
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 60 icmp name 'echo-request'
set security firewall name FIREWALL-INBOUND-TRAFFIC rule 60 state 'enable'

Configure masquerade exemption for Load Balancer, Ingress Controller and Node-Port subnets.

set service nat source rule 199 description 'exempt PUBLIC_INGRESS'
set service nat source rule 199 destination address '!10.0.0.0/8'
set service nat source rule 199 'exclude'
set service nat source rule 199 outbound-interface 'dp0bond1'
set service nat source rule 199 source address 'PUBLIC_INGRESS'

Commit the change

commit
save
exit

 

It is now possible to deploy the IBM Container Service Clusters to the Secure Perimeter Segment in a Secure Perimeter in IBM Cloud.

A sample solution that can be deployed in a Secure Perimeter Segment is described in Implementing a Dedicated Solution Pattern.

If you are an IBM Cloud Dedicated customer and want to use Kubernetes applications integrated with Cloud Foundry applications and services in your Dedicated environment, see Kubernetes and Cloud Foundry integration in IBM Cloud Dedicated.

Join The Discussion

Your email address will not be published. Required fields are marked *