Skill Level: Advanced

This article is meant for Infrastructure architects who have detailed knowledge on networking and networking aspects of IBM Cloud IaaS (Softlayer)

This article discusses various architecture patterns involving custom private addressing which can be used to build resilient application Infrastructure.


To undserstand this article one should have prior knowledge of cloud  and networking fundamenals. One should also know in details about IBM Cloud concepts.


  1. Introduction

    In this article I will discuss on how customers can build scalable and resilient application architectures leveraging custom private addressing on IBM Cloud Infrastructure As A Service (IaaS). These patterns should be tested prior to using them in production scenarios. Before we get into details of CPA architecture patterns lets discuss on IBM cloud IaaS capabilities. IBM provides range of services offerings which can be used for building application Infrastructure with no long-term commitments. IBM has broad and deep set of services that it provides out of its global presence. As of now IBM has 50 data centers across geographies with several POP’s (Point of Presence). IBM Cloud has 10gbps connection running inside its data centers. One of the key benefits that customers can leverage out of IBM is free Ingress and egress bandwidth within IBM data centers along with 100% network uptime guarantee, which also adds to lot of cost savings in longer run.  Before getting deeper into application resiliency it is Important to understand IBM Cloud data center architecture.



    IBM Cloud private backbone connects all the POP’s into one globe of spanning network, enabling customers to move workloads and data between all our data centers without additional cost. The POP’s also provides network edge functionality allowing customers to connect its network to IBM cloud data centers and private backbone. POP’s are spread around the globe to ensure Public and Private connectivity from anywhere with minimum number of hops. They also provides local Internet breakout through multiple peering and transit partners.



    All IBM Cloud data centers and POP are connected by the private network backbone. This private network is separate from the public network, and it enables connectivity to services in IBM Cloud data center around the world. Moving data between data center is done via multiple 10Gbps or 40GBps connections to the private network.


  2. IBM Cloud Data Center Architecture

    IBM Cloud data centers is based on POD architecture where in each data center comprises of minimum three POD’s, with each POD comprising of racks, network, security and storage along with Power backup generators and Environmental controls. All the facility components of data center are highly available by the factor of n+1. These includes PDU’s, backup power generator, UPS battery backup unit etc.




     **With IBM Cloud private network, you can transfer a large size of data within a secured environment for no charge.

    Each POD has a rack and each server rack holds approximately 5000 servers. Each POD has following characteristics:

    1. At least one FCR and BCR.
    2. FCR manages public VLAN.
    3. BCR manages private VLAN.
    4. FCR and BCR can have multiple VLANs.
    5. If a resource needs to be assigned to both public and private IP addresses then select the VLAN of FCR and BCR in the same POD.




  3. IBM Cloud Networking

    The resiliency of customer’s application Infrastructure would depend primarily on its network design along with resiliency of compute and storage. When customer requests for Infrastructure customer is provided with one public and one private VLAN along with one no charge management VLAN which is meant to manage Infrastructure components. Customer can request further VLAN’s (public or private) based on his requirements. Each VLAN can accommodate maximum of 64 machines per VLAN within CIDR range allocated by IBM. These VLAN’s are issued in a same data center (in same or different POD’s). All the servers are provisioned in these VLAN’s. IBM holds responsibility to manage SLA of 100% for availability of VLAN’s which constitutes the network. So IBM guarantees resiliency of its Network.



    Host networking: Each host within the design has two redundant pairs of ethernet connections into customer switch.

    Public VLAN: The purpose of public VLAN is to expose application workload to Internet.

    Private VLAN: The purpose of private VLAN is in contrast to what public VLAN does i.e one can’t reach directly to private VLAN from public Internet. While Implementing Hybrid Infrastructure with customer data center private VLAN is been used.  Customer’s first server purchase will generate a private VLAN for their account with 64 private IPs. All of the future server purchases will be placed on the same VLAN so your servers can communicate over the private network. If customer expect to have more than 64 servers in same private VLAN, IBM support helps to increase the size of the private network to allow for future growth. If customer’s servers are in multiple locations then one would need to Enable Private Network Spanning to enable communication between these networks. The three key areas of private network are:

    1. Server-to-Server: Two or more servers can communicate over the private network at Gigabit speed. Bandwidth is free and unmetered.
    2. Server-to-Services: Customers private VLAN and all connected servers have access to NAS, SAN, DNS resolver, OS update servers etc.
    3. VLAN to VLAN: Customers can use SSL, PPTP and IPSec VPN for their workloads Integration and access requirements.

    Management VLAN: The purpose of management VLAN is for the customer to get access to his machines and to administrative Jobs. This is also referred to as an outoftheband network.

    The public, private and management network constitutes a triple network architecture of IBM Cloud.


  4. Custom Private Addressing (CPA)

    Custom private addressing (CPA) is a SoftLayer feature that lets the customer choose the private IP address range used by their servers. The customer may define up to 5 CPA Networks using any RFC 1918-valid IP address range (10.x.x.x, 192.168.x.x, 172.16.x.x). Customer-defined subnets split each CPA Network into smaller address ranges, with each subnet associated with a location—a data center and Pod—in IBM Cloud’s  global private network.  The customer CPA account may have up to 5 Networks. Each Network’s size may vary from a /24 (255 hosts) to a /16 (65,535 hosts). Your CPA account may have a maximum of 200 subnets, spread across as many Networks as you use. Subnet size may vary from /29 (6 hosts) to /24 (255 hosts). CPA could be used alternatively to private VLAN as in VLAN customer has to use IBM provided range and use Portable subnets for secondary IP’s.  

    All private subnets within a given CPA Network are routed together, creating a flat, globally-routed private address space. Private IPs are not routed across CPA Networks, making each Network a segregated private routing domain. By default, the private interfaces of all servers that share a CPA Network are routed together, regardless of server location. There is no additional charge for custom private addressing.

    To custom private addressing, an external SoftLayer account is precisely like another private Network: CPA communicates freely within a Network using private IPs, but it cannot use them to communicate outside that Network.



  5. Subnets

    Subnets are used within VLAN’s to provide additional IP’s for virtual machines hosted on these VLAN’s or for secondary IPs on servers or secondary interfaces. Another typical purpose of these addresses would be for cluster or HA IPs as they are tracked by our routers via ARP so they can quickly and easily be moved around. The standard format of these subnets will includes its own Network, Gateway, and Broadcast address so three of the IPs will be used right away. The gateway points to FCR or BCR associated with this VLAN and helps in routing with other VLAN’s. This type of subnet is also called Secondary on VLAN.

    There is another type of subnet called Routed on VLAN which allow you to use all the IPs in the subnet because it would no longer have its own Network, Gateway, or Broadcast IPs. The purpose of this type of subnet it to allow you to have secondary alias IPs on a server that are only used for things such as webserver IPs or for HA. The benefit of this type of subnet is you get more IPs and they can be moved around without manual steps (adding entries in OS networking) in the customer portal or via a ticket since they are tracked by our routers using ARP. 

    Static subnets are the second type of subnet that are intended to be used for webservers, email, or other services hosted on a server that needs several additional IPs to allow connections too. Static Subnets are unique in that you have to specify another existing IP that is already assigned to a server that the subnet is directly routed too. This allows you the flexibility to route the subnet as you need and allows you to move it around to different servers, but in order to do that you have to change the routed location in the customer portal.

  6. Designing resilient application Infrastructure solutions on IBM Cloud

    The best solution is the one which has best network design. When designing the solution on IBM cloud the placement of VLAN’s matters the most. The VLAN’s could be placed as follows:

    a)      Same POD different row in a rack.

    b)      Different POD in same data center.

    c)       Different POD in different data center in same region.

    d)      For DR the workloads could be placed in POD’s spanning data centers across regions.


  7. Designing Patterns using Custom Private Addressing

    Cloud solutions are based on three major components – Compute, Storage and Network. Compute and Storage plays a small role in overall cloud solution design. The best Infrastructure solution created would depend upon how you plan your network design where in compute and storage are part of this network. This will also help reduce the cost of solution. As discussed in the article above there are options like VLAN and CPA which one can use to design network architecture for their solution. CPA could be one of the choice.  


    Pattern1: Custom Private Addressing (CPA) network spanning within same DC and POD and holding multiple subnets which holds application Infrastructure workload. Internally all the workload can Interact with each other directly among the subnets.




    Pattern2: Custom Private Addressing (CPA) network spanning two PODs in same data center with each CPA in different subnet range. Workload in one CPA network could interact with the other workload through L3 routing where as within CPA they can interact directly.




    Pattern 3: CPA spanning across IBM cloud data centers with same subnet range. The workload in each subnet could interact with each other directly without need for L3 routing .


    Pattern 4: Two CPA networks with different subnets Interacting with each other through L3 routing. Since both are different CPA networks with different subnets they communicate via L3 routing.



    Pattern 5: Using Citrix Netscalar Load Balancing Service to load balance application workload.Fig9a

    Citrix NetScaler is a powerful web application delivery appliance and load balancer. A Citrix NetScaler is capable of being deployed using high availability (HA) by configuring an HA VPX pair.Citrix NetScaler is the only product that can load balance traffic on both the public and private networks.

    NetScaler also offers full Layer 4 and Layer 7 load balancing functionality. Utilizing a web application firewall, SSL offloading, and custom load balancing rule definitions – the NetScaler is a network protection and traffic optimization toolkit.

    Pattern 6: Balancing traffic for workload in private network.


    The IBM Cloud load balancer service distributes traffic among multiple server instances (bare metal and virtual server) that reside locally, within the same data center. A publicly-accessible, fully qualified domain name is assigned to your load balancer service instance. You may use this domain name to access your applications hosted behind the load balancer service. The backend compute instances hosting your application must be on an IBM Cloud private network.

    As a best practice, it is recommended that one should provision their backend servers as ‘private-only’, unless they require direct public connectivity. This practice helps achieve better security, and it preserves your public IP address. The applications hosted on these backend servers are still accessible over public network using the load balancer.

    One may define up to ten front-end application ports (protocols) or proxy ports and map them to respective ports (protocols) on the back-end application servers. The fully qualified domain name assigned to your load balancer service instance and the front-end application ports are exposed to the external world. The incoming user requests are received on these ports.

    On the other hand, the back-end ports are only known internally. These back-end ports may or may not be the same as the front-end ports. As an example, the load balancer may be configured to receive incoming web/HTTP traffic on front-end port 80, while the back-end servers are listening on custom port 81.

  8. References

    a)       https://knowledgelayer.softlayer.com/es/faqs/1487#7339

    b)      https://knowledgelayer.softlayer.com/procedure/allowing-servers-communicate-private-network-across-multiple-vlans

    c)       https://knowledgelayer.softlayer.com/learning/utilizing-subnets-and-ips

    d) https://knowledgelayer.softlayer.com/articles/web-application-hosting-softlayer


Join The Discussion