This article presupposes that you already have some familiarity with Kubernetes deployments on IBM Cloud Kubernetes Service (IKS) clusters. Developers with knowledge using Kubernetes secrets and how they are configured in Kubernetes deployments will find the information herein useful.


Kubernetes ingress is a collection of routing rules that govern how external users access services running in a Kubernetes cluster. Ingress, added in Kubernetes v1.1, exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.

In IBM Cloud-based Kubernetes deployments, a public ingress ALB (Application Load Balancer) is provided by default. The deployment is exposed to the public by using a public cloud domain. If we want to secure the deployment on a dedicated cluster using https on a custom domain name, we need to configure a private Ingress ALB using the custom domain certificate. An app-private secret needs to be created and used in the ingress resource configuration for the incoming SSL connections to work. This is described in

An ingress YAML file defines the Kubernetes ingress with a configured secret. Here is a sample ingress YAML file that does the following:

"spec": {
    "tls": [
        "hosts": [
          "<domain name>"
        "secretName": "app-private"

"app-private" is a Kubernetes ‘secret’ file. For reference, Kubernetes ‘secrets’ are explained here:

The "app-private" is a YAML file whose format looks like this:

  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "app-private",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/secrets/app-private",
    "uid": "2f7f08b6-deb1-11e8-8884-56886f4688e7",
    "resourceVersion": "11421125",
    "creationTimestamp": "2018-11-02T15:08:42Z"
  "data": {
    "tls.crt": "<base64-encoded-cert>",
    "tls.key": "<base64-encoded-client key>"
  "type": "Opaque"

The “tls.crt” file referenced above is the domain certificate issued by the CA (Certificate Authority). The CA may issue a hierchical chain of certificates instead of a single certificate on the domain. In a Kubernetes cluster using private Ingress ALB, configuring secure access for a domain that has chain certicates issued for it requires additional steps that are not currently available in one comprehensive place. A developer may find the configuration steps outlined in this document useful in such a situation.

Working with a chain of certificates to configure secrets and the ingress

Chain certificates are a hierarchy of certificates on a domain. They usually consist of a root certificate, an intermediate certficate and a domain certificate issued by a certificate authority.

For example, IBM Internal Certificate Authority (IBMCAPKI) issues 3 server certificates on internal IBM domains: domain certificates, root certificates, and intermediate certificates. Deatils on the IBMCAPKI certificate process are documented at:

The below steps are illustrated using the IBM Internal Certificate Authority issued Server Certificates. Note that the concept is also applicable to another CA (Certificate Authority) provided chain certificates.

The steps below assume that the chain server certificates have already been requested and issued by IBMCAPKI for an intranet IBM domain and that these certificates will need to be deployed in the Ingress private ALB (Application Load Balancer) on a Kubernetes cluster.

  1. Download the domain certificate as a cert.crt (or using any name.crt). Download the .der files for the root and intermediate certificates from the IBMCAPKI website (
  2. Generate the app-bundle.pem file from cert.crt, carootcert.der, and caintermediatecert.der: i. Create cert.pem, carootcert.pem and caintermediatecert.pem first by converting a DER file (.crt .cer .der) to PEM:

       openssl x509 -inform der -in certificate.cer -out certificate.pem

    Note: Use this syntax on all the 3 .crt files one by one.

       openssl x509 -inform der -in cert.crt -out cert.pem
       openssl x509 -inform der -in carootcert.der -out carootcert.pem
       openssl x509 -inform der -in caintermediatecert.der -out caintermediatecert.pem

    ii. Then create app-bundle.pem as follows:

       cat cert.pem caintermediatecert.pem carootcert.pem >>app-bundle.pem
  3. Get the modulus of the private key.

    openssl rsa -modulus -noout -in private.key | openssl md5

    The modulus of the key should match modulus of app-bumdle.pem for verifications on the certs and the private key.

  4. Create the app-bundle.pem.base64 file.

    openssl base64 -in app-bundle.pem -out app-bundle.pem.base64
  5. Remove new lines from the app-bundle.pem.base64 file by using your favorite editor and then save it. Note: If using the vi editor, you can open the file in vi and remove new lines by using: %s/$\n//g and wq.

    The resulting file should look like this:

    cat app-bundle.pem.base64
  6. Similarly, create the base64 file for the private key.

    openssl base64 -in private.key -out private.key.base64

    Remove new lines from this file by using an editor and save it.

  7. Create a secret file (for example, app_private file) that will be used by Kubernetes.

    Edit your app_private file with the tls.crt set to the contents of app-bundle.pem.base64, and the tls.key set to the contents of private.key.base64.

    cat app_private
    apiVersion: v1
    kind: Secret
     name: app-private
    type: Opaque
     tls.crt: <copy/paste contents of app-bundle.pem.base64 file>
     tls.key: <copy/paste contents of private.key.base64 file>
  8. Now, apply the app_private file to the Kubernetes back end. This creates/updates the secret named app_private in the Kubernetes backend.

    kubectl apply -f app_private
  9. Check logs on the node to verify that the Ingress is successfully configured for SSL.

    kubectl logs -n kube-system <private system pod name> nginx-ingress

    Note: To get a list of system nodes, use kubectl -n kube-system get pods.

    I1009 13:11:55.600034       1 nginx.go:394] Updating NGINX configuration
    I1009 13:11:55.600050       1 nginx.go:396] AddOrUpdateIngress filename /etc/nginx/conf.d/default-ingress.conf
    Entry ::::::  {true map[KEYLESS:[False]]}
     upstream {
       server  ;
       keepalive 64;
     upstream {
       server  ;
       server  ;
       server  ;
       server  ;
       server  ;
       keepalive 64;
     server {
       listen 80;
           listen 443 ssl;
           ssl_certificate /etc/nginx/ssl/default-app-private.pem;
           ssl_certificate_key /etc/nginx/ssl/default-app-private.pem;
     . . . .  
     . . . .
    I1009 13:11:55.611004       1 nginx.go:662] NGINX configuration file had been updated
    2018/10/09 13:11:55 [notice] 22383#22383: signal process started
    I1009 13:11:55.743831       1 status.go:200] Generating a kube event for ingress ingress
    I1009 13:11:55.750209       1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress",    UID:"4b0701f3-c81c-11e8-ad39-d6197118b088", APIVersion:"extensions", ResourceVersion:"1879336", FieldPath:""}): type: 'Normal'  reason: 'Success' Successfully applied ingress resource.
    10. [OPTIONAL] Additional checks can be done by ‘bash’ing into the system node as follows:
    $ kubectl exec -ti -n kube-system <private system pod name> bash
    Defaulting container name to nginx-ingress.
    Use 'kubectl describe pod/private-cr9f334d3775cd45499ac92dac627a6a2f-alb1-6fcb58c7f9c7x84' to see all of the containers in this pod.
    root@private-cr9f334d3775cd45499ac92dac627a6a2f-alb1-6fcb58c7f9c7x84:/# cd /etc/ingress/ssl
    bash: cd: /etc/ingress/ssl: No such file or directory
    root@private-cr9f334d3775cd45499ac92dac627a6a2f-alb1-6fcb58c7f9c7x84:/# cd /etc/nginx/ssl
    root@private-cr9f334d3775cd45499ac92dac627a6a2f-alb1-6fcb58c7f9c7x84:/etc/nginx/ssl# ls -l
    total 12
    -rw-r--r-- 1 root root 6995 Oct  9 16:17 default-app-private.pem
    -rw-r--r-- 1 root root 2967 Aug 27 21:50 default.pem
    root@private-cr9f334d3775cd45499ac92dac627a6a2f-alb1-6fcb58c7f9c7x84:/etc/nginx/ssl# cat default-app-private.pem
    . . . .
    -----END RSA PRIVATE KEY-----
    . . . .
    -----END CERTIFICATE-----
    . . . .
    -----END CERTIFICATE-----
    . . . .
    -----END CERTIFICATE-----

The above code sample confirms a successful configuration of nginx by using the above certificate and private key.


This article is based on the steps I developed to configure and deploy chain certificates on a Kubernetes cluster in the IBM GDPR services project for the IBM CDO successfully. This write-up is intended to be a valuable reference and time saver for developers that need to perform similar configuration that involve a hierarchy of chain certificates that need to be configured in a privare Ingress ALB in Kubernetes.