Ambassador is an API gateway technology that is built on top of Envoy with first-class Kubernetes integration. In this tutorial, we’ll go through the steps of setting up Ambassador, integrating it with the IBM Cloud Kubernetes Service (IKS), and showing a brief example of it in use. The authoritative documentation on use and configuration will be on the Ambassador website.

Ambassador is used for forwarding and filtering North/South traffic in your Kubernetes cluster. Web requests come through Ambassador where they are forwarded to the correct backend deployment inside Kubernetes. Ambassador takes over a LoadBalancer external IP. Many other software pieces do this work, including Kubernetes Ingress, the IKS Application Load Balancer (ALB), and the Istio-ingressgateway. There is a lot of innovation happening at this part of the stack as the industry iterates in the space.

Brief comparison to other tools

Both Ambassaodor and Istio-ingressgateway leverage Envoy(https://www.envoyproxy.io/). IKS ALB leverages Nginx and is configured through Kubernetes Ingress. Kubernetes Ingress is just an API, but the default implementation is based on Nginx. With pretty much any of these tools, you can do common edge load balancer operations including rewrite, ssl termination/upgrade, header management, traffic splitting, etc. Ambassador and istio-ingressgateway boast gRPC support due to their use of Envoy. The real difference is in flexibility and expressiveness of the configuration. Ingress configuration has largely remained stagnant at the API layer (upstream in Kubernetes) so most of the common actions e.g. rewrite, are done as vendor-specific extensions. Istio-ingressgateway leverages a CRD called gateway to provide configuration. Finally, as we’ll see, Ambassador embeds yaml directly into an annotation on the Kubernetes service, identifying it for external publishing and providing configuration.

Installation and configuration

The official quick start guide on the Ambassador website works wonderfully on IKS. This tutorial goes further and integrates Ambassador with the IKS automatic DNS name and automatic Let’s Encrypt SSL certificate. Make sure you’ve completely uninstalled Ambassador if you had it installed before starting this tutorial.

  1. Get the source code.

     $ git clone https://gitlab.com/nibalizer/ambassador-iks
     $ cd ambassador-iks
    
  2. Apply the ambassador yaml configuration.

     $ kubectl apply -f ambassador-rbac.yaml
     service/ambassador-admin created
     clusterrole.rbac.authorization.k8s.io/ambassador created
     serviceaccount/ambassador created
     clusterrolebinding.rbac.authorization.k8s.io/ambassador created
       deployment.extensions/ambassador created
    

    This creates a service, a ClusterRole, a ServiceAccount, a ClusterRoleBinding, and a deployment. ClusterRoles and ClusterRoleBindings are not namespaced. The other resources are created in the kube-system namespace. This is both to keep them out of the way, and to better integrate with the existing IKS configuration.

  3. Verify that the daemon came up by checking that the pods are in Status: Running.

     $ kubectl -n kube-system get pod -l service=ambassador
     NAME                         READY   STATUS    RESTARTS   AGE
     ambassador-f6b9b96cc-49854   1/1     Running   0          4h8m
     ambassador-f6b9b96cc-5t2bl   1/1     Running   0          4h8m
     ambassador-f6b9b96cc-9jw4f   1/1     Running   0          4h8m
    
  4. Inspect the deployment for ambassador (snipped for brevity/clarity)

     $ kubectl -n kube-system get -o yaml deploy/ambassador
    
     yaml
     apiVersion: extensions/v1beta1
     kind: Deployment
     metadata:
       annotations:
         deployment.kubernetes.io/revision: "2"
       creationTimestamp: "2019-03-27T16:26:10Z"
       generation: 2
       labels:
         service: ambassador
       name: ambassador
       namespace: kube-system
       resourceVersion: "56509"
       selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/ambassador
       uid: 08058fd9-50ad-11e9-851d-da993851ee33
     spec:
       progressDeadlineSeconds: 2147483647
       replicas: 3
       revisionHistoryLimit: 10
       selector:
         matchLabels:
           service: ambassador
       strategy:
         rollingUpdate:
           maxSurge: 1
           maxUnavailable: 1
         type: RollingUpdate
       template:
         metadata:
           annotations:
             consul.hashicorp.com/connect-inject: "false"
             sidecar.istio.io/inject: "false"
           creationTimestamp: null
           labels:
             service: ambassador
         spec:
           containers:
           - env:
               - name: AMBASSADOR_NAMESPACE
               value: default
             image: quay.io/datawire/ambassador:0.51.2
             imagePullPolicy: IfNotPresent
             livenessProbe:
               failureThreshold: 3
               httpGet:
                 path: /ambassador/v0/check_alive
                 port: 8877
                 scheme: HTTP
               initialDelaySeconds: 30
               periodSeconds: 3
               successThreshold: 1
               timeoutSeconds: 1
             name: ambassador
             ports:
             - containerPort: 80
               name: http
               protocol: TCP
             - containerPort: 443
               name: https
               protocol: TCP
             - containerPort: 8877
               name: admin
               protocol: TCP
             readinessProbe:
               failureThreshold: 3
               httpGet:
                 path: /ambassador/v0/check_ready
                 port: 8877
                 scheme: HTTP
               initialDelaySeconds: 30
               periodSeconds: 3
               successThreshold: 1
               timeoutSeconds: 1
             resources:
               limits:
                 cpu: "1"
                 memory: 400Mi
               requests:
                 cpu: 200m
                 memory: 100Mi
             terminationMessagePath: /dev/termination-log
             terminationMessagePolicy: File
           dnsPolicy: ClusterFirst
           restartPolicy: Always
           schedulerName: default-scheduler
           securityContext: {}
           serviceAccount: ambassador
           serviceAccountName: ambassador
     status:
       availableReplicas: 3
       conditions:
       readyReplicas: 3
       replicas: 3
       updatedReplicas: 3
    

    The first thing to notice are the namespace and labels. This is running in the kube-system namespace and is labeled with service: ambassador.

     metadata:
       generation: 2
       labels:
         service: ambassador
       name: ambassador
       namespace: kube-system
    

    The next thing to notice is that we’re setting the namespace for Ambassador to scan to the default namespace. If you want to run your apps in a different namespace, configure that here.

     - env:
       - name: AMBASSADOR_NAMESPACE
         value: default
    

    Finally, the service account configuration is available as well. Ambassador uses this service account to connect to the Kubernetes API and watch for changes to service and other objects.

     serviceAccount: ambassador
     serviceAccountName: ambassador
    

    We can further inspect the ClusterRoleBinding, and more importantly, the ClusterRole as well.

     $ kubectl get ClusterRole/ambassador -o yaml
    
     apiVersion: rbac.authorization.k8s.io/v1
     kind: ClusterRole
     metadata:
       annotations:
       creationTimestamp: "2019-03-27T16:26:10Z"
       name: ambassador
       resourceVersion: "52692"
       selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/ambassador
       uid: 07d033bd-50ad-11e9-851d-da993851ee33
     rules:
     - apiGroups:
       - ""
       resources:
       - namespaces
       - services
       - secrets
       verbs:
       - get
       - list
       - watch
    

    The above says it will grant get, watch, and list operations on all namespaces, services, and secrets to the ServiceAccount. These operations are necessary so that Ambassador can pick up SSL private data out of a secret and so that Ambassador can watch service objects for changes and apply configuration to the primary daemon.

    The next step is to configure a service of type LoadBalancer in Kubernetes to accept external traffic for the Ambassador deployment. In the quick-start guide, you create a custom service just for this, however we won’t do this. Instead, what we’ll do is re-use the existing service currently used by the IKS ALB. This gives us the IP already assigned to the DNS name of the cluster.

  5. Inspect the existing cluster and networking setup

     $ ibmcloud ks cluster get nibz-nightly-2019-03-26
     Retrieving cluster nibz-nightly-2019-03-26...
     OK
    
     Name:                   nibz-nightly-2019-03-26   
     ID:                     c1961757afbb44aba41ab089562bba91   
     State:                  normal   
     Created:                2019-03-26T08:02:42+0000   
     Location:               wdc06   
     Master URL:             https://c1.us-east.containers.cloud.ibm.com:30437   
     Master Location:        Washington D.C.   
     Master Status:          Ready (12 hours ago)   
     Ingress Subdomain:      nibz-nightly-2019-03-26.us-east.containers.appdomain.cloud   
     Ingress Secret:         nibz-nightly-2019-03-26   
     Workers:                3   
     Worker Zones:           wdc06   
     Version:                1.12.6_1546   
     Owner:                  skrum@us.ibm.com   
     Monitoring Dashboard:   -   
     Resource Group ID:      2a926a9173174d94a6eb13284e089f88   
     Resource Group Name:    default   
     $ host nibz-nightly-2019-03-26.us-east.containers.appdomain.cloud
     nibz-nightly-2019-03-26.us-east.containers.appdomain.cloud has address 169.63.132.38
     $ kubectl get svc -n kube-system
     NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
     kube-dns                                         ClusterIP      172.21.0.10      <none>          53/UDP,53/TCP                12h
     kubernetes-dashboard                             ClusterIP      172.21.207.120   <none>          443/TCP                      12h
     metrics-server                                   ClusterIP      172.21.226.71    <none>          443/TCP                      12h
     public-crc1961757afbb44aba41ab089562bba91-alb1   LoadBalancer   172.21.197.63    169.63.132.38   80:31574/TCP,443:31121/TCP   12h
    

    We can see that the domain name generated for the cluster nibz-nightly-2019-03-26.us-east.containers.appdomain.cloud is already mapped to the External IP of the ALB LoadBalancer (svc/public-crc1961757afbb44aba41ab089562bba91-alb1 in this example).

    Before we can proceed, we need to disable the IBM managed ALB behind that LoadBalancer.

  6. Get the ALB name.

     $ kubectl get svc -n kube-system | grep alb
     public-crc1961757afbb44aba41ab089562bba91-alb1   LoadBalancer   172.21.197.63    169.63.132.38   80:31574/TCP,443:31121/TCP   13h
    
  7. Set the env variable.

     $ ALB_ID=$(kubectl get svc -n kube-system | grep alb | cut -d " " -f 1)
     $ echo $ALB_ID 
     public-crc1961757afbb44aba41ab089562bba91-alb1
    
  8. Disable alb.

     $ ibmcloud ks alb configure --albID $ALB_ID --disable-deployment
     Configuring ALB...
     OK
    

    Note the IKS_BETA_VERSION=1.0 command structure. (ibmcloud ks alb-configure would be the old way.)

    It might take some time for these pods to come down, so just be patient.

  9. Monitor the ALB pods until they go away

     $ kubectl get pods -n kube-system | grep alb
     public-crc1961757afbb44aba41ab089562bba91-alb1-89d685cfb-fvvpl   4/4     Running            0          13h
     public-crc1961757afbb44aba41ab089562bba91-alb1-89d685cfb-h6z5p   4/4     Running            0          13h
    
     $ kubectl get pods -n kube-system | grep alb
    
  10. Delete the svc/ambassador from kube-system (if it’s there).

    $ kubectl -n kube-system delete svc/ambassador
    
  11. Find the alb service provided by IBM.

    $ kubectl -n kube-system get svc| grep alb
    public-cr1d251d6b8c9f4ab093405333a4570d83-alb1   LoadBalancer   172.21.138.109   169.63.140.238   80:30549/TCP,443:30345/TCP   8h
    $ kubectl -n kube-system get svc| grep alb | cut -d " " -f 1
    public-cr1d251d6b8c9f4ab093405333a4570d83-alb1
    
  12. Set the env variable.

    $ ALB_SVC=$(kubectl -n kube-system get svc| grep alb | cut -d " " -f 1)
    $ echo $ALB_SVC 
    public-cr1d251d6b8c9f4ab093405333a4570d83-alb1
    
  13. Edit the ALB_SVC.

    $ kubectl -n kube-system edit svc/${ALB_SVC}
    

    Change:

    selector:
      app: public-cr1d251d6b8c9f4ab093405333a4570d83-alb1
    

    to

    selector:
      service: ambassador
    

    Ambassador should now be installed! Now deploy a simple test application and do some testing.

    $ kubectl apply -f qotm.yaml 
    service/qotm created
    deployment.extensions/qotm created
    

    In the service of the qotm configuration, you can see the Ambassador configuration. This will expose the qotm service publicly at /qotm/. Note that because we are running the ambassador pods in the kube-system namespace, we need to specify the namespace on in the qotm service entry (Line 13).

    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: qotm
      annotations:
        getambassador.io/config: |
          ---
          apiVersion: ambassador/v1
          kind:  Mapping
          name:  qotm_mapping
          prefix: /qotm/
          service: qotm.default
    spec:
      selector:
        app: qotm
      ports:
      - port: 80
        name: http-qotm
        targetPort: http-api
    

    Now, test!

  14. Use curl to test the service we deployed.

    $ curl -I nibz-nightly-2019-03-27.us-east.containers.appdomain.cloud/qotm/
    
    HTTP/1.1 200 OK
    content-type: application/json
    content-length: 172
    server: envoy
    date: Wed, 27 Mar 2019 23:02:17 GMT
    x-envoy-upstream-service-time: 3
    

    Here we can see that we got a 200 OK response, the server was Envoy, and everything seems to be working!

Enable Ambassador to use SSL

IBM Cloud creates a DNS name and a Let’s Encrypt certificate for you. Usually these are used by the IBM ALB, but in this case we’ll use Ambassador. We already set up DNS for use with IBM Cloud, so let’s now use the SSL certificate.

The certificate is stored in a secret in the default namespace. The secret is the same name as your cluster.

$ kubectl get secret | grep Opaque
nibz-nightly-2019-03-27                Opaque
  1. Inspect the certificate with openssl.

     $ kubectl get secret nibz-nightly-2019-03-27 -o yaml | grep tls.crt: | cut -d " " -f 4 | base64 -d | openssl x509 -text -noout -in - 
     Certificate:
         Data:
             Version: 3 (0x2)
             Serial Number:
                 04:80:c8:b2:e1:e0:f6:02:29:ff:57:e6:f6:15:a7:8b:6f:72
             Signature Algorithm: sha256WithRSAEncryption
             Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
             Validity
                 Not Before: Mar 27 07:28:45 2019 GMT
                 Not After : Jun 25 07:28:45 2019 GMT
             Subject: CN = nibz-nightly-2019-03-27.us-east.containers.appdomain.cloud
             Subject Public Key Info:
                 Public Key Algorithm: rsaEncryption
                     RSA Public-Key: (2048 bit)
                     Modulus:
    

In order to turn on TLS for Ambassador, we need to modify the ALB service definition again.

  1. Edit the ALB_SVC.

     $ kubectl -n kube-system edit svc/${ALB_SVC}
    
  2. Add the getambassador.io/config section to the annotations. Note: You need to use your cluster name not nibz-nightly-2019-03-27. You do need the .default at the end as this refers to the secret in the default namespace.

     apiVersion: v1
     kind: Service
     metadata:
       annotations:
         getambassador.io/config: |
           ---
           apiVersion: ambassador/v1
           kind: Module
           name: tls
           config:
             server:
               enabled: True
               redirect_cleartext_from: 80
               secret: nibz-nightly-2019-03-27.default
         service.kubernetes.io/ibm-ingress-controller-public: 169.63.140.238
         service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: wdc06
       creationTimestamp: "2019-03-27T08:23:25Z"
    

    With that, SSL should be enabled!

  3. Test again by using curl.

     $ curl -Li nibz-nightly-2019-03-27.us-east.containers.appdomain.cloud/qotm/
     HTTP/1.1 301 Moved Permanently
     location: https://nibz-nightly-2019-03-27.us-east.containers.appdomain.cloud/qotm/
     date: Wed, 27 Mar 2019 17:45:52 GMT
     server: envoy
     content-length: 0
    
     HTTP/1.1 200 OK
     access-control-allow-credentials: true
     access-control-allow-origin: *
     content-type: text/html; charset=utf-8
     date: Wed, 27 Mar 2019 17:45:52 GMT
     server: envoy
     content-length: 9593
     x-envoy-upstream-service-time: 4
    

    It works! We’re getting a 301 redirect from port 80 to port 443, and the service on 443 has a valid x509 certificate.

Using Ambassador

Now let’s use Ambassador to actually do something. Ambassador supports lots of configuration opportunities, which are better documented on the main site: https://www.getambassador.io. To get our feet wet though, lets take the simple example of adding a response header when the qotm app is hit. This is documented on the website. Add the following configuration to the Ambassador annotations to the qotm service entry.

  1. Edit the qotm service object.

     $ kubectl edit svc/qotm
    
  2. Add the following to the ambassador configuration.

     add_response_headers:
       x-test-static: Ambassador on IKS
    

    You can also change the resource with kubectl apply -f httbin-headers.yaml.

    Now test the result:

     $ curl -LI nibz-nightly-2019-03-27.us-east.containers.appdomain.cloud/qotm/
     HTTP/1.1 301 Moved Permanently
     location: https://nibz-nightly-2019-03-27.us-east.containers.appdomain.cloud/qotm/
     date: Wed, 27 Mar 2019 18:10:45 GMT
     server: envoy
     transfer-encoding: chunked
    
     HTTP/1.1 200 OK
     access-control-allow-credentials: true
     access-control-allow-origin: *
     content-length: 9593
     content-type: text/html; charset=utf-8
     date: Wed, 27 Mar 2019 18:10:45 GMT
     server: envoy
     x-envoy-upstream-service-time: 4
     x-test-static: Ambassador on IKS
    

Note the x-test-static at the end of the response headers.

In this tutorial, you learned how to set up Ambassador on the IBM Cloud Kubernetes Service. You also went beyond the basics and integrated the built-in DNS and TLS certificate functionality. To go further, you can follow some of the guides on the Ambassador website or try out a tool that builds on top of Ambassador like Seldon.

Special thanks to Gregory Hanson, who figured out the alb-disable trick and documented it, and the awesome folks over at Ambassador for writing great docs.

References: