Skill Level: Beginner

Basic understanding of Kubernetes is required.

The recipe describes how to configure IBM Cloud Object Storage as a destination of Ark, a Heptio open source program.
It demonstrates how to deploy Ark to IBM Cloud Private and/or IBM Public Cloud Container Service, and run backups and restores.


In order to start, you need an IBM Cloud Account.

We will use:

  • IBM Cloud Private
  • IBM Cloud Object Storage
  • IBM Container Service
  • Heptio Ark

Heptio Ark helps manage disaster recovery for Kubernetes cluster resources and its persistent volumes from a series of checkpoints. In addition, it can be used for migration of Kubernetes resources from one cluster or namespace to another. Ark helps to replicate Kubernetes deployments to different DevOps environments.


  1. Create an IBM Cloud Object Storage place

    To store Kubernetes backups, you need a destination bucket in an instance of Cloud Object Storage (COS) and you have to configure service credentials to access this instance.


    • The last step in the COS configuration is to define a service that can store data in the bucket. The process of creating service credentials is described in Service credentials. Several comments:
      • Your Ark service will write its backup into the bucket, so it requires the “Writer” access role.
      • Ark uses an AWS S3 compatible API. Which means it authenticates using a signature created from a pair of access and secret keys – a set of HMAC credentials. You can create these HMAC credentials by specifying “{“HMAC”:true}” as an optional inline parameter. See step 3 in the Service credentials guide.


    • After successfully creating a Service credential, you can view the JSON definition of the credential. Under the “cos_hmac_keys” entry there are “access_key_id” and “secret_access_key”. We will use them later.


  2. Install IBM Cloud Private

    If you have not installed IBM Cloud Private before, you can install it according to the instructions on the following page: Cloud Private v2.1.0.1 documentation.

  3. Download Ark

  4. Create an Ark configuration file for IBM Cloud Object Storage

    Ark comes with several examples of integration with different cloud providers, such as AWS, GCP, Azure, and Mino. In this step, we will create an Ark configuration file for your IBM COS instance.

    • Create a directory for IBM COS configuration, e.g.,  <ark_root_directory>/examples/ibm
    • In the created directory, create a configuration file 10-ark-config.yaml with the following content:

    apiVersion: v1

    kind: Secret


      namespace: heptio-ark-server

      name: cloud-credentials


      cloud: |


    # UPDATE ME: the value of “access_key_id” of your COS service credential

    aws_access_key_id = 72c5b9fdfcaf4d0387da8607ba00b2fc

    # UPDATE ME: the value of “secret_access_key” of your COS service credential

    aws_secret_access_key = 0800b6ef46fdd1c44a3759b7ecefee6a3ec5f2deff5b9012


    apiVersion: ark.heptio.com/v1

    kind: Config


      namespace: heptio-ark

      name: default


      name: aws

      # UPDATE ME: name of your bucket

    bucket: ark                           


        # UPDATE ME: region of your COS deployment

    region: us-geo                       

        s3ForcePathStyle: "true"

        # UPDATE ME: the public endpoint of your COS

        s3Url: http://s3-api.us-geo.objectstorage.softlayer.net

    backupSyncPeriod: 1m

    gcSyncPeriod: 1m

    scheduleSyncPeriod: 1m

    restoreOnlyMode: false
    • You should update the credentials (aws_access_key_id, aws_secret_access_key), COS bucket name, its region, and public endpoint. You can find the region and public endpoint in the COS console.

    NOTE: please put attention on different namespaces of the Secret and the Config. Ark from the version 0.7.0 and later allows to run it in any namespace. Heptio recommends “to run the Ark server in one namespace, and place your backups, schedules, restores, and config in a different namespace. You might encounter issues with deleting a single Ark namespace that contains everything.” For more information see Ark namespace recommendations


  5. Deploy Ark into your IBM Cloud Private cluster

    • Configure your kubectl client to access your IBM Cloud Private deployment. See the IBM Cloud Private documentation.
    • Run the following commands from the root Ark directory:
    kubectl apply -f examples/common/00-prereqs.yaml

    kubectl apply -f examples/ibm/10-ark-config.yaml

    kubectl apply -f examples/common/10-deployment.yaml

    NOTE: If you encounter an error related to Config creation, wait for a minute and run the command again. (The Config CRD does not always finish registering in time.)

    • Check that the deployments succeeded:
    kubectl get deployments -n heptio-ark-server -l component=ark
  6. Let's validate the backup-restore capabilities

    We can use the example from Ark distribution. See https://github.com/heptio/ark

    • Deploy the example nginx app:
    kubectl apply -f examples/nginx-app/base.yaml
    • Check the deployment:
    kubectl get deployments -n nginx-example
    • Create a backup for any object matching the app=nginx label selector:
    ark backup create nginx-backup --selector app=nginx

    NOTE: “ark” is the Ark client program that we installed in the Step 3.

    • Check the backup status:
    ark backup get

    The output should be something like


    • In addition, you can see the backup files in your COS console.


    • Now, let’s emulate a disaster with the following:
    kubectl delete namespace nginx-example

    We deleted our nginx deployment.

    • To validate that our example nginx deployment no longer exists, run the following command again:
    kubectl get deployments -n nginx-example
    • Now restore the deployment:
    ark restore create nginx-backup
    • You can check the restore status, by running:
    ark restore get

    The output should be something like:


    • If the restore status is not “Completed” and/or there are warnings or errors, you can get more information by
    ark restore get <RESTORE NAME> -o yaml

    For more information, see the Ark documentation.

    Congratulations!!! You deployed and configured Ark to store its backups on the IBM Cloud Object Storage.

  7. Deploy Ark to the IBM Public Container Service

    • Create an instance of IBM Container Service in the IBM Public Cloud. See Getting started with IBM Cloud Container Service.
    • You will need to download the Kubernetes configuration file and set the KUBECONFIG environment variable, as described in Configuring the CLI to run kubectl.
    • Once you set the KUBECONFIG environment variable, you can deploy Ark into the public container service instance exactly as you deployed it into the private cloud.
    • The Ark CLI command uses the same KUBECONFIG env variable. Therefore, after setting the variable, the Ark CLI will work with your deployment in the public cloud.
    • During deployment of the Ark server, it synchronizes with the backup storage. In our example we used the same storage areas; therefore, if you run “ark backup get” against the public container service, you will get the same list of available backups.
  8. Recovery an IBM Cloud Private deployment on Public Cloud

    Now, let’s demonstrate recovery of a deployment from IBM Cloud Private to the IBM Public Cloud.

    • After you set the KUBECONFIG environment variable to work with the public cloud container service, just repeat the ark CLI restore command:
    ark restore create nginx-backup
    • Now, if you check deployments in your public cloud container service, you will see the nginx deployment there.
    kubectl get deployments -n nginx-example
  9. Conclusion

    We demonstrated how IBM Cloud Object Storage and Heptio Ark can be used to back up and restore Kubernetes deployments in IBM Private and Public Clouds.

    The backup-restore mechanism allows us to move deployments among clouds.

    Ark supports a lot of other features, such as partial backup and restore, restore to another namespace, backup of persistent volumes, and automatic backup scheduler. For more details, see the Ark documentation.

2 comments on"Integration of IBM Kubernetes deployments with IBM Cloud Object Storage via Heptio Ark"

  1. Thanks for the article. Using ark with COS is great combination. I’m using it with the IBM Cloud Container Service.

    Note, the “cloud-credentials” secret in the 10-ark-config.yaml file at the beginning of the article needs to be modified to be stored in the `heptio-ark-server` namespace in order for the heptio ark server to properly deploy.
    For those picking up and using ark do note that it is still under development. There are some issues where kube backups and restores are not perfect but this project is definitely a step in the right direction.

    • Alexey Roytman March 02, 2018

      Hi Dan,
      Thanks for the comments. Heptio changed the Ark namespace policy. Ark from the version 0.7.0 and later allows to run it in any namespace. Heptio recommends “to run the Ark server in one namespace, and place your backups, schedules, restores, and config in a different namespace. You might encounter issues with deleting a single Ark namespace that contains everything.”
      I updated the recipe.
      Thanks again.

Join The Discussion