In this tutorial, you will install and configure the Kasten K10 by Veeam backup service to a Red Hat OpenShift single-node cluster. You will also configure the backup of a container application running on the single-node cluster and store the backup to external Amazon S3-compatible IBM Cloud Object Storage.

Kasten K10 is available in two main editions, Free Kasten K10 and Kasten K10 Enterprise. The Kasten product page contains a side-by-side comparison of the K10 editions, but each edition uses the same container images and follows an identical installation process.

Free: The default Kasten K10 starter edition is provided at no charge and is intended for evaluation or for use in smaller or non-production clusters. It is functionally the same as the Enterprise edition but limited from a support and scale perspective.

Enterprise: Customers choosing to upgrade to the Enterprise edition can obtain a license key from Kasten or install from cloud marketplaces.

High-level architecture diagram

alt text

Backup flow

alt text

Prerequisites

Kasten K10 can be installed in a variety of different environments and on a number of Kubernetes distributions.

To ensure a smooth installation experience, be sure to review the pre-flight checks in the Kasten K10 documentation.

To install Kasten K10, you first need the Helm package manager (v3.0.0+) and the Kasten Helm charts repository. Installing K10 using helm will also automatically create a new service account to grant K10 the required access to Kubernetes resources.

Note: You must have S3-compatible Object Storage available along with its access key, secret key, endpoint URL, region and bucket name.

Install the Helm package manager.
Add the Kasten Helm charts repository using the following command:
```
$ helm repo add kasten https://charts.kasten.io/
```
Create the namespace where Kasten will be installed. By default, the installation creates the namespace kasten-io.
```
$ kubectl create namespace kasten-io
```
The Kasten K10 installer creates the persistent volume using the default storage class in the Red Hat OpenShift cluster, so you must ensure that the TopoLVM storage class is set as the default. To set a StorageClass as the cluster-wide default, add the following annotation to TopoLVM storage class metadata:
```
storageclass.kubernetes.io/is-default-class: "true"
```
Install the Red Hat OpenShift CLI.

Steps

There are two methods for installing Kasten K10 on Red Hat OpenShift:

Helm-based installation
Operator-based installation

Note that operator-based installation is not available for Red Hat OpenShift 4.11. This tutorial provides the steps for Helm-based installation.

Step 1. Install Kasten K10

To install Kasten K10 to Red Hat OpenShift, complete the following steps:

Log in to the Red Hat OpenShift cluster using the following command:
```
$ oc login -u <username> -p <password>  <sno api address>:6443
```

Install Kasten K10 in Red Hat OpenShift using the following Helm command:

$ helm install k10 kasten/k10 --namespace=kasten-io \
 --set scc.create=true

Check that installation is complete and all pods are up and running in the kasten-io namespace:
```
$ oc get pods –namespace kasten-io
```

Create a route for accessing the Kasten dashboard:

$ oc expose svc  gateway  --name  <route name>  --namespace kasten-io

Get the route URL to connect to the Kasten K10 dashboard. The value displayed in the host/port column is the route created by OpenShift.
```
$ oc get route
```

Note: You can now access the K10 dashboard using the route URL: http://<host and port value displayed by oc get route>/k10/#/

Step 2. Configure the object storage profile

To store the backup of your container application and its storage volume to external object storage, you must create the Location Profile.

Log in to the K10 dashboard.
In the menu bar, click Settings.
In the left menu, click Locations and in the Location Profiles page, click New Profile.
In the New Profile pop-up window, type a profile name. For storage provider, select S3 Compatible. Enter your S3 access key, S3 secret, endpoint URL, region name and bucket name. When you have added all the profile information, click Save.

Step 3. Create a backup policy

To protect the container application with K10, create a backup policy from the K10 dashboard. Note: To create a valid policy definition and successfully run your backup and restore, be sure to complete the following steps exactly as described.

Log in to the K10 dashboard.
In the Policies panel, click Backup Policies.
In the Policies window, click Create New Policy.
In the New Policy pop-up window, enter a name for your new policy. In the Action section, select Snapshot. For backup frequency, select the frequency and time you want to run the backup; for example, Daily at 10:00 p.m.

Note: Do not click Create Policy yet. If you click Create Policy at this point, the snapshot will be created. However, snapshots are not always durable. First, catastrophic storage system failure will destroy your snapshots along with your primary data. Additionally, in a number of storage systems, a snapshot's lifecycle is tied to the source volume. If the volume is deleted, all related snapshots might automatically be garbage-collected at the same time. We highly recommend that you create backups of your application snapshots.
In the Snapshot Retention panel, click to Enable Backup via Snapshot Exports and complete the following fields and sections:
- Export Frequency: Select Every daily snapshot.
- Export Location Profile: Select the location where you want to export to; for example, ibmcos.
- Select Applications: Select By Name. (This will list namespaces; select the namespace where the application is deployed.)
- Select Application Resources: Select All Resources.
Click to enable the Snapshot Cluster-Scoped Resources and then select All Cluster-Scoped Resources.
In the Location profile for Kanister Action drop-down menu, select the same location profile that you created earlier; for example, ibmcos.
When you have entered all the policy information, click Create Policy.

Step 4. Configure an application-consistent backup

Kasten K10 defaults to volume snapshot operations when capturing data, but there are some situations where it requires customization. For example, a database application requires an application-consistent backup.

For more details, including information on quiescing functions, see Application-consistent backups in the Kasten K10 documentation.

Kanister uses open source Blueprints to define database-specific workflows. The Kanister Blueprint contains pre- and post-backup hooks that trigger the action before and after the volume snapshot is taken.

The process for creating the application-consistent backup is as follows:

Create the Application Blueprint that performs the pre- and post-backup tasks in kasten-io namespace.
Add the annotation to the container application to instruct K10 to use the above hooks when performing operations.

Use case 1: Create an application-consistent PostgreSQL backup

To create a Kanister Blueprint, create a postgresql-hooks.yaml file with the following contents:

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
name: postgresql-hooks
namespace: kasten-io
actions:
backupPrehook:
phases:
- func: KubeExec
name: makePGCheckPoint
args:
namespace: "{{ .DeploymentConfig.Namespace }}"
pod: "{{ index .DeploymentConfig.Pods 0 }}"
container: postgresql
command:
- bash
      - -o
      - errexit
      - -o
      - pipefail
      - -c
      - |
      psql -c "select pg_start_backup('app_cons');"
backupPosthook:
phases:
- func: KubeExec
name: afterPGBackup
args:
namespace: "{{ .DeploymentConfig.Namespace }}"
pod: "{{ index .DeploymentConfig.Pods 0 }}"
container: postgresql
command:
- bash
      - -o
      - errexit
      - -o
      - pipefail
      - -c
      - |
      psql -c "select pg_stop_backup();"

Apply the postgresql-hooks.yaml file:

$ kubectl --namespace=kasten-io create -f postgresql-hooks.yaml

Add the following annotation to the PostgreSQL DeploymentConfig to instruct K10 to use the above hooks when performing operations on this PostgreSQL instance.
```
$ kubectl annotate DeploymentConfig postgresql kanister.kasten.io/blueprint='postgresql-hooks' \
--namespace=postgresql
```

Use case 2: Create an application-consistent MySQL backup

To create the Kanister Blueprint, create a mysql-hooks.yaml file with the following contents:

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
name: mysql-hooks
actions:
backupPrehook:
phases:
- func: KubeExec
name: makeMSCheckPoint
args:
  namespace: "{{ .DeploymentConfig.Namespace }}"
  pod: "{{ index .DeploymentConfig.Pods 0 }}"
  container: mysql
  command:
  - bash
  - -o
  - errexit
  - -o
  - pipefail
  - -c
  - |
  mysql -uroot -p$MYSQL_ROOT_PASSWORD -h $HOSTNAME -e "FLUSH TABLES WITH READ LOCK;”
backupPosthook:
phases:
- func: KubeExec
name: afterMSBackup
args:
  namespace: "{{ .DeploymentConfig.Namespace }}"
  pod: "{{ index .DeploymentConfig.Pods 0 }}"
  container: mysql
  command:
  - bash
  - -o
  - errexit
  - -o
  - pipefail
  - -c
  - |
  mysql -uroot -p$MYSQL_ROOT_PASSWORD -h $HOSTNAME -e "UNLOCK TABLES;" -v

Apply the mysql-hooks.yaml file:

   $ kubectl --namespace=kasten-io create -f mysql-hooks.yaml

Add the following annotation to the MySQL DeploymentConfig to instruct K10 to use the above hooks when performing operations on this PostgreSQL instance:
```
$ kubectl annotate DeploymentConfig mysql kanister.kasten.io/blueprint='mysql-hooks' \
--namespace=mysql
```
To run an on-demand backup, click run once in the appropriate backup policy. Ideally, the backup job will run according to the schedule that you configured when you created the backup policy backup policy.
Now that you have protected your applications with a backup policy, you can restore them in place or clone them into a different namespace. To restore an existing application, go to the Kasten K10 dashboard and click the Applications tile.
Click “restore” in the panel of the application (namespace) that you want to restore. A list of all the previous available restore points is displayed.
Click the desired restore point. In the Application Name field, verify the application namespace that you are restoring to and then click Restore. (See the next step if you want to restore to a different namespace.)
Optional: If you want to restore to a different namespace, in the Application Name field, select the target namespace that you want to restore to. Click Restore.

Notes

The restore process can take a few minutes depending on the amount of data captured by the restore point. The restore time is also dependent on the speed of the underlying storage infrastructure.
The process of restoring a deleted application is nearly identical to restoring a running application. The only difference is that, by default, removed applications are not shown on the Applications page. To discover them, you simply need to filter and select Removed. You will see all applications that K10 has previously protected but which have been deleted. These applications can now be restored using the normal restore process.

Use case 3: Create an etcd backup on Red Hat OpenShift

Before completing a backup of the etcd cluster, you need to create a Secret in an existing or new temporary namespace containing details about the authentication mechanism used by etcd. In the case of OCP, it is likely that etcd pods have labels app=etcd,etcd=true and are running in the namespace openshift-etcd.

To create a temporary namespace and a Secret to access the etcd member, run the following command (replace the IP address with your single-node OpenShift (SNO) address for endpoints in the command.):

$ oc create ns etcd-backup
$ oc create secret generic etcd-details \
--from-literal=endpoints=https://10.0.133.5:2379 \
--from-literal=labels=app=etcd,etcd=true \
--from-literal=etcdns=openshift-etcd \
--namespace etcd-backup

To determine the value for the endpoints flag, run the following command:

oc get node -l node-role.kubernetes.io/master="" -ojsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'

To avoid any other workloads from the etcd-backup namespace being backed up, label the etcd-details Secret by running the following command:
```
$ oc label secret -n etcd-backup etcd-details include=true
```

To create the Blueprint resource that will be used by K10 to back up etcd, run the following command:

$ oc --namespace kasten-io apply -f \
https://raw.githubusercontent.com/kanisterio/kanister/0.84.0/examples/etcd/etcd-in-cluster/ocp/blueprint-v2/etcd-incluster-ocp-blueprint.yaml

You need to annotate the Secret to instruct K10 to use the Blueprint to perform backups on the etcd pod. Run the following command with the name of the Blueprint that you created:
```
$ oc annotate secret -n etcd-backup etcd-details kanister.kasten.io/blueprint='etcd-blueprint'
```

When the Secret is annotated, you can use K10 to back up etcd using the new namespace. If the Secret has a label (see above), you can include that Secret in the backup when you create the policy: alt text

Restore the etcd cluster

To restore the etcd cluster, you can follow the same instructions provided in the Red Hat OpenShift documentation, with some minor modifications. The Red Hat OpenShift documentation provides a cluster restore script (cluster-restore.sh). That script requires minor modifications because it expects the backup of static pod manifests, which we are not completed in this tutorial.

Before starting the restore process, make sure you have:

Created a namespace (for example etcd-restore) where the K10 restore can be completed:
```
$ oc create namespace etcd-restore
```

Created a Persistent Volume and Persistent Volume Claim in the etcd-restore namespace (for copying the backed-up etcd data to the leader node to perform restore operation):

$ oc --namespace etcd-restore apply -f \
https://raw.githubusercontent.com/kanisterio/kanister/0.84.0/examples/etcd/etcd-in-cluster/ocp/blueprint-v2/pv-etcd-backup.yaml
#
$ oc --namespace etcd-restore apply -f \
https://raw.githubusercontent.com/kanisterio/kanister/0.84.0/examples/etcd/etcd-in-cluster/ocp/blueprint-v2/pvc-etcd-backup.yaml

SSH connectivity to all the leader nodes.

Choose one node to be the restore node and download the etcd backup file to that node:

To restore the etcd cluster, complete the following steps:

Add an etcd-restore label to the node that you chose as the restore node:
```
$ oc label node <your-leader-node-name> etcd-restore=true
```
Perform the restore action on K10 by selecting etcd-restore as the target namespace. Note: The K10 restore action in this step only downloads the backup file from the external storage to the restore node.
Confirm that the backup file has been downloaded and is available in the directory /mnt/data on the restore node.

Stop the static pods from all other leader nodes by moving them out of the staticPodPath directory (for example, /etc/kubernetes/manifests):

# Move etcd pod manifest
$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp
#
# Make sure etcd pod has been stopped. The output of this
# command should be empty. If it is not empty, wait a few
# minutes and check again.
$ sudo crictl ps | grep etcd | grep -v operator
#
# Move api server pod manifest
$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp
#
# Verify that the Kubernetes API server pods are stopped. The output of
# this command should be empty. If it is not empty, wait a few minutes
# and check again.
$ sudo crictl ps | grep kube-apiserver | grep -v operator

Move the etcd data directory to a different location for all leader nodes that are not the restore node:
```
$ sudo mv /var/lib/etcd/ /tmp
```
Run the modified cluster-ocp-restore.sh script with the location of the etcd backup:
```
$ sudo ./cluster-ocp-restore.sh /mnt/data
```

Check the nodes to ensure they are in the Ready state:

$ oc get nodes -w
#
NAME                STATUS  ROLES          AGE     VERSION
host-172-25-75-28   Ready   master         3d20h   v1.23.3+e419edf
host-172-25-75-38   Ready   worker         3d20h   v1.23.3+e419edf
host-172-25-75-40   Ready   master         3d20h   v1.23.3+e419edf
host-172-25-75-65   Ready   master         3d20h   v1.23.3+e419edf
host-172-25-75-74   Ready   worker         3d20h   v1.23.3+e419edf
host-172-25-75-79   Ready   worker         3d20h   v1.23.3+e419edf
host-172-25-75-86   Ready   worker         3d20h   v1.23.3+e419edf
host-172-25-75-98   Ready   worker         3d20h   v1.23.3+e419edf

If any nodes are in the NotReady state, log in to the nodes and remove all of the PEM files from the /var/lib/kubelet/pki directory on each node.
Restart the Kubelet service on all of the leader nodes:
```
     $ sudo systemctl restart kubelet.service
```

Approve the pending CSRs (If there are no CSRs, you can skip this step):

# Check for any pending CSRs
$ oc get csr
#
# Review the details of a CSR to verify that it is valid
$ oc describe csr <csr_name>
#
# Approve all the pending CSRs
$ oc adm certificate approve <csr_name>

On the restore node, verify that the etcd container is running:
```
$ sudo crictl ps | grep etcd | grep -v operator
```

Verify that the single etcd node has been started by running the following command from a host that can access the cluster:

$ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd
#
NAME                              READY   STATUS      RESTARTS   AGE
etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1          2m47s

Force the etcd deployment:

$ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
#
# Verify all nodes are updated to latest version
$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
#
# To make sure all etcd nodes are on the latest version wait for a message like below
AllNodesAtLatestRevision
1 nodes are at revision 3

Force the rollout of the API Server control plane component:

# API Server
$ oc patch kubeapiserver cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

Wait for all the API server pods to get to the latest revision:

$ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

Force the rollout of the Controller Manager control plane component:

$ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

Wait for all Controller Manager pods to get to the latest revision:

$ oc get kubecontrollermanager -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

Force the rollout for the Scheduler control plane component:

 $ oc patch kubescheduler cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

Wait for all the Scheduler pods to get to the latest revision:

$ oc get kubescheduler -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

Verify that all the etcd pods are in the "Running" state:

 $ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd
 etcd-sno.ocp411.edgenius-ocp-ha.cloud                5/5     Running     0          9h

If all etcd pods are running, the etcd cluster has been restored successfully.

Summary

This tutorial walked you through the steps to install Veeam Kasten K10 in a single-node OpenShift cluster. You then configured your object storage as the backup target and configured a backup for container applications using a backup policy with multiple use cases:

Application-consistent backup using Kanister Blueprint for PostgreSQL
Application-consistent MySQL backup
Red Hat OpenShift etcd backup and restore

Now that you are comfortable with installing Kasten K10 and backing up and restoring your container applications, you can explore more IBM Security options. Check out the following pages:

Products

Languages

Technologies

Events

External Resources

Install Kasten K10 to back up and restore Kubernetes workloads