Overview

Aspera is very excited to release our first deliverable to support IBM Cloud Private (ICP).

ICP is built on open standards using Docker and Kubernetes. The Aspera CLI can be easily provisioned within ICP environments to provide fast and secure data transport into or out of private cloud environments.

One of Docker’s best practices is that containers should be as ephemeral and as stateless as possible. To help facilitate this best practice, Kubernetes provides the capability for Persistent Volumes as a mechanism that Kubernetes pods of docker containers can use to store or share state. The Aspera CLI can be initiated via a Helm job (manually or as a scheduled CronJob). This provisions an ephemeral container of the Aspera CLI. You attach it to a shared persistent volume, and instruct the CLI to send or receive data from an Aspera server.

Aspera is a true game changer for transporting your data. The Aspera FASP transport protocol gracefully handles loss and latency on the network to more efficiently utilize the existing wide-area network (WAN). Transporting data using Aspera can be up to hundreds of times faster than with legacy technology like FTP, SFTP, SCP, etc. Aspera transfers provide enterprise-grade security with data-at-rest and data-in-motion encryption. Finally, using Aspera, data can be moved from and to any cloud environment – public, private, or hybrid – regardless of distances and network conditions.

Use Cases

A variety of data transport use cases are enabled with the Aspera CLI, including:

  • A database stores a backup snapshot into a Persistent Volume. The Aspera CLI can send the database backup to another location quickly and securely.
  • An application is storing business data into a Persistent Volume. The Aspera CLI can send that data into another cloud environment for analysis or enrichment.
  • An application in ICP needs to pull in data for a batch processing job. The Aspera CLI can retrieve the required data and store it in the Persistent Volume for processing.
  • An application in ICP is producing large data that needs to be emailed out. The Aspera CLI can email out this dataset (regardless of size) to one or more recipients.

Obtaining the Aspera CLI for ICP

The Aspera CLI for ICP can be found on the ICP v3.1 Catalog. The Aspera CLI itself is free, but it must be used in conjunction with a licensed or metered Aspera transfer server. To get started, click here to provision a free trial of Aspera on Cloud to use in conjunction with the Aspera CLI.

You can also retrieve the Aspera CLI Docker image and Helm Chart via the following:

Preliminary Setup

1. In Aspera on Cloud (AoC), your AoC organization admin must create and provide you with a client ID and client secret:

       

2. Your AoC organization admin must enable JSON Web Token Auth (JWT):

       

3. The AoC user must upload their public SSH key to their Aspera on Cloud account profile:

       

4. On your system, install kubectl and Helm to launch the CLI on the command line.

5. In IBM Cloud Private, configure your command line as an ICP client.

       

6. Then copy and paste the configuration commands into your shell environment:

       

7. Finally, install Tiller in your ICP namespace by running helm init:

ameloy@DESKTOP-0L5BES2:~/charts$ helm init

8. In the ICP kubectl command line, create a single Secret that contains the AoC client ID, AoC client secret, and the user’s private SSH key (the same key as in Step 3 above):

kubectl create secret generic aoc-creds --from-file=ACLI_AOC_PRIVATE_KEY="/mnt /c/temp/private_key.pem" --from-literal=ACLI_AOC_CLIENT_SECRET="vFsdkjhsdkjhsdkjhdT-LrWsdkjhsdkjhdskjhsdkjhdskjhdskjhdskjhdskjhdskjhdskjhdskjhdskjc2TdZKRgJl" --from-literal=ACLI_AOC_CLIENT_ID="djhsjhdh_X4Q"

9. Add the IBM Charts as a Helm repository:

helm repo add ibm-charts https://raw.githubusercontent.com/IBM/charts/master/repo/stable/
helm repo update

10. In the ICP UI, go to Platform > Storage > New and create a Persistent Volume. Below is an example JSON:

{
    "apiVersion": "v1",
    "kind": "PersistentVolume",
    "metadata": {
        "name": "xfer-volume",
        "resourceVersion": "2925434",
        "annotations": {
            "pv.kubernetes.io/bound-by-controller": "yes"
        },
        "finalizers": [
            "kubernetes.io/pv-protection"
        ]
    },
    "spec": {
        "capacity": {
            "storage": "1Gi"
        },
        "nfs": {
            "server": "10.24.33.10",
            "path": "/export/storage/arthur"
        },
        "accessModes": [
            "ReadWriteMany"
        ],
        "claimRef": {
            "kind": "PersistentVolumeClaim",
            "namespace": "arthur",
            "name": "xfer-volume-claim",
            "uid": "5c25ec2a-b208-11e8-9827-005056ad1d7e",
            "apiVersion": "v1",
            "resourceVersion": "2925431"
        },
        "persistentVolumeReclaimPolicy": "Retain"
    }
}

11. Create a Persistent Volume Claim. Below is an example JSON:

{
    "apiVersion": "v1",
    "kind": "PersistentVolumeClaim",
    "metadata": {
        "name": "xfer-volume-claim",
        "namespace": "arthur",
        "resourceVersion": "2925436",
        "annotations": {
            "pv.kubernetes.io/bind-completed": "yes",
            "pv.kubernetes.io/bound-by-controller": "yes"
        },
        "finalizers": [
            "kubernetes.io/pvc-protection"
        ]
    },
    "spec": {
        "accessModes": [
            "ReadWriteMany"
        ],
        "resources": {
            "requests": {
                "storage": "1Gi"
            }
        },
        "volumeName": "xfer-volume"
    }
}

Examples

This section offers some sample YAML files to kick off Helm jobs of the Aspera CLI. Note that all of these examples can also be done in the ICP UI.

Example A: Upload a File (From the ICP Persistent Volume to an AoC Folder)

NOTE: In this example, there is a 100MB file in the user’s ICP persistent mount.

ameloy@DESKTOP-60DJUQK:~/icp$ helm install -f /home/ameloy/icp/upload.yaml ibm-charts/ibm-aspera-cli --tiller-namespace your-namespace
ameloy@DESKTOP-60DJUQK:~/icp$ cat upload.yaml

image:
    repository: ibmcom/aspera-cli
    pullPolicy: IfNotPresent
    secretName:
    tag: latest

cronjob:
    enabled: false
    schedule: "*/5 * * * *"
    successfulJobsHistoryLimit: 1
    failedJobsHistoryLimit: 1
    concurrencyPolicy: Forbid

job:
    backoffLimit: 2
    restartPolicy: Never

cli:
    subcommand: aoc
    direction: upload
    remoteHost: aoc-demo.ibmaspera.com
    remotePort: ''
    username: aspera-sales@ibm.com
    passwordSecretName: aoc-creds
    localPath: /100MB
    remotePath: Files/IBM-CLOUD-ICOS-DALLAS (Shared)
    debugLevel: 2
    additionalOptions: ''
    aoc:
        org: aoc-demo
        workspace: MyWorkspace
        secret: aoc-creds
        package:
            id: ''
            name: ''
            recipients: ''

resources:
    requests:
        memory: '1024Mi'
        cpu: '1'
    limits:
        memory: '2048Mi'
        cpu: '2'

volume:
    name: xfer-volume
    claimName: xfer-volume-claim
    existingClaimName: xfer-volume-claim
    mountPath: /mount

Example B: Send a Package (From Data in an ICP Persistent Volume to an AoC Recipient)

NOTE: In this example, there is a 100MB file in the user’s ICP persistent mount.

ameloy@DESKTOP-60DJUQK:~/icp$ helm install -f /home/ameloy/icp/send-package.yaml ibm-charts/ibm-aspera-cli --tiller-namespace your-namespace
ameloy@DESKTOP-60DJUQK:~/icp$ cat send-package.yaml

image:
    repository: ibmcom/aspera-cli
    pullPolicy: Always
    secretName:
    tag: latest

cronjob:
    enabled: false
    schedule: "*/5 * * * *"
    successfulJobsHistoryLimit: 1
    failedJobsHistoryLimit: 1
    concurrencyPolicy: Forbid

job:
    backoffLimit: 2
    restartPolicy: Never

cli:
    subcommand: aoc
    direction: send
    remoteHost: aoc-demo.ibmaspera.com
    remotePort: ''
    username: aspera-sales@ibm.com
    passwordSecretName: aoc-creds
    localPath: /100MB
    remotePath: ''
    debugLevel: 2
    additionalOptions: -l 500m
    aoc:
        org: aoc-demo
        workspace: MyWorkspace
        secret: aoc-creds
        package:
            id: test
            name: test
            recipients: aspera-sales@ibm.com

resources:
    requests:
        memory: '2048Mi'
        cpu: '2'
    limits:
        memory: '2048Mi'
        cpu: '2'

volume:
    name: xfer-volume
    claimName: xfer-volume-claim
    existingClaimName: xfer-volume-claim
    mountPath: /mount

Example C: Download a File (From an AoC Folder to an ICP Persistent Volume)

NOTE: In this example, there is a 200MB file in the user’s AoC shared folder.

ameloy@DESKTOP-60DJUQK:~/icp$ helm install -f /home/ameloy/icp/download.yaml ibm-charts/ibm-aspera-cli --tiller-namespace your-namespace
ameloy@DESKTOP-60DJUQK:~/icp$ cat download.yaml

image:
    repository: ibmcom/aspera-cli
    pullPolicy: IfNotPresent
    secretName:
    tag: latest

cronjob:
    enabled: false
    schedule: "*/5 * * * *"
    successfulJobsHistoryLimit: 1
    failedJobsHistoryLimit: 1
    concurrencyPolicy: Forbid

job:
    backoffLimit: 2
    restartPolicy: Never

cli:
    subcommand: aoc
    direction: download
    remoteHost: aoc-demo.ibmaspera.com
    remotePort: ''
    username: aspera-sales@ibm.com
    passwordSecretName: aoc-creds
    localPath: /
    remotePath: Files/AWS-S3-OREGON (Shared)/200MB
    debugLevel: 2
    additionalOptions: ''
    aoc:
        org: aoc-demo
        workspace: MyWorkspace
        secret: aoc-creds
        package:
            id: ''
            name: ''
            recipients: ''

resources:
    requests:
        memory: '1024Mi'
        cpu: '1'
    limits:
        memory: '2048Mi'
        cpu: '2'

volume:
    name: xfer-volume
    claimName: xfer-volume-claim
    existingClaimName: xfer-volume-claim
    mountPath: /mount

Example D: Download a Package (From an AoC Inbox to an ICP Persistent Volume)

ameloy@DESKTOP-60DJUQK:~/icp$ helm install -f /home/ameloy/icp/get-package.yaml ibm-charts/ibm-aspera-cli --tiller-namespace your-namespace
ameloy@DESKTOP-60DJUQK:~/icp$ cat get-package.yaml

image:
    repository: ibmcom/aspera-cli
    pullPolicy: Always
    secretName:
    tag: latest

cronjob:
    enabled: false
    schedule: "*/5 * * * *"
    successfulJobsHistoryLimit: 1
    failedJobsHistoryLimit: 1
    concurrencyPolicy: Forbid

job:
    backoffLimit: 2
    restartPolicy: Never

cli:
    subcommand: aoc
    direction: get
    remoteHost: aoc-demo.ibmaspera.com
    remotePort: ''
    username: aspera-sales@ibm.com
    passwordSecretName: aoc-creds
    localPath: /
    remotePath: ''
    debugLevel: 2
    additionalOptions: -l 500m
    aoc:
        org: aoc-demo
        workspace: MyWorkspace
        secret: aoc-creds
        package:
            id: BwjqQcNQlA
            name: ''
            recipients: ''

resources:
    requests:
        memory: '2048Mi'
        cpu: '2'
    limits:
        memory: '2048Mi'
        cpu: '2'

volume:
    name: xfer-volume
    claimName: xfer-volume-claim
    existingClaimName: xfer-volume-claim
    mountPath: /mount