IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

Learn more about how I began prototyping OpenWhisk runtimes to build, deploy, and serve OpenWhisk actions on Knative.


Earlier last year, I began prototyping Apache OpenWhisk runtimes to build, deploy, and serve OpenWhisk actions on Knative. The initial prototype was designed using a Knative Build for an OpenWhisk Node.js runtime due to Node.js being the most popular OpenWhisk runtime. The idea was to create a Knative Build template for one runtime and extend the same template to include other OpenWhisk runtimes, such as Java, Python, Swift, and more.

Around the same time, the Tekton pipeline project, originating from a knative/build-pipeline, began evolving and enhancing Knative build functionality by providing advanced continuous integration (CI) and continuous delivery (CD) features on Kubernetes. Soon after, Knative Build was deprecated in favor of the Tekton pipeline. By following a migration guide, I converted Knative Build and BuildTemplate resources to Tekton TaskRuns and Tasks which were added to the Tekton catalog.

The Tekton pipeline for an OpenWhisk Node.js runtime is a collection of Tekton tasks performing the following steps:

  1. Clone a Node.js application source.
  2. Install dependent NPM packages specified in package.json.
  3. Build an archive .zip with the application source, including all the dependencies.
  4. Clone an OpenWhisk Node.js runtime.
  5. Embed a .zip file built in the previous task into the runtime before building and pushing the image to a registry.

The pipeline takes as an input git resource representing the application GitHub repository with application code and an OpenWhisk Node.js runtime. The output is an image resource which can be deployed and served on Knative.

A Node.js serverless pipeline before workspace

Successful but unpleasant

Below you can see this additional step in every task within a pipeline is extra overhead and makes the whole pipeline difficult. Notice that each task has an input and an output. The output of a task is linked to the input of a previous task. For example, task build-archive has an input git resource which is linked to the output of the previous task install-dependent-packages. This is because a .zip file should include all dependencies from NPM packages. When two tasks are connected using inputs and outputs, the resources have to be manually copied using the copy command:

# Task to install NPM packages
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: pull-dependencies
spec:
  inputs:
    resources:
    - name: source
      type: git
  outputs:
    resources:
    - name: source
      type: git
  steps:
  - name: install-dependencies
    image: nodejs
    script: |
      cd $(inputs.resources.source.path)/
      npm install --production --loglevel=error
      cp -r $(inputs.resources.source.path)/* $(outputs.resources.source.path)
  - name: copy-source-to-output
    image: ubuntu
    script: |
      cp -avr $(inputs.resources.source.path)/ $(outputs.resources.source.path)/
---
# Task to build archive from the source including its dependencies
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-archive
spec:
  inputs:
    resources:
    - name: source
      type: git
  outputs:
    resources:
    - name: source
      type: git
  steps:
  - name: install-dependencies
    image: nodejs
    script: |
      cd $(inputs.resources.source.path)/
      zip source.zip -r *
  - name: copy-zip-to-output
    image: ubuntu
    script: |
      cp -avr $(inputs.resources.source.path)/ $(outputs.resources.source.path)/
---
...

Ultimately, my goal was to extend the same pipeline to other OpenWhisk runtimes. So, simplifying such a complex prototype was a must.

While I was waiting for the next Tekton release, I started implementing a pipeline for an OpenWhisk Java runtime with the following list of tasks:

Task 1: create-jar-with-maven

An application with a Java runtime was first compiled and a .jar file was built using Maven if a Project Object Model (POM) file existed at the root of the application repo. With access to common workspace, .jar was created under $(workspaces.workspace.path).

Task 2: build-runtime-with-gradle

I had to select the JDK version, optional framework, and optional profile libraries before building a Java runtime in the common workspace.

Task 3: build-shared-class-cache

A shared class cache needed to be created and made available for the next task.

Task 4: finalize-runtime-with-function

I injected .jar, along with cached libraries before building and publishing an image to the registry.

Those tasks combined look like this:

A Java serverless pipeline before workspace

Workspace to the rescue

Similar to an OpenWhisk Node.js pipeline, the tasks in a Java pipeline are connected using inputs and outputs. Just when I finished implementing these pipelines, Tekton’s 0.10.0 release Bombay Robbie came to the rescue with the added workspace (shared persistent volume) support to tasks and a pipeline, which simplified everything. After introducing workspace to every task in a pipeline, I declared directories which were created using volumes at runtime. Now, instead of looping tasks through inputs and outputs, they all are connected through workspace. The below images depict both a Node.js pipeline and a Java pipeline featuring the new workspace support enhancements.

A Node.js serverless pipeline after workspace

A Java serverless pipeline after workspace

And this is how tasks are connected using workspace:

# Task to clone source repo
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: clone-source-to-workspace
spec:
  workspaces:
    - name: scratchpad
  inputs:
    resources:
      - name: source
        type: git
        targetPath: source
  steps:
    - name: clone-source-to-scratchpad
      image: ubuntu
      script: |
        cp -avr $(inputs.resources.source.path)/ $(workspaces.scratchpad.path)/
---
# Task to install NPM packages
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: pull-dependencies
spec:
  workspaces:
    - name: scratchpad
  steps:
  - name: install-dependencies
    image: nodejs
    script: |
      cd $(workspaces.scratchpad.path)/
      npm install --production --loglevel=error
---
# Task to build archive from the source including its dependencies
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-archive
spec:
  workspaces:
    - name: scratchpad
  steps:
  - name: install-dependencies
    image: nodejs
    script: |
      cd $(workspaces.scratchpad.path)/
      zip source.zip -r *
---

Hooray! There are now two OpenWhisk runtimes implemented using a single pipeline which can be easily extended for more OpenWhisk runtimes. Now, an OpenWhisk application source decides which OpenWhisk runtime needs to be executed. I implemented Tekton conditions for such detection, which discovered the type of application based on the file extension. So, that means a Java runtime for files with .java and a Node.js runtime for files with a .js extension. After identifying the runtime, the OpenWhisk Tekton pipeline executed tasks specific to that runtime. Here is a high-level overview of the OpenWhisk Tekton pipeline for multiple runtimes:

An OpenWhisk Tekton pipeline

Summary

I hope you now know how workspace helps create one single pipeline for multiple OpenWhisk runtimes. If you’d like to build on what you learned throughout this blog, refer to the Building an OpenWhisk Application with Tekton for Knative GitHub repository for the entire pipeline and example usage. If you’re interested in a hands-on approach, try this tutorial Run OpenWhisk actions on managed Knative on a Kubernetes cluster.