Win $20,000. Help build the future of education. Answer the call. Learn more

What’s new in Elyra 2.0

Earlier this month JupyterLab 3.0 was released, providing a slew of new features. In this blog post, I recap a few highlights and how the Elyra 2.0 extensions take advantage of them.

Note: New to Elyra? Take a look at the overview documentation. TL;DR: Elyra provides productivity features for common data science and AI tasks.

Support for Language Servers

Many modern integrated development environments (IDEs) simplify development by providing editor features like autocomplete, “go to definition”, “find references”, or linting.

JupyterLab optionally supports these features by integrating with Language Servers using the jupyterlab-lsp extension. Language Servers are available for many languages, including Python and R. Note that some Language Servers might not support every convenience feature.

Elyra 2.0 includes the Python Language Server, enabling these features in the notebook editor and Python script editor, as shown below.

Lint code, use auto-complete, open signature preview, find references and more using the Python Language Server integration

You can add support for other languages by installing the appropriate Language Server and its prerequisites.

Visual debugger

JupyterLab includes a visual debugger front-end that provides common debugging tasks. For example, you can set breakpoints, step through code, or inspect data. The front-end requires a kernel with support for debugging.

As of January 2021, the only Jupyter Python kernel that supports debugging is xeus-python.

Note: Note that this kernel currently does not support all iPython features, such as cell magics.

Neither JupyterLab 3.0 nor Elyra 2.0 install xeus-python by default. You therefore have to install it separately from PyPI, conda, or build from source code.

Debugging a notebook

To debug a notebook, associate it with a kernel that supports debugging and enable the debugger. The official debugger example Python notebook provides a short introduction and is highly recommended!

Debug notebooks using debugger supporting kernels

Debugging a Python script

Elyra’s Python editor is also enabled for debugging. To debug a Python script:

  1. Open the script in the Python editor.
  2. Right click on the canvas and select Create Console for Editor.
  3. Select a kernel that supports debugging.
  4. Enable the debugger in the console.
  5. Set one or more breakpoints in the source code.
  6. Run code to breakpoint and step through the code.

Debug a Python script using the debugger

Elyra 2.0 also includes several enhancements that are not specific to JupyterLab 3.0.

Pipeline editor enhancements for Kubeflow Pipelines

Elyra’s visual pipeline editor, which I introduced in this article, was extended to support custom resource allocation and the Tekton workflow engine.

Customize resource allocation

When a pipeline node is executed on Kubeflow Pipelines, it is processed in a container. The amount of CPU, GPU, and memory (RAM) resources that are allocated for the container is governed by configuration settings in Kubernetes.

With the latest enhancement in Elyra, you can request additional CPU, GPU, or memory resources to speed up processing of your notebook or Python script during pipeline execution. You configure those resource requirements in the node’s properties view:

  1. Select the notebook or Python script node in the pipeline editor.
  2. Right click, and select Properties.
  3. Enter the desired values.

Defining custom resource requirements for the load_data Python script

If you leave the properties empty, the Kubernetes default settings are applied.

Support for Tekton workflow engine

Kubeflow Pipelines utilizes a container-native workflow engine to execute pipelines. As of January 2021, Argo and Tekton are the only two supported engines.

When you run (or export) a pipeline in the Elyra pipeline editor, it is compiled into a workflow engine-specific format. In previous releases, Elyra could only produce Argo-compatible outputs, but this restriction is lifted in version 2.0. Since Elyra cannot automatically determine which workflow engine your Kubeflow Pipelines installation leverages, you need to specify the type when you create a runtime configuration for your Kubeflow Pipelines deployment.

This runtime configuration in Elyra is associated with a Kubeflow Pipelines deployment that leverages Tekton as workflow engine

If you don’t specify an engine type in your configuration, Argo is assumed as default to retain backward compatibility with older Elyra releases.

Note If you are unsure what engine your Kubeflow Pipelines deployment utilizes, check with the administrator. If you select the wrong engine type, your pipelines will fail to execute.

Code snippets

Elyra 2.0 makes it easier to create code snippets. The code snippet editor now allows for the creation of code snippets from highlighted code.

To create a code snippet within an editor, select the desired piece of code, right click, and choose from the context menu Save as Code Snippet.

Create a code snippet from code, paste it as a new cell and run it

To learn more about other enhancements or bug fixes that went into Elyra 2.0 please check the changelog.

Open Data Hub, Watson Studio, Cloud Pak for Data, and Watson AI Platform Pipelines

Open Data Hub

If you are considering a JupyterHub/JupyterLab/Elyra deployment on the OpenShift Container Platform, Open Data Hub might fit the bill. Based on Kubeflow, the project includes open source AI tools for running large and distributed AI workloads.

IBM Cloud Pak for Data and Watson Studio

IBM Cloud Pak for Data and its as-a-service companion Watson Studio incorporate many Elyra features.

Watson AI Platform Pipelines

With Elyra’s Visual Pipeline editor, you can easily assemble basic machine learning pipelines from Jupyter Notebooks and Python scripts. The recently announced Watson AI Platform Pipelines takes this to the next level by providing enterprise-grade features, such as a pipeline components registry, production-grade logging, and governance capabilities. It’ll integrate with Watson Studio, Watson AutoAI, Watson Knowledge Catalog, SPSS, and others.

Get started

If you are new to Elyra, it’s easy to get started.

Install from PyPI or conda

You can install the Elyra JupyterLab extensions from PyPI, conda, or from source code. Refer to the installation guide for details.

Quick side note. JupyterLab 3.0 introduces a new extension type named prebuilt extensions. These extensions don’t need a JupyterLab rebuild or a Node.js installation. Unfortunately, we are unable to distribute Elyra 2.0 as prebuilt extensions, but are considering doing it for future releases.

Run JupyterLab/Elyra in a container image

The Elyra community publishes Elyra container images on DockerHub and Quay.io. Take a look at these instructions to learn more about the images and how to run them.

Get involved

During the past year we have received many suggestions and feedback from users like you. We appreciate your input and encourage you to stay involved. Open GitHub issues, reach out on gitter, post in our new forum, or join our weekly community meeting.

Use the forum to ask general questions or share things you’ve created, like Nick Burdakous did. He recently published a pipeline viewer for Google Chrome and Mozilla Firefox. This viewer visualizes pipeline files on GitHub, like shown in the screen shot below. The depicted pipeline was created by Nick Pentreath and presented at the Big Things conference in November 2020.

Elyra pipeline preview on GitHub

The Elyra community has lot planned for the first half of this year. Follow us on Twitter to stay in the loop.