Earlier this month JupyterLab 3.0 was released, providing a slew of new features. In this blog post, I recap a few highlights and how the Elyra 2.0 extensions take advantage of them.
Note: New to Elyra? Take a look at the overview documentation. TL;DR: Elyra provides productivity features for common data science and AI tasks.
Support for Language Servers
Many modern integrated development environments (IDEs) simplify development by providing editor features like autocomplete, “go to definition”, “find references”, or linting.
JupyterLab optionally supports these features by integrating with Language Servers using the
jupyterlab-lsp extension. Language Servers are available for many languages, including Python and R. Note that some Language Servers might not support every convenience feature.
Elyra 2.0 includes the Python Language Server, enabling these features in the notebook editor and Python script editor, as shown below.
You can add support for other languages by installing the appropriate Language Server and its prerequisites.
JupyterLab includes a visual debugger front-end that provides common debugging tasks. For example, you can set breakpoints, step through code, or inspect data. The front-end requires a kernel with support for debugging.
As of January 2021, the only Jupyter Python kernel that supports debugging is
Note: Note that this kernel currently does not support all iPython features, such as cell magics.
Neither JupyterLab 3.0 nor Elyra 2.0 install
xeus-python by default. You therefore have to install it separately from PyPI, conda, or build from source code.
Debugging a notebook
To debug a notebook, associate it with a kernel that supports debugging and enable the debugger. The official debugger example Python notebook provides a short introduction and is highly recommended!
Debugging a Python script
Elyra’s Python editor is also enabled for debugging. To debug a Python script:
- Open the script in the Python editor.
- Right click on the canvas and select
Create Console for Editor.
- Select a kernel that supports debugging.
- Enable the debugger in the console.
- Set one or more breakpoints in the source code.
- Run code to breakpoint and step through the code.
Elyra 2.0 also includes several enhancements that are not specific to JupyterLab 3.0.
Pipeline editor enhancements for Kubeflow Pipelines
Elyra’s visual pipeline editor, which I introduced in this article, was extended to support custom resource allocation and the Tekton workflow engine.
Customize resource allocation
When a pipeline node is executed on Kubeflow Pipelines, it is processed in a container. The amount of CPU, GPU, and memory (RAM) resources that are allocated for the container is governed by configuration settings in Kubernetes.
With the latest enhancement in Elyra, you can request additional CPU, GPU, or memory resources to speed up processing of your notebook or Python script during pipeline execution. You configure those resource requirements in the node’s properties view:
- Select the notebook or Python script node in the pipeline editor.
- Right click, and select
- Enter the desired values.
If you leave the properties empty, the Kubernetes default settings are applied.
Support for Tekton workflow engine
When you run (or export) a pipeline in the Elyra pipeline editor, it is compiled into a workflow engine-specific format. In previous releases, Elyra could only produce Argo-compatible outputs, but this restriction is lifted in version 2.0. Since Elyra cannot automatically determine which workflow engine your Kubeflow Pipelines installation leverages, you need to specify the type when you create a runtime configuration for your Kubeflow Pipelines deployment.
If you don’t specify an engine type in your configuration, Argo is assumed as default to retain backward compatibility with older Elyra releases.
Note If you are unsure what engine your Kubeflow Pipelines deployment utilizes, check with the administrator. If you select the wrong engine type, your pipelines will fail to execute.
Elyra 2.0 makes it easier to create code snippets. The code snippet editor now allows for the creation of code snippets from highlighted code.
To create a code snippet within an editor, select the desired piece of code, right click, and choose from the context menu
Save as Code Snippet.
To learn more about other enhancements or bug fixes that went into Elyra 2.0 please check the changelog.
Open Data Hub, Watson Studio, Cloud Pak for Data, and Watson AI Platform Pipelines
Open Data Hub
If you are considering a JupyterHub/JupyterLab/Elyra deployment on the OpenShift Container Platform, Open Data Hub might fit the bill. Based on Kubeflow, the project includes open source AI tools for running large and distributed AI workloads.
IBM Cloud Pak for Data and Watson Studio
IBM Cloud Pak for Data and its as-a-service companion Watson Studio incorporate many Elyra features.
Watson AI Platform Pipelines
With Elyra’s Visual Pipeline editor, you can easily assemble basic machine learning pipelines from Jupyter Notebooks and Python scripts. The recently announced Watson AI Platform Pipelines takes this to the next level by providing enterprise-grade features, such as a pipeline components registry, production-grade logging, and governance capabilities. It’ll integrate with Watson Studio, Watson AutoAI, Watson Knowledge Catalog, SPSS, and others.
If you are new to Elyra, it’s easy to get started.
Install from PyPI or conda
You can install the Elyra JupyterLab extensions from PyPI, conda, or from source code. Refer to the installation guide for details.
Quick side note. JupyterLab 3.0 introduces a new extension type named prebuilt extensions. These extensions don’t need a JupyterLab rebuild or a Node.js installation. Unfortunately, we are unable to distribute Elyra 2.0 as prebuilt extensions, but are considering doing it for future releases.
Run JupyterLab/Elyra in a container image
The Elyra community publishes Elyra container images on DockerHub and Quay.io. Take a look at these instructions to learn more about the images and how to run them.
During the past year we have received many suggestions and feedback from users like you. We appreciate your input and encourage you to stay involved. Open GitHub issues, reach out on gitter, post in our new forum, or join our weekly community meeting.
Use the forum to ask general questions or share things you’ve created, like Nick Burdakous did. He recently published a pipeline viewer for Google Chrome and Mozilla Firefox. This viewer visualizes pipeline files on GitHub, like shown in the screen shot below. The depicted pipeline was created by Nick Pentreath and presented at the Big Things conference in November 2020.
The Elyra community has lot planned for the first half of this year. Follow us on Twitter to stay in the loop.