Introducing Elyra pipelines with custom component support
Tags: jupyter, kubeflow, apacheairflow, machinelearning
Published: 2021-08-10 13:59:36

The [Elyra open source project](https://github.com/elyra-ai/elyra) for [JupyterLab](https://jupyter.org/) aims to simplify common data science tasks. Its most popular feature is the Visual Pipeline Editor, which is used to create pipelines without the need for coding. You can run these pipelines in JupyterLab or on [Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/overview/pipelines-overview/) or [Apache Airflow](https://airflow.apache.org/).

![pipelines-and-runtimes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g3pkl1uensn84x3rzdur.png)
 

Elyra 3.0 extends the pipeline capabilities by adding _experimental_ support for custom components. Before I dive into specifics and outline why support is still experimental in the initial releases, let's recap a few concepts.

A [_pipeline_](https://elyra.readthedocs.io/en/stable/user_guide/pipelines.html) comprises _nodes_ that are connected with each other to form a graph. The graph defines _dependencies_ between the nodes, governing the order in which the nodes are executed. The example pipeline shown below executes a Python script and several Jupyter notebooks.

![pipeline-nodes-dependencies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/phpahcsrcs3dcbscuoya.png) 

Nodes are implemented using [components](https://elyra.readthedocs.io/en/stable/user_guide/pipeline-components.html). To create the pipeline shown above, you'll need components that can execute Python scripts and Jupyter notebooks. Most components are configurable to make them re-usable. For file-based components, such a configuration might include the file name and the container image where the file is executed in.

![nodes-components](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/awvvk2yh291v0v4t6poe.png) 

In Elyra, the processing of Jupyter notebooks, Python scripts, and R scripts is implemented using a single component. This component is referred to as a _generic component_ because it is supported in all runtime environments.

The pipeline editor then exposes this component under different names in the palette, which is located on the left-hand side. (You can add nodes to the pipeline by selecting a component from the palette and dropping it on the canvas.)

![generic-component-pipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4ijl7j5qto0d0yybajcp.png)
 

Pipelines that only include generic components are referred to as _generic pipelines_ because you can run them in any runtime environment Elyra supports. 

Take a look at the [tutorials](https://elyra.readthedocs.io/en/stable/getting_started/tutorials.html#tutorials) if you are new to Elyra and would like to learn more about how to use the Visual Pipeline Editor to create a pipeline. If you've used Elyra before, we recommend reviewing the  recently published [_Best practices_ topic in the _User Guide_](https://elyra.readthedocs.io/en/stable/user_guide/best-practices-file-based-nodes.html). We've only now gotten around to documenting some of the things that make your life easier!

### Experimental support for custom components

Custom components are similar to generic components in that they only implement a single task, such as load data, train a model, or send an email. However, these components are only supported for [Kubeflow Pipelines
](https://www.kubeflow.org/docs/components/pipelines/sdk/component-development/) and [Apache Airflow]
(https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/index.html) and are implemented in a runtime specific form.

The screen capture below depicts the pipeline editor for Apache Airflow pipelines. The palette, shown on the left, is by default divided into two categories — one for generic components and one for custom components. Note the Airflow specific components in the second category, such as the _BashOperator_ and the _SimpleHttpOperator_, which process a bash command and an HTTP request, respectively.

![airflow-example-pipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3i0hhbjk0t1nl4ajxpl4.png) 

Pipelines that utilize custom components are called _runtime specific_ pipelines because it is not possible to run a pipeline that was created for Kubeflow Pipelines on Apache Airflow and vice versa.

### Get started with pipelines

Once you've [installed Elyra](https://elyra.readthedocs.io/en/stable/getting_started/installation.html), it is easy to get started with pipelines. The JupyterLab launcher now includes tiles for each pipeline type: one for generic pipelines, and one for each supported pipeline runtime platform.

![pipeline-editor-launchers](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/41yljwt149i4bd2frevt.png) 

Click the desired pipeline editor tile and you are ready to compose a pipeline from the components that are supported for the selected platform.

To get you going quickly, the component registry includes a few example custom components for each runtime platform. The [Elyra examples GitHub repository](https://github.com/elyra-ai/examples) includes information about those components and pipelines that illustrate their usage. These components are included for illustrative purposes only. Unless stated otherwise, the components were not created by the Elyra community and are therefore provided as is.

### Opportunities for growth

In the initial 3.0 release, Elyra’s support for custom components is rather limited. Many features are still under development, planned for a future release, or in the backlog without a specific target release. Some of the high priority features for the next releases are:

- **Data exchange between custom components:** Components commonly produce outputs that other components require as input. Currently, custom components are isolated from each other and cannot exchange data. (Data exchange between generic components is already supported.)
- **Data exchange between generic components and custom components**: Same as above.
- **Manage component registry**: Provide a UI and/or CLI that allows for the addition, editing, or deletion of components. Currently, components [can only be managed manually](https://github.com/elyra-ai/elyra/discussions/1881).

For an up-to-date feature status please refer to [this forum thread](https://github.com/elyra-ai/elyra/discussions/1890).

Use Watson Studio services in pipelines
Pipelines can also take advantage of external services using custom components. If you are looking for a managed solution for Watson Studio services, check out [this IBM Watson Studio Pipeline article](https://medium.com/ibm-data-ai/automating-the-ai-lifecycle-with-ibm-watson-studio-orchestration-flow-4450f1d725d6). It illustrates how to run notebooks, refine data, [run AutoAI experiments](https://medium.com/ibm-data-ai/watson-autoai-series-part-1-24be02171e51), and deploy a model.

![Watson Studio Pipelines](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/becraacjq3qh8js98wme.png)

### Your opportunity to help us improve Elyra

Elyra is a fairly new open source project that is currently maintained by a small community of JupyterLab enthusiasts. We welcome contributions of any kind, such as feedback, bug reports, bug fixes, features, or documentation. To learn more about how you can make a difference refer to the [_Getting help_ topic in the documentation](https://elyra.readthedocs.io/en/stable/getting_started/getting-help.html).

On behalf of the community: Thank You!