Repository: tensorflow/model-analysis
Issue: JupyterLab support?
Root cause: Unimplemented Feature Gap


### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: N/A
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux CentOS 7
-   **TensorFlow Model Analysis installed from (source or binary)**: pypi binary
-   **TensorFlow Model Analysis version (use command below)**: tensorflow-model-analysis 0.14.0
-   **Python version**: 3.6
-   **Jupyter Notebook version**: 7.0.0
-   **Exact command to reproduce**: N/A

### Describe the problem

At Twitter, we primarily use the JupyterLab front-end for our notebook-based workflows. TFMA currently only supports running as an `nbextension` for the "Classic" Notebook UI - vs providing a `labextension` for e.g. JupyterLab.

Thus, consuming TFMA currently requires that our ML practitioners revert to the "Classic" Notebook UI which has largely been deprecated internally. It'd be great if TFMA could provide a JupyterLab plugin so that our users didn't have to switch UIs and interrupt their typical workflow.

### Source code / logs

N/A

+1 for this feature!

I need this feature too!

Seems like using `InteractiveContext.show` from TFX pipelines is also breaking because of this. Any chance to take a look at this?

Pinging back on this. Any chance we can prioritize this?


+1, this is a big gap in the UX for utilizing TFMA in any modern Jupyter Notebook environment (including AI Platform Notebooks, etc).

An additional +1 for this now since KubeFlow has release Kale to easily define and deploy pipelines directly from a JupyterLab notebook.

atn832@ is working on this, but no ETA. A partial fix has been submitted, but is not working in all cases. Feel free to try out the latest as it may work in some setups.

@mdreves I would be happy to try it. Is there an extension that needs to be installed as with the nbextension? Where can I find it?

Seems this is the PR being mentioned: https://github.com/tensorflow/model-analysis/pull/84

After pip installing jupyterlab and tensorflow_model_analysis do the following:

jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install tensorflow_model_analysis

Running into an issue here:

```
vulcanized_tfma.js:1 Uncaught SyntaxError: Identifier 'wj' has already been declared
    at vulcanized_tfma.js:1
```
any ideas?

This error is due to Facets (another project) loading by default before TFMA. Facets somehow automatically loads if you run Beam in Jupyter Lab ([code](https://github.com/apache/beam/blob/cb6d22ea5a6587e479505af3e368fba7df9200c1/sdks/python/apache_beam/runners/interactive/interactive_environment.py#L206)).

If you are able to split your notebook into two, one for processing data, and the other for rendering the results, you should be able to avoid the error.

I submitted a workaround for this issue at https://github.com/tensorflow/model-analysis/commit/19b4f4af0a0072b0194031d49718dc868c316c5d. It'll be released on the next version of TFMA. But it's only a partial fix. With this fix, TFMA will load show the components, but the CSS is broken.

I've also reached out to the Facets team, but they might not be able to easily remove the auto-load behavior.

Or you can try forcing TFMA to load first by adding this snippet at the very top of your notebook. Let me know if it works!

```py
from IPython.core.display import Javascript
from IPython.core.display import display_javascript
display_javascript(Javascript("""
    var link = document.createElement('link');
    link.rel = 'import'
    link.href = '/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js';
    document.head.appendChild(link);
"""))

```

@atn832 I'm having problems with the plugin. It installs as it should but in jupyterlab I have run into issues.
First, jupyterlab is unable to launch a terminal while I have this plugin installed. If I disable, uninstall and rebuild jupyterlab, I can launch terminals again. There are no errors thrown in the log.
Second, invoking tfma.load_eval_result load the eval result but then invoking  tfma.view.render_slicing_metrics yields "Loading widget..." where it is stuck. Note that the cell is marked as complete and no other output is shown.

Have you encountered anything similar? Is there a specific node version that one should use, etc.?
 

In the company I work for I need this feature too. We want to leverage TF/TFX stack along with JupyterLab for our AI Platform. 

> Or you can try forcing TFMA to load first by adding this snippet at the very top of your notebook. Let me know if it works!
> 
> ```python
> from IPython.core.display import Javascript
> from IPython.core.display import display_javascript
> display_javascript(Javascript("""
>     var link = document.createElement('link');
>     link.rel = 'import'
>     link.href = '/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js';
>     document.head.appendChild(link);
> """))
> ```

I tried to work around this problem using this but unfortunately it did not work for me. Thanks though!

@mdreves @atn832 what's the latest on this feature request? any progress on the facets issue?

Seems Beam fixed it, but still not released https://github.com/apache/beam/pull/13045 (included in Beam 2.26 which is under release candidate)

> After pip installing jupyterlab and tensorflow_model_analysis do the following:
> 
> jupyter labextension install @jupyter-widgets/jupyterlab-manager
> jupyter labextension install tensorflow_model_analysis

@ConverJens I was able to install it as a labextension, but I'm also unable to open up the terminal after enabling the extension. This is the error I get in my web developer console tools. 
```
Failed to configure @jupyterlab/terminal-extension:plugin: Loading chunk 5 failed
```
@mdreves What jupyterlab version do we need to make this work as a labextension?
I tested with `Jupyterlab 2.x`.

As @casassg mentioned we are waiting on a beam release.

@fhuanming is currently testing this and can provide more details once we have it working.

> > After pip installing jupyterlab and tensorflow_model_analysis do the following:
> > jupyter labextension install @jupyter-widgets/jupyterlab-manager
> > jupyter labextension install tensorflow_model_analysis
> 
> @ConverJens I was able to install it as a labextension, but I'm also unable to open up the terminal after enabling the extension. This is the error I get in my web developer console tools.
> 
> ```
> Failed to configure @jupyterlab/terminal-extension:plugin: Loading chunk 5 failed
> ```
> 
> @mdreves What jupyterlab version do we need to make this work as a labextension?
> I tested with `Jupyterlab 2.x`.

I was testing it on JupyterLab v1.2.16 and can render the TFMA UI if TFMA>=0.25.0 and Beam is not imported. To fully solve the issue, we are still waiting for the Beam next release (should be happening very soon).


@mdreves thanks for the update!
@fhuanming We need this labextension to work for Jupyterlab 2.x.  Could you also test that it's working for that version?  
Currently, I'm not able to open up the terminal when I have this extension enabled. 

> @mdreves thanks for the update!
> @fhuanming We need this labextension to work for Jupyterlab 2.x. Could you also test that it's working for that version?

I tried to update the jupyterlab version. Then it asked rebuild extension but failed eventually.

> Currently, I'm not able to open up the terminal when I have this extension enabled.

Good catch. I have the same issue but didn't realize that it's caused by this extension. Thanks. I will report the bug.


@mdreves @fhuanming But does the actuall model visualisation work for your? When I tested it with TFMA 0.24.0 and JupyterLab 2.X the plugin failed to load the eval result. I will run another test  with TFMA 0.25. next week.

@ConverJens Hey, to make the visualization work, the version requirements are TFMA >= 0.25.0, Apache-Beam >=2.26.0, Jupyterlab==1.x.

Please let me know if it does not work for you.

@fhuanming Thank you for the clarification! I haven't tested this as we are using JupyterLab 2.X. Is it on the roadmap to support JupyterLab==2.X versions as well? Seeing as 3.X versions are already available.

Would be nice to actually support JupyterLab 3 as it allows for prebuilt extensions similar to Jupyter Notebooks: https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#prebuilt-extensions 

Has anyone found a configuration for this that works with TFX as well? Currently it seems the latest version of TFX (0.26.0) is recommended to be used with apache-beam 2.25.0.

@ConverJens Hi, yes, the JupyterLab 2.x support is on the roadmap. We will consider the support for JupyterLab 3.x after we make sure TFMA is compatible with 2.x.

We recently just fixed the JupyterLab 1.x terminal bug (https://github.com/tensorflow/model-analysis/issues/56#issuecomment-743429245) and will be looking into JupyterLab 2.x now.

Hi, TFMA 0.27.0 is released today and the "Jupyter 1.x console cannot open once TFMA extension is enabled" problem mentioned in https://github.com/tensorflow/model-analysis/issues/56#issuecomment-743429245 is fixed with the new release.

@fhuanming we're currently blocked on the 0.27.0 upgrade, but want to get TFMA working in our notebook environment ASAP. if we wanted to cherry pick your fix for the above terminal bug what would be the best commit/series of commits for that?  the issue->repair PR or commit linkage is not super clear here.

is that https://github.com/tensorflow/model-analysis/commit/19c552e8c0813a356d18ea9a7717d2292f5c2015 ?

@fhuanming also, Jupyterlab 2.x support is critical for us - any ETA on when that will be validated for TFMA?

this is creating a huge UX problem for our adoption of TFX as we have to ask users to context switch into alternative notebook environments just to utilize TFMA. so we're hoping Google can help actively unblock us from a streamlined experience here.

> @fhuanming we're currently blocked on the 0.27.0 upgrade, but want to get TFMA working in our notebook environment ASAP. if we wanted to cherry pick your fix for the above terminal bug what would be the best commit/series of commits for that? the issue->repair PR or commit linkage is not super clear here.

Yes. You're correct. That's the one. That single commit is all you need.

However, (I'm not a frontend engineer, so my understand could be wrong here) when you install the JupyterLab extension, the JupyterLab will pull the extension code from NPM. So you might need a way to let JupyterLab install your local version (which including the commit that you cherry picked above) or you can just manually modify the installed JupyterLab extension locally with the content of commit above.

Sorry I'm not familiar with how github issue works. Let me see if I can link the commit with this issue.

Thanks.

> @fhuanming also, Jupyterlab 2.x support is critical for us - any ETA on when that will be validated for TFMA?
> 
> this is creating a huge UX problem for our adoption of TFX as we have to ask users to context switch into alternative notebook environments just to utilize TFMA. so we're hoping Google can help actively unblock us from a streamlined experience here.

Understand, let me reach out to our frontend engineer about the ETA and get back to you very soon.

> @fhuanming also, Jupyterlab 2.x support is critical for us - any ETA on when that will be validated for TFMA?
> 
> this is creating a huge UX problem for our adoption of TFX as we have to ask users to context switch into alternative notebook environments just to utilize TFMA. so we're hoping Google can help actively unblock us from a streamlined experience here.

We're in the same situation. +1

> > @fhuanming also, Jupyterlab 2.x support is critical for us - any ETA on when that will be validated for TFMA?
> > this is creating a huge UX problem for our adoption of TFX as we have to ask users to context switch into alternative notebook environments just to utilize TFMA. so we're hoping Google can help actively unblock us from a streamlined experience here.
> 
> We're in the same situation. +1

@ConverJens @kwlzn 

Hi, we have verified that TFMA 0.26.0 and 0.27.0 work with JupyterLab 2.x. 

The TFMA 0.26.0 has the "cannot open up jupyter terminal" bug mentioned in  https://github.com/tensorflow/model-analysis/issues/56#issuecomment-743429245. However, if you don't use JupyterLab terminal or you have access to OS terminal, TFMA 0.26.0 will not be an issue to you.

To make it work:

```
$ pip install tensorflow_model_analysis==0.27.0
$ jupyter labextension install tensorflow_model_analysis@0.27.0
$ jupyter labextension install @jupyter-widgets/jupyterlab-manager
$ jupyter lab
```

**The `jupyter labextension install @jupyter-widgets/jupyterlab-manager` is the key here**. We didn't install this and TFMA extension failed to load due to this.

I won't close this issue for now. So please let us know if you still cannot use TFMA extension on JupyterLab. In the meanwhile, we will improve our doc to explicitly add the instruction about how to use TFMA on JupyterLab. 

Thanks! 










@fhuanming thanks a ton for the quick turnaround! we'll get our TFMA version bumped ASAP and validate that E2E before we circle back to close this issue.

@fhuanming I tested TFMA==0.27.0 and the jupyter lab extension as you described above but I still have issues. The terminal bug is fixed, TFMA loads succesfully, I can load TFMA results from disc but when trying to visualise it (`tfma.view.render_slicing_metrics`) nothing shows.

Any idea how I can proceed and debug this?

> @fhuanming I tested TFMA==0.27.0 and the jupyter lab extension as you described above but I still have issues. The terminal bug is fixed, TFMA loads succesfully, I can load TFMA results from disc but when trying to visualise it (`tfma.view.render_slicing_metrics`) nothing shows.
> 
> Any idea how I can proceed and debug this?

Hi @ConverJens, 

- What's the jupyterlab version that you are using? 
- Did you install the TFMA and jupyterlab-manager as I introduced in https://github.com/tensorflow/model-analysis/issues/56#issuecomment-774299762 ? 
- If you install these packages through the JupyteLab console, you may need to refresh the browser after the installation. 
- And what's the error message (either on the UI or on the browser developer console) that you saw after you run the `tfma.view.render_slicing_metrics`?

Thanks!

> > @fhuanming I tested TFMA==0.27.0 and the jupyter lab extension as you described above but I still have issues. The terminal bug is fixed, TFMA loads succesfully, I can load TFMA results from disc but when trying to visualise it (`tfma.view.render_slicing_metrics`) nothing shows.
> > Any idea how I can proceed and debug this?
>
@fhuanming  Thank you for your response!
> Hi @ConverJens,
> 
> * What's the jupyterlab version that you are using?
> * Did you install the TFMA and jupyterlab-manager as I introduced in [#56 (comment)](https://github.com/tensorflow/model-analysis/issues/56#issuecomment-774299762) ?
> * If you install these packages through the JupyteLab console, you may need to refresh the browser after the installation.
> * And what's the error message (either on the UI or on the browser developer console) that you saw after you run the 
> 
> Thanks!

- I using jupyterlab 2.2.9
- Yes
- I install all packages while building my Jupyter image so that is not an issue. I have also reloaded the page multiple times.
- When starting up jupyter I get:
`Uncaught SyntaxError: Unexpected token '<'          vulcanized_tfma.js:1 `
in the dev console.
When executing the cell I get no error messages, not in the UI nor the console. If I reload the page I get the outline of a small red square in the cell output.
When I reopened my notebook today, the following message had appeared: 
`Error displaying widget: model not found`

Anything else I can do to troubleshoot?

I tested yesterday with:
- `pip install jupyterlab==2.2.9`
- `pip install tensorflow-model-analysis==0.26.0`
- `jupyter labextension install @jupyter-widgets/jupyterlab-manager@2`
- For TFMA npm package, we backported the changes to 0.26.0 and used local path to install it: `jupyter labextension install tensorflow_model_analysis/notebook/jupyter/js` - Patched branch: https://github.com/twitter-forks/model-analysis/tree/nle/0.26patched

Posting here to see if this is helpful. I get TFMA fully working with the above steps locally.

@fhuanming I tested again today and specifically specified the jupyterlab-manager version as @casassg had done but with no luck.

I still get an empty output but if I then reload the page, the output now says:
`SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'Overall', 'metrics':…`
so some data seems to be there but the viewer isn't rendered.

> @fhuanming I tested again today and specifically specified the jupyterlab-manager version as @casassg had done but with no luck.
> 
> I still get an empty output but if I then reload the page, the output now says:
> `SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'Overall', 'metrics':…`
> so some data seems to be there but the viewer isn't rendered.

Hi @ConverJens, I still cannot reproduce what you met. I have talked with some of my teammates and it sounds like this is a different problem. Would you mind creating a new issue and if would be great if you can share with us 

- a step-by-step instruction of cmd of how you build your runtime
  - starts with creating a clean python virtual environment
- a minimum notebook to reproduce your case if possible.

So that we can reproduce your cases. Please let us know if you have any questions. Thank you!

@fhuanming Thank you for your help! I will create a clean runtime and test again. 

@fhuanming @casassg Btw, which node version are you using? I've been using node 15.

@fhuanming Issue still remains for me. I created https://github.com/tensorflow/model-analysis/issues/112 with a minimal reproducible example.

npm --version -> 6.14.5
node --version -> v12.18.2

Not sure if this can be it.

> @fhuanming @casassg Btw, which node version are you using? I've been using node 15.

npm --version  -> 6.14.10
node --version -> v14.15.4



@casassg @fhuanming I don't think this is the issue. As I mentioned in the other issue, I've now tested with both node 12 and 15 with the same result. I also tested in both Chrome and Safari.

Not sure if this may work, but I also ran `python setup.py jsdeps` when using local repo to re-compile the vulcanized_tfma.js. If you are using the prebuilt package, then this probs is not needed (no idea here tbh). 

we've been able to make this work with a workaround, so I'm closing this issue for now - however we're still seeing an upstream issue wrt lab vs nb extensions that I've described in https://github.com/tensorflow/model-analysis/issues/112#issuecomment-797141302 - so will track this separate problem in #112.

thanks a ton for all the help!