Repository: evidentlyai/evidently
Issue: Error in DataDrift dashboard show() on databricks notebook
Root cause: Architectural Constraint

Hi Evidently team,

I'm trying to show the datadrift dashboard I created on a databricks notebook. Using show(mode="inline") works but it also has a 20MB limit, in my case I can only use one integer column up to 75 reference samples and 75 current samples out of my 200k dataset.
`data_drift_dashboard.calculate(df[:75], df[75:150], column_mapping=None)`
See error screenshot below:
`Command result size exceeds limit: 20971520 bytes (current 20974461)`
<img width="648" alt="Screen Shot 2022-05-11 at 6 06 33 PM" src="https://user-images.githubusercontent.com/90450365/167835546-9baf1478-cc44-4379-b6d0-f86736fb756f.png">

I tried using the mode="auto and mode="nbextension" but I get this error:
`Uncaught ReferenceError: requirejs is not defined`
<img width="648" alt="Screen Shot 2022-05-11 at 7 01 13 PM" src="https://user-images.githubusercontent.com/90450365/167835964-3e3ad41f-812d-4fb4-8962-176dce05d793.png">

Using the colab method is not an option for me as I need to have it done on a databricks notebook.
Thank you and please let me know if there's an existing answer for this already in the documentation or previous issues. 

Kind regards





Hi @shebna-kumu , 
first of all, apologies for the late response.

It looks like a limitation from Databricks side (a reasonable one, we know that our dashboards are currently too large 😳).

I am afraid there is no clear-cut solution to compress the dashboard to 20MB. I can suggest a few workarounds. I hope one of them can help:
- generate the DataDrift Profile instead of a Dashboard, then derive drift scores from it (you'd need to write custom code for it), and visualize the results using Matplotlib or Plotly.
- split the dataframe by columns, for example, most important features VS the rest, and create 2 separate reports. This should help reduce each report's size so that each one fits into the limitation.

Also, you can try to save the Dashboard as HTML if it is possible to do so in the Databricks environment and then open it in the browser. 

By the way, we are working on Dashboards size optimization and hope to share some good news in the next releases!

 

> Hi @shebna-kumu , first of all, apologies for the late response.
> 
> It looks like a limitation from Databricks side (a reasonable one, we know that our dashboards are currently too large 😳).
> 
> I am afraid there is no clear-cut solution to compress the dashboard to 20MB. I can suggest a few workarounds. I hope one of them can help:
> 
> * generate the DataDrift Profile instead of a Dashboard, then derive drift scores from it (you'd need to write custom code for it), and visualize the results using Matplotlib or Plotly.
> * split the dataframe by columns, for example, most important features VS the rest, and create 2 separate reports. This should help reduce each report's size so that each one fits into the limitation.
> 
> Also, you can try to save the Dashboard as HTML if it is possible to do so in the Databricks environment and then open it in the browser.
> 
> By the way, we are working on Dashboards size optimization and hope to share some good news in the next releases!

Thanks for the HTML solution - displayHTML(data_drift_dashbaord.get_html()) works for me to get the visual on databricks.