Repository: evidentlyai/evidently
Issue: DataDrift not working
Root cause: External Dependency Breakage

Hello I'm trying to use some methods but there are not working anymore is seems from evidently.report import Report and 
from evidently.metric_preset import DataDriftPreset. Did anyone has the same issue ? 

Very sorry that's for the drift and other issues related to ML system, I recommended you to fix the version to `evidently==0.6.7` (as the lastest support on ML) and related to the docs at https://docs-old.evidentlyai.com/user-guide/how-to-guides, after `0.7.x`, everything like `spark,...` has moved to the `legacy` module, marked for no more support.

`evidently` is not like the original anymore, while most enterprise use cases focus on ML, the community is increasingly integrating LLM components, even the documentation is like 70-80% LLM stuffs

Hi @thangbuiq, 

This isssue is closed since I believe the author found the answer, but I'd like to clarify for anyone else who comes across it.

➡️ **Evidently still actively supports ML and data drift use cases**.

v0.6.7 simply introduced a huge refactor of the metric API. All core ML monitoring features, including data drift and quality checks, were refactored and are actively maintained. 

(Spark module is indeed no longer actively developed. But it was partial even previously and unfortunately we do not have the capacity to test and maintain it for all metrics).

The new Evidently version includes many requested features, like joining of Reports with Test Suites to avoid duplication; ability to render custom ML metrics in self-hosted UI, etc. 

For ML use cases, here are some updated resources:
- [Quickstart for ML](https://docs.evidentlyai.com/quickstart_ml) – how to run and use the updated import paths.
- [Cookbook of individual ML metrics](https://github.com/evidentlyai/evidently/blob/main/examples/cookbook/metrics.ipynb) – including drift, data quality, classification, regression, etc.
- [Complete list of data ML-focused metrics in the docs](https://docs.evidentlyai.com/metrics/all_metrics)

Is there anything specific you are missing beyond Spark?

> Hi [@thangbuiq](https://github.com/thangbuiq),
> 
> This isssue is closed since I believe the author found the answer, but I'd like to clarify for anyone else who comes across it.
> 
> ➡️ **Evidently still actively supports ML and data drift use cases**.
> 
> v0.6.7 simply introduced a huge refactor of the metric API. All core ML monitoring features, including data drift and quality checks, were refactored and are actively maintained.
> 
> (Spark module is indeed no longer actively developed. But it was partial even previously and unfortunately we do not have the capacity to test and maintain it for all metrics).
> 
> The new Evidently version includes many requested features, like joining of Reports with Test Suites to avoid duplication; ability to render custom ML metrics in self-hosted UI, etc.
> 
> For ML use cases, here are some updated resources:
> 
> * [Quickstart for ML](https://docs.evidentlyai.com/quickstart_ml) – how to run and use the updated import paths.
> * [Cookbook of individual ML metrics](https://github.com/evidentlyai/evidently/blob/main/examples/cookbook/metrics.ipynb) – including drift, data quality, classification, regression, etc.
> * [Complete list of data ML-focused metrics in the docs](https://docs.evidentlyai.com/metrics/all_metrics)
> 
> Is there anything specific you are missing beyond Spark?

Thanks for the reply, @elenasamuylova 

We're currently using evidently version `0.6.7` for data validation in a large-scale, big data platform. The setup is working decently, and I believe many enterprise environments are using a similar approach.

We’ve avoided migrating to `0.7.x`, not because of the import path changes like `evidently.spark` to `evidently.legacy.spark`, that part is manageable - but mainly due to performance issues and lack of documentations about spark module after `0.6.7`. The Spark module has been quite poor performance, generating a very high number of stages (sometimes up to 800) during report.run(), which makes it impractical at scale.

There's also a bit of a trust gap, since the updated documentation for the Spark module is missing—unlike the older one here: https://docs-old.evidentlyai.com/user-guide/tests-and-reports/spark

Would it be okay if I opened a new issue to discuss this in more detail? I'd be happy to help provide more context or examples if needed.

Hi @thangbuiq, 

Thanks for the thoughtful reply!

You are absolutely welcome to open a new issue or drop by [Discord](https://discord.gg/xZjKRaNp8b). But as a heads-up: as I mentioned, evaluations on Spark are no longer maintained in open-source (even before the API update), so we won’t be able to offer engineering help or debugging for Spark-specific issues. 

This is mainly due to the high cost of testing across all metrics and support needed for our users on how to run it efficiently depending on Spark cluster configuration. Unfortuntaley maintaining this in open-source just hasn’t been feasible for us.