Repository: evidentlyai/evidently
Issue: null handling in data drift check
Root cause: Validation Gap

i saw in the code that rows are dropped in nulls in categorical columns and records with infinite values are also not handled for numerical columns.
what happens if the reference data has no nulls and the test data has nulls?
it wont find a difference or it will have a difference because the whole row was dropped because of the null value ?

Hello!

The short answer is yes, we have a problem in this case.

We count all `null` and `inf` rows for data quality dashboards and add them to statistics.

But for data drift, we filter the rows with such values and do not consider them. It is a problem - if we got many dropped values in the test/current dataset, we might not detect data drift if reference values do not drift other data.

You can try to replace the values with some meaningless values in your datasets, but it can affect other metrics.

thanks for the answer .
i suggest to parameterize the behaviour so we can choose to ignore the nulls or not.
i think in the majority of the use cases it is crucial to see the difference with null percent in each feature , and it should affect the distribution , even more than a regular shift in the feature value (if we impute null with constant value like 0 or -1 we will see a big difference in the data that is supplied to the model we train)

Hi @yairVanti!
We agree here - it’s really important to control the ratio of nan values. It's something we've discussed and would like to do. Could you share, how would you prefer this option to work?

As @Tapot mentioned, in the meantime you can use a data quality report https://docs.evidentlyai.com/reports/data-quality where we calculate the ratio of nan values and other statistics.

my opinion is that the behaviour for drift detection needs to give a big weight for a use case where in one dataset there is null ratio much larger than the other.
i think in such a case imputing the null to a value with the biggest shift from the mean of the value will be the right choice.
like smaller than the min value or bigger than the max value from the reference dataset.

Hi @yairVanti 

Wanted to share some thoughts here! 

We considered adding some modification like this (null pre-processing before applying the drift test), but decided against. Mainly, to make sure the drift detection test does not exhibit some unpredictable behavior, as such imputation of NULLs is likely not expected by most users, and it can impact different statistical tests differently (we are now adding a few more tests to choose from).  

We agree that in general appearance of a large number of NULLs is an issue to react to, but more often this is a data quality/integrity issue rather than a pure "data drift" (change in the real-world population). 

What we suggest to do here, is to group the data drift test with data quality tests: where one test would check for increase in NULLs beyond a certain threshold, and another to check for statistical distribution drift. 

Right now, these are two different reports in Evidently (Data Drift and Data Quality) that you can group together. If you run such check as a part of pipeline, you can also use JSON Profile and only extract the metrics you need. 

We are also now working on making such "testing" use case easier by bundling different metrics in test suites. Hope to release something in a matter of weeks!

Hope that makes sense!