Repository: evidentlyai/evidently
Issue: TestColumnsType -> extract exact column that made the test fail
Root cause: Validation Gap

Hi,

Started to try your package and this amazing !
Have two questions

Q1:
Tried to use the TestSuite with number of tests.
For example the 'TestColumnsType' was failed ( changed it manually ) , how can I extract the **exact column**  that made the test fail?

My tests :

set_of_tests_for_data_quality =\
TestSuite(tests=[
     evi_tests.TestColumnsType(), 
  
])

Q2:
Using the same data ( with mismatched column type ) the report was crushed . 

Error => ValueError: Column Total_intl_calls have different types in reference int64 and current float16

I assumed it would show me the problematic column.

data_drift_report = Report(metrics=[ DataDriftPreset(),]) 
data_drift_report.run(reference_data = train_df , current_data= test_df, column_mapping=column_mapping) 

Will appreciate your response,
Boris


Hi @borisRa, 

**1. Extracting the names of columns with type mismatch.** 

If you generate the visual test suite output (HTML file or directly, e.g., in the Jupyter notebook) for this test, on expand, it contains a table that shows each expected/actual column type. You can explore it visually to see for which columns it failed. 

![image](https://user-images.githubusercontent.com/67064421/225712183-54f89461-6a98-4943-8efc-e74ec6eb2b78.png)

If you want to extract the list programmatically, you’d need to parse the JSON / Python dictionary output. 

(Here is how to get the output of the test suite as Python dictionary or JSON https://docs.evidentlyai.com/user-guide/tests-and-reports/run-tests#output-formats)

The easiest way is to get the output as a Python dictionary and extract the necessary information using Python.

**2. Data drift report** 

Data drift report performs statistical tests / calculates distance metrics to detect the difference in feature distributions between two datasets. It expects the data schema and column types to match; otherwise, it is not possible to perform the computation. 

It is expected behavior; this report/metric is not meant to help discover data quality issues (with that purpose, it is best to use tests from the data quality/data integrity group). 

Thanks @elenasamuylova !

About **Extracting the names of columns with type mismatch** :  Is there a way to get mismatched columns directly without looping through all columns ?

About  **Data drift report** : I tried to use the `TestColumnsType` test but got this error : 
`ValueError: Column Total_intl_calls have different types in reference int64 and current float16` I expected to get this test fail but not crash.

My code is :
```

set_of_tests = TestSuite(tests=[ evi_tests.TestColumnsType(),])		   
set_of_tests.run(reference_data=train_df, current_data=test_df)
```

							

Hi @borisRa, 

1. Right now, you can only work with the existing JSON/Python dict output, so all further transformations must happen externally. (There will be some changes to the JSON format of all metrics/tests in the near future to make it easier to parse). 

2. Sorry, I must have misunderstood your question since the code snippet mentioned `DataDriftPreset()`. Don't have an immediate answer here as to why this error might come up with `TestColumnsType` test. 

Does it occur when you run a single `TestColumnsType`  test in a test suite, or do you have multiple tests in a suite? 

CC @Liraim and @emeli-dral to take a look when they can!
 

Thanks @elenasamuylova  ,

About "Does it occur when you run a single TestColumnsType test in a test suite, or do you have multiple tests in a suite?" 
Both as a single test showed above and as multiple test.

Hi @borisRa,

Re: TestColumnsType test throwing an error: we ran a number of tests (e.g., changing the int64 to float16 in current data) but could not reproduce the error. Is there a chance you could share a few lines from your dataframe, including nulls if any?

@borisRa Quick question: what evidently version do you use? I believe we cannot reproduce the issue, because it was solved in the 0.2.7 version

Hi @emeli-dral ,

Thanks this issue was resolved !
Though a new one was  arrived :)

`set_of_tests_for_data_quality.save_html(path)`

Getting this error => `AttributeError: 'TestResult' object has no attribute 'features'`

hey @borisRa ,
could you please also share with us the full stack trace error?

We struggle to reproduce this one as well.