Repository: evidentlyai/evidently
Issue: Data Definition Object Useless or Not Working
Root cause: Validation Gap

# Overview of Issue

The DataSummaryPreset is calculating metrics on columns not defined in the data definition, but just happen to exist on the dataset.   

# Sample 

```python
    data_definition = DataDefinition(
        numerical_columns=numeric_features,  # list
        categorical_columns=categorical_features, # list
        regression=regression,  # [Regression(name='default', target='acquisition_price', prediction='predicted')]
    )


# Convert datasets to evidently objects
    reference_evidently = Dataset.from_pandas(
        reference,  # pd.DataFrame
        data_definition=data_definition,  # DataDefinition object
    )
    current_evidently = Dataset.from_pandas(
        current,   # pd.DataFrame
        data_definition=data_definition,  # DataDefinition object
    )

# List of reports
report_list = [
   DataDriftPreset(columns=model_features_list),
   DataDriftPreset(columns=target_predictions_cols),
   DataSummaryPreset(columns=model_features + target_prediction_cols),
   RegressionPreset()
]

# Report 
report = Report(report_list, include_tests=True)
executed_report = report.run(current_data=current_evidently, reference_data=reference_evidently) 
```

# Issue 

The DatasetSummaryPreset() is calculating summary metrics and tests are failing on columns that are not defined in my data definition. **If this is because the dataset has these extra columns that are not defined in model_features or target_prediction_cols, I ask what is the point of the DataDefintion object**? I am hoping I am just doing something wrong and it's not using it. Otherwise, for the DataDriftPreset, I already need to define the columns in the Report object, so I don't see the necessity of the DataDefinition object there either,.  

Hello! I saw this issue has been open for some time. I’d be happy to contribute and work on it—please let me know if it’s still available.


@yiannimercer,
Thanks you waiting!
I am investigating why `DatasetSummaryPreset` can use columns that aren’t explicitly mentioned. What I’ve found so far:
- The preset reads the **entire dataset** to compute overall dataset metrics (e.g., counts of columns by type, etc.)
- It uses **provided column list** for `ValueStats` calculations over those columns (e.g., min/max values, etc.)

Because of this, tests in this preset should compute only for given metrics. Failures related to unmentioned columns are expected only in first set of metrics. Does that behavior match what you’re seeing?

It would also really help if you could share which tests are failing along with actual vs. expected outputs. That will let me confirm I’ve caught all the issues in this preset. Thanks!

> I already need to define the columns in the Report object, so I don't see the necessity of the DataDefinition object there either
Some presets and metrics allow omitting the columns list and we can infer it from `DataDefinition` object for further use. For example:
- `DataDriftPreset` - if you omit `columns` argument - it would automatically use all numerical, categorical and text columns from `DataDefinition` to calculate metrics.


Apologies, just seeing this. I can't recall how to replicate the tests, but I think we found a work around.  Ultimately though, this boiled up into the issue I just left here: #1765 

What is the point of the `DataDefinition` if at the end of the day, the preprocessing, the tests (example on this issue) are using all the columns you provide in the DataFrame. The `DataDefintion` effectively does nothing.  

**If I tell evidently, "these are my categorical columns, these are my numeric columns, this is my target column, this column has my predictions, this is my Date column, .., etc." (what the `DataDefinition` seems to be for), why would things like data preprocessing (see linked issue), drift tests, or really anything at all consider columns that are defined in this object?**