Repository: evidentlyai/evidently
Issue: Problem getting ClassificationPreset to recognize Binary Probabilistic ColumnMapping
Root cause: Validation Gap

Hi. I'm attempting to use the ClassificationPreset to generate some reports on a model running in production.  Evidently version = 0.4.2. The general flow looks like this:

```python
from evidently import ColumnMapping
from evidently.report import Report
from evidently.metric_preset import ClassificationPreset

col_map = ColumnMapping()
col_map.target = 'actual'
col_map.prediction = 'model_score'
col_map.datetime = 'SERVICE_DATE'
col_map.pos_label = 1
col_map.target_names = [f'Not outcome', 'outcome']
col_map.task = 'classification'
classification_perf = Report(metrics=[ClassificationPreset(probas_threshold=0.671)]) # tried without the threshold too.
```

In this paradigm the following is true:

* `df['model_score']` is a calibrated probability with range (0,1) 
* `df['actual']` is a binary outcome [0 or 1]
* `df['SERVICE_DATE']` is a datetime column of when a record was created.

When I run the "runner code" below, getting this error:

```python
classification_perf.run(current_data=df[['actual','model_score','SERVICE_DATE']], reference_data=None, column_mapping=col_map)
classification_perf.save_html('/PATH/TO/FILE/test_class_report.html')
```
Error:
```bash
ValueError: Top K / Threshold parameter can be used only with binary classification with probas
```

I'm able to get it to run if I don't provide a probability and instead just provide a pre-threshold'd value, but I would like to get the full report based on probabilities as the confusion matrix and recall/precision are insufficient for my level of monitoring.

---

So two questions and a comment:

1. How do I get it to recognize that this is a probabilistic binary classification? Based on the docs here: [col mapping binary Option 3](https://docs.evidentlyai.com/user-guide/input-data/column-mapping#binary-probabilistic-classification-option-3), I believe I'm doing all of the things correctly.
2. Could you move to a paradigm where I can specify in the column mapping two types of inputs: `col_map.prediction` and `col_map.predict_proba`? I'd love to be able to provide both and have the applicable one used behind the scenes for each plot. Then the plots could be dynamic to "do I have the right type of predict" in the metrics and reports. That also allows me to apply different thresholds to different rows if needed since I can provide the "actual prediction" in one column instead of trying to set a single flat threshold. 

Comment: I believe your examples on `pos_label` in the link (and for all of the Column Mapping example page) above are missing the `column_map` component. It just sets `pos_label = 1` when I believe it should say (if I'm reading the source code correctly): `column_mapping.pos_label = 1`. If that's not true, I'm not sure how the `pos_label` in your examples are being passed into the Report runs. 

EDIT: I moved to [Style 2: Probabilistic Classification](https://docs.evidentlyai.com/user-guide/input-data/column-mapping#binary-probabilistic-classification-option-2) and was able to get it to work. That version adds a lot of extra passes through the data to relabel binary outcomes to have a specific name - so I'd still like help on how to use version 3 which is more direct for my data. Thanks!

Hi @ZWMiller, 

Thanks for sharing! 

**1. Mapping the binary probabilistic classification.** We could not immediately reproduce the issue. Here is the notebook where we run a similar column mapping (option 3) on a dummy dataset and it works correctly https://colab.research.google.com/drive/1KPX60oD7DRykyMKUnjxq-0oUkumJSBhl

Could you try coming up with a reproducible example?

And thanks for noting the error in the docs about `pos.label` - you are right, it should be `column_mapping.pos_label = 1` !
 


**2. Suggestion on adding `predict_proba` to column mapping** - cc @emeli-dral to take a look! 

Hi @elenasamuylova , Thanks for the link. I was unable to replicate (just as you were) in the collab, so maybe it's something with my dataset. I will attempt to find a replicable simple example that I can share. Hopefully it's not something weird with my data and I just pinged y'all for nothing.

Cheers.

I actually had this same issue today, and it seems the issue for me was that my prediction column was a `float32`. When I explicitly cast it as a `float64` I get no error. Here is a reproducible example:

```python
import pandas as pd

from evidently import ColumnMapping
from evidently.report import Report
from evidently.metric_preset import ClassificationPreset

df = pd.DataFrame.from_dict({'label': {0: 1, 1: 1, 2: 0},
 'prediction': {0: 0.9780731797218323,
  1: 0.8408827185630798,
  2: 0.02511681616306305}})

# df["prediction"] = df["prediction"].astype("float32") # doesn't work
# df["prediction"] = df["prediction"].astype("float64") # works

column_mapping = ColumnMapping()
column_mapping.target = "label"
column_mapping.prediction = "prediction"
column_mapping.pos_label = 1

classification_performance_report = Report(metrics=[
    ClassificationPreset(probas_threshold=0.5),
])

classification_performance_report.run(
    reference_data=None,
    current_data=df,
    column_mapping=column_mapping,
)
classification_performance_report.show(mode="inline")
```

Thanks for sharing @kingdavescott - let us take a look! cc @0lgaF 

Well spotted @kingdavescott , I checked that it was being treated as a float but didn't check the full typing. Good eye!

I can confirm that was also the issue on my side.

Thanks for confirming @ZWMiller - we also reproduced the error and will address it in the next release!