Repository: evidentlyai/evidently
Issue: Input for ProbClassificationPerformanceAnalyzer
Root cause: Validation Gap

I posted same question at discord first, but this maybe a more appropriate place. We'll see ;-)

I'm writing some tests for dashboards and profiles but stumbled upon a probable bug in `ProbClassificationPerformanceAnalyzer`.
https://github.com/burkovae/evidently/blob/ab-fix-classification-bugs/evidently/analyzers/test_classification_analyzers.py

Since I'm not sure, how a correct input would look like, I hope that someone can post ist here. I will be happy with a very basic data for an analysis of a single dataframe. Hence, I hope to get some correct input, so that I can narrow down what's happening and what components fails.

Especially I would like know, what the `ProbClassificationPerformanceAnalyzer` computes exactly (@emeli-dral it seems you wrote this piece). I would be happy with few inputs and expected outputs. 

If you already know the fix to

```
Error
Traceback (most recent call last):
  File "...\evidently\evidently\analyzers\test_classification_analyzers.py", line 21, in test_single_dataset_with_two_classes
    blubb = analyzer.calculate(df, None, ColumnMapping())
  File "...\evidently\analyzers\prob_classification_performance_analyzer.py", line 32, in calculate
    prediction_labels = [prediction_column[x] for x in prediction_ids]
TypeError: 'numpy.intc' object is not iterable
```

then I urge you to write the tests with inputs and expected outputs first, please. It helps down the line and improves robustness and safeguard against unwanted changes.

If you can point me in the right direction, I will fix any issues along the road. I just need to know what you compute specifically with some explicit examples.

Thanks @Liraim for pointing out the example https://github.com/evidentlyai/evidently/blob/main/evidently/examples/iris_probabilistic_classification_performance.ipynb

```python
    def test_single_dataset_with_two_classes(self):
        df = pd.DataFrame(
            {
                'target': ['label_a', 'label_a', 'label_a', 'label_b', 'label_b', 'label_b'],
                'label_a': [.1, .2, .3, .4, .5, .6],
                'label_b': [.9, .8, .7, .6, .5, .4],
            }
        )
        df_column_mapping = ColumnMapping(
            target='target',
            prediction=['label_a', 'label_b'],
        )
        analyzer = ProbClassificationPerformanceAnalyzer()
        result = analyzer.calculate(df, None, df_column_mapping)
        ...
```
That's the essential part...


We are planning to write tests for our components along with documentation about its usage and inputs / outputs soon.

I think we can discuss inside team about writing documentation first, if it can provide some information that you ask for.

I can suggest, that you rethink writing documentation _first_. [Good tests](https://www.testim.io/blog/unit-testing-best-practices/) are kind of good place to start with documentation. That works for me anyway: Inputs -> Behavior -> Outputs as test.

See point 4 for example
> Test cases double as a form of documentation. And they are the best type of documentation since they’re executable and don’t get out of sync with what they’re documenting. But in order for the team to be able to reap the benefits of these executable specifications, they obviously need to be readable.

evidently obviously has readable tests in form notebooks. They are just not very convenient for testing _behavior_ of the system. However, given some time and contributions from the community we will eventually get there. Just not tomorrow ;-)

Because evidently progressed a lot in the codebase, it will take a good chunk of time (writing test + refactoring + rewriting tests) until the unit tests serve as splendid documentation, however.  See [dplyr's tests](https://github.com/tidyverse/dplyr/tree/main/tests/testthat) for example - and even if you are not R native, you basically have examples how each function works as a documentation (and of course RMarkdown vignettes serve similar purpose as note books in evidently 👍). 

But take it with a grain of salt, that's just my opinion.

Probably my previous answer was misleading:

You are totally right that good tests can be good documentation, and we definitely will write this kind of tests. But it is time-consuming to write proper tests, so I mentioned documentation.

And I didn't mean that documentation will be substitution for tests in any way, docs that I meant is some kind of high-level description about how given component work, and how you can use it (maybe with code snippets). And it is definitely not a way to test our code.

In short:
- tests is our priority for code-quality
- documentation also our priority but for usability

thb, 4 years ago, I though writing proper tests would be time consuming as well until a co-worker introduced me to test driven development. Since then I cannot help myself but to write tests first :-) 
We all have our journey and the community is here to help. Because I believe in evidently's approach ❤, I will contribute to tests whenever I can.