Repository: huggingface/evaluate
Issue: Feature: calibration error estimators
Root cause: Unimplemented Feature Gap

While implementations of standard metrics are scattered, this is certainly the case for L^p calibration error estimators.

Why is it important to measure model calibration? 
When fine-tuning a model with cross-entropy loss (or any other strictly proper loss optimizing both accuracy and calibration) there is no guarantee that your model will turn out well-calibrated. Empirically, large NNs were shown to "overfit" on accuracy, leading to sub-optimal calibration.

Additionally, binned estimators typically require setting some arguments such as the binning scheme (equal-range/equal-mass/...), number of bins, and the p-norm. More advanced settings include debiasing ([1] Kumar et al. 2019) or the proxy used for average bin probability (bin center or bin left/right edge). 
This library might provide the right standardization and documentation on which arguments are important and how they impact estimation and comparisons.

Without complicating matters from the start, it might already be nice to have a simple calibration error estimator along the lines of `ECE` ([2] Guo et al. 2017), which (despite some flaws and differing implementations) is commonly used to report top-1 miscalibration.  Some "reasonable" defaults are equal-range binning with 15 bins and p-norm 1, to be discussed :).

With regards to the implementation, there is a clean way to create a hashmap of unique bin assignments which keeps running averages of the conditional expectation and average bin probabilities. In the future, the hashmap can even be created on the validation set and when evaluating on the test set retrieves values by the hash, resulting in unbiased estimates.

I would very much appreciate a discussion on this so that the community is aligned on the approach.
Over the weekend, I will get my hands dirty for a first PR, maybe better to discuss it there :)
First functional code dump here: [https://huggingface.co/spaces/jordyvl/ece](ece)

[1] Kumar, A., Liang, P.S. and Ma, T., 2019. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32.
[2]  Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q., 2017, July. On calibration of modern neural networks. In International Conference on Machine Learning (pp. 1321-1330). PMLR.


Hi @Jordy-VL that sounds like an interesting addition! Let me know if you need any help with setting up a custom metric. It will be indeed simpler to discuss on a PR. 

cc @douwekiela 

Hi @lvwerra, I created the first proof of concept demo [here](https://huggingface.co/spaces/jordyvl/ece/blob/main/local_app.py ) (run locally for now).

The space allows calculating ECE with sliders and dropdowns for varying keyword arguments to `_compute`.
Currently, it gives both the metric and a reliability plot that renders dynamically: 
![image](https://user-images.githubusercontent.com/16034009/172860873-daa4a628-962c-4b6d-8ae9-16e2b180a7a4.png)

I have not yet touched the `evaluate/utils/gradio.py` file, but in order to create this demo with the original code, `launch_module] needs to accept both new `inputs`, `outputs`, `function`. I do not know if you would rather want to keep this clean or allow for extensions :)



Hi @Jordy-VL, this looks super cool! 

The `launch_module` function is mainly a helper to setup the gradio app without much customization and serves no functional purpose when loading the metric with `evaluate`. So you can write your `app.py` with Gradio and setup your own widget like you want. I think It would be great to keep the structure similar to the other metrics so it's easily recognizable. In particular parsing the README and metric description, see here:
https://github.com/huggingface/evaluate/blob/76218cf1ea5e757feaaabe96132391a5005aa84c/src/evaluate/utils/gradio.py#L119-L121

It could also be nice to add examples to the Gradio interface so you get a nice default plot. 

Also, I don't think it is not necessary to merge your metric into `evaluate` - we could keep it where it is under your name. That way it stays connected to you and is a nice example of a community metric. What do you think?

Continued discussion here: `https://huggingface.co/spaces/jordyvl/ece/discussions/1`

Closing this for now as the discussion is continuing on hf.co.