Repository: huggingface/evaluate
Issue: Feature: compose multiple metrics into single object
Root cause: Unimplemented Feature Gap

Often models are evaluated on multiple metrics in a project. E.g. a classification project might always want to report the Accuracy, Precision, Recall, and F1 score. In `scikit-learn` one use the classification report for that which is widely used. This takes this a step further and allows the user to freely compose metrics. Similar to a `DatasetDict` one could use the `MetricSuite` like a `Metric` object. 

```Python
metrics_suite = MetricsSuite(
     {
        "accuray": load_metric("accuracy"),
        "recall": load_metric("recall")
     }
)

metrics_suite = MetricsSuite(
     {
        "bleu": load_metric("bleu"),
        "rouge": load_metric("rouge"),
        "perplexity": load_metric("perplexity")
     }
)

metrics_suite.add(predictions, references)
metrics_suite.compute()
>>> {"bleu": bleu_result_dict, "rouge": roughe_result_dict, "perplexity": perplexity_result_dict}
```

Alternatively, we could also flatten the return dict or have it as an option. We could also add a `summary` option that defines how an overall result is calculated. E.g. `summary="average"` averages all the metrics into a summary metric or a custom function with `summary=lambda x: x["bleu"]**2 + 0.5*["rouge"]+2`. This would allow to create simple, composed metrics without the needing to define a new metric (e.g. for a custom benchmark). 

cc @douwekiela @lewtun 

I like it! I think we've been calling this an "aggregation function" in Dynabench (eg the Dynascore aggregates scores for several metrics and datasets with different weightings into a single scalar). In your snippet above, is "references" the ground truth? We may not always have a ground truth (e.g. for a perplexity metric) I guess.

>  In your snippet above, is "references" the ground truth? We may not always have a ground truth (e.g. for a perplexity metric) I guess.

Indeed, more subtle would be metrics that expect different inputs (e.g. full strings vs. tokenized strings, or one-hot predictions vs. logits). I think at the beginning this could just be a power user feature and we can figure out how to support different input types or analyze metric compatibility later. E.g. https://github.com/huggingface/evaluate/issues/9 could help with that. 

From our discussion, here are some API ideas:

I. For metrics with the same inputs

```python
>>> from evaluate import load_metrics
>>> 
>>> metric = load_metrics(["bleu", "rouge"])
>>> metric.compute(predictions=..., references=...)
{"rouge1": ..., "rouge2": ... , ... , "bleu": ...}
```

PS: it fails if you mix metrics with incompatible inputs like "accuracy" and "bleu". We would need a function for users to redefine their inputs if we want to have this in the end:
```python
>>> metric.compute(text_predictions=..., text_references=..., labels_predictions=..., labels_references=...)
```

II. If input names don't match, it's ok

```python
>>> from evaluate import load_metrics
>>>
>>> metric = load_metrics(["bleu", "perplexity"])
>>> metric.compute(predictions=..., references=..., input_texts=...)  # input_texts is the perplexity input
{"bleu": ..., ..., "perplexity": ...}
```

PS 2: would it be nice to rename the input of perplexity to be one of references or predictions ?

III. If you need more control over the metric, you can load them separately

```python
>>> from evaluate import load_metric, combine_metrics
>>> 
>>> bleu = load_metric("bleu")
>>> bleurt = load_metric("bleurt", "bleurt-large-512")
>>> metric = combine_metrics([bleu, bleurt])
```

IV. Aggregate metrics

```python
>>> from evaluate import load_metrics
>>> 
>>> metric = load_metrics(["bleu", "rouge"]).add_mean(["rouge1", "rouge2", "bleu"], weights=[...])
>>> metric.compute(predictions=..., references=...)
{"rouge1": ..., "rouge2": ... , ... ,"mean_rouge1_rouge2_bleu": ...}
```

PS 3: we could also have a function `.apply()` if users want to define their own aggregation functions.

PS 4: we could also allow users to discard and/or rename output values

Feel free to comment/edit this if you have other ideas :)

For PS comment on I) we could also do this if we have some more specific `Features` in `MetricsInfo`, right? Then we could have:

```python
>>> metric.compute(predictions=[texts, logits, ...], references=[texts, label_ids, ...])
```

Then we could dispatch the inputs based on feature type to each metric - but maybe that's too implicit? Also wouldn't work if two metrics have the same type but still different format. 

I haven't seen any other library or project approach mix of metrics using an API like this, I feel like it would be hard to have an intuition that it must be used this way. Therefore I would be in favor of having something more explicit.

Anyway it's ok to not focus on this case right now, I don't think it would have much usage and users can still handle two separate metrics separately