Repository: huggingface/evaluate
Issue: Defining and standardizing metric input structures
Root cause: Unimplemented Feature Gap

### Documenting the types of inputs and data structures used for each metric 

[accuracy](https://github.com/huggingface/evaluate/tree/main/metrics/accuracy)
```
predictions = Sequence(Value("int32"))
references = Sequence(Value("int32"))
```
or if `"multilabel"` mode:
```
predictions= Value("int32")
references = Value("int32")
```

[bertscore](https://github.com/huggingface/evaluate/tree/main/metrics/bertscore)
```
predictions = Value("string", id="sequence")
references = Sequence(Value("string", id="sequence"), id="references")
```

[bleu](https://github.com/huggingface/evaluate/tree/main/metrics/bleu)
```
predictions = Sequence(Value("string", id="token"), id="sequence")
references = Sequence(Value("string", id="token"), id="sequence"), id="references"
```

[bleurt](https://github.com/huggingface/evaluate/tree/main/metrics/bleurt)
```
predictions = Value("string", id="sequence")
references = Value("string", id="sequence")
```

[cer](https://github.com/huggingface/evaluate/tree/main/metrics/cer)
```
predictions = Value("string", id="sequence")
references = Value("string", id="sequence")
```

[chrf](https://github.com/huggingface/evaluate/tree/main/metrics/chrf)
```
predictions = Value("string", id="sequence")
references = Sequence(Value("string", id="sequence"), id="references")
```

[code_eval](https://github.com/huggingface/evaluate/tree/main/metrics/code_eval)
```
predictions = Sequence(Value("string"))
references = Value("string")
```

[comet](https://github.com/huggingface/evaluate/tree/main/metrics/comet)
```
sources = Value("string", id="sequence")
predictions = Value("string", id="sequence")
references = Value("string", id="sequence")
```

[competition_math](https://github.com/huggingface/evaluate/tree/main/metrics/competition_math)
```
predictions = Value("string")
references = Value("string")

```

[coval](https://github.com/huggingface/evaluate/tree/main/metrics/coval)
```
predictions = Sequence(Value("string"))
references = Sequence(Value("string"))
```
N.B. The sentences have to be in CoNLL format, which may be tricky to handle in some cases

[cuad](https://github.com/huggingface/evaluate/tree/main/metrics/cuad)
```
"predictions": {
    "id": Value("string"),
    "prediction_text": Sequence(Value("string")),
}
  "references": {
      "id": Value("string"),
      "answers": Sequence(
          {
              "text": Value("string"),
              "answer_start": Value("int32"),
          }
      ),
  },
}
```

[exact_match](https://github.com/huggingface/evaluate/tree/main/metrics/exact_match)
```
predictions = Value("string", id="sequence")
references = Value("string", id="sequence")

```

[f1](https://github.com/huggingface/evaluate/tree/main/metrics/f1)
```
predictions = Sequence(Value("int32")
references = Sequence(Value("int32"))
```

[frugalscore](https://github.com/huggingface/evaluate/tree/main/metrics/frugalscore)
```
references = Value("string")
predictions = Value("string")
```

[gleu](https://github.com/huggingface/evaluate/tree/main/metrics/gleu)
```
predictions = Sequence(Value("string", id="token"), id="sequence")
references = Sequence(Sequence(Value("string", id="token"), id="sequence"), id="references")
```

[glue](https://github.com/huggingface/evaluate/tree/main/metrics/glue)
```
predictions = Value("int64" if self.config_name != "stsb" else "float32")
references = Value("int64" if self.config_name != "stsb" else "float32")
```
The type of input depends on the [GLUE](https://huggingface.co/datasets/glue) subset used.

[google_bleu](https://github.com/huggingface/evaluate/tree/main/metrics/google_bleu)
```
predictions = Sequence(Value("string", id="token"), id="sequence")
references = Sequence(Sequence(Value("string", id="token"), id="sequence"), id="references")
```

[indic_glue](https://github.com/huggingface/evaluate/tree/main/metrics/indic_glue)
```
predictions = Value("int64") if self.config_name != "cvit-mkb-clsr" else Sequence(Value("float32"))
references = Value("int64") if self.config_name != "cvit-mkb-clsr" else Sequence(Value("float32"))
```

[mae](https://github.com/huggingface/evaluate/tree/main/metrics/mae)
```
predictions = Value("float")
references = Value("float")
```
or if `multilist`:
```
predictions = Sequence(Value("float"))
references = Sequence(Value("float"))
```

[mahalanobis](https://github.com/huggingface/evaluate/tree/main/metrics/mahalanobis)
```
"X": Sequence(Value("float", id="sequence"), id="X")
reference_distribution = np.array(reference_distribution)
```
N.B. the names for `references` and `predictions` are different here -- maybe we should standardize? wdyt @lhoestq 

[matthews_correlation](https://github.com/huggingface/evaluate/tree/main/metrics/matthews_correlation)
```
predictions = Value("int32")
references = Value("int32")
```

[mauve](https://github.com/huggingface/evaluate/tree/main/metrics/mauve)
```
predictions = Value("string", id="sequence")
references = Value("string", id="sequence")
```

[mean_iou](https://github.com/huggingface/evaluate/tree/main/metrics/mean_iou)
```
predictions = Sequence(Sequence(Value("uint16")))
references = Sequence(Sequence(Value("uint16")))
```
What's a `unit16`? unicode? this is the only metric with a unicode restriction (so far).

[meteor](https://github.com/huggingface/evaluate/tree/main/metrics/meteor)
```
predictions = Value("string", id="sequence")
references = Value("string", id="sequence")
```

[mse](https://github.com/huggingface/evaluate/tree/main/metrics/mse)
```
predictions = Value("float")
references = Value("float")
```
or if `multilist`:
```
predictions = Sequence(Value("float"))
references = Sequence(Value("float")),
```

[pearsonr](https://github.com/huggingface/evaluate/tree/main/metrics/pearsonr)
```
references = Value("float")
predictions = Value("float")
```

[perplexity](https://github.com/huggingface/evaluate/tree/main/metrics/perplexity)
```
input_texts = Value("string")
```

[precision](https://github.com/huggingface/evaluate/tree/main/metrics/precision)
```
predictions = Value("int32")
references = Value("int32")
```
or if `multilist`:
```
predictions = Sequence(Value("int32"))
references = Sequence(Value("int32"))
```
[recall](https://github.com/huggingface/evaluate/tree/main/metrics/recall)
```
predictions = Value("int32")
references = Value("int32")
```
or if `multilist`:
```
predictions = Sequence(Value("int32"))
references = Sequence(Value("int32"))
```

[rouge](https://github.com/huggingface/evaluate/tree/main/metrics/rouge)
```
predictions = Value("string", id="sequence")
references = Value("string", id="sequence")
```

[sacrebleu](https://github.com/huggingface/evaluate/tree/main/metrics/sacrebleu)
```
predictions = Value("string", id="sequence")
references = Sequence(Value("string", id="sequence"), id="references")
```

[sari](https://github.com/huggingface/evaluate/tree/main/metrics/sari)
```
sources = Value("string", id="sequence")
predictions = Value("string", id="sequence")
references = Sequence(Value("string", id="sequence"), id="references")
```

[seqeval](https://github.com/huggingface/evaluate/tree/main/metrics/seqeval)
```
predictions = Sequence(Value("string", id="label"), id="sequence")
references = Sequence(Value("string", id="label"), id="sequence")
```
N.B. both `predictions` and `references` are in [IOB format](https://github.com/huggingface/datasets/tree/master/metrics/seqeval)

[spearmanr](https://github.com/huggingface/evaluate/tree/main/metrics/spearmanr)
```
predictions = Value("float")
references = Value("float")
```

[squad](https://github.com/huggingface/evaluate/tree/main/metrics/squad)
```
predictions = {"id": Value("string"), "prediction_text": Value("string")}
"references": {
"id": Value("string"),
"answers": features.Sequence(
    {
        "text": Value("string"),
        "answer_start": Value("int32"),
    }
)
```

[squad_v2](https://github.com/huggingface/evaluate/tree/main/metrics/squad_v2)
```
"predictions": {
    "id": Value("string"),
    "prediction_text": Value("string"),
    "no_answer_probability": Value("float32"),
}
"references": {
    "id": Value("string"),
    "answers": features.Sequence(
      {"text": Value("string"), "answer_start": Value("int32")}
                        ),
                    }
```
N.B. SQuAD and SQuAD v2. formats differ in the fact that v2 has the `'no_answer_probability'` tag in `predictions`.

[super_glue](https://github.com/huggingface/evaluate/tree/main/metrics/super_glue)
```
if self.config_name == "record":
        return {
            "predictions": {
                "idx": {
                    "passage": Value("int64"),
                    "query": Value("int64"),
                },
                "prediction_text": Value("string"),
            },
            "references": {
                "idx": {
                    "passage": Value("int64"),
                    "query": Value("int64"),
                },
                "answers": Sequence(datasets.Value("string")),
            },
        }
    elif self.config_name == "multirc":
        return {
            "predictions": {
                "idx": {
                    "answer": Value("int64"),
                    "paragraph": Value("int64"),
                    "question": Value("int64"),
                },
                "prediction": Value("int64"),
            },
            "references": Value("int64"),
        }
    else:
        return {
            "predictions": Value("int64"),
            "references": Value("int64"),
        }
```

[ter](https://github.com/huggingface/evaluate/tree/main/metrics/ter)
```
predictions = Value("string", id="sequence")
references = Sequence(Value("string", id="sequence"), id="references")
```

[wer](https://github.com/huggingface/evaluate/tree/main/metrics/wer)
```
predictions = Value("string", id="sequence"),
references = Value("string", id="sequence")
```

[wiki_split](https://github.com/huggingface/evaluate/tree/main/metrics/wiki_split)
```
predictions = Value("string", id="sequence")
references = Sequence(Value("string", id="sequence"), id="references")
```

[xnli](https://github.com/huggingface/evaluate/tree/main/metrics/xnli)
```
predictions = Value("int64" if self.config_name != "sts-b" else "float32")
references = Value("int64" if self.config_name != "sts-b" else "float32")
```

[xtreme_s](https://github.com/huggingface/evaluate/tree/main/metrics/xtreme_s)
```
pred_type = "int64" if self.config_name in ["fleurs-lang_id", "minds14"] else "string"
```
```
predictions = Value(pred_type)
references = Value(pred_type)
```
N.B. the input depends on the [XTREME-S dataset](https://huggingface.co/datasets/google/xtreme_s) selected 

## Proposing categories of metrics (WIP)

### Number-based metrics

**2 lists of numbers**: `accuracy`, `f1`, `glue`, `indic_glue`, `mae`, `mahalanobis`, `matthews_correlation`, `mse`, `pearsonr`, `precision`, `recall`, `spearmanr`, `super_glue`, `xnli`

**2 lists of lists of ints**: `mean_iou`

### Text-based metrics

**1 list of strings** : `perplexity`

**2 lists of strings** : `bertscore`, `bleurt`, `cer`, `competition_math`, `exact_match` (*maybe?*), `frugalscore`, `mauve`, `meteor`,  `rouge`, `wer`, 

**2 lists of lists of strings**:  `seqeval`

**2 lists of tokenized strings**:   `google_bleu` (maybe?) 
 
**3 lists of strings**: `comet`, `sari` 

**prediction list of strings, reference list of list of strings**: `chrf`, `sacrebleu`, `ter`, `wiki_split`.

**prediction list of tokenized strings, reference list of list of tokenized strings**: `bleu`

### Structured metrics 

**2 lists of dictionaries**: `cuad`, `squad`, `squad_v2`. 

### Other

**hybrid metrics (strings and ints)** : `xtreme_s`,  `code_eval`

**NER metrics (structured format)**: `coval`

Thanks a lot @sashavor to work on this! Regarding the metrics that take a tokenized list of strings maybe the following would be a viable step to allow for more standardisation:

- either pass the list of tokenized strings
- or pass the list of stings along with a tokenization function

That way we could better combine e.g. `GLEU` and `bertscore` as they would take the same predictions and references. What do you think? I think it would be great if all the text metrics were compatible. cc @lhoestq 

Shouldn't the tokenization function be part of the metric implementation itself ?

That can work! But currently there are metrics that take lists of tokenized sentences (or do `string.split()` somewhere in the code.

Ok, so for all the text-based metrics, I propose we have 1 mandatory input (`prediction`) and 2 optional ones (`references` and `sources`). 

- All 3 would be *lists of strings* (either a single string or multiple). 
- Tokenization would be handled within the metric implementation.   
- For those metrics that are implemented elsewhere (e.g. `bleu`), we will add a transform within the `compute()` function that will convert this format to the format needed by the function.
- For those metrics that we have implemented internally, we can make the input structure changes directly in the code.

@lvwerra what do you think? is this what you had in mind?

A few questions:
* About "sources", stupid question but what are those?
* What about non-text-based metrics?
* About tokenization: it could be that a specific metric has a specific tokenization scheme, or that if we allow the user to do the tokenization then different scores come out depending on how they tokenize? I guess we can solve this by allowing users to specify a tokenizer, and defining a default? Or what's best?
* About "metrics that are implemented elsewhere" - what does that mean? You mean when we have a dependency on some other library like sacrebleu?

Hi @douwekiela 

> * About "sources", stupid question but what are those?

As far as I know these are the source texts in translation tasks. 

> * What about non-text-based metrics?

There are already a few non-text-based metrics (e.g. the classification metrics `accuracy`, `recall` etc.). We aim to standardize the metrics that we currently have and then in a next step add new metrics and come up with standards as we go.

> * About tokenization: it could be that a specific metric has a specific tokenization scheme, or that if we allow the user to do the tokenization then different scores come out depending on how they tokenize? I guess we can solve this by allowing users to specify a tokenizer, and defining a default? Or what's best?

Yep, discussed this yesterday with @sashavor and that's the way we will probably go: have a default (whitespace) tokenizer for those and have an option to pass a custom tokenization function.

> * About "metrics that are implemented elsewhere" - what does that mean? You mean when we have a dependency on some other library like sacrebleu?

Yes, that's them. The idea is that we add a layer between passing the preds/refs to the metric and passing them on to external metric function and transform the inputs from our to their standard there.


@lhoestq we've been discussing standardazing the text based metrics a bit. Some of them currently take a text as reference and others a list of texts. The idea for standardization was to always use a list (even if it only has one element) for all text metrics such as `bleu`, `rouge` etc. What do you think?

How would you handle several references in rouge ? AFAIK the `rouge_score` tool only takes one reference.

`predictions` would be a list, and `references` would be a list of lists, does that make sense?

In the following I try to explain why I think it can be nice to have a difference between BLEU and ROUGE inputs. I try to keep in mind that what matters is:
- it should be easy to use metrics together when they are compatible
- the dev experience should be easy, and in particular users expect to be able to know in advance what will happen when they call a function

When evaluating text generation capabilities, there are two types of datasets:
1. datasets with a single reference per sample;
2. datasets with possibly several references per sample.

BLEU can be used for 1 and 2, and ROUGE only for 1.

Therefore depending on the dataset, you can or cannot use BLEU and ROUGE together.

It would be nice to be able to tell if you can use BLEU and ROUGE together in advance given a certain dataset. This will be helpful to automate evaluation.

If you have a dataset with several references per sample (case 2.), you suggested to always require lists of lists, which for me doesn't allow to know if advance if BLEU and ROUGE can be used together. It's even the contrary: it looks like they can be used together _a priori_ since they require the same input, while they actually don't work together.

In particular I find that running this code should fail and return an error:
```python
rouge = load_metrics("rouge")

# should fail
rouge.compute(references=[["hello there", "hey there"]], predictions=["hello there"])

# should also fail since it supposes that there could be several references
rouge.compute(references=[["hello there"]], predictions=["hello there"])
```

Because of this, I would keep the inputs different to account for those two cases, and define the metrics this way:
- allow two input types for BLEU:
  - one single reference per sample (1.) -> List[str]
  - possibly several references per sample (2.) -> List[List[str]]
- allow only one input type for ROUGE:
  - one single reference per sample (1.) -> List[str]

This way:
- ROUGE fails when it should fail
- BLEU and ROUGE still be used together for 1.

In the code it could look like
```python
>>> rouge = load_metric("rouge")
>>> bleu = load_metric("bleu")
>>> allowed_inputs_in_common = set(rouge.allowed_inputs) & set(bleu.allowed_inputs)
>>> if len(allowed_inputs_in_common) > 0:
...     print(f"ROUGE and BLEU can be used together if you provide this input:\n{allowed_inputs_in_common[0]}")
ROUGE and BLEU can be used together if you provide this input:
predictions: Value("string")
references: Value("string")
```

The code above would be helpful for automatic evaluation, since you can tell in advance if you can use certain metrics together on a given dataset.

Let me know what you think, and feel free to comment ideas and suggestions.

Thanks @lhoestq, that makes a lot of sense, I hadn't taken into account the case where a dataset has multiple references and ROUGE would unexpectedly fail. 

A comment on your code example to test for compatibility: I guess we would need a bit more logic as we would need to check if all required inputs are present in `allowed_inputs_in_common`. E.g. if we did that on the two metrics how they are now (incompatible in references) the output would be:

```Python
ROUGE and BLEU can be used together if you provide this input:
predictions: Value("string")
```

From a practical perspective, is it currently possible to set a feature key to multiple possible types? E.g. the references of BLUE could be of type `Sequence(Value("string"))` or `Value("string")`, right? Also, is there already a way to test if an Python object is of a certain Feature type? Asking this since we would need to some transformation if e.g. only one sample is passed to BLUE to make sure it's compatible BLUE code. 

Maybe the role of `bleu.allowed_inputs` wasn't clear: it would be a list of all possible input types combinations. So it would be a list of two elements for BLEU:
```python
[
    Features({"predictions": Value("string"), "references": Value("string")}),
    Features({"predictions": Value("string"), "references": Sequence(Value("string"))}),
]
```

Two metrics would be compatible if they have at least one Features in common.

> From a practical perspective, is it currently possible to set a feature key to multiple possible types? E.g. the references of BLUE could be of type Sequence(Value("string")) or Value("string"), right? 

We could have a list of Features as I described above, or define Union or something like that ?

> Also, is there already a way to test if an Python object is of a certain Feature type? Asking this since we would need to some transformation if e.g. only one sample is passed to BLUE to make sure it's compatible BLUE code.

I don't think so but it would be easy to add. For BLEU in particular we could add the check in the BLEU code directly, we don't need to verify the types for every single metric in a first step

Bouncing off of @lhoestq 's comment above, could we have a generic `input_checker()` function that compares the structure of the inputs to metrics and those defined in their `Features`? 
That way we can establish whether a metric can be used at all (e.g. if it needs `int`s and the user provides `str`s, it's not possible).
And maybe doing this *before* calling `compute()` can be more efficient since : 
1) sometimes calling `compute` involves loading a model 
2) sometimes the error messages thrown by the `compute` function aren't very clear, e.g. `ValueError` or `TypeError` isn't **super** informative.

Thoughts, @lhoestq and @lvwerra ?

One thing to do would be to add something like

```
if not self._check(inputs):
  raise ValueError(f"Incompatible metric inputs: {inputs} [... etc ...]")
```

somewhere around here: https://github.com/huggingface/evaluate/blob/main/src/evaluate/metric.py#L418? And then let every metric `_check` for itself whether the inputs are in the correct format?

`compute()` already does basic type checking before calling the underlying `_compute()` that may load a model ;)

It's done here:

https://github.com/huggingface/evaluate/blob/df3d20712df202b586f73cf45a66b65652e45d5b/src/evaluate/metric.py#L420

which calls the ArrowWriter that does type checking:

https://github.com/huggingface/evaluate/blob/df3d20712df202b586f73cf45a66b65652e45d5b/src/evaluate/metric.py#L468

@lhoestq that's great, however, I encountered with strings there are some implicit conversions going on which is not ideal for input testing. E.g. if a field is of type `"references": Value("string")` but if you pass `references=[1]` it will be converted to `["1"]` internally. Similarly if a list of strings is past to that field: `references =[["a", "b"]]` then become `["['a', 'b']"]` internally. Is this by design or is this something we could fix? 

This is bad for metrics like `rouge` which don't take a list of reference but it can still be passed without an error.

Indeed currently it tries to cast the type here:

https://github.com/huggingface/evaluate/blob/df3d20712df202b586f73cf45a66b65652e45d5b/src/evaluate/metric.py#L464

You can try removing this line, it should fix this issue :)

@lhoestq indeed the data is cast there, however, when deactivating it is still cast later on by `pyarrow.array` (see [docs](https://arrow.apache.org/docs/python/generated/pyarrow.array.html)). The main issue is string types as everything can be cast to a string. 

Ideally we would have a mechanism that checks all types but this could be quite extensive. So maybe to start we could just check if something that should be a string is really a string. What do you think?