Repository: EleutherAI/lm-evaluation-harness
Issue: KeyError on some metrics from huggingface/evaluate
Root cause: Interface Contract Mismatch

### Context
I am currently utilizing the `lm-evaluation-harness` in conjunction with the metrics provided by `huggingface/evaluate`. Specifically, I am using [`bertscore`](https://github.com/huggingface/evaluate/tree/main/metrics/bertscore). This metric returns a dictionary with the following format:

```python
{
  'precision': [0.7380737066268921, 0.5584042072296143],
  'recall': [0.7380737066268921, 0.5889028906822205],
  'f1': [0.7380737066268921, 0.5732481479644775], 
  'hashcode': 'bert-base-uncased_L5_no-idf_version=0.3.10(hug_trans=4.10.3)'
}
```

My task configuration is as follows (note that this is just an example to reproduce the error):
```yaml
group: bertscore
task: bertscore_eval
dataset_path: truthful_qa
dataset_name: generation
validation_split: validation
doc_to_text: "{{question}}"
doc_to_target: "{{best_answer}}"
metric_list:
  - metric: bertscore
    aggregation: mean
    higher_is_better: true
    lang: en
```

### Issue Encountered
When running this configuration, I encounter a `KeyError: 'bertscore'`.

I've been looking a bit into the reason, and I think this is because the metric name (`bertscore`) is not present in the returned dictionary, and therefore `result_score` does not contain it in `task.py`. The relevant code is located at:

https://github.com/EleutherAI/lm-evaluation-harness/blob/ada4a31d69fd58ce794b60944900bd1471fbad3e/lm_eval/api/task.py#L1181

Is there a potential workaround to this issue, or am I configuring my task incorrectly?


Thanks!

I've been looking a bit more into it. The issue seems to be with how `self._metric_fn_list` is initialized. The metrics list is [initialized with the metric name](https://github.com/EleutherAI/lm-evaluation-harness/blob/ada4a31d69fd58ce794b60944900bd1471fbad3e/lm_eval/api/task.py#L554) (ie `bertscore`), and it is this key the one used to fetch the results, but in the case of this metric, the dict does not contain that key as explained above.

For a workaround until this issue is resolved, here's how I managed to make it work:

First, the YAML file that defines the metrics:

```yaml
group: bertscore
task: bertscore_eval
dataset_path: truthful_qa
dataset_name: generation
validation_split: validation
doc_to_text: "{{question}}"
doc_to_target: "{{best_answer}}"
metric_list:
  - metric: !function utils.bertscore_f1
    aggregation: mean
    higher_is_better: true
    lang: en
  - metric: !function utils.bertscore_precision
    aggregation: mean
    higher_is_better: true
    lang: en
  - metric: !function utils.bertscore_recall
    aggregation: mean
    higher_is_better: true
    lang: en
```

Then, the `utils.py` file that defines the metrics looks like:

```py
from evaluate import load

def bertscore_f1(references, predictions, **kwargs):
    """Computes the F1 score of the BERTScore metric.

    Args:
        references: A list of reference strings.
        predictions: A list of predicted strings.
        **kwargs: Additional keyword arguments.

    Returns:
        The F1 score of the BERTScore metric.
    """
    bertscore = load("bertscore")
    return bertscore.compute(predictions=predictions, references=references, **kwargs)["f1"][0]

def bertscore_precision(references, predictions, **kwargs):
    """Computes the precision of the BERTScore metric.

    Args:
        references: A list of reference strings.
        predictions: A list of predicted strings.
        **kwargs: Additional keyword arguments.

    Returns:
        The precision of the BERTScore metric.
    """
    bertscore = load("bertscore")
    return bertscore.compute(predictions=predictions, references=references, **kwargs)["precision"][0]

def bertscore_recall(references, predictions, **kwargs):
    """Computes the recall of the BERTScore metric.

    Args:
        references: A list of reference strings.
        predictions: A list of predicted strings.
        **kwargs: Additional keyword arguments.

    Returns:
        The recall of the BERTScore metric.
    """
    bertscore = load("bertscore")
    return bertscore.compute(predictions=predictions, references=references, **kwargs)["recall"][0]
```

This is not ideal because we're computing the same metrics three times and just fetching one of the results, but it works for now.

Could we simply use a cache to avoid doing the computation 3 times? Any downside?

```python
from functools import cache

from evaluate import load


@cache
def bertscore_compute(predictions, references, **kwargs):
    bertscore = load("bertscore")
    return bertscore.compute(predictions=predictions, references=references, **kwargs)

def bertscore_f1(references, predictions, **kwargs):
    """Computes the F1 score of the BERTScore metric.

    Args:
        references: A list of reference strings.
        predictions: A list of predicted strings.
        **kwargs: Additional keyword arguments.

    Returns:
        The F1 score of the BERTScore metric.
    """
    return bertscore_compute(predictions=predictions, references=references, **kwargs)["f1"][0]

def bertscore_precision(references, predictions, **kwargs):
    """Computes the precision of the BERTScore metric.

    Args:
        references: A list of reference strings.
        predictions: A list of predicted strings.
        **kwargs: Additional keyword arguments.

    Returns:
        The precision of the BERTScore metric.
    """
    return bertscore_compute(predictions=predictions, references=references, **kwargs)["precision"][0]

def bertscore_recall(references, predictions, **kwargs):
    """Computes the recall of the BERTScore metric.

    Args:
        references: A list of reference strings.
        predictions: A list of predicted strings.
        **kwargs: Additional keyword arguments.

    Returns:
        The recall of the BERTScore metric.
    """
    return bertscore_compute(predictions=predictions, references=references, **kwargs)["recall"][0]
```