Repository: EleutherAI/lm-evaluation-harness
Issue: Possible bug in TriviaQA: gold answer not always included in candidates
Root cause: Algorithmic Error

When looking through model predictions I noticed that some predictions in TriviaQA are not counted as correct even though they match the gold answer exactly.

This can be easily checked by adding the following assert to the `process_results` function before the return statement:

```
gold = doc["answer"]["value"].lower().translate(str.maketrans('', '', string.punctuation))
assert gold in list_of_candidates, f"Error: {gold} was not found in list of candidates {list_of_candidates}"
```

https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/triviaqa.py#L91

This results in the following assertion error:

```
AssertionError: Error: montreal was not found in list of candidates ['city of montreal', 'city of montréal', 'funtreal', 'montracal', 'montrea', 'montrela', 'montreol', 'montréal', 'motnreal', 'sister cities of montreal', 'team990', 'the weather in montreal', 'the weather in montréal', 'tiohtiáke', 'unlocodecamtr', 'ville de montreal', 'ville de montréal']
```

Unless I'm missing something I think that since the exact match score is computed simply as `float(continuation in list_of_candidates)` this gold string should be in the list of candidates, no? In fact it seems to be contained in the list before the `_remove_prefixes` function is called. For instance in the above case the raw list of aliases is the below and does in fact contain the ground truth answer "Montreal":

```
['City of Montreal', 'City of Montréal', 'Funtreal', 'MontrA(c)al', 'Montrea', 'Montreal', 'Montreal (QC)', 'Montreal Skyline', "Montreal's", 'Montreal, Canada', 'Montreal, PQ', 'Montreal, QB', 'Montreal, QC', 'Montreal, Qc', 'Montreal, Qc, Canada', 'Montreal, Que.', 'Montreal, Quebec', 'Montreal, Quebec, Canada', 'Montreal, Québec', 'Montreal, qc', 'Montreal, quebec', 'Montrealer', 'Montrealers', 'Montrela', 'Montreol', 'Montréal', 'Montréal (QC)', 'Montréal (Québec)', "Montréal's", 'Montréal, Canada', 'Montréal, PQ', 'Montréal, QB', 'Montréal, QC', 'Montréal, Quebec', 'Montréal, Quebec (Province)', 'Montréal, Québec', 'Montréal, Québec (Province)', 'Montréal, Québec, Canada', 'Motnreal', 'Sister Cities of Montreal', 'Team990', 'The weather in Montreal', 'The weather in Montréal', 'Tiohtiá:ke', 'UN/LOCODE:CAMTR', 'Ville de Montreal', 'Ville de Montréal']
```

I have run the evaluation with Llama-1-7B both with TQA as is and when manually appending the normalized gold string to the list of candidates and results change as follows:

| Llama-1-7B  | 0-shot | 1-shot | 5-shot |
|-------------|--------|--------|--------|
| TQA         | 47.3   | 55.3   | 57.9   |
| TQA w/ gold | 48.9   | 57.5   | 60.2   |

Should the `_remove_prefixes` be called on the list of candidates here at all?

Updated results when removing `_remove_prefixes` entirely:

| Llama-1-7B  | 0-shot | 1-shot | 5-shot |
|-------------|--------|--------|--------|
| TQA         | 47.3   | 55.3   | 57.9   |
| w/ gold | 48.9   | 57.5   | 60.2   |
| no `_remove_prefixes` | 50.0   | 58.3   | 60.9   |

You are correct. Using `_remove_prefixes` is only valid for log-likelihood evaluations and not exact match. I've opened a PR removing it.