Repository: EleutherAI/lm-evaluation-harness
Issue: wrong regular expression for exact match scoring
Root cause: Validation Gap

Hi, I would like to report a problem about Natural Question evaluation, especially about `regexes_to_ignore`. 

## TL;DR
I believe we should change `regexes_to_ignore` in [nq_open.yaml](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/nq_open/nq_open.yaml)
from
```
- "\ban|a|the\b"
```
to
```
- "\\b(?:The |the |An |A |The |a |an )"
```

And I think this modification should also be considered for other tasks using `exact_match` such as math reasoning.

## Why
Firstly I git clone the latest version(ada4a31d69fd58ce794b60944900bd1471fbad3e) and ran the 0-shot `meta-llama/Llama-2-7b-hf` evaluation on `nq_open` dataset with this command:
```bash
lm-eval --model hf --batch_size auto --task nq_open --model_args pretrained=meta-llama/Llama-2-7b-hf \
--output output/llama2-7b-0-shot-no-escape --log_samples
```

When manually checking the output, I found there are some weird `exact_match` score calculation, for example this one:
```json
  {
    "doc_id": 160,
    "doc": {
      "question": "who is the biggest selling female group of all time",
      "answer": [
        "Spice Girls"
      ]
    },
    "target": [
      "Spice Girls"
    ],
    "arguments": [
      [
        "Answer these questions:\n\nQ: who is the biggest selling female group of all time?\nA:",
        {
          "until": [
            "\n",
            ".",
            ","
          ],
          "do_sample": false,
          "temperature": 0.0
        }
      ]
    ],
    "resps": [
      [
        "The Spice Girls"
      ]
    ],
    "filtered_resps": [
      "The Spice Girls"
    ],
    "exact_match": 0.0
  },
```

## Adding An|A|The
Although we have set `ignore_case=True` and regular expression for filter out the as in the [config file](https://github.com/EleutherAI/lm-evaluation-harness/blob/ada4a31d69fd58ce794b60944900bd1471fbad3e/lm_eval/tasks/nq_open/nq_open.yaml#L27-L30C21), it still have `"exact_match": 0.0`.

After digging into the code, I found the problem lies in the `evaluate` package. As stated [here](https://github.com/huggingface/evaluate/tree/main/metrics/exact_match), it would do `regex filter` **before** lowercasing. So I believe we should also add `An|A|The` into the `regexes_to_ignore` term. 
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/38466901/bf827422-5abe-409e-8c9d-2faec530a12a)

## Adding whitespace
And since there is also no `white_space_fixing` operation as in the [KILT](https://github.com/facebookresearch/KILT/blob/2664322b9a994be686e4c3a9e8f75a0b70927f22/kilt/eval_downstream.py#L53-L69), it simply uses `re.sub`, as shown [here](https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/metrics/exact_match/exact_match.py#L112-L115), it would cause problems such as `"the Animal"` to `" Animal"`, which is incompatible with `"Animal"`.

## Adding extra backslash for `\b`
The last problem is that `lm-eval` would read the config using `yaml.load` method, and directly sending this regex string to third party evaluation toolkit (huggingface/evaluate). However, the evaluate function needs a raw string. So we should add other `\` to `\b`. 

BTW, I think this would also affect other tasks requiring `exact_match` score.

After doing all the modification above, I got the right outcome:
```json
 {
    "doc_id": 160,
    "doc": {
      "question": "who is the biggest selling female group of all time",
      "answer": [
        "Spice Girls"
      ]
    },
    "target": [
      "Spice Girls"
    ],
    "arguments": [
      [
        "Answer these questions:\n\nQ: who is the biggest selling female group of all time?\nA:",
        {
          "until": [
            "\n",
            ".",
            ","
          ],
          "do_sample": false,
          "temperature": 0.0
        }
      ]
    ],
    "resps": [
      [
        "The Spice Girls"
      ]
    ],
    "filtered_resps": [
      "The Spice Girls"
    ],
    "exact_match": 1.0
  },
```


Thanks for investigating this and for the thorough reporting! 

Would you be willing to PR this fix? I can proceed to look at whether it affects other tasks.