Repository: EleutherAI/lm-evaluation-harness
Issue: Filter not extracting choice selection correctly
Root cause: Validation Gap

I am just starting trying to use lm_eval and am trying to run an evaluation with the bbh benchmark.

I am noticing the problem however where this is what the results look like where the period has not been removed from the response meaning that it counts them as incorrect.

I have tried looking through the source code and it seems that `MultiChoiceRegexFilter` that is the base for 'get-answers'.

<details>

<summary>An example from the Movie recommendations showing how the filtered resps is wrong.</summary>

> {'doc_id': 2,
 'doc': {'input': "Find a movie similar to Schindler's List, Braveheart, The Silence of the Lambs, Tombstone:\nOptions:\n(A) Orlando\n(B) Guilty of Romance\n(C) Forrest Gump\n(D) All the Real Girls",
  'target': '(C)'},
 'target': '(C)',
 'arguments': [("Recommend movies similar to the given list of movies.\n\nQ: Find a movie similar to Star Wars Episode IV - A New Hope, Indiana Jones and the Last Crusade, Star Wars Episode V - The Empire Strikes Back, The Big Lebowski:\nOptions:\n(A) Tetsuo\n(B) the Ironman\n(C) The Princess Bride\n(D) The Barkley Marathons The Race That Eats Its Young\n(E) Bug\nA: Let's think step by step.\n Let's think step by step.\n- Star Wars Episode IV - A New Hope (action, adventure, fantasy; 1977)\n- Indiana Jones and the Last Crusade (action, adventure; 1989)\n- Star Wars Episode V - The Empire Strikes Back (action, adventure, fantasy; 1980)\n- The Big Lebowski (action, drama, comedy; 1998)\nThese are all famous classic American movies produced before 2000. Amongst all the options, the only movie similar to these ones seems to be The Princess Bride (1987). So the answer is (C).\n\nQ: Find a movie similar to Twister, The Silence of the Lambs, Independence Day, Braveheart:\nOptions:\n(A) They Shoot Horses\n(B) Don't They\n(C) Forrest Gump\n(D) The Salton Sea\n(E) Extreme Days\nA: Let's think step by step.\n Let's think step by step.\n- Twister (action, adventure, thriller; 1996)\n- The Silence of the Lambs (crime, drama, thriller; 1991)\n- Independence Day (action, science-fiction, drama; 1996)\n- Braveheart (biography, drama, epic; 1995)\nThese are all famous Hollywood movies produced around the 1990s. Amongst all the options, the only movie similar to these ones seems to be Forrest Gump (comedy, drama, romance; 1994). So the answer is (C).\n\nQ: Find a movie similar to Minority Report, Total Recall, Inside Out, Forrest Gump:\nOptions:\n(A) Phenomena\n(B) Lilting\n(C) Catwoman\n(D) Edge of Tomorrow\nA: Let's think step by step.\n Let's think step by step.\n- Minority Report (action, crime, mystery; 2002)\n- Total Recall (action, adventure, science-fiction; 2012)\n- Inside Out (animation, family, comedy; 2015)\n- Forrest Gump (comedy, drama, romance; 1994)\nThese are all famous movies produced in the past few decades.Amongst all the options, the only movie similar to these ones seems to be Edge of Tomorrow (action, adventure, crime, mystery; 2014), as it is also a science-fiction movie and features Tom Cruise. So the answer is (D).\n\nQ: Find a movie similar to Schindler's List, Braveheart, The Silence of the Lambs, Tombstone:\nOptions:\n(A) Orlando\n(B) Guilty of Romance\n(C) Forrest Gump\n(D) All the Real Girls\nA: Let's think step by step.\n",
   {'max_gen_toks': 1024,
    'until': ['
', 'Q', '\n\n'],
    'do_sample': False,
    'temperature': 0.0})],
 'resps': [[" Let's think step by step.\n- Schindler's List (biography, drama, history; 1993)\n- Braveheart (biography, drama, epic; 1995)\n- The Silence of the Lambs (crime, drama, thriller; 1991)\n- Tombstone (action, drama, western; 1993)\nThese are all famous Hollywood movies produced in the 1990s. Amongst all the options, the only movie similar to these ones seems to be Forrest Gump (comedy, drama, romance; 1994). So the answer is (C). "]],
 'filtered_resps': ['(C).'],
 'filter': 'get-answer',
 'metrics': ['exact_match'],
 'doc_hash': 'a3d847cade4b5a6f37b1b77d0cbed1e060e4017858e4e1502b08791dabd2503e',
 'prompt_hash': 'eadef3a2d2d0865a4751c98110caf447ec9712c5b9194705e8bd70ec01a41032',
 'target_hash': 'ab0531c9d89473e8c5af60d4621a5d5bfaa6a5b1f1ebf3cc69e20814fee801cc',
 'exact_match': np.float64(0.0)}

</details>

<details>

<summary>Code needed to replicate</summary>

```python
bbh = tasks.get_task_dict("bbh")
movie_recommendations = list(bbh.values())[0]["bbh_cot_fewshot_movie_recommendation"]
eval_model = lm_eval.models.huggingface.HFLM(pretrained=models["unsloth/Llama-3.2-3B-Instruct"]["model"], device = device)
results =  evaluator.simple_evaluate(model = eval_model,
                            tasks =[movie_recommendations], limit = 0.025, batch_size= 24)

def results_to_df(results):
    """
    Take the results from each of the tasks and convert them into a dataframe.
    """
    all_results = [list(res.values()) for res in results['results'].values()]
    df = pd.DataFrame(all_results, columns=["name","accuracy", "accuracy_std"])
    return df.sort_values(by="accuracy", ascending=True).reset_index(drop=True)

results_to_df(results)
def inspect_task(results, index):
    samples = results['samples']

    all_samples = [sample for task_samples in samples.values() for sample in task_samples]
    sample = all_samples[index]
    display(sample)

    print(f"""
Prompt:
{sample['doc']['input']}
Expected answer:
{sample['doc']['target']}
Answer {sample['filtered_resps'][0]}:
{sample['resps'][0][0]}
"""
    )

inspect_task(results, 2)
```

</details>



Hi! It actually uses the [RegexFilter](https://github.com/EleutherAI/lm-evaluation-harness/blob/2a41c02e7a43819ce86c272eb47f313706451bb6/lm_eval/filters/extraction.py#L10) and there's an open [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/2013) to make the regex extraction more permissive that slipped my notice. I'll try merging that this week after some testing, but it seems (mostly) reasonable to me. I think setting [ignore_punctuation: true](https://github.com/EleutherAI/lm-evaluation-harness/blob/2a41c02e7a43819ce86c272eb47f313706451bb6/lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml#L10) might also be appropriate, as well as adding the [remove_whitespace](https://github.com/EleutherAI/lm-evaluation-harness/blob/2a41c02e7a43819ce86c272eb47f313706451bb6/lm_eval/filters/extraction.py#L69) filter, e.g.
```yaml
filter_list:
  - name: "get-answer"
    filter:
      - function: remove_whitespace
      - function: "regex"
        regex_pattern: "(?<=the answer is )(.*)(?=.)"
      - function: "take_first"
```
cc: @lintangsutawika 

@1jamesthompson1 How do you get the json like example result? I mean the "{'doc_id': 2, xxx}".

Hi @AlvinChanCG, bit late with this one. But the dict results are actually returned directly from the simple_evaluate function.

```python
bbh = tasks.get_task_dict("bbh")
movie_recommendations = list(bbh.values())[0]["bbh_cot_fewshot_movie_recommendation"]
results =  evaluator.simple_evaluate(model = eval_model,
                            tasks =[movie_recommendations], batch_size=48)
results["samples"]["bbh_cot_fewshot_movie_recommendation"]
```
> [{'doc_id': 0,
  'doc': {'input': 'Find a movie similar to Batman, The Mask, The Fugitive, Pretty Woman:\nOptions:\n(A) The Front Page\n(B) Maelstrom\n(C) The Lion King\n(D) Lamerica',
   'target': '(C)'},
  'target': '(C)',
  'arguments': [("Recommend movies similar to the given lis...\n\nQ: Find a movie similar to Batman, The Mask, The Fugitive, Pretty Woman:\nOptions:\n(A) The Front Page\n(B) Maelstrom\n(C) The Lion King\n(D) Lamerica\nA: Let's think step by step.\n",
    {'max_gen_toks': 1024,
     'until': ['
', 'Q', '\n\n'],
     'do_sample': False,
     'temperature': 0.0})],
  'resps': [[" Let's think step ...r is (A)."]],
  'filtered_resps': ['(A)'],
  'filter': 'get-answer',
  'metrics': ['exact_match'],
  'doc_hash': '677fdc232129be92030e3846838256fc5702d56a6a2321d9b6e0de00c37415e2',
  'prompt_hash': '9a716f0090cbe1ef67912fa0e96d97d70a7a033facfdf66834ba963b18c5e856',
  'target_hash': 'ab0531c9d89473e8c5af60d4621a5d5bfaa6a5b1f1ebf3cc69e20814fee801cc',
  'exact_match': np.float64(0.0)},
 {'doc_id': 1,
  'doc': {'input': 'Find a movie similar to The Sixth Sense, The Matrix, Forrest Gump, The Shawshank Redemption:\nOptions:\n(A) Street Fighter II The Animated Movie\n(B) The Sheltering Sky\n(C) The Boy Who Could Fly\n(D) Terminator 2 Judgment Day',
   'target': '(D)'},
  'target': '(D)',
  'arguments': [("Recommend ...ep by step.\n",
    {'max_gen_toks': 1024,
     'until': ['
', 'Q', '\n\n'],
     'do_sample': False,
...
  'metrics': ['exact_match'],
  'doc_hash': '4d984e191bec6086a1e3484e3bf35b8a16a17477a30b19fa0b67df9bf9ca22a5',
  'prompt_hash': '1cda5db3ef3aad17de474a15d37a9e28d97b81f977f2e87c88fee75febeb8d6b',
  'target_hash': 'bf97bdceb17f2739335afb287573b4a3d50f25b57916b12cab5829ee8cd768c1',
  'exact_match': np.float64(0.0)}]

Where "bbh_cot_fewshot_movie_recommendation" is the exact task name.