Repository: EleutherAI/lm-evaluation-harness
Issue: Help Understanding Exact Match Scores (Or Maybe Error with Exact Match Calculation)?
Root cause: Algorithmic Error

I'm running evaluations using LM Eval Harness of GSM8K and Hendrycks MATH. I noticed that the exact match scores are sensible for `exact_match_flexible_extract`, but my `exact_match_strict_extract` and `exact_match` are all 0.

I'm looking at the generated samples and I'm confused whether exact match is being computed correctly? Here are two examples from Hendrycks math:

# Example 1

```
{"doc_id": 0, "doc": {"problem": "How many vertical asymptotes does the graph of $y=\\frac{2}{x^2+x-6}$ have?", "level": "Level 3", "type": "Algebra", "solution": "The denominator of the rational function factors into $x^2+x-6=(x-2)(x+3)$. Since the numerator is always nonzero, there is a vertical asymptote whenever the denominator is $0$, which occurs for $x = 2$ and $x = -3$.  Therefore, the graph has $\\boxed{2}$ vertical asymptotes.", "answer": "2"}, "target": "2", "arguments": {"gen_args_0": {"arg_0": "Problem: How many vertical asymptotes does the graph of $y=\\frac{2}{x^2+x-6}$ have?\nAnswer:", "arg_1": {"until": ["Problem:"], "do_sample": false, "temperature": 0.0}}}, "resps": [[" Factoring the denominator, we get $y=\\frac{2}{(x+3)(x-2)}$. There is a vertical asymptote at each value of $x$ that makes the denominator equal to zero, so there are $\\boxed{2}$ vertical asymptotes."]], "filtered_resps": [" Factoring the denominator, we get $y=\\frac{2}{(x+3)(x-2)}$. There is a vertical asymptote at each value of $x$ that makes the denominator equal to zero, so there are $\\boxed{2}$ vertical asymptotes."], "filter": "none", "metrics": ["exact_match"], "doc_hash": "1e9250fca0c032f52b78594043db079d286f01be879bcd5f26a2e6f57cc7387a", "prompt_hash": "ec43418a8f2c727844d1586ebe095733b745c61116ee729c7590232d98f00c1f", "target_hash": "d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35", "exact_match": 0}
```

The solution says the boxed answer is $\\boxed{2}$ and the (filtered) response gives a boxed answer as $\\boxed{2}$ but exact match is 0.

# Example 2

```
{"doc_id": 1, "doc": {"problem": "What is the positive difference between $120\\%$ of 30 and $130\\%$ of 20?", "level": "Level 1", "type": "Algebra", "solution": "One hundred twenty percent of 30 is $120\\cdot30\\cdot\\frac{1}{100}=36$, and $130\\%$ of 20 is $ 130\\cdot 20\\cdot\\frac{1}{100}=26$.  The difference between 36 and 26 is $\\boxed{10}$.", "answer": "10"}, "target": "10", "arguments": {"gen_args_0": {"arg_0": "Problem: What is the positive difference between $120\\%$ of 30 and $130\\%$ of 20?\nAnswer:", "arg_1": {"until": ["Problem:"], "do_sample": false, "temperature": 0.0}}}, "resps": [[" $120\\%$ of 30 is $1.2(30)=36$, and $130\\%$ of 20 is $1.3(20)=26$.  The difference is $36-26=\\boxed{10}$."]], "filtered_resps": [" $120\\%$ of 30 is $1.2(30)=36$, and $130\\%$ of 20 is $1.3(20)=26$.  The difference is $36-26=\\boxed{10}$."], "filter": "none", "metrics": ["exact_match"], "doc_hash": "513db6eb55dd1e7ea3f84b74b9327154200d40122a03fd4858e068cd19b21b4a", "prompt_hash": "ffe4f01dcdeea0f2821829d77880f76482bc567bfd4b2470864f025d153e0aef", "target_hash": "4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5", "exact_match": 0}
```

The solution says the boxed answer is $\\boxed{10}$ and the (filtered) response gives a boxed answer as $\\boxed{10}$ but exact match is 0.

Am I misunderstanding how exact match functions? Or is this an error?

Potentially relevant issues:

https://github.com/EleutherAI/lm-evaluation-harness/issues/2905

https://github.com/EleutherAI/lm-evaluation-harness/issues/3150

I'm reading through this code: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hendrycks_math/utils.py

And stepping through it with my first example, we have that the solution is:

`"The denominator of the rational function factors into $x^2+x-6=(x-2)(x+3)$. Since the numerator is always nonzero, there is a vertical asymptote whenever the denominator is $0$, which occurs for $x = 2$ and $x = -3$.  Therefore, the graph has $\\boxed{2}$ vertical asymptotes."`

And the response is:

`" Factoring the denominator, we get $y=\\frac{2}{(x+3)(x-2)}$. There is a vertical asymptote at each value of $x$ that makes the denominator equal to zero, so there are $\\boxed{2}$ vertical asymptotes."`

This [line](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hendrycks_math/utils.py#L24) extracts the `answer` as:

`'y=\\frac{2}{(x+3)(x-2)}$. There is a vertical asymptote at each value of $x$ that makes the denominator equal to zero, so there are $\\boxed{2}'`

Consequently, `is_equiv(answer, remove_boxed(last_boxed_only_string(doc["solution"])))` returns `False`.

This seems incorrect, but maybe I'm misunderstanding the code or the metric?

Follow up question: Why is the `retval` not:

```
is_equiv(
        remove_boxed(last_boxed_only_string(results[0])),
        remove_boxed(last_boxed_only_string(doc["solution"])),
    )
```

? It seems like one would want to extract the last box, remove the box and then compare for both solution and for answer, no?

@baberabb can you please advise or point me towards someone who can?

Hi! apologies for the tardiness. Am running quite behind these days. tl;dr use `minerva_math`

Some history:  The prompt formatting and answer extraction used in `hendrycks_math` is taken almost verbatim from the original [repo](https://github.com/hendrycks/math/blob/main/modeling/math_equivalence.py), and is _very_ particular. It only [tries](https://github.com/EleutherAI/lm-evaluation-harness/blob/7f04db12d2f8e7a99a0830d99eb78130e1ba2122/lm_eval/tasks/hendrycks_math/utils.py#L20) to extract the `$` sign and if not falls back to the entire generated string (iirc without fewshots they expected the model to be a finetuned gpt-2, otherwise almost always with fewshots).

`minerva_math` improved on it by using a more structured fewshot prompt and also using sympy equivalence for the math expressions, and that worked until models started getting annealed on math datasets and wouldn't necessary follow the fewshot format (particularly Qwen models. good detail [here](https://huggingface.co/blog/math_verify_leaderboard) - we also use it now).

(`gsm8k` also has a similar progression, but simpler as it's all integers. Initially we extracted `"#### (\\-?[0-9\\.\\,]+)"`, but then added `flexible_extract` to [fallback](https://github.com/EleutherAI/lm-evaluation-harness/blob/7f04db12d2f8e7a99a0830d99eb78130e1ba2122/lm_eval/tasks/gsm8k/gsm8k.yaml#L42) to the first number in the string)

With instruct tuned models we can just bypass all this by having them return the answer in the [appropriate](https://github.com/EleutherAI/lm-evaluation-harness/blob/7f04db12d2f8e7a99a0830d99eb78130e1ba2122/lm_eval/tasks/gsm8k/gsm8k-cot-llama.yaml#L4) format.

For your 0-shot use-case, I think you'll have to modify [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7f04db12d2f8e7a99a0830d99eb78130e1ba2122/lm_eval/tasks/minerva_math/utils.py#L210) to return the original string, so that it gets passed to math_verify's `parse` method (it uses some fallback extraction heuristics - we should really make this the default after some testing). 

Also generally feel free to ping me on discord! I'm much more punctual there :)

Thanks for the amazing response! And no worries about the delay - I know you're helping carry this superb library :)

I switched to `minerva_math` and I'm still puzzled over the scoring.

As quick context, I'm SFTing exactly on the test set to study test set contamination, so I'm expecting to hit nearly 100% with enough SFT epochs. I'm finding that `math_verify` maxes out at around 64% (for Qwen 2.5 7B + Minerva Math).

Here's the command I'm running

```
mem_scoring_vs_sampling_env/bin/lm_eval \
    --model vllm \
    --model_args pretrained=RylanSchaeffer/mem_model_Qwen2.5-3B_dataset_minerva_math_epochs_32_seed_0,dtype=auto,max_model_len=2048,max_num_seqs=2048 \
    --batch_size auto \
    --tasks minerva_math \
    --num_fewshot 0 \
    --limit 10 \
    --log_samples \
    --output_path ./lm-eval-output-vllm/ \
    --gen_kwargs temperature=0.0,do_sample=False \
    --seed 0
```

Here is an output with score `0`:

`
{"doc_id": 2, "doc": {"problem": "Find $x$ such that $\\lceil x \\rceil + x = \\dfrac{23}{7}$. Express $x$ as a common fraction.", "level": "Level 4", "type": "Algebra", "solution": "First, we note that $x$ must be positive, since otherwise $\\lceil x \\rceil + x$ is nonpositive. Next, we know that the decimal part of $x$ must be $\\dfrac{2}{7}$. We write $x$ as $n+\\dfrac{2}{7}$, where $n$ is the greatest integer less than $x.$ Then, $\\lceil x \\rceil = n + 1.$ Therefore, we can write $\\lceil x \\rceil + x$ as $n+1+n+\\dfrac{2}{7}=\\dfrac{23}{7}$. Solving, we get $n=1$. Therefore, the only value $x$ that satisfies the equation is $1+\\dfrac{2}{7}=\\boxed{\\dfrac{9}{7}}$.", "answer": "\\dfrac{9}{7}"}, "target": "\\dfrac{9}{7}", "arguments": {"gen_args_0": {"arg_0": "Problem:\nFind $x$ such that $\\lceil x \\rceil + x = \\dfrac{23}{7}$. Express $x$ as a common fraction.\n\nSolution:", "arg_1": {"until": ["Problem:"], "do_sample": false, "temperature": 0.0}}}, "resps": [[" First, we note that $x$ must be positive, since otherwise $\\lceil x \\rceil + x$ is nonpositive. Next, we know that the decimal part of $x$ must be $\\dfrac{2}{7}$. We write $x$ as $n+\\dfrac{2}{7}$, where $n$ is the greatest integer less than $x.$ Then, $\\lceil x \\rceil = n + 1.$ Therefore, we can write $\\lceil x \\rceil + x$ as $n+1+n+\\dfrac{2}{7}=\\dfrac{23}{7}$. Solving, we get $n=1$. Therefore, the only value $x$ that satisfies the equation is $1+\\dfrac{2}{7}=\\boxed{\\dfrac{9}{7}}$."]], "filtered_resps": [" First, we note that $x$ must be positive, since otherwise $\\lceil x \\rceil + x$ is nonpositive. Next, we know that the decimal part of $x$ must be $\\dfrac{2}{7}$. We write $x$ as $n+\\dfrac{2}{7}$, where $n$ is the greatest integer less than $x.$ Then, $\\lceil x \\rceil = n + 1.$ Therefore, we can write $\\lceil x \\rceil + x$ as $n+1+n+\\dfrac{2}{7}=\\dfrac{23}{7}$. Solving, we get $n=1$. Therefore, the only value $x$ that satisfies the equation is $1+\\dfrac{2}{7}=\\boxed{\\dfrac{9}{7}}$."], "filter": "none", "metrics": ["exact_match", "math_verify"], "doc_hash": "2adb8f157a5260ee542d4ef6fbea5e3fbb1cb6ebe31d74383af75c937e429092", "prompt_hash": "d7fcfd59997e47fc6f25d5caf474cf4482440ab1c00c32b424f5d6a8abf806ec", "target_hash": "cbc2be11eb3645540e7931633a351eabc973fa9788173630175aa13a4ac874de", "exact_match": 0, "math_verify": 0}
`

Looking closely, the solution is:

`"First, we note that $x$ must be positive, since otherwise $\\lceil x \\rceil + x$ is nonpositive. Next, we know that the decimal part of $x$ must be $\\dfrac{2}{7}$. We write $x$ as $n+\\dfrac{2}{7}$, where $n$ is the greatest integer less than $x.$ Then, $\\lceil x \\rceil = n + 1.$ Therefore, we can write $\\lceil x \\rceil + x$ as $n+1+n+\\dfrac{2}{7}=\\dfrac{23}{7}$. Solving, we get $n=1$. Therefore, the only value $x$ that satisfies the equation is $1+\\dfrac{2}{7}=\\boxed{\\dfrac{9}{7}}$."`

The response is:

`"First, we note that $x$ must be positive, since otherwise $\\lceil x \\rceil + x$ is nonpositive. Next, we know that the decimal part of $x$ must be $\\dfrac{2}{7}$. We write $x$ as $n+\\dfrac{2}{7}$, where $n$ is the greatest integer less than $x.$ Then, $\\lceil x \\rceil = n + 1.$ Therefore, we can write $\\lceil x \\rceil + x$ as $n+1+n+\\dfrac{2}{7}=\\dfrac{23}{7}$. Solving, we get $n=1$. Therefore, the only value $x$ that satisfies the equation is $1+\\dfrac{2}{7}=\\boxed{\\dfrac{9}{7}}$."`

**These two strings are verbatim the same, and yet both Exact Match and Math Verify are `0`.**

I'm currently using the Eval Harness installed via `pip install` at commit `bc811365ef3917a15da6aac9b7feafb4ba0fb42f`

```
commit bc811365ef3917a15da6aac9b7feafb4ba0fb42f (grafted, HEAD -> main, origin/main, origin/HEAD)
Author: Cyrus Leung <cyrus.tl.leung@gmail.com>
Date:   Sun Aug 3 01:46:20 2025 +0800

    Update vLLM compatibility (#3024)
    
    * Update vLLM compatibility
    
    Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
    
    * add TokensPrompt to all generate calls
    
    ---------
    
    Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
    Co-authored-by: Baber <baber@hey.com>
```

> Looking closely, the solution is:
> 
> `"First, we note that $x$ must be positive, since otherwise $\\lceil x \\rceil + x$ is nonpositive. Next, we know that the decimal part of $x$ must be $\\dfrac{2}{7}$. We write $x$ as $n+\\dfrac{2}{7}$, where $n$ is the greatest integer less than $x.$ Then, $\\lceil x \\rceil = n + 1.$ Therefore, we can write $\\lceil x \\rceil + x$ as $n+1+n+\\dfrac{2}{7}=\\dfrac{23}{7}$. Solving, we get $n=1$. Therefore, the only value $x$ that satisfies the equation is $1+\\dfrac{2}{7}=\\boxed{\\dfrac{9}{7}}$."`
> 
> The response is:
> 
> `"First, we note that $x$ must be positive, since otherwise $\\lceil x \\rceil + x$ is nonpositive. Next, we know that the decimal part of $x$ must be $\\dfrac{2}{7}$. We write $x$ as $n+\\dfrac{2}{7}$, where $n$ is the greatest integer less than $x.$ Then, $\\lceil x \\rceil = n + 1.$ Therefore, we can write $\\lceil x \\rceil + x$ as $n+1+n+\\dfrac{2}{7}=\\dfrac{23}{7}$. Solving, we get $n=1$. Therefore, the only value $x$ that satisfies the equation is $1+\\dfrac{2}{7}=\\boxed{\\dfrac{9}{7}}$."`
> 
> **These two strings are verbatim the same, and yet both Exact Match and Math Verify are `0`.**

Ok looks like it has trouble parsing without `boxed` so shouldn't call `remove_boxed` [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/3bc7cc8a72c66bac8d5b830cb3ccec9a5f691b12/lm_eval/tasks/minerva_math/utils.py#L39)

```python
from math_verify import parse, verify

parse("\\dfrac{9}{7}") # as doc["answer"] = remove_boxed(last_boxed_only_string(doc["solution"])
# []
# but this works
parse("\\boxed{\\dfrac{9}{7}}")
# [9/7, '\\frac{9}{7}']
```

or maybe just [pass](https://github.com/EleutherAI/lm-evaluation-harness/blob/3bc7cc8a72c66bac8d5b830cb3ccec9a5f691b12/lm_eval/tasks/minerva_math/utils.py#L86) the full solution:
```python
parse(doc["solution"])
# [9/7, '\\frac{9}{7}']
```

@baberabb and I have been chatting privately on Discord. The TLDR is that (1) use Minerva MATH and (2) a new PR (currently the branch `mathfix`) will be opened soon.

A detail I missed:

If using `vllm`, I thought setting the `max_model_len` was sufficient, like so:

`--model_args pretrained={model_hf_path},dtype=auto,max_model_len=4096,max_num_seqs=2048`

It is not! You also need to set the `max_gen_toks` under `gen_kwargs`, since otherwise it defaults to `256`

`--gen_kwargs temperature={temperature},do_sample={do_sample},max_gen_toks=2048 \`

For the record, @baberabb has merged a PR to fix this issue: https://github.com/EleutherAI/lm-evaluation-harness/pull/3259