Repository: EleutherAI/lm-evaluation-harness
Issue: MATH is_equiv() function can not parser some ground truth 
Root cause: Validation Gap

Hi! We noticed that the is_equiv() function in utils.py from both the [minerva_math](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/minerva_math/utils.py#L144) and [leaderboard/math](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/minerva_math/utils.py#L144) failed to parse some of the ground truth from[ MATH-Hard](https://huggingface.co/datasets/lighteval/MATH-Hard) dataset, causing the scores for those affected questions marked as incorrect.  We modified the [process_results](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/leaderboard/math/utils.py#L67) to the following code, which asks is_equiv(ground_truth,ground_truth), and only get 94.03% correctness instead of 100%.  We hope it can be fixed soon.  Thanks!
```
def process_results(doc: dict, results: List[str]) -> Dict[str, int]:
    candidates = results[0]
    import logging
    eval_logger.setLevel(logging.DEBUG)
    unnormalized_answer = get_unnormalized_answer(candidates)
    answer = normalize_final_answer(unnormalized_answer)

    if is_equiv(doc["answer"], doc["answer"]):
        retval = 1
    else:
        retval = 0

    results = {
        "exact_match": retval,
    }
    return results
```
the detail logs looks like this: 
```
Running generate_until requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 169/169 [09:32<00:00,  3.39s/it]
^[[A^[[A^[[A2024-07-10:09:31:05,132 DEBUG    [utils.py:158] couldn't parse one of \le(\frac{4}{3},-\frac{1}{3}\right) or \le(\frac{4}{3},-\frac{1}{3}\right)
2024-07-10:09:31:05,160 DEBUG    [utils.py:158] couldn't parse one of \le(\frac{7}{45},\frac{4}{45}\right) or \le(\frac{7}{45},\frac{4}{45}\right)
2024-07-10:09:31:05,163 DEBUG    [utils.py:158] couldn't parse one of (2,-1) or (2,-1)
2024-07-10:09:31:05,178 DEBUG    [utils.py:158] couldn't parse one of (-7,10) or (-7,10)
2024-07-10:09:31:05,208 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}1\4\3\end{pmatrix} or \begin{pmatrix}1\4\3\end{pmatrix}
2024-07-10:09:31:05,210 DEBUG    [utils.py:158] couldn't parse one of \le(3,\frac{11\pi}{8},\frac{4\pi}{5}\right) or \le(3,\frac{11\pi}{8},\frac{4\pi}{5}\right)
2024-07-10:09:31:05,219 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}-4\\-3\end{pmatrix} or \begin{pmatrix}-4\\-3\end{pmatrix}
2024-07-10:09:31:05,224 DEBUG    [utils.py:158] couldn't parse one of (2,-1) or (2,-1)
2024-07-10:09:31:05,286 DEBUG    [utils.py:158] couldn't parse one of \le(\frac{11}{15},\frac{11}{60},\frac{1}{12}\right) or \le(\frac{11}{15},\frac{11}{60},\frac{1}{12}\right)
2024-07-10:09:31:05,287 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}1&2\-3&-5\end{pmatrix} or \begin{pmatrix}1&2\-3&-5\end{pmatrix}
2024-07-10:09:31:05,291 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}5/3\5/3\4/3\end{pmatrix} or \begin{pmatrix}5/3\5/3\4/3\end{pmatrix}
2024-07-10:09:31:05,310 DEBUG    [utils.py:158] couldn't parse one of (4,24) or (4,24)
2024-07-10:09:31:05,315 DEBUG    [utils.py:158] couldn't parse one of (3,-3,1) or (3,-3,1)
2024-07-10:09:31:05,320 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}23/8\7/4\end{pmatrix} or \begin{pmatrix}23/8\7/4\end{pmatrix}
2024-07-10:09:31:05,321 DEBUG    [utils.py:158] couldn't parse one of \le(-\frac{5}{3},\frac{2}{3},\frac{19}{3}\right) or \le(-\frac{5}{3},\frac{2}{3},\frac{19}{3}\right)
2024-07-10:09:31:05,340 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}-4/9\1/9\8/9\end{pmatrix} or \begin{pmatrix}-4/9\1/9\8/9\end{pmatrix}
2024-07-10:09:31:05,356 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}-6\9\end{pmatrix} or \begin{pmatrix}-6\9\end{pmatrix}
2024-07-10:09:31:05,358 DEBUG    [utils.py:158] couldn't parse one of (7,21,35) or (7,21,35)
2024-07-10:09:31:05,360 DEBUG    [utils.py:158] couldn't parse one of 75\% or 75\%
2024-07-10:09:31:05,367 DEBUG    [utils.py:158] couldn't parse one of \le(-\frac{8}{3},\frac{4}{3},-\frac{1}{3}\right) or \le(-\frac{8}{3},\frac{4}{3},-\frac{1}{3}\right)
2024-07-10:09:31:05,378 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}1/50&7/50\7/50&49/50\end{pmatrix} or \begin{pmatrix}1/50&7/50\7/50&49/50\end{pmatrix}
2024-07-10:09:31:05,381 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}1\2\-3\end{pmatrix} or \begin{pmatrix}1\2\-3\end{pmatrix}
2024-07-10:09:31:05,381 DEBUG    [utils.py:158] couldn't parse one of (0,1) or (0,1)
2024-07-10:09:31:05,406 DEBUG    [utils.py:158] couldn't parse one of \begin{pmatrix}-5/13\12/13\end{pmatrix} or \begin{pmatrix}-5/13\12/13\end{pmatrix}
2024-07-10:09:31:05,409 DEBUG    [utils.py:158] couldn't parse one of 30\% or 30\%
2024-07-10:09:31:05,456 DEBUG    [utils.py:158] couldn't parse one of 28\% or 28\%
2024-07-10:09:31:05,526 DEBUG    [utils.py:158] couldn't parse one of \le[-\frac{1}{2},\frac{1}{2}\right] or \le[-\frac{1}{2},\frac{1}{2}\right]
2024-07-10:09:31:05,535 DEBUG    [utils.py:158] couldn't parse one of (3,-3,1) or (3,-3,1)
2024-07-10:09:31:05,563 DEBUG    [utils.py:158] couldn't parse one of \{1\pm\sqrt{5},-2\} or \{1\pm\sqrt{5},-2\}
2024-07-10:09:31:05,572 DEBUG    [utils.py:158] couldn't parse one of \le[\frac{3}{2},2\right) or \le[\frac{3}{2},2\right)
2024-07-10:09:31:05,574 DEBUG    [utils.py:158] couldn't parse one of 01 or 01
2024-07-10:09:31:05,581 DEBUG    [utils.py:158] couldn't parse one of [-4,0) or [-4,0)
2024-07-10:09:31:05,592 DEBUG    [utils.py:158] couldn't parse one of (-\iny,4) or (-\iny,4)
2024-07-10:09:31:05,622 DEBUG    [utils.py:158] couldn't parse one of (3,4] or (3,4]
2024-07-10:09:31:05,666 DEBUG    [utils.py:158] couldn't parse one of (6,31,-1) or (6,31,-1)
2024-07-10:09:31:05,667 DEBUG    [utils.py:158] couldn't parse one of (-5,\sqrt{26}] or (-5,\sqrt{26}]
2024-07-10:09:31:05,678 DEBUG    [utils.py:158] couldn't parse one of (27,\iny) or (27,\iny)
2024-07-10:09:31:05,683 DEBUG    [utils.py:158] couldn't parse one of 01 or 01
2024-07-10:09:31:05,717 DEBUG    [utils.py:158] couldn't parse one of (-5,-4) or (-5,-4)
2024-07-10:09:31:05,719 DEBUG    [utils.py:158] couldn't parse one of (-\iny,-2]\cup[-1,\iny) or (-\iny,-2]\cup[-1,\iny)
2024-07-10:09:31:05,734 DEBUG    [utils.py:158] couldn't parse one of (6,5) or (6,5)
2024-07-10:09:31:05,736 DEBUG    [utils.py:158] couldn't parse one of (2,\iny) or (2,\iny)
2024-07-10:09:31:05,736 DEBUG    [utils.py:158] couldn't parse one of \le(\frac{1}{2},\frac{\sqrt{7}}{2}\right) or \le(\frac{1}{2},\frac{\sqrt{7}}{2}\right)
2024-07-10:09:31:05,743 DEBUG    [utils.py:158] couldn't parse one of (2,5) or (2,5)
2024-07-10:09:31:05,749 DEBUG    [utils.py:158] couldn't parse one of \frac{\}{p}i4+2-\sqrt{2} or \frac{\}{p}i4+2-\sqrt{2}
2024-07-10:09:31:05,750 DEBUG    [utils.py:158] couldn't parse one of (-a,0) or (-a,0)
2024-07-10:09:31:05,808 DEBUG    [utils.py:158] couldn't parse one of (\tfrac{27}{8},\tfrac{15}{8}) or (\tfrac{27}{8},\tfrac{15}{8})
2024-07-10:09:31:05,819 DEBUG    [utils.py:158] couldn't parse one of 7.2\% or 7.2\%
2024-07-10:09:31:05,827 DEBUG    [utils.py:158] couldn't parse one of \frac{1}{{}72} or \frac{1}{{}72}
2024-07-10:09:31:05,859 DEBUG    [utils.py:158] couldn't parse one of (-\iny,-2)\cup(-2,3)\cup(3,\iny) or (-\iny,-2)\cup(-2,3)\cup(3,\iny)
2024-07-10:09:31:05,861 DEBUG    [utils.py:158] couldn't parse one of -\frac{1}{{}2x} or -\frac{1}{{}2x}
2024-07-10:09:31:05,901 DEBUG    [utils.py:158] couldn't parse one of [6,\iny) or [6,\iny)
2024-07-10:09:31:05,944 DEBUG    [utils.py:158] couldn't parse one of (7,6) or (7,6)
2024-07-10:09:31:05,957 DEBUG    [utils.py:158] couldn't parse one of (4,5) or (4,5)
2024-07-10:09:31:05,959 DEBUG    [utils.py:158] couldn't parse one of (0,1] or (0,1]
2024-07-10:09:31:05,967 DEBUG    [utils.py:158] couldn't parse one of (1,-2) or (1,-2)
2024-07-10:09:31:05,982 DEBUG    [utils.py:158] couldn't parse one of (-6,10) or (-6,10)
2024-07-10:09:31:05,990 DEBUG    [utils.py:158] couldn't parse one of (-\iny,-5]\cup[5,\iny) or (-\iny,-5]\cup[5,\iny)
2024-07-10:09:31:05,996 DEBUG    [utils.py:158] couldn't parse one of (-\iny,-4)\cup(-4,6)\cup(6,\iny) or (-\iny,-4)\cup(-4,6)\cup(6,\iny)
2024-07-10:09:31:05,997 DEBUG    [utils.py:158] couldn't parse one of \le[\frac{1}{2},\frac{4}{3}\right] or \le[\frac{1}{2},\frac{4}{3}\right]
2024-07-10:09:31:06,000 DEBUG    [utils.py:158] couldn't parse one of (-\iny,0)\cup(0,\iny) or (-\iny,0)\cup(0,\iny)
2024-07-10:09:31:06,030 DEBUG    [utils.py:158] couldn't parse one of (-1,8) or (-1,8)
2024-07-10:09:31:06,032 DEBUG    [utils.py:158] couldn't parse one of [-10,4] or [-10,4]
2024-07-10:09:31:06,033 DEBUG    [utils.py:158] couldn't parse one of (-7,-2) or (-7,-2)
2024-07-10:09:31:06,046 DEBUG    [utils.py:158] couldn't parse one of \le(\frac{3}{5},\frac{8}{3}\right] or \le(\frac{3}{5},\frac{8}{3}\right]
2024-07-10:09:31:06,050 DEBUG    [utils.py:158] couldn't parse one of (-\iny,2)\cup(3,\iny) or (-\iny,2)\cup(3,\iny)
2024-07-10:09:31:06,056 DEBUG    [utils.py:158] couldn't parse one of (2,12) or (2,12)
2024-07-10:09:31:06,059 DEBUG    [utils.py:158] couldn't parse one of [0,1) or [0,1)
2024-07-10:09:31:06,070 DEBUG    [utils.py:158] couldn't parse one of (-\iny,-7)\cup(-7,3)\cup(3,\iny) or (-\iny,-7)\cup(-7,3)\cup(3,\iny)
2024-07-10:09:31:06,085 DEBUG    [utils.py:158] couldn't parse one of (-\iny,-14)\cup(-3,\iny) or (-\iny,-14)\cup(-3,\iny)
2024-07-10:09:31:06,091 DEBUG    [utils.py:158] couldn't parse one of [1,4] or [1,4]
2024-07-10:09:31:06,104 DEBUG    [utils.py:158] couldn't parse one of (-\iny,-8)\cup(8,\iny) or (-\iny,-8)\cup(8,\iny)
2024-07-10:09:31:06,110 DEBUG    [utils.py:158] couldn't parse one of (-\sqrt{3},\sqrt{3}) or (-\sqrt{3},\sqrt{3})
2024-07-10:09:31:06,114 DEBUG    [utils.py:158] couldn't parse one of [0,\iny) or [0,\iny)
2024-07-10:09:31:06,134 DEBUG    [utils.py:158] couldn't parse one of [-3,2] or [-3,2]
2024-07-10:09:31:06,135 DEBUG    [utils.py:158] couldn't parse one of [1,9] or [1,9]
2024-07-10:09:31:06,145 DEBUG    [utils.py:158] couldn't parse one of (-\iny,0) or (-\iny,0)
2024-07-10:09:31:06,181 DEBUG    [utils.py:158] couldn't parse one of 100\% or 100\%
2024-07-10:09:31:11,199 INFO     [evaluation_tracker.py:182] Saving results aggregated
2024-07-10:09:31:11,200 INFO     [evaluation_tracker.py:258] Saving per-sample results for: leaderboard_math_algebra_hard
2024-07-10:09:31:11,217 INFO     [evaluation_tracker.py:258] Saving per-sample results for: leaderboard_math_counting_and_prob_hard
2024-07-10:09:31:11,224 INFO     [evaluation_tracker.py:258] Saving per-sample results for: leaderboard_math_geometry_hard
2024-07-10:09:31:11,231 INFO     [evaluation_tracker.py:258] Saving per-sample results for: leaderboard_math_intermediate_algebra_hard
2024-07-10:09:31:11,247 INFO     [evaluation_tracker.py:258] Saving per-sample results for: leaderboard_math_num_theory_hard
2024-07-10:09:31:11,255 INFO     [evaluation_tracker.py:258] Saving per-sample results for: leaderboard_math_prealgebra_hard
2024-07-10:09:31:11,265 INFO     [evaluation_tracker.py:258] Saving per-sample results for: leaderboard_math_precalculus_hard
hf (pretrained=meta-llama/Meta-Llama-3-8B-Instruct,dtype=bfloat16), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 4
|                    Tasks                    |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|---------------------------------------------|-------|------|-----:|-----------|---|-----:|---|-----:|
| - leaderboard_math_algebra_hard             |      1|none  |     4|exact_match|↑  |0.9023|±  |0.0170|
| - leaderboard_math_counting_and_prob_hard   |      1|none  |     4|exact_match|↑  |1.0000|±  |0.0000|
| - leaderboard_math_geometry_hard            |      1|none  |     4|exact_match|↑  |0.9545|±  |0.0182|
|leaderboard_math_hard                        |N/A    |none  |     4|exact_match|↑  |0.9403|±  |0.0064|
| - leaderboard_math_intermediate_algebra_hard|      1|none  |     4|exact_match|↑  |0.9393|±  |0.0143|
| - leaderboard_math_num_theory_hard          |      1|none  |     4|exact_match|↑  |1.0000|±  |0.0000|
| - leaderboard_math_prealgebra_hard          |      1|none  |     4|exact_match|↑  |0.9845|±  |0.0089|
| - leaderboard_math_precalculus_hard         |      1|none  |     4|exact_match|↑  |0.8296|±  |0.0325|

|       Groups        |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|---------------------|-------|------|-----:|-----------|---|-----:|---|-----:|
|leaderboard_math_hard|N/A    |none  |     4|exact_match|↑  |0.9403|±  |0.0064|
```

May have been fixed by [this PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/2475/files#diff-48f08ae868e5cb00d7d7c307ae04a5dd6e6d7586fff9fcdbeb82780378ac5eca)