Repository: EleutherAI/lm-evaluation-harness
Issue: Error when evaluating leaderboard_math
Root cause: External Dependency Breakage

Hi!

I am using the lm eval framework to run the `leaderboard_math_hard` task but I get this error:


```
  File "/pscratch/sd/p/prajwal/conda/yalis-env-latest/lib/python3.11/site-packages/math_verify/grader.py", line 284, in sympy_solve_and_compare
    for g, p in zip(sorted(solved_gold), sorted(solved_pred))
                    ^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'dict' and 'dict'
```

Do I have to keep something in mind when using this task. Also, I am using a custom evaluation backend (like vLLM) which works fine for other tasks. 

Hi! I can't reproduce this. Do you have the full stack trace?

Hi! I get the same error when using `vllm` to run `leaderboard_math_hard` with Qwen2.5-3B-Instruct, but there are results shown below. I'm not sure if potential issues exist. 


```
2025-05-30:16:22:22 INFO     [evaluator:559] Running generate_until requests
Processed prompts: 100%|███████████████████████████████████████████████| 1324/1324 [02:36<00:00,  8.44it/s, est. speed input: 6141.08 toks/s, output: 7403.49 toks/s]
Running generate_until requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 1324/1324 [02:37<00:00,  8.42it/s]
2025-05-30:16:25:10 ERROR    [math_verify.grader:813] Error during comparison
Traceback (most recent call last):
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 809, in compare_single_extraction_wrapper
    return compare_single_extraction(g, t)
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/utils.py", line 51, in wrapper
    return func(*args, **kwargs)
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 789, in compare_single_extraction
    return sympy_expr_eq(
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 667, in sympy_expr_eq
    return sympy_compare_relational(gold, pred, float_rounding, numeric_precision)
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 344, in sympy_compare_relational
    if sympy_solve_and_compare(gold, pred, float_rounding, numeric_precision):
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 284, in sympy_solve_and_compare
    for g, p in zip(sorted(solved_gold), sorted(solved_pred))
TypeError: '<' not supported between instances of 'dict' and 'dict'
2025-05-30:16:25:12 ERROR    [math_verify.grader:813] Error during comparison
Traceback (most recent call last):
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 809, in compare_single_extraction_wrapper
    return compare_single_extraction(g, t)
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/utils.py", line 51, in wrapper
    return func(*args, **kwargs)
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 789, in compare_single_extraction
    return sympy_expr_eq(
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 667, in sympy_expr_eq
    return sympy_compare_relational(gold, pred, float_rounding, numeric_precision)
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 344, in sympy_compare_relational
    if sympy_solve_and_compare(gold, pred, float_rounding, numeric_precision):
  File "/home/nfs05/anaconda3/envs/xch-trl/lib/python3.10/site-packages/math_verify/grader.py", line 284, in sympy_solve_and_compare
    for g, p in zip(sorted(solved_gold), sorted(solved_pred))
TypeError: '<' not supported between instances of 'dict' and 'dict'
fatal: detected dubious ownership in repository at '/home/nfs05/xiangch/lm-evaluation-harness'
To add an exception for this directory, call:

        git config --global --add safe.directory /home/nfs05/xiangch/lm-evaluation-harness
[rank0]:[W530 16:25:40.135536777 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-05-30:16:25:43 INFO     [loggers.evaluation_tracker:272] Output path not provided, skipping saving results aggregated
vllm (pretrained=/home/nfs02/model/Qwen2.5/Qwen2.5-3B-Instruct,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=1,max_model_len=8192), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto
|                    Tasks                    |Version|Filter|n-shot|       Metric       |   |Value |   |Stderr|
|---------------------------------------------|------:|------|-----:|--------------------|---|-----:|---|-----:|
|leaderboard_math_hard                        |       |none  |      |exact_match         |↑  |0.2674|±  |0.0113|
| - leaderboard_math_algebra_hard             |      3|none  |     4|exact_match         |↑  |0.4853|±  |0.0286|
|                                             |       |none  |     4|exact_match_original|↑  |0.2117|±  |0.0234|
| - leaderboard_math_counting_and_prob_hard   |      3|none  |     4|exact_match         |↑  |0.2114|±  |0.0370|
|                                             |       |none  |     4|exact_match_original|↑  |0.0813|±  |0.0247|
| - leaderboard_math_geometry_hard            |      3|none  |     4|exact_match         |↑  |0.0985|±  |0.0260|
|                                             |       |none  |     4|exact_match_original|↑  |0.0379|±  |0.0167|
| - leaderboard_math_intermediate_algebra_hard|      3|none  |     4|exact_match         |↑  |0.0929|±  |0.0174|
|                                             |       |none  |     4|exact_match_original|↑  |0.0143|±  |0.0071|
| - leaderboard_math_num_theory_hard          |      3|none  |     4|exact_match         |↑  |0.2468|±  |0.0349|
|                                             |       |none  |     4|exact_match_original|↑  |0.0519|±  |0.0179|
| - leaderboard_math_prealgebra_hard          |      3|none  |     4|exact_match         |↑  |0.4611|±  |0.0360|
|                                             |       |none  |     4|exact_match_original|↑  |0.2280|±  |0.0303|
| - leaderboard_math_precalculus_hard         |      3|none  |     4|exact_match         |↑  |0.0963|±  |0.0255|
|                                             |       |none  |     4|exact_match_original|↑  |0.0148|±  |0.0104|

|       Groups        |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|---------------------|-------|------|------|-----------|---|-----:|---|-----:|
|leaderboard_math_hard|       |none  |      |exact_match|↑  |0.2674|±  |0.0113|
```