Repository: EleutherAI/lm-evaluation-harness
Issue: Can't find the dataset lighteval/MATH-Hard in the huggingface
Root cause: External Dependency Breakage

when I run the code to get the results of the tasks in the open llm leaderboard v2, I can't find lighteval/MATH-Hard in the huggingface, how should I solve this problem?
Thanks！

Have the same issue

Hi! yeah, it doesn't seem to be up anymore. cc: @clefourrier @NathanHB 

Did one find a temporary solution for this with an exactly similar dataset? I reckon this dataset would probably be uploaded to HF multiple times, so forking lm_eval or monkey patching with the reference to that one would be feasible for now. How long does it usually take before these kind of problems are resolved? Thanks in advance.

As it's the same dataset as MATH, but they only use level-5 problems, you could change the dataset_path [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/bb098f13b05e361f01a5afe7b612779ce362b3f2/lm_eval/tasks/leaderboard/math/_template_yaml#L1) to `EleutherAI/hendrycks_math` and pipe a filter in the `process_docs` function (`return dataset.filter(lambda x: x["level"] == "Level 5").map(...)`), in the utils script (same folder).

Also add `"level": "Level 5",` field to all the samples in `list_fewshot_samples`

Thank you very much, will fork it and implement a custom solution!

Thank you for the answer！I will try it.

hello?

There was a DMCA takedown notice on the MATH dataset, information was quite publicised at the time, more info here: https://huggingface.co/datasets/hendrycks/competition_math

Yeah but - the reference to the dataset should be removed or changed to fix the broken tasks

https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval%2Ftasks%2Fleaderboard%2Fmath%2F_template_yaml

The tasks can't be left broken indefinitely

Can we replace the missing dataset with this one: <https://huggingface.co/datasets/DigitalLearningGmbH/MATH-lighteval>? It seems to be a drop-in replacement.

It also looks like others are doing the same: <https://github.com/stanfordnlp/dspy/issues/3358> 

This should be fixed by #2719 and can be closed now.

I got different results after changing to `DigitalLearningGmbH/MATH-lighteval`

lighteval/MATH-Hard:
```
hf (pretrained=huggingface/cache/hub/models--microsoft--Phi-3.5-mini-instruct/snapshots/af0dfb8029e8a74545d0736d30cb6b58d2f0f3f0/,dtype=bfloat16,attn_implementation=flash_attention_2), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|                    Tasks                    |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|---------------------------------------------|-------|------|-----:|-----------|---|-----:|---|-----:|
|leaderboard_math_hard                        |    N/A|none  |      |exact_match|↑  |0.1586|±  |0.0094|
| - leaderboard_math_algebra_hard             |      2|none  |     4|exact_match|↑  |0.2834|±  |0.0258|
| - leaderboard_math_counting_and_prob_hard   |      2|none  |     4|exact_match|↑  |0.1707|±  |0.0341|
| - leaderboard_math_geometry_hard            |      2|none  |     4|exact_match|↑  |0.0606|±  |0.0208|
| - leaderboard_math_intermediate_algebra_hard|      2|none  |     4|exact_match|↑  |0.0250|±  |0.0093|
| - leaderboard_math_num_theory_hard          |      2|none  |     4|exact_match|↑  |0.1039|±  |0.0247|
| - leaderboard_math_prealgebra_hard          |      2|none  |     4|exact_match|↑  |0.3523|±  |0.0345|
| - leaderboard_math_precalculus_hard         |      2|none  |     4|exact_match|↑  |0.0222|±  |0.0127|

|       Groups        |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|---------------------|-------|------|------|-----------|---|-----:|---|-----:|
|leaderboard_math_hard|    N/A|none  |      |exact_match|↑  |0.1586|±  |0.0094|
```

DigitalLearningGmbH/MATH-lighteval:

```
hf (pretrained=huggingface/cache/hub/models--microsoft--Phi-3.5-mini-instruct/snapshots/af0dfb8029e8a74545d0736d30cb6b58d2f0f3f0/,dtype=bfloat16,attn_implementation=flash_attention_2), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|                    Tasks                    |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|---------------------------------------------|-------|------|-----:|-----------|---|-----:|---|-----:|
|leaderboard_math_hard                        |    N/A|none  |      |exact_match|↑  |0.3768|±  |0.0065|
| - leaderboard_math_algebra_hard             |      2|none  |     4|exact_match|↑  |0.5307|±  |0.0145|
| - leaderboard_math_counting_and_prob_hard   |      2|none  |     4|exact_match|↑  |0.3671|±  |0.0222|
| - leaderboard_math_geometry_hard            |      2|none  |     4|exact_match|↑  |0.3090|±  |0.0211|
| - leaderboard_math_intermediate_algebra_hard|      2|none  |     4|exact_match|↑  |0.1661|±  |0.0124|
| - leaderboard_math_num_theory_hard          |      2|none  |     4|exact_match|↑  |0.2778|±  |0.0193|
| - leaderboard_math_prealgebra_hard          |      2|none  |     4|exact_match|↑  |0.5924|±  |0.0167|
| - leaderboard_math_precalculus_hard         |      2|none  |     4|exact_match|↑  |0.2125|±  |0.0175|

|       Groups        |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|---------------------|-------|------|------|-----------|---|-----:|---|-----:|
|leaderboard_math_hard|    N/A|none  |      |exact_match|↑  |0.3768|±  |0.0065|
```

Also, I noticed that the samples are larger (333 to 1252) after changing to the new repo.