Repository: EleutherAI/lm-evaluation-harness
Issue: Some LongBench scores on Llama-3.1-8B-Instruct are far off
Root cause: Configuration Error

I've highly appreciated that longbench has been added as a dataset recently. I was trying to reproduce baselines for my own work and found that some scores were pretty far off of results I'd seen in other papers. Is there something wrong with the way the benchmarks are being run? [`results.json` file here](https://github.com/user-attachments/files/19917869/results.json).

```
|            Tasks             |Version|Filter|n-shot|       Metric       |   |Value |   |Stderr|
|------------------------------|------:|------|-----:|--------------------|---|-----:|---|-----:|
|longbench_2wikimqa            |      2|none  |     0|qa_f1_score         |↑  |0.5510|±  |0.0281|
|longbench_dureader            |      2|none  |     0|rouge_zh_score      |↑  |0.0639|±  |0.0039|
|longbench_gov_report          |      2|none  |     0|rouge_score         |↑  |0.2930|±  |0.0052|
|longbench_hotpotqa            |      2|none  |     0|qa_f1_score         |↑  |0.5819|±  |0.0273|
|longbench_lcc                 |      2|none  |     0|code_sim_score      |↑  |0.3842|±  |0.0143|
|longbench_lsht                |      2|none  |     0|classification_score|↑  |0.2617|±  |0.0305|
|longbench_multi_news          |      2|none  |     0|rouge_score         |↑  |0.2202|±  |0.0054|
|longbench_multifieldqa_en     |      2|none  |     0|qa_f1_score         |↑  |0.4471|±  |0.0310|
|longbench_multifieldqa_zh     |      2|none  |     0|qa_f1_zh_score      |↑  |0.3921|±  |0.0334|
|longbench_musique             |      2|none  |     0|qa_f1_score         |↑  |0.3829|±  |0.0228|
|longbench_narrativeqa         |      2|none  |     0|qa_f1_score         |↑  |0.3691|±  |0.0228|
|longbench_passage_count       |      2|none  |     0|count_score         |↑  |0.0195|±  |0.0082|
|longbench_passage_retrieval_en|      2|none  |     0|retrieval_score     |↑  |0.3833|±  |0.0288|
|longbench_passage_retrieval_zh|      2|none  |     0|retrieval_zh_score  |↑  |0.6183|±  |0.0267|
|longbench_qasper              |      2|none  |     0|qa_f1_score         |↑  |0.4302|±  |0.0269|
|longbench_qmsum               |      2|none  |     0|rouge_score         |↑  |0.1392|±  |0.0037|
|longbench_repobench-p         |      2|none  |     0|code_sim_score      |↑  |0.2965|±  |0.0122|
|longbench_samsum              |      2|none  |     0|rouge_score         |↑  |0.2087|±  |0.0092|
|longbench_trec                |      2|none  |     0|classification_score|↑  |0.3785|±  |0.0259|
|longbench_triviaqa            |      2|none  |     0|qa_f1_score         |↑  |0.6548|±  |0.0269|
|longbench_vcsum               |      2|none  |     0|rouge_zh_score      |↑  |0.0427|±  |0.0021|
```

A few other sources where I found Llama-3.1-8B-Instruct baselines on LongBench
* [Table 3 here](https://arxiv.org/pdf/2502.14051)
* [Table 1 here](https://arxiv.org/pdf/2410.00161)
* These two separate papers report similar baselines, but my results have
  * Musique is ~10 points higher
  * MultiFieldQA-en is ~7 points lower 

@baberabb I know you worked on adding longbench, did you reproduce any results? were they closer to mine or to the papers I linked?

Hi! Thanks for those references. I'll have a look!

Hi! Wanted to provide an update. Sorry for the delay - had limited bandwidth recently. 

Made some fixes in #2983:
1. Now using all `answers` when calculating metrics rather than the first answer index.
2. truncation issue: for cxt_len > max_len they truncate from the middle, while we always left-truncate.
3. trim whitespace from the preds.

Noticed some inconsistencies in `max_length` settings, for example for [Table 1](https://arxiv.org/pdf/2410.00161) from above, they say they use max length of 19000 (Sec 5.2), but use 130,000 in their [code](https://github.com/IsaacRe/PyramidKV/blob/1ca16d6ea0bc3e48fc4cc8623bc4901bc131d511/run_longbench.py#L79). And some tasks are run without chat template as in the original longbench [code](https://github.com/THUDM/LongBench/blob/2e00731f8d0bff23dc4325161044d0ed8af94c1e/LongBench/pred.py#L63-L64).

Will try to provide updated scores once i get compute access back!



Thank you for the update. I will also re-run it to verify your fixes.

added results to #2983. Most scores are close (musique still ~5% higher), but code tasks are quite a bit off.

Ran myself finally and my results were very close to what you got in #2983 

I suggest correcting `do_sample` to `false` in all the LongBench YAML config files, as suggested in the official [LongBench repo's eval script](https://github.com/THUDM/LongBench/blob/main/LongBench/pred.py#L88). Currently, lm_eval uses `do_sample=true`, which contributes to bad reproducibility.