Repository: EleutherAI/lm-evaluation-harness
Issue: about nq_open results
Root cause: Configuration Error

Hi,
I have problems reproducing the results of Natural Question dataset from this [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/789). This is the command I use:

For llama1-0-shot
```bash
lm-eval --model hf --batch_size auto --task nq_open --model_args pretrained=huggyllama/llama-7b
```

For llama1-5-shot
```bash
lm-eval --model hf --batch_size auto --task nq_open --model_args pretrained=huggyllama/llama-7b --num_fewshot 5
```

For llama2-0-shot:
```bash
lm-eval --model hf --batch_size auto --task nq_open --model_args pretrained=meta-llama/Llama-2-7b-hf
```

For llama2-5-shot
```bash
lm-eval --model hf --batch_size auto --task nq_open --model_args pretrained=meta-llama/Llama-2-7b-hf --num_fewshot 5
```

This is the results I got:

|             | 0-shot-llama1-7b | 5-shot-llama1-7b | 0-shot-llama2-7b | 5-shot-llama2-7b |
|:-----------:|:----------------:|:----------------:|:----------------:|:----------------:|
| original PR |       18.3       |       23.9       |       17.5       |       26.1       |
|  reproduced |       13.3    |       21.5       |       14.5       |       25.2      |

I notice there was a refactor of the database, I am not sure if this would cause discrepancy here. 

Hi! Thanks for opening this. Are you using the most recent version of this codebase?

I think the discrepancy may be due to the lack of passing a prepended description (see https://github.com/EleutherAI/lm-evaluation-harness/pull/789#issuecomment-1685487046 )--we changed from expecting users to pass this via the CLI, to allowing it to be specified for a task in that task's config. Would you be willing to add `description: "Answer these questions:\n\n"` into the NaturalQs YAML file and see if you can replicate one of these results?

If so then we can merge that fix. I'm sorry for the issues!

Hi, thanks for response! I used the latest version by `git clone` the repo and `pip install -e`. The commit id is `588a493c41b58d048d3994f99800edd27223bd3a`.

The YAML file I use for nq_open is this: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/nq_open/nq_open.yaml.

There is already `description` filed. https://github.com/EleutherAI/lm-evaluation-harness/blob/588a493c41b58d048d3994f99800edd27223bd3a/lm_eval/tasks/nq_open/nq_open.yaml#L6

Do you mean I should change it to `"Answer these questions:\n\n"`? I would try it out. 

Thanks again for your help!

Haha, this is crazy. Simply adding a `\n` to the current description make llama-2-7b-0-shot from 14.5 to 18.4.

*sigh* Sometimes I hate LLM evals. I'm glad this works to reproduce though!

To match the old `master` implementation, `\n\n` is needed. I'll PR this fix!

Hi, thanks for fixing! 

For anyone who is also interested in the nq_open results:

| **llama1** |       **Description**       | **0-shot** | **1-shot** | **5-shot** |
|:----------:|---------------------------  |:----------:|:----------:|:----------:|
|     7b     |Answer these questions:\n    |    13.4    |    17.3    |    21.4    |
|     7b     |Answer these questions:\n\n  |    17.3    |    18.8    |    23.0    |
|     13b    |Answer these questions:\n    |    14.7    |    21.5    |    26.5    |
|     13b    |Answer these questions:\n\n  |    22.2    |    23.5    |    28.0    |


| **llama2** |       **Description**       | **0-shot** | **1-shot** | **5-shot** |
|:----------:|---------------------------  |:----------:|:----------:|:----------:|
|     7b     |Answer these questions:\n    |    14.5    |    21.3    |    25.2    |
|     7b     |Answer these questions:\n\n  |    18.4    |    22.5    |    25.5    |
|     13b    |Answer these questions:\n    |    14.4    |    23.9    |    28.5    |
|     13b    |Answer these questions:\n\n  |    22.8    |    26.0    |    29.7    |

Slightly lower than results in this [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/789#issuecomment-1683453138), reason unknown