Repository: EleutherAI/lm-evaluation-harness
Issue: Getting different results on standard benchmarks for Llama-2 7B
Root cause: Documentation Deficiency

Hi,

I've setup the harness to run some standard benchmarks on the base Llama-2 7B (HF) model. I had some confusion as I wasn't getting the same results as what's been reported in the paper. Note that I am using the same files that are hosted [here](https://huggingface.co/meta-llama/Llama-2-7b-hf).

Here are the results from the paper for quick reference:
![Screenshot 2023-08-24 at 3 18 00 PM](https://github.com/EleutherAI/lm-evaluation-harness/assets/47084919/c015962d-7bb0-4344-8928-c976a2cc0462)

Here is how I am running the harness as well as the outputs.
`python main.py --model hf-causal --model_args pretrained=/path_to_llama/Llama-2-7b-hf/ --tasks hellaswag,boolq,piqa,winogrande,arc_easy,arc_challenge --device cuda:0`

```
{
  "results": {
    "arc_challenge": {
      "acc": 0.4351535836177474,
      "acc_stderr": 0.014487986197186045,
      "acc_norm": 0.46245733788395904,
      "acc_norm_stderr": 0.014570144495075578
    },
    "arc_easy": {
      "acc": 0.7634680134680135,
      "acc_stderr": 0.008719840797175742,
      "acc_norm": 0.7466329966329966,
      "acc_norm_stderr": 0.008924765424529265
    },
    "boolq": {
      "acc": 0.7773700305810397,
      "acc_stderr": 0.007276093141006333
    },
    "hellaswag": {
      "acc": 0.571400119498108,
      "acc_stderr": 0.004938643787869544,
      "acc_norm": 0.7600079665405298,
      "acc_norm_stderr": 0.0042620545265771035
    },
    "piqa": {
      "acc": 0.780739934711643,
      "acc_stderr": 0.009653357463605317,
      "acc_norm": 0.7905331882480957,
      "acc_norm_stderr": 0.009494302979819805
    },
    "winogrande": {
      "acc": 0.6906077348066298,
      "acc_stderr": 0.012991329330823007
    }
  },
  "versions": {
    "arc_challenge": 0,
    "arc_easy": 0,
    "boolq": 1,
    "hellaswag": 0,
    "piqa": 0,
    "winogrande": 0
  },
  "config": {
    "model": "hf-causal",
    "model_args": "pretrained=/model-weights/Llama-2-7b-hf/",
    "num_fewshot": 0,
    "batch_size": null,
    "batch_sizes": [],
    "device": "cuda:0",
    "no_cache": false,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal (pretrained=/model-weights/Llama-2-7b-hf/), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
|    Task     |Version| Metric |Value |   |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge|      0|acc     |0.4352|±  |0.0145|
|             |       |acc_norm|0.4625|±  |0.0146|
|arc_easy     |      0|acc     |0.7635|±  |0.0087|
|             |       |acc_norm|0.7466|±  |0.0089|
|boolq        |      1|acc     |0.7774|±  |0.0073|
|hellaswag    |      0|acc     |0.5714|±  |0.0049|
|             |       |acc_norm|0.7600|±  |0.0043|
|piqa         |      0|acc     |0.7807|±  |0.0097|
|             |       |acc_norm|0.7905|±  |0.0095|
|winogrande   |      0|acc     |0.6906|±  |0.0130|
```

Any ideas as to what might be causing the difference here? Is there any potential issue with the tokenizer or maybe a difference in the evaluated dataset?

I think the `hf-causal` model is deprecated. 

> I think the `hf-causal` model is deprecated.

I'm not sure I follow. Isn't that the recommended API usage?

I see only minor differences. Just curious -- what difference are you specifically talking about? 

One potential reason: you are using num_fewshot=0. The [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) uses non-zero num_fewshot for many tasks. 

For Llama-2, from their description, they were using their "internal evaluations library" (Sec 2.3) for which there's seems to be no publicly available info.

Given the potential discrepancies in implementation, I think it's reasonable we see some minor diff. Or, is there anything that you find particularly weird?

> I think the hf-causal model is deprecated.

`hf-causal` and `hf-causal-experimental` both remain supported while we near the release of a new version (WIP found in `big-refactor` branch)!

For differences between Llama 2's paper and our repository, Llama 1 and 2 seem to use [Task Descriptions](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/description_guide.md) prefixed to the front of their fewshot prompts. though for most tasks it's impossible for us to know what precise prompt + description they used for evaluations we've tried to track down ones that we discover. 



> > I think the hf-causal model is deprecated.
> 
> `hf-causal` and `hf-causal-experimental` both remain supported while we near the release of a new version (WIP found in `big-refactor` branch)!
> 
> For differences between Llama 2's paper and our repository, Llama 1 and 2 seem to use [Task Descriptions](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/description_guide.md) prefixed to the front of their fewshot prompts. though for most tasks it's impossible for us to know what precise prompt + description they used for evaluations we've tried to track down ones that we discover.

Could you inform me the difference between `hf-causal` and `hf-causal-experimental`? `hf-causal-experimental` seems to be for multi-GPUs and multi stop criterion. The results of gsm8k with llama-v2 are different for `hf-causal` and `hf-causal-experimental`. Which one am I supposed to use?

The answer to the original question is that the LLaMA 2 paper uses some unreleased custom prompts that we are not privy to. You can see this in action by comparing the results listed for the LLaMA 1 model between the LLaMA 1 and LLaMA 2 papers... they change their custom promoting, resulting in a massive swing in some scores!

For this reason we strongly urge all researchers to evaluate all models on the same codebase and not copy numbers from other papers. We know it's annoying and expensive, but it's necessary to have a meaningful comparison (unless you're both using this library!)