Repository: EleutherAI/lm-evaluation-harness
Issue: Does lm-eval support models like OPT or LLama? 
Root cause: Environment Incompatibility

I am trying to evaluate LLM models like OPT, LLama, Alpaca using the existing tasks. I am trying to see whether I need to extend models or just use existing HuggingFaceAutoLM and convert original model to HG format. 

Hi, we do support all Huggingface `transformers.AutoModelForCausalLM` and `transformers.AutoModelForSeq2SeqLM` classes via the `AutoCausalLM` and `AutoSeq2SeqLM` LM subclasses in `huggingface.py`. 

You should be able to just run these by passing `--model hf-causal` or `--model hf-seq2seq` to `main.py` but if you run into any issues running LLaMA (I believe it's merged into HF transformers main now?) do let me know!

I tried running LLaMA with the following command:
`python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,copa,openbookqa`
And I got results which are pretty far from what reported in the paper. Do you have any idea if LLaMA used some modifications in the evaluation code to produce the numbers we see in the paper?
Were you able to reproduce the numbers shown in the paper?

I tried `python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq`, but got key error.

```
(model_eval) root1@root1-System-Product-Name:~/lx/lm-evaluation-harness$ python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq
Selected Tasks: ['boolq']
Traceback (most recent call last):
  File "/home/root1/lx/lm-evaluation-harness/main.py", line 108, in <module>
    main()
  File "/home/root1/lx/lm-evaluation-harness/main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/models/huggingface.py", line 149, in __init__
    self._config = self.AUTO_CONFIG_CLASS.from_pretrained(
  File "/home/root1/software/miniconda3/envs/model_eval/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 917, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/home/root1/software/miniconda3/envs/model_eval/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 623, in __getitem__
    raise KeyError(key)
KeyError: 'llama'
```

You need to install transformers from source to have the llama model architecture and tokenizer. Also, you need to change the name of the tokenizer in the tokenizer_config.json file to LlamaTokenizer to get it to work

Hello, @ofirzaf, I download the model from huggingface and change tokenizer name in tokenizer_config.json to `LlamaTokenizer `
![image](https://user-images.githubusercontent.com/42756849/227902895-1f783909-43fc-41c2-b3c4-ea764fff8354.png)
![image](https://user-images.githubusercontent.com/42756849/227902989-9f0bfc26-0854-4241-9d0c-4cd85c405cd8.png)

after that i try:

```
python main.py --model hf-causal --model_args pretrained="/home/root1/models/llama-7b-hf",tokenizer="LlamaTokenizer" --device cuda:0  --tasks boolq
```

but got same error:

```
(model_eval) root1@root1-System-Product-Name:~/lx/lm-evaluation-harness$ python main.py --model hf-causal --model_args pretrained="/home/root1/models/llama-7b-hf",tokenizer="LlamaTokenizer" --device cuda:0  --tasks boolq
Selected Tasks: ['boolq']
Traceback (most recent call last):
  File "/home/root1/lx/lm-evaluation-harness/main.py", line 108, in <module>
    main()
  File "/home/root1/lx/lm-evaluation-harness/main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/models/huggingface.py", line 149, in __init__
    self._config = self.AUTO_CONFIG_CLASS.from_pretrained(
  File "/home/root1/software/miniconda3/envs/model_eval/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 917, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/home/root1/software/miniconda3/envs/model_eval/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 623, in __getitem__
    raise KeyError(key)
KeyError: 'llama'
```

Transformers version 4.27.3


You need to install main branch from source, download transformers repo and install it with `pip install -e <transformers directory`

Thanks for your advice, i installed from main branch

```
Building wheels for collected packages: transformers
  Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.28.0.dev0-0.editable-py3-none-any.whl size=35081 sha256=a38464d59fdcf8c9303bf51611e6dc8343bf420de052b6c5dbba6cdc3fade016
  Stored in directory: /tmp/pip-ephem-wheel-cache-q1f3ggem/wheels/a7/27/25/00b53be48807b094c61303dbf14ff5e73cca23134830b4a53a
Successfully built transformers
Installing collected packages: transformers
  Attempting uninstall: transformers
    Found existing installation: transformers 4.28.0.dev0
    Uninstalling transformers-4.28.0.dev0:
      Successfully uninstalled transformers-4.28.0.dev0
Successfully installed transformers-4.28.0.dev0
```

but when i try to load tokenizer and model, it occur another error, have you ever encountered this problem before?

```
>>> tokenizer = LlamaTokenizer.from_pretrained("/home/root1/models/llama-7b-hf")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/root1/lx/transformers/src/transformers/tokenization_utils_base.py", line 1805, in from_pretrained
    return cls._from_pretrained(
  File "/home/root1/lx/transformers/src/transformers/tokenization_utils_base.py", line 1959, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/root1/lx/transformers/src/transformers/models/llama/tokenization_llama.py", line 71, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/root1/software/miniconda3/envs/model_eval/lib/python3.9/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
  File "/home/root1/software/miniconda3/envs/model_eval/lib/python3.9/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
```

I have solved it, thanks~

```
python main.py --model hf-causal --model_args pretrained="/home/root1/models/llama-7b-hf",tokenizer="/home/root1/models/llama-7b-hf" --device cuda:0  --tasks boolq
```

i miss `tokenizer.model` in my local model dir.

> I tried running LLaMA with the following command: `python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda --tasks boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,copa,openbookqa` And I got results which are pretty far from what reported in the paper. Do you have any idea if LLaMA used some modifications in the evaluation code to produce the numbers we see in the paper? Were you able to reproduce the numbers shown in the paper?

I tested llama-7b-hf on **boolq** using lm-evaluation-harness and the result was 73.09%, just 3% less than the original paper. But when I tried it on **triviaqa** dataset, the result came to 0%......which is obviously abnormal. What's your testing result?

> Hi, we do support all Huggingface `transformers.AutoModelForCausalLM` and `transformers.AutoModelForSeq2SeqLM` classes via the `AutoCausalLM` and `AutoSeq2SeqLM` LM subclasses in `huggingface.py`.
> 
> You should be able to just run these by passing `--model hf-causal` or `--model hf-seq2seq` to `main.py` but if you run into any issues running LLaMA (I believe it's merged into HF transformers main now?) do let me know!

https://huggingface.co/docs/transformers/main/en/model_doc/llama#transformers.LlamaForSequenceClassification

Just a summarization for people who are new for LLM evaluation. 

### Preparation

```
# 1. Install lm-eval
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .

# 2. Install nightly transformer. (4.27.x doesn't have the LLM support, it's only on master at this moment)
git clone https://github.com/huggingface/transformers.git
cd transformers
pip3 install -e .
pip3 list | grep transformers
transformers             4.28.0.dev0  /root/transformers

# 3. Install other required libraries
sentencepiece

# 4. (Optional) Update Tokenizer name
If you download from decapoda-research/llama-7b-hf. This is not using up to date transformer library so tokenizer name needs to be changed.  Change the name of the tokenizer in the tokenizer_config.json file to `LlamaTokenizer`. 
If you plan to use hugging face to automatically download the mode, that's ok. You can make the change after you first time run the code and encounter such failures. Then update the name

vim ~/.cache/huggingface/hub/models--decapoda-research--llama-7b-hf/snapshots/5f98eefcc80e437ef68d457ad7bf167c2c6a1348/tokenizer_config.json

------
Selected Tasks: ['arc_easy', 'hellaswag', 'piqa', 'openbookqa', 'arc_challenge', 'boolq', 'winogrande', 'copa']
Downloading (…)lve/main/config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 427/427 [00:00<00:00, 124kB/s]
Downloading (…)okenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 141/141 [00:00<00:00, 49.9kB/s]
Traceback (most recent call last):
  File "main.py", line 108, in <module>
    main()
  File "main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/huggingface.py", line 155, in __init__
    self.tokenizer = self._create_auto_tokenizer(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/huggingface.py", line 374, in _create_auto_tokenizer
    tokenizer = super()._create_auto_tokenizer(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/huggingface.py", line 221, in _create_auto_tokenizer
    tokenizer = self.AUTO_TOKENIZER_CLASS.from_pretrained(
  File "/home/ubuntu/transformers/src/transformers/models/auto/tokenization_auto.py", line 686, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

```

### Running Evaluation

```
python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq
```

@ofirzaf Here's my evaluation results. Is your data similar like mine? 

```
python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,copa,openbookqa
.....
.....
Running loglikelihood requests
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 62732/62732 [29:06<00:00, 35.93it/s]
{
.....
  "config": {
    "model": "hf-causal",
    "model_args": "pretrained=decapoda-research/llama-7b-hf",
    "num_fewshot": 0,
    "batch_size": null,
    "device": "cuda",
    "no_cache": false,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal (pretrained=decapoda-research/llama-7b-hf), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
|    Task     |Version| Metric |Value |   |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|openbookqa   |      0|acc     |0.2800|±  |0.0201|
|             |       |acc_norm|0.4240|±  |0.0221|
|hellaswag    |      0|acc     |0.5644|±  |0.0049|
|             |       |acc_norm|0.7296|±  |0.0044|
|arc_easy     |      0|acc     |0.6730|±  |0.0096|
|             |       |acc_norm|0.5248|±  |0.0102|
|boolq        |      1|acc     |0.7315|±  |0.0078|
|winogrande   |      0|acc     |0.6709|±  |0.0132|
|piqa         |      0|acc     |0.7835|±  |0.0096|
|             |       |acc_norm|0.7731|±  |0.0098|
|copa         |      0|acc     |0.8400|±  |0.0368|
|arc_challenge|      0|acc     |0.3823|±  |0.0142|
|             |       |acc_norm|0.4138|±  |0.0144|
```

![image](https://user-images.githubusercontent.com/4739316/228375025-6cb5fb54-b0e7-40f9-965f-35b777348d36.png)
> note: screenshot is from paper https://arxiv.org/pdf/2302.13971.pdf

I create the comparison 

|                  | BoolQ | PIQA | HellaSwag | WinoGrande | Arc-challenges | OBQA |
|------------------|-------|------|-----------|------------|----------------|------|
| LLaMA-7B-paper   | 76.5  | 79.8 | 76.1      | 70.1       | 47.6           | 57.2 |
| LLaMA-7B-lm-eval | 73.1  | 78.3 | 56.4      | 67.1       | 38.2           | 28.0 |


> I tested llama-7b-hf on **boolq** using lm-evaluation-harness and the result was 73.09%, just 3% less than the original paper. But when I tried it on **triviaqa** dataset, the result came to 0%......which is obviously abnormal. What's your testing result?

Here's my result on **triviaqa**, it looks same. It's probably a bug there.

```
Running loglikelihood requests
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 116146/116146 [51:58<00:00, 37.24it/s]
{
  "results": {
    "triviaqa": {
      "acc": 0.0,
      "acc_stderr": 0.0
    }
  },
  "versions": {
    "triviaqa": 1
  },
  "config": {
    "model": "hf-causal",
    "model_args": "pretrained=decapoda-research/llama-7b-hf",
    "num_fewshot": 0,
    "batch_size": null,
    "device": "cuda",
    "no_cache": false,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal (pretrained=decapoda-research/llama-7b-hf), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
|  Task  |Version|Metric|Value|   |Stderr|
|--------|------:|------|----:|---|-----:|
|triviaqa|      1|acc   |    0|±  |     0|
```


> @ofirzaf Here's my evaluation results. Is your data similar like mine?
> 
> ```
> python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,copa,openbookqa
> .....
> .....
> Running loglikelihood requests
> 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 62732/62732 [29:06<00:00, 35.93it/s]
> {
> .....
>   "config": {
>     "model": "hf-causal",
>     "model_args": "pretrained=decapoda-research/llama-7b-hf",
>     "num_fewshot": 0,
>     "batch_size": null,
>     "device": "cuda",
>     "no_cache": false,
>     "limit": null,
>     "bootstrap_iters": 100000,
>     "description_dict": {}
>   }
> }
> hf-causal (pretrained=decapoda-research/llama-7b-hf), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
> |    Task     |Version| Metric |Value |   |Stderr|
> |-------------|------:|--------|-----:|---|-----:|
> |openbookqa   |      0|acc     |0.2800|±  |0.0201|
> |             |       |acc_norm|0.4240|±  |0.0221|
> |hellaswag    |      0|acc     |0.5644|±  |0.0049|
> |             |       |acc_norm|0.7296|±  |0.0044|
> |arc_easy     |      0|acc     |0.6730|±  |0.0096|
> |             |       |acc_norm|0.5248|±  |0.0102|
> |boolq        |      1|acc     |0.7315|±  |0.0078|
> |winogrande   |      0|acc     |0.6709|±  |0.0132|
> |piqa         |      0|acc     |0.7835|±  |0.0096|
> |             |       |acc_norm|0.7731|±  |0.0098|
> |copa         |      0|acc     |0.8400|±  |0.0368|
> |arc_challenge|      0|acc     |0.3823|±  |0.0142|
> |             |       |acc_norm|0.4138|±  |0.0144|
> ```
> 
> ![image](https://user-images.githubusercontent.com/4739316/228375025-6cb5fb54-b0e7-40f9-965f-35b777348d36.png)
> 
> > note: screenshot is from paper https://arxiv.org/pdf/2302.13971.pdf
> 
> I create the comparison
> 
> BoolQ	PIQA	HellaSwag	WinoGrande	Arc-challenges	OBQA
> LLaMA-7B-paper	76.5	79.8	76.1	70.1	47.6	57.2
> LLaMA-7B-lm-eval	73.1	78.3	56.4	67.1	38.2	28.0

Thanks a lot for your sharing. I got similar results using lm-eval, except that I think accuracy_norm results are better to be compared in your table, because the llama paper said it used the same normalization as lm-eval. 
And I noticed other details that may affect the results:
1. The paper used different normalization method on **boolq** and **OpenBookQA** dataset, as illustrated here:
![image](https://user-images.githubusercontent.com/109711569/228444118-3d3abdee-49fa-42ff-827f-ce6913c63346.png)
2. The released llama code adds <bos> token at the beginning of a scentence, so maybe the same pattern should be added as well:
![image](https://user-images.githubusercontent.com/109711569/228444564-cc4557f8-fa69-4700-b719-b34145e04620.png)

I added the tricks above and got slight boosts, but still had 1%-2% difference between the results in the paper. Maybe you can have a try.

> > I tested llama-7b-hf on **boolq** using lm-evaluation-harness and the result was 73.09%, just 3% less than the original paper. But when I tried it on **triviaqa** dataset, the result came to 0%......which is obviously abnormal. What's your testing result?
> 
> Here's my result on **triviaqa**, it looks same. It's probably a bug there.
> 
> ```
> Running loglikelihood requests
> 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 116146/116146 [51:58<00:00, 37.24it/s]
> {
>   "results": {
>     "triviaqa": {
>       "acc": 0.0,
>       "acc_stderr": 0.0
>     }
>   },
>   "versions": {
>     "triviaqa": 1
>   },
>   "config": {
>     "model": "hf-causal",
>     "model_args": "pretrained=decapoda-research/llama-7b-hf",
>     "num_fewshot": 0,
>     "batch_size": null,
>     "device": "cuda",
>     "no_cache": false,
>     "limit": null,
>     "bootstrap_iters": 100000,
>     "description_dict": {}
>   }
> }
> hf-causal (pretrained=decapoda-research/llama-7b-hf), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
> |  Task  |Version|Metric|Value|   |Stderr|
> |--------|------:|------|----:|---|-----:|
> |triviaqa|      1|acc   |    0|±  |     0|
> ```

The same. And I notice another thing: llama filtered the triviaqa dev set, maybe that affects too. But 0% definitely has bugs.
![image](https://user-images.githubusercontent.com/109711569/228468077-ae885dd2-808f-43d3-b5cb-45fbabbb8a73.png)


> I added the tricks above and got slight boosts, 

@yysjasmine 

I added `special_tokens_map={"bos_token": "
","eos_token": "
"}` to tokenizer initialization and added ` --model_args pretrained=decapoda-research/llama-7b-hf,add_special_tokens=True` in the running scripts but get errors like this 

```
Traceback (most recent call last):
  File "main.py", line 108, in <module>
    main()
  File "main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/huggingface.py", line 142, in __init__
    assert (
AssertionError: Evaluating causal models with `add_special_tokens=True` is currently not supported.
```

This looks expected in current code base, https://github.com/EleutherAI/lm-evaluation-harness/blob/50eb80ba5f23bde87cb80bbe1f1633bb98038e58/lm_eval/models/huggingface.py#L137-L144. Could you share your changes to make it work? I would like to reproduce and see the numbers. Thanks! 

@Jeffwan 
In fact, I extracted the logic of loglikelihood calculation from here:
https://github.com/EleutherAI/lm-evaluation-harness/blob/50eb80ba5f23bde87cb80bbe1f1633bb98038e58/lm_eval/base.py#L172-L185
and adapt them into the original llama code, so `
` could be directly added to the begining of the sentence (no need to chage mask or position according to llama original code in generation).
Maybe you can have a try to view `
` as a normal token and directly add it to the original encoding when calculating loglikelihood, like:
`context_enc = [1] + self.tok_encode(context)`
But I don't know if that works.

I will give it a try

By the way, have you figured out why the results on triviaqa dataset is 0? @Jeffwan 

> @ofirzaf Here's my evaluation results. Is your data similar like mine?
> 
> ```
> python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,copa,openbookqa
> .....
> .....
> Running loglikelihood requests
> 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 62732/62732 [29:06<00:00, 35.93it/s]
> {
> .....
>   "config": {
>     "model": "hf-causal",
>     "model_args": "pretrained=decapoda-research/llama-7b-hf",
>     "num_fewshot": 0,
>     "batch_size": null,
>     "device": "cuda",
>     "no_cache": false,
>     "limit": null,
>     "bootstrap_iters": 100000,
>     "description_dict": {}
>   }
> }
> hf-causal (pretrained=decapoda-research/llama-7b-hf), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
> |    Task     |Version| Metric |Value |   |Stderr|
> |-------------|------:|--------|-----:|---|-----:|
> |openbookqa   |      0|acc     |0.2800|±  |0.0201|
> |             |       |acc_norm|0.4240|±  |0.0221|
> |hellaswag    |      0|acc     |0.5644|±  |0.0049|
> |             |       |acc_norm|0.7296|±  |0.0044|
> |arc_easy     |      0|acc     |0.6730|±  |0.0096|
> |             |       |acc_norm|0.5248|±  |0.0102|
> |boolq        |      1|acc     |0.7315|±  |0.0078|
> |winogrande   |      0|acc     |0.6709|±  |0.0132|
> |piqa         |      0|acc     |0.7835|±  |0.0096|
> |             |       |acc_norm|0.7731|±  |0.0098|
> |copa         |      0|acc     |0.8400|±  |0.0368|
> |arc_challenge|      0|acc     |0.3823|±  |0.0142|
> |             |       |acc_norm|0.4138|±  |0.0144|
> ```
> 
> ![image](https://user-images.githubusercontent.com/4739316/228375025-6cb5fb54-b0e7-40f9-965f-35b777348d36.png)
> 
> > note: screenshot is from paper https://arxiv.org/pdf/2302.13971.pdf
> 
> I create the comparison
> 
> BoolQ	PIQA	HellaSwag	WinoGrande	Arc-challenges	OBQA
> LLaMA-7B-paper	76.5	79.8	76.1	70.1	47.6	57.2
> LLaMA-7B-lm-eval	73.1	78.3	56.4	67.1	38.2	28.0

hi, @Jeffwan  I got the same result on BoolQ, PIQA, HellaSwag, WinoGrande  tasks, which had ~3% difference from the results in the paper. But I cannot reproduce the result on RACE task, the result is as follows:
![image](https://user-images.githubusercontent.com/6135246/233844470-11899587-8a47-4ae2-8948-423e70f04dd2.png)


Do you have any idea?