Repository: EleutherAI/lm-evaluation-harness
Issue: Some questions on the DROP evaluations
Root cause: Documentation Deficiency

Hi！
I just evaluated Microsoft/Orca-2-7b on the DROP dataset. And the evaluation is quite different to [Microsoft's paper](https://arxiv.org/pdf/2311.11045.pdf). They show that the zero-shot performance of Orca 2 7-B on the DROP is about 60.26.

However, my evaluation result is as follows:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/76684848/2c6c4784-5f77-4e06-a767-a661fc448cd4)

I have some hypotheses about the discrepancies in the results, such as differences in the prompts used in the paper and the harness. However, I did not anticipate such a significant difference in the evaluation results. 

**environment setting & args**
I use the latest master branch of EleutherAI/lm-evaluation-harness
with the command:
```
python main.py --model hf-causal-experimental --model_args pretrained=microsoft/Orca-2-7b,use_accelerate=True,dtype=half  --task drop --device cuda:0,1 --no_cache --batch_size auto
```



--------
Some extra information:

will it be because of the tokenizer? 
I install the `sentencepiece` to avoid the following error:


    ValueError: Couldn't instantiate the backend tokenizer from one of:                                                                                                                                                             
    (1) a `tokenizers` library serialization file,                                                                                                                                                                                  
    (2) a slow tokenizer instance to convert or                                                                                                                                                                                     
    (3) an equivalent slow tokenizer class to instantiate and convert.                                                                                                                                                              
    You need to have sentencepiece installed to convert a slow tokenizer to a fast one.                                                                                                                                             
    (lm-eval) python main.py --model hf-causal-experimental --model_args pretrained=microsoft/Orca-2-7b,use_accelerate=True --task drop --device cuda:0,1 --no_cache --batch_size auto                                              
    [dynet] random seed: 1234                                                                                                                                                                                                       
    [dynet] allocating memory: 32MB                                                                                                                                                                                                 
    [dynet] memory allocation done.                                                                                                                                                                                                 
    Selected Tasks: ['drop']                                                                                                                                                                                                        
    Traceback (most recent call last):                                                                                                                                                                                              
      File "main.py", line 105, in <module>     
      
  and also the  `protobuf` with the version `4.25.1` to avoid the error below:
  
        ImportError:                                                                                                                                                                                                                    
      The new behaviour of LlamaTokenizer (with `self.legacy = False`) requires the protobuf library but it was not found in your environment. Checkout the instructions on the                                                       
      installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones                                                                                                  
      that match your environment. Please note that you may need to restart your runtime after installation.  
Thank you for your attention.

It looks like they use both a system prompt and a custom prompt when evaluating DROP. You can write a variant on the DROP benchmark using these things via the `big-refactor` branch. Give that a try and if the numbers are still different let us know.

They also use custom answer parsing in the paper. We currently support the official answer parsing, but have been doing some investigation with HuggingFace and have concluded that it's really bad. We will be pushing an updated parsing process soon which differs from the official implementation.

See #1050 for further discussion