Repository: EleutherAI/lm-evaluation-harness
Issue: Evalution of Llama on Hellaswag
Root cause: Documentation Deficiency

Hi, there, how do you evaluate Llama on Hellaswag? Llama does not have an API that can pass in an argument called echo and returns the log probs. To the best of my knowledge, if we are using Llama from Huggingface, we can only get logits as model output. How to make it work on Hellaswag?

You should be able to use [HF's implementation of Llama](https://huggingface.co/docs/transformers/model_doc/llama) 

Also note there is an issue with tokenization which is currently a work-in-progress.
https://github.com/EleutherAI/lm-evaluation-harness/pull/531

I evaluated it recently like so:

```bash
python main.py --model hf-causal-experimental --model_args pretrained=huggyllama/llama-7b,use_accelerate=True --tasks hellaswag --batch_size auto
```

```
|  Task   |Version| Metric |Value|   |Stderr|
|---------|------:|--------|----:|---|-----:|
|hellaswag|      0|acc     | 0.57|±  |0.0049|
|         |       |acc_norm| 0.76|±  |0.0043|
```

For the [implementation](https://github.com/EleutherAI/lm-evaluation-harness/blob/23f30926f3ce738e3eee4e6be5c29fb3467e3a6e/lm_eval/base.py#L317), you compute summed log probabilities (`log_softmax` of logits) corresponding to each answer by calling `model(question + answer)` on each answer, then the `argmax` of that is what the model "thinks" is the right answer, even if it generates some nonsense not matching with the actual answer, we just use it as a classifier.

As en example

```py
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('huggyllama/llama-7b')
model = AutoModelForCausalLM.from_pretrained('huggyllama/llama-7b', device_map='auto', load_in_8bit=True)

test = [
    ('Paris is the capital of', ' England.'),
    ('Paris is the capital of', ' Germany.'),
    ('Paris is the capital of', ' France.'),
    ('Paris is the capital of', ' Japan.'),
]

# encode full sentences, questions, and answers, no padding for simplicity
batched_sentences = tokenizer.batch_encode_plus([q + a for q, a in test], add_special_tokens=False, return_tensors='pt')['input_ids']
batched_questions = tokenizer.batch_encode_plus([q for q, _ in test], add_special_tokens=False, return_tensors='pt')['input_ids']

# run the model on full sentences and get the log probabilities
batched_logprobs = F.log_softmax(model(batched_sentences.cuda())['logits'], dim=-1).cpu()

# take log probabilities corresponding to possible answer tokens
batched_logprobs = batched_logprobs[:, len(batched_questions[0]) - 1 : -1, :]

# get the scores by summing log probabilities corresponding to correct answer tokens, unvectorized
scores = []
for sentence, question, logprobs in zip(batched_sentences, batched_questions, batched_logprobs):
    answer = sentence[len(question):]
    guess = logprobs.argmax(dim=-1)
    print(tokenizer.decode(guess), bool((guess == answer).all()))
    scores.append(float(torch.gather(logprobs, 1, answer.unsqueeze(-1)).sum()))

# predict the answer
test[torch.tensor(scores).argmax()]
```

Can answer correctly even if the `guess` in `print` is wrong.

> As en example
> 
> ```python
> import torch
> import torch.nn.functional as F
> from transformers import AutoTokenizer, AutoModelForCausalLM
> 
> tokenizer = AutoTokenizer.from_pretrained('huggyllama/llama-7b')
> model = AutoModelForCausalLM.from_pretrained('huggyllama/llama-7b', device_map='auto', load_in_8bit=True)
> 
> test = [
>     ('Paris is the capital of', ' England.'),
>     ('Paris is the capital of', ' Germany.'),
>     ('Paris is the capital of', ' France.'),
>     ('Paris is the capital of', ' Japan.'),
> ]
> 
> # encode full sentences, questions, and answers, no padding for simplicity
> batched_sentences = tokenizer.batch_encode_plus([q + a for q, a in test], add_special_tokens=False, return_tensors='pt')['input_ids']
> batched_questions = tokenizer.batch_encode_plus([q for q, _ in test], add_special_tokens=False, return_tensors='pt')['input_ids']
> 
> # run the model on full sentences and get the log probabilities
> batched_logprobs = F.log_softmax(model(batched_sentences.cuda())['logits'], dim=-1).cpu()
> 
> # take log probabilities corresponding to possible answer tokens
> batched_logprobs = batched_logprobs[:, len(batched_questions[0]) - 1 : -1, :]
> 
> # get the scores by summing log probabilities corresponding to correct answer tokens, unvectorized
> scores = []
> for sentence, question, logprobs in zip(batched_sentences, batched_questions, batched_logprobs):
>     answer = sentence[len(question):]
>     guess = logprobs.argmax(dim=-1)
>     print(tokenizer.decode(guess), bool((guess == answer).all()))
>     scores.append(float(torch.gather(logprobs, 1, answer.unsqueeze(-1)).sum()))
> 
> # predict the answer
> test[torch.tensor(scores).argmax()]
> ```
> 
> Can answer correctly even if the `guess` in `print` is wrong.

Hello, I was wondering how could we alter this code for few-shot prompting? Thank you so much.