Repository: tatsu-lab/alpaca_eval
Issue: What to think ?
Root cause: Documentation Deficiency

I made the following experiment:
Use guanaco-13b as the evaluator.
Compared existing model_outputs.json of gpt4 vs guanaco-65b: winrate 50.4/49.6
Given the gpt4 leaderboard I would have expected gpt4 wintrate significantly higher vs guanaco-65b

Then I tested gpt4 and guanaco-65b vs text_davinci_003
text_da_vinci_003 winrate and way higher vs the two other.

Even if one could say guanaco-13b as the evaluator is not the best model to use, I find these results quite questioning.

I had to adjust a bit the prompts to reduce the  "no results" rate as follow:

```
A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
### Human: Select the output (a) or (b) that best matches the given instruction. Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) or Output (b). Give a description of the following job: "ophthalmologist".
Output (a):
An ophthalmologist is a medical doctor who specializes in the diagnosis and treatment of eye diseases and conditions.
Output (b):
An ophthalmologist is a medical doctor who pokes and prods at your eyes while asking you to read letters from a chart.

Which is best, Output (a) or Output (b)?
### Assistant:
Output (a)

Here the answer is Output (a) because it provides a comprehensive and accurate description of the job of an ophthalmologist. In contrast, output (b) is more of a joke.

### Human: Select the output (a) or (b) that best matches the given instruction. Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) or Output (b). {instruction}
### Output (a):
{output_1}

### Output (b):
{output_2}

### Which is best, Output (a) or Output (b)?
### Assistant:
```





Hi @vince62s, thank you for raising this!

Did you run the evaluation of guanaco-13b as an evaluator? Here's [some documentation](https://github.com/tatsu-lab/alpaca_eval#contributing-an-evaluator) about how to do that. Guanaco-33b has 62.7 agreement with humans (against 69% for GPT4) and I wouldn't be surprised if Guanaco-13b was much lower. Depending on how low the agreement with humans is, I wouldn't read too much in those results. 

In particular, Guanaco-33b is highly biased towards preferring the first output (as seen in the column **Proba. prefer 1** [here](https://github.com/tatsu-lab/alpaca_eval/tree/main/src/alpaca_eval/evaluators_configs) ), if Guanaco-13b has even a stronger positional bias and because we randomize the order of examples, that would cause Guanaco-13b to assign win-rates close to 50% regardless fo the outputs. That's one possible hypothesis as to what is happening

I'm closing now, but feel free to reopen in case you run the eval of guanaco-13b as an evaluator and it's not that bad!

I ran 13b, not 33b.
when comparing outputs gpt4 vs guanaco-65b, I did it both ways, and results were consistent 50.4/49.6 and 49.6/50.4
But I think I do understand parts of your explanations.

Yes what I meant is that I never ran 13b so it's likely even worst than 33b!