Repository: tatsu-lab/alpaca_eval
Issue: Huge performance gap when using annotator weighted_alpaca_eval_gpt4_turbo and alpaca_eval_gpt4_turbo_fn 
Root cause: Documentation Deficiency

Hi, thanks for your nice work.

I am evaluating my model against GPT-4 turbo generations on the AlpacaEval 2.0. I observed that there is a huge gap when using `weighted_alpaca_eval_gpt4_turbo` and `alpaca_eval_gpt4_turbo_fn`.   Specifically, when evaluating the same model output, they achieve:

|                                    | Lc win rate | Win rate | n_total | avg_len |
| ---------------------------------- | ----------- | -------- | ------- | ------- |
| my_model_weighted_alpaca_eval_gpt4 | 22.63       | 18.38    | 650     | 1366    |
| My_model_alpaca_eval_gpt4_turbo_fn | 29.10       | 22.56    | 749     | 1366    |

Is this performance gap strange?

Besides, there is another issue. For several recent days, I consistently got the warning that `xx samples had no auto annotation. we are filtering them for now.`  So, I got `n_total`, sometimes much smaller than 805, as shown in the table above. However, I have not used a CoT-style annotator. Do you have any ideas for this?

There are two things here:
1. The absolute scores are not comparable between annotators. Does the relative ranking between models change much?
2. The second row in your table is evaluated on ~100 instructions more (749 vs 650) It’s not surprising that with different instructions you get different scores.


Now as to why you are not getting all the annotations, my guess is that it’s an issue with the API you are using. Are you using OpenAI API directly or some proxy?

Thanks for your quick response and questions!

1. The relative ranking also changes if I change the evaluator configuration. In most cases, I observe some improvements if using `alpaca_eval_gpt4_turbo_fn`, but the improvements vanish if I use `weighted_alpaca_eval_gpt4_turbo`. 
2. I agree that the absolute scores are not directly comparable across different annotators. However, both evaluation configurations employed the `gpt-1104-preview` model as the judge. I think the difference between them is that prompt and `weighted_alpaca_eval_gpt4_turbo` only return one token. Given that both configurations use the generation of `gpt4-1104-preview` as the reference output,  it seems a little weird to me that changing the prompt could lead to such a difference in `win_rate` against the same reference output.
3. I use the proxy. Do you happen to have any idea how to fix this? If I fix this, I will immediately reproduce the result with the full dataset to see if such a gap still exists. AlpacaEval2 is of significant importance to the LLMs community. Maybe we need to show its robustness to the evaluator's different prompt (evaluation mode)?

Hi Dubois,

I got some results with the full alpaca eval set: 

|                                    | Lc win rate | Win rate | n_total | avg_len |
| ---------------------------------- | ----------- | -------- | ------- | ------- |
| my_model_weighted_alpaca_eval_gpt4 | 23.39       | 23.39    | 805     | 1878    |
| my_model_alpaca_eval_gpt4_turbo_fn | 32.33       | 31.55    | 805     | 1878    |

If we change the evaluator configuration from `weighted_alpaca_eval_gpt4_turbo` to `alpaca_eval_gpt3_turbo_fn`, the win rate varies by nearly 10%. Both use the same model, `gpt4-1106-preview`, so I assume the difference comes from how we prompt them. 

For `weighted_alpaca_eval_gpt4`, we prompt it to make sure its "output only contains one of the following model identifiers and nothing else". And for `alpaca_eval_gpt4_turbo_fn`, we prompt it to "Evaluate and rank the models based on the quality and relevance of their outputs. The ranking should be such that the model with the highest quality output is ranked first." The latter one makes more sense to me as it does not introduce strong restriction to the model's output, while most models are evaluated using the former. 



The absolute value really cannot be compared between both. For example the first one will actually not rate preferences based on a token but based on the probability of preferring a certain model. E.g. if `gpt4-1106-preview` prefers model A with 60% then the first one will increase the win rate of model A by 60 while the other will the latter will increase the win rate by 100. This is what `weighted_` refers to. To easily work with a weighted prompt you need to generate a single token which is why the first model does that. 

as seen in the two leaderboard, the relative ranking doesn't change much:
- https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/leaderboards/data_AlpacaEval_2/alpaca_eval_gpt4_turbo_fn_leaderboard.csv
- https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/leaderboards/data_AlpacaEval_2/weighted_alpaca_eval_gpt4_turbo_leaderboard.csv


In your table you are only considering absolute ranking which doesn't give much insight into the leaderboard