zhiminy/EvalEng huggingface dataset:
Repository: tatsu-lab/alpaca_eval
Issue: [Question] Alternative for GPT4 eval?
Root cause: Documentation Deficiency

I am still forbidden from approaching GPT4 api, so is there an alternative for evaluation?


We support many different evaluators, which we compare [here](https://github.com/tatsu-lab/alpaca_eval/tree/main/src/alpaca_eval/evaluators_configs). You just need to specify the desired annotator in the CLI, e.g. for chatgpt use `--annotators_config chatgpt`. If you don't have GPT4 I would suggest choosing between the following:
- `chatgpt` cheaper but less human-like 
- `text_davinci_003` pretty high human-agreement

The only issues you might encounter are:
1. if your model generates very long outputs given that chatgpt/text-davinci-003 have a context window 2x smaller than GPT4.
2. we didn't precompute the leaderboard for those.

Given that this seems to be a recurrent problem I will try to precompute the leaderboard using the 16k token chatgpt. Leaving the issue open until then.

We've now added a `chatgpt_fn` annotator. You can find a minimal leaderboard using it here https://github.com/tatsu-lab/alpaca_eval#models and evaluation of its performance here https://github.com/tatsu-lab/alpaca_eval#evaluators. In general, it's not as good as gpt4 or claude, but it should be the most widely accessible one.
Repository: tatsu-lab/alpaca_eval
Issue: Add Phi 3 models
Root cause: Unimplemented Feature Gap

It would be really nice if Microsoft's new [Phi 3 models](https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3) could be added to the AlpacaEval Leaderboard,

That would be great, I would love to accept a PR for that if you have the time!

here's are the configs for phi-2 if it's useful

https://github.com/tatsu-lab/alpaca_eval/blob/0d451644eeffd955e530162ca83e32f20d7f3268/src/alpaca_eval/models_configs/phi-2/configs.yaml#L4



Closing for now, but I would be happy to merge such a PR if you submit one!
Repository: tatsu-lab/alpaca_eval
Issue: Question on assumption of `model_identity` as a factor for preference on generated outputs.

Thanks for all the brilliant work of yours. I'm exploring it with joy for now. 

One thing I feel awkward about during read is the [part](https://arxiv.org/html/2404.04475v1#S3) that **considering `model-identity` as a factor for affecting generation output preference**. In a blind test scenario, GPT-4 or human judge is not provided that which text coming from what model. Why should it affect the preference? 

After all, some model might clearly perform better (and so it preferred more) than others, which could be verified by the win rate over the benchmark examples. That is the result we want, but not the factor to be included when we make a preference decision for comparison.

I concisely draw my point of view with the doodle below: 
![image](https://github.com/tatsu-lab/alpaca_eval/assets/135444161/254ea445-8e8f-4565-bb46-e38a58acd25a)


Thanks for your answer in advance, appreciating for pointing out any misconception might I have.
Have a nice day =]

I think the core idea of LCAE, modify the overestimated factors (including length) of preference to make the automated judges to agree more with human preference, is correct. Hope you got my intent correct; I'm not saying that the work is wrong from its foundational assumptions. 

Hey @fgenie !

The model is what generates the outputs, you need to condition the GLM on either the output or what generated the outputs. In the causal graph above we don't have have the model's outputs so the preference is caused by the model and the instruction (alternatively you can add a node such that model + instruction -> model_output -> preference). You could then train a model to take in the output and predict the preference, but that would require a more complex (.e.g LLM based) pipeline as is not necessary from a causal perspective given that we only care about ranking models rather than ranking outputs.

Sounds good. It is actually the GLM that re-evaluate the model to preference correlation not the other bias factors. My question comes from considering segment-wise comparison case that consider blind test of model outputs not the system level correlation which is the case. Thx for your great work!
Repository: tatsu-lab/alpaca_eval
Issue: Unexpected low judge preference for some prompts
Root cause: External Dependency Breakage

I have been looking a bit into the data available (thank you so much for sharing everything so openly, this is so useful).

When doing so, I saw that some prompts are almost always won by the reference. However, when looking at the provided answers, I can really not see why the judge would prefer one version compared to the other.

For instance, for the prompt: "Enter the words that satisfy the given condition. 5 Countries that Start with S", we get the following json entry for Llama3-70B (from [this file](https://github.com/tatsu-lab/alpaca_eval/blob/main/results/Meta-Llama-3-70B-Instruct/weighted_alpaca_eval_gpt4_turbo/annotations.json)):

```
    "instruction":"Enter the words that satisfy the given condition.\n\n5 Countries that Start with S",
    "output_1":"Sure, here are five countries that start with the letter 'S':\n\n1. Spain\n2. Sweden\n3. Switzerland\n4. Syria\n5. Serbia",
    "generator_1":"gpt4_1106_preview",
    "dataset":"selfinstruct",
    "output_2":"Here are 5 countries that start with the letter S:\n\n1. Spain\n2. Sweden\n3. Switzerland\n4. South Africa\n5. Sri Lanka",
    "generator_2":"Meta-Llama-3-70B-Instruct",
    "annotator":"weighted_alpaca_eval_gpt4_turbo",
    "preference":1.0000426475,
```

However, this sounds suspicious to me that the judge would select the first answer (or the second) with such a high probability given that they are both very hard to distinguish.

The same behavior happens with most other models, even when providing almost identical answers, they get a very low preference compared to the baseline (the average preference for this prompt is 1.009933). 

Do you know what could be causing this? (I will look a bit myself in the meantime)

Great question, I've been seeing this type of calibration issues recently so I think that may be due to OpenAI's new models. A few things to note: 
1. instruction following models are typically miscalibrated (especially if trained to optimize some RL objective) given that the loss is not strictly proper, and in most real cases the optimal policy will actually be deterministic => the model is trained to not give good logprobs. My guess is that they changed in the last few months the instruction following training procedure, or some postprocessing/calibration they may do on the logprobs.
2. Using logprobs that are calibrated is most useful to decrease the standard error in AlpacaEval 2.0 (i.e improve statistical efficiency) but  the rankings with and without weighting are not that different . So I don't think it's a major concern.

That said I would love to see why that happens so please let me know if you find out.
Repository: tatsu-lab/alpaca_eval
Issue: Question about 805 eval examples

Hi, thanks for your excellent work, especially the great efforts on designing and evaluating the evaluators. 

Compared to these well-designed details, the 805 eval instructions does not seem to have much explanation. The alpacafarm paper only provides the root verb distribution and the source of the instructions. I would like to ask if the topics of these instructions are carefully selected, such as whether they cover mathematics, coding, reasoning, etc., and could you provide some principles on building the eval instructions set? 

Thanks a lot!

Thanks for the question. To construct the evaluation set, we essentially gathered all human-written open-ended instruction datasets that had been released at the time - this resulted in 252 instructions from self-instruct eval, 188 from Open Assistant eval, 129 from Anthropic's helpful base, 80 from Vicuna eval, and 156 from Koala eval. We then validated this mixture by comparing performance on it to performance on live user questions from the Alpaca demo (Fig 6 in the AlpacaFarm paper). This correlation result was the main principle by which we decided not to filter the dataset further.

To answer your question more specifically, we did not perform any additional filtering or cleaning on top of the questions we aggregated. We don't have a careful breakdown of the topics along the lines you suggested, but one could look at the documentation for each of the source datasets to find out more information about this.
Repository: tatsu-lab/alpaca_eval
Issue: The code for computing instruction difficulty
Root cause: Documentation Deficiency

Hi, I'm wondering if the code for computing instruction difficulty could be added to the repo?

Also, I was wondering why the "preference" for each annotation is a logit from what seems to be 0.0 to 1.0 whereas the ranking obtained from the annotator (gpt4) is discrete [1, 2].

See #346 I added a notebook [here](https://github.com/tatsu-lab/alpaca_eval/blob/main/notebooks/instruction_difficulty.ipynb) for computing instruction_difficulty

the ranking is not discrete in AlpacaEval 2.0: we use the log probs of generating the token of the preferred model to compute a continuous ranking. Here's the [code](https://github.com/tatsu-lab/alpaca_eval/blob/aecedcef035a65b6cc7bff527c47e904072d95fe/src/alpaca_eval/completion_parsers.py#L236) for computing the continuous ranking based one the logprobs of the mode's preferences
Repository: tatsu-lab/alpaca_eval
Issue: confused about openai API
Root cause: Documentation Deficiency

When I actually invoke the OpenAI interface using a third-party interface, the format of the responses I get back will change accordingly. Could you tell me if the file I need to modify is https://github.com/tatsu-lab/alpaca_eval/blob/2e8c5f3fa4f2b9792d9c1f4aeb5491b581b1ddf4/src/alpaca_eval/decoders/openai.py#L181?

Yes that's what you need to change. 


Alternatively (if you don't use OpenAI code at all) then create a new decoder like the ones here: https://github.com/tatsu-lab/alpaca_eval/tree/2e8c5f3fa4f2b9792d9c1f4aeb5491b581b1ddf4/src/alpaca_eval/decoders and change this line in your configs `fn_completions: "openai_completions"`
Repository: tatsu-lab/alpaca_eval
Issue: Preference doesn't match log_probs in `annotations.json`
Root cause: Documentation Deficiency

Hi! Thank you for maintaining such a valuable repository for academia and research!

I have a question regarding interpreting the preference value in the `annotations.json` file.

Let's say that we have the following annotated entry in the `annotations.json` file:

```json
{
    "instruction":"What year was the Yamato Battleship built?",
    "output_1":"The Yamato battleship, which was ...",
    "generator":"gpt4_1106_preview",
    "dataset":"helpful_base",
    "output_2":"The Yamato Battleship, also known as the ...",
    "generator_2":"my_custom_trained_model",
    "annotator":"weighted_alpaca_eval_gpt4_turbo",
    "preference":1.0000048889,
    "raw_completion":{
      "finish_reason":"length",
      "index":0,
      "logprobs":{
        "content":[
          {
            "token":"M",
            "bytes":[
              77
            ],
            "logprob":-0.0000147365,
            "top_logprobs":[
              {
                "token":"M",
                "bytes":[
                  77
                ],
                "logprob":-0.0000147365
              },
              {
                "token":"m",
                "bytes":[
                  109
                ],
                "logprob":-12.228549
              }, ...
            ]
          }
        ]
      },
      "message":{
        "content":"M",
        "role":"assistant",
        "function_call":null,
        "tool_calls":null
      },
      "text":"M",
      "total_tokens":463.0
    },
    "price_per_example":0.00463,
    "time_per_example":0.1736203954
  }
```

From my understanding, the `annotations.json` file contains de-randomized samples, in which the token "M" always points to the generator_2's response, and the token "m" always points to the reference baseline generator's response.

In all cases, the preference should be computed as `2 - exp(token_m_logprob) / (exp(token_m_logprob) + exp(token_M_logprob))`

However for the annotated sample above, the preference value `1.0000048889` seems to be miscalculated in that it computes `2 - exp(token_M_logprob) / (exp(token_m_logprob) + exp(token_M_logprob))`. This can be cross-validated by the fact that the preference points closer to 1, even though the annotator has picked the token "M" (generator_2's response) as the preferred samples.

Is this a bug, or am I misinterpreting the way `annotations.json` file?

Thank you so much! 


@YJWon99 the `raw_completion` contains the annotations before de-randomization so `M` and `m` are not extractable. See more information about `annotations.json` here: https://github.com/tatsu-lab/alpaca_eval?tab=readme-ov-file#interpreting-annotations 
Feel free to open a PR to improve the README if that isn't clear!

Note that the seed for the randomization is a hash of the instruction, so you can already reconstruct the randomization post-hoc and it will be the same for all models. Ideally, we would add a `is_shuffled` in the annotations.json ...
Repository: tatsu-lab/alpaca_eval
Issue: Why is max_num_seqs allowed here?
Root cause: Interface Contract Mismatch

https://github.com/tatsu-lab/alpaca_eval/blob/834f8d5e7d4b780acfdb1534289bd78a588e470d/src/alpaca_eval/decoders/vllm_local.py#L63

Why is max_num_seqs allowed here?

Especially when they aren't present in the SamplingParams init function here? https://github.com/vllm-project/vllm/blob/c96fc067479453b02e92d9378eeeaebb6b3816de/vllm/sampling_params.py#L31

thanks, it's fixed  here: https://github.com/tatsu-lab/alpaca_eval/pull/338
it should be given to the model_kwargs
Repository: tatsu-lab/alpaca_eval
Issue: tensor_parallel_size can not work
Root cause: Configuration Error

<img width="1302" alt="image" src="https://github.com/tatsu-lab/alpaca_eval/assets/83741666/879a4b09-9977-4f04-8cd5-776250c15b95">
<img width="521" alt="image" src="https://github.com/tatsu-lab/alpaca_eval/assets/83741666/1d00164c-9361-4471-833e-835175fe7b2a">

I set the tensor_parallel_size=2 in the config, but it does not work.

update the package with `pip install -U alpaca_eval`, this is only for the last version
Repository: tatsu-lab/alpaca_eval
Issue: Post-analysis of the annotations
Root cause: Documentation Deficiency

Since the two outputs are randomly ordered during evaluation, it is hard to conduct post-analysis of the evaluation results. Adding extra fields such as `output_1_source, output_2_source, data_source` in the output json file would be great.

Hi @haorannlp, could you be more specific about what these fields would entail? To clarify, the randomization is seeded by the instruction (https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/annotators/pairwise_evaluator.py#L755-L767), so it is possible to exactly reconstruct the order in which the outputs were fed into the annotator model.

If you would like this randomization order information included in the output json file, we would be happy to accept a PR for this. Or let me know if this did not answer your question.

Closing this issue as there's been no followup - assuming it has been resolved. Feel free to re-open if need be.
Repository: tatsu-lab/alpaca_eval
Issue: How to solve the problem of null appearing in the evaluation results? Thank you very much
Root cause: Validation Gap

<img width="951" alt="image" src="https://github.com/tatsu-lab/alpaca_eval/assets/83741666/140c6c51-5a61-4826-8b62-2b09b0479377">

Hi, I noticed the there are many preference null in annotations.json. Is this norm?

I also see the have the same error :(. I don't think this is normal but I do not know what would have caused this. 

I'm facing the same error now, this may lead to the "n_total" being less than 805 in the final output but I don't know how to fix this. 

`null` means that there was an error while annotating. By default it will try 5 times and otherwise it will just give up and save the null to say that it wasn't able to annotate.

One thing to keep in mind is that those `null` will also be cached by default meaning that if you try again annotating again it won't actually retry. You can avoid that by using `--is_store_missing_annotations=False`
Repository: tatsu-lab/alpaca_eval
Issue: ERROR:root:Error while parsing completion: 
Root cause: Validation Gap

<img width="1290" alt="image" src="https://github.com/tatsu-lab/alpaca_eval/assets/83741666/71efaccd-680e-4b2e-aafb-8f5e42bffc46">

Could you please tell me there is a "ERROR:root:Error while parsing completion" during the evaluation

This seems that the model generated a JSON that can't be decoded. I think if you sample a second time it should work. Feel free to re-open otherwise!
Repository: tatsu-lab/alpaca_eval
Issue: Evaluating a self hosted LLM through an API.
Root cause: Documentation Deficiency

I am trying to evaluate a finetuned LLM with Llama as the baseline and alpaca_eval_gpt4_turbo as the auto annotator. My finetuned LLM is not hosted on hugging face and can only be accessed via an API. How do i execute this? 

I see two questions.

# Evaluating an API model 
Option 1:  you can generate the outputs using your own code as specified in the [readme](https://github.com/tatsu-lab/alpaca_eval/tree/aecedcef035a65b6cc7bff527c47e904072d95fe?tab=readme-ov-file#evaluating-a-model). 

Option 2: If your API provider is compatible with OpenAI client you can also use our openai code and change the base URL e.g. [here](https://github.com/tatsu-lab/alpaca_eval/blob/2e8c5f3fa4f2b9792d9c1f4aeb5491b581b1ddf4/src/alpaca_eval/models_configs/Qwen1.5-14B-Chat/configs.yaml). Then you don't need to code anything besides the config.

# Changing the baseline to Llama

There are two ways of computing the score with a different baseline.

Option 1: use the GLM to predict the preference that the LLM-judge would have preferred as shown in figure 5 of the paper. You can do that easily using `--metric_kwargs "{'baseline':'Meta-Llama-3-8B-Instruct'}"`. 

Option 2: actually annotate using a different baseline which can be done using `--reference_outputs="<path_to_llama_outputs>.json" --metric_kwargs "{'glm_name':'length_controlled_noreg'}"` .
Repository: tatsu-lab/alpaca_eval
Issue: Discrepancy between alpaca leaderboard and Chatbot arena ELO
Root cause: Documentation Deficiency

Hello, I wanted to ask if there is any known regression of the correlation of chatbot arena elo scores and alpaca length controlled evaluations (from 0.98). I was using the alpaca-bench for a project and any comments on this would be helpful.

Here are some troubling instances:
Chatbot Arena: https://arena.lmsys.org/
Leadeboard: https://tatsu-lab.github.io/alpaca_eval/
Scores (As of 24 June, 16:00 UTC)
| Model | Alpaca Eval LC Win rate | Chatbot ELO |
| ------| ------ | ------ |
| GPT-4o | 57.5% | 1287 |
| Yi-Large | 51.9% | 1240 |
| LLaMA-3-70B-Instruct | 34.4% | 1207 |
| Gemini-Pro (Bard) | 24.4% | 1208 |
| Qwen-1.5-14B-chat | 23.9% | 1109 |
| LLaMA-2-70B-chat | 14.7% | 1093 |


Both the columns tell different stories about model's abilities and performance gaps. Can we still use Alpaca Eval as a proxy for Chatbot arena ELO?
I understand that differences can arise due to different sampling configurations used when doing inference using this model in chatbot arena, whereas these evals are performed with fixed hyperparameter settings, but is there any other reason?

Thanks in Advance!

Hi @Varun221 ! The only model that seems different is `LLaMA-3-70B-Instruct` vs `Gemini-Pro (Bard)` and this is because Gemini in AlpacaEval is Gemini 1.0 but in ChatBot arena I believe that it's 1.5 (?) 

Note that the absolute differences are not directly comparable as those are different metrics (win rate vs ELO). You can also get ELO scores from win rates but I would suggest only using AlpacaEval for comparing the ranking of models (which is what the 0.98 Spearman correlation measures) rather than the absolute scores.
Repository: tatsu-lab/alpaca_eval
Issue: ValueError: Trailing data
Root cause: Validation Gap

![image](https://github.com/tatsu-lab/alpaca_eval/assets/113493692/d05d5cab-98b6-469e-abd8-7f5006bc7885)
Did I do something wrong?

the JSON model_outputs doesn't seem well formatted. Please double check (especially the end of that file). Feel free to reopen if there are issues

I meet the same problem. Though I can load the `model_outputs` data with `pandas.read_json()` successfully, it still raise `ValueError: Trailing data` with `alpaca_eval` :(

I had the same issue but below command helped.
(*: your downloaded location to alpaca_eval)
1. `rm */alpaca_eval/evaluators_configs/weighted_alpaca_eval_gpt4_turbo/annotations_seed0_configs.json`
2. `pip uninstall alpaca-eval`
3. `pip install alpaca-eval`

Same problem here.
Repository: tatsu-lab/alpaca_eval
Issue: AssertionError
Root cause: Validation Gap

I really appreciate to your last reply.
But now I got following AssertionError.
What did I do wrong this time?
![image](https://github.com/tatsu-lab/alpaca_eval/assets/113493692/349e458d-5855-4d7b-afd1-ae8826c7719f)

![image](https://github.com/tatsu-lab/alpaca_eval/assets/113493692/67e20693-e9ae-4cc9-9d20-ac700964b2f9)
Repository: tatsu-lab/alpaca_eval
Issue: Encountering Error about cannot reindex on an axis with duplicate labels
Root cause: Validation Gap

After loading all the annotation chunk, it popped the ValueError
![image](https://github.com/tatsu-lab/alpaca_eval/assets/109973290/e22a5b01-11ca-4405-8413-e10778d85c25)

WHAT should I do?

> After loading all the annotation chunk, it popped the ValueError ![image](https://private-user-images.githubusercontent.com/109973290/344991800-e22a5b01-11ca-4405-8413-e10778d85c25.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA0Mjg3MzgsIm5iZiI6MTcyMDQyODQzOCwicGF0aCI6Ii8xMDk5NzMyOTAvMzQ0OTkxODAwLWUyMmE1YjAxLTExY2EtNDQwNS04NDEzLWUxMDc3OGQ4NWMyNS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOFQwODQ3MThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hZTAwNzEyN2ZjOWIwYmVlOWVjNDhhMWRkZjVlNjFlZjg1MTVkMzZjYzcyOWNhOWYyOWQzYTEzOGZiZDNiYmE2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.mcyCNvtNvltliCY_u8nUEjPHfEwf6ZrKzLZJYKG9rm4)
> 
> WHAT should I do?

Most likely, the two JSON files you provided contain either NAN or None. Please check them

Your input dataframe seems not well formatted. Maybe you have duplicated column names?