zhiminy/EvalEng huggingface dataset:
Repository: tatsu-lab/alpaca_eval
Issue: Negative Correlation while using `alpaca_eval analyze_evaluators`
Root cause: Documentation Deficiency

I am currently analyzing the human agreement of the evaluator llama3-70b-instruct based on my research interests. Here is my approach:

Given one prompt (P) and two responses (A and B), I use two separate prompt+response (P+A and P+B) pairs as inputs to llama3.
I calculate the average log likelihood of responses (avg_logp of A and B) as scores. This is the same as [SimPO](https://arxiv.org/abs/2405.14734). 
The response with the higher score is considered the preferred one.

In this way, I have found that the human agreement is approximately 60%. However, both the Spearman correlation and Pearson correlation are negative.
<img width="1264" alt="alpaca_eval_2_llama3i_simpo" src="https://github.com/tatsu-lab/alpaca_eval/assets/137851425/dbd3f57a-ffa5-4089-b0a1-404232644321">


I am unsure how these two metrics function and why they might be negative in this context. Could you kindly help me understand this?

Mhh not sure why it's negative, also your variance is really low. My guess is that you haven't validated on all the data, can you check `# parsed` in `src/alpaca_eval/leaderboards/evaluators/evaluators_leaderboard.csv`

The correlation and all those columns are explained [here](https://github.com/tatsu-lab/alpaca_eval?tab=readme-ov-file#evaluators).

<img width="1067" alt="Screenshot 2024-07-18 at 19 54 51" src="https://github.com/user-attachments/assets/fbe247bf-edb3-48cf-92cb-a51e9dc784b7">
Repository: tatsu-lab/alpaca_eval
Issue: Argument `do_sample` is removed from `vllm_local_completions`
Root cause: Interface Contract Mismatch

Hi! I'm updating new model results to the repo, but have found the recent updates of the repo might lead to some reproducibility issues. 

- According to this [PR](https://github.com/tatsu-lab/alpaca_eval/pull/338/commits/7fa3b3de33e203d62434e1bbaac056d79018762d#diff-cf19f0575f27d7400500c9e14398d36a428ee7dbc08e9b8aa6170f889d9ef18eL65), in a previous version, do_sample set use_beam_search to True, but this has been removed in the latest PR. Why is this the case? How would setting use_beam_search to True after initializing SamplingParams affect the decoding?
- Now that do_sample is removed from the vllm_local_completions function, including do_sample in the config file will cause an error during decoding with vllm. This could lead to compatibility issues with existing model configs.

I could have missed something here, and would love to hear your thoughts! Thanks!

Thanks for letting me know: you are absolutely correct. That was a mistake on my side while I was simplifying the vllm code! That said I don't think we should use `do_sample` to mean`use_beam_search`. I changed the old configs to use `use_beam_search`. Lmk if you have other issues!
Repository: tatsu-lab/alpaca_eval
Issue: assert output_path is not None and name in leaderboard
Root cause: Validation Gap

When I use the following shell script to launch a evaluation, I get this error: `AssertionError`
```shell
export OPENAI_API_KEY="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
alpaca_eval --model_outputs "autodl-tmp/alpaca_eval_results/mistral-swpo03/outputs.json" --is_recompute_metrics_only True
```

![image](https://github.com/user-attachments/assets/1f477e60-eb0f-4e8f-a2b0-0e40ed802c51)


I meet the same problem, did you solve it?
Repository: tatsu-lab/alpaca_eval
Issue: Why do we set do_sample to true during evaluation?
Root cause: Documentation Deficiency

https://github.com/tatsu-lab/alpaca_eval/blob/d7747c36934ad0b216fdb263d59e4e554411a739/src/alpaca_eval/models_configs/alpaca-7b/configs.yaml#L11

I saw that do_sample is set to True for almost all the model configs, and I'm curious why do we add such randomness to the evaluation process instead of just getting determinant answers?

Most models perform better with sampling. We do not have any requirements on the model/system being evaluated so people submitting the models can choose what parameter to set.
Repository: tatsu-lab/alpaca_eval
Issue: Extra '\n' in the prompt of Llama-3-Instruct-8B-SimPO
Root cause: Configuration Error

https://github.com/tatsu-lab/alpaca_eval/blob/d7747c36934ad0b216fdb263d59e4e554411a739/src/alpaca_eval/models_configs/Llama-3-Instruct-8B-SimPO/prompt.txt#L3

There seems to be three '\n' s at the end of the prompt, but the official tokenizer only add two '\n'. 

From the github website, the prompt file has 5 lines. However, if you pull the repo to local (or download the .zip) and open Llama-3-Instruct-8B-SimPO/prompt.txt, there will be 6 lines. I'm not sure about the cause of it. My OS systems are Mac and Linux.

I’m not familiar with that level of detail for this model. Tagging @xiamengzhou as this is a community addition

@gohsyi I only used two \n at the end for the llama checkpoints, but it's very strange because I noticed a similar issue with the [zephyr template](https://github.com/tatsu-lab/alpaca_eval/blob/d7747c36934ad0b216fdb263d59e4e554411a739/src/alpaca_eval/models_configs/zephyr-7b-alpha/prompt.txt). If you download the file, you'll see an extra \n. I'm not sure why this happens, but I hope this extra \n won't significantly affect the decoding process.

Closing although I'm not sure why that happens either...
Repository: tatsu-lab/alpaca_eval
Issue: Questions about the 'use_beam_search' configuration.
Root cause: Documentation Deficiency

Hello,

I notice that in many models' configs.yaml, when use_beam_search is set to true (#373), the temperature is also set to a non-zero value. This causes conflicts when I use it with vLLM. For vLLM, when use_beam_search is set to true, the best_of parameter needs to be set to represent the beam size (which is missing in the configs.yaml), and the temperature should be set to 0. My vLLM version is 0.5.

Since use_beam_search and temperature parameters appear together in the configs.yaml, I am unsure which one to disable when reproducing the results. I would like to know what the correct settings should be.

Disclaimer: All the vLLM configs are community additions so I haven't followed those closely nor tried to replicate those.

What do you mean by "and the temperature should be set to 0"? are you getting an error? If not then that's what you should use for replicating results, and it just means that someone added the temperature configs even if it's not used. Others probably then copy pasted the configs.

If you want to make sure (e.g. it may be different with different version of vLLM), I would simply suggest running both beam search and temperature sampling and selecting the results that are closest to the one on the leaderboard. If you do so, could you please comment on this GH issue for future documentation? thanks!
Repository: tatsu-lab/alpaca_eval
Issue: Using local models as evaluators
Root cause: Documentation Deficiency

Hi there,

As GPT-4 (and some other API-based models) might be too expensive for testing purposes or for running thousands of experiments in research works, I would like to know if is it possible to use a local model (e.g., Llama3-70B deployed using **textgenwebui** or **vLLM** as a serving engine) as an evaluator. 

If so, could you please provide me with additional details on how to do this (e.g., setting up the configs.yaml file)?

Thanks in advance!


Here's an example: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/alpaca_eval_vllm_llama3_70b_fn/configs.yaml

You just need to change the path to the local model, and optionally the decoding configs. This uses vLLM

That's great! Thank you!
Repository: tatsu-lab/alpaca_eval
Issue: Non-reproducible results
Root cause: Documentation Deficiency

Dear maintainers,

I am running AlpacaEval 2.0 using GPT-4o-mini as a judge. Every time I run the evaluation I get a different result. Is that a reason for that and how can I fix it?

Thanks!
Felipe


you shouldn't because evaluations are cached. If you remove caching it is normal that results are not deterministic given that the API models are not deterministic. But AlpacaEval's standard errors are low and so results shouldn't be too different. 

Can you share the results you get? In particular please share the entire row that you see in the terminal (I would like to see the win-rates, the standard errors, and the number of annotated examples)



I also have different results when calling the same model each time.

Same issue. 
I am running AlpacaEval with the **default annotator** (weighted_alpaca_eval_gpt4_turbo) to evaluate the outputs in `./results/gemma-2-9b-it-SimPO/model_outputs.json`.

I plotted the histogram of the absolute differences of the preference scores below:

![Screenshot 2024-08-07 at 7 26 33 PM](https://github.com/user-attachments/assets/41d3b87d-d03b-4edb-b59e-1769d121bdb4)


the generated leaderboard (I reformated the csv into markdown):

|Model | win_rate| standard_error| n_wins| n_wins_base| n_draws| n_total| discrete_win_rate| mode| avg_length| length_controlled_winrate| lc_standard_error|
|---|---|---|---|---|---|---|---|---|---|---|---|
|gemma-2-9b-it-SimPO| 65.86422561532919| 1.423459922555078| 540| 264| 1| 805| 67.14285714285714| community| 1833| 72.3508446939842| 0.5167873784867067|
|**gemma-2-9b-it-SimPO-rerun**| 65.90176174567401| 1.4214682843719195| 550| 254| 1| 805| 68.38509316770185| community| 1833| 72.23893471246227| 0.5333697317163058|
|openpipe-moa-gpt-4-turbo-v1| 63.15493451236265| 1.422980098799326| 515| 283| 7| 805| 64.40993788819875| community| 1856| 68.37866250336802| 0.7309418614587613|
|gemma-2-9b-it-DPO| 65.35922380122982| 1.402802336467638| 536| 268| 1| 805| 66.64596273291924| community| 2016| 67.6620382198043| 0.6605613085864308|



A difference of 0.04 win rate is absolutely normal and comes from stochasticity of the API model even with temperature 0 (due to many reasons like GPU stochasticity, batching stochasticity, and other inference optimisations from OpenAI)

Note that if you use the same model name then you will hit AlpacaEval’s cache and thus not see any stochasticity (because you won’t relabel anything)
Repository: tatsu-lab/alpaca_eval
Issue: all the following results are about AlpacaEval 1.0 and have not been updated since?

Check https://github.com/tatsu-lab/alpaca_eval?tab=readme-ov-file#additional-analysis-and-plots but found results about Alpaca 2.0:
![image](https://github.com/user-attachments/assets/a207285b-f6fd-4735-be1d-037a4268224b)


thanks updated
Repository: tatsu-lab/alpaca_eval
Issue: After I finished evaluating Meta-Llama3-8B-Instruct

After I finished evaluating Meta-Llama3-8B-Instruct, I found that it didn't match the leaderboard. Can you tell me why this is? 
Win rate ~39、LC-win rate ~40、length ~1900

Did you regenerate the outputs? I used Togeter.ai for llama and they do some quantization of the weights so it may be that..

closing as no answer.
Repository: tatsu-lab/alpaca_eval
Issue: I can't duplicate your results locally
Root cause: External Dependency Breakage


![image](https://github.com/user-attachments/assets/2525b853-6652-4d7a-827c-382f34bb0a19)

Model results use the results in the results directory. But the difference in scores between the results is too big

Did you regenerate the outputs or is that on the same outputs? If the model's outputs are very different (OpenAI API changed a good amount) there's nothing we can do 

closing as no answer.
Repository: tatsu-lab/alpaca_eval
Issue: Question about Leaderboard

First of all, thank you very much for your work. I have a question: based on the current leaderboard, it seems that DPO/SimPO can improve a LLM's performance by 20%-40%, and even many smaller models with 7B-9B parameters seem to far exceed GPT-4 turbo in performance. I find this quite unreasonable, as it appears that Alpaca eval is overly biased toward these DPO/SimPO-tuned models. What is your take on this?








I think that many models that perform DPO/SimPO specifically aim to improve those benchmark numbers (AlapcaEval / MTBench / ...), while the other models you are comparing with (the company models) don't optimize for those benchmarks.
Repository: tatsu-lab/alpaca_eval
Issue: Will the annotator alpaca_eval_gpt4_turbo_fn also change?
Root cause: External Dependency Breakage

May I confirm if the annotator alpaca_eval_gpt4_turbo_fn will also consistently change and affect the final (length control) win rates?

Yes, that depends on OpenAI API and there's nothing we can do about the model silently changing.
Repository: tatsu-lab/alpaca_eval
Issue: AttributeError: module 'alpaca_eval.metrics' has no attribute 'get_length_controlled_winrate'
Root cause: Environment Incompatibility

I installed `alpaca_eval==0.6.2` to ensure consistency with my previous results. However, I encountered an issue. And `alpaca_eval==0.6.2` performs well on another Linux platform.

Traceback (most recent call last):
  File "/home/xxx/anaconda3/bin/alpaca_eval", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/xxx/anaconda3/lib/python3.11/site-packages/alpaca_eval/main.py", line 610, in main
    fire.Fire(evaluate)
  File "/home/xxx/anaconda3/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/anaconda3/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/anaconda3/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/anaconda3/lib/python3.11/site-packages/alpaca_eval/main.py", line 175, in evaluate
    fn_metric_ = getattr(metrics, fn_metric)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'alpaca_eval.metrics' has no attribute 'get_length_controlled_winrate'

Do you still have that error @sunjie279 ? sounds like an installation & env issue
Repository: tatsu-lab/alpaca_eval
Issue: ValueError: Trailing data
Root cause: External Dependency Breakage

I run Alpaca Eval 2 successfully 2 months ago, but this time when I run it, using the same code in the same env, there is an error: 
"ValueError: Trailing data" 

It should not be the problem of my outputs.json file since it has not changed since the last time. 

Is there any one meeting similar problems here? 


See : https://github.com/tatsu-lab/alpaca_eval/issues/353#issuecomment-2381406436

I think pandas changed their json reading between different versions

Hi, thanks for your reply. 
I degrade pandas to 2.2.2/2.2.1, pip uninstall alpaca-eval, and then pip install alpaca-eval. 

But, still having this error. 

I guess there should be more people having this error. Do you know how to fix it?
Repository: tatsu-lab/alpaca_eval
Issue: Possibility of Re-running Only Failed Queries After Rate Limit Reached
Root cause: Documentation Deficiency

Hi,

Thanks for the brilliant evaluation framework! Recently, I encountered an issue where around 40 queries failed due to rate limits and the maximum number of retries, while approximately 760 queries were successfully processed. It would be very cost-effective if there were an option to re-run only the failed queries instead of the entire batch.

Is there a way to achieve this?

Thanks!

I found a potential solution! It seems we can manually remove entries from the cache file where `raw_completion` is `null`. 

I'm using `weighted_alpaca_eval_gpt4_turbo` as my evaluator, so the cache file is located at `evaluators_configs/weighted_alpaca_eval_gpt4_turbo/annotations_seed0_configs.json`.

However, I'm not entirely sure if this approach is correct. Am I on the right track?

Yes that works. Alternatively you should be able to achieve it with `is_store_missing_annotations=False` see [here](https://github.com/tatsu-lab/alpaca_eval/blob/0b4af7601853e718b72a8da6a35460bda3015754/src/alpaca_eval/annotators/base.py#L63C5-L63C33)
Repository: tatsu-lab/alpaca_eval
Issue: TypeError: LogisticRegressionCV.fit() got an unexpected keyword argument 'groups'
Root cause: External Dependency Breakage

Error when execute alpaca_eval --model_outputs 'results/Meta-Llama-3.1-8B-Instruct/model_outputs.json'

File "/home/anaconda3/envs/factory/lib/python3.12/site-packages/alpaca_eval/metrics/glm_winrate.py", line 105, in get_length_controlled_winrate
    model = fit_LogisticRegressionCV(
  File "/home/anaconda3/envs/factory/lib/python3.12/site-packages/alpaca_eval/metrics/glm_winrate.py", line 354, in fit_LogisticRegressionCV
    model.fit(
  File "/home/anaconda3/envs/factory/lib/python3.12/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
TypeError: LogisticRegressionCV.fit() got an unexpected keyword argument 'groups'

[https://github.com/tatsu-lab/alpaca_eval/blob/0b4af7601853e718b72a8da6a35460bda3015754/src/alpaca_eval/metrics/glm_winrate.py#L348C13-L348C24](url)
‘groups’ seems to be incompatible with LogisticRegressionCV? How to use it correctly?

Hi, I have the same problem. Did you solve this?

can't reproduce, try `pip install -U scikit-learn`
Repository: tatsu-lab/alpaca_eval
Issue: Can we have a self-hosted evaluator for avoiding loading a large model every evaluating?   
Root cause: Documentation Deficiency

Recently, I set out to evaluate a large number of model outputs—around 1,000 files—using the llama3-70b-instruct model as the evaluator. However, I found existing solutions to be less than ideal for the following reasons:

1. Using an external host (e.g., from api.together.xyz/v1) can be prohibitively expensive.
2. Loading a 70B model repeatedly for each evaluation incurs significant I/O overhead and wastes valuable time.

To address these issues, I propose a more efficient approach:

1. Launch a local LLM host using a library like vllm. For instance, the following command could be used to serve the model locally:
```
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
    --dtype auto \
    --host '127.0.0.1' \
    --port 6379 \
    --max-model-len 700 \
    --tensor-parallel-size 4 \ # I have four gpus for evaluation
    --enforce-eager \
    --gpu-memory-utilization 0.95 \
    --chat-template ./alpaca_eval_self_hosted_llama3_70b_fn/alpaca_eval_fn.txt
```

2. Request evaluations or annotations from the locally hosted model by running:
`alpaca_eval --model_outputs "your_model_outputs.json" --annotators_config "./alpaca_eval_vllm_llama3_70b_fn"
`
with the `alpaca_eval_vllm_llama3_70b_fn`:
```
alpaca_eval_self_hosted_llama3_70b_fn:
  prompt_template: "alpaca_eval_self_hosted_llama3_70b_fn/alpaca_eval_fn.txt"
  fn_completions: "openai_completions"
  completions_kwargs:
    requires_chatml: True
    model_name: "meta-llama/Meta-Llama-3-70B-Instruct"
    max_tokens: 100
    temperature: 0
    client_kwargs:
      base_url: 'http://localhost:6379/v1'
      api_key: 'i_dont_need_one'
  fn_completion_parser: "ranking_parser"
  batch_size: 1
```

However, I find I cannot access the local host 70B model using `alpaca_eval --annotators_config xxx`. (I can access to the local host model using OpenAI client interface) Seems there are something wrong in annotators_config. Do you have suggestion?  Thanks in advance. 

I just realize that achieving this would require a lot of efforts. Seems the best solution is to reload the evaluator each time the evaluation is run. Hope you don't mind if I close this issue.
Repository: tatsu-lab/alpaca_eval
Issue: why judge has a default temperature of 1？

it doesn't make sense to make winrate even more volatile.

which judge/annotator are you referring to? for the default one `weighted_alpaca_eval_gpt4_turbo` we use logprobs not temperature based sampling so there's no stochasticity from sampling. 

As noted from the comment here https://github.com/tatsu-lab/alpaca_eval/blob/474386e3be4159b3ebe8f755071e706c47ea7f4f/src/alpaca_eval/evaluators_configs/weighted_alpaca_eval_gpt4_turbo/configs.yaml#L7 
temperature shouldn't make any difference with log probs. I used 1 just in case the API changes at some point (we want to access the real log probs with the temperature that was used during training)

> which judge/annotator are you referring to? for the default one `weighted_alpaca_eval_gpt4_turbo` we use logprobs not temperature based sampling so there's no stochasticity from sampling.
> 
> As noted from the comment here
> 
> [alpaca_eval/src/alpaca_eval/evaluators_configs/weighted_alpaca_eval_gpt4_turbo/configs.yaml](https://github.com/tatsu-lab/alpaca_eval/blob/474386e3be4159b3ebe8f755071e706c47ea7f4f/src/alpaca_eval/evaluators_configs/weighted_alpaca_eval_gpt4_turbo/configs.yaml#L7)
> 
> Line 7 in [474386e](/tatsu-lab/alpaca_eval/commit/474386e3be4159b3ebe8f755071e706c47ea7f4f)
> 
>  temperature: 1 # temperature should be applied for sampling, so that should make no effect. 
> 
> temperature shouldn't make any difference with log probs. I used 1 just in case the API changes at some point (we want to access the real log probs with the temperature that was used during training)

thanks, that makes sense
Repository: tatsu-lab/alpaca_eval
Issue: How to overwrite the leaderboard and reannotate the missing question?
Root cause: Unimplemented Feature Gap

First of all, I have the request to overwrite the leaderboard due to the update of my model, but when I specify --is_overwrite_leaderboard, it will directly load the cache instead of reannotate, because my model name is the same as the one in the leaderboard.

Second, I wonder if there is a way to only reannotate the missing question? Because I found the n_total is slightly smaller than 805, and I just want to reannotate the missing part. 

Thank you for your help!

see https://github.com/tatsu-lab/alpaca_eval/issues/419#issuecomment-2467410687

Thank you for answering my second point, but my first question is remain unsolved, it seems like I can not reannotate a same file with the same generator_name, is this correct?
Repository: tatsu-lab/alpaca_eval
Issue: How is the win rate (not LC) calculated?
Root cause: Documentation Deficiency

Hi all, I understand that the LC win rate will utilize a GLM to do regression to get rid of the length factors, however, I am wondering how is the standard win rate calculated. I found some models on the leaderboard has the exact same number of "n_wins" "n_wins_base" "n_draws", but they end up with a slight different win rate. I think it means that the standard win rate is not calculate by `n_wins / (n_wins + n_wins_base + n_draws)`, Do anyone knows that how the standard win rate is calculated? THANKS!!


For the default annotator in AlpacaEval 2.0 (i.e. `weighted_alpaca_eval_gpt4_turbo`) it's weighted by the probability that the annotator assigns to winning or losing. See https://github.com/tatsu-lab/alpaca_eval/blob/474386e3be4159b3ebe8f755071e706c47ea7f4f/src/alpaca_eval/metrics/winrate.py#L18

What you are describing is called `discrete_win_rate` in the code see https://github.com/tatsu-lab/alpaca_eval/blob/474386e3be4159b3ebe8f755071e706c47ea7f4f/src/alpaca_eval/metrics/winrate.py#L19 for annotators that don't start with `weighted_*` the `preference` is not a probability so `win_rate` and `discrete_win_rate` coincide.