zhiminy/EvalEng huggingface dataset:
Repository: tatsu-lab/alpaca_eval
Issue: Logistic regression for length-controlled winrate

Hi,

I'm trying to understand how the new length-controlled winrate metric works. Is the logistic regression model supposed to be trained on the comparisons of a single model vs the baseline, or all comparisons from all evaluated models vs the baseline?

Hi @normster, the model is trained on the comparisons of a single model. This is important to make sure that the leaderboard doesn't change when adding new models. 
If you want more information check out the pdf below that should be on arxiv on Tuesday. 

[Length_controlled_AlpacaEval.pdf](https://github.com/tatsu-lab/alpaca_eval/files/14895204/Length_controlled_AlpacaEval.pdf)

Let me know if you have more questions

Thank you, makes sense!
Repository: tatsu-lab/alpaca_eval
Issue: Question on Using Character-Level Length
Root cause: Documentation Deficiency

Hello,

I have noticed that when printing tables in the command line, the output for `avg_length` is based on character-level length, as implemented here: 

https://github.com/tatsu-lab/alpaca_eval/blob/a291ea779e35e1e5533e492ac5bde6d958808cdc/src/alpaca_eval/main.py#L160

I have two questions regarding this implementation:
1. Word-level length seems like a more appropriate metric. Why was character-level length chosen instead?
2. For calculating `length_controlled_win_rate`, is character-level or word-level length used? Where is this implemented in the code?

Thank you!

The length is character-based indeed. The number of words is harder to estimate or makes sense outside of English chats (e.g. for coding, or for non-Western languages). That said, I agree that using the number of tokens rather than characters would make more sense. I used characters because it's faster and easier to compute, and for large enough sequences it should only differ in approximately a constant factor (in practice 1 token ≈ 4 characters).

Code for length control: [here](https://github.com/tatsu-lab/alpaca_eval/blob/a291ea779e35e1e5533e492ac5bde6d958808cdc/src/alpaca_eval/metrics/glm_winrate.py#L34) and more information about the method [here](https://github.com/tatsu-lab/alpaca_eval/issues/269#issuecomment-2041227499).
Repository: tatsu-lab/alpaca_eval
Issue: With unstable GPT-4 API, I encounterd a tricky problem
Root cause: Validation Gap

![image](https://github.com/tatsu-lab/alpaca_eval/assets/84949179/fdf2db45-98d3-4390-9663-5f390e3dd77b)
Because my GPT-4 API is unstable, I often encounter the problem above when using \texttt{alpaca_eval --model_outputs ...}. I have checked the file \texttt{/alpaca_eval/decoders/openai.py}, but it really puzzled me. How could I fix the problem without changing my GPT-4 API? 

I'm looking forward to your reply. Thanks very much!

reopen latter
Repository: tatsu-lab/alpaca_eval
Issue: With unstable GPT-4 API, I encounterd a tricky problem
Root cause: Validation Gap

![image](https://github.com/tatsu-lab/alpaca_eval/assets/84949179/2c8ad298-6d41-4830-bb50-5b8c106bb3cc)
Because of my unstable GPT-4 API , I often encounter the problem above when using **alpaca_eval --model_outputs ....**. I have checked the file _alpaca_eval/decoders/openai.py_, but it really puzzled me. How could I fix the problem without changing my GPT-4 API?

I'm looking forward to your reply. Thanks very much!


Note sure what is happening, I've never seen that error and it's hard to debug without being able to replicate..

Do you know if it always comes for the same examples?

Can you try:
1. set `export OPENAI_MAX_CONCURRENCY=1` which will make it easy to debug
2. add after `choices = completion_batch.choices` [here](https://github.com/tatsu-lab/alpaca_eval/blob/a291ea779e35e1e5533e492ac5bde6d958808cdc/src/alpaca_eval/decoders/openai.py#L229) the following:
```
if choices is None:
    print(completion_batch)
```

and then paste again the logs when it errors out? Thanks.

Ohh I see you are not using the base OpenAI but `flag.smarttrot`. This is likely your server error not an AlpacaEval error. If you can reproduce the error with openai link then feel free to reopen.
Repository: tatsu-lab/alpaca_eval
Issue: Question about the GPT-4 API
Root cause: Environment Incompatibility

Thank you for your amazing work!
I use the command 'alpaca_eval --model_outputs xxxx.json' to evaluate the metrics. But I got the following issues:
![image](https://github.com/tatsu-lab/alpaca_eval/assets/94901161/5a38bc57-3f84-470d-9378-bde4c2c63ca4)
This causes the following problems:
![image](https://github.com/tatsu-lab/alpaca_eval/assets/94901161/c94e48ed-c8ed-415e-8610-3c2bb5fbcd04)
It makes only 272 samples be used for calculating the winrate. Is there an issue with my GPT4 API? Thank you very much for your response.

That's surprising given that it works for the other examples. It's very hard to help without being able to reproduce. 

- Does it always error out on the same examples? if so can you send some to me?
- can you try `export OPENAI_MAX_CONCURRENCY=1` and see if the error persits
- can you paste the error/warnings you get during the annotation?

Thanks! 

Thank you for your reply! I will try again and give more information latter.

I have some questions about the dropping of samples during the evaluation of Alpaca eval. Is it normal to have some dropped samples, or is the normal condition that all 805 samples are correct?

Usually we don't drop anything, but it may happen that GPT4 API fails and then you drop 2-3 examples, which we don't care much about. But you are dropping 500 examples here, which is definitely not expected and I wouldn't trust the win-rates you are currently getting.

I think it could be an issue with my API or occurring during the request process. Now I have a unified platform for requesting GPT-4. The process involves uploading a text file with prompts and inputs, and the platform will return the generated results from GPT-4. Then, I utilize the results from GPT-4 to parse and calculate the win rate. Do you have any suggestions on how I should unpack and execute the process you've encapsulated?

![image](https://github.com/tatsu-lab/alpaca_eval/assets/94901161/1c2f2e9f-0e4c-4332-ab86-d5db0b6b68e6)
This is part of the process of requesting GPT-4, and no other errors have occurred. However, the following error subsequently appeared.
![image](https://github.com/tatsu-lab/alpaca_eval/assets/94901161/cbdd25c7-c5be-4166-bcd4-f98a248d2a6b)


Do you have the error if you use the base OpenaAI api? Please try without using `lonlie.plus7`, I can't help for other APIs


This is likely your server error not an AlpacaEval error. If you can reproduce the error with the openai base then feel free to reopen.

Thank you for all your patient responses. I'm currently trying out the openai base API to test the results.

Have you found a solution?

> Have you found a solution?

I used to use an API provided by my lab, which required setting up an openai_api_base. This was a URL that may not have been the official API. When I switched to the official OPENAI GPT-4 API, the problem was resolved.
Repository: tatsu-lab/alpaca_eval
Issue: Confusion in Model Evaluation Results Due to GPT Updates
Root cause: Configuration Error

The actual models pointed to by `gpt-turbo` and ``gpt-preview`` are frequently updated, leading to confusion in the results.

When using the latest `gpt-turbo` (defaulting to gpt-4-turbo-2024-04-09) and claude-3-opus-20240229, the result is 47.38, significantly higher than the 40.39 on the leaderboard.
![image](https://github.com/tatsu-lab/alpaca_eval/assets/37578514/8dd982c0-59df-4346-b5fd-690db1924304)

When using the latest `gpt-preview` (defaulting to gpt-4-0125-preview) and claude-3-opus-20240229, the result is 41.09, which is similar to the 40.39 on the leaderboard.
![image](https://github.com/tatsu-lab/alpaca_eval/assets/37578514/4a80bfad-d78a-413c-8a02-89c9b51c2813)

Based on this, I speculate that **alpaca_eval is using `gpt-preview` for evaluation**.

However, there are two different settings displayed on the webpage.
![image](https://github.com/tatsu-lab/alpaca_eval/assets/37578514/991117fb-c116-4580-a389-f973c44f6942)
![image](https://github.com/tatsu-lab/alpaca_eval/assets/37578514/095d45f9-a523-4c1d-8eef-f39d24b2e4c5)
The first image is after switching to alpaca-eval and then back to alpaca-eval 2.0, the displayed settings are different.

Given the above situation, I suggest:

Avoid using `gpt-turbo` (symbolic links), and instead use specific version numbers like gpt-4-turbo-2024-04-09.

Also, what are the current settings for the leaderboard?

Hi @yifan123, thanks a lot for flagging this. 

What happened is that since gpt-4-turbo came out a few data ago, it means that preview would soon get discontinued. So I tested the annotations of turbo 3 different models to see how much the model is different (I thought it would mostly be the same model). I tested it on three models from Mistral, OpenAI, Contextual and saw little difference so I switched the annotator to gpt-4-turbo. But given your post I tested differences more thoroughly on 15 models and see that actually for a few models this makes a large difference. 

#283 reverted those and reannotated the models that were added in the last few days with preview.

Thanks for your response. Alpaca_eval is a very nice leaderboard, hope to maintain and improve it
Repository: tatsu-lab/alpaca_eval
Issue: How instruction_difficulty feature is obtained



That's actually the only sentence about **instruction_difficulty** feature.
![image](https://github.com/tatsu-lab/alpaca_eval/assets/56322857/4859201e-3e9d-4d34-8d87-436c81b963df)

I can't really understand how author obtained it. Can somebody explain or add information into [notebook](https://github.com/tatsu-lab/alpaca_eval/blob/main/notebooks/length_controlled.ipynb).

Thanks for attention and explanation.


Updated the notebook

> First, let's load the instruction complexity ($\mathbf{w'}_x^T$).
> This was precomputed by fitting the logistic regression as the one above with the following changes:
> - replace the instruction embedding by a one hot encoding of the instruction 
> - tie the weights of the instruction across all the models and fit jointly across all the models
> 
> Here's the equation:
> 
> 
> $$win\_rate(m,b) = \frac{1}{N} \sum_{i=1}^{N} logistic( \mathbf{w}_l[(m,b)]  * tanh(standardized(length(m(x_i)) - length(b(x_i)))) + \mathbf{w}_x*I(x_i)  + (\mathbf{w}_m[m] - \mathbf{w}_m[b]))$$ 
> 
> Here $\mathbf{w}_x$ is shared across all models and quantifies how good the baseline win-rate on a certain instruction is. 
> We then extract this weight and use it as $embedding(x)$ because we want to fit all models disjointly.
> 

Also the paper has more explanation about why: https://arxiv.org/abs/2404.04475
Repository: tatsu-lab/alpaca_eval
Issue: Llama-3-Instruct not using official prompt template?
Root cause: Configuration Error

The evaluation of Llama-3-Instruct does not seem to use the official chat template

Llama-3-Instruct config from `alpaca_eval`
https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/models_configs/Meta-Llama-3-8B-Instruct/configs.yaml#L2

Llama-3-Instruct official chat template
https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/tokenizer_config.json#L2053

We use the together prompt: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/models_configs/Mixtral-8x7B-Instruct-v0.1/togetherai_prompt.txt which is blank because I believe Together.ai adds the prompt template on their side!
Repository: tatsu-lab/alpaca_eval
Issue: Unable to reproduce results
Root cause: Documentation Deficiency

Dear developers,

Thank you for your amazing work with AlpacaEval!

I am failing to reproduce the winrate results for one of the models (Snorkel-Mistral-PairRM-DPO). If I use the results from results/Snorkel-Mistral-PairRM-DPO, I get a winrate of 29.93% vs 30.2% from the leaderboard. I am using the formula (preference-1).mean() suggested in your documentation to compute the score. Could you help me understand why my results do not match for that LLM? Thank you!

Best,
Felipe

Hi Felipe.

This is probably because you have to also deal with equalities. In particular 0 (which means equality when preferences are between 1 and 2) should be mapped to 0.5 (ie average, when the rest is mapped in 0 and 1). This is for annoying historical reasons. Also make sure you remove nans from the average.

Here's the actual function we use: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/metrics/winrate.py#L18
Repository: tatsu-lab/alpaca_eval
Issue: Missing item in results/llama-2-70b-chat-hf
Root cause: External Dependency Breakage

Hi authors,

Thanks for your contributions to this interesting work. However, I found there are only 804 items in ```results/llama-2-70b-chat-hf/model_outputs.json``` while as said there should be totally 805 items in this benchmark. The missing item is ```{'dataset': 'selfinstruct', 
       'instruction': 'A user story is an informal, general explanation of a software feature written from the perspective of the end user or customer. Write a user story for a given software.\n\nGmail'}```

Could you please check and update if possible? Thanks.

This means that either the GPT4 or LLaMA API had an error with decoding/evaluation. This could be because of context limits, API down, or safety flags. If you look at https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/leaderboards/data_AlpacaEval_2/weighted_alpaca_eval_gpt4_turbo_leaderboard.csv you will see that there are a few models for which n_total≠805. 

We try 10 times when there's an error so that was a recurring issue. I'm not planning on trying again as this wouldn't change significantly the score.
Repository: tatsu-lab/alpaca_eval
Issue: Missing result file in notebook
Root cause: Documentation Deficiency

This repo is so cool, thanks for sharing!

I saw that this file is missing: `results/all_annotations_alpaca_eval_gpt4.json` from [this notebook](https://github.com/tatsu-lab/alpaca_eval/blob/main/notebooks/analyzing_evalset.ipynb).

Would it be possible to add it or to document how to get it?

Good catch. It should be this one: [all_annotations_alpaca_eval_gpt4.json](https://github.com/tatsu-lab/alpaca_eval/files/15198046/all_annotations_alpaca_eval_gpt4.json)
Repository: tatsu-lab/alpaca_eval
Issue: What to do if the log prob is not returned?
Root cause: External Dependency Breakage

Thanks for the great repo.

I was using alpaca_eval to evaluate a large batch of custom data, but when I was calling the GPT-4-Turbo API, sometimes, the log probs were not returned, as a result, there are no auto-annotations for the data without log probs, but the costs were incurred. 

Is it possible to add a feature to return the output of these data even though the log probs are not there?

Why don't you get log probs as outputs though? Are you not using OpenAI API directly? I've never had that issue...

But you can always decide to label without logprobs by using `--annotators_configs alpaca_eval_gpt4_turbo``

I am using a proxy website to access the API. It turned out to be some bugs with their API service and it has been fixed. Thanks a lot for your reply.

> I am using a proxy website to access the API. It turned out to be some bugs with their API service and it has been fixed. Thanks a lot for your reply.

I have encountered the same issue with a proxy website to access the API. Could you share the proxy website you used? I would appreciate it!

> > I am using a proxy website to access the API. It turned out to be some bugs with their API service and it has been fixed. Thanks a lot for your reply.
> 
> I have encountered the same issue with a proxy website to access the API. Could you share the proxy website you used? I would appreciate it!

https://api.ai-gaochao.cn/

> > > I am using a proxy website to access the API. It turned out to be some bugs with their API service and it has been fixed. Thanks a lot for your reply.
> > 
> > 
> > I have encountered the same issue with a proxy website to access the API. Could you share the proxy website you used? I would appreciate it!
> 
> https://api.ai-gaochao.cn/

Thank you very much!!! :)

> > > I am using a proxy website to access the API. It turned out to be some bugs with their API service and it has been fixed. Thanks a lot for your reply.
> > 
> > 
> > I have encountered the same issue with a proxy website to access the API. Could you share the proxy website you used? I would appreciate it!
> 
> https://api.ai-gaochao.cn/

Hi, im using this proxy api but there's still some bug that some logprob is not returned.
Can you share how you fix it?
Thanks mate!

> > > > I am using a proxy website to access the API. It turned out to be some bugs with their API service and it has been fixed. Thanks a lot for your reply.
> > > 
> > > 
> > > I have encountered the same issue with a proxy website to access the API. Could you share the proxy website you used? I would appreciate it!
> > 
> > 
> > https://api.ai-gaochao.cn/
> 
> Hi, im using this proxy api but there's still some bug that some logprob is not returned. Can you share how you fix it? Thanks mate!

Need to contact the support staff and ask them to update their keys or service
Repository: tatsu-lab/alpaca_eval
Issue: How to get the LC Win Rate in AlpacaEval 1.0 version?
Root cause: Documentation Deficiency

I run the AlpacaEval 1.0 to evaluate my models. But it does not output the LC Win Rate (as shown in the figure)
<img width="842" alt="image" src="https://github.com/tatsu-lab/alpaca_eval/assets/41265224/6813acb5-f567-46fe-8032-11eaaf5538d4">
I find the LC Win Rate of AlpacaEval 1.0 version in the leaderboard website https://tatsu-lab.github.io/alpaca_eval/.
So, how to get the LC Win Rate in AlpacaEval 1.0 version?

you can use the parameter `--fn_metric 'get_length_controlled_winrate'` and potentially `--sort_by 'length_controlled_winrate'` see: https://github.com/tatsu-lab/alpaca_eval/blob/f693f1f7ec23b0d53e51339e7eca55d9453c2b86/src/alpaca_eval/main.py#L28

Note that for many models in AlpacaEval we can't compute LC because we used to not require the per sample annotation. 

Out of curiosity: why are you using AE1 instead of AE2?

> you can use the parameter `--fn_metric 'get_length_controlled_winrate'` and potentially `--sort_by 'length_controlled_winrate'` see:
> 
> https://github.com/tatsu-lab/alpaca_eval/blob/f693f1f7ec23b0d53e51339e7eca55d9453c2b86/src/alpaca_eval/main.py#L28
> 
> Note that for many models in AlpacaEval we can't compute LC because we used to not require the per sample annotation.
> 
> Out of curiosity: why are you using AE1 instead of AE2?

Thank you for your detailed reply.

Would it re-request the GPT4 API and cost again using the parameter `fn_metric` in the same models?

For your curiosity, we don't use AE2 because we cannot get the logprobs of GPT4 by the proxy website in AE2.

No everything should be cached and will thus be instantaneous/free if you haven't deleted the cache

ohh but actually you also need to use `--is_recompute_metrics_only True` if not it won't even rerun the metrics
Repository: tatsu-lab/alpaca_eval
Issue: How to change cache path when evaluating multi-models
Root cause: Documentation Deficiency

Hi, I have noticed that the following codes control for the cache file, https://github.com/tatsu-lab/alpaca_eval/blob/4ced94f63cec6de3ac9e69ee59634d43e1bcba5c/src/alpaca_eval/annotators/base.py#L508
 and I want to know that if I simultaneously start evaluation for multiple models with the same annotator, will the cache for these multiple models be stored in the same file? Could there be cache conflicts? Additionally, how do I specify the cache file path from the command line?”
For example, my program submits the following three command lines simultaneously. These three models use the same annotator for evaluation, so the cache file names appear to be the same.
```
alpaca_eval --model_outputs {model1} --annotators_config {annotators} --output_path {output_path1}
alpaca_eval --model_outputs {model2} --annotators_config {annotators} --output_path {output_path2}
alpaca_eval --model_outputs {model3} --annotators_config {annotators} --output_path {output_path3}
```
Can I specify the storage path by using a ```--cache-file``` option?

It's possible but unlikely that you will get caching conflicts. In particular, I reload the cache right before appending the new annotations. Note that worse case if there's a caching conflict, you will lose some of the cached annotations which isn't too bad (you will still have all the annotations & score, but if you want to reannotate it will call GPT4 again).

For the CLI use `--caching_path {path}`
Repository: tatsu-lab/alpaca_eval
Issue: How to use AE1 to evaluate model
Root cause: Documentation Deficiency

Hello！

I'm trying to use AE1 to evaluate my model, I've looked through the read me and related questions and can't find exactly where to modify the code.

I tried to modify IS_ALPACA_EVAL_2 = ast.literal_eval(os.environ.get("IS_ALPACA_EVAL_2", "False")) in "alpaca_eval-main/src/alpaca_eval/constants.py", but I found that the result was close to that of AE2.

I would like to know if I have successfully used AE1, if not, I would appreciate your guidance, thanks.

What you did should work. Alternatively, you can use:

`IS_ALPACA_EVAL_2=False alpaca_eval --model_outputs <model_outputs.json>` 

what do you mean by "result was close to that of AE2"? That shouldn't be the case. Maybe you used a flag that changed the baseline.

> What you did should work. Alternatively, you can use:
> 
> `IS_ALPACA_EVAL_2=False alpaca_eval --model_outputs <model_outputs.json>`
> 
> what do you mean by "result was close to that of AE2"? That shouldn't be the case. Maybe you used a flag that changed the baseline.

Thank you! 😊
Repository: tatsu-lab/alpaca_eval
Issue: possibility of adding llama3-70b as the evaluator?
Root cause: Unimplemented Feature Gap

GPT-4 is so expensive if we have to run hundreds of experiments for scientific studies. I wonder whether you have tried using Llama3-70b, which performs comparably to the older version of GPT-4, as the evaluator and seen its correlation with humans. That is a more affordable replacement for GPT-4, given that Llama3-70b on Together AI is only 1/12 the price of GPT-4.


I've analyzed some older open source models [here](https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/README.md) but not llama3

It should be really easy to make and evaluate a new evaluator though. I'm very happy to accept a PR for that. This section of the readme explains how to make a new evaluator and how to analyze it (ie compare it to human annotations): https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/README.md

Lmk if you have any issues running the above!

Thanks! I just realized that I could do it myself. I have tried with llama3. The results are quite impressive. The human agreement is 67.53 (without using COT), which is slightly lower than GPT4_turbo version and Claude 3. But the speed is 10 times faster, and the price is 10 times cheaper. I will upload the evaluation result later.

Great news, I would love to accept a PR for that!