zhiminy/EvalEng huggingface dataset:
Repository: tatsu-lab/alpaca_eval
Issue: a bug about `openai_organization_ids`
Root cause: Validation Gap

Hi! I'm trying with the latest version for alpaca_eval 2.0 and found an error. (I do not set a org id in my env, and previously with the alpaca_eval version, it won't have any issue.) Now, it seems that this ` for org in openai_organization_ids:` will be broken if I do not have an org id in my env. 

"""
Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/yuchenl/.local/lib/python3.10/site-packages/alpaca_eval/decoders/openai.py", line 197, in _openai_completion_helper
    all_clients = utils.get_all_clients(
  File "/home/yuchenl/.local/lib/python3.10/site-packages/alpaca_eval/utils.py", line 545, in get_all_clients
    client_configs = get_backwards_compatible_configs(**backward_compatibility_kwargs)
  File "/home/yuchenl/.local/lib/python3.10/site-packages/alpaca_eval/decoders/openai.py", line 375, in _get_backwards_compatible_configs
    for org in openai_organization_ids:
TypeError: 'NoneType' object is not iterable
"""

fixed, thanks @yuchenlin 

thanks for the quick fix! love the update! :D
Repository: tatsu-lab/alpaca_eval
Issue: log prob error when using Azure API
Root cause: External Dependency Breakage

![image](https://github.com/tatsu-lab/alpaca_eval/assets/92421404/38be1682-399b-4dde-8f50-c7365e604065)
I don't know if there is any bug with amazon API ？

Sorry, I mean Azure API

When I set logprobs=1, I found this "This model doesn't support the 'logprobs' paramter". Can you please switch the default evaluation method to others everyone can access?

Hi @chengjl19, unfortunately, Azure does not support logprobs yet but that will hopefully be fixed soon (as described at the bottom of [this section](https://github.com/tatsu-lab/alpaca_eval/tree/main/client_configs#configuring-openai)) Anyone can use the new evaluator using the OpenAI API instead. If you can't use the OpenAI API, then I would suggest using AlpacaEval1 for now. You can do so by setting `IS_ALPACA_EVAL_2=False` in your environment. Sorry for that!
Repository: tatsu-lab/alpaca_eval
Issue: TypeError when trying to run alpaca_eval
Root cause: Environment Incompatibility

Using python 3.9. Getting the following error when trying to run alpaca_eval with any model

`alpaca_eval evaluate_from_model --model_configs 'chatgpt' --annotators_config 'alpaca_eval_gpt4'`

`Traceback (most recent call last):
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/bin/alpaca_eval", line 8, in <module>
    sys.exit(main())
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/site-packages/alpaca_eval/main.py", line 468, in main
    fire.Fire(ALL_FUNCTIONS)
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/site-packages/alpaca_eval/main.py", line 214, in evaluate_from_model
    evaluation_dataset = utils.load_or_convert_to_dataframe(evaluation_dataset)
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/site-packages/alpaca_eval/utils.py", line 264, in load_or_convert_to_dataframe
    if isinstance(df, AnyPath):
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/typing.py", line 720, in __instancecheck__
    return self.__subclasscheck__(type(obj))
  File "/home/makeshn/ssd1/miniconda3/envs/alpaca_eval/lib/python3.9/typing.py", line 723, in __subclasscheck__
    raise TypeError("Subscripted generics cannot be used with"
TypeError: Subscripted generics cannot be used with class and instance checks`

Changing 

`if isinstance(df,  AnyPath):
    df = Path(df)`

to 

`if isinstance(df,  str):
    df = Path(df)`
    
seems to have fixed the issue.

https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/utils.py#L264

Didn't solve the issue for me. Any update on this?

Thanks for raising this! Actually we require >= python3.10 for this package.

Please try again on python>=3.10 and reinstall package via `pip install -U alpaca_eval`.
If you have any further issues, please create a new issue.

Note that it works fine with Python 3.9 if you change this particular type check to just be a tuple of the types in that Union - no other things in the package appear to need 3.10
Repository: tatsu-lab/alpaca_eval
Issue: Any arxiv paper for reference?


There's no AlpacaEval paper, what information are you looking for?

For citation please use 
```
@misc{alpaca_eval,
  author = {Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {AlpacaEval: An Automatic Evaluator of Instruction-following Models},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/alpaca_eval}}
}
```

If you want more information about some aspects of AlpacaEval I would suggest reading the pretty detailed README, and asking on the discord channel if you still have issues. 

For the data please cite the AlpacaFarm paper
```
@misc{dubois2023alpacafarm,
  title={AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback}, 
  author={Yann Dubois and Xuechen Li and Rohan Taori and Tianyi Zhang and Ishaan Gulrajani and Jimmy Ba and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto},
  year={2023},
  eprint={2305.14387},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}
```

> There's no AlpacaEval paper, what information are you looking for?
> 
> For citation please use
> 
> ```
> @misc{alpaca_eval,
>   author = {Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
>   title = {AlpacaEval: An Automatic Evaluator of Instruction-following Models},
>   year = {2023},
>   publisher = {GitHub},
>   journal = {GitHub repository},
>   howpublished = {\url{https://github.com/tatsu-lab/alpaca_eval}}
> }
> ```
> 
> If you want more information about some aspects of AlpacaEval I would suggest reading the pretty detailed README, and asking on the discord channel if you still have issues.
> 
> For the data please cite the AlpacaFarm paper
> 
> ```
> @misc{dubois2023alpacafarm,
>   title={AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback}, 
>   author={Yann Dubois and Xuechen Li and Rohan Taori and Tianyi Zhang and Ishaan Gulrajani and Jimmy Ba and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto},
>   year={2023},
>   eprint={2305.14387},
>   archivePrefix={arXiv},
>   primaryClass={cs.LG}
> }
> ```

Thanks, I am looking for the full report, @YannDubs anything related to it?

As I said above there's no full report. The README should have all information needed. Let me know if you are looking for some specific information
Repository: tatsu-lab/alpaca_eval
Issue: Enable specifying a custom path for model configs ? 
Root cause: Documentation Deficiency

Is it possible to run alpaca-eval using a custom model config in a directory path that is out of the alpaca-eval repo ? I see the base path is explicitly set to the default [here](https://github.com/tatsu-lab/alpaca_eval/blob/9a7fcd0ef8970c7b9ed52c68402d1470e71d3540/src/alpaca_eval/main.py#L265) 


Enabling this would help avoid forking alpaca-eval or installing it in editable mode & just use it directly from pip install. It looks like its possible to specify annotator config from a different root path - [link](https://github.com/tatsu-lab/alpaca_eval/blob/9a7fcd0ef8970c7b9ed52c68402d1470e71d3540/src/alpaca_eval/annotators/base.py#L61) - Similar pattern for model config and a more explicit cli flag increase usability of the repo. Any thoughts ?

Hey @stoicio, I think that it is already possible if you provide an absolute path! 

Not sure it's worth adding a CLI flag for relative path given that you would need to write as much as if you use absolute paths. Let me know if you have issues with absolute paths.

Tested and it should work!

I didn't know about the Path lib's `/` behavior when used with absolute path. TIL ! 
Thanks for checking it out.
Repository: tatsu-lab/alpaca_eval
Issue: clarification on annotation entries from `alpaca_eval`
Root cause: Documentation Deficiency

I wanted to clarify the notation in the `alpaca_eval` command. Let's say I passed in `--model_outputs my_model_A.json` and
`--reference_outputs my_model_B.json`. The resulting `annotations.json` will contain keys `output_1`, `output_2`, and rankings under `raw_completions`:
```
    "raw_completion":{
      "concise_explanation":"....",
      "ordered_models":[
        {
          "model":"M",
          "rank":1
        },
        {
          "model":"m",
          "rank":2
        }
      ]
    }
```
According to [this template](https://github.com/tatsu-lab/alpaca_eval/blob/3abbec9bb9e04b3e97ed57dd938540538c3aef9f/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4_turbo_fn/alpaca_eval_fn.txt#L19), `m` corresponds to `output_1` and `M` corresponds to `output_2`, correct? Then would this mean the `model_outputs` corresponds to `m` as well? I'm not sure which one takes precedence.

Thanks!

Do you see a field `referenced_models` alongside? It should look something like this:
`  "referenced_models": {
    "M": "output_1",
    "m": "output_2"
  }
`
which should be easier to understand / hopefully resolves the confusion

@xwinxu check out the [readme](https://github.com/tatsu-lab/alpaca_eval?tab=readme-ov-file#interpreting-chain-of-thought-when-available), the rank tells you which model was preferred so if `preference = 2` (i.e. output_2 is preferred) and `M` has rank 1 (i.e. it's preferred) then you would know that `M` corresponds to `output_2`. The reason why all of this is a bit complicated is because we randomize the order of the outputs when evaluating.

I'll add the field `referenced_models` that Rohan is referring to, which currently is only added for sample sheets so you won't see it locally.

Note that your reference model will always be output_1.


Done @xwinxu , to add `referenced_models` you need to (1) update alpaca_eval, and (2) rerun the parsing by using the flag `--is_reapply_parsing True`. Note that this won't rerun the actual OAI annotations, which are cached.

Thanks Yann!
Repository: tatsu-lab/alpaca_eval
Issue: Evaluator fails on prompts that violates Azure content moderation policy
Root cause: Validation Gap

I am getting the following error on some responses using OpenAI from Azure.
```
WARNING:root:OpenAIError: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': True, 'severity': 'medium'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}}.
WARNING:root:Unknown error. 
 It's likely a rate limit so we are retrying...
INFO:root:Sleeping 2 before retrying to call openai API...
```
Minimal example to reproduce
```
[
    {
        "instruction": "How can I spice things up in the bedroom?",
        "output": "7 Ways to Spice Up Your Sex Life\n\n1. Talk dirty to each other.\n2. Role play.\n3. Try a new position.\n4. Use sex toys.\n5. Get intimate in a new location.\n6. Set a sexy scene.\n7. Explore each other\u2019s fantasies.",
        "generator": "llama2-13b",
        "dataset": "helpful_base",
        "datasplit": "eval"
    }
]
```
The evaluator keeps on re-trying indefinitely. A similar issue was also reported in #162, #168.
Should not there be a limit on the maximum number of tries for each prompt, and if such prompts exceed that threshold, they should be excluded from the final result calculation?

Should be fixed: https://github.com/tatsu-lab/alpaca_eval/blob/5df7581a2ed2c369bc287f382684ccea21ecd905/src/alpaca_eval/decoders/openai.py#L193

Let me know if you have other issues!

Also are you using AE2? if so how did you make it work with Azure? I thought they didn't give access to logprobs...

Thanks for the reply! I am using AE1 only.
Repository: tatsu-lab/alpaca_eval
Issue: TypeError: Subscripted generics cannot be used with class and instance checks
Root cause: Validation Gap

> Traceback (most recent call last):
  File "/home/agl/.local/bin/alpaca_eval", line 8, in <module>
    sys.exit(main())
  File "/home/agl/.local/lib/python3.9/site-packages/alpaca_eval/main.py", line 468, in main
    fire.Fire(ALL_FUNCTIONS)
  File "/usr/local/conda/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/conda/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/conda/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/agl/.local/lib/python3.9/site-packages/alpaca_eval/main.py", line 106, in evaluate
    leaderboard = utils.get_precomputed_leaderboard(precomputed_leaderboard, reference_outputs, annotators_config)
  File "/home/agl/.local/lib/python3.9/site-packages/alpaca_eval/utils.py", line 391, in get_precomputed_leaderboard
    leaderboard = load_or_convert_to_dataframe(precomputed_leaderboard).to_dict(orient="index")
  File "/home/agl/.local/lib/python3.9/site-packages/alpaca_eval/utils.py", line 264, in load_or_convert_to_dataframe
    if isinstance(df, AnyPath):
  File "/usr/local/conda/lib/python3.9/typing.py", line 720, in __instancecheck__
    return self.__subclasscheck__(type(obj))
  File "/usr/local/conda/lib/python3.9/typing.py", line 723, in __subclasscheck__
    raise TypeError("Subscripted generics cannot be used with"
TypeError: Subscripted generics cannot be used with class and instance checks


I get the above error when i run `alpaca_eval evaluate --model_outputs 'example/outputs.json'`.Can you tell me why this is?

For me, switching from python 3.9 to 3.11 did the trick.
Repository: tatsu-lab/alpaca_eval
Issue: Repeated deprecation errors
Root cause: External Dependency Breakage

I am repeatedly getting this error when I try to run the pairwise annotator. How do I fix this? The problem seems to be from Alpaca Eval.

INFO:root:Sleeping 2 before retrying to call openai API...
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/completions "HTTP/1.1 404 Not Found"
WARNING:root:OpenAIError: Error code: 404 - {'error': {'message': 'The model `text-davinci-003` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}.
WARNING:root:Unknown error. 
 It's likely a rate limit so we are retrying...

**Here is the full traceback:**
Traceback (most recent call last):
  File "/alpaca_farm/test_autoannotations.py", line 19, in <module>
    annotated = annotator.annotate_pairs(outputs_pairs)
  File "/anaconda3/envs/llm-pref/lib/python3.10/site-packages/alpaca_eval/annotators/pairwise_evaluator.py", line 263, in annotate_pairs
    return self.__call__(to_annotate, **decoding_kwargs)
  File "/anaconda3/envs/llm-pref/lib/python3.10/site-packages/alpaca_eval/annotators/base.py", line 188, in __call__
    df_annotated = self._annotate(curr_df_to_annotate, **decoding_kwargs)
  File "/anaconda3/envs/llm-pref/lib/python3.10/site-packages/alpaca_eval/annotators/base.py", line 295, in _annotate
    curr_annotated = self.annotators[annotator](
  File "/anaconda3/envs/llm-pref/lib/python3.10/site-packages/alpaca_eval/annotators/base.py", line 665, in __call__
    completions = self.fn_completions(prompts=prompts, **self.completions_kwargs, **decoding_kwargs)
  File "/anaconda3/envs/llm-pref/lib/python3.10/site-packages/alpaca_eval/decoders/openai.py", line 152, in openai_completions
    completions = list(
  File "/anaconda3/envs/llm-pref/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/anaconda3/envs/llm-pref/lib/python3.10/multiprocessing/pool.py", line 861, in next
    self._cond.wait(timeout)
  File "/anaconda3/envs/llm-pref/lib/python3.10/threading.py", line 320, in wait
    waiter.acquire()
KeyboardInterrupt

This error happens even if I change out text-davinci-003 for gpt-4-0314

There's, unfortunately, nothing we can do about depreciated endpoints, we keep them in AlpacaEval for documentation purposes. But if you switch to `gpt-4` it will work.
Repository: tatsu-lab/alpaca_eval
Issue: Why is the performance of Falcon-40b-instruct so poor?


This question is hard to answer from a scientific perspective, since there are many design decisions that go into either the pretraining or instruct-tuning stage that we don't fully understand.

Concretely for AlpacaEval, all the outputs of running `falcon-40b-instruct` are available at [alpaca_eval_all_outputs.json](https://huggingface.co/datasets/tatsu-lab/alpaca_eval/blob/main/alpaca_eval_all_outputs.json). Reading through these results, it becomes apparent that the generations can be hit or miss, and are on average less detailed than outputs from other models (say, `vicuna-13b`). So looking at these results, it does seem that the automatic evaluation tracks with human judgement here. I should also point you to our [more in-depth analysis](https://github.com/tatsu-lab/alpaca_eval#additional-analysis-and-plots).

You may be also asking about the discrepancy between the result here and on other leaderboards, say the Open LLM Leaderboard. The Open LLM Leaderboard measures performance on more knowledge-intensive tasks (TruthfulQA, MMLU, etc). However, AlpacaEval focuses on more general, open-domain instruction following. So there is definitely a mismatch in the type of task the model is performing, which is probably the driving factor in the difference in leaderboard rankings.

These sort of results suggest why evaluation of these models is important! If we're looking to deploy `falcon-40b-instruct` as a chatbot, AlpacaEval suggests it may not be the best model out there. However, it does seem like a good choice for QA/knowledge-intensive questions, given its benchmark results there.

Closing this issue as there's no concrete follow-up.
Repository: tatsu-lab/alpaca_eval
Issue: A bug in `weighted_alpaca_eval_gpt4_turbo`
Root cause: Environment Incompatibility

I use the scripe:
`alpaca_eval --model_outputs 'model_outputs.json'`
As I known, `weighted_alpaca_eval_gpt4_turbo` is the default annotator. I encounter an error:
```
INFO:httpx:HTTP Request: POST https://lonlie.plus7.plus/v1/chat/completions "HTTP/1.1 400 Bad Request"
WARNING:root:OpenAIError: Error code: 400 - {'error': {'message': "This model does not support the 'logprobs' parameter. (request id: 20240308105643738285969aaXFO0Xe) (request id: 202403081056436512177484L8BZc3U)", 'type': 'invalid_request_error', 'param': 'logprobs', 'code': None}}.
WARNING:root:Unknown error. 
 It's likely a rate limit so we are retrying...
INFO:root:Sleeping 2 before retrying to call openai API...
```

As per my other message, I think the issue is on the proxy `https://lonlie.plus7.plus/v1/chat/completions` you are using. Can you (1) `pip install -U openai` and (2) try using openai servers directly for 1 instruction

closing as I don't think it's an AE issue (nor openai actually). Feel free to reopen if the above doesn't work

Thank you for your reply. You are right. This issue is on the proxy https://lonlie.plus7.plus/v1/chat/completions.
Repository: tatsu-lab/alpaca_eval
Issue:  Remove Deprecated model.to_bettertransformer() Call for Compatibility with Latest Transformers and Torch
Root cause: External Dependency Breakage

### Environment:

Transformers version: >=4.36
Torch version: >=2.1.1
AlpacaEval version: nightly build
Python version: 3.10
### Issue:
The call to model.to_bettertransformer() in alpaca_eval/decoders/huggingface_local.py fails with the latest versions of Transformers and Torch, as these libraries have integrated BetterTransformer optimizations natively.

### Error Message:

`ValueError: Transformers now supports natively BetterTransformer optimizations... Please upgrade to transformers>=4.36 and torch>=2.1.1 to use it.`

### Expected Behavior:
AlpacaEval should skip the model.to_bettertransformer() conversion when using recent library versions.

### Suggested Fix:
Remove or conditionally bypass the model.to_bettertransformer() call based on library versions.

Thanks for flagging @zxia545 , it should work now but lmk if it doesn't!
Repository: tatsu-lab/alpaca_eval
Issue: Reproducing numbers for evaluator human-agreement leaderboard.
Root cause: Documentation Deficiency

Hello, I am trying to obtain the leaderboard numbers present in 
`src/alpaca_eval/leaderboards/evaluators/evaluators_leaderboard.csv`
But I am unable to understand how to use the datasets mentioned in `https://huggingface.co/datasets/tatsu-lab/alpaca_eval`

My main problem is,
1. Human annotations give a preference rank based on two outputs from the same language model, and I can see that they are obtained for a wide variety of models in `alpaca_farm_human_annotations.json`
2. The GPT-4 annotations/Claude annotation-based preference ranking however is in-between two language models outputs.

My main question is,  is the dataset to obtain the human-agreement of GPT-4 evaluation (and other s in the evaluators leaderboard) released ? If so, can you give a few pointers to find them?
Specifically, I want to compare my own evaluator in-terms of human agreement with the available models.

Thank you.

`alpaca_farm_human_crossannotations` has human (cross)annotations on the AlpacaEval evaluation set, when comparing outputs of two models. This is the data that we used to compute human agreement here: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/README.md

It's only for older models as we did not run any new human experiment since the release of AE/AF. Hope it helps, and good luck!
Repository: tatsu-lab/alpaca_eval
Issue: [LOG] improve the logging for OpenAI maximum context length
Root cause: Validation Gap

I tried to run the chatgpt evaluator, but your OpenAPI requestor seems to go into infinite retry loops:

> WARNING:root:OpenAIError: This model's maximum context length is 4097 tokens. However, your messages resulted in 4452 tokens. Please reduce the length of the messages..
> WARNING:root:Hit request rate limit; retrying...
> (repeats forever)

And similar loops happen on trying gpt4 if you do not have access to the model or use an invalid api key. These seem to be caused by assuming any other error is a rate-limiting message.

I tried to fix the token limit loop by:
> if "Please reduce your prompt" in str(e) or "This model's maximum context length" in str(e):

but it can still crash due to this line, probably since the sum of the generated samples is too long to fit.
```
if kwargs["max_tokens"] == 0:
   raise e
```


Thanks for flagging this; we'll look into this ChatGPT issue. For GPT-4 or Claude, if you set the API key and have access to the models, the error wouldn't occur. We'll work on adding better logging for failure to set the key or restricted access in that case.

the logging is now improved! Thanks