zhiminy/EvalEng huggingface dataset:
Repository: stepjam/RLBench
Issue: Changing robotic arm in RLBench
Root cause: Documentation Deficiency

Hello, thanks for making the great repo.  
I want to change the default arm to the UR3.ttm which is already provided in CoppeliaSim(V-rep).  
Can I just change the codes in environment.py to add the UR3 in SUPPORTED_ROBOTS and add UR3.ttm to the robot_ttms directory? 
I hope there can be more documentations on swapping the robotic arms since I might need adding more arms in the future.

Hi, sorry for the delay in getting back to you. 
The 'tip' of the arm would need to be oriented to be the same as the Panda arm, and then saved to the robot_ttms directory. I can add the UR3; but bear in mind that many of the tasks will not be achievable given the small workspace of the ur3

UR5 is now in version (branch) 1.1.0. See the examples/swap_arm.py to see how to swap the arm.
Please note that alternative arms have **limited** support; they may not work with all tasks, as all of the tasks have been designed to be completed with the Panda.
There was no point adding in UR3 because the UR3 workspace is so small that (almost) all RLBench tasks are impossible.
Repository: stepjam/RLBench
Issue: Enhance randomness of environments in different processes
Root cause: Unimplemented Feature Gap

When running multiple instances on different processes, I noticed that episodes are always initialized the same in the all of processes. That is, when using tasks where the target changes in each episode (like in ReachTarget), all processes have the same target all the time. I would appreciate a possibility to enhance the randomness, e.g., by passing different seeds to each process, since I am running a bunch of environments in parallel. I couldn't find a way to do it yet.

Setting the seed in numpy with `np.random.seed(worker_id)` for each process differently solved the issue.
Repository: stepjam/RLBench
Issue: would it possible for more realistic grasping as in real world
Root cause: Unimplemented Feature Gap

dear author, 
would it possible for more realistic grasping as in the real world, since rlbench give fake grasping by proximity sensor in the middle of the gripper tips.  

Thanks,

I can certainly provide a function that would allow you to turn off the fake grasping. Would this be OK?
Bare in mind that some of the tasks' demos will most likely fail in this mode though.

@stepjam yes, I would like to avoid the fake grasping in our env.  Thanks very much! 
So, where can I found the function?

This is now in 1.0.9 (which will soon be merged to main). You can disable the fake grasping by:

```
e = Environment(..., attach_grasped_objects=False)
```
Repository: tatsu-lab/alpaca_eval
Issue: Add documentation / process for contributing 

- [x] discord
- [x] README (outputs / number)

Ideally, we would have the outputs but I don't want the package to be too large so I don't want the package to have the outputs. One possibility is to make it optional but say that outputs need to be on hugging face hub. Then in the leaderboard we say whether the data is there, which would make the results more believable

done
Repository: tatsu-lab/alpaca_eval
Issue: Use chatGPT as baseline?
Root cause: Documentation Deficiency

the number are getting close to 100% win rate we should consider recalibrating win rates by comparing to chatGPT

+1 

although it seems that we can do that by ourselves via customizing the output json file. ~~It would be better to have an official release of the ChatGPT references though~~ 

The references using ChatGPT: https://github.com/tatsu-lab/alpaca_eval/blob/main/results/chatgpt/model_outputs.json 
So I guess we can directly set the reference to this file for using ChatGPT as the baseline 

Yup this is supported as a setting. We currently don't have plans to make chatGPT as the default, so closing this for now.
Repository: tatsu-lab/alpaca_eval
Issue: freewilly missing?
Root cause: Unimplemented Feature Gap

https://stability.ai/blog/freewilly-large-instruction-fine-tuned-models
shouldnt that be on here since they are winning on many benchmarks?

We are open to PRs :) Contribution docs: https://github.com/tatsu-lab/alpaca_eval#contributing-a-model

Hi @jtoy closing for now, but if you are interested we'd love to merge a PR for freewilly!!
Repository: tatsu-lab/alpaca_eval
Issue: [DOC] Update Anthropic docstring
Root cause: Documentation Deficiency

Anthropic changed their python sdk - making this code line outdated.

https://github.com/tatsu-lab/alpaca_eval/blob/7e85b5e85f241d983d2bf292bc5ee4a4cc2a8b86/src/alpaca_eval/decoders/anthropic.py#L38

---

Would love to know if this might help - https://github.com/BerriAI/litellm

~Simple I/O library, that standardizes all the llm api calls to the OpenAI call

```
from litellm import completion

## set ENV variables
# ENV variables can be set in .env file, too. Example in .env.example
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["ANTHROPIC_API_KEY"] = "anthropic key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# anthropic call
response = completion("claude-v-2", messages)

```


cc: @YannDubs 

Thanks @krrishdholakia,
Note that the line you linked to is the docstring and the code is actually up to date, but thanks for the heads up I'll update the docstring!

Concerning litellm seems like a great and clean library 💯 ! We already have cohere / claude / openai and want to avoid any dependencies for those. But if you want to send a PR for a new completion function and test it on a model we haven't evaluated yet (e.g. `palm-2-chat-bison`) we'd love to merge both the completion function and the evaluations.

- Documentation for contributing a [completion function](https://github.com/tatsu-lab/alpaca_eval#contributing-a-completion-function)
- Documentation for contributing a [model](https://github.com/tatsu-lab/alpaca_eval#contributing-a-model)

If you open a PR I'd be happy to provide more help on how to go about it!



closing because we updated the docstring. 

@krrishdholakia if you end up adding `litellm` feel free to open a PR and I'll help you there !
Repository: tatsu-lab/alpaca_eval
Issue: Separate files per provider?

hey @YannDubs - we should have palm-2-chat-bison in by the end of this week, will add it then.

Y'all have an interesting approach of an individual file per provider
![image](https://github.com/tatsu-lab/alpaca_eval/assets/17561003/e9050002-7d58-4313-8a43-4f55f3bca863)


Why do it this way - vs. making the completion call inside the __init__.py?

I also noticed you're calculating cost per provider <- why is that?

Separate files: we wanted to keep things separate to make it more readable especially if we scaled to many more providers.

Cost per provider: this is just additional useful information. Especially useful to select the evaluator. E.g as you see in [this table](https://github.com/tatsu-lab/alpaca_eval/tree/main#evaluators)  when deciding on the evaluator you should likely take into the cost & tie for evaluation in addition to quality (the "agreement" with humans).

Thanks, looking forward for `palm-2-chat-bison`!



closing issue. please open another for future issues
Repository: tatsu-lab/alpaca_eval
Issue: GPT4 rate limit
Root cause: Documentation Deficiency

**Hi, I am trying to evaluate our model output using alpaca_eval.**

## Here is the command:
```
export OPENAI_API_KEY="sk-xxxxxx"
alpaca_eval --model_outputs 'output/alpaca_eval/outputs.json'
```

## Problem: While running, there are always "Rate limit reached" error. 
```
NFO:openai:error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-4 in organization org-0Ibh47ogWcbeM3DJsyr3EC29 on tokens per min. Limit: 40000 / min. Please try again in 1ms. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False
WARNING:root:OpenAIError: Rate limit reached for default-gpt-4 in organization org-0Ibh47ogWcbeM3DJsyr3EC29 on tokens per min. Limit: 40000 / min. Please try again in 1ms. Contact us through our help center at help.openai.com if you continue to have issues..
WARNING:root:Hit request rate limit; retrying...
```

## Question:
1) How could I control the GPT4 access rate (e.g. add some delay in the code)? 
2) After evaluation, I got n_total = 800, does this mean that there are 5 tests failed? Are these failures related to GPT4 rate limit error?


Great questions!

1. Concerning the rate limit, there are 3 ways to avoid/mitigate it.
    - **Decreasing number of processes**: by default we use 4 concurrent processes for querying GPT4. This will hit the rate limit faster. If that's an issue you can change it in the evaluators configs. e.g. add `num_procs: 2` in the [completion kwargs](https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/configs.yaml#L5)
    - **Increasing sleep time**: we are already adding a 2 seconds sleep time [here](https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/decoders/openai.py#L233). If this is not sufficient for you, you can change it in the evaluators configs. e.g. add `sleep_time: 10` in the [completion kwargs](https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/configs.yaml#L5)
     - **Using multiple API keys**: in case you have multiple API keys you can set them separated by commas `OPENAI_API_KEYS=key1,key2` and we will switch between them when you get rate limited.
     
2. When you hit a rate limit we will simply retry (as seen in the warning) so that's not the issue. If there are missing evaluation results it typically means that the annotations could not be parsed. we also raise a warning for that so you see the reason in your terminal! 

Feel free to reopen if you have further questions

For future reference, here's the way to deal with multiple keys now: https://github.com/tatsu-lab/alpaca_eval/tree/main/client_configs

one can also use Azure
Repository: tatsu-lab/alpaca_eval
Issue: add leaderboard of base models
Root cause: Unimplemented Feature Gap

Not needed for monday.

- [ ] simple script to perform SFT
- [ ] create leaderboard of models

benefit is that new base models will directly be evaluated on our leaderboard => no work for us to do

done @rtaori ?

no there's no SFT script. unclear why it's needed here
Repository: tatsu-lab/alpaca_eval
Issue: Consider using a fixed version of GPT-4
Root cause: Configuration Error

Since the OpenAI API is changing, please consider using a fixed GPT-4 version such as `gpt-4-0314`. Using `gpt-4` can cause fluctuating results and make old results stale.

Related: https://github.com/imoneoi/openchat/issues/42

Thanks @imoneoi, we added both version gpt4 in the configs. Note that all outputs are there so you can always evaluate any model with any evaluator. 

We decided to use the latest gpt4 as default to always have best possible evaluator 

Thanks! Keeping the latest gpt4 as default is a good idea, but maybe we should use a fixed version for the leaderboard to ensure reproducibility and fair comparison between old/new model results, since the behavior [changes hugely between versions](https://arxiv.org/abs/2307.09009) .

I would recommend `gpt-4-0314` or the version used for human agreement experiments.
Repository: tatsu-lab/alpaca_eval
Issue: API call fails with long outputs
Root cause: Unimplemented Feature Gap

When the evaluated model outputs long responses, the evaluation API call will fail and keep retrying. Consider counting tokens with `tiktoken` and truncating the trailing X tokens of the evaluated model to reduce the total length to 8192 tokens.

```
WARNING:root:Unknown error This model's maximum context length is 8192 tokens. However, your messages resulted in 8344 tokens. Please reduce the length of the messages..
```

Can you provide a command to reproduce this issue?

Can you try evaluating this model output? This model often output very long responses because it cannot stop correctly.

```
python -m alpaca_eval.main evaluate --model_outputs model_outputs.json --annotators_config alpaca_eval_gpt4
```

[model_outputs.zip](https://github.com/tatsu-lab/alpaca_eval/files/12416468/model_outputs.zip)



thanks @imoneoi! This is indeed something that would be really nice, although it's a bit annoying to do with the current implementation (i.e. the error comes up after you already formatted the string). I'll leave it open for now as we might add that in the near future but probably not in the next few weeks :/  

Hi @YannDubs, we need this feature for experiments. We can implement it and contribute a pull request, what do you think?

My idea is to use binary search with `tiktokens`, and cut the trailing content of evaluated model output until it fits in GPT4 context length.

Hi @imoneoi ,

Your solution sounds like it could work. Although, I think the current instructions are not *that* complex that they would really benefit from models generating extremely long/detailed responses. If you are running into this issue, in my experience the most common issue is excessive repetition from the model. Looking through some of the outputs in the json you attached, this is exactly what seems to be happening (lots of gibberish and repeated tokens).

Additionally, evaluating models based on truncated version of the outputs would break the json outputs -> leaderboard number mapping we currently have. However, this is not a big issue, and so if you submit a PR, we'd be happy to include that.

Closing, if you anyone needs long outputs I would use GPT4-32K or the 128K
Repository: tatsu-lab/alpaca_eval
Issue: Add a fixed version GPT4 evaluator for the leaderboard
Root cause: Unimplemented Feature Gap

See [#126 ](https://github.com/tatsu-lab/alpaca_eval/issues/126#issuecomment-1689359237)

Thanks for flagging this. If you'd like to submit a PR for a frozen leaderboard by rerunning `gpt-4-0314` or `gpt-4-0613` on all of the model outputs, we'd happily merge that in as a separate set of results

This is indeed a huge problem. I used both `alpaca_eval_gpt4_0314` and `alpaca_eval_gpt4` for testing the same output json file and the win rate can be super different. One is 70ish and the other is 50+%...

@yuchenlin which model were you evaluating?
It sounds like the best option here may be to fix the version as `gpt-4-0613` and rerun the evaluator on all model jsons. Would that resolve both your concerns? Am very happy to accept a PR for this!

@rtaori  I tried to use gpt-4-0613 to reproduce the results, but I only obtained the last two lines. There might be something wrong. Could you help me?

|                      | Win Rate | Std Error |
| -------------------- | -------- | --------- |
| chatgpt              | 86.1     | 1.2       |
| text_davinci_003     | 50.0     | 0.0       |
| chatgpt-own          | 96.77    | 0.61      |
| text_davinci_003-own | 49.87    | 0.089     |

@wlhgtc  here you are fixing the version of the evaluator (gpt4) but not the model being evaluated (chatgpt) right?

> @wlhgtc here you are fixing the version of the evaluator (gpt4) but not the model being evaluated (chatgpt) right?

I directly use the json file in [here](https://github.com/tatsu-lab/alpaca_eval/blob/main/results/chatgpt).

I'm closing the issue given that:
1. we can't rely on a fixed version of OAI given how often they depreciate models & that newer versions are often cheaper
2. AlpacaEval2 already decreased standard errors (although it's unclear if it does between different models)
Repository: tatsu-lab/alpaca_eval
Issue: How to change GPT-4 API url?
Root cause: Documentation Deficiency

For example, I want to access some proxy GPT-4 API node.

Hi @hijkzzz if I understand correctly you can set `openai_api_base` as in the following: https://github.com/tatsu-lab/alpaca_eval/blob/afdef274e00c1904cf712a8d45f7751126dcb77b/src/alpaca_eval/models_configs/openchat-13b/configs.yaml#L5

feel free to reopen if that's not what you wanted


just set environment variable in your os:
```export OPENAI_BASE_URL=xxxx```
Repository: tatsu-lab/alpaca_eval
Issue: Is there a way to re-generate the leaderboard's index.html?
Root cause: Documentation Deficiency

Hi ,

After evaluating my model. Apart from the leaderboard.csv output , I would like to ask if there's a way to re-generate the leaderboard's index.html within /docs with the updated leaderboard's data?

Regards,
Vincent

Hi @VincentV89,

The index.html is never modified or generated, the only thing that changes is the leaderboard csv in the docs: https://github.com/tatsu-lab/alpaca_eval/blob/main/docs/alpaca_eval_gpt4_leaderboard.csv. You can regenerate that using `python docs/format_export_leaderboards.py`
Repository: tatsu-lab/alpaca_eval
Issue: what is alpaca_eval_gpt4

Hi,
I saw the difference when using `GPT4` and `alpaca_eval_gpt4` to eval the response from any model, but I am not sure what is `alpaca_eval_gpt4`. Do I missing the intruction of this?

Hi @zhixiongzh, `alpaca_eval_gpt4` uses our prompt which performs better than `gpt4` for eval. `gpt4` uses the same prompt as all the other evaluators, which is useful to understand which models are the best for automatic evaluation. All our evaluator results can be found at: https://github.com/tatsu-lab/alpaca_eval/tree/main/src/alpaca_eval/evaluators_configs
Repository: tatsu-lab/alpaca_eval
Issue: Modify the baseline to a stronger model, such as ChatGPT
Root cause: Unimplemented Feature Gap

The performance of 003 is too poor for today's models, it can be surpassed by simply using ChatGPT to get a few thousand data points, the current list already looks bloat, I don't see much point in continuing to top the list.

Thanks @chengjl19, we agree and had an issue for this: https://github.com/tatsu-lab/alpaca_eval/issues/101. Unfortunately, we currently don't have the bandwidth to rerun everything though. 

Note that the ranking still seems to be able to distinguish between the quality of models. This is because our eval set is relatively large so there are still some example for which 003 is better.

I believe if we want to keep this benchmark up to date, at some point we must raise the bar. Most leading open source LLM at this moment are already rival to chatgpt3.5 if not gpt4

Yes definitely agree, we are discussing something in the works. Will let you know of the update

Thanks @chengjl19 and @yhyu13 , we just updated the leaderboard with GPT-4 turbo as baseline
Repository: tatsu-lab/alpaca_eval
Issue: alpaca_eval_gpt4_turbo as evaluator quality

have we run any study on the human agreement if we use alpaca_eval_gpt4_turbo as the evaluator config? thanks

Yes, you can see `alpaca_eval_gpt4_turbo_fn ` here: https://github.com/tatsu-lab/alpaca_eval/tree/main/src/alpaca_eval/evaluators_configs

Note that:
- I only evaluated it once per example so I don't have the estimates of bias and variance
- I haven't tuned the prompt

You can see that it performs pretty well, it's cheaper, and has fewer biases than GPT4 (length and list preference)
Repository: tatsu-lab/alpaca_eval
Issue: OpenAI response error
Root cause: Environment Incompatibility

I keep getting this error whenever I evaluate to a specific sample.

```INFO:openai:error_code=None error_message='OpenAI response error' error_param=prompt error_type=invalid_request_error message='OpenAI API error received' stream_error=False WARNING:root:OpenAIError: OpenAI response error.```

I don't know how to fix it.

is that the same as https://github.com/tatsu-lab/alpaca_eval/issues/162 ? The error message seems different, can you send me a minimal reproducible example? also pleasure update `pip install -U alpaca_eval`

> is that the same as #162 ? The error message seems different, can you send me a minimal reproducible example? also pleasure update `pip install -U alpaca_eval`

Yes, it is the same issue. It's different because of my old version of alpaca_eval and openai.

Thank you for your reply.

ok then it should be solved
Repository: tatsu-lab/alpaca_eval
Issue: Data parallelism in `evaluate_from_model`?
Root cause: Unimplemented Feature Gap

Hi, 

Thanks for the repo! I am wondering if `evaluate_from_model` supports splitting the query data across GPUs, and generating responses independently on each GPU. 

I see that we can do tensor parallelism by setting `tp > 1`, like in [here](https://github.com/tatsu-lab/alpaca_eval/blob/bac7e0e383064aa8ea283c74f0df4a98eae81064/src/alpaca_eval/models_configs/openbuddy-llama2-13b-v11.1/configs.yaml#L8). But to my understanding, this is model parallelism. The model that I wanted to evaluate is quite small, and can be fitted into a single GPU, so there is no need to do model parallelism. I wish to instead do data parallelism --- parallelize answer generation with multiple GPUs.

Thanks for your help in advance!


We can't currently with `alpaca_eval`. See my answer here: https://github.com/tatsu-lab/alpaca_eval/issues/29#issuecomment-1825282350. 
TL;DR, I would suggest doing inference using other tools such as TGI, and just the evaluation step with AlpacaEval, i.e. using `alapca_eval --model_outputs '.../outputs.json' `. 

`evaluate_from_model`  is especially useful for API models. For open models there are too many improvements and new tricks for us to keep up, so  the best is to use a library that focuses on that. We would love to add a new TGI [completion function](https://github.com/tatsu-lab/alpaca_eval#contributing-a-completion-function) though, any PR would be really welcome.

Sorry for that!

Thanks!
Repository: tatsu-lab/alpaca_eval
Issue: How to rate limit for gpt4 ?
Root cause: Documentation Deficiency

I could not find any such information in help command ?
How to rate limit the request for gpt4 (else it hit maximum request already) ?

See https://github.com/tatsu-lab/alpaca_eval/issues/115#issuecomment-1670968126

The easiest is just to chang your environ variable `OPENAI_MAX_CONCURRENCY`: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/constants.py#L14C46-L14C68
Repository: tatsu-lab/alpaca_eval
Issue: [ENH] potential improvements for AlpacaEval 2.0
GitHub labels: enhancement

Keeping track of things we should do for the next release. @rtaori lmk what you think. Tagging @lxuechen in case you also want to chime in. We can chat on Slack but let's keep the todos here:
- [x] change baseline to GPT 4 
- [x] prompt tuning for GPT 4 turbo
- [x] precompute the full minimal leaderboard with `alpaca_eval_cot_gpt4_turbo_fn`
- [x] precompute the full minimal leaderboard with `alpaca_eval_gpt4_turbo_fn`
- [x] decide which annotator to use 
- [x] update the website.: instead of toggle for evaluator, add toggle for version. Remove minimal toggle.
- [x] update the readme.
- [x] (potentially out of scope) run some important missing models, e.g., Mistral or Gemini https://github.com/tatsu-lab/alpaca_eval/pull/194
- [x] (probably out of scope) test out log probs for the evaluation https://github.com/tatsu-lab/alpaca_eval/pull/189
- [x] close and comment on previous PRs that are affected by it
Repository: tatsu-lab/alpaca_eval
Issue: Dataset 'tatsu-lab/alpaca_eval' doesn't exist on the Hub
Root cause: Environment Incompatibility

Hello,

I'm trying to evaluate my model using alpaca_eval, but I'm getting an error when loading the evaluation set. `Dataset 'tatsu-lab/alpaca_eval' doesn't exist on the Hub`. I checked the tatsu-lab huggingface account and it is indeed not available. Can you please advise?

Best regards,
Hani

Sorry about that! Forgot to make it public. Should work now