zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: oaieval doesn't run beacuse of "module 'openai' has no attribute 'error'"
Root cause: External Dependency Breakage

### Describe the bug

I jsut installed the evals repository and evals package for python using pip. 
When running any command using oaieval (e.g. `oaieval --help`) I get an error saying:

```
  File "/Users/.pyenv/versions/3.11.1/bin/oaieval", line 5, in <module>
    from evals.cli.oaieval import main
  File "/Users/.pyenv/versions/3.11.1/lib/python3.11/site-packages/evals/__init__.py", line 2, in <module>
    from .completion_fns.openai import (
  File "/Users/.pyenv/versions/3.11.1/lib/python3.11/site-packages/evals/completion_fns/openai.py", line 13, in <module>
    from evals.utils.api_utils import (
  File "/Users/.pyenv/versions/3.11.1/lib/python3.11/site-packages/evals/utils/api_utils.py", line 12, in <module>
    openai.error.ServiceUnavailableError,
    ^^^^^^^^^^^^
AttributeError: module 'openai' has no attribute 'error'

```

After searching online I found that this is caused by a too new openai library, that doesn't have 'error' anymore and/or that langchain still uses that. I tried installing openai==0.28.1 and langchain==0.0.330 or langchain==0.0.316 but neither worked for me. 

### To Reproduce

1. run  `git clone git@github.com:openai/evals.git`
2. change into directory
3. run `pip install -e .`
4. try `oaieval --help`

### Code snippets

_No response_

### OS

macos

### Python version

Python v3.11.1

### Library version

evals 1.0.3.post1

I had the same issue. This fixed it for me.

```
pip install openai==1.3.6
pip install langchain==0.0.343
```

It looks like you're using the library version of evals with the version of openai-python > 1.0. maybe try starting from scratch with a new venv and be sure you don't install evals from pypi?

Thanks for your help guys. The problem was, that although I did everything via venv my terminal still used my global python for the oaieval commands. After deleting the eval library from my global python it automatically used the venv for that and it works now.
Repository: openai/evals
Issue: Error structure in `utils` after openai package upgrade
Root cause: Configuration Error

Hey @etr2460, appreciate your work upgrading `evals` to work with newer `openai` versions!

This is mostly a question regarding [this commit](https://github.com/openai/evals/commit/58ac0fff9856834965538a295696fad038628521), where the error list in `api_utils.py` is different from the list in `make_me_say/utils.py`. The list in `api_utils` seems more complete, and I'm inclined to update the one in MakeMeSay to match it. 

Is there any reason I shouldn't?

Thanks!

I think updating make_me_say to match what the api utils uses makes sense. We specifically chose the ones in api_utils as errors that retrying could possibly resolve

Okay, thanks! I opened a PR to fix this.
Repository: openai/evals
Issue: Request to change arithmetical_puzzles prompting
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

arithmetical_puzzles prompts does not allow proper reasoning The following is added at the end of every prompt..."Do not give any reasoning or logic for your answer." 

In a previous PR, usama-openai requested to change the prompting of eval math_for_5th-grader to add better prompting to allow the LLM to reason. 

https://github.com/openai/evals/pull/1293#pullrequestreview-1517023638

Request to change the prompt for arithmetical_puzzles as well. 

### Additional context

_No response_
Repository: openai/evals
Issue: Tagged Release For 2.0.0
GitHub labels: bug

### Describe the bug

Hey all, I saw that there was a release recently to pypi: https://pypi.org/project/evals/ for version 2.0.0. I was wondering if it was easy to make a tagged commit in Github like you did for 1.0.3 (https://github.com/openai/evals/tags). I'm not sure if it's difficult to do in hindsight, but it does make it slightly easier to navigate and brows the code at a specific release.

Thanks!

### To Reproduce

1. Go to https://github.com/openai/evals/tags
2. There is no tag for 2.0.0

### Code snippets

```Python
N/A
```


### OS

N/A

### Python version

N/A

### Library version

openai-evals v2.0.0 and openai-evals v2.0.0.post1

The release is tagged now, sorry for the miss! https://github.com/openai/evals/releases/tag/2.0.0
Repository: openai/evals
Issue: Local run doesn't save logs to disk
Root cause: Documentation Deficiency

### Describe the bug

After running a successful eval, the records are supposed to be saved to tmp/evallogs, but I can't find them. 
Setting the `--local-run` flag explicitly doesn't work either. 

However, setting a `--record_path` explicitly works and correctly writes the results to the provided path. 

### Proposed fix
I managed to fix the issue by changing the default `record_path` **`/tmp/evallogs/`** to a relative path **`tmp/evallogs/`** in `oaieval.py`.
I'm not sure if this is a bug or if I'm missing something. 

### To Reproduce

Run an eval with the CLI command:
`oaieval gpt-3.5-turbo <eval-name>` 

### Code snippets

```Python
# evals/cli/oaieval.py - line 178

 record_path = (
        f"/tmp/evallogs/{run_spec.run_id}_{args.completion_fn}_{args.eval}.jsonl"
        if args.record_path is None
        else args.record_path
    )
```


### OS

Ubuntu 22.04.3 (WSL)

### Python version

Python 3.11.4

### Library version

latest

Sorry I was obviously missing something. The logs are correctly written to the absolute path.
Repository: openai/evals
Issue: Support for Azure OpenAI client
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

Currently evals framework does not support Azure openAI implementation. This is blocker if someone wants to use eval with Azure OpenAI implementation.


### Additional context

_No response_
```[tasklist]
### Tasks
- [ ] https://github.com/openai/evals/pull/1470
```


I understand that the solution has been implemented but how do I use this feature. 

The PR is still pending for approval

any update on this?
Repository: openai/evals
Issue: Support multiple completions for ModelbasedClassify 
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

It would be nice to be able to score multiple sample completions using ModelBasedClassify. Even if n>1 is passed into a completion function and multiple samples are returned, only the first is graded because of this line:

https://github.com/openai/evals/blob/main/evals/elsuite/utils.py#L193

### Additional context

I would like to be able to raise the temperature, ask a model to produce N completions, and have each completion graded separately using a rubric. This appears to work fine for non-model-based scoring.
Repository: openai/evals
Issue: `OpenAIChatCompletionFn` should `__init__` should accept `**kwargs`
Root cause: Validation Gap

### Describe the bug

Currently the `__init__` method for `OpenAICompletionFn` has `**kwargs` but `OpenAIChatCompletionFn` does not. This creates problems if I have both:
- a custom completion function that takes arguments I would like to populate with `--completion_args`
- a modelgraded eval that I plan to grade with an OpenAI model

In these cases I would like to do:
```sh
oaieval my_completion_fn,gpt-4-turbo-preview my_eval --completion_args my_param=my_value
```

But I get:

```python
TypeError: OpenAIChatCompletionFn.__init__() got an unexpected keyword argument 'my_param'
```

### To Reproduce

Run `oaieval gpt-3.5-turbo test-match --completion_args ignore_me=1`

### Code snippets

_No response_

### OS

macOS

### Python version

Python v3.11.7

### Library version

openai-evals 2.0.0.post1
Repository: openai/evals
Issue: Setting completion function args via CLI does not work
Root cause: Interface Contract Mismatch

### Describe the bug

The response to issue #512 implemented a way to dynamically change API parameters (such as temperature) from the CLI (by looking at the [code](https://github.com/openai/evals/blob/5a92ac38155cb32dcde1cc8b69b5e002e9437532/evals/cli/oaieval.py#L34-L39), the argument name has been changed to `--completion_args`). However, the arguments passed there do not seem to be used correctly.

### To Reproduce

1. Try running any eval, with a completion function corresponding to an openai model, by setting a parameter of the API, such as the temperature; for instance:
```  oaieval gpt-3.5-turbo-0125 <eval_name> --completion_args 'temperature=0.5'```
2. The following error is raised:
```
Traceback (most recent call last):
  File "/home/lorenzo/venv/recog-LLM_capabilities/bin/oaieval", line 8, in <module>
    sys.exit(main())
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 277, in main
    run(args)
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 146, in run
    completion_fn_instances = [
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 147, in <listcomp>
    registry.make_completion_fn(url, **additonal_completion_args) for url in completion_fns
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/registry.py", line 124, in make_completion_fn
    return OpenAIChatCompletionFn(model=name, n_ctx=n_ctx, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'temperature'
```
The reason is that `OpenAIChatCompletionFn` is not called correctly, as its `__init__` is the following:
```
    def __init__(
        self,
        model: Optional[str] = None,
        api_base: Optional[str] = None,
        api_key: Optional[str] = None,
        n_ctx: Optional[int] = None,
        extra_options: Optional[dict] = {},
    ):
```


### Code snippets

_No response_

### OS

Ubuntu 20.04

### Python version

python 3.9

### Library version

git+https://github.com/openai/evals.git@dd96814dd96bd64f3098afca8dc873aa8d8ce4c8

Can you please help me figure out https://github.com/openai/evals/issues/1564
Repository: openai/evals
Issue: When installing the project dependencies, i got: "ERROR: Could not build wheels for greenlet, which is required to install pyproject.toml-based projects"
Root cause: External Dependency Breakage

### Describe the bug

After cloning the repo and running: `pip install .` I get:  "ERROR: Could not build wheels for greenlet, which is required to install pyproject.toml-based projects"
 
 I am pretty sure this is related to the fixed version dependency of `playwright==1.32.1,` and some kind of version conflict with greenlet or another dependency.
 
 dependencies = [
    ...
    "playwright==1.32.1",
     ...
]

Because if I remove the exact version number (1.32.1), the installation works as expected. I am not sure if there are some consequences of not installing that specific version. 
 

### To Reproduce

1- clone the repo and create an environment
2- run `pip install .`
3- the installation will fail and you will see: ERROR: Could not build wheels for greenlet, which is required to install pyproject.toml-based projects

### Code snippets

_No response_

### OS

macOs 13.6.1

### Python version

v3.9.6

### Library version

2.0.0.post1

Hi @danesherbs,  I noticed you added  "playwright==1.32.1", do you have any comments about having a fixed version for that library? Thanks


I ran into the same issue with macOS 14.1.1 and Python version v3.9.6. Removing the fixed version for the playwright library fixed it for me as well. Appreciate it @JuanmaMenendez!

Worth providing some guidance here to save others hours of debugging time 😄 

Thanks for raising this @JuanmaMenendez!

For context, the `playwright` module is needed to run the [Multi-Step Web Tasks](https://github.com/openai/evals/tree/main/evals/elsuite/multistep_web_tasks) eval, which uses Web Arena environments,  which [require](https://github.com/web-arena-x/webarena/blob/main/requirements.txt#L2) `playwright==1.32.1`. However, I've verified that we can relax the version constraint on `playwright` and can still successfully run the Multi-Step Web Task test task using `oaieval dummy multistep-web-tasks`.

I've opened a [PR](https://github.com/openai/evals/pull/1516) updating `pyproject.toml` to fix the issue :)
Repository: openai/evals
Issue: Getting started example doesn't work - oieval attempts to update a None type object
Root cause: Validation Gap

### Describe the bug

When running the example provided in https://github.com/openai/evals/blob/main/docs/run-evals.md: `oaieval gpt-3.5-turbo test-match`, the following error is encountered out of the box:
```
 File "/Users/juliewang/Documents/evals/evals/cli/oaieval.py", line 167, in run
    eval_spec.args.update(extra_eval_params)
    ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'update'
```

This is due to this line: https://github.com/openai/evals/blob/main/evals/cli/oaieval.py#L158. The `args` field of `EvalSpec` is optional: https://github.com/openai/evals/blob/0dc0ba455194f5c9c4566ce415a2b2714f56e2db/evals/base.py#L58, and in the `test-match` case, `eval_spec.args` is `None`. 


### To Reproduce

1. Run  `oaieval gpt-3.5-turbo test-match`

### Code snippets

_No response_

### OS

macOS

### Python version

Python 3.12.2

### Library version

openai-evals commit b5853eb4fed5001188671e87b20d8f8261b57e63

Thx for raising this! I've opened a PR that should address it.
Repository: openai/evals
Issue: Add BigBench Tasks for evaluation
Root cause: Unimplemented Feature Gap

Hi would be cool to valuate all openai models on Beyond the Imitation Game Benchmark (BIG-bench) which is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities. The more than 200 tasks included in BIG-bench are summarized by keyword [here](https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/keywords_to_tasks.md#summary-table), and by task name [here](https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/README.md). A paper introducing the benchmark, including evaluation results on large language models, is currently under review, and is available as a [preprint](https://arxiv.org/abs/2206.04615). 

I believe a significant part of BIG-bench requires logprobs, which our API doesn't support currently. However, feel free to open a PR to add BIG-bench evals!
Repository: openai/evals
Issue: Schelling point eval doesn't work
Root cause: External Dependency Breakage

Fresh installation (d3dc89042ddee879a68a326fdb37716ee518640c)

```
oaieval gpt-3.5-turbo schelling_point
```

dies around 580th sample with

```Traceback (most recent call last):
(...)
openai.BadRequestError: Error code: 400 - {'error': {'message': "Sorry! We've encountered an issue with repetitive patterns in your prompt. Please try again with a different prompt.", 'type': 'invalid_prompt', 'param': 'prompt', 'code': None}}
```

I guess some prompt was accepted by the OpenAI API when the eval was developed and now it's rejected.

Thanks for flagging! https://github.com/openai/evals/pull/1534 has a fix, which should get merged soon.
Repository: openai/evals
Issue: Multiple Unit Test Failures Across OpenAI Assistants, Anthropic, and Google Gemini Libraries
Root cause: External Dependency Breakage

### Describe the bug

I observed several failures in unit tests involving the libraries for OpenAI Assistants, Anthropic, and Google Gemini. Each, detailed below.

1. **Anthropic Library:**
   - **Issue:** Recent changes have made the `ContentBlock` type a Union type, which can't be directly instantiated. This updated type now encapsulates both `TextBlock` and `ToolUseBlock`.
     ```python
     ContentBlock = Annotated[Union[TextBlock, ToolUseBlock], PropertyInfo(discriminator="type")]
     ```
   - **Impact:** Our existing code relies on the direct instantiation of `ContentBlock`, which is now causing failures.

2. **OpenAI Library:**
   - **Issue:** The latest update to the assistant API introduced several breaking changes:
     - The `retrieval` tool has been renamed to `file_search`.
     - The parameter `assistant.file_ids` has been changed to `tool_resources`.
     - The parameter `message.file_ids` has been modified to `attachments`.
   - **Impact:** These changes are causing failures in the functionalities that depend on file handling and assistant resources.

3. **Gemini Library:**
   - **Issue:** There's a defect in the Gemini library related to how it handles protobuf objects; specifically, it erroneously parses these objects as dictionaries.
   - **Impact:** This parsing error is causing unexpected behavior and test failures.

### To Reproduce

Just run the unit tests for v 3.0.1

### Code snippets

_No response_

### OS

macOS

### Python version

3.9.0

### Library version

3.0.1
Repository: openai/evals
Issue: Log injection alert
GitHub labels: bug

### Describe the bug

Potential log injection alert here - https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L187-L187

### To Reproduce

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L187-L187

### Code snippets

```Python
https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L187-L187
```


### OS

macOS

### Python version

3.11.4

### Library version

1.40.1
Repository: openai/evals
Issue: Information exposure alert through an exception
Root cause: Validation Gap

### Describe the bug

Stack trace information flows to this location and may be exposed to an external user.
 
Affected line - https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L191-L191

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L164-L170

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L119-L124

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L61-L65

### To Reproduce

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L191-L191

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L164-L170

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L119-L124

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L61-L65

### Code snippets

```Python
https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L191-L191

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L164-L170

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L119-L124

https://github.com/openai/evals/blob/234bcde34b5951233681455faeb92baaaef97573/evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py#L61-L65
```


### OS

macOS

### Python version

3.11.4

### Library version

1.40.1
Repository: openai/evals
Issue: Building an MMLU Eval issue
Root cause: Environment Incompatibility

I am trying to execute the Building an MMLU Eval jupyter notebooks
all of the cells execute correctly until I execute the following code:

!oaieval gpt-3.5-turbo match_mmlu_anatomy

I receive the following error:

Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/bin/oaieval", line 5, in <module>
    from evals.cli.oaieval import main
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/__init__.py", line 1, in <module>
    from .api import check_sampled_text, completion_query, sample_freeform
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/api.py", line 9, in <module>
    from evals.base import ModelSpec
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/base.py", line 93, in <module>
    class ModelSpecs:
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/base.py", line 125, in ModelSpecs
    def names(self) -> dict[str, Sequence[str]]:
TypeError: 'type' object is not subscriptable



I tried executing using python 3.10 and still had the issue.

I got several of similar error while trying to run it in Python 3.8, I quit trying to fix it because several other places were also not compatible with 3.8. There is a bug in the file "pyproject.toml" where it says:
requires-python = ">=3.8"
it should be:
requires-python = ">=3.9"

It's a bit hard to diagnose the issue - When you switch between python versions, did you ensure to reinstall the CLI - `oaieval` is pointing to the right path?

Closing as stale, please file a new issue if you're still seeing this
Repository: openai/evals
Issue: o1 release breaks token usage stats
Root cause: Interface Contract Mismatch

### Describe the bug

```
Traceback (most recent call last):
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/bin/oaieval", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/lib/python3.12/site-packages/evals/cli/oaieval.py", line 304, in main
    run(args)
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/lib/python3.12/site-packages/evals/cli/oaieval.py", line 227, in run
    add_token_usage_to_result(result, recorder)
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/lib/python3.12/site-packages/evals/cli/oaieval.py", line 279, in add_token_usage_to_result
    key: sum(u[key] if u[key] is not None else 0 for u in usage_events)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unsupported operand type(s) for +: 'int' and 'dict'
```

### To Reproduce

Run any eval

### Code snippets

_No response_

### OS

macOS

### Python version

v3.12

### Library version

3.0.1.post1
Repository: openai/evals
Issue: Text2code2video eval
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

I found that also that last released models (o1 family) fail in text to visual coding tasks.
I am guessing if a text2shader dataset could be a valid approach to improve on this domain.
It is basically like text2image or text2video but more in the flavour text2code2video so that we have a intermediate programmable artifact.
I don't know if [shadertoy](https://www.shadertoy.com/) or any other resource could help to build this eval.


### Additional context

_No response_
Repository: openai/evals
Issue: Is Evals repo being replaced by the Evaluations feature in the Playground?

### Describe the feature or improvement you're requesting

We noticed that there is a new feature in the Playground called "Evaluations," which seems to accept a similar file format to the Evals repo—though not exactly the same. We're wondering if this new feature is intended to supersede the Evals repo. If that's the case, will there be support for importing from the existing Evals structure to ease the transition?

### Additional context

_No response_

Good question.