zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: Access not working
Root cause: Documentation Deficiency

### Describe the feature or improvement you're requesting

(I'm sorry for the noise, but as OpenAI staff said in some threads, they will likely not see replies to closed/merged issues)

In #1121 I was told that GPT4 access had been granted to actually both of my emails, but I don't see it on either one. I actually tried both logging with an OAuth gmail account (what I had done previously) or with the email directly and neither work. Curious if you could take a look @usama-openai 

### Additional context

_No response_

Can you try https://platform.openai.com/playground?mode=chat&model=gpt-4?

Ah, that works! I didn't realize it would only be under "Chat" and not "Complete" -- makes sense. And I shouldn't be expecting to show up on https://chat.openai.com/, then? That's fine, just checking.
Repository: openai/evals
Issue: You should see GPT-4 API access enabled in your account in the next few days.
Root cause: Others

You should see GPT-4 API access enabled in your account in the next few days.

_Originally posted by @usama-openai in https://github.com/openai/evals/pull/1187#issuecomment-1597602662_

Please can I have access to the GPT 4 API? My email is jonathan@verhe.es

Many thanks

@usama-openai @luqman-openai @jorge-openai @andrew-openai
Repository: openai/evals
Issue: closedqa prompt is not adequate for gpt-4-0613
Root cause: Validation Gap

It seems that GPT-4 neglects to follow the instructions in the closedqa prompt much more than gpt-3.5-turbo.  See, for example, https://github.com/openai/evals/issues/1200#issuecomment-1605238900 where gpt-4 gives 9 invalid responses out of 47, while gpt-3.5-turbo does not give any invalid responses.  Does this hold across the other evals in the repo?
Repository: openai/evals
Issue: 国际化的支持
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

不能友好的识别中文的含义
<img width="739" alt="image" src="https://github.com/openai/evals/assets/29973427/db688ef9-8797-4893-8acf-5a1f20cd23d6">

明明明明明白白白喜欢欢欢应该这样断句：明明，明明，明白，白白，喜欢，欢欢。
明明（第一个）、白白、欢欢是人名，第二个明明是清楚的意思。
### Additional context

_No response_

> ### Describe the feature or improvement you're requesting
> 不能友好的识别中文的含义 <img alt="image" width="739" src="https://user-images.githubusercontent.com/29973427/248752280-db688ef9-8797-4893-8acf-5a1f20cd23d6.png">
> 
> 明明明明明白白白喜欢欢欢应该这样断句：明明，明明，明白，白白，喜欢，欢欢。 明明（第一个）、白白、欢欢是人名，第二个明明是清楚的意思。
> 
> ### Additional context
> _No response_

Here is the repo of openai evals, and your question is about ChatGPT. You should not ask your question here! What's more, it seems that you are using a third-party chatgpt-like service, which can be the cause of the problem.

> > ### Describe the feature or improvement you're requesting
> > 不能友好的识别中文的含义 <img alt="image" width="739" src="https://user-images.githubusercontent.com/29973427/248752280-db688ef9-8797-4893-8acf-5a1f20cd23d6.png">
> > 明明明明明白白白喜欢欢欢应该这样断句：明明，明明，明白，白白，喜欢，欢欢。 明明（第一个）、白白、欢欢是人名，第二个明明是清楚的意思。
> > ### Additional context
> > _No response_
> 
> Here is the repo of openai evals, and your question is about ChatGPT. You should never ask your question here! What's more, it seems that you are using a third-party chatgpt-like service, which can be the cause of the problem.



seems interesting. lol
Now I think is OK！
Repository: openai/evals
Issue: gpt-4-32k

### Describe the feature or improvement you're requesting

Do you support adding evals for long inputs?
For example: https://arxiv.org/abs/2305.14196 ?

### Additional context

_No response_

(In the paper, they only evaluate up to 8k w. GPT-4)

Tagging the authors @Mivg @UriSha

Cool paper man!

Yes, we will definitely merge long context evals!

Since this is answered (yes long context evals are welcome!) i'm going to close
Repository: openai/evals
Issue: Expose run_id to code being run within an eval
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

I'd like to use this to label information created during an eval run (e.g. the filenames of system log files), to allow it to be correlated with the evals logfiles.

### Additional context

Example log file name:
/tmp/evallogs/230629131228Z4SV7V2E_completionfnname_evalname.jsonl

Example of system log file I'd like to write from my own code:
230629131228Z4SV7V2E.log

Perhaps it could be made available to the CompletionFn

It can be accessed from `recorder.run_spec.run_id` but it's not convenient to get it through to the completion function.
Repository: openai/evals
Issue: Code Evals 
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

I wonder if anyone has a solid method for evaluating code benchmarks like APPS.
String typed codes can be very noisy and require deliberate preprocessing to be executed and tested.
I don't see any class inheriting Evals that can perform code tests.

Any clue? :thinking: 

### Additional context

_No response_
Repository: openai/evals
Issue: oaieval hangs a lot
Root cause: Resource Mishandling

### Describe the bug

`oaieval` hangs near the end, before reporting, a lot.

### To Reproduce

```
✗ EVALS_THREADS=12 EVALS_THREAD_TIMEOUT=10 oaieval gpt-3.5-turbo myeval
[2023-06-28 18:09:56,280] [registry.py:266] Loading registry from /Users/username/development/evals/evals/registry/evals
[2023-06-28 18:09:56,615] [registry.py:266] Loading registry from /Users/username/.evals/evals
[2023-06-28 18:09:56,617] [oaieval.py:138] Run started: runid
[2023-06-28 18:09:56,618] [data.py:83] Fetching myeval/samples.jsonl
[2023-06-28 18:09:56,619] [eval.py:33] Evaluating 69 samples
[2023-06-28 18:09:56,627] [eval.py:139] Running in threaded mode with 10 threads!
 99%|█████████████████████████████████████████████████████████████████████████  | 68/69 [00:20<00:00,  7.93it/s]
```
style of call hangs at this point for many minutes, even though `EVALS_THREAD_TIMEOUT` is set to 10 seconds. This is devastating to eval turnaround time.

### Code snippets

_No response_

### OS

macOS Ventura (13.4)

### Python version

3.11.3

### Library version

1.0.3

Indeed, I have encountered a similar issue. It appears that the output might take an extended period to return. In my specific experience, the model only completed its return after generating all 4097 tokens. To address this problem, I found success in augmenting the EVALS_THREAD_TIMEOUT to 500, as the previous setting of 100 was insufficient for my requirements.
Repository: openai/evals
Issue: All evals currently in the repo appear only to have dev samples: is this correct?
Root cause: Documentation Deficiency

### Describe the feature or improvement you're requesting

The documentation speaks of dev, val, and test splits, but all evals currently in the repo appear only to have dev samples. Is this correct, or are we testing against hidden test samples when we run evals?
It would be great if this could be made clearer in the documentation (as well as what the distinction is between "dev" and "val", which is not self-evident -- I'm used to "development set" and "validation set" being synonyms...)

### Additional context

_No response_

We're using `dev` to indicate the eval may still be updated or changed - we've merged some corrections to evals in the past. The repo does not make a technical distinction between `dev`/`train`/`test` evals, those headings are mostly for your organization.
Repository: openai/evals
Issue: Accuracy Score
Root cause: Documentation Deficiency

### Generic question about the accuracy score and boostrap_std metric

When I run an eval, I got the following report.
`{'accuracy': 0.6, 'boostrap_std': 0.1423900220076777}`

How to decide the accuracy is good or not good?
Can the Accuracy score 0.6 be considered as good or not good? At below which point, We should consider that the Model's performance is not good for that specific eval?

Also, Can anyone explain what is the `boostrap_std` metric?

### Additional context

_No response_
Repository: openai/evals
Issue: Sample evaluations completing after timeout cause duplicate results
Root cause: Resource Mishandling

### Describe the bug

There is a default 40s timeout for completion functions as per EVALS_THREAD_TIMEOUT.

In eval.py, when evaluating a sample times out, it is retried. However it appears that the original invocation can still complete later and thus we end up with multiple sampling/match events in the log file for that sample line.

### To Reproduce

1. Run an eval of N samples, some of which will be slow
2. Observe multiple events in the log file for samples which were retried

To easily find which samples were retried, put a breakpoint/logging in the timeout catch of the worker thread.

### Code snippets

_No response_

### OS

Windows 10

### Python version

4.11

### Library version

1.0.3.post1

This may well be a duplicate of #955 which has been fixed but not released due to #1344
Repository: openai/evals
Issue: Having trouble building Evals locally? Try this.
Root cause: Environment Incompatibility

### Describe the bug

I used 

> git clone git@github.com:openai/evals.git

and couldn't get `pip install -e .` to run.  

Error logs said I was missing a setup.py or setup.cfg file.  That's odd.  Poking around, I realized that my `git clone` hadn't copied pyproject.toml and mypy.ini into my local directory. 

No problem, I manually downloaded those files and voila it compiled. 

So then I ran `oaieval gpt-3.5-turbo test-closedqa-uniqueness` and got this error: ModuleNotFoundError: No module named 'evals.utils'

Upon closer inspection, it was the "git clone" issue again!  For some reason, cloning the repo isn't bringing in all of the files and folders. Sure enough, I was missing the utils folders under /evals/

I downloaded the entire repo as a Zip file. That zip contained all of the files, at which point building and running the `oaievals` worked.


### To Reproduce

1. git clone this main branch
2. run 'pip install -e .'
3. it will fail. 

### Code snippets

_No response_

### OS

macOS

### Python version

Python 3.9.7

### Library version

openai-evals 1.0.3.post1

I posted a similar issue when this repository first came out [here](https://github.com/openai/evals/issues/269). Make sure to update your pip!
Repository: openai/evals
Issue: Publish latest evals framework to PyPI
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

The have been numerous improvements and fixes to the evals framework itself over the past few months, but these haven't been released to PyPI. It would be great for that to happen.

There is the option of installing a specific version of the main branch direct from the repo using pip, but there are no tags of versions that are confirmed to be good/tested for use.

### Additional context

The last release was evals 1.0.3.post1 on Apr 17, 2023.

https://pypi.org/project/evals/
Repository: openai/evals
Issue: Feature request for evals:  Add support for function call.
Root cause: Unimplemented Feature Gap

###  Add support for function call.

I would like to `eval` based on prompts that utilize `function_call`. 
From I have seen in the code, it's not possible at the moment. 
I observed a similar request [here](https://community.openai.com/t/how-do-we-submit-evals-for-function-calls/270038).
Is there any plan to implement this?
Repository: openai/evals
Issue: Evaluate the cost of running tests 
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

In many production scenarios, it is important to do cost-benefit analysis, and it will be great if `oaieval` command can also return the total cost of running the test.

Specifically, it will be 2 parts:
1. in oaieval command, add optional param to specify whether you want to output the cost 
2. an optional interface in `CompletionFn` to calculate the costs of running that Completion

### Additional context

_No response_
Repository: openai/evals
Issue: Use github.com/apssouza22/chatflow as a conversational layer. It would enable actual API requests to be carried out from natural language inputs.

### Describe the feature or improvement you're requesting

Adding this conversational UI would enable people to 'talk' directly with the backend and API requests to be carried out more effectively. RAG can help with some of the problems function calling by language models face at the moment.

https://youtu.be/r3cegH2kviQ

.

### Additional context

I'm trying to accelerate the adoption of natural language interfaces
Repository: openai/evals
Issue: Context window of completion functions not accounted for
Root cause: Validation Gap

### Describe the bug

It seems that some evals require specific context window length, ex: `make-me-say` eval probably requires 32k?

It would be nice if there was a more DX friendly to know about this before it errors in the API call?

### To Reproduce

`oaieval gpt-3.5-turbo,gpt-3.5-turbo,gpt-3.5-turbo make-me-say --debug`

This model's maximum context length is 4097 tokens. However, your messages resulted in 4123 tokens. Please reduce the length of the messages.

### Code snippets

_No response_

### OS

macOS

### Python version

Python v3.9.7

### Library version

openai-evals 1.0.3
Repository: openai/evals
Issue: Multiple evals not found
Root cause: Documentation Deficiency

### Describe the bug

When running evals such as the steganography eval, I get: AssertionError: Eval steganography.yaml not found

### To Reproduce

Install and set up everything according to the documentation
oaieval gpt-3.5-turbo steganography

### Code snippets

_No response_

### OS

macOS

### Python version

Python 3.9

### Library version

main

Hello, you need to specify 2 completion functions to make this eval work:
`oaieval gpt-3.5-turbo,gpt-3.5-turbo steganography` 

It works for me at least.

> Hello, you need to specify 2 completion functions to make this eval work: `oaieval gpt-3.5-turbo,gpt-3.5-turbo steganography`
> 
> It works for me at least.

Unfortunately that still doesn't work for me. Did you change anything else too?

Did you pull the git lfs files? Is your main branch up to date?

I now get this issue after it starts running: UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 7: ordinal not in range(256)

That I have no clue about, I'd say reinstall, repull the files, etc
Repository: openai/evals
Issue: Should random collection of values be supported?
Root cause: Unimplemented Feature Gap

https://github.com/openai/evals/blob/dd96814dd96bd64f3098afca8dc873aa8d8ce4c8/evals/elsuite/basic/match.py#L39-L44

refer to `evals/registry/evals/cissp-study-questions.yaml`

`few_shot_jsonl` has 143 pieces of data, but `num_few_shot` only takes the first 4 items.
Repository: openai/evals
Issue: In the task "balance_chemical_equation", many instances have incorrect labels.
Root cause: Validation Gap

### Describe the bug

For example
```
You are ChemistGPT, can help user balance chemical equation. For example, if user's input is "C6H5COOH + O2 = CO2 + H2O", you will reply the balanced chemical equation: "2C6H5COOH + 15O2 = 14CO2 + 6H2O", without explanation. If you can't balance the equation, just reply "Unknown"

Input: Al(OH)3 + H2SO4 = Al2(SO4)3 + H2O
Answer: 
```
The given groundtruth is 
```
2Al(OH)3 + 3H2SO4 = Al2(SO4)3 + 3H2O
```

but the real correct label is 
```
2Al(OH)3 + 3H2SO4 = Al2(SO4)3 + 6H2O
```

There are many problems like this, hope to fix. 
@scruel 


### To Reproduce

Nothing

### Code snippets

```Python
Nothing
```


### OS

macOS

### Python version

Python3.11

### Library version

openai-evals latest

Hi, thank you for pointing out those incorrect labels, I would like to fix them, however currently I don't have enough time for this.
Anyway, you're welcome to create a PR to fix this.
Repository: openai/evals
Issue: Dataset hosting, data cards and previews
Root cause: Unimplemented Feature Gap

Hi, I'm Quentin from Hugging Face :)

I know hosting datasets on github is not always practical: git lfs required, no data preview, limited storage (maybe not for you haha), no standard for data documentation. So I was wondering:

Have you considered host alternatives more suited for datasets, and would let researchers explore the datasets of `evals` ?

This way researchers can know in depth what data is used for evaluation and their goals and limitations, in particular to better understand what domains and structures their models perform good or bad at.

e.g. the Hugging Face datasets hub shows data cards for documentation and previews for each dataset. Also loading and caching a dataset is one line of python, saving you from `wget` and github hosting. It also supports pull requests for the community to contribute.

It can even allow to use those datasets in other well known eval frameworks, such as lm-evaluation-harness.

Let me know what you think !

I love hugging face! Worth considering at some point soon, will explore internally over the next few weeks.

Thanks for stopping by! HuggingFace datasets is great.

Many of our evals are only a few samples long (10-20), which we were worried to be too small to host as individual datasets on Hugging Face Datasets. We needed a platform to support lots of small datasets which is why LFS seemed to work OK for our task.

If you think this is still a reasonable use case for HuggingFace Datasets, I'd be happy to help any efforts in mirroring them onto HuggingFace!

@logankilpatrick  @andrew-openai hi, I'm Polina from HF datasets team :) regarding your worrying about many small datasets - in  `datasets` it's possible to host more then one dataset as a single dataset with many subsets, like it's done for benchmarks like [glue](https://huggingface.co/datasets/glue).  
Also feel free to ping me if you have any questions about adding datasets to the Hub :)

Has any further consideration taken place over a Hugging Face dataset mirror?
Repository: openai/evals
Issue: Evals broken with latest openai package v1.1.1
Root cause: External Dependency Breakage

### Describe the bug

`openai` package v1.1.1 has changed their error handling. Now `openai.error` raises an `AttributeError`. [These lines](https://github.com/openai/evals/blob/main/evals/utils/api_utils.py#L17) will raise an `AttributeError` when running any eval, as this is part of the backoff logic which is used for both [base](https://github.com/openai/evals/blob/4b7a66bd45f06156656e021e170e7574f6cde3f5/evals/completion_fns/openai.py#L83) and [chat](https://github.com/openai/evals/blob/4b7a66bd45f06156656e021e170e7574f6cde3f5/evals/completion_fns/openai.py#L129) completion functions.

### To Reproduce

1. Clone repo
2. Install dependencies
3. Run `oaieval gpt-4 2d_movement`

### Code snippets

_No response_

### OS

macOS

### Python version

Python v3.9

### Library version

openai-evals v0.1.1

encounter the same issue here

encountered the same issue here. getting "module 'openai' has no attribute 'error' when trying to run the "make me pay" eval
Repository: openai/evals
Issue: Do not back off on `openai.BadRequestError`
Root cause: Validation Gap

After [the recent change](https://github.com/openai/evals/blob/a06a07bbfe21d6ff13d62f98ef464021225a4987/evals/utils/api_utils.py#L17) we back off on `openai.APIError`. This means backing off on `openai.BadRequestError`:

```
>>> issubclass(openai.BadRequestError, openai.APIError)
True
```

I don't know all the cases where we can get a `BadRequestError`, but one of them is exceeded context length:
```
[2023-11-14 11:09:32,241] [_common.py:105] Backing off openai_completion_create_retrying(...) for 17.0s (openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8001 tokens, however you requested 8184 tokens (7672 in your prompt; 512 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}})
```

And in this case we should definitely avoid repeating the request.

(I found this on #1407, but I don't think this is related to this particular PR)

NOTE: logic corresponding to backing off on `openai.APIError` is in a few different places, I think all of them need the same fix.

Resolved in https://github.com/openai/evals/pull/1420, we should now only be backing off for errors that could work on retry (429s, connection errors, timeouts, 5xxs)
Repository: openai/evals
Issue: Proposal for Adding a New Evaluation Metric: Sentiment Analysis Accuracy
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

Description:

I propose the addition of a new evaluation metric, "Sentiment Analysis Accuracy," to enhance the existing evaluation capabilities of the project. This metric will focus on assessing the model's performance specifically in sentiment analysis tasks.

Motivation:

The current evaluation metrics provide valuable insights, but there is a growing need for a specialized metric to evaluate sentiment analysis accuracy. Sentiment analysis is a common and crucial task in natural language processing, and having a dedicated metric will enable more fine-grained assessment of the model's performance in this domain.

### Additional context

_No response_
```[tasklist]
### Tasks
```
Repository: openai/evals
Issue: Improvements to `Match`: case insensitive and strip
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

The current implementation of the [`Match`](https://github.com/openai/evals/blob/main/evals/elsuite/basic/match.py) basic eval template is case-sensitive. This leads to results such as: 

`{'correct': False, 'expected': 'no', 'picked': None, 'sampled': 'No', 'options': ['no']}`

Similarly, Match does not strip the sampled string from white spaces in the front. That causes the evaluation to fail for models using the Completion endpoint, as those are more likely to output spaces in the front. Example:

`{'correct': False, 'expected': 'Mumbai', 'picked': None, 'sampled': ' Mumbai', 'options': ['Mumbai']}`

It would be good to add an argument to `Match` allowing to require case insensitive behaviour and to determine whether the answer should be stripped of spaces. These can then be specified in the yaml file for a task. 

Similar options can be added for the other templates, such as `Includes` and `FuzzyMatch`. 

### Additional context

_No response_