zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: Suggestion: add git-hook hook for pre-commit

Since a run of pre-commit is required before creating a PR, it would be reasonable to add pre-commit as git-hook. We use this in our development repos and have very good experience since it ensures everyone is actually running pre-commit.

At the moment, it looks like only part of the repo is actually conveying the code standards enforced by the pre-commit tasks, as can be verified by running `pre-commit run --all-files`. Running pre-commit in the CI might help-as well.

If you think that sounds reasonable, I'd be happy to create a PR.

I think ```pre-commit install``` will install it as an actual git-hook, but I see your point that it is not committed into the repo by default.

There may be some intention or concern to not do so because the comments mention that it is slightly dangerous:

https://github.com/openai/evals/blob/f118fca38e3dde127d2eba44374806f581b0da1c/.pre-commit-config.yaml#L16-L29

A manual run adds a kind of self-check to confirm the existence of unused-in-file imports. I see this visual check as a good thing in library imports like `__init__.py` (which is excluded in the code above) where the intention is to expose functionality but not necessarily use them within the file.
Repository: openai/evals
Issue: Doesn't answer every aspect of the task

On the openai playground, the ai doesn't answer every aspect of the question. I asked it to do a 20 line essay about how ww2 was a annihilation war (for school) in French - and it didn't answer the entire task. Missed out on defining the term "annihilation war" like I asked it to and did only 5 lines about the subject. I asked the same thing to the public version of chatgpt and it did it no problem. Might be a bug.
Thanks for your great product. Truly a step forward in the history of mankind.
Matt
Repository: openai/evals
Issue: GPT 4 Image Evals
Root cause: Unimplemented Feature Gap

Hello, I wanted to open an issue/question regarding GPT-4 evals with images. I have a few questions as I'm looking to write some evals to test a few ideas I had. 

1. Is there a rough timeline for when GPT-4 with image scanning will be available?
2. When this feature comes out, will queries have to be done individually? Can we create a batch of images and include a prompt?
3. If someone were to process large quantities of large images, say 10k images a day, what would that look like cost wise?

I plan to write an eval to test a significantly large dataset of high-quality 360 images of cars and answer questions the end user of a site like Carvana may have. Questions like:

- Does this car have any visible markings, dents, or damage on the surface?
- Will this car be able to take me off the road when I go camping?
- Is this a good fuel-efficient car for me to use?
- ...

Would an evaluation like this be helpful to the public? I want to test this for my needs, but if this would be useful, I can take a fragment of the set and upload it somewhere once I write the prompts. 

Great Idea

That is a great question regarding the cost of image processing.

I'm curious; I've reached out to the OpenAI team but haven't heard back. My guess is they won't have pricing for a while, it seems like getting GPT-4 to scale for more users is the current challenge. 

I think one of the issues is the availability of the Nvidia chips needed to power the platform.

We are interested in supporting image evals as soon as that feature becomes more widely available and the API is finalized.

@andrew-openai that's great! We are super excited for that! We're looking forward to the image feature as well as taking advantage of plugins to create tools that can interact with our vehicle databases. Hopefully this is something we can set up soon to start testing!

Going to close for now, please feel free to reopen once you start building the eval :)
Repository: openai/evals
Issue: The metrics valus are not right
Root cause: Algorithmic Error

When running `python oaieval.py gpt-3.5-turbo chess-match     --record_path ./output/chess-match` the evaluation value is not right, see the specific examples:

![image](https://user-images.githubusercontent.com/16098181/227223413-daa98ad4-f338-4218-85bb-5636329c105a.png)

![image](https://user-images.githubusercontent.com/16098181/227223585-b82997de-c013-4808-b53a-cead3ec5bc67.png)
Repository: openai/evals
Issue: Not helpful for professionals.

Chat gpt fails to help professionals. 
If a Lawyer, Chartered Accountant while in search of case laws, judgements, Acts and other similar things.
 Or  A  Doctor while in search of medical test, article's or other such things Chat gpt did not have the capability to provide personalised results. 

If chat gpt trained to provide personalised results for professionals it will hit both bigger and targeted markets.

Chat gpt can be trained to be an ai that works differently based on the category of the user. If a user is a medical professional then it can give more advanced results in the medical and related categories. If a user is a lawyer he can be provided with more advanced results in a law and legal categories. 

And for the categories that a user was not identified as an expert in it, then he can be provided with the simple and easy understandable results. 

The above capability will benefit in larger scale.

This repository is not a general issue tracker for suggestions or feedback for ChatGPT. Please either give a context to the evals in this repository or close this issue.

Chat GPT is helpful in general I would conflict with what you are saying chat gpt is certainly helpful so I would say that your headline chatgpt fails to help professionals is wrong as chatgpt is actively helping software develoeprs and professionals.
Repository: openai/evals
Issue: Modelgraded often selects the wrong choice when using cot_classify
Root cause: Algorithmic Error

## Problem

When using cot_classify, the get_choice function will often match the wrong answer. 

get_choice works correctly when the model produces an answer on a different line

<img width="819" alt="correct" src="https://user-images.githubusercontent.com/50199412/227408811-159dfa59-a596-4ecf-911f-7d45c7f06ab0.png">

However, when the model writes the response on the same line, the get_choice function matches on the 'A' in answer.

<img width="1099" alt="incorrect" src="https://user-images.githubusercontent.com/50199412/227408813-7a1dedc2-941d-4867-a01b-b52df2a35584.png">

## Cause

The match function `"starts_or_endswith": lambda x, y: x.startswith(y) or x.endswith(y)` will match on the capital A in Answer from the examples.


Help

,, neuron "

Oops, found the way to select a match function. 😂
Repository: openai/evals
Issue: Neuron 


Help

@SierotkaM Hey! How can I help you?

@SierotkaM A neuron, also known as a nerve cell, is a specialized cell that serves as the basic building block of the nervous system. Neurons transmit information through electrical and chemical signals, allowing communication between the brain, spinal cord, and other parts of the body.

The primary components of a neuron include:

1. Cell body (soma): The central part of the neuron, which contains the nucleus and other cell organelles necessary for the cell's functioning and maintenance.

2. Dendrites: Short, branch-like structures that extend from the cell body. They receive signals from other neurons or sensory receptors and transmit the information toward the cell body.

3. Axon: A long, slender projection that extends from the cell body. The axon carries electrical signals (called action potentials) away from the cell body to other neurons, muscles, or glands.

4. Myelin sheath: A fatty, insulating layer that wraps around the axon in segments, helping to speed up the transmission of electrical signals.

5. Nodes of Ranvier: Small gaps between the myelin sheath segments where the axon is exposed, which help to facilitate the rapid transmission of electrical signals along the axon.

6. Axon terminals (synaptic terminals): The ends of the axon that form connections with other neurons, muscles, or glands. These connections, called synapses, allow for the transmission of chemical signals (neurotransmitters) between cells.

Neurons can be classified based on their function:

1. Sensory neurons: These neurons detect and transmit information from sensory receptors (such as those in the eyes, ears, and skin) to the central nervous system (brain and spinal cord).

2. Motor neurons: These neurons transmit signals from the central nervous system to muscles or glands, causing them to respond accordingly.

3. Interneurons: These neurons serve as connectors between other neurons, allowing for information processing and communication within the central nervous system.

@Ein-Tim, it has to be a bot. It gave "X" to pull requests that didn't meet the criteria. It would post blank on PRs with the evals empty. Does not seem like human behaviour. Must be some experiement of some sorts.

@mmtmn Either something like this, but I think it's rather a really confused user or a troll, I already pinged @andrew-openai in another PR/issue so that he takes a look at it. 

@Ein-Tim a confused user can't review pull requests or ping Andrew...

@mmtmn He definitely performs tasks, creates issues and uses GitHub's e-mail reply feature to spam unnecessary stuff in these repositories... 

@Ein-Tim I think the account has access on purpose, not sure. @SierotkaM I made a [neurons eval](https://github.com/openai/evals/pull/474)

Since this user was spamming a few PRs, we restricted their ability to interact with the repo.
Repository: openai/evals
Issue: openai key works, but not work for oaieval
Root cause: Configuration Error

when I run : ```oaieval gpt-3.5-turbo test-match ``` , I always have "openai.error.AuthenticationError: No API key provided. " but I can get response running the official examples like 
```curl https://api.openai.com/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer $OPENAI_API_KEY"   -d '{
  "model": "gpt-3.5-turbo",
  "messages": [{"role": "user", "content": "Say this is a test!"}],
  "temperature": 0.7
}'
```
It seems the openai key works, but not work for oaieval
Repository: openai/evals
Issue: Handling new lines in evaluation?
Root cause: Documentation Deficiency

Hi!

I am planning on creating an evaluation but I am still thinking about how to write new lines in the evaluation _.jsonl_ file. Would the model see "\n" exactly the same as I do when I enter this:

![image](https://user-images.githubusercontent.com/32551374/227522289-5d16a787-1750-45dd-8cf1-f5c3a94b2449.png)

into the prompt? How about with "\t"?

Regards, Rasmus

Mi kiedyś Gpt 3 taka wykreował
r  n  b  q  k  b  n  r
p  p  p  p  p  p  p  p
.  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .
P  P  P  P  P  P  P  P
R  N  B  Q  K  B  N  R
Jakoś trzeba mu oznaczenia szachowe  jak na szachownicy zrobic

pt., 24 mar 2023, 13:34 użytkownik Rasmus Lindahl ***@***.***>
napisał:

> Hi!
>
> I am planning on creating an evaluation but I am still thinking about how
> to write new lines in the evaluation *.jsonl* file. Would the model see
> "\n" exactly the same as I do when I enter this:
>
> [image: image]
> <https://user-images.githubusercontent.com/32551374/227522289-5d16a787-1750-45dd-8cf1-f5c3a94b2449.png>
>
> into the prompt? How about with "\t"?
>
> Regards, Rasmus
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/431>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6PJHHXLYSIJTIWMIYYXO5DW5WIE3ANCNFSM6AAAAAAWGQDFJQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Yes, '\n' will work in the "ideal" part of the `.jsonl` file. If you ask it to output values on newlines, you can see that the model outputs '\n' between lines in the 'sampled` part of the logs.

Gogle robi problemy z czego nie korzystam przy szkoleniu Gpt jest opatrzone
informacja ,że to ich i niby tylko informują Ale czuje się presję .proszę
oi szachy do dyspozycji i tłumaczenie ekranu

pt., 24 mar 2023, 16:49 użytkownik Chris Sypherd ***@***.***>
napisał:

> Yes, '\n' will work in the "ideal" part of the .jsonl file. If you ask it
> to output values on newlines, you can see that the model outputs '\n'
> between lines in the 'sampled` part of the logs.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/431#issuecomment-1483025394>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6PJHHR3U6G257AOJHWH6MTW5W67NANCNFSM6AAAAAAWGQDFJQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
Repository: openai/evals
Issue: Database Not updated in 2 years

I was looking something up that is pretty current relevant knowledge and Chat got wasn't able to work with that prompt because it cut off in 2021 and can't actually visit or scrub websites even though I just uploaded to the Plus version for $20 and I thought it was supposed to be more comprehensive. Anyone can let me know more about this because I'm fairly new to this world but want to understand.

You can use the GPT-4 model, which has a more recent date cut-off. Anyway, this repository is not intended for general issue tracking of problems/questions reg. ChatGPT, but is specifically for evals. I therefore suggest closing this issue.

I thought I was evaluating so I could get the api faster but I'm sorry I don't think I've got it right
Repository: openai/evals
Issue: Generate JSONL from other formats: xlsx, csv, tsv, json
Root cause: Unimplemented Feature Gap

Hello, while I wrote PR, I had a problems with jsonl generate.

Personally, I wrote simple generator that converts yml to jsonl, with repeat of the system message.

Then I found that python `openai` package have cli tool for generate jsonl for training from different formats.

Example:
```sh
openai tools fine_tunes.prepare_data -f date-booking.xlsx
```

I think that generator from xlsx will make evals creating more simple, so professionals, but not programmers can make tests better.

Just as idea, I don't need for it anymore, but I would be glad if it was already written before my generator :) So just issue and no PR from me.

Created issue in `evals` project instead of https://github.com/openai/openai-python, because I think people will search it here.

Thanks for flagging! Great find, we'll add it to the README
Repository: openai/evals
Issue: Wolfram Plugin - Should it be taken into account when making evals? [Question]

I was curious about this part:

README.md
"(...)measure the quality of completions provided by an OpenAI model, and compare performance across different datasets and models"

So, should the Wolfram Plugin be taken into consideration when making new evals?

If you're using wolfram lang for data analysis then you can use it else not necessary.
Repository: openai/evals
Issue: A little Bug

On the openai playground, the ai doesn't answer every aspect of the question. I asked it to do a 20 line essay about how ww2 was a annihilation war (for school) in French - and it didn't answer the entire task. Missed out on defining the term "annihilation war" like I asked it to and did only 5 lines about the subject. I asked the same thing to the public version of chatgpt and it did it no problem. Might be a bug.
Thanks for your great product. Truly a step forward in the history of mankind.
YX


niedz., 26 mar 2023, 14:04 użytkownik YX2953714689 ***@***.***>
napisał:

> On the openai playground, the ai doesn't answer every aspect of the
> question. I asked it to do a 20 line essay about how ww2 was a annihilation
> war (for school) in French - and it didn't answer the entire task. Missed
> out on defining the term "annihilation war" like I asked it to and did only
> 5 lines about the subject. I asked the same thing to the public version of
> chatgpt and it did it no problem. Might be a bug.
> Thanks for your great product. Truly a step forward in the history of
> mankind.
> YX
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/458>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6PJHHT76M33ZGV6LWO57KLW6AWDLANCNFSM6AAAAAAWIEM6XI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


S

This repository is not a general issue tracker for suggestions or feedback for ChatGPT. Please either give a context to the evals in this repository or close this issue.
Repository: openai/evals
Issue: Doesn't answer every aspect of the task

On the openai playground, the ai doesn't answer every aspect of the question. I asked it to do a 20 line essay about how ww2 was a annihilation war (for school) in French - and it didn't answer the entire task. Missed out on defining the term "annihilation war" like I asked it to and did only 5 lines about the subject. I asked the same thing to the public version of chatgpt and it did it no problem. Might be a bug.
Thanks for your great product. Truly a step forward in the history of mankind.

Duplicate of #458
Repository: openai/evals
Issue: Unable to run evals
Root cause: Environment Incompatibility

I could use some help figuring out the cause of the following error.

Whenever i try to run an eval, i get an error similar to the one below:
```
FileNotFoundError: [Errno 2] No such file or directory: '/Users/wmn/.pyenv/versions/3.11.2/lib/python3.11/site-packages/evals/registry/data/test_match/samples.jsonl'
```

I followed the instructions to download all evals; running:
```
git lfs fetch --all
git lfs pull
```

I can also confirm that the `test_match/samples.jsonl` file does exist. 

Right
Repository: openai/evals
Issue: "Includes" evaluation returns False when "ideal" text is at the end of generated text
Root cause: Algorithmic Error

I have a dataset for evaluation where the "ideal" answer is expected to be at the end of the generated text. E.g. if "XXX" is the ideal answer, generated text should look like: "some generated text XXX".
**eval_sample** function in **Includes** class calls **evals.elsuite.utils.get_answer** function:
```
def get_answer(text, answer_prompt):
    idx = text.rfind(answer_prompt)
    if idx == -1:
        return None
    return text[idx + len(answer_prompt) :]
```
In my scenario, _idx + len(answer_prompt)_ equals the lengths of the text, and the **get_answer** function returns empty string. This leads to **includes_answer** being False:
```
        includes_answer = any(
            [evals.elsuite.utils.get_answer(sampled, ref) for ref in sample["ideal"]]
        )
```
But the generated text actually does include my ideal snippet, so I believe includes_answer should be True here.

FYI since been fixed here https://github.com/openai/evals/pull/972
Repository: openai/evals
Issue: "Includes" evaluation does not work if sample["ideal" ] is str and not list
Root cause: Validation Gap

eval_sample function in Includes evaluation does not work adequately if sample["ideal"] is a string. The code below checks whether each item in sample["ideal"] occurs in the generated text. If sample["ideal"] is a string, it loops through its characters (leading to a wayyy higher evaluation scores than expected).

```
        includes_answer = any(
            [evals.elsuite.utils.get_answer(sampled, ref) for ref in sample["ideal"]]
        )
```

Believe this is fixed with https://github.com/openai/evals/pull/800!
Repository: openai/evals
Issue: Support for or examples of Evals with image prompts?
Root cause: Unimplemented Feature Gap

Perhaps dumb question, feel free to close, but watching the GPT-4 livestream it seems that there is the ability to take an image prompt and do processing thereof which is our desired use case. Is there an analogous mechanism to create eval data sources with an image instead of text input?

The current model available via our API and ChatGPT does not support image inputs. We will be adding that but there's no way to do evals for it right now.

@logankilpatrick ok cool thanks for the info
Repository: openai/evals
Issue: Should data be balanced in a classification evals?

I have a classification eval with possible answers Y or N. Is it okay if I have more N than Y examples? What if the model is more likely to produce Y due to hallucinations and having more N would artificially lower the score?

If you're using this for personal use cases, you can also test for precision/recall to get better metrics with unbalanced data. If it's for a public submission, that's fine anyway to just submit as is (we would run P/R as well internally, maybe make a comment in the PR)

thanks!
Repository: openai/evals
Issue: `argparse.BooleanOptionalAction` is not available in Python 3.8
Root cause: Environment Incompatibility

`argparse.BooleanOptionalAction` is used in the module, for example, in `evals/cli/oaieval.py`. However, `BooleanOptionalAction` was not added till Python 3.9. 

The Python version requirements in the `pyproject.toml` is specified as https://github.com/openai/evals/blob/b297ecd5a613bcb502f7bdb362fc67af5143ea71/pyproject.toml#L4
Might this require changing?

They could be replaced by `action='store_true'` to ensure compatibility, but I am not sure this is worth the pain.
Repository: openai/evals
Issue: How to dynamically change API parameters through `oaieval` CLI tool?
Root cause: Documentation Deficiency

I'm trying to run an eval based on `evals.elsuite.basic.match:Match` class but I can't find a way to modify API parameters such as `temperature` and `max_tokens` through the CLI. What is the current solution that the framework offers to this problem?

It's a good idea! I added a `--modelspec_extra_options` that would support setting those values from the CLI, e.g.

```
oaieval gpt-3.5-turbo test-match --modelspec_extra_options max_tokens=50,temperature=1
```

Edit: Although that said, I am curious why you want nonzero temperature for evals, usually we want to set that value to 0 for determinism across runs.

Wow! Thank you so much for your help @andrew-openai . About the temperature, it doesn't make any sense changing it at all for match evals. You're completely right. Actually, I was just trying to mess around with API parameters in general to better understand how to work with Portuguese prompts and responses. For a moment I thought temperature could be something to that I should change but I wasn't thinking straight anymore I guess.

Again, thank you so much @andrew-openai !
Repository: openai/evals
Issue: 📌 Contributing to Project Documentation on Readthedocs??

Dear project developers,

I hope this message finds you well. I am interested in contributing to your project by creating documentation on the Readthedocs site. 

I would greatly appreciate your thoughts on this matter and any guidance you can offer. Thank you for your time and consideration.

Best regards,
TKK 

Hey, thanks for the offer, we will pass for now.
Repository: openai/evals
Issue: Performing evals on other models for comparison
Root cause: Unimplemented Feature Gap

Hi all, 

Is there an easy way to add another LLM to compare performance via an API endpoint? I've tried playing around with the repo some, but the task seems nontrivial due to the use of the openai library.

We are adding this functionality soon, you can track the PR progress here! https://github.com/openai/evals/pull/515

Closing since https://github.com/openai/evals/pull/515 is merged