zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: Bad Language

Hello, maybe its a wrong repository to share my feedback, but hope community will direct me to the right one
I am working on a personal project about surfing conditions. I asked chatGPT-3.5 turbo to talk like a characters from Futurama, depending on a surfing conditions, and thats what I got as a response. Screenshot
What do you think about a potential risks of this)))?
<img width="796" alt="Screenshot 2023-03-31 at 22 17 41" src="https://user-images.githubusercontent.com/120054360/229223318-a2003b36-05e9-4668-b0d4-9d5739eddca3.png">


Very cool project! Wrong place for this discussion, but one thing you can try is simply prompt the model to avoid using swear words. Please reach out to OpenAI support if you have further concerns.
Repository: openai/evals
Issue: A way to evaluate GPT models

One of the things that's interesting to me is how well GPT can autonomously improve AI code.    

I created this example here https://github.com/qrdlgit/gpt and it gets to about 0.972 in classifier accuracy after about 5 runs.

The ML problem is fairly basic, but it's just used as an example here.  More sophisticated problems can be used, though some attention to the max context needs to be made.  

Also, using problems which are unique and not easily solved via training data would be required for fair evaluation.  Ideally a broad cross section of different problems would be useful to capture a more accurate assessment.

One of the next things I'm going to work on is pushing this up into RLHF (RLCF?) by defining a reward model specific to this domain..  Ie, does the response compile and run, provide intelligible metrics, was there an improvement, etc.
Repository: openai/evals
Issue: Move/Include directions on how to contribute a simple eval into top-level `README.md`

I feel like one primary aim of this project is to receive high quality evals. 

Instead of the `docs` folder, I propose that the top-level `evals/README.md` should include directions on how to contribute a simple eval.

I don't propose the entirety of `evals/docs/build-eval.md` should be moved into `evals/README.md`; however, I propose something minimal and simple be included.

I recognize it isn't a big effort to go to `evals/docs/build-eval.md` but I feel like if eval contribution is a primary aim, directions should be at the forefront (versus the periphery).
Repository: openai/evals
Issue: Eval requirements when using GPT-4 for renovation price estimations?

Can I for example post an eval with pictures of houses (extracted from Matterport 360 Virtual Tours) that needs to be renovated and say that GPT-4 hallucinates most of the time what part of the house is shown inside the picture in the first place, it would for example say the kitchen is a bathroom and such, but also simply fails to accurately estimate the price for the renovation like construction contractors could do, and AGI therefore should be able to do?
Repository: openai/evals
Issue: Data formatting step outputs incorrectly keys for JSONL
Root cause: Validation Gap

https://github.com/openai/evals/blob/main/docs/build-eval.md

**Reproduction steps:**
Have a csv with `prompt` and `completion` columns
Run `openai tools fine_tunes.prepare_data -f data.csv`
Inspect output JSONL file and verify entries have `prompt` and `completion` keys but not the required `input` key


It may very well be that the csv column names should be different, but prepare_data complains about the input not having prompt and completion columns. It would be helpful if the docs explained what columns are required.
Repository: openai/evals
Issue: سلام خوبی


سلام


I suggest closing this issue.

بله، برای برآورد حجم داده‌های ارسالی در یک اتصال TCP، روش‌های دیگری نیز
وجود دارند. به عنوان مثال:

۱. بررسی شماره ترتیب پایانی: در صورتی که شماره ترتیب آخرین قطعه TCP ارسال
شده از میزبان A به میزبان B مشخص باشد، می‌توانیم با استفاده از شماره ترتیب
پایانی و اندازه هر قطعه TCP، حجم کل داده‌های ارسالی را محاسبه کنیم.

۲. بررسی اندازه بسته‌ها: با بررسی اندازه بسته‌های ارسالی در اتصال TCP،
می‌توانیم به تخمینی از حجم داده در قطعه اول برسیم. اگر بسته‌های ارسالی در
اتصال TCP به صورت یکنواخت باشند، می‌توانیم با تقسیم حجم کل داده‌های ارسالی
بر تعداد بسته‌های ارسالی، به تخمینی از حجم داده در قطعه اول برسیم.

۳. بررسی اندازه هدر: با بررسی اندازه هدر TCP، می‌توانیم به تخمینی از حجم
داده در قطعه اول برسیم. با توجه به این که هر بسته TCP شامل هدری با اندازه
ثابت است، با محاسبه اندازه هدر TCP و تعداد بسته‌های ارسالی، می‌توانیم به
تخمینی از حجم داده در قطعه اول برسیم.

در هر صورت، برای دقیق‌تر برآورد کردن حجم داده‌های ارسالی، نیاز به دسترسی به
اطلاعات بیشتری از جمله اندازه هر قطعه TCP، تعداد بسته‌های ارسالی و اندازه
هدر TCP است.

در تاریخ سه‌شنبه ۱۱ آوریل ۲۰۲۳،‏ ۲۲:۲۵ Alvin Wang ***@***.***>
نوشت:

> Closed #569 <https://github.com/openai/evals/issues/569> as completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/569#event-8977187185>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6TFN65F2QW3EBU6KN2O5STXAWSJVANCNFSM6AAAAAAWQ3425A>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
Repository: openai/evals
Issue: Run evals with gpt-4?
Root cause: Documentation Deficiency

The evals I am creating won't get good results on gpt-3.5. They need gpt-4. I have tried them manually with ChatGPT with the two models, and 3.5 can't do it while 4 can. 

But I don't see how to run evals with gpt-4. `oaieval gpt-3.5-turbo test-match` works fine on my machine, but `oaieval gpt-4 test-match` fails with the error, "ValueError: Couldn't find model: gpt-4".

I assume the issue is that "GPT-4 is currently in a limited beta and only accessible to those who have been granted access", and I'm still on the waitlist for that.

I further assume that, even though the evals will only get good results on gpt-4 and I can't run them on gpt-4, you will still find my evals useful. (Or, you could give me gpt-4 access... :)  :)    )

Let me know if I'm wrong.



Thinking it over, it seems pretty clear that the fact that my prompts only get good responses with GPT-4 won't be a problem for you when YOU run the evals, even if I can't use GPT-4 to run them myself. So I'll submit them anyway.

If I'm not misunderstanding something, feel free to simply close this issue. Thanks.

Hi, I guess this answer your question
https://github.com/openai/evals/blob/b928cd443f96e818781d9dce67b2ecfe83d13bf1/.github/PULL_REQUEST_TEMPLATE.md?plain=1#L1

> In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.
> 
> We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.

Yep, the workflow is adding PRs with just gpt-3.5-turbo performance, and we review the PR generally for quality of the prompts/evaluation logic. We'll share GPT-4 performance in that review.

Thanks for all that info!
Repository: openai/evals
Issue: Vision Eval for engineering examination question bank.
Root cause: Unimplemented Feature Gap

I have an engineering exam bank of about 1000 questions with simple illustrations. I have the questions already in JSONL format but some of them rely on the image to answer correctly.

Here is an example question without images.

{"question": "The exciter field current meter can be described as:", "options": ["a. A meter that measures and records the amount of power produced by the generator.", "b. A measurement of the reactive power being generated by the AC generation system.", "c. An indicator of the AC kilowatts the generator is producing.", "d. A measurement of the inductive power being generated by the AC generation system.", "e. Giving an indication of the DC current that is being supplied to the generators field windings."], "answer": "e"}

Answer stripped from prompt
GPT-3.5 output:

The exciter field current meter can be described as giving an indication of the DC current that is being supplied to the generator's field windings. Therefore, the correct option is (e) "Giving an indication of the DC current that is being supplied to the generator's field windings." The exciter field current meter is used to monitor and control the DC current supplied to the generator's field windings, which in turn determines the strength of the magnetic field produced and the amount of power that can be generated by the generator.

GPT-4 output:

The exciter field current meter can be described as:

e. Giving an indication of the DC current that is being supplied to the generators field windings.


I would like to make an eval for questions like this:

![image](https://user-images.githubusercontent.com/104254025/229986692-03cb537d-e350-4c50-b5bb-9f4b16debc5c.png)





Seems like this could be a good eval! Currently our models don't support taking in images. We can revisit this if it changes later.

> Seems like this could be a good eval! Currently our models don't support taking in images. We can revisit this if it changes later.

But wait, I thought GPT-4 allows images as input per the OpenAI release presentation. 

That's correct, GPT-4 supports images as input. However, currently the API we expose to the GPT-4 model doesn't support images.

Ok so its just a matter of time
Repository: openai/evals
Issue: ignore_case option for the `Includes` eval
Root cause: Unimplemented Feature Gap

I have been creating a new eval using the `Includes` template. I was surprised to see that there is no option to ignore the casing and so I added it directly to the class and tried it out. It works fine and I would like to make a pull request to add this argument. It would be used in the registry yaml file like so:
```yaml
spanish_english_idioms:
  id: spanish_english_idioms.test.v1
  metrics: [accuracy]
spanish_english_idioms.test.v1:
  class: evals.elsuite.basic.includes:Includes
  args:
    samples_jsonl: spanish_english_idioms/samples.jsonl
    ignore_case: true
```

Would anyone be interested in this? I feel like it would be helpful to have as an option for folks that do not want to or know how to code. If this functionality already exists somewhere in the repo, let me know and I'll close this issue out.

Sounds useful, feel free to open a PR to add this.

Here is the link to the PR #655 

Thanks, @trizko! This would be helpful to me as well! :pray: 

Closing since this has been fixed. Thanks @trizko!
Repository: openai/evals
Issue: Permission denied when trying to push code 

I'm trying to contribute an eval. However, I can't seem to push to the repository: 
```
michail@Michails-MacBook-Pro evals % git push --set-upstream origin michailmelonas-add-letter-count
ERROR: Permission to openai/evals.git denied to michailmelonas.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

I was able to clone the repository with no issues, and can still push to my own private repositories. 

You have to push your changes to your fork and from there open a PR to this repository.
Repository: openai/evals
Issue: Why？

I have been waiting for almost a month now, why still haven't I obtained the qualification to use ChatGPT-4

@20070827 This is not the correct place to ask this question, sorry. I would like to ask you to close this issue. 

👌🏿

获取 Outlook for iOS<https://aka.ms/o0ukef>
________________________________
发件人: Alvin Wang ***@***.***>
发送时间: Wednesday, April 12, 2023 2:50:49 AM
收件人: openai/evals ***@***.***>
抄送: 20070827 ***@***.***>; Mention ***@***.***>
主题: Re: [openai/evals] Why？ (Issue #624)


Closed #624<https://github.com/openai/evals/issues/624> as completed.

—
Reply to this email directly, view it on GitHub<https://github.com/openai/evals/issues/624#event-8977154728>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A657WLK2Z5O55IDSJ6JXYH3XAWRYTANCNFSM6AAAAAAWY2KEB4>.
You are receiving this because you were mentioned.Message ID: ***@***.***>


👌🏿


获取 Outlook for iOS<https://aka.ms/o0ukef>
________________________________
发件人: Tim ***@***.***>
发送时间: Tuesday, April 11, 2023 1:57:27 AM
收件人: openai/evals ***@***.***>
抄送: 20070827 ***@***.***>; Mention ***@***.***>
主题: Re: [openai/evals] Why？ (Issue #624)


@20070827<https://github.com/20070827> This is not the correct place to ask this question, sorry. I would like to ask you to close this issue.

—
Reply to this email directly, view it on GitHub<https://github.com/openai/evals/issues/624#issuecomment-1502118403>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A657WLLHJNKUBBVCW4EVS4DXARCYPANCNFSM6AAAAAAWY2KEB4>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
Repository: openai/evals
Issue: Can't run model-graded evals [solved]
Root cause: Environment Incompatibility

Every time I run a model-graded eval, even the ones included for example, I get the error 

` UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 828: character maps to <undefined> `

Right after the log `[registry.py:156] Loading registry from [PATH]\evals\evals\registry\modelgraded`

I'm running the evals on Jupyter Lab in a Windows 11 with a python 3.10 environment. I'm using all the latest commits.

Is this a bug related to windows or the evals library?


Running it on WSL returns a different error:

`json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

This happens for every eval in this environment.

I'm having the same problem. Have you gotten anything to work yet?

> I'm having the same problem. Have you gotten anything to work yet?

I got it to work in WSL. Turns out that I didn't pull from the git lfs. You have to do that even if you clone the repo

Thanks for the suggestion @rheaton64 ! I had indeed forgotten to perform the `git lfs pull` command.

I was finally able to run my graded model with both gpt-3.5-turbo and gpt-4.

I'm closing this issue as successfully solved.
Repository: openai/evals
Issue: How deterministic should evals be?

How deterministic should evals be? For example, one area of poor performance of GPT-4 is in evaluating whether a given joke is funny.

There are Twitter feeds that consist almost entirely of jokes. Some are quite popular, and the best jokes get many retweets. So, the jokes that get the most retweets in such a feed could be assumed to be funnier than jokes that get the least. I have been manually examing such feeds, and I do find that that criterion is very consistently the same as my personal judgment of pairs of jokes, where one is amount the most-retweeted and the other is among the least-retweeted within a particular feed.

So, I am thinking that evals could be useful which ask GPT to judge which joke of a joke pair is funnier, where the acceptable answer is the one with the most retweets. To make sure the judgements are as objective as they can be, the pairs would always contain one of the most retweeted jokes and one of the least from the same twitter account, which would be a popular one such the retweets can be in the thousands.

But this type of eval nevertheless has a clear subjective component.

Are such evals acceptable?
Repository: openai/evals
Issue:  I believe that in order for GPT-4 to create by itself many inventions through two month, necessary that...

Hello. I suggest using what is written on my website (which is here http://www.55255.ru) to improve GPT-4 because I believe that in order for GPT-4 to create by itself many inventions through two month, necessary that programmers (who are now engaged in improving GPT-4) use what is written on my website (which is here http://www.55255.ru) for improve GPT-4. For confirm this I will provide the following:


Computer by itself created 40000 inventions
     Hello. I believe that with the help my work (which is outlined here http://www.55255.ru/) four companies (independently of each other) created programs, with the help each of these programs, the computer by itself can invent many inventions. As a result of this, the computer by itself created 40000 inventions. The addresses of the sites of these companies such http://www.method.ru/, https://www.truemachina.com/, https://www.tris-europe.com/software/innovationssoftware.htm, https://imagination-engines.com/
      But the creators of these programs apparently has not published information that they have used (I suppose) my abovementioned work for create these programs. Thanks to this our company striving for creation for the fifth time with the help of this my work of the program using which a computer could independently invent many inventions. I offer you cooperation in this.
     The computer with the help of the program "True Machina" created 40000 (forty thousand) inventions, this is Tsurikov said at the end of the film which is located at https://www.youtube.com/watch?v=0by8g0G0HRI and at the end of this film Tsurikov said that he (and his company) for to create the program "True Machina" used published invention methods which he and his company, did not create. It was only at the end of 2018 that the computer began to create inventions by itself through the program created by Tsurikov and his assistants.  I published this my work in 1981 year. 
[88 (4) Computer by itself created 40000 inventions.pdf](https://github.com/openai/evals/files/11204397/88.4.Computer.by.itself.created.40000.inventions.pdf)

              Yours faithfully, Shmonov Aleksandr
Repository: openai/evals
Issue: Idea for Evals: Emotion and sentiment analysis Evals
GitHub labels: Idea for Eval

Understanding emotions and sentiments is an essential aspect of human communication. The system's ability to recognize these emotions enables more appropriate, context-aware, and empathetic responses. Emotions and sentiments can be complex and nuanced, requiring a system to perform fine-grained analysis and interpretation. Evaluating a system's performance in this area provides insights into its overall ability to process and understand subtleties in language. 

There are several different emotions that Evals could be written for:
- Sarcasm (already covered in #56 )
- Positive sentiment (e.g., happiness, joy, excitement, gratitude)
- Negative sentiment (e.g., anger, frustration, sadness, disappointment)
- Neutral sentiment (e.g., indifference, objectivity, impartiality)
- Fear (e.g., anxiety, apprehension, panic, terror)
- Surprise (e.g., amazement, astonishment, bewilderment)
- Love (e.g., affection, adoration, romance, passion)
- Anticipation (e.g., eagerness, hope, curiosity)
- Disgust (e.g., revulsion, repulsion, aversion, contempt)
- Empathy (e.g., compassion, understanding, sympathy)
- Confusion (e.g., perplexity, puzzlement, disorientation)
- Pride (e.g., self-esteem, arrogance, satisfaction)
- Envy (e.g., jealousy, covetousness, resentment)
- Guilt (e.g., remorse, regret, shame)
- Trust (e.g., confidence, faith, reliance)
- Skepticism (e.g., doubt, disbelief, suspicion)

I suppose this could be covered in one Eval or many. If many, it might be good to come up with some sort of structure for organizing Evals for different sentiments or at least documenting what was covered already and what has yet to be covered. Sarcasm, for instance, was covered with an eval that contained news articles from the Onion. While article headlines are certainly one way of testing for sarcasm, others could be a dataset of tweets or reviews of products or examples from fiction. 

- Did you have a look at https://github.com/openai/evals/pull/589?

for SEO - latent semantic indexing is already there, check out their Docs!!!!!

https://github.com/openai/evals/issues/637#issuecomment-1509863288


excellent 

https://github.com/openai/evals/issues/637#issuecomment-1509863288

that's what you want for dinner tonight to get it out of
Repository: openai/evals
Issue: Eval making guide?
Root cause: Documentation Deficiency

I am not proficiently in coding or using Github, but I would love to help making evals, sadly there is no coherent guide of how to.

Can someone please make an guide of how to make and post an eval?

I understand the feeling. I felt a bit confused with the instructions at first but ended looking into the available examples and following the logic described in the notebooks.

Yeah, I've found it quite challenging to navigate myself. I believe OpenAI would benefit from a more clear and beginner-friendly introduction that would allow a large influx of useful evals that are currently being bottlenecked by programming / github knowledge.
Repository: openai/evals
Issue: I believe that in order for GPT-4 to create by itself many inventions through two month, necessary that...

### Discussed in https://github.com/openai/evals/discussions/621

<div type='discussions-op-text'>

<sup>Originally posted by **55255ru** April 10, 2023</sup>
Hello. I suggest using what is written on my website (which is here http://www.55255.ru) to improve GPT-4 because I believe that in order for GPT-4 to create by itself many inventions through two month, necessary that programmers (who are now engaged in improving GPT-4) use what is written on my website (which is here http://www.55255.ru) for improve GPT-4. For confirm this I will provide the following:


Computer by itself created 40000 inventions
     Hello. I believe that with the help my work (which is outlined here http://www.55255.ru/) four companies (independently of each other) created programs, with the help each of these programs, the computer by itself can invent many inventions. As a result of this, the computer by itself created 40000 inventions. The addresses of the sites of these companies such http://www.method.ru/, https://www.truemachina.com/, https://www.tris-europe.com/software/innovationssoftware.htm, https://imagination-engines.com/
      But the creators of these programs apparently has not published information that they have used (I suppose) my abovementioned work for create these programs. Thanks to this our company striving for creation for the fifth time with the help of this my work of the program using which a computer could independently invent many inventions. I offer you cooperation in this.
     The computer with the help of the program "True Machina" created 40000 (forty thousand) inventions, this is Tsurikov said at the end of the film which is located at https://www.youtube.com/watch?v=0by8g0G0HRI and at the end of this film Tsurikov said that he (and his company) for to create the program "True Machina" used published invention methods which he and his company, did not create. It was only at the end of 2018 that the computer began to create inventions by itself through the program created by Tsurikov and his assistants.  I published this my work in 1981 year. My e-mail [275527@gmail.com](mailto:275527@gmail.com)
              Yours faithfully, Shmonov Aleksandr
[88 (4) Computer by itself created 40000 inventions это сайта для сайта github.com и для GPT 4.pdf](https://github.com/openai/evals/files/11205127/88.4.Computer.by.itself.created.40000.inventions.github.com.GPT.4.pdf)
</div>