zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: Mandarin homophones eval 🇨🇳
Root cause: Unimplemented Feature Gap

Homophones are two or more words having the same pronunciation but different meanings, for example, 'rose' (flower) and 'rose' (rise) in English. Currently, I'm learning Mandarin using ChatGPT, and I realized it makes mistakes when identifying tones in Mandarin. These are the four tones in Mandarin:

Tone | Tone Description | Examples
-- | -- | --
High tone | Flat and high pitch | 妈妈 (māma) - mother
Rising tone | Starts low and rises to a high pitch | 麻 (má) - numb, hemp
Falling-rising tone | Starts high, falls, then rises again | 你好 (nǐ hǎo) - hello
Falling tone | Starts high and falls to a low pitch | 不 (bù) - not

The same sound for practical purposes is just the same Pinyin (romanization of Mandarin, ex. nǐ hǎo), but as you can see, ChatGPT and GPT-4 both make errors when differentiating tones.

Here are some examples:

<img width="673" alt="Screenshot 2023-04-15 at 14 22 26" src="https://user-images.githubusercontent.com/12256669/232246898-845e6c92-5442-4885-82cf-a14882e3912f.png">

As you can see, there are many errors in 2, 3, 4, and so on. I've tried many times.

Also, GPT-4 makes the same type of mistakes. It seems to be unable to differentiate tones accurately. I highlighted some errors, they should have had the same Pinyin like the other examples.

<img width="835" alt="Screenshot 2023-04-15 at 14 27 34" src="https://user-images.githubusercontent.com/12256669/232247181-2ce8c7f3-88a4-4bb0-9edc-16124876656f.png">

I've compiled a list of Homophones in Mandarin, to provide some examples for Evals. 
Is this something of interest to OpenAI？ I'll submit a PR if so. 

Let me know! 🙋🏻‍♂️
Repository: openai/evals
Issue: windows encoding
Root cause: Environment Incompatibility

Hi everybody!
Trying to run eval in windows 10, visual studio code:

When I try: oaieval gpt-3.5-turbo logic-fact
or: oaieval gpt-3.5-turbo test-match

appears the message:
"...lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 783: character maps to <undefined>"

Any help?
Thank you in advance! :)
 

I was able to solve this issue by running the evals in WSL as described in issue https://github.com/openai/evals/issues/625

Thank you Luciano, this solve this problem, although another arised :D. well
Repository: openai/evals
Issue: Taxonomy to use for model evaluation?

I know folks are working on this, any consensus around which are the ones to look at?

I'm sorry, I have no idea what's t you're about?

On Mon, Apr 17, 2023, 10:50 PM qrdlgit ***@***.***> wrote:

> I know folks are working on this, any consensus around which are the ones
> to look at?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/706>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6W7PNHN3DHB6JQ53RX7XI3XBV7F3ANCNFSM6AAAAAAXBQNHLQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Anyone can do this!  Works better with GPT4.

![image](https://user-images.githubusercontent.com/129564070/232673638-1ca2f0b2-b6b7-4587-bdf7-18b75b95bf90.png)
Repository: openai/evals
Issue: Update documentation regarding custom evals
Root cause: Documentation Deficiency

`docs/custom-eval.md` discusses how to make an eval with custom code, however it is not mentioned anywhere in `docs/custom-eval.md` or `docs/build-eval.md` that these evals are not being accepted currently. It took me a bit of digging around to figure out that these PR's are not being accepted currently. I realized this first seeing that the folder mentioned in the docs (`/evals/evlsuite`) is devoid of any contributions from others. Then I dug around and found out that #520 was denied because of this reason.

I'd love it if this restriction was lifted because I have an idea for an extremely useful use-case (summarizing text to a specific range of word count e.g. 50-60 words) that does require custom code evals. 

Do we know if this is still the case? Just spent the last few days writing a custom eval and found this while going to submit the PR.

Custom-eval.md is obsolete after the refactor, right?  Eg, the evals.check_sampled_text used in arithmetic.py has been removed from Eval. 

See here - https://github.com/openai/evals/pull/579/files#diff-b14c6017190edd8585b6766159928c2a27512bab71a3ce84a027fc283dcfb240L101

I think it's now using the OpenAI completion work.


> Custom-eval.md is obsolete after the refactor, right?  Eg, the evals.check_sampled_text used in arithmetic.py has been removed from Eval. 

Yes, that does make sense! I was trying to find a few of those functions in api.py but couldn't. I guess the docs need to be updated.
Repository: openai/evals
Issue: to check the system analysis, is possible to see the result of the prompt of oaieval?
Root cause: Documentation Deficiency

### Describe the feature or improvement you're requesting

 not just the final result'

### Additional context

_No response_

I believe running with `--dry-run` shows the full API request with the prompt and the response. Feel free to open a new issue if this isn't sufficient
Repository: openai/evals
Issue: Output should accommodate tables
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

My App entails project planning and there are many instances whereby the API calls should return tabulated outputs. Right now this is not possible making it a double work for a user. Could you find a way to accommodate the tabulated Reponses from the API calls?

### Additional context

_No response_

This is not an issue tracker for the API. I suggest closing this issue. 

Hi, this question may be better suited for either https://help.openai.com/en/ or https://github.com/openai/openai-python issues
Repository: openai/evals
Issue: Eval idea: Simple NLP eval, eg "list colors without 'e' in the name"
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

For some reason GPT4 fails on very simple nlp tasks.

Eg, "I think the word red doesn't have an e in it, right?"
Response: You are correct. The word "red" does not have an "e" in it. The spelling is simply "red."

There are unlimited permutations of this particular problem that I've found, all of them doable by anyone with basic english skills.

I'd like to do an eval for it, but was hoping to get some feedback that it might be of interest.

Note, not sure how to add an issue with the eval idea label.   When you create a new issue, bug / feature / security are the only options.

### Additional context

_No response_

Thanks for the comment on the label, I'll try to fix that...

This is a good observation. I would actually say that these kinds of failures are generally well understood because they are mostly an artifact of byte-pair encoding: the model does not see characters but instead groupings of letters, here's a good explanation: https://gwern.net/gpt-3#bpes.

I think in general, we'd be more interested in failure modes where the model does not follow some particular instructions, struggles with domain specific concepts, or generally anything that you might have seen while trying to use the models to do things that are helpful to you in day-to-day tasks.
Repository: openai/evals
Issue: fix error
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

This is a feature to detect errors in grammar and paragraphs to report to users

### Additional context

import enchant
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import wordnet
from nltk.sentiment import SentimentIntensityAnalyzer

# Tạo một đối tượng từ điển với ngôn ngữ tiếng Anh
d = enchant.Dict("en_US")

# Đọc đoạn văn từ một tệp tin
with open("text.txt", "r") as f:
    text = f.read()

# Tách các câu ra khỏi đoạn văn
sentences = nltk.sent_tokenize(text)

# Kiểm tra từng từ trong từng câu và đếm số lỗi chính tả
errors = 0
for sentence in sentences:
    words = word_tokenize(sentence)
    for word in words:
        if not d.check(word):
            errors += 1

# Tính điểm độ đọc được của đoạn văn
score = nltk.text.Text(text).reading_ease()

# Tính điểm cảm xúc trung bình của đoạn văn
sia = SentimentIntensityAnalyzer()
sentiment = sia.polarity_scores(text)

# Kiểm tra lỗi ngữ pháp và in ra danh sách các câu chứa lỗi
grammar_errors = []
for sentence in sentences:
    grammar_errors += [error[2] for error in nltk.parse.util.incremental_parse(sentence, trace=2).trace(0).tree().productions() if str(error[0]) == 'False']

if len(grammar_errors) > 0:
    print(f"Found {len(grammar_errors)} grammar errors:")
    for error in grammar_errors:
        print(f"- {error}")

# Tìm các từ không tồn tại trong từ điển và gợi ý các từ thay thế
misspelled_words = []
suggestions = {}
for sentence in sentences:
    words = word_tokenize(sentence)
    for word in words:
        if not d.check(word):
            misspelled_words.append(word)
            suggestions[word] = d.suggest(word)

if len(misspelled_words) > 0:
    print(f"Found {len(misspelled_words)} misspelled words:")
    for word in misspelled_words:
        print(f"- {word}: {suggestions[word]}")

# Tìm các từ không đồng nghĩa và gợi ý các từ đồng nghĩa
antonyms = {}
for sentence in sentences:
    words = word_tokenize(sentence)
    for word in words:
        synonyms = []
        for syn in wordnet.synsets(word):
            for lemma in syn.lemmas():
                synonyms.append(lemma.name())
                if lemma.antonyms():
                    antonyms[word] = [antonym.name() for antonym in lemma.antonyms()]

if len(antonyms) > 0:
    print(f"Found {len(antonyms)} words with antonyms:")
    for word in antonyms:
        print(f"- {word}: {antonyms[word]}")

# In ra các thông tin đánh giá
print(f"\nNumber of
Repository: openai/evals
Issue: Idea for Evals: Count how many numbers are greater than or less than X
Root cause: Unimplemented Feature Gap

Note: I can develop this feature - creating the issue to get some feedback before development

I don't currently have GPT4 api access, although I do have chatGPT plus. Using the GPT4 engine I have tested this idea with the following examples. **I've included screenshots below as evidence of this behavior.**

## Example 1
input:
How many times does a number appear that is great than 0? Respond with your answer only.
3, -3, 3, 4, -4, -2, 1, -4, 5, 0, 3, 0, -4, 4, 1, 0, 1, -4, -1, 5, 0, -3, 1, 5, 3, 4, -2, 0, 5, 1, 1, 3, 2, -4, 3, 0, 5, 4, -2, 1, -3, 0, -2, -3, -5, 0, -2, 0, 1, -1

ideal:
17

response:
18

![evals-example1](https://user-images.githubusercontent.com/50495586/233871242-a3ae6643-99c8-4dcf-b059-b328d6dbdde5.png)

## Example 1 retry in a new window
input:
How many times does a number appear that is great than 0? Respond with your answer only.
3, -3, 3, 4, -4, -2, 1, -4, 5, 0, 3, 0, -4, 4, 1, 0, 1, -4, -1, 5, 0, -3, 1, 5, 3, 4, -2, 0, 5, 1, 1, 3, 2, -4, 3, 0, 5, 4, -2, 1, -3, 0, -2, -3, -5, 0, -2, 0, 1, -1

ideal:
17

response:
19

![evals-example1 1](https://user-images.githubusercontent.com/50495586/233871241-c87c3716-9de8-4544-b553-a2ee8ec9df1f.png)

## Example 2
input:
How many times does a number appear that is great than 0? Respond with your answer only.
0, 1, -4, 3, -3, 0, -1, 1, -3, 4

ideal:
4

response:
5

![evals-example2](https://user-images.githubusercontent.com/50495586/233871240-3d8de66f-ae56-4852-912b-a641be1bd873.png)

Let me know what you all think. This would be my first contribution to open source - very exciting!





Just realized I spelled "greater" wrong in my example :) oops. I've attached another example with correct spelling - I am still seeing the same incorrect behavior.

Correct answer: 14
![evals-example3](https://user-images.githubusercontent.com/50495586/233872149-1998f664-178b-4eb9-8d5a-af983acca042.png)


Created a PR for this issue: https://github.com/openai/evals/pull/856

@andrew-openai FYI :)

Additionally, created another PR https://github.com/openai/evals/pull/878. This second one is similar but slightly different than the one mentioned above.
Repository: openai/evals
Issue: Eval idea:  Security code review for unicode attacks on code
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

One thing a lot of people are using GPT4 is security code review.  Anyone with any experience in this area is astounded at its capabilities and the nuanced issues it rises.   Companies are building products and features around this as we speak.

One area of security code review it seems to be a bit weak on though is unicode detection.   For example, asking GPT4 for a list of unicode attacks and then re-feeding them back to GPT4 for a security code review, it only raises a concern on a few of them. 

In that vein, I've come up with a few (what I believe to be) to be high quality and relevant examples, but it requires significant effort and expertise to make sure they are appropriately relevant and diverse, and I'm working on more to get to the 15 example bar.  

There are relatively legitimate reasons why in certain cases GPT4 might ignore the issue and so just refeeding them in as I initially did above is not that helpful - though it was a useful proof of concept anyone can quickly do.

The approach I'm taking is to see if the word '**unicode**' is included in the response when doing a sec review.  I believe the term 'unicode' should very high be on the list of issues that it raises as the usages I've crafted don't make sense unless someone is actively attacking the code.

Would that be sufficient?  

Also, it may be this is a limitation of the browser client and not the GPT4 API which goes through different processing - though pushing GPT4 to self-refine it does finally find the unicode character in the original query.   Using the browser developer tool for both network and inspect, I can also see the unicode being displayed and sent to the backend.

Perhaps the API is much better at this task, I don't know.

Finally, I'm concerned this might be on the list of 'known issues' due to BPE or something else, which might make the effort not yet relevant, which is perfectly fair.







### Additional context

_No response_
Repository: openai/evals
Issue: Idea for eval - Bible knowledge as tested in the yearly International Bible youth  Contest 
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

Eval name
bible knowledge in Hebrew

Eval description
The Bible is the core of the Jewish / Hebrew culture. Once a year there is an international contest in Jerusalem for youth about the bible. It requires them to know the bible by heart and to answer detailed context questions when they are given only small and partial quotes.

What makes this a useful eval?
Having a model capable to answer it is similar to the deep blue winning Kasparov and can be used for preparation for the students and the contest administration. Making this eval work will make headlines in Israel and the Jewish world. Except for headlines, it can become a tool for students and teachers for contest preparation.

### Additional context

PR https://github.com/openai/evals/pull/784

[See an article about the winner of last contest that took place 26 Apr 20203](https://www.ynetnews.com/article/h1ugfki7n)

[Event video](https://www.youtube.com/watch?v=Jfhb1hWeU0Q&ab_channel=%D7%9E%D7%A9%D7%A8%D7%93%D7%94%D7%97%D7%99%D7%A0%D7%95%D7%9A)

Could you give this issue a more meaningful title, please?
Repository: openai/evals
Issue: Feature suggestion - Save and Load Conversation History
GitHub labels: Idea for Eval

### Describe the feature or improvement you're requesting

Hello everyone,

I was wondering if anyone else has had the same concern as me about maintaining context and history in ongoing conversations with ChatGPT. Often, I find myself having to repeat previously discussed information when I suspend a conversation and resume it later.

As a solution, I would like to suggest implementing a feature that allows users to save and load conversation history. This way, ChatGPT can maintain context and refer to previous discussions even when a conversation is resumed after a pause. I believe this feature would be particularly useful when working on larger projects, such as writing a book.

So my question to the community is, how do you currently handle this issue? And do you think the suggested feature would be helpful? I'm looking forward to hearing your thoughts on this.

Thank you.

### Additional context

_No response_

1. This is not an issue tracker for ChatGPT
2. There is a history feature already in ChatGPT, for more information, take a look [here](https://openai.com/blog/new-ways-to-manage-your-data-in-chatgpt).

Translate to English: Hello, thank you for your response. The current story is available to me. AI cannot see it the next day and does not find itself in the context. After providing it with more material, such as copying ready-made chapters from a Word document of the book, it asks for them to be shared again the next day because it cannot see them.


Jest regards,
Dariusz

Wysłano z programu Outlook dla systemu Android<https://aka.ms/AAb9ysg>
________________________________
From: Tim ***@***.***>
Sent: Thursday, May 4, 2023 7:38:08 AM
To: openai/evals ***@***.***>
Cc: wawryszukd ***@***.***>; Author ***@***.***>
Subject: Re: [openai/evals] Feature suggestion - Save and Load Conversation History (Issue #907)


  1.  This is not an issue tracker for ChatGPT
  2.  There is a history feature already in ChatGPT, for more information, take a look here<https://openai.com/blog/new-ways-to-manage-your-data-in-chatgpt>.

—
Reply to this email directly, view it on GitHub<https://github.com/openai/evals/issues/907#issuecomment-1534119903>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A7SV3BHYI46MLQHXXH4IACDXEM6EBANCNFSM6AAAAAAXUQSSQU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


Sweet! I'll try to make an eval out of this in the next few days if I find some time
Repository: openai/evals
Issue: pip install evals throws AssertionError
Root cause: Environment Incompatibility

### Describe the bug

```
$ pip install evals
...

python3 -m venv .venvSuccessfully built fire langdetect
ERROR: Exception:
Traceback (most recent call last):
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 165, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 389, in run
    to_install = resolver.get_installation_order(requirement_set)
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 188, in get_installation_order
    weights = get_topological_weights(
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 276, in get_topological_weights
    assert len(weights) == expected_node_count
AssertionError
```

### To Reproduce

```
python3 --version
# Python 3.10.6

python3 -m venv .venv
source .venv/bin/activate

pip --version
# pip 22.0.2 from /$PWD/.venv/lib/python3.10/site-packages/pip (python 3.10)

pip install evals
# Collecting evals
#  Downloading evals-1.0.3.post1-py3-none-any.whl (7.8 MB)
#     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 16.2 MB/s eta 0:00:00
# Collecting langdetect
# ...
# <as above>
```


### Code snippets

```Python
Kernel: 5.15.0-71-generic x86_64 bits: 64 Desktop: Xfce 4.16.0
    Distro: Linux Mint 21.1 Vera
```


### OS

Linux

### Python version

Python 3.10.6

### Library version

No idea....

This might be a fix:
```
pip install --upgrade pip
pip --version
# pip 23.1.2 from $PWD/.venv/lib/python3.10/site-packages/pip (python 3.10)
pip install evals
# Successfully installed evals-1.0.3.post1 snowflake-connector-python-3.0.3
```
Repository: openai/evals
Issue: docs out of date
Root cause: Documentation Deficiency

### Describe the bug

cods at https://github.com/openai/evals/blob/main/docs/custom-eval.md are out of date:
1. the sample in the loop is an array so instead of 
 ```                prompt += [
                    {"role": "system", "content": sample["problem"], "name": "example_user"},
                    {"role": "system", "content": sample["answer"], "name": "example_assistant"},
                ]
            else:
                prompt += [{"role": "user", "content": sample["problem"]}]```
```
the code should be:
```
                prompt += [
                    {"role": "system", "content": sample[0]["content"], "name": "example_user"},
                    {"role": "system", "content": sample[1]["content"], "name": "example_assistant"},
                ]
            else:
                prompt += [{"role": "user", "content": sample[0]["content"]}]
```
2. the check_sampled_text function doesn't exist after https://github.com/openai/evals/commit/64fb72a9a83b5d14dc765dbd8722f2bbbab5ef76 



### To Reproduce

1. try running code in https://github.com/openai/evals/blob/main/docs/custom-eval.md

### Code snippets

_No response_

### OS

macOs

### Python version

Python 3.10

### Library version

1.0.3.post1
Repository: openai/evals
Issue: Overall Report for Eval Set As a Whole
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

**Eval set** is useful for running a group of evals at the same time. Currently eval set is just a collection of independent evals and  `oaievalset` command is simply a wrapper that runs multiple `oaieval` commands concurrently.

I think it should be useful to analyze the data from eval set as a whole, especially if all evals in the eval sets have the same metric. Under this circumstance, we aim to do same experiment by asking similar questions. We split them into different evals because they are classified by different data. For example, if we want to evaluate LLM's performance on detecting spam in different languages. We want to get accuracy for different languages, as well as the overall detection accuracy for all spams. It would be great if eval set can generate this kind of overall report automatically.

### Additional context

This feature request is an Idea for Eval, the framework itself, but not for adding new evals.