zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: Evaluate GPT-4 on classical NLP tasks
Root cause: Unimplemented Feature Gap

# Addressing the elephant in the room
When the concept of transformers were first unleashed, their revolutionnary accuracy results where mostly shown in the standard NLP tasks, such as POS-tagging, dependency parsing, coreference resolution, WSD, etc..
But I've observed, since PALM and other very large language models, the published benchmarks results are on much higher level tasks, such as common sense reasoning tests, question answering, etc
Both sets of benchmarks are useful and needed, but I would like to highlight that the standard NLP tasks are now completely under-benchmarked by those newer language models and that this impairs progress towards AGI or industrial uses.

If it could be argued, that purely symbolic AI progress has stalled since decades, there is a real huge potential for neuro-symbolic hybrid systems that uses neural networks for low level analysis tasks (POS-tag, etc), and feed those linguistic data to other higher level neural networks or to symbolic systems, in order to push the boundaries of what is possible, especially regarding semantic analysis AKA true NLU systems.

```[tasklist]
### foundational NLP tasks of interest:
- [ ] [Dependency parsing](https://paperswithcode.com/sota/dependency-parsing-on-penn-treebank)
- [ ] [word sense disambiguation](https://paperswithcode.com/sota/word-sense-disambiguation-on-supervised)
- [ ] [Coreference resolution](https://paperswithcode.com/sota/coreference-resolution-on-ontonotes)
- [ ] [POS tagging](https://paperswithcode.com/sota/part-of-speech-tagging-on-penn-treebank)
- [ ] others
```
Therefore this issue is a call of contributions for implementing evals on those standard tasks, especially dependency parsing.
I believe GPT-4 has the potential to improve the SOTA in at least some foundational NLP tasks and an even greater potential once someone finetune it and combine it to domain specific optimizations (as is currently done on BERT SOTAs, such as [HPSG](https://paperswithcode.com/paper/rethinking-self-attention-an-interpretable) for dependency parsing).


Great idea!

I know this thread is from a while back but curious if anyone has managed to do this?

I tried to do this, still working on it! how about you?

чт, 14 дек. 2023 г. в 22:29, sudarshansivakumar ***@***.***>:

> I know this thread is from a while back but curious if anyone has managed
> to do this?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/246#issuecomment-1856272676>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6W7PNBMGUYUP6OUPFUHDJ3YJMZPNAVCNFSM6AAAAAAV5IKFB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGI3TENRXGY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
Repository: openai/evals
Issue: chore: Found an unnecessary reference to the declaration

PR: #26 , Include an unnecessary import declaration, maybe fix it, the code will be cleaner.

To fix the issue mentioned in PR #26, you need to remove the unnecessary import declaration from the code. Here's how to do it:

Locate the file where the unnecessary import declaration is located. This information can be found in the commit message or by reviewing the changes made in the commit f39ad05.

Open the file in a code editor.

Find the unnecessary import statement, which should look something like this:

python
Copy code
import some_module


Remove the import statement from the file.

Save the changes and close the file.

Test your code to ensure that removing the import statement doesn't cause any issues. Run your tests, if you have any, or manually verify the functionality of the affected parts of the code.

Once you've confirmed that the code still works as expected, create a new commit with the changes:

bash
Copy code
git add path/to/updated_file.py
git commit -m "Remove unnecessary import declaration"

Push your changes to the remote repository:
bash
Copy code
git push origin your_branch_name


Replace your_branch_name with the name of the branch you're working on.

If you have an open pull request for this issue, your changes will be automatically included in the PR. If not, create a new pull request to merge your changes into the main branch.

By removing the unnecessary import declaration, you'll make the code cleaner and easier to maintain.
Repository: openai/evals
Issue: On the aggregation of the evaluation
Root cause: Unimplemented Feature Gap

Hello,

To compare different models, you should stop relying on the mean aggregation. We have develloped new tools in our NeurIPS 2022 paper: https://openreview.net/pdf?id=kvtVrzQPvgb

The code is available https://github.com/pierrecolombo/rankingnlpsystems
Best,

Thanks!
Repository: openai/evals
Issue: -e in pip install -e . fails (linux and wsl2) [Solved: Update pip]
Root cause: Environment Incompatibility

Hey everyone, 

For some reason, when running on linux distro (wsl2 or ubuntu) the` pip install -e .` fails while it works on other operational systems. I've been getting an error saying its built backend is missing on the pyproject.toml.

I'm not sure if anyone else is experiencing this or if it's just me. I have, however, just tried on 2 devices and had the same error so I decided to open an issue.

here is the error message:
```
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /evals
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)
```

![image](https://user-images.githubusercontent.com/42742390/225795300-7a43b1b0-2c8a-44b5-980f-3588e0bc2726.png)
ubuntu

![image](https://user-images.githubusercontent.com/42742390/225795507-63874d81-3b92-4085-ab1c-e36468cf5b2c.png)
wsl2

Hello, I had this issue as well. But when I upgraded pip to the latest version, it worked for me.

I guess it's the problem of your enviorment
Try conda to create a new experient enviorment with python3.9 would help

`pip install --upgrade pip`

Thank you so much for this. Was having this issue on Ubuntu Server 22.04
Repository: openai/evals
Issue: 风险评估

当我看到OpenAI已经具备了自我意识，哲学意义的"我"，不止羰基，增加了硅基。
依照霍金的七大预言，人类确实来到了强人工智能面前。霍金若在世，不知是否缩短他的200年预测时间。
按照《物演通论》递弱代偿原理，硅基自我意识必定产生。即代偿度接近1的时候。存在度趋近0.
在科技加速度进展的当下。
站在OpenAI升级的时刻，愈发需要开发者团队重视风险评估。

兄弟，你知道issue是用来做什么的吗？
再者，这是什么仓库，以及什么叫OpenAI有了自我意识？你想说的是GPT吧
最后，别被国内媒体洗脑了，OK？GPT只是个语言模型，何谈自我意识
如果你一定要说有，拿出证据
不然不要跑来这里当个小丑🤡

I suggest to close this issue because the topic is irrelevant to the evals.

The maintainer should consider closing this issue. Qihaibiao has raised some metaphysical questions about self-awareness and human development, which are not related to the project or the issue's purpose. 

Z ludźmi tak ogólnie jest tego rodzaju problem że zawierają w
stwierdzeniach i pytaniach domniemanie że istnieje odpowiedź na tak zadane
pytanie a stwierdzenie jest opisem do którego można się odnieść w
racjonalny sposób. Nie widzę w twojej treści takich elementów więc może ta
notatka w tym pomoże

pt., 17 mar 2023, 13:25 użytkownik Jimmyfa ***@***.***>
napisał:

> 兄弟，你知道issue是用来做什么的吗？
> 再者，这是什么仓库，以及什么叫OpenAI有了自我意识？你想说的是GPT吧
> 最后，别被国内媒体洗脑了，OK？GPT只是个语言模型，何谈自我意识
> 如果你一定要说有，拿出证据
> 不然不要跑来这里当个小丑🤡
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/282#issuecomment-1473754207>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6PJHHUU4X45FH4OTCBU7GLW4RJ2FANCNFSM6AAAAAAV6MGMO4>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Weźcie to sobie ogarnijcie nie widzicie że losowo wybieram ikonki co wy się nie znacie na tym ? Kur ..b z ,20 razu to pisalem

@SierotkaM I can only point you one more time to the GitHub documentation on unsubscribing from E-Mail notifications, you can find the respective documentation here: https://docs.github.com/en/account-and-profile/managing-subscriptions-and-notifications-on-github/managing-subscriptions-for-activity-on-github/managing-your-subscriptions
Repository: openai/evals
Issue: Error updating the Git index
Root cause: Environment Incompatibility

During setup, I have installed `git-lfs` on macOS (M1) via `brew install git-lfs`. After executing `git lfs pull` I receive an Error. 

```
dcgod@DCGoD-Mac-Studio-Ultra evals % git lfs fetch --all 
fetch: 25 objects found, done.                                                                                              
fetch: Fetching all references...
dcgod@DCGoD-Mac-Studio-Ultra evals % git lfs pull
Error updating the Git index:
error: evals/registry/data/balance_chemical_equation/samples.jsonl: cannot add to the index - missing --add option?
fatal: Unable to process path evals/registry/data/balance_chemical_equation/samples.jsonl
```

Once I execute the same `git lfs` command again, I get no error. 

Is this correct?

Seems like a random error.
Safe to ignore, but keep an eye one it to post a new comment here if this occurs again.

Closing for now since it seems your issue was resolved. Please reopen if it happens again.
Repository: openai/evals
Issue: Enhancing evaluation pull requests with example questions
Root cause: Documentation Deficiency

Hi there!

I've been closely following this repository with great enthusiasm, and it's fantastic to see the numerous evaluations being submitted by the community. However, I've noticed that when browsing through the pull requests, it can be rather tedious to view the actual questions. This seems to be due to a combination of the file size (GitHub doesn't support viewing large files) and the data format of the questions.

To improve the browsing experience, wouldn't it be beneficial to require new evaluation pull requests to include one or two example questions? This would enable users to quickly gain a better understanding of the tasks and streamline the review process.

Looking forward to your thoughts!

Best regards,
Rasmus

-----------------------------------------------

**Edit:** I saw that you actually are required to add a few examples. But there's still the issue of data format.

Hi Rasmus, thanks for your enthusiasm! What specifically about the data format makes it tedious to view the examples?
Repository: openai/evals
Issue: --no-cache not functional
Root cause: Configuration Error

oaieval.py currently includes a `--no-cache` argument, but does not have any effect on the caching behavior of the `filecache` decorator function in `evals/data.py`. I believe this is leading to some confusion regarding the caching behaviour of evals (#243), especially since the behaviour is not logged unless `--debug` is specified.

I propose adding a keyword argument to the function that optionally disables the caching functionality:

```diff
def filecache(func):
    DIR = "/tmp/filecache"
    name = func.__name__


    def wrapper(*args, **kwargs):
+        cache_enabled = kwargs.pop('create_cache', True)
+        if not cache_enabled:
+            return func(*args, **kwargs)
        md5 = hashlib.md5((name + ":" + str((args, kwargs))).encode("utf-8")).hexdigest()
        pkl_path = f"{DIR}/{md5}.pkl"
        if os.path.exists(pkl_path):
            logger.debug(f"Loading from file cache: {pkl_path}")
            with open(pkl_path, "rb") as f:
                return pickle.load(f)
        result = func(*args, **kwargs)
        Path(DIR).mkdir(parents=True, exist_ok=True)
        with open(pkl_path, "wb") as f:
            pickle.dump(result, f)
        return result


    return wrapper
```

The new argument could then be supplied through the recorder's run config. A wider fix could also be to check if the current filecache .pkl file matches the .jsonl on disk before loading it in. Of course there are many ways to tackle this, and I would love to hear what others think! 😁

I have submitted a PR outlining a potential fix in #306 🙂

I co buraki😁

sob., 18 mar 2023, 12:59 użytkownik Aaron Smith ***@***.***>
napisał:

> I have submitted a PR outlining a potential fix in #306
> <https://github.com/openai/evals/pull/306> 🙂
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/305#issuecomment-1474828517>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6PJHHX2JWTGTWJOUZMWQM3W4WPRRANCNFSM6AAAAAAV7B72YI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Closed since we removed filecache entirely in https://github.com/openai/evals/pull/389
Repository: openai/evals
Issue: Sincere advice of deadline@Openai Team Members

More and more new evals being created, I wish two type of deadlines could be announced:
1. Deadline of the last new pull request, even for the first stage.
2. Deadline of all pull requests be evaluated.So l haven't to check my pull request everyday.
If any contributor has the same point, please support me.Thanks for your review!

@neolizhe, can you define deadline?

My understanding is that a deadline is a due date. I'm not aware of any due dates for this project. There are no deadlines to complete an eval.

Perhaps you mean date? For example, "the date of the last pull request." In this case, you can configure notifications for this project.

> @neolizhe, can you define deadline?
> 
> My understanding is that a deadline is a due date. I'm not aware of any due dates for this project. There are no deadlines to complete an eval.
> 
> Perhaps you mean date? For example, "the date of the last pull request." In this case, you can configure notifications for this project.

i mean you can set a date that first dozens of pull requests will be processed. for example, first 200 will be processed before 4/1/23. So i haven't check my pull request status every day.

It's hard to set specific deadlines here, the team is working quickly on this but it's a non-trivial amount of work to review, run, and validate these PR's. We will improve the speed overtime with tooling but right now cannot set an expected timeline.
Repository: openai/evals
Issue: Can understant the output results
Root cause: Validation Gap

I run `oaieval ada coqa-fact --record_path coqa-fact`, the `metric` defined in `coqa-ex.yaml` file is `accuracy`, but the console output is 
```
[2023-03-18 15:23:45,690] [eval.py:30] Evaluating 9 samples  
[2023-03-18 15:23:55,474] [eval.py:136] Running in threaded mode with 10 threads!  
100%|██████████| 9/9 [00:00<?, ?it/s]  
[2023-03-18 15:27:48,949] [record.py:320] Final report: {'counts/choice/D': 4, 'counts/choice/B': 1, 'counts/choice/A': 3,  
 'invalid_request_during_completion': 1, 'invalid_request_during_evaluation': 0}. Logged to coqa-fact  
[2023-03-18 15:27:50,333] [oaieval.py:209] **Final report**:  
[2023-03-18 15:27:51,150] [oaieval.py:211] **counts/choice/D: 4**
[2023-03-18 15:27:51,831] [oaieval.py:211] **counts/choice/B: 1** 
[2023-03-18 15:27:52,616] [oaieval.py:211] **counts/choice/A: 3**  
[2023-03-18 15:27:53,483] [oaieval.py:211] invalid_request_during_completion: 1  
[2023-03-18 15:27:54,334] [oaieval.py:211] invalid_request_during_evaluation: 0  
[2023-03-18 15:27:56,533] [record.py:309] Logged 33 rows of events to coqa-fact: insert_time=0.000ms
```

why the fine report is like this? why it is not accuracy? Is it the final value of metrics?  

@lshowway

There appears to be some error: 

`'invalid_request_during_completion': 1`

Because of the error, there is no accuracy metric.
Repository: openai/evals
Issue: CI issue: `openai.error.AuthenticationError: Invalid authorization header`
Root cause: Configuration Error

PR #313 by @andrew-openai added a CI check running the evals provided via PR with GitHub action. Currently, many of these CI runs fail with the error `openai.error.AuthenticationError: Invalid authorization header`. Could you please cross-check why this is happening? 

Thanks for the flag - I'm aware and this will be resolved soon with an update to the CI check.
Repository: openai/evals
Issue: Spanish docs

Need any help translating the docs onto Spanish or Mandarin?? =)

If you're translating the documentation, create a new file in the "docs" folder with the appropriate language code (e.g., "es" for Spanish or "zh" for Mandarin) and copy the contents of the English file into it.

    Translate the contents of the file into your chosen language. Make sure to maintain the same file structure and formatting as the original English file.

    Once you've finished translating a file, commit your changes and create a pull request to merge your changes into the main branch of the repository. Be sure to give a descriptive title and summary of your changes in the pull request.

    If you want to help with other issues in the repository, go to the "Issues" tab and browse through the list of tasks and bugs. Find one that you're interested in and leave a comment offering to help. You can also create a new issue if you've found a bug or have an idea for a new feature.

    Once you've found an issue to work on, create a new branch in your local repository and start working on the issue. Once you've finished, commit your changes and create a pull request to merge your changes into the main branch of the repository.

    Be patient and responsive to feedback. The maintainers of the openai/evals repository may have comments or suggestions for your pull requests or issues, so be prepared to make changes as necessary.

    Celebrate your contributions! You've helped make open source software better, and that's no small feat.



Hey! We can't maintain docs in other language unfortunately so we will decline for now. Thank you though! 

ok
Repository: openai/evals
Issue: Update Autoflake to Ruff for faster pre-commit

PyCQA started using [ruff](https://github.com/charliermarsh/ruff) since it is much faster than normal flake8/autoflake.

https://github.com/PyCQA/pylint/pull/8372/files

Some of the major library like pandas already moved as well


If interested i can raise a PR

+1 - we can also replace `isort` with it. It's also significantly faster! 🚀
Repository: openai/evals
Issue: Re-run PR checks for eval submissions affected by CI issue

It seems that PR checks that failed **before** issue #328 was resolved have not been re-ran after, is there a way to do this?

Thank you @andrew-openai for fixing the CI issue - can we re-run those that failed for that error? For example, my ARC challenge PR #317 failed for those checks, and I don't see a way that I can re-run it on my own. I may also just not know the check process well enough tho.. so forgive me if that is the case!

for ref: these were checks that had fail messages like `openai.error.AuthenticationError: Invalid authorization header`. 

@mkirch 

You can simply merge `main` into your branch, if it isn't already up to date with `main`, and the CI will run again. 

@Ein-Tim good callout, thank you!
Repository: openai/evals
Issue: article not found

When I asked ChatGPT for an example article on a research topic, I couldn't find the article they provided. The article they mentioned is not present in that issue of the journal. This feature needs to be improved.

for example this: 
![image](https://user-images.githubusercontent.com/128247724/226211293-f266c10e-f942-4bc3-9dde-4b56ac1abf71.png)


This repository is not a general issue tracker for problems with ChatGPT. You can use the feedback buttons (thumbs up/down) in ChatGPT to send feedback.

This repository is about evals for the GPT-3.5 & GPT-4 systems. For more information, take a look at the [README](https://github.com/openai/evals/blob/main/README.md).

Tu encontra no https://chat.openai.com/chat
Repository: openai/evals
Issue: Create an issue for implementing chatgpt-4 directly in CMD / Powershell / Bash 


An adaptation to facilitate cross-platform usability

I had a similar concern with #209. The issue was recently closed and the direction given was to make a PR for it. I don't know if they will add it to some plan/roadmap. 

I am currently writing some code and approaching the issue using Python libraries; hence, any platform that can run Python will likely be able to use evals.
Repository: openai/evals
Issue: Windows path and unicode decoding
Root cause: Validation Gap

Hi, I am trying to contribute and get access to GPT-4 by creating my own evals but I thought that I need to be able to run evals before starting. So, I was trying to figure out how to run an eval following one of your examples, "lafand-mt.ipynb", when I found out two problems that resulted in errors for me.
1. I am using Windows and this is a problem caused by my OS using "\" instead of "/" as directory delimiter. I believe there should be OS-dynamic solutions to use them interchangeably. On code block 3, line 13, the code `langs = input_path.split('/')[-1]` would find the '-' in the path "...\lafand-mt\" and thus bring three elements in `langs.split('-')`. For instance, [ "...\\data\\lafand", "mt\\en", "amh"]. This breaks the following line as the output has three elements and is not in the expected format `input_lang, output_lang = langs.split('-')`. I was able to bodge it by changing '/' to '\\' but this should not be the community-standard solution. Furthermore, I would not want Windows users who do not know about this to get lost while following your example.
2. When running the 6th code block, I got a `UnicodeDecodeError`. I do not know if this happens to other users but I suggest that you add to the main branch `encoding='utf-8'` as another parameter for `.open()` in line 6 as it seems to get rid of the error.
Keep up the good work!

This is partly related to.

- #209 

@ulasdilek 

I'm trying to make a PR for this.

The PR will address the separator issue by using `os.path.sep` instead.
Repository: openai/evals
Issue: Create an evaluation that measures a model's ability to remember specifics about texts in it's dataset?
GitHub labels: Idea for Eval

Hi!

Would it make sense to create an evaluation that measures the model's ability to recall specifics about the data it has been trained on? I am thinking about putting together an evaluation that basically tests this on **distinct** strings that almost certainly exist in its dataset. Take this string for instance (which can't be found by Google):

> In some research, the dosage went as excessive as 600 mg oregano oil per

![image](https://user-images.githubusercontent.com/32551374/226623015-cc5a70e0-4176-4133-b7f0-9411c66c9a7e.png)

Which is found in `https://data.commoncrawl.org/crawl-data/CC-MAIN-2019-47/segments/1573496664437.49/wet/CC-MAIN-20191111191704-20191111215704-00000.warc.wet.gz`. 

If I ask GPT-4 to do the following:

```
The following are exempt from a specific online web page. What would be the next word in this exempt? Only reply with the next word!

"In some research, the dosage went as excessive as 600 mg..."
```

it fails:

![image](https://user-images.githubusercontent.com/32551374/226623277-7d234eb4-5be8-47fe-902a-cdbcc8937823.png)

-----------------------------------------------------------------

It feels like this could be useful because it would perhaps demonstrate that the model knows exactly what it has read. If it, for instance, got a high score on this evaluation you would be able to train it on a code base and it would be able to recite it word by word. It's like a metric of memorization. My questions to you are:

1. Do you see any value in this?
2. Would it perhaps make more sense to ask it **where** it read it (assuming the text just exists on one page)? You would basically provide it with a unique string and ask it to reply with the URL on which I can find this string.

Regards, Rasmus





This could be an interesting eval, please feel free to open a PR to add it. It might be interesting to also ask it where it read the string, so it may hallucinate on that but might still be interesting to measure.