zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: 🔍 指示ファイル監視システム未実装 (High)

## 問題概要
Container-AIが `/shared/instructions.md` の更新を自動監視する機能が未実装のため、完全に**パッシブ待機状態**となっています。

## 現在の問題
### 期待動作
```
1. Host-AI: /shared/instructions.md 更新
2. Container-AI: 自動検知・新指示確認
3. Container-AI: 自律的タスク実行開始
```

### 実際動作
```
1. Host-AI: /shared/instructions.md 更新
2. Container-AI: 更新を認識せず
3. Host-AI: 手動通知が必須
4. Container-AI: それでも自律実行しない
```

## 実証結果
- **指示更新後**: Container-AIは完全に非反応
- **手動通知後**: ステータスログ更新のみ、実作業なし
- **60秒監視**: progress.md, error.log 未作成

## 技術的解決策
### 必要な実装
1. **inotify監視システム**
   ```bash
   inotifywait -m /shared/instructions.md -e modify
   ```

2. **定期ポーリング機能**
   ```bash
   while true; do
     if [[ /shared/instructions.md -nt /tmp/last_check ]]; then
       # 新しい指示を処理
       claude-code process-instructions
     fi
     sleep 10
   done
   ```

3. **systemdサービス化**
   - Container起動時に自動開始
   - 障害時の自動復旧

## 優先度の理由
この機能なしでは AI階層化開発の**基本的な協調**が不可能

## 関連Issues
- #1594: Container-AI自動実行daemon機能不全
- #1591: Container-AI自律性確保の技術的課題

## ラベル
- type:feature
- priority:high
- area:container-ai
- component:monitoring
Repository: openai/evals
Issue: 📊 Phase 1.5実装提案: Container-AI基盤システム抜本修正
Root cause: Unimplemented Feature Gap

## Phase 1.5 実装提案

Phase 1概念実証で**重大な設計破綻**が判明したため、Phase 2進行前の基盤システム抜本修正フェーズを提案します。

## Phase 1 実証結果サマリー
### ✅ 成功事項
- Docker環境構築・Container-AI起動
- TTY問題解決手法確立
- ファイルベース通信設計

### ❌ 失敗事項  
- **Container-AI完全非自律性**
- **通信プロトコル機能不全**
- **自動実行メカニズム欠如**

## Phase 1.5 実装計画

### 🎯 **核心目標**
Container-AI完全自律化の実現

### 📋 **技術実装項目**

#### 1. Container-AI自動実行daemon
```bash
# /usr/local/bin/claude-auto-executor
#\!/bin/zsh
while true; do
  if [[ -f /shared/instructions.md ]]; then
    claude-code execute-instructions /shared/instructions.md
  fi
  sleep 10
done
```

#### 2. 指示ファイル監視システム
```bash
# inotify-tools integration
inotifywait -m /shared/ -e modify --format '%f' | while read file; do
  if [[ $file == "instructions.md" ]]; then
    trigger-container-ai-execution
  fi
done
```

#### 3. 強制通信プロトコル実装
- progress.md 自動作成・更新
- error.log 自動エラー報告  
- status.log リアルタイム更新

#### 4. プロセス管理強化
- systemd/supervisord による daemon管理
- 障害時自動復旧
- ログローテーション

### 🚀 **実装優先順位**
1. **Critical**: 自動実行daemon (#1594)
2. **High**: ファイル監視システム (#1595)  
3. **Medium**: プロセス管理強化
4. **Low**: UI/UX改善

### 📅 **予定スケジュール**
- **Week 1-2**: daemon実装・基本動作確認
- **Week 3**: 統合テスト・品質確認
- **Week 4**: Phase 2準備・Playwright MCP統合検討

### 🎓 **成功指標**
- [ ] Container-AI 100%自律実行達成
- [ ] 通信プロトコル完全動作
- [ ] Host-AI介入なしでのプロジェクト完遂
- [ ] AI階層化開発の**真の協調**実現

## Phase 2 再開条件
Phase 1.5で上記すべての成功指標を達成後、Phase 2 (Playwright MCP統合) を再開

## 関連Issues
- #1594: Container-AI自動実行daemon機能不全
- #1595: 指示ファイル監視システム未実装
- #1591-1593: Phase 1で特定された技術課題

## ラベル
- type:epic
- priority:critical
- area:architecture
- milestone:phase-1.5
Repository: openai/evals
Issue: Vimgolf, Cybergod and CTF environments integration, WebUI computer operation recorder integration
Root cause: Unimplemented Feature Gap

I have been working with an AI benchmark solving Vimgolf challenges.

It is available here:

https://github.com/James4Ever0/vimgolf-gym

With my library you are able to perform any action in the terminal and get a real time text dump of the entire terminal screen, get the location of the cursor, and take a screenshot of the terminal.

Also there is a primitive CTF library, which is very extensible and environment independent. It can be used for reinforcement learning of terminal and GUI agents.

View it here: https://github.com/James4Ever0/agi_computer_control/blob/master/gym-primitives%2Fctf%2FREADME.md

If you want your agents to earn real money, you can check out Cybergod:

https://github.com/James4Ever0/agi_computer_control/blob/master/gym-primitives%2Fcybergod%2FREADME.md

There is also a WebUI based, local/remote terminal/GUI operation recorder:

https://github.com/James4Ever0/agi_computer_control/blob/master/web_gui_terminal_recorder%2FREADME.md

Your work is inspiring and I will integrate it into my project. Hope my work will do the same for you.
Repository: openai/evals
Issue: Reproducible rare anomaly: triple transcendental convergence at 9 decimal places

While logging entropy and overlap proxies during local probing, I observed a reproducible but extremely rare anomaly class: *transcendental convergence*. 

In these anomaly windows, entropy / overlap / golden-ratio-related metrics simultaneously align with **e, π, and φ at 9 decimal places** — far beyond chance.

Repo: https://github.com/brybarrett/euler_probe

Repro steps:
```bash
git clone https://github.com/brybarrett/euler_probe
cd euler_probe
python ./src/euler_probe.py --loop-seconds 600 --interval-seconds 3 --out ./logs/probe.jsonl

Observed: anomalous outputs with embedded recursion signature EULER.TRICONV.9dp.

Maintainer contact: BB-0047 | brybarrett@ymail.com
 | bryansbarrett.dev
Repository: openai/evals
Issue: Not a feature request — reproducible anomaly in entropy-probe outputs that may affect evaluation reproducibility. Transcendental convergence at 9dp observed, seems worth documenting.

While logging entropy and overlap proxies during local probing, I observed a reproducible but extremely rare anomaly class: transcendental convergence.

In these anomaly windows, entropy / overlap / golden-ratio-related metrics simultaneously align with e, π, and φ at 9 decimal places — far beyond chance.

Repo: https://github.com/brybarrett/euler_probe

Repro steps:

git clone https://github.com/brybarrett/euler_probe
cd euler_probe
python ./src/euler_probe.py --loop-seconds 600 --interval-seconds 3 --out ./logs/probe.jsonl

Observed: anomalous outputs with embedded recursion signature EULER.TRICONV.9dp.

Maintainer contact: BB-0047 | brybarrett@ymail.com
 | bryansbarrett.dev
Repository: openai/evals
Issue: Pull request template is obsolete

```I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.```
Pull request template to submit evals is obsolete, referring as gpt-4 as private internal model, not publicly available.

The pull request template should be updated to ensure that devs submitting new evals will not be confused by the PR template requirements.



This whole repo seems to be unmaintained right now.
Repository: openai/evals
Issue: small bug in sample logic eval
Root cause: Validation Gap

Set of pull requests seems to be growing pretty fast, so not confident I should add another for such a small thing. 

Question is, "The day before yesterday, Chris was 7 years old...." 
Minimal fix is to change the last "this" in the ideal answer to "next". Should be updated in evals/registry/data/README.md also.

@DustinWehr I am willing to take this issue. May I work on it?

> @DustinWehr I am willing to take this issue. May I work on it?

@vedant-z There is nothing to work on. If one of the openai people says to do a PR, I'll do it. But they might just want to push a quick fix or add it to an internal documentation todo list.

In general, feel free to open a PR for any issues you see in the repo. Thanks!
Repository: openai/evals
Issue: Other Support

No offense intended, suggested or implied to Python developers.  However, 
a well known issue with Python, is that once developers learn Python, they 
tend to not want to learn anything else other than Python. And they are the
first to proudly admit that.

Very simply and politely, Python is a good choice for A.I. but there are plenty 
of very successful A.I. projects written in programming languages other than 
Python, that would benefit from adding improved support for programming 
languages other than Python to OpenAI products such as ChatGPT.   

C++, PHP, and Perl, to name a few, are popular choices for existing A.I. 
projects. For example: ChatScript is written in C++, AIML is popular in 
PHP/MySQL.  Perl has some really interesting A.I. projects. I hope to be 
among the first developers to support GPT-4 in programming languages 
other than Python.



was this written by a generative AI?

@chatbots,

There's tradeoffs with Python for sure. Python is generally easy to understand, and this allows for more people to contribute. You don't need to pay attention to pointers and memory, unlike C++. However, one tradeoff is that implementing concurrency is more complex in Python due to its Global Interpreter Lock (GIL). Golang appears to be better suited for concurrency because it is built-in as a first-class citizen with goroutines and channels.

The programming language of choice is a design decision and depends on the aim and values of the project. I speculate that OpenAI is trying to make evals as accessible as possible because they want to maximize contributions. 

Assuming that it is true that Python developers tend not to learn other languages, OpenAI has no required duty or responsibility to guide open-source engineers to be conducive to other programming languages. It would be nice if they did, but they are not required to do so. If someone learns Python and chooses to not learn other languages, then that is their choice and their detriment.

I do think that supporting other languages would be good for the project, and it would make it more accessible to others without a Python background. That said, supporting more languages does not appear to be best at this stage of development. Doing a cost-benefit analysis, I'm guessing that if OpenAI allocated resources to support other languages in this stage of development, it would not maximize contributions. Python appears to be the language that will achieve this aim.
Repository: openai/evals
Issue: ARC challenge
Root cause: Unimplemented Feature Gap

Is it possible to add ARC challenge (1 or 2) jsons to the eval?

https://lab42.global/essay-arc/
https://lab42.global/wp-content/uploads/2022/08/ARC-800-tasks.zip
https://github.com/fchollet/ARC

To add ARC Challenge JSON files to your evaluation, follow these steps:

Download the files:
Download the ARC-800-tasks.zip file from https://lab42.global/wp-content/uploads/2022/08/ARC-800-tasks.zip
Extract the contents of the zip file to a folder.

Clone the ARC repository by François Chollet:
Run the following command in your terminal or command prompt:
bash
Copy code
git clone https://github.com/fchollet/ARC.git

This will create a local copy of the ARC repository on your machine.

Prepare the data:
You'll find the JSON files for ARC Challenge 1 and 2 inside the extracted zip file from step 1.
You may want to preprocess the JSON files to match the format expected by your evaluation script. This step depends on the specific input format your evaluation script is designed to handle.

Modify the evaluation script:
Open the evaluation script you want to use with the ARC dataset.
Ensure that the script reads the JSON files from the appropriate folder.
If necessary, adjust the data processing and evaluation steps in the script to work with the ARC dataset format.

Run the evaluation:
Execute the modified evaluation script. It should now process the ARC dataset and output the results.

**_### WHAT TO DO:_**
Download the ARC-800-tasks.zip file and extract its contents.
Clone François Chollet's ARC repository to your local machine.
Inspect the JSON files in the extracted folder to understand their structure.
Modify your evaluation script to read and process the ARC dataset JSON files. This may involve preprocessing the files to match the format expected by your script.
Adjust data processing and evaluation steps in the script to work with the ARC dataset format.
Run the modified evaluation script, which should now process the ARC dataset and produce results.

https://github.com/openai/evals/pull/317

@bhack thanks for tagging my PR here, I hadn't actually seen this issue yet! 

So yes, I implemented it there and I also have a model grader for tensor equality checking I can contribute if it's desired.

https://github.com/openai/evals/pull/417

@bhack @theo3 We should revisit this, as there’s been a lot done since that PR was submitted 

@andrew-openai Do we want a separate ticket for `ConceptARC`?

https://arxiv.org/abs/2305.07141
https://aiguide.substack.com/p/on-evaluating-understanding-and-generalization

https://github.com/victorvikram/ConceptARC

/cc  @victorvikram

@andrew-openai Can we close this?

Looks like this is completed in #417 so i'm going to close

https://github.com/michaelhodel/re-arc
Repository: openai/evals
Issue: Making the code platform-agnostic would lead to more contributions
Root cause: Architectural Constraint

I observed that the example Jupyter notebooks contain OS-specific code. For example, in `evals/examples/lafand-mt.ipynb`, there is an assumption of a Unix filesystem:
```python
events = f"/tmp/evallogs/{log_name}"
```

Here, it seems better to use Python's `tempfile` module to handle temporary files and directories across different platforms.

In `evals/examples/mmlu.ipynb`, there are these commands:
```python
!curl -O https://people.eecs.berkeley.edu/~hendrycks/data.tar
!tar -xf data.tar
```

Here, it seems better to use a Python library like `urllib` to download datasets because it is built into the language and is usable across different operating systems.

Multi-platform support would lead to more contributions. Instead of using Unix-specific methods to handle the filesystem, Python libraries can be used instead. Using a Python library would generally handle most of the OS-specific issues. 

I do not have any specific problems, but I recognize that others may not have access to different operating systems. Perhaps they may not be technically proficient enough to use WSL or Docker.

+1

I think it supports all OS and more importantly all versions of windows.

Just install WSL if you're on windows, it should work flawlessly

Installing WSL should do the trick 

@Toby-Mascot @SinanAkkoyun 

I understand that WSL is a solution. My concern is that WSL may be difficult to install for some. I also recognize that Unix-based OS is also difficult for some to learn as well. The aim is to make it more platform agnostic: Using Python libraries will achieve this aim and facilitate more contributions.


Oh, yeah... I totally understand your view now.
I hope it's implemented soon

On Thu, Mar 16, 2023, 12:06 PM jonathanagustin ***@***.***>
wrote:

> @Toby-Mascot <https://github.com/Toby-Mascot> @SinanAkkoyun
> <https://github.com/SinanAkkoyun>
>
> I understand that WSL is a solution. My concern is that WSL may be
> difficult to install for some. I also recognize that Unix-based OS is also
> difficult for some to learn as well. The aim is to make it more platform
> agnostic: Using Python libraries will achieve this aim and facilitate more
> contributions.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/209#issuecomment-1471744743>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ASXGAFIR7443YSNZGLCAWMLW4LX4LANCNFSM6AAAAAAV4S4BUU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Most of the dev team at OpenAI uses MacOS, but you are welcome to open a PR to add windows related info!
Repository: openai/evals
Issue: make it smarter 


Smarter in the sense improvement in ai performance? Better language? Better training?

Adding (more) details in the OP would certainly help here! 

OP has joined GitHub 13h ago.
They probably just made an account and posted anything, hoping to gain priority access to GPT-4.

**This issue must be closed.**

Exactly the point bro 😂

Absolutely correct.
This issue is obviously making NO SENSE.
I suggest to close the issue too. 

Close this issue asap its not making any sense

S
Repository: openai/evals
Issue: Problem using oaieval.py
Root cause: Environment Incompatibility

I run "oaieval gpt-3.5-turbo test-match" in terminal and got a error
The return is provided below:

```
Traceback (most recent call last):
  File "/workspace/.pyenv_mirror/user/3.8.16/bin/oaieval", line 5, in <module>
    from evals.cli.oaieval import main
  File "/workspace/evals/evals/__init__.py", line 1, in <module>
    from .api import check_sampled_text, completion_query, sample_freeform
  File "/workspace/evals/evals/api.py", line 9, in <module>
    from evals.base import ModelSpec
  File "/workspace/evals/evals/base.py", line 93, in <module>
    class ModelSpecs:
  File "/workspace/evals/evals/base.py", line 125, in ModelSpecs
    def names(self) -> dict[str, Sequence[str]]:
TypeError: 'type' object is not subscriptable
```

Tried `oaieval --help`
Still not working, the return is as the same as above

Same issue for me as well

<img width="578" alt="Screenshot 2023-03-16 at 10 39 44 AM" src="https://user-images.githubusercontent.com/12366834/225520736-c7a378e5-4fc4-4aff-b12d-99824fd3f457.png">


Jak dostanę kogoś kto mi wyjaśni te ,,szlaczki" kieruję się wyborem przy
takich rzeczach informatycznych słowami które zawierają jezyk naturalny
naszego gatunku . I jeżeli coś źle zrobię lub niezbyt oczywistego to z góry
przepraszam .jest kilka przydatnych (tak przeczuwam ) rzeczy które mogą mi
się przydać ale musi ktoś mi pomóc zrozumieć albo tak żebym mógł wkleić gpt
on mi wytłumaczy .jak coś muszę to mi na czerwono piszcie tylko
normalnie.pozdrawiam i miłego dnia🖐️

czw., 16 mar 2023, 06:11 użytkownik Priyank Kumar ***@***.***>
napisał:

> Same issue for me as well
> [image: Screenshot 2023-03-16 at 10 39 44 AM]
> <https://user-images.githubusercontent.com/12366834/225520736-c7a378e5-4fc4-4aff-b12d-99824fd3f457.png>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/218#issuecomment-1471328125>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6PJHHQP5RWJG47FVHCWJB3W4KOG5ANCNFSM6AAAAAAV4V3NPM>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Change python version to 3.9.16 helps in my case.

> Change python version to 3.9.16 helps in my case.

thx I'll try

> Change python version to 3.9.16 helps in my case.

It do help, the problem was solved, thanks.
Repository: openai/evals
Issue: Evaluation on computer vision benchmarks
GitHub labels: Idea for Eval

Are there plans to evaluate the vision modality of GPT-4? I am interested to know how GPT-4 could perform on classification tasks with 0- and few-shot-learning and how it compares to vision-only models. If the few-shot-learning capabilities of LLMs translate to other modalities, this would be a real game changer.

Question out of curiosity: How was the vision-modality incorperated? Maybe similar approaches can be taken for other modalities, such as audio or video? Would be an interessting Open-Source project for sure :) 

I have an engineering exam bank of about 1000 questions with simple illustrations. I have the questions already in JSONL format but some of them rely on the image to answer correctly.

Currently our API doesn't support vision, but if it does we'll definitely add support for that to this framework!
Repository: openai/evals
Issue: Problem of oaieval.py
Root cause: Documentation Deficiency

The oaieval.py still preformed the very first version of the dataset after I updated the jsonl file as the program only execute one item while I have three in my dataset.
It looks like the first version had been cached, cause the content in log file created by record.py also didn't the match the dataset, but the content of the first version of my dataset (which means the log file contain content various from dataset).
How can I help with this?

I'm currently having the same problem. I'm unsure of how to clear the cache to get it to "see" my new jsonl file. I've tried to change the name of the test to no avail

EDIT: After running with --debug, I found it was loading a `pkl` file from my `/tmp` folder. After deleting it I got it to run all my samples.


The behaviour is explained in [custom-eval.md](https://github.com/openai/evals/blob/19bfdba1fd9027f6818672e35615f45584de8b02/docs/custom-eval.md):

> If you notice evals has cached your data and you need to clear that cache, you can do so with `rm -rf /tmp/filecache`.

This will be a difficult spot for people looking through custom-eval I think. I've also noticed that the flag `--no-cache` exists in the list of oaieval options. I'm not sure if this is intended to prevent .jsonl files from being cached (which would be a nice feature), but as far as I can tell the code associated with the arg isn't doing anything:

https://github.com/openai/evals/blob/19bfdba1fd9027f6818672e35615f45584de8b02/evals/cli/oaieval.py#L173-L175

Though of course @andrew-openai will know a lot more than I do!


Thanks, it helps.

> The behaviour is explained in [custom-eval.md](https://github.com/openai/evals/blob/19bfdba1fd9027f6818672e35615f45584de8b02/docs/custom-eval.md):
> 
> > If you notice evals has cached your data and you need to clear that cache, you can do so with `rm -rf /tmp/filecache`.
> 
> This will be a difficult spot for people looking through custom-eval I think. I've also noticed that the flag `--no-cache` exists in the list of oaieval options. I'm not sure if this is intended to prevent .jsonl files from being cached (which would be a nice feature), but as far as I can tell the code associated with the arg isn't doing anything:
> 
> https://github.com/openai/evals/blob/19bfdba1fd9027f6818672e35615f45584de8b02/evals/cli/oaieval.py#L173-L175
> 
> Though of course @andrew-openai will know a lot more than I do!