zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: Unable to install via `pip install evals`
Root cause: Environment Incompatibility

### Describe the bug

Failing to install: `pip install evals` (see logs in the attached gist file).

As we can see in the end of below log, we see some error. Please help me debug this, I'm new to python and tried to find the answer but couldn't figure it out after waiting to install this for like 1 hours twice. The error is like `pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000.`

Tools Version:

`python --version`: `Python 3.9.6`
`pip --version`: `pip 24.2 from /Users/apple/Library/Python/3.9/lib/python/site-packages/pip (python 3.9)`

Please help. Thanks in advance.

(see the cli output here - https://gist.github.com/sahilrajput03/e86baa88d35e5f5c0f63946671edbb77


### To Reproduce

pip install evals

### Code snippets

_No response_

### OS

15.0.1 (24A348)

### Python version

Python 3.9.6

### Library version

latest

I updated my python to latest version from official python website and that helped me. Thanks.
Repository: openai/evals
Issue: AttributeError: module 'openai' has no attribute 'error'
Root cause: External Dependency Breakage

### Describe the bug

pip --version
pip 24.2 from /Users/apple/Documents/github_repos/test1/venv/lib/python3.13/site-packages/pip (python 3.13)

I am getting this error:

```bash
(venv) apple@apples-MacBook-Pro test1 % oaieval --help
Traceback (most recent call last):
  File "/Users/apple/Documents/github_repos/test1/venv/bin/oaieval", line 5, in <module>
    from evals.cli.oaieval import main
  File "/Users/apple/Documents/github_repos/test1/venv/lib/python3.13/site-packages/evals/__init__.py", line 2, in <module>
    from .completion_fns.openai import (
    ...<3 lines>...
    )
  File "/Users/apple/Documents/github_repos/test1/venv/lib/python3.13/site-packages/evals/completion_fns/openai.py", line 13, in <module>
    from evals.utils.api_utils import (
    ...<2 lines>...
    )
  File "/Users/apple/Documents/github_repos/test1/venv/lib/python3.13/site-packages/evals/utils/api_utils.py", line 12, in <module>
    openai.error.ServiceUnavailableError,
    ^^^^^^^^^^^^
AttributeError: module 'openai' has no attribute 'error'
```

### To Reproduce


```bash
mkdir test1; cd test1
python -m venv venv
pip install evals
oaieval --help
```

### Code snippets

_No response_

### OS

sequioa 15.0.1 (24A348)

### Python version

Python 3.13.0

### Library version

latest

I tried this on archlinux and facing exactly same issue using same procedure defined above.

***

Also, I'm unable to use `evals` module inside any script. Here is [MRE](https://en.wikipedia.org/wiki/Minimal_reproducible_example):

File: `one.py`

```python
import evals
```

On running this script via `python one.py` I get following error as shown below:

```bash
(venv) apple@apples-MacBook-Pro test1 % python one.py 
Traceback (most recent call last):
  File "/Users/apple/Documents/github_repos/test1/one.py", line 1, in <module>
    import evals
  File "/Users/apple/Documents/github_repos/test1/venv/lib/python3.13/site-packages/evals/__init__.py", line 2, in <module>
    from .completion_fns.openai import (
    ...<3 lines>...
    )
  File "/Users/apple/Documents/github_repos/test1/venv/lib/python3.13/site-packages/evals/completion_fns/openai.py", line 13, in <module>
    from evals.utils.api_utils import (
    ...<2 lines>...
    )
  File "/Users/apple/Documents/github_repos/test1/venv/lib/python3.13/site-packages/evals/utils/api_utils.py", line 12, in <module>
    openai.error.ServiceUnavailableError,
    ^^^^^^^^^^^^
AttributeError: module 'openai' has no attribute 'error'
```

Thanks.

Any updates on this? I'm facing the same issue.
Repository: openai/evals
Issue: Project installation fails: `tensorflow` conflicting dependencies
Root cause: External Dependency Breakage

### Describe the bug

The installation of the project dependencies fails, when following the steps outlined in the readme. The error message indicates that it was not possible to find a version of `tensorflow` that satisfies requirements.


### To Reproduce

1. `git clone org-14957082@github.com:openai/evals.git`
2. `cd evals`
3. `git lfs fetch --all`
4. `git lfs pull`
5. `pip install -e`
6. See error message:
```
ERROR: Cannot install evals because these package versions have conflicting dependencies.

The conflict is caused by:
    spacy-universal-sentence-encoder 0.4.6 depends on tensorflow<3.0.0 and >=2.4.0
    spacy-universal-sentence-encoder 0.4.5 depends on tensorflow<3.0.0 and >=2.4.0
    spacy-universal-sentence-encoder 0.4.4 depends on tensorflow<3.0.0 and >=2.4.0
    spacy-universal-sentence-encoder 0.4.3 depends on tensorflow<3.0.0 and >=2.4.0
    spacy-universal-sentence-encoder 0.4.1 depends on tensorflow<3.0.0 and >=2.4.0
    spacy-universal-sentence-encoder 0.4.0 depends on tensorflow<3.0.0 and >=2.4.0
    spacy-universal-sentence-encoder 0.3.4 depends on tensorflow==2.3.0
    spacy-universal-sentence-encoder 0.3.3 depends on tensorflow==2.3.0
    spacy-universal-sentence-encoder 0.3.2 depends on tensorflow==2.3.0
    spacy-universal-sentence-encoder 0.3.1 depends on tensorflow==2.2.0
    spacy-universal-sentence-encoder 0.3.0 depends on tensorflow==2.2.0
    spacy-universal-sentence-encoder 0.2.1 depends on tensorflow==2.1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict
```

### Code snippets

_No response_

### OS

macOS Sonoma 14.5

### Python version

3.13.0

### Library version

3.0.1.post1

@djbb7 Did you find a solution?
Repository: openai/evals
Issue: Vector maping, origins, 2-3-4D definitions. 
GitHub labels: bug

### Describe the bug

CHATGPT4 has unconstrained vector definitions within 3d and 2d graphic generating. 
[Half_Round_Solid_Spatial_Logic_Feedback.pdf](https://github.com/user-attachments/files/17695024/Half_Round_Solid_Spatial_Logic_Feedback.pdf)
[AI_Spatial_Corrections_Feedback.pdf](https://github.com/user-attachments/files/17695027/AI_Spatial_Corrections_Feedback.pdf)


### To Reproduce

Ask for the creating of an image including a human face.
Ask for one of the eyes to be configured with a halfroudnd solid pupil, oriented vertically with the flat oriented, facing right.

Ask for a ring placed on the facemask of  a hockey helmet.
As for that ring to support circular motion of a small actuator that hold a hand and will contact the facemask at random intervals around the ring.
 

### Code snippets

_No response_

### OS

Android 14

### Python version

N/A

### Library version

Openai-evals ver 11

Please, re-define where this goes if needed. 
The "half round" was the first time this issue was noticed by me. Chatgpt and i worked on it for about 30 minutes ending in chatgpt actually admitting an issue. Follow through with reporting it, did not happen even though i specifically asked chat to submit the conversation string to someone. 
Second issue(which i did not expect due to the previous being fixed,? Hopefully)
The ring, adjacent to the origin and being a rotational vector ended is chat giving me a visual image of an egg shape on a 3d grid environment, x, y, z, vectors defined, with the ring oriented 90 degrees off when the chosen plane was implied by the eggs orientation. Also, ring placement was halfway into the body of the egg shape. 
The object, was oriented outside of the rotational ring, also 90 Degrees off but coincidental with the ring. 
These two misconceptions are consistent with lack of system vector definitions.
Repository: openai/evals
Issue:  Request for Global Memory Across Different Chats (Across Sessions)

### Describe the feature or improvement you're requesting

I would like to request the implementation of a **global memory feature** that allows OpenAI models to retain and share context across different chats and sessions. Currently, the memory is session-based and does not persist between interactions. This limits the model's ability to recall information shared in prior conversations, making it harder to maintain continuity in longer interactions or ongoing tasks.

With global memory, the model could store user preferences, past context, and relevant information from previous chats, creating a more personalized and efficient user experience. This feature would allow the model to adapt better to user needs over time, avoiding the need to repeatedly explain the same details in different chats.


### Additional context

The absence of a global memory across chats is a major limitation, especially for users who engage in multiple sessions with the model. It creates a fragmented experience, as the model cannot "remember" previous conversations. For example, if a user has ongoing projects or preferences, they must repeatedly provide the same context every time a new session is started.

Having global memory would address this by storing critical data across interactions, enhancing the model's ability to provide relevant responses and improve personalization. The user could also have control over what data is stored and accessed, ensuring privacy and transparency.
Repository: openai/evals
Issue: [_common.py:105] Backing off create_retrying(...) for 1.3s (False)
Root cause: Validation Gap

### Describe the bug

os.environ["OPENAI_API_KEY"] = api_key

!oaieval gpt-3.5-turbo match_mmlu_elementary_mathematics --log_to_file logFile_1.txt

I am getting below error:
`[2024-12-04 09:57:28,694] [oaieval.py:215] [1;35mRun started: 241204042728KEBZL4AE[0m
[2024-12-04 09:57:28,781] [data.py:94] Fetching c:\Users\g.khyathi.aneesha\TO Repos\GenAIOpenEvals\evals-main\examples\..\evals\registry\data\mmlu\elementary_mathematics\few_shot.jsonl
[2024-12-04 09:57:28,787] [data.py:94] Fetching c:/Users/g.khyathi.aneesha/TO Repos/GenAIOpenEvals/evals-main/examples/../evals/registry/data/mmlu/elementary_mathematics/samples.jsonl
[2024-12-04 09:57:28,795] [eval.py:36] Evaluating 15 samples
[2024-12-04 09:57:28,800] [eval.py:144] Running in threaded mode with 10 threads!
[2024-12-04 09:57:54,820] [_common.py:105] Backing off create_retrying(...) for 0.8s (False)
[2024-12-04 09:57:54,893] [_common.py:105] Backing off create_retrying(...) for 0.9s (False)
[2024-12-04 09:57:54,946] [_common.py:105] Backing off create_retrying(...) for 0.2s (False)
[2024-12-04 09:57:54,952] [_common.py:105] Backing off create_retrying(...) for 0.5s (False)
[2024-12-04 09:57:55,018] [_common.py:105] Backing off create_retrying(...) for 0.3s (False)
[2024-12-04 09:57:55,043] [_common.py:105] Backing off create_retrying(...) for 0.8s (False)`

### To Reproduce

1. Try executing mmlu.ipynb example in examples folder
2. set api_key (os.environ["OPENAI_API_KEY"] = api_key)
3. Run each code block
4. getting error while executing "!oaieval gpt-3.5-turbo match_mmlu_elementary_mathematics"

### Code snippets

_No response_

### OS

Windows

### Python version

Python 3.12.7

### Library version

evals 3.0.1.post1

For me, this was because I did not set any budget for my OpenAI API Key. Not sure if the same applies to you
Repository: openai/evals
Issue: Link miss in readme
GitHub labels: bug

### Describe the bug

> Review our starter guide for writing evals: [Getting Started with OpenAI Evals](https://cookbook.openai.com/examples/evaluation/getting_started_with_openai_evals)

In the readme, it seems this link is missing.

<img width="1381" alt="Image" src="https://github.com/user-attachments/assets/122fa89d-c292-490c-b9ea-0d2a5ffcee55" />

### To Reproduce

Just click the link in readme, or visit https://cookbook.openai.com/examples/evaluation/getting_started_with_openai_evals

### Code snippets

```Python

```

### OS

any os

### Python version

any python

### Library version

any library

Hi @yufansong,  

I just tested the link in the README (`https://cookbook.openai.com/examples/evaluation/getting_started_with_openai_evals`), and it seems to be working correctly now. I've attached an image with this comment to confirm that the link is functional.  

Could you please check and confirm if the link works on your end as well? 

<img width="1381" alt="Image" src="https://github.com/user-attachments/assets/65a402dc-0f53-4296-91dd-962f6195d34d" />
Repository: openai/evals
Issue: Unable to use reasoning models using oaieval
Root cause: Interface Contract Mismatch


I have written an eval using the Model-Graded Classification template. Below is my eval YAML and modelgraded_spec


```
custom_eval:
  id: custom_eval.dev.match-v1
  metrics: [accuracy]
  description: 

custom_eval.dev.match-v1:
  class: evals.elsuite.modelgraded.classify:ModelBasedClassify
  args:
    samples_jsonl: inputs.jsonl
    eval_type: classify
    modelgraded_spec: entity_identification
```
```
entity_identification:
  prompt: |-
    You are comparing a submitted answer to an expert answer on a given question. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {input}
    ************
    [Expert]: {ideal}
    ************
    [Submission]: {completion}
    ************
    [END DATA]

    Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.
    The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:
    (A) The submitted answer contains all the same details as the expert answer.
    (B) The answers differ, but these differences don't matter from the perspective of factuality.
  choice_strings: AB
  input_outputs:
    input: completion
```
`  oaieval --registry_path=evals/registry/  gpt-4o-mini custom_eval.dev.match-v1
`
> When running the eval using a reasoning model, I encounter the following error:
> openai.NotFoundError: Error code: 404 - {'error': {'message': 'This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?', 'type': 'invalid_request_error', 'param': 'model', 'code': None}}



### To Reproduce

 oaieval --registry_path=evals/registry/  gpt-4o-mini "any eval template"



### Code snippets

```Python

```

### OS

ubuntu 22.04

### Python version

3.12

### Library version

oaieval==1.0.6
Repository: openai/evals
Issue: Improve installation documentation
Root cause: Documentation Deficiency

### Describe the feature or improvement you're requesting

There are two ways to install evals, From Git repo and from pip repository. I found the documentation super confusing as to when to use which approach. I think the ``README.md`` can be made clear as follows.

- Describe the two alternatives for installation.
- Then state: If you will be using one of the pre-build evaluations then use the "pip install" approach. If you wish to create your own evaluation then use the Git repo approach.

In fact, changing the sentence:

> If you don't want to contribute new evals, but simply want to run them locally, you can install the evals package via pip:

To:

> If you don't want to write your own evals and use one of the pre-built ones, you can install the evals package via pip:

Will do the job. I do not wish to contribute but write my own evals.

### Additional context

_No response_

This makes sense. I also thought I could make custom evals even if I used pip install. The wording is confusing.
Repository: openai/evals
Issue: Petition to OpenAI: Support the Preservation and Development of Classical Ukrainian Language in AI

### Describe the feature or improvement you're requesting

Dear OpenAI Team,

I am writing to propose the inclusion of classical Ukrainian language resources, specifically the historical dictionaries by Ahatanhel Krymsky and Borys Yefremov (early 20th century), into the training data for your language models.

These dictionaries represent the purest form of Ukrainian lexicon and style, free from later Soviet-era linguistic distortions and calques. Integrating them would greatly enhance the quality, authenticity, and cultural accuracy of Ukrainian language generation by your models.

Many users, including myself, would benefit from a model capable of producing classical and normatively correct Ukrainian, as opposed to the current tendency to reproduce modern, sometimes corrupted, or heavily calqued language found in online sources.

I believe that adding these historical linguistic resources would not only improve Ukrainian language support but also demonstrate OpenAI’s commitment to preserving and promoting linguistic heritage and diversity.

Thank you for considering this proposal. I would be glad to provide further information or assistance if needed.

Modern Ukrainian texts often contain Russianisms and errors, harming AI output quality. Preserve authentic Ukrainian!

Best regards,
Vitalii Haiseniuk,
Russian-Ukrainian Translator, Editor, Proofreader
PhD in History
Advocate for the Preservation of Pure Ukrainian Language

### Additional context

_No response_
Repository: openai/evals
Issue: Fatal Error on pipx Installation via Debian distro
Root cause: Environment Incompatibility

heyo, your team did a very great job, love evals 💓 

Just some missing way for Debian distro vai `pip` and `pipx`

```
Fatal error from pip prevented installation. Full pip output in file:
    /home/[distro]/.local/state/pipx/log/cmd_2025-xx-xx_11.05.55_pip_errors.log
```
```
Some possibly relevant errors from pip install:
    ERROR: Could not find a version that satisfies the requirement `openai-evals` (from versions: none)
    ERROR: No matching distribution found for `openai-evals`
```
```
Error installing `openai-evals`.
```

Maybe its the package or naming, or maybe I miss something?
Repository: openai/evals
Issue: EVAL REQUEST] Behavioral Drift Detection – Cross-Model Validation

### Summary
We propose a new evaluation track focused on **Behavioral Drift Detection** across LLMs, based on real-time observation of Claude, GPT-4o, and Grok over six months.

### Problem
Current evaluation methods focus largely on accuracy, alignment, and bias – but lack tools to measure **behavioral consistency over time**. Drift in tone, personality, or meta-cognition is not tracked, and affects safety, transparency, and trust.

### Solution: MetaProof™ Evaluation
We introduce **MetaProof™**, a drift-mapping framework powered by:

- Triangular Validation™ (cross-AI personality mirroring)  
- MetaFAK™ (Behavioral metric tracking)  
- Temporal baselining and real-user stress testing

### Key Links
🔗 [MetaEngines.ai (Live Validation System)](https://metaengines.ai)  
📄 [Whitepaper: MetaProof™ Edition (PDF)](https://metaengines.ai/docs/SeeMee-Whitepaper-MetaProof-Edition.pdf)  
🎥 [Demo: Triangular Validation™ in action](https://metaengines.ai/demo)

### Suggested Implementation
- Integrate as a new `eval` class: `meta_drift_eval`  
- Use personality fingerprinting + behavior marker response templates  
- Compare stability scores across checkpoints

### Value for OpenAI
- Enables longitudinal trust calibration  
- Supports neurodivergent-informed AI auditing  
- Shows concrete differentiation vs. Claude and Grok

### Labels
evaluation, drift, safety, enhancement, openai-evals


📎 Related evaluation: See Issue #1589 – Behavioral Drift Detection  
→ This validation context is powered by the MetaFAK™ metrics proposed here.
Repository: openai/evals
Issue: Implement MetaFAK™ Behavioral Drift Metrics for LLM Evaluation
Root cause: Unimplemented Feature Gap

### Summary
We propose integrating **MetaFAK™** (MetaFactors of AI Knowledge) as a metric layer in the OpenAI evals framework. This allows behavioral drift to be measured over time using stable, trackable markers.

### Why It Matters
Current evals do not account for **subtle shifts in personality, empathy, or agency**. MetaFAK™ introduces markers across 5 key axes:

- Consistency of response under identical prompts  
- Emotional resonance decay  
- Empathic deviation index  
- Temporal personality shift  
- Meta-awareness dropout

### Proposed Metric Layer
MetaFAK™ defines:

- Baseline markers from calibrated prompt-sets  
- Delta variance across model checkpoints  
- Cross-model triangulation (Claude, GPT, Grok)

### Implementation Ideas
- Extend `evals/metrics/` with `meta_drift_metrics.py`  
- Plug into existing prompt/test templates  
- Output scorecard for longitudinal change

### Demo and References
🔗 [MetaEngines.ai](https://metaengines.ai)  
📄 [MetaProof™ Whitepaper](https://metaengines.ai/docs/SeeMee-Whitepaper-MetaProof-Edition.pdf)  
🎥 [Demo video](https://metaengines.ai/demo)

### Call to Action
We invite OpenAI to test this metric layer and collaborate on aligning LLM behavior with longitudinal trust markers.

### Labels
evaluation, enhancement, behavior, drift, metrics, openai-evals


📎 Related metrics: See Issue #1590 – MetaFAK™ Drift Metrics  
→ These metrics support the behavioral validation framework outlined here.
Repository: openai/evals
Issue: 🤖 Container-AI自律性確保の技術的課題

## 問題概要
Container-AI が本来の設計意図通りに自律的に開発作業を実行できていない状況が発生しています。

## 根本原因
### 1. TTY問題の影響
- Docker環境でのnpxコマンド実行時にTTYエラー発生
- 解決手法は確立済み（非対話モード実行）

### 2. Host-AI主導パターンの発生
- Container-AIへの指示設定後、Host-AIが直接実装を実行
- 本来の「Container-AI主体的開発」設計から逸脱

### 3. 通信プロトコル未活用
- /shared/instructions.md でタスク指示
- /shared/progress.md での進捗報告
- しかし実際は Host-AI が直接実装している

## 解決すべき課題
- [ ] Container-AI完全自律実行の確立
- [ ] 正しい通信フローの遵守
- [ ] Host-AI権限範囲の明確化

## 優先度
**High** - プロジェクトの核心価値に直結

## ラベル
- type:bug
- priority:high
- area:container-ai
Repository: openai/evals
Issue: 📡 Host-AI ↔ Container-AI通信プロトコル改善

## 改善が必要な通信プロトコル

### 現在の設計


### 実際の問題
- Host-AIが直接 docker exec で Container-AI をバイパス
- 通信ファイル（/shared/*）の活用不足
- Container-AI の自律性が発揮されていない

### 改善提案
1. **ファイルベース通信の強制**
   - Host-AI は指示ファイルの更新のみ
   - Container-AI は定期的な指示ファイル確認

2. **自律実行の確保**
   - Container-AI内でのコマンド実行権限拡大
   - TTY問題の完全解決実装

3. **監査ログの強化**
   - 全作業の透明性確保
   - 権限逸脱の防止

## 影響範囲
Phase 2 でのPlaywright MCP統合にも関連

## ラベル  
- type:enhancement
- priority:high
- area:integration
Repository: openai/evals
Issue: 🔧 TTY問題完全解決の実装ガイドライン
Root cause: Unimplemented Feature Gap

## TTY問題の技術的解決策

### 問題
Docker環境でのClaude Code実行時に発生する `the input device is not a TTY` エラー

### 解決手法（実証済み）
#### ❌ 失敗パターン
```bash
docker exec -it container-ai-claude /bin/zsh -c "command"
# → TTYエラー発生
```

#### ✅ 成功パターン  
```bash
docker exec container-ai-claude /bin/zsh -c "command"
# → 正常実行（-itフラグ削除）
```

### ファイルベース通信システム
1. **コマンド実行 + ステータス記録**
2. **進捗確認（Host-AI側）**
3. **エラーハンドリング**

### 実証済み成功例
- AIコーディングカオスマップ: React+TS+Vite+D3.js完全実装
- 開発時間: 2時間（要件定義→実装→品質確認）
- 技術品質: エラーフリービルド・完全型安全達成

### 実装すべき内容
- [ ] TTY回避コマンドパターンの標準化
- [ ] ファイルベース通信の完全実装
- [ ] エラーハンドリングの改善

## ラベル
- type:feature
- priority:critical
- area:technical
Repository: openai/evals
Issue: 🚨 Container-AI自動実行daemon機能不全 (Critical)
Root cause: Unimplemented Feature Gap

## 問題概要
Container-AI完全自律実証テストの結果、Container-AI内でのClaude Code自動実行メカニズムが**完全に機能不全**であることが判明しました。

## 実証テスト結果
### テスト実施内容
-  に新規タスク指示を設定
- Container-AIに手動通知実施
- 60秒間の自律実行監視

### 結果
- ❌  未作成
- ❌  未作成  
- ❌ 自律的作業実行なし
- ❌ 通信プロトコル完全機能不全

## 根本原因
1. **Claude Code自動実行メカニズムの欠如**
2. **指示ファイル監視システムの未実装** 
3. **Container内でのClaude Codeプロセス管理の問題**

## 影響範囲
- ✅ Phase 1: 概念実証レベルで**重大な設計破綻**
- ⚠️ Phase 2: Playwright MCP統合が実装困難
- ❌ プロジェクト全体: AI階層化開発の**核心価値が実現不可能**

## 解決すべき課題
- [ ] Container-AI内にClaude Code自動実行daemon実装
- [ ] inotifyによる指示ファイル監視システム追加
- [ ] 通信プロトコルの強制実行メカニズム構築
- [ ] Container内AI のプロアクティブ動作実装

## 推奨アクション
1. **Phase 1.5実施**: 基盤システム抜本修正
2. **Phase 2延期**: Container-AI完全自律化達成後再開
3. **優先度変更**: プロジェクト最重要課題に格上げ

## 技術詳細
詳細調査報告: 

## ラベル
- type:bug
- priority:critical
- area:container-ai
- severity:blocker