zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: openCompass中若使用CMMLU数据集对大模型进行评测，最终得分50分，这个50的得分是怎么计算的？具体代码在哪
Root cause: Documentation Deficiency

### Describe the feature

openCompass中若使用CMMLU数据集对大模型进行评测，最终得分50分，这个50的得分是怎么计算的？具体代码在哪

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can refer to this summarizer config:
/opencompass/configs/summarizers/groups/cmmlu.py

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: 在CPU上运行 - launch on CPU
Root cause: Configuration Error

![截屏2024-03-19 下午10 56 58](https://github.com/open-compass/opencompass/assets/62199595/7a3b079b-122d-4e3c-8f57-59addf26e927)


Please make sure that you have installed the torch(cuda version) successfully.

> Please make sure that you have installed the torch(cuda version) successfully.

Thanks! I found out that this error was caused by GPU settings.

@qianghuangwhu 你好我也遇到了这个问题，你能够说一下你怎么解决的吗？

> @qianghuangwhu 你好我也遇到了这个问题，你能够说一下你怎么解决的吗？

@202030481266 这个应该是你推理时候你设置的的GPU卡数的问题，最好先用一张卡试试。

opencompass 必须用gpu吗，能不能用cpu，模型用的是api

> opencompass 必须用gpu吗，能不能用cpu，模型用的是api

他这个没有GPU的话，应该默认用cpu的；如果你有GPU的话，最好设置GPU环境变量为单卡：
`os.environ["CUDA_VISIBLE_DEVICES"] = "0"`
Repository: open-compass/opencompass
Issue: [Bug] missing mtbench dataset
Root cause: Unimplemented Feature Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

OpenCompassData-complete-20240207.zip

### Reproduces the problem - code/configuration sample

the "OpenCompassData-complete-20240207.zip" missing the mtbench dataset, which mentioned in this changelog: https://github.com/open-compass/opencompass/releases/tag/0.2.2.rc1.

### Reproduces the problem - command or script

none

### Reproduces the problem - error message

none

### Other information

none

Can you show me your config? By the way, which model are you running, huggingface model or API-based model?

Hello, I am using `configs/eval_subjective_mtbench.py` and I noticed that the `subject_datasets` variable `data_path` is configured as `data/subjective/mtbench`. However, I did not find the corresponding dataset file in this directory. BTW, I have downloaded the complete dataset package [OpenCompassData-complete-20240207.zip](https://github.com/open-compass/opencompass/releases/tag/0.2.2.rc1).

> Hello, I am using `configs/eval_subjective_mtbench.py` and I noticed that the `subject_datasets` variable `data_path` is configured as `data/subjective/mtbench`. However, I did not find the corresponding dataset file in this directory. BTW, I have downloaded the complete dataset package [OpenCompassData-complete-20240207.zip](https://github.com/open-compass/opencompass/releases/tag/0.2.2.rc1).

Well, this is because we reorganized the datapath and have not release the new files yet. Thanks for your reminder, we will add it later, and you can also put these file in your path.
[mtbench.zip](https://github.com/open-compass/opencompass/files/14739752/mtbench.zip)
By the way, there are four files in this zip, the mtbench.json is the whole json data, and the others are the part of the data, which split by the different temperature settings in MTBench.

Thanks a lot！
Repository: openai/evals
Issue: Special Token Encoding Error in OpenAI Evals Library
Root cause: Validation Gap

### Describe the bug

When attempting to encode a string containing a special token using the Transformers library in the OpenAI Evals, an error message is displayed. The error seems to indicate that the library is not able to handle the encoding of certain special tokens properly. Here's the error message:
', ...}`.
If you want this text to be encoded as a special token, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<your_special_token>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.


### To Reproduce

1. Import the necessary libraries and load a pretrained model tokenizer.
2. Create a string containing a special token.
3. Attempt to encode the string using the tokenizer.
4. Observe the error.

### Code snippets

```Python
from transformers import AutoTokenizer

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# Create a string with a special token
text = 'This is a [SEP] test.'

# Attempt to encode the string
tokens = tokenizer.encode(text, disallowed_special=tokenizer.special_tokens_set - {'[SEP]'})
```


### OS

macOS

### Python version

Python 3.11.1

### Library version

openai-evals 1.0.3
Repository: openai/evals
Issue: Travando Chat GPT
GitHub labels: bug

### Describe the bug

Quando digito o seguinte "como eu posso You can also gain priority access if you contribute model evaluations to OpenAI Evals" misturando em Português e em inglês o site do chat GPT bug.



### To Reproduce

![bug](https://github.com/openai/evals/assets/79059166/a3f749d4-209a-460b-88b0-853d3dd28db1)


### Code snippets

_No response_

### OS

Windows 

### Python version

11

### Library version

chat.openai.com

Please use the feedback tool in ChatGPT. Thanks. 

This issue should be closed.
Repository: openai/evals
Issue: The `eval_type` within  ModelBasedClassify is always None
Root cause: Documentation Deficiency

### Describe the bug

I've been running OpenAI model-graded evaluations locally. I've noticed that the `eval_type` instance variable is always set to `None` regardless of what value is specified in the spec.

Take for example this eval.

```yaml
coqa-fact-expl:
  id: coqa-fact-expl.dev.v0
  metrics: [accuracy]
coqa-fact-expl.dev.v0:
  class: evals.elsuite.modelgraded.classify:ModelBasedClassify
  args:
    samples_jsonl: coqa/samples.jsonl
    eval_type: classify_cot
    modelgraded_spec: fact
```

The `eval_type` is set to `classify_cot` but if you place a print statement before `self.eval_type`'s creation, you'll see that it's None within the eval class.

```python
print("Eval type passed into ModelBasedClassify", eval_type) # Shows None
self.eval_type = eval_type
```

This results in the relevant `ANSWER_PROMPT` not being appended to the prompt in the original spec.

### To Reproduce

1. Place `print()` statement to reveal the value of `eval_type`

```python
print("Eval type passed into ModelBasedClassify", eval_type)
```

### Code snippets

_No response_

### OS

macOS

### Python version

Python v3.11.0

### Library version

openai-evals v1.0.3.post1

I apologize, I misunderstood the interface. This is working correctly
Repository: openai/evals
Issue: Exact Match template
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

The documentation in eval-templates.md describes `basic/match.py` as `Match: any([b.startswith(a) for b in B])` "[f]or a model completion `a` and a reference list of correct answers `B`.  This is a poor fit for arithmetic and other algorithmic tasks, where we want the model response to exactly match some ideal answer, i.e., `any([b == a for b in B])`

### Additional context

_No response_

For arithmetic and other complex tasks, it is recommended to ask the model for reasoning before answering. For such cases, proper prompt instructions are used, like asking the model to enclose the final answer in square brackets and formatting the ideal answer like `[5]`.
Repository: openai/evals
Issue: main is broken for model graded 
Root cause: Validation Gap

### Describe the bug

running model graded fact eval on main fails on python exception 
```
in ".../openai/evals/evals/registry/modelgraded/sql.yaml", line 5, column 5
expected alphabetic or numeric character, but found '*'
```


### To Reproduce

git pull upstream main

From github.com:openai/evals

branch main -> FETCH_HEAD
Already up to date.
oaieval gpt-3.5-turbo  coqa-fact

...
[2023-05-30 08:23:59,114] [registry.py:250] Loading registry from .../openai/evals/evals/registry/modelgraded
Traceback (most recent call last):
File ".../openai/evals/.venv/bin/oaieval", line 8, in
sys.exit(main())
...
yaml.scanner.ScannerError: while scanning an alias
in ".../openai/evals/evals/registry/modelgraded/sql.yaml", line 5, column 5
expected alphabetic or numeric character, but found '*'
in "...openai/evals/evals/registry/modelgraded/sql.yaml", line 5, column 6

### Code snippets

_No response_

### OS

Ubuntu 22.04.1 LTS (on WSL Windows11) 

### Python version

Python 3.10.6

### Library version

latest main

Believe this is fixed now that we merged https://github.com/openai/evals/pull/1052
Repository: openai/evals
Issue: Communicate GPT-4 access grant timelines

### Describe the feature or improvement you're requesting

I've observed that several authors of recently merged evaluation PRs have requested GPT-4 access. However, these requests seem to have been met with limited response from the team since May. For example:
- https://github.com/openai/evals/pull/675#issuecomment-1567847490
- https://github.com/openai/evals/pull/1034#issuecomment-1565123305

Contrastingly, during April, it was quite typical to see responses granting GPT-4 access to authors of merged PRs, as evidenced here: https://github.com/openai/evals/pull/821#issuecomment-1524146854

Given that most eval contributors are eagerly anticipating GPT-4 access as a result of their merged PRs, a more transparent communication strategy from the team would be appreciated. At present, it appears most contributors anticipate a delay of O(days) to gain access based on past access grants in April.

If this timeframe does not align with the current reality, it would be immensely beneficial for contributors to receive clarity on the expected wait times. This adjustment in expectation would allow them to better plan their experiments and subsequent activities.

Thank you for your attention to this matter!

Best regards,

### Additional context

_No response_

I've been granting access usually within a day of when the eval is merged, just the comments haven't been explicitly mentioning it as so. We'll be more explicit going forward, thanks for the feedback. The accounts associated with the two PRs you linked already had GPT-4 added. You can see your access here: https://platform.openai.com/playground?mode=chat

> I've been granting access usually within a day of when the eval is merged, just the comments haven't been explicitly mentioning it as so. We'll be more explicit going forward, thanks for the feedback. The accounts associated with the two PRs you linked already had GPT-4 added. You can see your access here: https://platform.openai.com/playground?mode=chat

I'm having the same issue here, I got my PR merged almost two weeks ago, #1006 and I still haven't gotten access - email is correct on the commit, as I checked the [patch](https://github.com/openai/evals/commit/b91292c803af2bdadeec3853ab03514b73310109.patch) of the commit here.
Repository: openai/evals
Issue: Requesting for GPT-4 access for already merged PR

### Describe the feature or improvement you're requesting

Hi @andrew-openai ,

As mentioned in #1066 , GPT-4 access is usually granted within a day of when the eval is merged. But my merged PRs:

#837 was merged over a week,

#644 was merged two days ago,

and I still can't find my GPT-4 access at https://platform.openai.com/playground?mode=chat. I've commented in theses PR, but no one replied to me.

Are there any other actions from my side to get GPT-4 API access?

Thank you for your time to resolve this issue!

Best regards,

### Additional context

_No response_

I added access to the account associated with the email on your commits, it was an @outlook address. There are no guarantees I'll see a comment that asks for access to be added to a different account due to the volume of PRs we are looking through (we also mention this in the PR template).

I can add it to the account you mentioned now.
Repository: openai/evals
Issue: pip package is outdated (1.0.3.post1) - has broken elsuite/basic/includes.py
Root cause: External Dependency Breakage

### Describe the bug

Running an eval that uses the `evals.elsuite.basic.includes:Includes` class via the latest pip package (1.0.3.post1) as a standalone app gives misleading results.

### To Reproduce

1. Install evals 1.0.3.post1: `pip install evals`.
2. Create file `test.yaml` in `$(HOME)/.evals` with contents in code snippets section.
3. Create file `samples.json` in `/tmp/samples.json` with contents below:
~~~
{"input": [{"role": "system", "content": "Answer the following questions as concisely as possible."}, {"role": "system", "content": "What's the capital of France?", "name": "example_user"}, {"role": "system", "content": "Paris", "name": "example_assistant"}, {"role": "system", "content": "What's 2+2?", "name": "example_user"}, {"role": "system", "content": "4", "name": "example_assistant"}, {"role": "user", "content": "Who is the girl who plays eleven in stranger things?"}], "ideal": ["FAKE"]}
~~~
4. In a shell run `oaieval gpt-3.5-turbo mytest`.

The tool reports an accuracy of 1.0 instead of 0.


### Code snippets

```Python
my-test:
  id: my-test.dev.v0
  description: Test
  metrics: [accuracy]

game-theory.dev.v0:
  class: evals.elsuite.basic.includes:Includes
  args:
    samples_jsonl: /tmp/samples.jsonl
    ignore_case: true
```


### OS

Linux pop-os 6.2.6-76060206-generic #202303130630~1683753207~22.04~77c1465 SMP PREEMPT_DYNAMIC Wed M x86_64 x86_64 x86_64 GNU/Linux

### Python version

Python 3.10

### Library version

1.0.3-post1
Repository: openai/evals
Issue: Allow easier way to package local evals with samples not in the repo
Root cause: Architectural Constraint

### Describe the feature or improvement you're requesting

Currently when using the pip package as a standalone app, the path to the samples_jsonl file has to be either an absolute path, or is relative to the installed module directory.  This makes it hard to portable evals that can be used independently of those in the original repo, since the path to the jsonl must be a hard-coded absolute path.  It would be more convenient if the tool first looked for the samples file relative to the module dir, and then if it cannot find it there it could look in `$(HOME)/.evals/data`.

### Additional context

See #1098 for an example where we create a local eval.  Notice the hard-coded absolute path to the samples.jsonl file.

> or is relative to the installed module directory

Specifically, I found that it's relative to the `registry/data` folder in the default registry. So in my case, I needed 6x `../` to traverse upwards from `venv\\lib\\site-packages\\evals\\registry\\data\\`

Encountered the same problem. The data directory is hard-coded in the code, this part should be registered like /completion_fns, /evals.

```
def open_by_file_pattern(filename: str, mode: str = "r", **kwargs: Any) -> Any:
....
            if scheme == "" or scheme == "file":
                return open_fn(
                    os.path.join(
                        os.path.dirname(os.path.abspath(__file__)), "registry", "data", filename
                    ),
                    mode=mode,
                )
            else:
                return open_fn(filename, mode=mode)
....
```

I had the same problem.

The workaround I got was using gz file.

In the eval registry file I used the path `<local_registry_relative_path>/data/<eval>/samples.jsonl.gz` then I call `oaieval --registry_path ./< local_registry_relative_path> ...` from the parent folder of my local registry.

It works because gz files are returned before the path modification.

https://github.com/openai/evals/blob/259992faf4830e39455a5d2ac22437f5d6487b77/evals/data.py#L53-L54

I also ran into this issue and would appreciate if this was solved. The solution described in the first post here seems like the right thing.

> Specifically, I found that it's relative to the `registry/data` folder in the default registry. So in my case, I needed 6x `../` to traverse upwards from `venv\\lib\\site-packages\\evals\\registry\\data\\`

N.B. This is a different path on a non-Windows machine
Repository: openai/evals
Issue: Requesting GPT-4 access for merged PR - two week silence

### Describe the feature or improvement you're requesting

Hello @andrew-openai ,

I'd hate to bother anyone, but mentioned in [issue 1066](https://github.com/openai/evals/issues/1066), GPT-4 access is usually granted within a day of when the eval is merged. However, I got #1006 merged two weeks ago, and I still don't have any GPT4 access. This issue is almost identical to [this issue](https://github.com/openai/evals/issues/1068). I also tried leaving a comment on an issue to try to ask the timelines, but got nothing. Should I do anything on my side to get GPT-4 API access?

Thank you!

### Additional context

_No response_

That's weird, does this work? syn@zbeacon.org
Repository: openai/evals
Issue: Awaiting GPT4 access
Root cause: Configuration Error

PR #500 was merged a week ago, and I don't have GPT4 access on my account alex dot meiburg at gmail -- which is the address listed on my Github account, and my openai account. After checking, I realized that the email attached to the _commit_ was different, timeroot dot alex at gmail. So I made an OpenAI account with that email, to see if it has access, but it doesn't. Wondering where the access ended up, or when I should expect it. (Maybe because the timeroot openai account didn't exist at the time of merge, something didn't go through properly?)

Added to both emails!

can u check ***@***.*** also

On Fri, Jun 9, 2023 at 12:20 AM Andrew ***@***.***> wrote:

> Closed #1121 <https://github.com/openai/evals/issues/1121> as completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/1121#event-9477047147>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AZO4SQZNFW5CUBDWG5FNL2DXKI6ZVANCNFSM6AAAAAAY6TLVEY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


> Added to both emails!

I'm not seeing access, despite giving it a few days to update. Can you confirm exactly which account(s) have access now @andrew-openai ? Or, is there something I have to click or enable? When I go to https://chat.openai.com/ I still see "Upgrade to Plus!" as an option, and on the https://platform.openai.com/playground page I only see "GPT-3" based models.
Repository: openai/evals
Issue: Extracting and analyzing data from scientific articles
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

It will be possible to create a benchmark that will evaluate the ability of the model to extract key information from abstracts of scientific articles.  For example, information such as:

 1. Object of study (what was studied in the article)
 2. Methodology (what methods were used to conduct the study)
 3. Results (what were the main results of the study)
 4. Conclusions (what conclusions were drawn by the authors based on the results)

 Abstracts from real scientific articles can be used as input, and pre-prepared answers that accurately represent the information to be extracted can be used as the ideal answer.  This will allow you to assess how well the model is able to understand and analyze scientific texts.

### Additional context

_No response_
Repository: openai/evals
Issue: stock picking as model eval
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

An idea for an eval which won't suffer from data contamination or overfitting.

maybe use financial news before the opening bell for some top S&P stocks + historical movements.

no need to trade, just use appropriate and standard portfolio metrics

at the very least, it would be an entertaining eval!

### Additional context

_No response_

Some more, brainstorming with gpt4:

Sports Betting or Fantasy Sports,Social Media Trend Prediction, Climate change temps, CDC stuff (flus, etc), Polling (rcp daily averages?)

An idea might be to use the various prediction markets around the web for sources of ideas and data.
Repository: openai/evals
Issue: Registry path CLI option for oaievalset
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

It should take an `--registry_path` like `oaieval` does

### Additional context

Useful when using as a library, to run eval sets stored in my own project outside this repo.

Looking at the source `oaievalset` passes unknown arguments through to `oaieval`. But I think it needs to add that path to its own registry before that so it can find the evalset.
Repository: openai/evals
Issue: Awaiting GPT-4 Access For Further Development
Root cause: Documentation Deficiency

PR #1013 was merged weeks ago and I still haven't been granted access to GPT-4. I mentioned I was interested in further testing and training as a developer, and I made sure to include numerous test cases in my last PR, far more than the requirement. I checked the email on my commit and it was the same as my OpenAI account email, grahamzemel at gmail. Was there an issue on the back-end that didn't match up my account correctly or was I just not granted access at all? If possible, I'd like to make further contributions once GPT-4 becomes available to my email. Thank you. 

Hi, you should have gpt-4 access with that email, can you access it at https://platform.openai.com/playground?mode=chat&model=gpt-4 for example?

Ah, got it. I was looking on the main website of chatGPT, I can access it
with the playground link. Thanks.
Repository: openai/evals
Issue: Functions: Support for minItems and maxItems for json schema array
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

First off - let me just say how great functions are! Game changer!

It would be really cool if the functions would respect validation like `minItems` and `maxItems` for an array. 
If I explicitly put the number of questions I want in the function description it works, but I half expected it to work when setting `minItems`. Unfortunately, if I specify `minItems: 10` it only returns 5 questions. 

I'd imagine other json schema validation options are ignored, and this could be a more generic ticket. 

```js
const func = {
  name: "generateQuiz",
  description: "Generate a questions about a given topic",
  parameters: {
    type: "object",
    properties: {
      topics: {
        type: "array",
        items: {
          type: "string",
          description: "Topic of the lesson, e.g. Planes",
        },
      },
      questions: {
        type: "array",
        minItems: 10,
        maxItems: 20,
        description:
          "Questions, e.g. What is the primary function of an airplane?",
        items: {
          type: "object",
          properties: {
            question: {
              type: "string",
              description:
                "Question, e.g. What is the primary function of an airplane?",
            },
            answer: {
              type: "string",
              description: "Answer for the question",
            },
          },
          required: ["question", "answer"],
        },
      },
    },
    required: ["topic", "questions"],
  },
};
```

### Additional context

_No response_