zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug] /bin/sh: torchrun: command not found
Root cause: Environment Incompatibility

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

同官方

### 重现问题 - 代码/配置示例

#修改了task.py，希望评估qwen-vl-chat
from mmengine.config import read_base

with read_base():
from .qwen.qwenvl_chat_7b_mmbench_cn import (qwen_mmbench_dataloader,
qwen_mmbench_evaluator,
qwen_model)

models = [qwen_model]
datasets = [qwen_mmbench_dataloader]
evaluators = [qwen_mmbench_evaluator]

num_gpus = 1
num_procs = 1
launcher = 'pytorch'

### 重现问题 - 命令或脚本

python configs/multimodal/tasks.py --mm-eval

### 重现问题 - 错误信息

stdout如下:
launch qwen-vl-chat-opencompass.MMBenchDataset-opencompass.DumpResults on GPU 0
11/09 16:33:55 - OpenCompass - WARNING - task qwen-vl-chat-opencompass.MMBenchDataset-opencompass.DumpResults fail, see
/data/oss_bucket_0/guolianghui/outputs/default/20231109_163352/qwen-vl-chat/opencompass.MMBenchDataset/opencompass.DumpResults.out
11/09 16:33:55 - OpenCompass - ERROR - /checkpoint/binary/train_package/opencompass/runners/base.py - summarize - 63 - qwen-vl-chat-opencompass.MMBenchDataset-opencompass.DumpResults failed with code 127


stderr如下：
/root/.local/lib/python3.8/site-packages/colossalai/kernel/cuda_native/mha/flash_attn_2.py:28: UserWarning: please install flash_attn from https://github.com/HazyResearch/flash-attention
  warnings.warn("please install flash_attn from [https://github.com/HazyResearch/flash-attention")](https://github.com/HazyResearch/flash-attention%22))

  0%|          | 0/1 [00:00<?, ?it/s]
                                     

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:03<00:00,  3.13s/it]
100%|██████████| 1/1 [00:03<00:00,  3.14s/it]


在输出的日志文件opencompass.DumpResults.out中记录如下
/bin/sh: torchrun: command not found


### 其他信息

查阅资料说torch1.9.1以上可以支持torchrun，我的torch版本是2.0.1
另外，可否给出一个自定义模型，跑mmbench的step by step的详细介绍呀？十分感谢

Hi, @LianghuiGuo ,
请检查你的环境中是否有 torchrun 这条命令，理论上，这条命令是随你安装 torch 自动装上的
另外，我们正在准备一个新版的多模评测工具，预计下周开源，届时应可以满足你的需求


Hi, @LianghuiGuo , 
Please check our newly released codebase [VLMEvalKit](https://github.com/open-compass/VLMEvalKit), which is more straightforward and easy to use. 

虚拟环境是不是从别的地方移动过来的，先在命令行里面输入torchrun ，看是否会报错
Repository: open-compass/opencompass
Issue: [Feature] 
Root cause: Unimplemented Feature Gap

### 描述该功能

Support TydiQA?

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

hello，非常感谢您对OpenCompass支持,
首先，关于TydiQA-GoldP任务的支持马上就会上线了，我们非常欢迎您提出建议，
对于TydiQA的其他任务，比如Primary tasks中的SelectP，MinSpan任务，我们也欢迎您贡献代码，以完善TydiQA任务。
祝好

It's done in https://github.com/InternLM/opencompass/pull/75
Repository: open-compass/opencompass
Issue: [Bug] Links of documentation is misdirecting
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

I am browsing the leaderboard: https://opencompass.org.cn/leaderboard-llm. However, I found the links of benchmarks do not work as expected. Those circled ones all direct to https://github.com/nyu-mll/jiant when I click the GitHub website.
![image](https://github.com/open-compass/opencompass/assets/8592144/ce48a183-5ba2-40a6-8ae6-8a67f72be6b1)
![image](https://github.com/open-compass/opencompass/assets/8592144/1457d2b4-cdb4-4a63-9cf2-f2bef21fa2da)



### Reproduces the problem - code/configuration sample

Click on the following links:
![image](https://github.com/open-compass/opencompass/assets/8592144/b27150c9-1408-42ac-bcd2-e7c8aefd3c6d)


### Reproduces the problem - command or script

Click on any of those leaderboards:
![image](https://github.com/open-compass/opencompass/assets/8592144/ce48a183-5ba2-40a6-8ae6-8a67f72be6b1)

### Reproduces the problem - error message

They redirect me to https://github.com/nyu-mll/jiant

### Other information

_No response_

Thanks for your report. 

For the datasets AX-b, AX-g, RTE, and COPA, they are all part of SuperGLUE, and the evaluation tool and repo used by SuperGLUE is `nyu-mll/jiant`. We believe this is not a bug.

![image](https://github.com/open-compass/opencompass/assets/17680578/81ba162a-c8dc-4246-8d0d-4f52fcc6739f)


Feel free to re-open if needed.

> Thanks for your report.
> 
> For the datasets AX-b, AX-g, RTE, and COPA, they are all part of SuperGLUE, and the evaluation tool and repo used by SuperGLUE is `nyu-mll/jiant`. We believe this is not a bug.
> 
> ![image](https://user-images.githubusercontent.com/17680578/282645725-81ba162a-c8dc-4246-8d0d-4f52fcc6739f.png)

感谢回复，但是依旧不make sense啊，这样的话，更应该把它们redirect到supercglue的repo上吗？redirect到一个evaluation tool上依旧让人困惑吧。。。

- 而且ax-g其实就是Winograd Schema Challenge的衍生数据集，正确的link应该是https://github.com/rudinger/winogender-schemas
- RTE就是Recognizing Textual Entailment，根本不是出自superglue，而是出自https://tac.nist.gov//2011/RTE/index.html 
- COPA就更不是了，它来自https://people.ict.usc.edu/~gordon/copa.html
- ReCoRD也不是，应该是https://sheng-z.github.io/ReCoRD-explorer
- AX-b出自[CLUE](https://gluebenchmark.com/diagnostics)，这个倒是可以勉强算是SUPEGLUE的东西，但是redirect到tool还是非常不make sense

本质上，这些数据集都不来自superglue，不能因为一个榜单收录这个数据集就算出自人家的，而nyu-mll/jiant是mll实验室的一个evaluation tool，跟数据来源的github repo没什么关系吧。。

@Leymore @tonysy 

We updated the links following your instructions, plz check:
- https://opencompass.org.cn/dataset-detail/AX-b
- https://opencompass.org.cn/dataset-detail/AX-g
- https://opencompass.org.cn/dataset-detail/RTE
- https://opencompass.org.cn/dataset-detail/COPA
- https://opencompass.org.cn/dataset-detail/ReCoRD
Repository: open-compass/opencompass
Issue: [Bug] lark报告时，base,py L66会有bug
Root cause: Validation Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

-

### 重现问题 - 代码/配置示例

-

### 重现问题 - 命令或脚本

-

### 重现问题 - 错误信息

Traceback (most recent call last):
  File "/alg_vepfs/private/zhaoyu/opencompass/run.py", line 346, in <module>
    main()
  File "/alg_vepfs/private/zhaoyu/opencompass/run.py", line 298, in main
    runner(tasks)
  File "/alg_vepfs/private/zhaoyu/opencompass/opencompass/runners/base.py", line 39, in __call__
    self.summarize(status)
  File "/alg_vepfs/private/zhaoyu/opencompass/opencompass/runners/base.py", line 66, in summarize
    num_succeeded = len(status) - len(failed_logs)
TypeError: object of type 'generator' has no len()

### 其他信息

最后的这个status是预期外的值，并不是
```
status (list[tuple[str, int]]): A list of (task name, exit code).
```

<img width="619" alt="image" src="https://github.com/open-compass/opencompass/assets/30463691/c74d291f-b2d9-414d-8a33-36cd39eae64f">


<img width="770" alt="image" src="https://github.com/open-compass/opencompass/assets/30463691/5b164367-25ae-456c-91af-2f3363bb1704">

需要注释掉后，可以正常运行
Repository: open-compass/opencompass
Issue: [Bug] mmbench评测，一直连不上chatGPT，Got connection error
Root cause: External Dependency Breakage

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

同官方

### 重现问题 - 代码/配置示例

同官方

### 重现问题 - 命令或脚本

python tools/eval_mmbench.py

### 重现问题 - 错误信息

<img width="1561" alt="image" src="https://github.com/open-compass/opencompass/assets/87937587/5b12168b-c0fc-425f-aa9e-68ab3a0f140e">


### 其他信息

使用官方提供的脚本tools/eval_mmbench.py，评估mmbench，但无法连接chatGPT

The GPT service is temporarily unavailable, we are working on this problem.


> The GPT service is temporarily unavailable, we are working on this problem.

好的，感谢~

Feel free to re-open.
Repository: open-compass/opencompass
Issue: [Bug] `libGL.so.1: cannot open shared object file` error returned by following the Quick Start 
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

the latest version:
`048775192ba3c5e56708eb15788f01ead26d4e98`

### Reproduces the problem - code/configuration sample

```default```

### Reproduces the problem - command or script

```
python run.py --models hf_opt_125m hf_opt_350m --datasets siqa_gen winograd_ppl
```

### Reproduces the problem - error message

```
Traceback (most recent call last):
  File "run.py", line 9, in <module>
    from opencompass.partitioners import MultimodalNaivePartitioner
  File "/cpfs01/user//opencompass/opencompass/partitioners/__init__.py", line 1, in <module>
    from .mm_naive import *  # noqa: F401, F403
  File "/cpfs01/user//opencompass/opencompass/partitioners/mm_naive.py", line 8, in <module>
    from .base import BasePartitioner
  File "/cpfs01/user//opencompass/opencompass/partitioners/base.py", line 7, in <module>
    from opencompass.utils import get_logger, task_abbr_from_cfg
  File "/cpfs01/user//opencompass/opencompass/utils/__init__.py", line 4, in <module>
    from .collect_env import *  # noqa
  File "/cpfs01/user//opencompass/opencompass/utils/collect_env.py", line 2, in <module>
    from mmengine.utils.dl_utils import collect_env as collect_base_env
  File "/cpfs01/user//miniconda3/envs/lmdeploy006/lib/python3.8/site-packages/mmengine/utils/dl_utils/__init__.py", line 3, in <module>
    from .collect_env import collect_env
  File "/cpfs01/user//miniconda3/envs/lmdeploy006/lib/python3.8/site-packages/mmengine/utils/dl_utils/collect_env.py", line 8, in <module>
    import cv2
  File "/cpfs01/user//miniconda3/envs/lmdeploy006/lib/python3.8/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/cpfs01/user//miniconda3/envs/lmdeploy006/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
  File "/cpfs01/user//miniconda3/envs/lmdeploy006/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
(lmdeploy006) /cpfs01/user//opencompass> pip show opencompass
WARNING: Package(s) not found: opencompass
```

### Other information

_No response_

please use `pip install mmengine-lite`, we will fix this problem soon

> please use `pip install mmengine-lite`, we will fix this problem soon

After installing `mmengine-lite`, I still get the same error msg returned.

#622 We have updated the requirements, please pull the latest version and try again

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] Missing postprocessing of longbench trec, triviaqa, samsum, lsht
Root cause: Unimplemented Feature Gap

### Describe the feature

Regarding the official repo, https://github.com/THUDM/LongBench/blob/main/eval.py#L52
Opencompass misses the postprocessing of fetching the first row of prediction as the answer. This would affect the results of the exact match, so the final long bench evaluation. 

```python
prediction = prediction.lstrip('\n').split('\n')[0]
```

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thank you for your carefulness. We have fixed this issue in #625.
Repository: open-compass/opencompass
Issue: [Feature] How many shot in the leaderboard?
Root cause: Documentation Deficiency

### Describe the feature

I can't see how many shots in input prompt on each leaderboard.
For example, how to know shot number used in MATH leaderboard?
https://opencompass.org.cn/dataset-detail/MATH

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You may hover the header of each column. e.g. 4-shot in this case.

![image](https://github.com/open-compass/opencompass/assets/17680578/e6a09330-e19a-43af-8231-9cf62cce719a)
Repository: open-compass/opencompass
Issue: [Feature] Add GPQA benchmark?
Root cause: Unimplemented Feature Gap

### Describe the feature

There's a recent paper "GPQA: A Graduate-Level Google-Proof Q&A Benchmark" (https://arxiv.org/abs/2311.12022) which is useful for helping to devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the insightful suggestions. We will add this into our backlog. Contribution is also welcomed.
Repository: open-compass/opencompass
Issue: [Bug] Links of documentation is misdirecting
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

I am browsing the leaderboard: https://opencompass.org.cn/leaderboard-llm. However, I found the links of benchmarks do not work as expected. Those circled ones all direct to https://github.com/nyu-mll/jiant when I click the GitHub website.
![image](https://github.com/open-compass/opencompass/assets/8592144/ce48a183-5ba2-40a6-8ae6-8a67f72be6b1)
![image](https://github.com/open-compass/opencompass/assets/8592144/1457d2b4-cdb4-4a63-9cf2-f2bef21fa2da)

### Reproduces the problem - code/configuration sample

Click on the following links:
![image](https://github.com/open-compass/opencompass/assets/8592144/b27150c9-1408-42ac-bcd2-e7c8aefd3c6d)


### Reproduces the problem - command or script

Click on any of those leaderboards:
![image](https://github.com/open-compass/opencompass/assets/8592144/ce48a183-5ba2-40a6-8ae6-8a67f72be6b1)


### Reproduces the problem - error message

They redirect me to https://github.com/nyu-mll/jiant

### Other information

_No response_

Check the unfinished conversion at https://github.com/open-compass/opencompass/issues/584
Reopen it since it was not solved.

Thanks, we may continue the discussion in #584 .
Repository: open-compass/opencompass
Issue: [Bug] [doc] page not found, parent page: `https://opencompass.readthedocs.io/en/stable/`

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

-

### 重现问题 - 代码/配置示例

-

### 重现问题 - 命令或脚本

-

### 重现问题 - 错误信息

`https://opencompass.readthedocs.io/en/stable/`
点击以中文阅读，404
`https://opencompass.readthedocs.io/zh-cn/stable/`
点击以英文阅读，404


### 其他信息

_No response_

Thanks for your carefulness, it seems to be a trick one to solve directly, we will keep an eye for this.
Repository: open-compass/opencompass
Issue: [Feature] 如何在此项目上测试gpt和文心一言等APIhumaneval等代码生成数据集上的表现
Root cause: Documentation Deficiency

### 描述该功能

我希望能够在API测试代码生成数据集上有比较详细的介绍

### 是否希望自己实现该功能？

- [X] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

- You can refer https://github.com/open-compass/opencompass/blob/main/configs/api_examples/eval_api_baidu.py for how to use API model.
- Change CEval into HumanEval

You can contact us with opencompass@pjlab.org.cn or join wechat group, and we will contact you soon.

我参考后改的eval_api_baidu.py文件，但是还是不能运行，一直卡在11/30 10:26:50 - OpenCompass - INFO - Partitioned into 1 tasks.感谢解答
0%| | 0/1 [00:00<?, ?it/s]
11/30 10:26:50 - OpenCompass - INFO - All the logs and processes for each task should be checked in each infer/.out file.
Launch OpenICLInfer[erniebot/openai_humaneval] on CPU
0%| | 0/1 [00:00<?, ?it/s]卡在这里不动了
from mmengine.config import read_base
from opencompass.models import ERNIEBot
from opencompass.partitioners import NaivePartitioner
from opencompass.runners.local_api import LocalAPIRunner
from opencompass.tasks import OpenICLInferTask

with read_base():
    from ..summarizers.medium import summarizer
    from ..datasets.humaneval.humaneval_gen import   humaneval_datasets

datasets = [
    *humaneval_datasets,
]

models = [
    dict(
        abbr='erniebot',
        type=ERNIEBot,
        path='erniebot',
        key='erN0rhEMUSLTDOk1xKV2MC6R',  # please give you key
        secretkey='HPXg3WG7Z6A*****IQ5Vq407c0PQ9o',  # please give your group_id
        url='https://aip.baidubce.com/oauth/2.0/token',
        query_per_second=1,
        max_out_len=2048,
        max_seq_len=2048,
        batch_size=8),
]

infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(
        type=LocalAPIRunner,
        max_num_workers=2,
        concurrent_users=2,
        task=dict(type=OpenICLInferTask)),
)

work_dir = "/output_wenxin/"
Repository: open-compass/opencompass
Issue: [Feature] Support for Llama-2-7b-chat-hf
Root cause: Unimplemented Feature Gap

### 描述该功能

Add support for [Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf).

This is a very popular branch and should be supported for standardized evaluation.

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Please follow `https://github.com/open-compass/opencompass/blob/main/configs/models/llama/llama2_7b_chat.py`, use original repo will be much faster

 Feel free to re-open if needed
Repository: open-compass/opencompass
Issue: [Feature] Maybe some quantized and/or pruned models can be added?
Root cause: Unimplemented Feature Gap

### Describe the feature

Maybe add some quantized and/or pruned models to see the effects of different quantization and pruning levels on the output?
For example Llama2-13b vs the same quantized to int8, etc.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

If the model is defined with transformers format, OpenCompass can be used directly.
Repository: open-compass/opencompass
Issue: 为何看不到每一个数据集的平均分值？
Root cause: Documentation Deficiency

### Describe the feature

运行评估测试
python run.py configs/test_internlm.py -w outputs/001
cat configs/test_internlm.py 
from mmengine.config import read_base

with read_base():
    from .datasets.ceval.ceval_gen import ceval_datasets
    from .datasets.mmlu.mmlu_ppl import mmlu_datasets


datasets = [*ceval_datasets, *mmlu_datasets]

from opencompass.models import HuggingFaceCausalLM


_meta_template = dict(
    round=[
        dict(role='HUMAN', begin='
:', end='<eoh>\n'),
        dict(role='BOT', begin='
:', end='<eoa>\n', generate=True),
    ],
)

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='internlm-chat-7b-8k-hf',
        path="/media/root/sdc/data/model/internlm-chat-7b-8k",
        tokenizer_path='/media/root/sdc/data/model/internlm-chat-7b-8k',
        tokenizer_kwargs=dict(
            padding_side='left',
            truncation_side='left',
            use_fast=False,
            trust_remote_code=True,
        ),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=8,
        meta_template=_meta_template,
        model_kwargs=dict(trust_remote_code=True, device_map='auto'),
        run_cfg=dict(num_gpus=1, num_procs=1),
    )
]
然后获得评测结果
测试结果里面看不到ceval或者mmlu的测试平均值，需要如何设置，还是需要自己实现代码

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

The relevant logic has been implemented in summarizer, but needs an extra line to import the configuration.

You can modify your config, and just put this line within the block of `read_base()`:

https://github.com/InternLM/opencompass/blob/e19a0c1cf8f72a25194151e9c32193ac72bdcaa1/configs/eval_internlm_7b.py#L9

Then summarize the results by using `-m viz`. Don't forget `-r` to reuse the evaluation results obtained before:

```bash
python run.py YOUR_CONFIG -r [TIMESTAMP] -m viz
```

非常感谢！

好烦，只能一次转一个模型的结果，当通过opencompass 生成多个模型的结果时，它转不了
Repository: open-compass/opencompass
Issue: [Feature] Subjective evaluation with single answer grading
Root cause: Unimplemented Feature Gap

### Describe the feature

The paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" proposed two subjective evaluation methods: 
1. pairwise comparison (An LLM judge is given two answers from two models to determine which one is better.)
2. Single answer grading (An LLM judge is asked to directly assign a score to a single answer.)

Any plans on support the single answer grading method? I think it may not be as accurate as pairwise comparison, but it would be more convenient and time-saving for evaluation. 
Thanks.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

The new version contains this two features is coming soon

#685 We have supported Subjective Evaluation. 
Doc: https://opencompass.readthedocs.io/en/latest/advanced_guides/subjective_evaluation.html
Repository: open-compass/opencompass
Issue: [Feature] Support InfiniteBench 
Root cause: Unimplemented Feature Gap

### InfiniteBench

OpenBMB has released a new long-context benchmark: https://github.com/OpenBMB/InfiniteBench

Some of its tasks such as `Retrieve.PassKey1`, `Retrieve.Number `, `Retrieve.KV2` has been used in many papers recently.
 
Task Name | Context | # Examples | Avg Input Tokens | Avg Output Tokens | Description
-- | -- | -- | -- | -- | --
En.Sum | Fake Book | 103 | 171.5k | 1.1k | Summarization of a fake book created with core entity substitution.
En.QA | Fake Book | 351 | 192.6k | 4.8 | Free-form question answering based on the fake book.
En.MC | Fake Book | 229 | 184.4k | 5.3 | Multiple choice questions derived from the fake book.
En.Dia | Script | 200 | 103.6k | 3.4 | Identification of talkers in partially anonymized scripts.
Zh.QA | New Book | 175 | 2068.6k | 6.3 | Question answering on a set of newly collected books.
Code.Debug | Code Document | 394 | 114.7k | 4.8 | Finding which function in a code repo contains an crashing error (in multiple choice form).
Code.Run | Synthetic | 400 | 75.2k | 1.3 | Simulating execution of multiple simple, synthetic functions.
Math.Calc | Synthetic | 50 | 43.9k | 43.9k | Calculations involving super-long arithmetic equations.
Math.Find | Synthetic | 350 | 87.9k | 1.3 | Finding special integers in a lengthy list.
Retrieve.PassKey1 | Synthetic | 590 | 122.4k | 2.0 | Retrieving hidden keys in a noisy long context.
Retrieve.Number | Synthetic | 590 | 122.4k | 4.0 | Locating repeated hidden numbers in a noisy long context.
Retrieve.KV2 | Synthetic | 500 | 89.9k | 22.7 | Finding the corresponding value from a dictionary and a key.



### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

#739