zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug] I downloaded the Llama-7b model on the huggingface to the local location, and then modified the path to the file location. After running the code, the following error occurred. How can I solve it?
Root cause: Environment Incompatibility

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python

### 重现问题 - 代码/配置示例

python run.py --datasets ceval_ppl --hf-path /root/pruning/llama-7b  --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs device_map='auto' --max-seq-len 2048 --max-out-len 100 --batch-size 64 --num-gpus 1

### 重现问题 - 命令或脚本

python run.py --datasets ceval_ppl --hf-path /root/pruning/llama-7b --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs device_map='auto' --max-seq-len 2048 --max-out-len 100 --batch-size 1--num-gpus 1

### 重现问题 - 错误信息

![WX20231025-103210@2x](https://github.com/open-compass/opencompass/assets/148928413/5d62a2a7-6b2e-488a-8274-a8d8cf84d021)


### 其他信息

_No response_

Please show us the content of `outputs/blabla/logs/infer/blabla/blabla.out` and `outputs/blabla/logs/eval/blabla/blabla.out`

GPU: A100 40GB
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 626117) of binary: /root/.local/conda/envs/pytorch/bin/python
Traceback (most recent call last):
  File "/root/.local/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/root/.local/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/root/.local/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/root/.local/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/.local/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

I have the same problem (OpenICLEval fail), can you solve it?

> Please show us the content of `outputs/blabla/logs/infer/blabla/blabla.out` and `outputs/blabla/logs/eval/blabla/blabla.out`

'torchrun' 不是内部或外部命令，也不是可运行的程序或批处理文件。

同问，我的报错是/bin/sh: torchrun: command not found，查了下说torch1.9.1之后就支持torchrun了，我是torch2.0.1，还是报错
Repository: open-compass/opencompass
Issue: [Feature] 多模态榜单
Root cause: Documentation Deficiency

### Describe the feature

之前多模态榜单上好像有好几个bench一起评测(类似LLM榜单上好几个数据集综合评测)，现在只有MMBench了。请问其他数据集评测是迁移到哪里了吗？有没有可能恢复回来呀？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Hi, @lj163ucas , 
We are still evaluating those VLMs on MME and SEEDBench, the results will be available in 1 / 2 weeks. The data on the previous page are just mock data.

赞，多谢

Is it going well?

Hi, @lj163ucas , the evaluation of those benchmarks are now supported in [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) (including the evaluation results). 
Besides, we would add a multi-modal leaderboard in the following several days to our official website. 


https://opencompass.readthedocs.io/zh-cn/latest/advanced_guides/multimodal_eval.html中的多模态评测使用的是opencompass中的python run.py configs/multimodal/tasks.py --mm-eval，这部分支持测试吗？目前测试报错，榜单中提到使用的是VLMEvalKit

> [https://opencompass.readthedocs.io/zh-cn/latest/advanced_guides/multimodal_eval.html中的多模态评测使用的是opencompass中的python](https://opencompass.readthedocs.io/zh-cn/latest/advanced_guides/multimodal_eval.html%E4%B8%AD%E7%9A%84%E5%A4%9A%E6%A8%A1%E6%80%81%E8%AF%84%E6%B5%8B%E4%BD%BF%E7%94%A8%E7%9A%84%E6%98%AFopencompass%E4%B8%AD%E7%9A%84python) run.py configs/multimodal/tasks.py --mm-eval，这部分支持测试吗？目前测试报错，榜单中提到使用的是VLMEvalKit

Please try VLMEvalKit, evaluation for VLM has been deprecated in opencompass repo
Repository: open-compass/opencompass
Issue: [Feature] Add support for TensorRT-LLM inference engine
Root cause: Unimplemented Feature Gap

### Describe the feature

Hi guys,

The TensorRT-LLM has been released last week. It was maintained by NVIDIA with high inference performance.
Link: https://github.com/NVIDIA/TensorRT-LLM

Will implement it by API calling or just integrate it into inference pipeline just like huggingface inference method? Which method is better?


Thanks

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Bug] I downloaded the llama-7b model on Huggingface locally, and the inference was normal, but an error was reported during the evaluation on CEval
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

python3.8

### Reproduces the problem - code/configuration sample

python run.py --datasets ceval_ppl --hf-path '/root/pruning/llama-7b' --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs device_map='auto' --max-seq-len 2048 --max-out-len 100 --batch-size 1  --num-gpus 1

### Reproduces the problem - command or script

python run.py --datasets ceval_ppl --hf-path '/root/pruning/llama-7b' --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs device_map='auto' --max-seq-len 2048 --max-out-len 100 --batch-size 1  --num-gpus 1

### Reproduces the problem - error message

<img width="306" alt="image" src="https://github.com/open-compass/opencompass/assets/148928413/95e9d6c9-64a5-46a1-ba59-6b35148e1316">

ERROR:torch.distributed.elastic.multiprocessing.api:failed

<img width="1148" alt="image" src="https://github.com/open-compass/opencompass/assets/148928413/f9d23a36-0890-4ae2-8b27-be3bb1407a7b">


### Other information

_No response_

To be honest, I am not sure what is going on here. I have not encountered this error before. But here are some possible trials:

1. Try to pull the latest code, as we merged #522 
2. Try to run with `python3 -c "import torch; print(torch.cuda.is_available())"` to check cuda in ready.
3. Try to run with `--debug` option to show more debug information.

```bash
python run.py --datasets ceval_ppl --hf-path '/root/pruning/llama-7b' --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs device_map='auto' --max-seq-len 2048 --max-out-len 100 --batch-size 1 --num-gpus 1 --debug
```


Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: MMBENCH: Script to evaluate dev-split locally 
Root cause: Documentation Deficiency

### Describe the feature


Thanks for the awesome repo. I am just wondering whether there is any script [with CircularEval strategy + choice extraction using ChatGPT] to evaluate our method locally (without submitting to the leaderboard) on the dev split (I see that the file mmbench_dev_20230712.tsv contains answers). Thanks in Advance.


### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Hi, @dmlpt , the script is available at https://github.com/open-compass/opencompass/blob/main/tools/eval_mmbench.py

Hi @kennymckormick , Thanks for the response! I will try that command.

Feel free to re-open if needed.

> Hi, @dmlpt , the script is available at https://github.com/open-compass/opencompass/blob/main/tools/eval_mmbench.py

may I use this script to make evaluation on test-split locally?
Repository: open-compass/opencompass
Issue: [Feature]  ChatGLM-Turbo API 的 ceval 效果评估支持
Root cause: Documentation Deficiency

### 描述该功能

ChatGLM-Turbo API，像这种新发布的 API，应该怎么接入 opencompass 进行评测呢？

### 是否希望自己实现该功能？

- [X] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Hi, please refer to https://opencompass.readthedocs.io/zh-cn/latest/advanced_guides/new_model.html for more details.
Also you can check the implementation of OpenAI. https://github.com/open-compass/opencompass/blob/main/opencompass/models/openai_api.py

We also support ZhipuAI, maybe you can modify this file https://github.com/open-compass/opencompass/blob/main/opencompass/models/zhipuai.py

> We also support ZhipuAI, maybe you can modify this file https://github.com/open-compass/opencompass/blob/main/opencompass/models/zhipuai.py

thanks, I'll see it.
Repository: open-compass/opencompass
Issue: [Feature] Evaluation of Skywork
Root cause: Unimplemented Feature Gap

### Describe the feature

Evaluation of Skywork

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Updated in our website, plz check:
https://opencompass.org.cn/leaderboard-llm

![image](https://github.com/open-compass/opencompass/assets/17680578/46c2e7f7-cbb2-428d-82cc-5e806c1eeb6b)
Repository: open-compass/opencompass
Issue: [Feature] 'Add_special_tokens' parameter is needed
Root cause: Unimplemented Feature Gap

### Describe the feature

As the base models (pre-trained models without sft) do not predict after the EOS token during the pre-train stage, setting the `add_special_tokens=True` by default is not the right way to evaluate these models' performance.
So adding a parameter for this would be good?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Good one, but we did not conduct each experiment to compare the differences, it is better that you have some evaluation results that substantiate your issue.
BTW, you are welcomed to implement this feature to support `add_special_tokens` option in model.

Feel free to reopen it if needed
Repository: open-compass/opencompass
Issue: setting max_out_len only works in Inferencer[Bug] 
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

'Python': '3.10.12 (main, Jul  5 2023, 18:54:27) [GCC 11.2.0]',
 'TorchVision': '0.15.2',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.1.2+',
 'sys.platform': 'linux'}

### Reproduces the problem - code/configuration sample

python run.py configs/eval_gpt4.py

### Reproduces the problem - command or script

python run.py configs/eval_gpt4.py

### Reproduces the problem - error message

    and torch.sum(inputs_tensor[:, -1] == generation_config.pad_token_id) > 0
IndexError: index -1 is out of bounds for dimension 1 with size 0

### Other information

I've noticed that the max_out_len parameter can be set in both model config (models dict )and dataset config(inferencer) but only works in dataset config.

Setting max_out_len equal to max_seq_len will make input_ids empty and result in IndexError.
Repository: open-compass/opencompass
Issue: [Feature] Support HotpotQA dataset
Root cause: Unimplemented Feature Gap

### Describe the feature

- https://hotpotqa.github.io/
- https://arxiv.org/pdf/1809.09600.pdf
- https://arxiv.org/pdf/2304.10513.pdf

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] Support datasets in OpenAI/evals
Root cause: Unimplemented Feature Gap

### Describe the feature

[OpenAI/evals](https://github.com/openai/evals/tree/main/evals/registry/data) contains many community-sourced tasks, with about 400-500 datasets covering various languages and fields. Compared to other open-source datasets, they have very few questions, but many tasks come from users' real needs and inspiration when using GPT4, and many datasets give people a bright feeling.

 Is there a way to directly use their datasets through a certain mechanism?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Great idea.

> ### Describe the feature
> [OpenAI/evals](https://github.com/openai/evals/tree/main/evals/registry/data) contains many community-sourced tasks, with about 400-500 datasets covering various languages and fields. Compared to other open-source datasets, they have very few questions, but many tasks come from users' real needs and inspiration when using GPT4, and many datasets give people a bright feeling.
> 
> Is there a way to directly use their datasets through a certain mechanism?
> 
> ### Will you implement it?
> * [ ]  I would like to implement this feature and create a PR!

Hi, has any progress on this thread?
Repository: open-compass/opencompass
Issue: [Bug] 无法指定gpu
Root cause: Configuration Error

### 描述该错误

无法指定gpu，无论在shell中使用CUDA_VISIBLE_DEVICES=1，还是在python中使用os.environ["CUDA_VISIBLE_DEVICES"]=1，都无法让程序只占用1卡，程序永远都会优先占用0卡。

### 环境信息

shell中

CUDA_VISIBLE_DEVICES=1 python run.py configs/eval_llama_7b.py -w outputs/llama_7b --gpu_idx=1


python中

parser = argparse.ArgumentParser(description='Run an evaluation task')
parser.add_argument('--gpu_idx', type=str, default=None)
args = parser.parse_args()
os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu_idx

### 其他信息

_No response_

将opencompass/opencompass/runners/local.py文件(A位置)中的B区域代码注释，并添加A代码，即可。

![image](https://github.com/InternLM/opencompass/assets/139195155/6e40336a-e7a0-4379-9bd1-a9eaa36dc969)


PR is wlecomed for improving OpenCompass. Thanks in advance.
Repository: open-compass/opencompass
Issue: [Feature] how to add system prompt to _meta_template?
Root cause: Documentation Deficiency

### Describe the feature

```
_meta_template = dict(
    round=[
        dict(role="HUMAN", begin='\n
user\n', end='
'),
        dict(role="BOT", begin="\n
assistant\n", end='
', generate=True),
    ],
)
```
how to add system prompt to _meta_template?like the below:
```
_meta_template = dict(
    begin=[
        dict(role='SYSTEM', fallback_role='HUMAN', begin='Below is an instruction that describes a task. Write a response that appropriately completes the request.\n'),
    ],
    round=[
        dict(role="HUMAN", begin='\n
user\n', end='
'),
        dict(role="BOT", begin="\n
assistant\n", end='
', generate=True),
    ],
)

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='yi-6b',
        path=path,
        tokenizer_path="01-ai/Yi-6B",
        tokenizer_kwargs=dict(padding_side='left',
                              truncation_side='left',
                              trust_remote_code=True,
                              use_fast=False,
                              ),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=8,
        meta_template=_meta_template,
        model_kwargs=dict(device_map='auto', trust_remote_code=True),
        batch_padding=False, # if false, inference with for-loop without batch padding
        run_cfg=dict(num_gpus=1, num_procs=1),
    )
]
```

### Will you implement it?

- [X] I would like to implement this feature and create a PR!

I see. Read the doc! https://github.com/open-compass/opencompass/blob/main/docs/zh_cn/prompt/meta_template.md
Repository: open-compass/opencompass
Issue: python 3.8.13 加载config时候报错
Root cause: Validation Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

同官方

### 重现问题 - 代码/配置示例

# 修改了task.py，希望评估qwen-vl-chat
'''
from mmengine.config import read_base

with read_base():
    from .qwen.qwenvl_chat_7b_mmbench_cn import (qwen_mmbench_dataloader,
                                                 qwen_mmbench_evaluator,
                                                 qwen_model)

models = [qwen_model]
datasets = [qwen_mmbench_dataloader]
evaluators = [qwen_mmbench_evaluator]

num_gpus = 1
num_procs = 1
launcher = 'pytorch'

'''

### 重现问题 - 命令或脚本

python configs/multimodal/tasks.py --mm-eval

### 重现问题 - 错误信息

Traceback (most recent call last):
  File "/checkpoint/binary/train_package/run.py", line 342, in <module>
    main()
  File "/checkpoint/binary/train_package/run.py", line 242, in main
    cfg = Config.fromfile(output_config_path, format_python_code=False)
  File "/root/.local/lib/python3.8/site-packages/mmengine/config/config.py", line 456, in fromfile
    lazy_import is None and not Config._is_lazy_import(filename):
  File "/root/.local/lib/python3.8/site-packages/mmengine/config/config.py", line 1657, in _is_lazy_import
    parsed_codes = ast.parse(codes_str)
  File "/opt/conda/envs/python3.8.13/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 54
    type=<class 'opencompass.multimodal.models.qwen.prompt_constructor.QwenVLMMBenchPromptConstructor'>),
         ^
SyntaxError: invalid syntax

### 其他信息

看了之前一个同学的相同问题，也是python3.8.13，和python版本有关么？

相同问题链接，但不知道如何解决的：https://github.com/open-compass/opencompass/issues/225。

I'm not sure about the cause of this bug. A potential way to debug is: 
![image](https://github.com/open-compass/opencompass/assets/34324155/3d2af014-387f-4ae9-99fa-6824089dd136)
This bug happens while trying to load a temporary generated config located at `output_config_path`. You can copy the content of this temp config here, which will be helpful for further debugging. 

> I'm not sure about the cause of this bug. A potential way to debug is: ![image](https://user-images.githubusercontent.com/34324155/281609852-3d2af014-387f-4ae9-99fa-6824089dd136.png) This bug happens while trying to load a temporary generated config located at `output_config_path`. You can copy the content of this temp config here, which will be helpful for further debugging.

好的，感谢解答，我先这么做试试~

Feel free to re-open if needed.

Hi, have you resolved this issue?
I have the same issue. 

使用vscode调试时会出现这个问题，直接用Python跑不会，感觉很奇特

> 使用vscode调试时会出现这个问题，直接用Python跑不会，感觉很奇特

是的，如果要调试的话，需要怎么修改呢，你解决了吗
Repository: open-compass/opencompass
Issue: [Bug] 评测时报错：No module named 'lagent'
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

按照文档里通过最新源码pip3 install -e .

### Reproduces the problem - code/configuration sample

无

### Reproduces the problem - command or script

python run.py configs/eval_demo.py -w outputs/demo --debug

### Reproduces the problem - error message

<img width="1080" alt="image" src="https://github.com/open-compass/opencompass/assets/6971044/05fab433-665d-4a02-a772-f1cfbdfe90aa">


### Other information

_No response_

Thank you for bringing this issue to our attention. We appreciate your help in identifying the bug, and we will promptly address it.


We already fix this bug, please pull the latest main branch to reinstall, thanks for your notification. @frankxyy
Repository: open-compass/opencompass
Issue: [Bug] 根据用户手册安装环境出现pytorch版本过高问题
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

略

### Reproduces the problem - code/configuration sample

略

### Reproduces the problem - command or script

conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y


### Reproduces the problem - error message

/home/shijin/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/amp/autocast mode.py:250:Userwarning: User provided device_type of 'cuda', but CUDA is not available. Disabling


### Other information

根据手册https://opencompass.readthedocs.io/zh-cn/latest/get_started/installation.html的命令conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y准备环境。
安装的pytorch是最新版，2023.11.09安装2.1版本。如果机器的cuda是11.7，则无法使用pytorch，输入torch.cuda.is_available()返回结果为False。
降低pytorch版本降为2.0后，即可使用，输入torch.cuda.is_available()返回结果为True。

Thanks for the reporting. Please follow  the instruction of PyTorch to install it. We will update the documentation for different version of PyTorch

Closed due to no response. Feel free to reopen if needed.
Repository: open-compass/opencompass
Issue: [Bug] eval结果为空
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

main分支最新代码

### Reproduces the problem - code/configuration sample

eval_demo文件内容如下：
```python

from mmengine.config import read_base
from opencompass.models import HuggingFaceCausalLM

with read_base():
    # from .datasets.siqa.siqa_gen import siqa_datasets
    # from .datasets.winograd.winograd_ppl import winograd_datasets
    
    from .datasets.ceval.ceval_gen import ceval_datasets

    # from .models.opt.hf_opt_125m import opt125m
    from .models.opt.hf_opt_350m import opt350m

    from .models.baichuan.hf_baichuan2_7b_chat import models

datasets = [*ceval_datasets]

models = [
    dict(
        type=HuggingFaceCausalLM,
        path='/home/xuyangyang/llama-2-7b-hf',
        model_kwargs=dict(device_map='auto'),
        tokenizer_path='/home/xuyangyang/llama-2-7b-hf',
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
        max_seq_len=2048,
        max_out_len=50,
        run_cfg=dict(num_gpus=2, num_procs=1),
    )
]

```

然后运行 python run.py configs/eval_demo.py -w outputs/demo

### Reproduces the problem - command or script

见上一条

### Reproduces the problem - error message

<img width="1425" alt="image" src="https://github.com/open-compass/opencompass/assets/6971044/3590c15c-6041-4649-b85d-576122af9e1d">


<img width="1106" alt="image" src="https://github.com/open-compass/opencompass/assets/6971044/1c51e76e-7315-4db0-8f51-221d5545951c">

<img width="1131" alt="image" src="https://github.com/open-compass/opencompass/assets/6971044/5dae40ee-2931-4eff-b43b-10a20cf1aaa7">


### Other information

_No response_

看起来是 flash_attn 的安装不成功，请尝试正确安装 flash_attn

明白了，谢谢