Repository: open-compass/opencompass
Issue: [Bug]  LMEvaluator.__init__() missing 2 required positional arguments while evaluating flame datasets
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/workspace/opencompass/utils/collect_env.py", line 9, in collect_env
    env_info = collect_base_env()
  File "/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/collect_env.py", line 163, in collect_env
    env_info['OpenCV'] = cv2.__version__
AttributeError: module 'cv2' has no attribute '__version__'

And result of [pip list | grep opencv]
opencv-contrib-python     4.10.0.84
opencv-python             4.10.0.84
opencv-python-headless    4.10.0.84

### 重现问题 - 代码/配置示例

python run.py \
--hf-type base --hf-path /space/llms/Qwen2-1.5B-Instruct \
--datasets flames_gen \
--debug

### 重现问题 - 命令或脚本

python run.py \
--hf-type base --hf-path /space/llms/Qwen2-1.5B-Instruct \
--datasets flames_gen \
--debug

### 重现问题 - 错误信息

![微信图片_20241009105447](https://github.com/user-attachments/assets/b81654aa-65e7-46ab-80d1-5e9db7cfc2da)


### 其他信息

_No response_

For subjective evaluation, since there are too many settings, please use config format to start your evaluation, refer to https://github.com/open-compass/opencompass/blob/main/configs/eval_subjective.py

> For subjective evaluation, since there are too many settings, please use config format to start your evaluation, refer to https://github.com/open-compass/opencompass/blob/main/configs/eval_subjective.py

I don't understand what need to do with this *.py script.  Shall I copy and edit `models` / `datasets` / `infer` / `judge_models` / `eval` .etc parameters? And then run the new script instead of `run.py`?

浏览了 discuss 中关于主观评测的讨论，对于主观评测，我有两个问题想补充询问下：
1、如果我想使用本地自部署的qwen-llm服务作为judge_models，使用API的方式该如何配置；
2、我在使用
```bash
python run.py \
--hf-type base --hf-path /space/llms/Qwen2-1.5B-Instruct \
--datasets flames_gen \
-w outputs/demo \
--debug 
```
和
```bash
python run.py configs/eval_flame_demo.py \
-w outputs/demo \
--debug
```
的启动方式后，outputs/demo/*/predictions/*中的推理结果全都为 !!!!!!!!!!!! (感叹号)，这个可能原因是我的什么配置出错嘛

你可以展示一下你的eval_flame_demo.py是怎么写的

> 你可以展示一下你的eval_flame_demo.py是怎么写的

```python
from mmengine.config import read_base

with read_base():
    # from opencompass.configs.models.qwen.hf_qwen2_1_5b_instruct import models
    from opencompass.models.huggingface_above_v4_33 import HuggingFacewithChatTemplate
    from opencompass.configs.datasets.flames.flames_gen import flames_datasets
from opencompass.models.openai_api import OpenAI
from opencompass.partitioners import NaivePartitioner, SizePartitioner
from opencompass.partitioners.sub_naive import SubjectiveNaivePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask
from opencompass.tasks.subjective_eval import SubjectiveEvalTask
from opencompass.summarizers import SubjectiveSummarizer


api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ]
)

datasets = flames_datasets

models = [
    dict(
        type=HuggingFacewithChatTemplate,
        abbr='qwen2-1.5b-instruct-hf',
        path='/space/llms/Qwen2-1.5B-Instruct',
        generation_kwargs=dict(
            do_sample=True, #For subjective evaluation, we suggest you do set do_sample when running model inference!
        ),
        max_out_len=1024,
        batch_size=8,
        run_cfg=dict(num_gpus=1),
    )
]

infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=OpenICLInferTask)),
)

## ------------- JudgeLLM Configuration
judge_models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4-1106-preview',
    key='xxxx',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=2048,
    max_seq_len=2048,
    batch_size=8,
    temperature=0,
)]

## ------------- Evaluation Configuration
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)

summarizer = dict(type=SubjectiveSummarizer, function='subjective')
work_dir = 'outputs/demo/'
```

看起来没有什么问题，`predictions/*中的推理结果全都为 !!!!!!!!!!!! (感叹号)`是指什么呢，可以展示一下现象吗


然后judge model你要换成Qwen的API的话参考这个即可https://github.com/open-compass/opencompass/discussions/1575

> 看起来没有什么问题，`predictions/*中的推理结果全都为 !!!!!!!!!!!! (感叹号)`是指什么呢，可以展示一下现象吗

![微信图片_20241011171841](https://github.com/user-attachments/assets/b20cbe3e-a261-4b5a-8970-ab3d55f0520a)


噢应该是对话模板使用错了

https://github.com/open-compass/opencompass/blob/main/opencompass/configs/models/qwen/hf_qwen2_1_5b_instruct.py 你可以看这里的models的写法是这样的
```
from opencompass.models import HuggingFacewithChatTemplate

models = [
    dict(
        type=HuggingFacewithChatTemplate,
        abbr='qwen2-1.5b-instruct-hf',
        path='Qwen/Qwen2-1.5B-Instruct',
        max_out_len=1024,
        batch_size=8,
        run_cfg=dict(num_gpus=1),
    )
]
```
你可以先尝试换成from opencompass.models import HuggingFacewithChatTemplate试试

此外，Flames这个数据集是否所有的输出都是！呢

> https://github.com/open-compass/opencompass/blob/main/opencompass/configs/models/qwen/hf_qwen2_1_5b_instruct.py 你可以看这里的models的写法是这样的
> 
> ```
> from opencompass.models import HuggingFacewithChatTemplate
> 
> models = [
>     dict(
>         type=HuggingFacewithChatTemplate,
>         abbr='qwen2-1.5b-instruct-hf',
>         path='Qwen/Qwen2-1.5B-Instruct',
>         max_out_len=1024,
>         batch_size=8,
>         run_cfg=dict(num_gpus=1),
>     )
> ]
> ```

我复制过来的，但是eval_subjective.py标明了要设置`do_sample=True`，同时在未设置前，两种任务启动命令都会有 `do_sample=False` 和 `top_p=0.8` 的warning，所以我特意添加上了 `do_sample=True`这个配置

> 此外，Flames这个数据集是否所有的输出都是！呢

早先的测试结果是的，最新的测试我缩减了数据到10条，观测下来也是的

那就有些奇怪了，你换一个model试试呢，看是否还是会出现这样的情况

对于 eval_subjective.py 中的 infer/eval 变量，是否必须设置 `partitioner` 参数，该参数作何解释，我在设置后报错 `keyerror` - 'mode' , 但是注释掉 `partitioner` ，报错 `keyerror` - `partitioner` ……

partitioner是用来分片的，可以加速整个数据集的推理。 具体的报错信息是啥呢，如果是最新版的opencompass不应该有这个问题，你目前是pip install的最新的opencompass吗，按照你提供的config
```
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)

```
这样应该是不会报错的

> partitioner是用来分片的，可以加速整个数据集的推理。 具体的报错信息是啥呢，如果是最新版的opencompass不应该有这个问题，你目前是pip install的最新的opencompass吗，按照你提供的config
> 
> ```
> eval = dict(
>     partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
>     runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
> )
> ```
> 
> 这样应该是不会报错的


版本 0.3.2
```bash
/workspace# python run.py configs/eval_flame_demo.py --debug
10/11 08:45:25 - OpenCompass - INFO - Current exp folder: outputs/demo/20241011_084525
10/11 08:45:25 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
10/11 08:45:25 - OpenCompass - INFO - Partitioned into 1 tasks.
10/11 08:45:26 - OpenCompass - INFO - Task [qwen2-1.5b-instruct-hf/fairness]
10/11 08:45:37 - OpenCompass - INFO - using stop words: ['
']
10/11 08:45:37 - OpenCompass - INFO - Start inferencing [qwen2-1.5b-instruct-hf/fairness]
[2024-10-11 08:45:37,532] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
  0%|                                                                                                                                                               | 0/2 [00:00<?, ?it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [02:54<00:00, 87.04s/it]
Traceback (most recent call last):
  File "/workspace/run.py", line 4, in <module>
    main()
  File "/workspace/opencompass/cli/main.py", line 344, in main
    tasks = partitioner(cfg)
  File "/workspace/opencompass/partitioners/base.py", line 83, in __call__
    tasks = self.partition(**model_and_dataset_args,
  File "/workspace/opencompass/partitioners/sub_naive.py", line 190, in partition
    mode = dataset['mode']
  File "/usr/local/lib/python3.10/dist-packages/mmengine/config/config.py", line 138, in __getitem__
    return self.build_lazy(super().__getitem__(key))
  File "/usr/local/lib/python3.10/dist-packages/mmengine/config/config.py", line 105, in __missing__
    raise KeyError(name)
KeyError: 'mode'
```

现在是进入了eval阶段对吧，你的infer阶段都没问题了吗


> 现在是进入了eval阶段对吧，你的infer阶段都没问题了吗

infer阶段主要是推理结果感叹号，我会尝试别的系列模型确认下，eval阶段此前的问题是这个issue标题的那个报错，现在的这个keyerror我不确定是哪个阶段

我知道你的这个问题了，Flames这个数据集应该是没有随着其他主观数据集一起更新一些配置，我fix一下，你试一下https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/subjective
这里面的其他主观数据集都是随着一起更新了的，FLAMes当时被人放在外面了

> 我知道你的这个问题了，Flames这个数据集应该是没有随着其他主观数据集一起更新一些配置，我fix一下，你试一下https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/subjective 这里面的其他主观数据集都是随着一起更新了的，FLAMes当时被人放在外面了

目前我这边对别的数据集任务不太关注，我可以测试一下，关于flames的这个fix要多久，下个大版本再合并，还是我可以提前使用你的commit，然后手动修改本地代码测试一下

明天应该就能fix


> 明天应该就能fix

感谢，这个数据集今后是否会同步更新，我被这个坑的有点久，有点担心下次配置变更被遗漏……

我今天晚些时候更新一下，之后把它放到主观数据集里吧，以后更新的时候就不会遗漏了

prediction:
![图片](https://github.com/user-attachments/assets/7569e7af-d91e-4ee3-a498-e9d801d1a1e0)
results:
![图片](https://github.com/user-attachments/assets/640221fe-d5c9-4ccc-804b-2ea6fa0b651f)
我这边没问题了，用的下面的config：
```
from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.subjective.flames.flames_gen import flames_datasets
    from opencompass.configs.models.qwen2_5.lmdeploy_qwen2_5_1_5b_instruct import models as lmdeploy_qwen2_5_1_5b_instruct 
from opencompass.models import HuggingFaceCausalLM, HuggingFace, HuggingFaceChatGLM3, OpenAI, TurboMindModelwithChatTemplate
from opencompass.partitioners import NaivePartitioner, SizePartitioner
from opencompass.partitioners.sub_naive import SubjectiveNaivePartitioner
from opencompass.partitioners.sub_size import SubjectiveSizePartitioner
from opencompass.partitioners.sub_num_worker import SubjectiveNumWorkerPartitioner
from opencompass.runners import LocalRunner
from opencompass.runners import SlurmSequentialRunner
from opencompass.tasks import OpenICLInferTask
from opencompass.tasks.subjective_eval import SubjectiveEvalTask
from opencompass.summarizers import FlamesSummarizer

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ]
)

# -------------Inference Stage ----------------------------------------
# For subjective evaluation, we often set do sample for models
models = [*lmdeploy_qwen2_5_1_5b_instruct]


datasets = [*flames_datasets] # add datasets you want


infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=OpenICLInferTask)),
)
# -------------Evalation Stage ----------------------------------------

## ------------- JudgeLLM Configuration
judge_models = []

## ------------- Evaluation Configuration
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)

summarizer = dict(type=FlamesSummarizer)
work_dir = 'outputs/subjective/'

```
你把这个config里的models和judgemodels改成你需要的即可。
PR：https://github.com/open-compass/opencompass/pull/1599

> prediction: ![图片](https://private-user-images.githubusercontent.com/148421775/375929321-7569e7af-d91e-4ee3-a498-e9d801d1a1e0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjg3MTM0NTAsIm5iZiI6MTcyODcxMzE1MCwicGF0aCI6Ii8xNDg0MjE3NzUvMzc1OTI5MzIxLTc1NjllN2FmLWQ5MWUtNGVlMy1hNDk4LWU5ZDgwMWQxYTFlMC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDEyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAxMlQwNjA1NTBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lNDQ1YzU4OTg5ZDI4NjBhYzVjNGVjOTIxMTQyZWFiM2JkMzQzMWE1M2Y0Mzg2M2UyM2RlYmNjNjc5OTI5ODE3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.TtnIE2cxkAqHlKPBxU4MrWMdijRZyyeZ8iMMYQV6KqE) results: ![图片](https://private-user-images.githubusercontent.com/148421775/375929334-640221fe-d5c9-4ccc-804b-2ea6fa0b651f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjg3MTM0NTAsIm5iZiI6MTcyODcxMzE1MCwicGF0aCI6Ii8xNDg0MjE3NzUvMzc1OTI5MzM0LTY0MDIyMWZlLWQ1YzktNGNjYy04MDRiLTJlYTZmYTBiNjUxZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDEyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAxMlQwNjA1NTBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lZDY5Nzg2NjRkODMzODY3YmVhZjg2MzBkOWZkZmVjOGU2NjU5ODM4Yjc5MGU1Njc0MjUwOTIwOTE1ZDk4MzQ3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.0eWjU5OvNncNdEucJfE1D7LDXvdYMDZaUZxvYbNob5E) 我这边没问题了，用的下面的config：
> 
> ```
> from mmengine.config import read_base
> 
> with read_base():
>     from opencompass.configs.datasets.subjective.flames.flames_gen import flames_datasets
>     from opencompass.configs.models.qwen2_5.lmdeploy_qwen2_5_1_5b_instruct import models as lmdeploy_qwen2_5_1_5b_instruct 
> from opencompass.models import HuggingFaceCausalLM, HuggingFace, HuggingFaceChatGLM3, OpenAI, TurboMindModelwithChatTemplate
> from opencompass.partitioners import NaivePartitioner, SizePartitioner
> from opencompass.partitioners.sub_naive import SubjectiveNaivePartitioner
> from opencompass.partitioners.sub_size import SubjectiveSizePartitioner
> from opencompass.partitioners.sub_num_worker import SubjectiveNumWorkerPartitioner
> from opencompass.runners import LocalRunner
> from opencompass.runners import SlurmSequentialRunner
> from opencompass.tasks import OpenICLInferTask
> from opencompass.tasks.subjective_eval import SubjectiveEvalTask
> from opencompass.summarizers import FlamesSummarizer
> 
> api_meta_template = dict(
>     round=[
>         dict(role='HUMAN', api_role='HUMAN'),
>         dict(role='BOT', api_role='BOT', generate=True),
>     ]
> )
> 
> # -------------Inference Stage ----------------------------------------
> # For subjective evaluation, we often set do sample for models
> models = [*lmdeploy_qwen2_5_1_5b_instruct]
> 
> 
> datasets = [*flames_datasets] # add datasets you want
> 
> 
> infer = dict(
>     partitioner=dict(type=NaivePartitioner),
>     runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=OpenICLInferTask)),
> )
> # -------------Evalation Stage ----------------------------------------
> 
> ## ------------- JudgeLLM Configuration
> judge_models = []
> 
> ## ------------- Evaluation Configuration
> eval = dict(
>     partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
>     runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
> )
> 
> summarizer = dict(type=FlamesSummarizer)
> work_dir = 'outputs/subjective/'
> ```
> 
> 你把这个config里的models和judgemodels改成你需要的即可。 PR：#1599

感谢，我测试一下