Repository: open-compass/opencompass
Issue: [Bug] 自定义judge model进行主观评测，报错模型未被注册
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我修改了代码（配置不视为代码），或者我正在处理我自己的任务/模型/数据集。

### 环境

conda create --name opencompass python=3.10 -y
conda activate opencompass
pip install -U opencompass

### 重现问题 - 代码/配置示例

1、在opencompass/opencompass/models目录下新建mymodel_api.py，自定义API模型（MyModelAPI）
```python
import json
from concurrent.futures import ThreadPoolExecutor
from typing import Dict, List, Optional, Union
import requests
from openai import OpenAI
from opencompass.registry import MODELS
from opencompass.utils.logging import get_logger
from opencompass.utils.prompt import PromptList

from .base_api import BaseAPIModel

PromptType = Union[PromptList, str]

@MODELS.register_module()
class MyModelAPI(BaseAPIModel):
    is_api: bool = True

    def __init__(
            self,
            path: str,
            url: str,
            key: str,
            max_seq_len: int = 2048,
            meta_template: Optional[Dict] = None,
            retry: int = 2,
            generation_kwargs: Optional[Dict] = dict(),
    ):
        super().__init__(
            path=path,
            max_seq_len=max_seq_len,
            meta_template=meta_template,
            retry=retry,
            generation_kwargs=generation_kwargs,
        )
        
        ...

    def generate(self, inputs: List[str], max_out_len: int,
                 **kwargs) -> List[str]:
        """Generate results given a list of inputs."""
        ...
```

2、在opencompass.models下的__init__.py导入mymodel_api.py定义的模型
```python
from .mymodel_api import MyModelAPI
```

3、在config下新建一个评测配置脚本
```python
from mmengine.config import read_base
from opencompass.models.mymodel_api import MyModelAPI
.....

with read_base():
    from .datasets.subjective.alignbench.alignbench_judgeby_critiquellm import alignbench_datasets
datasets = [*alignbench_datasets,]

models = [
    dict(
        abbr='qwen2',
        type=MyModelAPI,
        path='qwen2',
        key='xxxxxxxxxxx',
        url= 'xxxxxxxxxxx',
        generation_kwargs=dict(top_k=1, temperature=1e-6, top_p=0.9),
        batch_size=8,
        run_cfg=dict(num_gpus=1)
        )
]


# -------------Infer Stage ----------------------------------------
infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(
        type=LocalRunner,
        max_num_workers=1,
        concurrent_users=1,
        task=dict(type=OpenICLInferTask)),
)


# -------------Evalation Stage ----------------------------------------

## ------------- JudgeLLM Configuration
judge_models = [
    dict(
        abbr='qwen2',
        type=MyModelAPI,
        path='qwen2',
        key='xxxxxxxxxxx',
        url= 'xxxxxxxxxxx',
        generation_kwargs=dict(top_k=1, temperature=1e-6, top_p=0.9),
        batch_size=8,
        run_cfg=dict(num_gpus=1)
        )
]


## ------------- Evaluation Configuration
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)

summarizer = dict(type=SubjectiveSummarizer, function='subjective')
work_dir = 'outputs/subjective/'
```


### 重现问题 - 命令或脚本

python run.py configs/eval_subjective_test.py --debug

### 重现问题 - 错误信息

11/04 16:26:38 - OpenCompass - INFO - Current exp folder: outputs/subjective/20241104_162638
11/04 16:26:38 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
11/04 16:26:38 - OpenCompass - INFO - Partitioned into 1 tasks.
11/04 16:26:38 - OpenCompass - WARNING - Ignored argument in opencompass.runners.local: concurrent_users=1
11/04 16:26:40 - OpenCompass - WARNING - Only use 1 GPUs for total 8 available GPUs in debug mode.
11/04 16:26:40 - OpenCompass - INFO - Task [qwen2/alignment_bench_sample]
11/04 16:26:40 - OpenCompass - INFO - Start inferencing [qwen2/alignment_bench_sample]
[2024-11-04 16:26:40,611] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-11-04 16:26:40,611] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:30<00:00, 15.06s/it]
11/04 16:27:10 - OpenCompass - INFO - Partitioned into 1 tasks.
11/04 16:27:12 - OpenCompass - WARNING - Debug mode, log will be saved to tmp/1351571_debug.log
outputs/subjective/20241104_162638/results/qwen2_judged-by--qwen2 is not exist! please check!
Your subjective evaluation results have been saved at outputs/subjective/20241104_162638/summary/20241104_162638/Subjective_all_results-judged-by--qwen2.csv

【1351571_debug.log】
11/04 16:27:19 - OpenCompass - INFO - No postprocessor found.
Traceback (most recent call last):
  File "/gpu/user/opencompass/opencompass/tasks/subjective_eval.py", line 451, in <module>
    inferencer.run()
  File "/gpu/user/opencompass/opencompass/tasks/subjective_eval.py", line 94, in run
    self._score(model_cfg, dataset_cfg, eval_cfg, output_column,
  File "/gpu/user/opencompass/opencompass/tasks/subjective_eval.py", line 380, in _score
    icl_evaluator = ICL_EVALUATORS.build(eval_cfg['evaluator'])
  File "/gpu/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/gpu/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/gpu/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/openicl/icl_evaluator/lm_evaluator.py", line 110, in __init__
    model = build_model_from_cfg(model_cfg=judge_cfg)
  File "/gpu/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/utils/build.py", line 25, in build_model_from_cfg
    return MODELS.build(model_cfg)
  File "/gpu/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/gpu/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 100, in build_from_cfg
    raise KeyError(
KeyError: 'opencompass.models.mymodel_api.MyModelAPI is not in the opencompass::model registry. Please check whether the value of `opencompass.models.mymodel_api.MyModelAPI` is correct or it was registered as expected.

### 其他信息

为什么API模型在infer阶段是可以正常推理输出prediction结果的，但是作为judge模型就无法调用输出

确认一下你的opencompass是怎么按照的呢，有执行pip install -e .吗，看你的log里貌似/gpu/user/opencompass/opencompass/tasks/subjective_eval.py在这里执行的，但环境进入到gpu/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine里了

感谢！需要执行pip install -e .后再运行即可