Repository: open-compass/opencompass
Issue: [Bug] 增加新的评测数据集遇到了问题
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.

### Environment

null

### Reproduces the problem - code/configuration sample

null

### Reproduces the problem - command or script

python run.py --datasets mydataset  \
--hf-path /home/Mistral-7B-v0.1 \
--model-kwargs device_map='auto' trust_remote_code=True \
--tokenizer-kwargs padding_side='left' truncation='left' use_fast=False trust_remote_code=True \
--max-out-len 100 \
--max-seq-len 2048 \
--batch-size 8 \
--num-gpus 1 \

### Reproduces the problem - error message

您好，我尝试新添加数据集进行评测，遇到了上述报错。请问如何解决，我的哪个步骤有问题吗
我对项目的修改如下：

1、新建mydataset.py：
import csv
import os.path as osp

from datasets import Dataset, DatasetDict

from opencompass.registry import LOAD_DATASET

from .base import BaseDataset


@LOAD_DATASET.register_module()
class MyDataset(BaseDataset):

    @staticmethod
    def load(path: str, name: str):
        dataset = DatasetDict()
        for split in ['dev', 'test']:
            raw_data = []
            filename = osp.join(path, split, f'{name}.csv')
            with open(filename, encoding='utf-8') as f:
                reader = csv.reader(f)
                _ = next(reader)  # skip the header
                for row in reader:
                    assert len(row) == 7
                    raw_data.append({
                        'question': row[1],
                        'A': row[2],
                        'B': row[3],
                        'C': row[4],
                        'D': row[5],
                        'answer': row[6],
                    })
            dataset[split] = Dataset.from_list(raw_data)
        return dataset

2、在.__init__.py中添加：
from .mydataset import *  # noqa: F401, F403

3、新建./data/mydataset，把我的数据集放在这里



遇到的报错如下：
Traceback (most recent call last):
  File "/home/Project/opencompass/run.py", line 357, in <module>
    main()
  File "/home/Project/opencompass/run.py", line 224, in main
    cfg = get_config_from_arg(args)
  File "/home/Project/opencompass/opencompass/utils/run.py", line 73, in get_config_from_arg
    for dataset in match_cfg_file(datasets_dir, args.datasets):
  File "/home/Project/opencompass/opencompass/utils/run.py", line 47, in match_cfg_file
    raise ValueError(err_msg)
ValueError: The provided pattern matches 0 or more than one config. Please verify your pattern and try again. You may use tools/list_configs.py to list or locate the configurations.
+------------------------+
| Not matched patterns   |
|------------------------|
| mydataset              |

### Other information

_No response_

It seems that you have only defined your Dataset Class, but have not provided a dataset config. 
In 
`python run.py --datasets mydataset`, the mydataset should be the name of a config.
You should import the dataset class to a dataset config, just like this https://github.com/open-compass/opencompass/blob/main/configs/datasets/IFEval/IFEval_gen_3321a3.py 

> It seems that you have only defined your Dataset Class, but have not provided a dataset config. In `python run.py --datasets mydataset`, the mydataset should be the name of a config. You should import the dataset class to a dataset config, just like this https://github.com/open-compass/opencompass/blob/main/configs/datasets/IFEval/IFEval_gen_3321a3.py

I have defined the Dataset Class:
![image](https://github.com/open-compass/opencompass/assets/45348154/432e09ae-d867-4f96-bf45-a456a9315160)

mydataset_ppl.py:
```
from mmengine.config import read_base

with read_base():
    from .mydataset_ppl_8b9c76 import mydataset  # noqa: F401, F403
```

mydataset_ppl_8b9c76.py:
```
from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import FixKRetriever
from opencompass.openicl.icl_inferencer import PPLInferencer
from opencompass.openicl.icl_evaluator import AccEvaluator
from opencompass.datasets import MyDataset
from opencompass.utils.text_postprocessors import first_capital_postprocess

mydataset_subject_mapping = {
    'agronomy': '农学',
    'anatomy': '解剖学'
}


mydataset_all_sets = list(mydataset_subject_mapping.keys())

mydataset = []
for _name in mydataset_all_sets:
    _ch_name = mydataset_subject_mapping[_name]
    mydataset_infer_cfg = dict(
        ice_template=dict(
            type=PromptTemplate,
            template={
                answer: dict(
                    begin="</E>",
                    round=[
                        dict(
                            role="HUMAN",
                            prompt=f"以下是关于{_ch_name}的单项选择题，请直接给出正确答案的选项。\n题目：{{question}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}"
                        ),
                        dict(role="BOT", prompt=f'答案是: {answer}'),
                    ])
                for answer in ["A", "B", "C", "D"]
            },
            ice_token="</E>",
        ),
        retriever=dict(type=FixKRetriever, fix_id_list=[0, 1, 2, 3, 4]),
        inferencer=dict(type=PPLInferencer),
    )

    mydataset_eval_cfg = dict(evaluator=dict(type=AccEvaluator))

    mydataset.append(
        dict(
            type=MyDataset,
            path="./data/mydataset/",
            name=_name,
            abbr=f"mydataset-{_name}",
            reader_cfg=dict(
                input_columns=["question", "A", "B", "C", "D"],
                output_column="answer",
                train_split="dev",
                test_split='test'),
            infer_cfg=mydataset_infer_cfg,
            eval_cfg=mydataset_eval_cfg,
        ))

del _name, _ch_name

```

but it doesn't work  (My demo is choose a part of cmmlu dataset as my dataset)

> MyDataset

Emm, what is the error now

> mydataset_ppl.py

By the way, if your config name is mydataset_ppl.py, you should use --datasets mydataset_ppl

> > mydataset_ppl.py
> 
> By the way, if your config name is mydataset_ppl.py, you should use --datasets mydataset_ppl

I run the script:
```
CUDA_VISIBLE_DEVICES=1 python run.py --datasets mydataset_ppl  \
--hf-path /home/Mistral-7B-v0.1 \
--model-kwargs device_map='auto' trust_remote_code=True \
--tokenizer-kwargs padding_side='left' truncation='left' use_fast=False trust_remote_code=True \
--max-out-len 100 \
--max-seq-len 2048 \
--batch-size 8 \
--num-gpus 1 \
--max-partition-size 10000 \
```

Terminal output:
```
03/17 20:23:09 - OpenCompass - INFO - Loading mydataset_ppl: configs/datasets/mydataset/mydataset_ppl.py
03/17 20:23:09 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
03/17 20:23:09 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
03/17 20:23:09 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners
03/17 20:23:09 - OpenCompass - DEBUG - Get class `SizePartitioner` from "partitioner" registry in "opencompass"
03/17 20:23:09 - OpenCompass - DEBUG - An `SizePartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.size
03/17 20:23:09 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
03/17 20:23:09 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
03/17 20:23:09 - OpenCompass - DEBUG - Additional config: {}
03/17 20:23:09 - OpenCompass - INFO - Partitioned into 0 tasks.
03/17 20:23:09 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners
03/17 20:23:09 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
03/17 20:23:09 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
03/17 20:23:09 - OpenCompass - DEBUG - Get class `NaivePartitioner` from "partitioner" registry in "opencompass"
03/17 20:23:09 - OpenCompass - DEBUG - An `NaivePartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.naive
03/17 20:23:09 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
03/17 20:23:09 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
03/17 20:23:09 - OpenCompass - DEBUG - Additional config: {'eval': {'runner': {'task': {}}}}
03/17 20:23:09 - OpenCompass - INFO - Partitioned into 0 tasks.
03/17 20:23:09 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
03/17 20:23:09 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
03/17 20:23:09 - OpenCompass - DEBUG - An `DefaultSummarizer` instance is built from registry, and its implementation can be found in opencompass.summarizers.default
dataset    version    metric    mode    opencompass.models.huggingface.HuggingFace_full_Mistral-7B-v0.1
---------  ---------  --------  ------  -------------------------------------------------------------------------------------------------------------------
03/17 20:23:09 - OpenCompass - INFO - write summary to /home//opencompass/outputs/test/20240317_202309/summary/summary_20240317_202309.txt
03/17 20:23:09 - OpenCompass - INFO - write csv to /home/opencompass/outputs/test/20240317_202309/summary/summary_20240317_202309.csv
```


There is no predict and eval result. What should I do? 
My config file was given yesterday and I don't modify my file. My data is copied from cmmlu:
![image](https://github.com/open-compass/opencompass/assets/45348154/be96eded-fc4a-4df8-b55a-948dee2b996d)


> /home/opencompass/outputs/test/20240317_202309

Do you have any error info in this path"
`
/home/opencompass/outputs/test/20240317_202309/logs/infer
`

This script don't generate the dictionary of logs.
![image](https://github.com/open-compass/opencompass/assets/45348154/14c70b0e-7ee7-49db-822c-743ae78b75dd)


> This script don't generate the dictionary of logs. ![image](https://private-user-images.githubusercontent.com/45348154/313519204-14c70b0e-7ee7-49db-822c-743ae78b75dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTA3MzA1MjksIm5iZiI6MTcxMDczMDIyOSwicGF0aCI6Ii80NTM0ODE1NC8zMTM1MTkyMDQtMTRjNzBiMGUtN2VlNy00OWRiLTgyMmMtNzQzYWU3OGI3NWRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzE4VDAyNTAyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVkNjNlMDIyNmVjNDcxNmEzYmRiNjRhMTdhYjIwYmQyMTBiOWJjM2UzM2U5MDMzYzI5MWY3YjczODVhYjZkMTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JFeiVilkKhxuE-UU8Dc9MTKP41_jra_sm-ecP866N18)

See in here,
https://github.com/open-compass/opencompass/blob/89a8a8917b3135dec68c8de0922cdcb6e0f557a2/opencompass/utils/run.py#L72-L78,
If you use command line to run,
You need to change 'mydataset' to my_datasets

> > This script don't generate the dictionary of logs. ![image](https://private-user-images.githubusercontent.com/45348154/313519204-14c70b0e-7ee7-49db-822c-743ae78b75dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTA3MzA1MjksIm5iZiI6MTcxMDczMDIyOSwicGF0aCI6Ii80NTM0ODE1NC8zMTM1MTkyMDQtMTRjNzBiMGUtN2VlNy00OWRiLTgyMmMtNzQzYWU3OGI3NWRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzE4VDAyNTAyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVkNjNlMDIyNmVjNDcxNmEzYmRiNjRhMTdhYjIwYmQyMTBiOWJjM2UzM2U5MDMzYzI5MWY3YjczODVhYjZkMTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JFeiVilkKhxuE-UU8Dc9MTKP41_jra_sm-ecP866N18)
> 
> See in here,
> 
> https://github.com/open-compass/opencompass/blob/89a8a8917b3135dec68c8de0922cdcb6e0f557a2/opencompass/utils/run.py#L72-L78
> 
> ,
> If you use command line to run,
> You need to change 'mydataset' to my_datasets

Could you please give more detail? I renamed the folder and the path in my_dataset_ppl.py to my_datasets but it still doesn't work. No logs folder is generated.

```
03/18 13:54:35 - OpenCompass - INFO - Loading mydataset_ppl: configs/datasets/my_datasets/mydataset_ppl.py
03/18 13:54:35 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
03/18 13:54:35 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
03/18 13:54:35 - OpenCompass - INFO - Partitioned into 0 tasks.
0it [00:00, ?it/s]
03/18 13:54:35 - OpenCompass - INFO - Partitioned into 0 tasks.
0it [00:00, ?it/s]
dataset    version    metric    mode    opencompass.models.huggingface.HuggingFace_full_Mistral-7B-v0.1
```

> > > This script don't generate the dictionary of logs. ![image](https://private-user-images.githubusercontent.com/45348154/313519204-14c70b0e-7ee7-49db-822c-743ae78b75dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTA3MzA1MjksIm5iZiI6MTcxMDczMDIyOSwicGF0aCI6Ii80NTM0ODE1NC8zMTM1MTkyMDQtMTRjNzBiMGUtN2VlNy00OWRiLTgyMmMtNzQzYWU3OGI3NWRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzE4VDAyNTAyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVkNjNlMDIyNmVjNDcxNmEzYmRiNjRhMTdhYjIwYmQyMTBiOWJjM2UzM2U5MDMzYzI5MWY3YjczODVhYjZkMTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JFeiVilkKhxuE-UU8Dc9MTKP41_jra_sm-ecP866N18)
> > 
> > 
> > See in here,
> > https://github.com/open-compass/opencompass/blob/89a8a8917b3135dec68c8de0922cdcb6e0f557a2/opencompass/utils/run.py#L72-L78
> > 
> > ,
> > If you use command line to run,
> > You need to change 'mydataset' to my_datasets
> 
> Could you please give more detail? I renamed the folder and the path in my_dataset_ppl.py to my_datasets but it still doesn't work. No logs folder is generated.
> 
> ```
> 03/18 13:54:35 - OpenCompass - INFO - Loading mydataset_ppl: configs/datasets/my_datasets/mydataset_ppl.py
> 03/18 13:54:35 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
> 03/18 13:54:35 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
> 03/18 13:54:35 - OpenCompass - INFO - Partitioned into 0 tasks.
> 0it [00:00, ?it/s]
> 03/18 13:54:35 - OpenCompass - INFO - Partitioned into 0 tasks.
> 0it [00:00, ?it/s]
> dataset    version    metric    mode    opencompass.models.huggingface.HuggingFace_full_Mistral-7B-v0.1
> ```

Well, as we can see, the info is
`
Partitioned into 0 tasks.
`,
which means that the dataset config is not load successfully.
So if you want to use command line to start your tasks, you can rename the dataset name in your dataset config, just like the "_datasets"
`
    from .ceval_gen_5f30c7 import ceval_datasets 
` in here:
https://github.com/open-compass/opencompass/blob/89a8a8917b3135dec68c8de0922cdcb6e0f557a2/configs/datasets/ceval/ceval_gen.py#L4
By the way, you can also write down your dataset in a config, just like here:
https://github.com/open-compass/opencompass/blob/main/configs/eval_demo.py
and use python run.py configs/eval_demo.py to start your task

> > > > This script don't generate the dictionary of logs. ![image](https://private-user-images.githubusercontent.com/45348154/313519204-14c70b0e-7ee7-49db-822c-743ae78b75dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTA3MzA1MjksIm5iZiI6MTcxMDczMDIyOSwicGF0aCI6Ii80NTM0ODE1NC8zMTM1MTkyMDQtMTRjNzBiMGUtN2VlNy00OWRiLTgyMmMtNzQzYWU3OGI3NWRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzE4VDAyNTAyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVkNjNlMDIyNmVjNDcxNmEzYmRiNjRhMTdhYjIwYmQyMTBiOWJjM2UzM2U5MDMzYzI5MWY3YjczODVhYjZkMTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JFeiVilkKhxuE-UU8Dc9MTKP41_jra_sm-ecP866N18)
> > > 
> > > 
> > > See in here,
> > > https://github.com/open-compass/opencompass/blob/89a8a8917b3135dec68c8de0922cdcb6e0f557a2/opencompass/utils/run.py#L72-L78
> > > 
> > > ,
> > > If you use command line to run,
> > > You need to change 'mydataset' to my_datasets
> > 
> > 
> > Could you please give more detail? I renamed the folder and the path in my_dataset_ppl.py to my_datasets but it still doesn't work. No logs folder is generated.
> > ```
> > 03/18 13:54:35 - OpenCompass - INFO - Loading mydataset_ppl: configs/datasets/my_datasets/mydataset_ppl.py
> > 03/18 13:54:35 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
> > 03/18 13:54:35 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
> > 03/18 13:54:35 - OpenCompass - INFO - Partitioned into 0 tasks.
> > 0it [00:00, ?it/s]
> > 03/18 13:54:35 - OpenCompass - INFO - Partitioned into 0 tasks.
> > 0it [00:00, ?it/s]
> > dataset    version    metric    mode    opencompass.models.huggingface.HuggingFace_full_Mistral-7B-v0.1
> > ```
> 
> Well, as we can see, the info is `Partitioned into 0 tasks.`, which means that the dataset config is not load successfully. So if you want to use command line to start your tasks, you can rename the dataset name in your dataset config, just like the "_datasets" `from .ceval_gen_5f30c7 import ceval_datasets ` in here:
> 
> https://github.com/open-compass/opencompass/blob/89a8a8917b3135dec68c8de0922cdcb6e0f557a2/configs/datasets/ceval/ceval_gen.py#L4
> 
> 
> By the way, you can also write down your dataset in a config, just like here:
> https://github.com/open-compass/opencompass/blob/main/configs/eval_demo.py
> and use python run.py configs/eval_demo.py to start your task

It works. Thank you!

Feel free to reopen it if needed