Repository: open-compass/opencompass
Issue: No module named opencompass.datasets.IFEval
Root cause: Environment Incompatibility

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python 3.10

### 重现问题 - 代码/配置示例

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

### 重现问题 - 命令或脚本

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

### 重现问题 - 错误信息

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

### 其他信息

_No response_

Hello, please ensure you installed latest opencompass correctly, and you can provide your config about how you runned on IFEval

I use below command in my config and runned successfully
`
with read_base():
    from .datasets.IFEval.IFEval_gen import ifeval_datasets
datasets = [*ifeval_datasets]
`
Feel free to reopen an issue or reply here if you still have this problem

> I use below command in my config and runned successfully `with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets]` Feel free to reopen an issue or reply here if you still have this problem

I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked __init__.py later and found nothing. :( 

> > I use below command in my config and runned successfully `with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets]` Feel free to reopen an issue or reply here if you still have this problem
> 
> I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked **init**.py later and found nothing. :(

can you provide the detailed config you used?


> > > I use below command in my config and runned successfully `with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets]` Feel free to reopen an issue or reply here if you still have this problem
> > 
> > 
> > I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked **init**.py later and found nothing. :(
> 
> can you provide the detailed config you used?

`from mmengine.config import read_base

with read_base():
    from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets]
    from .models.qwen.hf_qwen_7b import models
    # from .datasets.collections.leaderboard.qwen import datasets    # 使用所有数据集
    # from .summarizers.leaderboard import summarizer    # 使用summarizer做全数据集展示
    # from .datasets.ceval.ceval_ppl_578f8d import ceval_datasets    # 使用ceval数据集
    from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets    # 使用ceval数据集
    from .datasets.cmmlu.cmmlu_gen_c13365 import cmmlu_datasets    # 使用cmmlu数据集
    # from .datasets.cmmlu.cmmlu_ppl_8b9c76 import cmmlu_datasets    # 使用cmmlu数据集
    


# datasets = [*piqa_datasets, *siqa_datasets]
# datasets = [*ceval_datasets]
datasets = [*cmmlu_datasets, *ceval_datasets]`

It seems no error in your config, if you use 
```
from mmengine.config import read_base

with read_base():
from .datasets.IFEval.IFEval_gen import ifeval_datasets 
datasets = [*ifeval_datasets]
from .models.qwen.hf_qwen_7b import models
```
these command, it is expected to run ifeval dataset with qwen7b model successfully
and what's the error in the logs

> It seems no error in your config, if you use
> 
> ```
> from mmengine.config import read_base
> 
> with read_base():
> from .datasets.IFEval.IFEval_gen import ifeval_datasets 
> datasets = [*ifeval_datasets]
> from .models.qwen.hf_qwen_7b import models
> ```
> 
> these command, it is expected to run ifeval dataset with qwen7b model successfully and what's the error in the logs

here it is:
Traceback (most recent call last):
  File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 366, in <module>
    inferencer.run()
  File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 112, in run
    self._score()
  File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 115, in _score
    test_set = build_dataset_from_cfg(self.dataset_cfg).test
  File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\opencompass\utils\build.py", line 13, in build_dataset_from_cfg
    return LOAD_DATASET.build(dataset_cfg)
  File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\build_functions.py", line 98, in build_from_cfg
    obj_cls = registry.get(obj_type)
  File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 451, in get
    self.import_from_location()
  File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 376, in import_from_location
    import_module(loc)
  File "C:\Users\86181\miniconda3\envs\opencompass\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\opencompass\datasets\__init__.py", line 52, in <module>
    from .IFEval.ifeval import IFEvalDataset, IFEvaluator  # noqa: F401, F403
ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

how did you installed open compass, I think this may caused by code not updated or not installed correctly, you can try to make sure your code is latest and use ```pip install -e .``` to reinstall open compass

well I just installed it by:conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y
conda activate opencompass
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
from official text

can you check whether there is IFEval files in opencompass/opencompass/datasets?

yes there it is

When was the last time you installed opencompass? I just tried it with your commands and had no issues. I think you can re-clone and re-install to try again.

hmmmmmmm, I installed it just now,

the error disappeared, but my out put is null

> When was the last time you installed opencompass? I just tried it with your commands and had no issues. I think you can re-clone and re-install to try again.



plus, logs below:
04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found.
04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

there must be some other errors, you can provide the information in the .out file

> there must be some other errors, you can provide the information in the .out file

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found.
04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

> > there must be some other errors, you can provide the information in the .out file
> 
> 04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

As you can see in the information, it turns into the eval stage, but there is no predictions files, so do you use ```--mode eval``` to just run eval stage?

> > > there must be some other errors, you can provide the information in the .out file
> > 
> > 
> > 04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s
> 
> As you can see in the information, it turns into the eval stage, but there is no predictions files, so do you use `--mode eval` to just run eval stage?

actually I ran"python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl" just to test it, from the official text.
I don't really know how it actually works, it's built-in

can you check this file for me: opencompass/outputs/defaults/xxx/ceval/infer/ceval-chinese_language_and_literature.out

sry, the file isn't even exist

that's ok, when you not use ```--debug```, the error info will stored in such a .out file.
If it isn't exist, you can use this command
```
python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug
```
 to run again and get the error info directly from terminal.

> can you check this file for me: opencompass/outputs/defaults/xxx/ceval/infer/ceval-chinese_language_and_literature.out

nvm,the logs I mentioned above actually came from the file you asked.

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found.
04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s


and here,04/29 19:06:44 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-clinical_medicine]: No predictions found.
04/29 19:06:44 - OpenCompass - INFO - time elapsed: 11.33s came from ceval-clinical_medicine.out

> and here,04/29 19:06:44 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-clinical_medicine]: No predictions found. 04/29 19:06:44 - OpenCompass - INFO - time elapsed: 11.33s came from ceval-clinical_medicine.out

I see, but it comes from ```infer/ceval-clinical_medicine.out``` or ```eval/ceval-clinical_medicine.out``` ?

the latter

and I need to see the info in infer/ceval-clinical_medicine.out

this is because although it run failed in ```eval stage```, it was caused by ```No predictions found```, so there must be some error in ```infer stage```

this file doesn't exist in the document you're talking about, the infer documents holds 4 files, with the same substance as"failed to create process." 

please try to run this command 
```
python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug --mode infer
```

> please try to run this command
> 
> ```
> python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug --mode infer
> ```

output below:
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
04/29 19:38:31 - OpenCompass - INFO - Loading mmlu_ppl: configs\datasets\mmlu\mmlu_ppl.py
04/29 19:38:32 - OpenCompass - INFO - Loading ceval_ppl: configs\datasets\ceval\ceval_ppl.py
04/29 19:38:32 - OpenCompass - INFO - Loading hf_llama_7b: configs\models\hf_llama\hf_llama_7b.py
04/29 19:38:32 - OpenCompass - INFO - Loading example: configs\summarizers\example.py
04/29 19:38:33 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners
04/29 19:38:33 - OpenCompass - DEBUG - Get class `SizePartitioner` from "partitioner" registry in "opencompass"
04/29 19:38:33 - OpenCompass - DEBUG - An `SizePartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.size
04/29 19:38:33 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
04/29 19:38:33 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
04/29 19:38:33 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored.
04/29 19:38:33 - OpenCompass - DEBUG - Additional config: {}
04/29 19:38:33 - OpenCompass - INFO - Partitioned into 4 tasks.
04/29 19:38:33 - OpenCompass - DEBUG - Task 0: [llama-7b-hf/lukaemon_mmlu_professional_law_0]
04/29 19:38:33 - OpenCompass - DEBUG - Task 1: [llama-7b-hf/lukaemon_mmlu_professional_law_1]
04/29 19:38:33 - OpenCompass - DEBUG - Task 2: [llama-7b-hf/lukaemon_mmlu_moral_scenarios,llama-7b-hf/lukaemon_mmlu_miscellaneous,llama-7b-hf/lukaemon_mmlu_professional_psychology,llama-7b-hf/lukaemon_mmlu_high_school_psychology,llama-7b-hf/lukaemon_mmlu_high_school_macroeconomics,llama-7b-hf/lukaemon_mmlu_elementary_mathematics,llama-7b-hf/lukaemon_mmlu_moral_disputes,llama-7b-hf/lukaemon_mmlu_prehistory,llama-7b-hf/lukaemon_mmlu_philosophy,llama-7b-hf/lukaemon_mmlu_high_school_biology,llama-7b-hf/lukaemon_mmlu_nutrition,llama-7b-hf/lukaemon_mmlu_professional_accounting,llama-7b-hf/lukaemon_mmlu_professional_medicine,llama-7b-hf/lukaemon_mmlu_high_school_mathematics,llama-7b-hf/lukaemon_mmlu_clinical_knowledge,llama-7b-hf/lukaemon_mmlu_security_studies,llama-7b-hf/lukaemon_mmlu_high_school_microeconomics,llama-7b-hf/lukaemon_mmlu_high_school_world_history,llama-7b-hf/lukaemon_mmlu_conceptual_physics,llama-7b-hf/lukaemon_mmlu_marketing,llama-7b-hf/lukaemon_mmlu_human_aging,llama-7b-hf/lukaemon_mmlu_high_school_statistics,llama-7b-hf/lukaemon_mmlu_high_school_us_history,llama-7b-hf/lukaemon_mmlu_high_school_chemistry,llama-7b-hf/lukaemon_mmlu_sociology,llama-7b-hf/lukaemon_mmlu_high_school_geography,llama-7b-hf/lukaemon_mmlu_high_school_government_and_politics,llama-7b-hf/lukaemon_mmlu_college_medicine,llama-7b-hf/lukaemon_mmlu_world_religions,llama-7b-hf/lukaemon_mmlu_virology,llama-7b-hf/lukaemon_mmlu_high_school_european_history,llama-7b-hf/lukaemon_mmlu_logical_fallacies,llama-7b-hf/lukaemon_mmlu_astronomy]
04/29 19:38:33 - OpenCompass - DEBUG - Task 3: [llama-7b-hf/lukaemon_mmlu_high_school_physics,llama-7b-hf/lukaemon_mmlu_electrical_engineering,llama-7b-hf/lukaemon_mmlu_college_biology,llama-7b-hf/lukaemon_mmlu_anatomy,llama-7b-hf/lukaemon_mmlu_human_sexuality,llama-7b-hf/lukaemon_mmlu_formal_logic,llama-7b-hf/lukaemon_mmlu_international_law,llama-7b-hf/lukaemon_mmlu_econometrics,llama-7b-hf/lukaemon_mmlu_machine_learning,llama-7b-hf/lukaemon_mmlu_public_relations,llama-7b-hf/lukaemon_mmlu_jurisprudence,llama-7b-hf/lukaemon_mmlu_management,llama-7b-hf/lukaemon_mmlu_college_physics,llama-7b-hf/lukaemon_mmlu_college_chemistry,llama-7b-hf/lukaemon_mmlu_college_computer_science,llama-7b-hf/lukaemon_mmlu_college_mathematics,llama-7b-hf/lukaemon_mmlu_abstract_algebra,llama-7b-hf/lukaemon_mmlu_global_facts,llama-7b-hf/lukaemon_mmlu_computer_security,llama-7b-hf/lukaemon_mmlu_medical_genetics,llama-7b-hf/lukaemon_mmlu_high_school_computer_science,llama-7b-hf/lukaemon_mmlu_business_ethics,llama-7b-hf/lukaemon_mmlu_us_foreign_policy,llama-7b-hf/ceval-college_economics,llama-7b-hf/ceval-accountant,llama-7b-hf/ceval-tax_accountant,llama-7b-hf/ceval-physician,llama-7b-hf/ceval-civil_servant,llama-7b-hf/ceval-urban_and_rural_planner,llama-7b-hf/ceval-teacher_qualification,llama-7b-hf/ceval-college_programming,llama-7b-hf/ceval-electrical_engineer,llama-7b-hf/ceval-business_administration,llama-7b-hf/ceval-art_studies,llama-7b-hf/ceval-fire_engineer,llama-7b-hf/ceval-environmental_impact_assessment_engineer,llama-7b-hf/ceval-education_science,llama-7b-hf/ceval-professional_tour_guide,llama-7b-hf/ceval-college_chemistry,llama-7b-hf/ceval-metrology_engineer,llama-7b-hf/ceval-mao_zedong_thought,llama-7b-hf/ceval-law,llama-7b-hf/ceval-veterinary_medicine,llama-7b-hf/ceval-modern_chinese_history,llama-7b-hf/ceval-chinese_language_and_literature,llama-7b-hf/ceval-legal_professional,llama-7b-hf/ceval-logic,llama-7b-hf/ceval-middle_school_history,llama-7b-hf/ceval-plant_protection,llama-7b-hf/ceval-clinical_medicine,llama-7b-hf/ceval-computer_architecture,llama-7b-hf/ceval-middle_school_biology,llama-7b-hf/ceval-middle_school_politics,llama-7b-hf/ceval-middle_school_chemistry,llama-7b-hf/ceval-high_school_history,llama-7b-hf/ceval-computer_network,llama-7b-hf/ceval-operating_system,llama-7b-hf/ceval-college_physics,llama-7b-hf/ceval-advanced_mathematics,llama-7b-hf/ceval-high_school_physics,llama-7b-hf/ceval-high_school_chemistry,llama-7b-hf/ceval-high_school_biology,llama-7b-hf/ceval-middle_school_mathematics,llama-7b-hf/ceval-middle_school_physics,llama-7b-hf/ceval-marxism,llama-7b-hf/ceval-high_school_politics,llama-7b-hf/ceval-high_school_geography,llama-7b-hf/ceval-ideological_and_moral_cultivation,llama-7b-hf/ceval-high_school_chinese,llama-7b-hf/ceval-sports_science,llama-7b-hf/ceval-basic_medicine,llama-7b-hf/ceval-probability_and_statistics,llama-7b-hf/ceval-high_school_mathematics,llama-7b-hf/ceval-discrete_mathematics,llama-7b-hf/ceval-middle_school_geography]
04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners
04/29 19:38:33 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
04/29 19:38:33 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's task registry have been automatically imported from opencompass.tasks
04/29 19:38:33 - OpenCompass - DEBUG - Get class `OpenICLInferTask` from "task" registry in "opencompass"
04/29 19:38:33 - OpenCompass - DEBUG - An `OpenICLInferTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer
Traceback (most recent call last):
  File "C:\Users\86181\opencompass\run.py", line 4, in <module>
    main()
  File "C:\Users\86181\opencompass\opencompass\cli\main.py", line 302, in main
    runner(tasks)
  File "C:\Users\86181\opencompass\opencompass\runners\base.py", line 38, in __call__
    status = self.launch(tasks)
  File "C:\Users\86181\opencompass\opencompass\runners\local.py", line 88, in launch
    assert len(all_gpu_ids) >= num_gpus
AssertionError

Are you running on local or slum? if local, try ```nvidia-smi``` to check whether your GPU is normal

> Are you running on local or slum? if local, try `nvidia-smi` to check whether your GPU is normal

local
Mon Apr 29 19:41:58 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 552.12                 Driver Version: 552.12         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3070 ...  WDDM  |   00000000:01:00.0 Off |                  N/A |
| N/A   54C    P8             11W /   55W |     959MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+---------------------- 
It's normal I guess

oh Ok, so you are running on windows right? there are some version incompatible between open compass and windows, so we suggest to run on linux

> oh Ok, so you are running on windows right? there are some version incompatible between open compass and windows, so we suggest to run on linux

gotcha, thx for ur patience

you're welcome, feel free to contact me if needed

The reason for this issue is IFEval's missing if you use "pip install opencompass". So If you're using conda, you can try as follow:  
Copy the whole folder "{opencompass_project_path}/opencompass/datasets" to your environment (eg. miniconda3/envs/opencompass_new/lib/python3.10/site-packages/opencompass/datasets/)