Repository: open-compass/opencompass
Issue: [Bug] 评测后报错，output 文件夹里不创建prediction文件夹 报FileNotFoundError: Prediction files not found:
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

 opencompass-0.4.2
Python 3.10.18

### Reproduces the problem - code/configuration sample

怎么修改output 的输出目录呢？
没有自动生成predictions/      # 模型预测结果results/          # 评测结果汇总和summary/  目录
跑的命令如下


### Reproduces the problem - command or script

python run.py --datasets ceval_gen --hf-path /autodl-tmp/models/Shanghai_AI_Laboratory/internlm2-chat-1_8b --tokenizer-path /autodl-tmp/models/Shanghai_AI_Laboratory/internlm2-chat-1_8b --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs device_map='auto' trust_remote_code=True --max-seq-len 1024 --max-out-len 16 --batch-size 2  --hf-num-gpus 1 --work-dir outputs/default/20250811_230820

### Reproduces the problem - error message

每一个output 文件夹下的数据集评测结果都报错 
Map:   0%|                                                                                                   | 0/5 [00:00<?, ? examples/s]
Map: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1627.97 examples/s]

Map:   0%|                                                                                                  | 0/49 [00:00<?, ? examples/s]
Map: 100%|███████████████████████████████████████████████████████████████████████████████████████| 49/49 [00:00<00:00, 9821.32 examples/s]

Map:   0%|                                                                                                 | 0/443 [00:00<?, ? examples/s]
Map: 100%|████████████████████████████████████████████████████████████████████████████████████| 443/443 [00:00<00:00, 17971.18 examples/s]
Traceback (most recent call last):
  File "/root/evalue-benchmark/opencompass-main/opencompass/tasks/openicl_eval.py", line 466, in <module>
    inferencer.run()
  File "/root/evalue-benchmark/opencompass-main/opencompass/tasks/openicl_eval.py", line 88, in run
    self._score()
  File "/root/evalue-benchmark/opencompass-main/opencompass/tasks/openicl_eval.py", line 94, in _score
    pred_dicts, pred_strs = self._load_predictions()
  File "/root/evalue-benchmark/opencompass-main/opencompass/tasks/openicl_eval.py", line 140, in _load_predictions
    raise FileNotFoundError(
**FileNotFoundError: Prediction files not found: neither outputs/default/20250811_230820/predictions/internlm2-chat-1_8b_hf/ceval-accountant.json nor outputs/default/20250811_230820/predictions/internlm2-chat-1_8b_hf/ceval-accountant_0.json exists**



### Other information

_No response_

我是提问者，这里还有个infer 目录下的报错
08/11 23:53:11 - OpenCompass - INFO - Task [internlm2-chat-1_8b_hf/ceval-computer_network,internlm2-chat-1_8b_hf/ceval-operating_system,internlm2-chat-1_8b_hf/ceval-computer_architecture,internlm2-chat-1_8b_hf/ceval-college_programming,internlm2-chat-1_8b_hf/ceval-college_physics,internlm2-chat-1_8b_hf/ceval-college_chemistry,internlm2-chat-1_8b_hf/ceval-advanced_mathematics,internlm2-chat-1_8b_hf/ceval-probability_and_statistics,internlm2-chat-1_8b_hf/ceval-discrete_mathematics,internlm2-chat-1_8b_hf/ceval-electrical_engineer,internlm2-chat-1_8b_hf/ceval-metrology_engineer,internlm2-chat-1_8b_hf/ceval-high_school_mathematics,internlm2-chat-1_8b_hf/ceval-high_school_physics,internlm2-chat-1_8b_hf/ceval-high_school_chemistry,internlm2-chat-1_8b_hf/ceval-high_school_biology,internlm2-chat-1_8b_hf/ceval-middle_school_mathematics,internlm2-chat-1_8b_hf/ceval-middle_school_biology,internlm2-chat-1_8b_hf/ceval-middle_school_physics,internlm2-chat-1_8b_hf/ceval-middle_school_chemistry,internlm2-chat-1_8b_hf/ceval-veterinary_medicine,internlm2-chat-1_8b_hf/ceval-college_economics,internlm2-chat-1_8b_hf/ceval-business_administration,internlm2-chat-1_8b_hf/ceval-marxism,internlm2-chat-1_8b_hf/ceval-mao_zedong_thought,internlm2-chat-1_8b_hf/ceval-education_science,internlm2-chat-1_8b_hf/ceval-teacher_qualification,internlm2-chat-1_8b_hf/ceval-high_school_politics,internlm2-chat-1_8b_hf/ceval-high_school_geography,internlm2-chat-1_8b_hf/ceval-middle_school_politics,internlm2-chat-1_8b_hf/ceval-middle_school_geography,internlm2-chat-1_8b_hf/ceval-modern_chinese_history,internlm2-chat-1_8b_hf/ceval-ideological_and_moral_cultivation,internlm2-chat-1_8b_hf/ceval-logic,internlm2-chat-1_8b_hf/ceval-law,internlm2-chat-1_8b_hf/ceval-chinese_language_and_literature,internlm2-chat-1_8b_hf/ceval-art_studies,internlm2-chat-1_8b_hf/ceval-professional_tour_guide,internlm2-chat-1_8b_hf/ceval-legal_professional,internlm2-chat-1_8b_hf/ceval-high_school_chinese,internlm2-chat-1_8b_hf/ceval-high_school_history,internlm2-chat-1_8b_hf/ceval-middle_school_history,internlm2-chat-1_8b_hf/ceval-civil_servant,internlm2-chat-1_8b_hf/ceval-sports_science,internlm2-chat-1_8b_hf/ceval-plant_protection,internlm2-chat-1_8b_hf/ceval-basic_medicine,internlm2-chat-1_8b_hf/ceval-clinical_medicine,internlm2-chat-1_8b_hf/ceval-urban_and_rural_planner,internlm2-chat-1_8b_hf/ceval-accountant,internlm2-chat-1_8b_hf/ceval-fire_engineer,internlm2-chat-1_8b_hf/ceval-environmental_impact_assessment_engineer,internlm2-chat-1_8b_hf/ceval-tax_accountant,internlm2-chat-1_8b_hf/ceval-physician]
Traceback (most recent call last):
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/transformers/utils/hub.py", line 479, in cached_files
    hf_hub_download(
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/autodl-tmp/models/Shanghai_AI_Laboratory/internlm2-chat-1_8b'. Use `repo_type` argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/evalue-benchmark/opencompass-main/opencompass/tasks/openicl_infer.py", line 161, in <module>
    inferencer.run()
  File "/root/evalue-benchmark/opencompass-main/opencompass/tasks/openicl_infer.py", line 73, in run
    self.model = build_model_from_cfg(model_cfg)
  File "/root/evalue-benchmark/opencompass-main/opencompass/utils/build.py", line 24, in build_model_from_cfg
    return MODELS.build(model_cfg)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/root/evalue-benchmark/opencompass-main/opencompass/models/huggingface_above_v4_33.py", line 176, in __init__
    self._load_tokenizer(tokenizer_path or path, tokenizer_kwargs, pad_token_id)
  File "/root/evalue-benchmark/opencompass-main/opencompass/models/huggingface_above_v4_33.py", line 196, in _load_tokenizer
    self.tokenizer = AutoTokenizer.from_pretrained(path, **tokenizer_kwargs)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 1049, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 881, in get_tokenizer_config
    resolved_config_file = cached_file(
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/transformers/utils/hub.py", line 321, in cached_file
    file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/transformers/utils/hub.py", line 531, in cached_files
    resolved_files = [
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/transformers/utils/hub.py", line 532, in <listcomp>
    _get_cache_file_to_return(path_or_repo_id, filename, cache_dir, revision, repo_type)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/transformers/utils/hub.py", line 144, in _get_cache_file_to_return
    resolved_file = try_to_load_from_cache(
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/autodl-tmp/models/Shanghai_AI_Laboratory/internlm2-chat-1_8b'. Use `repo_type` argument if needed.
 

可以关了，模型没跑通


@yangxinQA 你好，请问你怎么解决的？

我的模型没跑起来，你先测试一下本地确保模型跑通再评测



---原始邮件---
发件人: ***@***.***&gt;
发送时间: 2025年8月24日(周日) 下午5:10
收件人: ***@***.***&gt;;
抄送: ***@***.******@***.***&gt;;
主题: Re: [open-compass/opencompass] [Bug] 评测后报错，output 文件夹里不创建prediction文件夹 报FileNotFoundError: Prediction files not found: (Issue #2235)


manglu097 left a comment (open-compass/opencompass#2235)
 
@yangxinQA 你好，请问你怎么解决的？
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

好的，我也是。lmdeploy没装

I see, the docs is problemtic, don't tell us we need to run lmdeploy