Repository: open-compass/opencompass
Issue: [Bug] Unable to load model offline
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'Tesla V100-SXM2-32GB',
 'MMEngine': '0.10.5',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
 'OpenCV': '4.10.0',
 'PyTorch': '2.0.1+cu117',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 11.7\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n'
                              '  - CuDNN 8.5\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.7, '
                              'CUDNN_VERSION=8.5.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.11.4 (main, Jul  5 2023, 13:45:01) [GCC 11.2.0]',
 'TorchVision': '0.15.2+cu117',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.35.2'}

### Reproduces the problem - code/configuration sample

from opencompass.models import HuggingFaceBaseModel

models = [
    dict(
        type=HuggingFaceBaseModel,
        abbr='llama-2-7b-hf',
        path='/share--net=host/llama2-7B',    #local model path
        max_out_len=1024,
        batch_size=8,
        run_cfg=dict(num_gpus=1),
    )
]

from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.mmlu.mmlu_ppl_ac766d import mmlu_datasets
    from opencompass.configs.datasets.triviaqa.triviaqa_wiki_gen_d18bf4 import triviaqa_datasets
    #from opencompass.configs.datasets.nq.nq_open_gen_e93f8a import nq_datasets
    #from opencompass.configs.datasets.gsm8k.gsm8k_gen_3309bd import gsm8k_datasets
    #from opencompass.configs.datasets.humaneval.deprecated_humaneval_gen_a82cae import humaneval_datasets
    #from opencompass.configs.datasets.agieval.agieval_mixed_713d14 import agieval_datasets
    #from opencompass.configs.datasets.SuperGLUE_BoolQ.SuperGLUE_BoolQ_ppl_314797 import BoolQ_datasets
    #from opencompass.configs.datasets.hellaswag.hellaswag_ppl_a6e128 import hellaswag_datasets
    #from opencompass.configs.datasets.obqa.obqa_ppl_6aac9e import obqa_datasets
    #from opencompass.configs.datasets.winogrande.winogrande_ll_c5cf57 import winogrande_datasets
    from opencompass.configs.models.hf_llama.hf_llama2_7b import models
    from opencompass.configs.summarizers.example import summarizer

datasets = sum([v for k, v in locals().items() if k.endswith('_datasets') or k == 'datasets'], [])
work_dir = './outputs/llama2/'

### Reproduces the problem - command or script

HF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 \
> opencompass /share--net=host/opencompass/configs/eval_hf_llama2.py

### Reproduces the problem - error message

/share--net=host/opencompass/opencompass/__init__.py:19: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in `./configs/datasets`, `./configs/models`, and `./configs/summarizers` will be migrated to the `opencompass/configs/` package. Please update your configuration file paths accordingly.
  _warn_about_config_migration()
01/13 22:13:47 - OpenCompass - INFO - Task [llama-2-7b-hf/lukaemon_mmlu_college_biology,llama-2-7b-hf/lukaemon_mmlu_college_chemistry,llama-2-7b-hf/lukaemon_mmlu_college_computer_science,llama-2-7b-hf/lukaemon_mmlu_college_mathematics,llama-2-7b-hf/lukaemon_mmlu_college_physics,llama-2-7b-hf/lukaemon_mmlu_electrical_engineering,llama-2-7b-hf/lukaemon_mmlu_astronomy,llama-2-7b-hf/lukaemon_mmlu_anatomy,llama-2-7b-hf/lukaemon_mmlu_abstract_algebra,llama-2-7b-hf/lukaemon_mmlu_machine_learning,llama-2-7b-hf/lukaemon_mmlu_clinical_knowledge,llama-2-7b-hf/lukaemon_mmlu_global_facts,llama-2-7b-hf/lukaemon_mmlu_management,llama-2-7b-hf/lukaemon_mmlu_nutrition,llama-2-7b-hf/lukaemon_mmlu_marketing,llama-2-7b-hf/lukaemon_mmlu_professional_accounting,llama-2-7b-hf/lukaemon_mmlu_high_school_geography,llama-2-7b-hf/lukaemon_mmlu_international_law,llama-2-7b-hf/lukaemon_mmlu_moral_scenarios,llama-2-7b-hf/lukaemon_mmlu_computer_security,llama-2-7b-hf/lukaemon_mmlu_high_school_microeconomics,llama-2-7b-hf/lukaemon_mmlu_professional_law,llama-2-7b-hf/lukaemon_mmlu_medical_genetics,llama-2-7b-hf/lukaemon_mmlu_professional_psychology,llama-2-7b-hf/lukaemon_mmlu_jurisprudence,llama-2-7b-hf/lukaemon_mmlu_world_religions,llama-2-7b-hf/lukaemon_mmlu_philosophy,llama-2-7b-hf/lukaemon_mmlu_virology,llama-2-7b-hf/lukaemon_mmlu_high_school_chemistry,llama-2-7b-hf/lukaemon_mmlu_public_relations,llama-2-7b-hf/lukaemon_mmlu_high_school_macroeconomics,llama-2-7b-hf/lukaemon_mmlu_human_sexuality,llama-2-7b-hf/lukaemon_mmlu_elementary_mathematics,llama-2-7b-hf/lukaemon_mmlu_high_school_physics,llama-2-7b-hf/lukaemon_mmlu_high_school_computer_science,llama-2-7b-hf/lukaemon_mmlu_high_school_european_history,llama-2-7b-hf/lukaemon_mmlu_business_ethics,llama-2-7b-hf/lukaemon_mmlu_moral_disputes,llama-2-7b-hf/lukaemon_mmlu_high_school_statistics,llama-2-7b-hf/lukaemon_mmlu_miscellaneous,llama-2-7b-hf/lukaemon_mmlu_formal_logic,llama-2-7b-hf/lukaemon_mmlu_high_school_government_and_politics,llama-2-7b-hf/lukaemon_mmlu_prehistory,llama-2-7b-hf/lukaemon_mmlu_security_studies,llama-2-7b-hf/lukaemon_mmlu_high_school_biology,llama-2-7b-hf/lukaemon_mmlu_logical_fallacies,llama-2-7b-hf/lukaemon_mmlu_high_school_world_history,llama-2-7b-hf/lukaemon_mmlu_professional_medicine,llama-2-7b-hf/lukaemon_mmlu_high_school_mathematics,llama-2-7b-hf/lukaemon_mmlu_college_medicine,llama-2-7b-hf/lukaemon_mmlu_high_school_us_history,llama-2-7b-hf/lukaemon_mmlu_sociology,llama-2-7b-hf/lukaemon_mmlu_econometrics,llama-2-7b-hf/lukaemon_mmlu_high_school_psychology,llama-2-7b-hf/lukaemon_mmlu_human_aging,llama-2-7b-hf/lukaemon_mmlu_us_foreign_policy,llama-2-7b-hf/lukaemon_mmlu_conceptual_physics,llama-2-7b-hf/triviaqa_wiki_0shot,llama-2-7b-hf/triviaqa_wiki_1shot,llama-2-7b-hf/triviaqa_wiki_5shot,llama-2-7b-hf/triviaqa_wiki_25shot]
/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Traceback (most recent call last):
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error
    metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata
    r = _request_wrapper(
        ^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper
    response = _request_wrapper(
               ^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 395, in _request_wrapper
    response = get_session().request(method=method, url=url, **params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 77, in send
    raise OfflineModeIsEnabled(
huggingface_hub.errors.OfflineModeIsEnabled: Cannot reach https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json: offline mode is enabled. To disable it, please unset the `HF_HUB_OFFLINE` environment variable.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/transformers/utils/hub.py", line 430, in cached_file
    resolved_file = hf_hub_download(
                    ^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download
    return _hf_hub_download_to_cache_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1325, in _hf_hub_download_to_cache_dir
    _raise_on_head_call_error(head_call_error, force_download, local_files_only)
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1826, in _raise_on_head_call_error
    raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/share--net=host/opencompass/opencompass/tasks/openicl_infer.py", line 161, in <module>
    inferencer.run()
  File "/share--net=host/opencompass/opencompass/tasks/openicl_infer.py", line 73, in run
    self.model = build_model_from_cfg(model_cfg)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/share--net=host/opencompass/opencompass/utils/build.py", line 25, in build_model_from_cfg
    return MODELS.build(model_cfg)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
          ^^^^^^^^^^^^^^^
  File "/share--net=host/opencompass/opencompass/models/huggingface_above_v4_33.py", line 530, in __init__
    self.max_seq_len = _get_possible_max_seq_len(max_seq_len, path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/share--net=host/opencompass/opencompass/models/huggingface_above_v4_33.py", line 50, in _get_possible_max_seq_len
    config = AutoConfig.from_pretrained(path, trust_remote_code=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1048, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/transformers/configuration_utils.py", line 622, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/transformers/configuration_utils.py", line 677, in _get_config_dict
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/root/miniconda3/envs/DL/lib/python3.11/site-packages/transformers/utils/hub.py", line 470, in cached_file
    raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like meta-llama/Llama-2-7b-hf is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.


### Other information

_No response_

Since my server can't connect to the hugging face, I need to load the model offline, I changed the path to the model to a local path and set the offline mode, but it still reports an error, the log shows OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like meta-llama/Llama-2-7b-hf is not the path to a directory containing a file named config. json. json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline Checkout your internet connection or see how to run the library in offline mode at ' -mode'. It seems to still be trying to load the model from the hugging face

eval local model with this :
```
python run.py \
--hf-type base --hf-path /data/llms/Qwen2-1.5B-Instruct \
--datasets math_gen \
--debug
```

> eval local model with this :
> 
> ```
> python run.py \
> --hf-type base --hf-path /data/llms/Qwen2-1.5B-Instruct \
> --datasets math_gen \
> --debug
> ```

it works.   thanks a lot!

Close issue. Reopen if needed.

How can I use the local checkpoint for loading? For example, I have a checkpoint that can be loaded by transformers' autoclass,  but it fails to work with opencompass.

> How can I use the local checkpoint for loading? For example, I have a checkpoint that can be loaded by transformers' autoclass, but it fails to work with opencompass.

两条路

1、如果你的checkpoint能被lmdeploy/vllm直接拉起，那就把opencompass的推理引擎改成lmdeploy/vllm，比较简单
2、去看opencompass/models/huggingface.py、opencompass/models/huggingface_above_v4_33.py这块的源码，给你的checkpoint加一些自定义的需求改动