Repository: open-compass/opencompass
Issue: [Bug] windows下数据集的位置以及检测结果为0.0
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': True,
 'CUDA_HOME': 'D:\\CUDA\\CUDA12.1\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1',
 'GCC': 'n/a',
 'GPU 0': 'NVIDIA GeForce RTX 4070 Ti',
 'MMEngine': '0.10.5',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.105',
 'OpenCV': '4.10.0',
 'PyTorch': '2.0.0+cu118',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - C++ Version: 199711\n'
                              '  - MSVC 193431937\n'
                              '  - Intel(R) Math Kernel Library Version '
                              '2020.0.2 Product Build 20200624 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 2019\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 11.8\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
                              '  - CuDNN 8.7\n'
                              '  - Magma 2.5.4\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
                              'CUDNN_VERSION=8.7.0, '
                              'CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, '
                              'CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w '
                              '/bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG '
                              '-DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DUSE_FBGEMM '
                              '-DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE, '
                              'LAPACK_INFO=mkl, PERF_WITH_AVX=1, '
                              'PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=OFF, '
                              'TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.15 | packaged by Anaconda, Inc. | (main, Oct  3 2024, '
           '07:22:19) [MSC v.1929 64 bit (AMD64)]',
 'TorchVision': '0.15.1+cu118',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.6+',
 'sys.platform': 'win32',
 'transformers': '4.46.3'}

### Reproduces the problem - code/configuration sample

# For mymodel_cfg.py

from opencompass.models import HuggingFaceCausalLM
from opencompass.models import HuggingFaceBaseModel

# models = [
#     dict(
#         type=HuggingFaceCausalLM,
#         path='meta-llama/Llama-3.2-1B',
#         model_kwargs=dict(device_map='auto'),
#         tokenizer_path='meta-llama/Llama-3.2-1B',
#         tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
#         max_seq_len=2048,
#         max_out_len=50,
#         run_cfg=dict(num_gpus=1, num_procs=1),
#     )
# ]

models = [
    dict(
        type=HuggingFaceBaseModel,
        abbr='Llama-3.2-1B',
        path='meta-llama/Llama-3.2-1B',
        max_out_len=1024,
        batch_size=8,
        run_cfg=dict(num_gpus=1),
    )
]

# For myeval_demo.py

from mmengine.config import read_base

with read_base():
    from .mymodel_cfg import models  # mymodel_cfg.py 中的 models 被继承到本配置文件
    from opencompass.configs.datasets.demo.demo_gsm8k_base_gen import gsm8k_datasets
    from opencompass.configs.datasets.demo.demo_math_base_gen import math_datasets
    
datasets = []
datasets += gsm8k_datasets
datasets += math_datasets



### Reproduces the problem - command or script

python run.py eval_demo.py --debug

### Reproduces the problem - error message

F:\Env\LLM_env\lib\site-packages\opencompass\__init__.py:17: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in `./configs/datasets`, `./configs/models`, and `./configs/summarizers` will be migrated to the `opencompass/configs/` package. Please update your configuration file paths accordingly.
  _warn_about_config_migration()
11/29 01:36:09 - OpenCompass - INFO - Task [Llama-3.2-1B/demo_gsm8k,Llama-3.2-1B/demo_math]
F:\Env\LLM_env\lib\site-packages\torchvision\datapoints\__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
F:\Env\LLM_env\lib\site-packages\torchvision\transforms\v2\__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
11/29 01:36:12 - OpenCompass - WARNING - pad_token_id is not set for the tokenizer.
11/29 01:36:12 - OpenCompass - WARNING - Using eos_token_id 128001 as pad_token_id.
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
11/29 01:36:15 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator\.cache/opencompass/./data/gsm8k/
11/29 01:36:15 - OpenCompass - INFO - Start inferencing [Llama-3.2-1B/demo_gsm8k]
[2024-11-29 01:36:15,688] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-11-29 01:36:15,688] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...

  0%|          | 0/8 [00:00<?, ?it/s]F:\Env\LLM_env\lib\site-packages\transformers\generation\configuration_utils.py:590: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
F:\Env\LLM_env\lib\site-packages\transformers\generation\configuration_utils.py:595: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(

 12%|            | 1/8 [00:05<00:37,  5.33s/it]
 25%|             | 2/8 [00:19<01:02, 10.41s/it]
 38%|              | 3/8 [00:36<01:06, 13.39s/it]
 50%|               | 4/8 [00:52<00:58, 14.68s/it]
 62%|                 | 5/8 [01:09<00:45, 15.33s/it]
 75%|                  | 6/8 [01:28<00:33, 16.69s/it]
 88%|                   | 7/8 [01:47<00:17, 17.35s/it]
100%|                    | 8/8 [02:06<00:00, 17.88s/it]
100%|                    | 8/8 [02:06<00:00, 15.80s/it]
11/29 01:38:22 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator\.cache/opencompass/./data/math/
11/29 01:38:22 - OpenCompass - INFO - Start inferencing [Llama-3.2-1B/demo_math]
[2024-11-29 01:38:22,195] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-11-29 01:38:22,195] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...

  0%|          | 0/8 [00:00<?, ?it/s]
 12%|            | 1/8 [00:34<03:59, 34.22s/it]
 25%|             | 2/8 [01:08<03:24, 34.03s/it]
 38%|              | 3/8 [01:43<02:54, 34.82s/it]
 50%|               | 4/8 [02:17<02:17, 34.43s/it]
 62%|                 | 5/8 [02:51<01:42, 34.17s/it]
 75%|                  | 6/8 [03:24<01:07, 33.95s/it]
 88%|                   | 7/8 [03:58<00:33, 33.78s/it]
100%|                    | 8/8 [04:32<00:00, 33.75s/it]
100%|                    | 8/8 [04:32<00:00, 34.01s/it]
11/29 01:42:54 - OpenCompass - INFO - time elapsed: 404.59s
F:\Env\LLM_env\lib\site-packages\opencompass\__init__.py:17: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in `./configs/datasets`, `./configs/models`, and `./configs/summarizers` will be migrated to the `opencompass/configs/` package. Please update your configuration file paths accordingly.
  _warn_about_config_migration()
F:\Env\LLM_env\lib\site-packages\torchvision\datapoints\__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
F:\Env\LLM_env\lib\site-packages\torchvision\transforms\v2\__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
11/29 01:42:59 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator\.cache/opencompass/./data/gsm8k/
Parameter 'function'=<function OpenICLEvalTask._score.<locals>.postprocess at 0x0000013618439510> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.

Map:   0%|          | 0/64 [00:00<?, ? examples/s]
Map: 100%|                    | 64/64 [00:00<00:00, 21402.92 examples/s]
11/29 01:42:59 - OpenCompass - INFO - Task [Llama-3.2-1B/demo_gsm8k]: {'accuracy': 4.6875}
11/29 01:42:59 - OpenCompass - INFO - time elapsed: 2.10s
F:\Env\LLM_env\lib\site-packages\opencompass\__init__.py:17: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in `./configs/datasets`, `./configs/models`, and `./configs/summarizers` will be migrated to the `opencompass/configs/` package. Please update your configuration file paths accordingly.
  _warn_about_config_migration()
F:\Env\LLM_env\lib\site-packages\torchvision\datapoints\__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
F:\Env\LLM_env\lib\site-packages\torchvision\transforms\v2\__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
11/29 01:43:04 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator\.cache/opencompass/./data/math/
11/29 01:43:04 - OpenCompass - INFO - Task [Llama-3.2-1B/demo_math]: {'accuracy': 0.0}
11/29 01:43:04 - OpenCompass - INFO - time elapsed: 2.08s


### Other information

（1）我已经下载了离线数据集，但是似乎在运行中opencompass仍然重新下载并缓存在了C:\Users\Administrator\.cache\opencompass\data，我应如何修改离散数据集的路径，或者修改opencompass读取数据的路径？
（2）我使用的Llama3.2-1B模型在math_demo数据集上的acc为0，这是正常情况还是其他bug？