Repository: open-compass/opencompass
Issue: [Bug] torch.OutOfMemoryError
Root cause: Resource Mishandling

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0',
 'GPU 0,1': 'NVIDIA GeForce RTX 4090',
 'MMEngine': '0.10.5',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
 'OpenCV': '4.10.0',
 'PyTorch': '2.5.1+cu124',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2024.2-Product Build 20240605 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 12.4\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.4, '
                              'CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
                              'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
                              'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0]',
 'TorchVision': '0.20.1+cu124',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.8+',
 'sys.platform': 'linux',
 'transformers': '4.47.1'}

### 重现问题 - 代码/配置示例

CUDA_VISIBLE_DEVICES=0,1 python run.py \
    --datasets tydiqa_gen \
    --hf-type chat \
    --hf-path /public/zzy/model/Phi/Phi-3-mini-128k-instruct \
    --batch-size 1 \
    --debug

### 重现问题 - 命令或脚本

上面的python命令


### 重现问题 - 错误信息

/public/zzy/fintuning/opencompass/opencompass/__init__.py:19: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in `./configs/datasets`, `./configs/models`, and `./configs/summarizers` will be migrated to the `opencompass/configs/` package. Please update your configuration file paths accordingly.
  _warn_about_config_migration()
12/22 15:16:50 - OpenCompass - WARNING - Found ambiguous patterns, using the first matched config.
+----------------------+---------------------------------------------------------------------------------------+
| Ambiguous patterns   | Matched files                                                                         |
|----------------------+---------------------------------------------------------------------------------------|
| tydiqa_gen           | configs/datasets/tydiqa/tydiqa_gen.py                                                 |
|                      | /public/zzy/fintuning/opencompass/opencompass/configs/./datasets/tydiqa/tydiqa_gen.py |
+----------------------+---------------------------------------------------------------------------------------+
12/22 15:16:50 - OpenCompass - INFO - Loading tydiqa_gen: configs/datasets/tydiqa/tydiqa_gen.py
12/22 15:16:50 - OpenCompass - WARNING - Found ambiguous patterns, using the first matched config.
+----------------------+--------------------------------------------------------------------------------+
| Ambiguous patterns   | Matched files                                                                  |
|----------------------+--------------------------------------------------------------------------------|
| example              | configs/summarizers/example.py                                                 |
|                      | /public/zzy/fintuning/opencompass/opencompass/configs/./summarizers/example.py |
+----------------------+--------------------------------------------------------------------------------+
12/22 15:16:50 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
12/22 15:16:50 - OpenCompass - INFO - Current exp folder: outputs/default/20241222_151650
12/22 15:16:51 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
12/22 15:16:51 - OpenCompass - INFO - Partitioned into 1 tasks.
12/22 15:16:52 - OpenCompass - WARNING - Only use 1 GPUs for total 2 available GPUs in debug mode.
12/22 15:16:52 - OpenCompass - INFO - Task [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_arabic,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_bengali,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_english,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_finnish,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_indonesian,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_japanese,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_korean,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_russian,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_swahili,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_telugu,Phi-3-mini-128k-instruct_hf/tydiqa-goldp_thai]
`flash-attention` package not found, consider installing for better performance: No module named 'flash_attn'.
Current `flash-attenton` does not support `window_size`. Either upgrade or use `attn_implementation='eager'`.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████| 2/2 [01:20<00:00, 40.22s/it]
We've detected an older driver with an RTX 4000 series GPU. These drivers have issues with P2P. This can affect the multi-gpu inference when using accelerate device_map.Please make sure to update your driver to the latest version which resolves this.
12/22 15:18:13 - OpenCompass - INFO - using stop words: ['
', '
', '
']
12/22 15:18:14 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_arabic]
[2024-12-22 15:18:14,186] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 15:18:14,187] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
  0%|                                                                                                           | 0/921 [00:00<?, ?it/s]The `seen_tokens` attribute is deprecated and will be removed in v4.41. Use the `cache_position` model input instead.
`get_max_cache()` is deprecated for all Cache classes. Use `get_max_cache_shape()` instead. Calling `get_max_cache()` will raise error from v4.48
You are not running the flash-attention implementation, expect numerical differences.
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 921/921 [1:47:55<00:00,  7.03s/it]
12/22 17:06:09 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_bengali]
[2024-12-22 17:06:09,609] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 17:06:09,609] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 113/113 [17:35<00:00,  9.34s/it]
12/22 17:23:45 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_english]
[2024-12-22 17:23:45,217] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 17:23:45,217] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 440/440 [12:06<00:00,  1.65s/it]
12/22 17:35:51 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_finnish]
[2024-12-22 17:35:51,916] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 17:35:51,916] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [1:16:55<00:00,  5.90s/it]
12/22 18:52:47 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_indonesian]
[2024-12-22 18:52:47,531] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 18:52:47,531] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 565/565 [28:03<00:00,  2.98s/it]
12/22 19:20:50 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_japanese]
[2024-12-22 19:20:50,824] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 19:20:50,824] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 455/455 [25:30<00:00,  3.36s/it]
12/22 19:46:21 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_korean]
[2024-12-22 19:46:21,344] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 19:46:21,344] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 276/276 [27:02<00:00,  5.88s/it]
12/22 20:13:23 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_russian]
[2024-12-22 20:13:23,953] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 20:13:23,953] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 812/812 [23:46<00:00,  1.76s/it]
12/22 20:37:10 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_swahili]
[2024-12-22 20:37:10,967] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 20:37:10,967] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 499/499 [36:18<00:00,  4.37s/it]
12/22 21:13:29 - OpenCompass - INFO - Start inferencing [Phi-3-mini-128k-instruct_hf/tydiqa-goldp_telugu]
[2024-12-22 21:13:30,003] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-22 21:13:30,003] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
 16%|███████████████▌                                                                               | 110/669 [25:30<2:09:37, 13.91s/it]
Traceback (most recent call last):
  File "/public/zzy/fintuning/opencompass/run.py", line 4, in <module>
    main()
  File "/public/zzy/fintuning/opencompass/opencompass/cli/main.py", line 308, in main
    runner(tasks)
  File "/public/zzy/fintuning/opencompass/opencompass/runners/base.py", line 38, in __call__
    status = self.launch(tasks)
  File "/public/zzy/fintuning/opencompass/opencompass/runners/local.py", line 128, in launch
    task.run(cur_model=getattr(self, 'cur_model',
  File "/public/zzy/fintuning/opencompass/opencompass/tasks/openicl_infer.py", line 89, in run
    self._inference()
  File "/public/zzy/fintuning/opencompass/opencompass/tasks/openicl_infer.py", line 134, in _inference
    inferencer.inference(retriever,
  File "/public/zzy/fintuning/opencompass/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 153, in inference
    results = self.model.generate_from_template(
  File "/public/zzy/fintuning/opencompass/opencompass/models/base.py", line 201, in generate_from_template
    return self.generate(inputs, max_out_len=max_out_len, **kwargs)
  File "/public/zzy/fintuning/opencompass/opencompass/models/huggingface_above_v4_33.py", line 479, in generate
    outputs = self.model.generate(**tokens, **generation_kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/transformers/generation/utils.py", line 2252, in generate
    result = self._sample(
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/transformers/generation/utils.py", line 3251, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/Phi-3-mini-128k-instruct/modeling_phi3.py", line 1286, in forward
    outputs = self.model(
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/Phi-3-mini-128k-instruct/modeling_phi3.py", line 1164, in forward
    layer_outputs = decoder_layer(
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/Phi-3-mini-128k-instruct/modeling_phi3.py", line 885, in forward
    attn_outputs, self_attn_weights, present_key_value = self.self_attn(
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/Phi-3-mini-128k-instruct/modeling_phi3.py", line 405, in forward
    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(value_states.dtype)
  File "/root/anaconda/envs/opencompass/lib/python3.10/site-packages/torch/nn/functional.py", line 2142, in softmax
    ret = input.softmax(dim, dtype=dtype)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.83 GiB. GPU 0 has a total capacity of 23.65 GiB of which 9.06 GiB is free. Process 43896 has 954.00 MiB memory in use. Process 44069 has 13.65 GiB memory in use. Of the allocated memory 12.72 GiB is allocated by PyTorch, and 485.35 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

### 其他信息

_No response_

You can decrease the batch size to avoid out-of-memory

> You can decrease the batch size to avoid out-of-memory

但是我的batch size已经是1了呀


Get, if the OOM is still existing, maybe the tensor parallel is required.