Repository: pytorch
Issue: WSL2 RTX A6000 , CUDA out of memory.
Labels: module: windows,module: cuda,module: memory usage,triaged

### 🐛 Describe the bug

I fine-tuned and inferred Qwen-14B-Chat using LLaMA Factory. I constantly encounter out-of-memory issues in WSL2, and it can only run in a Windows environment. The following is the screen output during inference:
(base) wxy@DESKTOP-FEGFEAL:/mnt/d/LLaMA-Factory$ conda activate llmfactory
(llmfactory) wxy@DESKTOP-FEGFEAL:/mnt/d/LLaMA-Factory$ sh web_demo_qwen.sh
[INFO|tokenization_utils_base.py:2024] 2024-01-11 08:49:21,006 >> loading file qwen.tiktoken
[INFO|tokenization_utils_base.py:2024] 2024-01-11 08:49:21,006 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2024] 2024-01-11 08:49:21,006 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2024] 2024-01-11 08:49:21,006 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2024] 2024-01-11 08:49:21,006 >> loading file tokenizer.json
[INFO|configuration_utils.py:737] 2024-01-11 08:49:21,197 >> loading configuration file models/Qwen-14B-Chat/config.json
[INFO|configuration_utils.py:737] 2024-01-11 08:49:21,208 >> loading configuration file models/Qwen-14B-Chat/config.json
[INFO|configuration_utils.py:802] 2024-01-11 08:49:21,209 >> Model config QWenConfig {
  "_name_or_path": "models/Qwen-14B-Chat",
  "architectures": [
    "QWenLMHeadModel"
  ],
  "attn_dropout_prob": 0.0,
  "auto_map": {
    "AutoConfig": "configuration_qwen.QWenConfig",
    "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
  },
  "bf16": false,
  "emb_dropout_prob": 0.0,
  "fp16": false,
  "fp32": false,
  "hidden_size": 5120,
  "initializer_range": 0.02,
  "intermediate_size": 27392,
  "kv_channels": 128,
  "layer_norm_epsilon": 1e-06,
  "max_position_embeddings": 8192,
  "model_type": "qwen",
  "no_bias": true,
  "num_attention_heads": 40,
  "num_hidden_layers": 40,
  "onnx_safe": null,
  "rotary_emb_base": 10000,
  "rotary_pct": 1.0,
  "scale_attn_weights": true,
  "seq_length": 2048,
  "softmax_in_fp32": false,
  "tie_word_embeddings": false,
  "tokenizer_class": "QWenTokenizer",
  "transformers_version": "4.36.2",
  "use_cache": true,
  "use_cache_kernel": false,
  "use_cache_quantization": false,
  "use_dynamic_ntk": true,
  "use_flash_attn": "auto",
  "use_logn_attn": true,
  "vocab_size": 152064
}

[INFO|modeling_utils.py:3341] 2024-01-11 08:49:21,251 >> loading weights file models/Qwen-14B-Chat/model.safetensors.index.json
[INFO|modeling_utils.py:1341] 2024-01-11 08:49:21,255 >> Instantiating QWenLMHeadModel model under default dtype torch.float16.
[INFO|configuration_utils.py:826] 2024-01-11 08:49:21,255 >> Generate config GenerationConfig {}

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [01:42<00:00,  6.82s/it]
[INFO|modeling_utils.py:4185] 2024-01-11 08:51:03,756 >> All model checkpoint weights were used when initializing QWenLMHeadModel.

[INFO|modeling_utils.py:4193] 2024-01-11 08:51:03,756 >> All the weights of QWenLMHeadModel were initialized from the model checkpoint at models/Qwen-14B-Chat.
If your task is similar to the task the model of the checkpoint was trained on, you can already use QWenLMHeadModel for predictions without further training.
[INFO|configuration_utils.py:779] 2024-01-11 08:51:03,761 >> loading configuration file models/Qwen-14B-Chat/generation_config.json
[INFO|configuration_utils.py:826] 2024-01-11 08:51:03,761 >> Generate config GenerationConfig {
  "chat_format": "chatml",
  "do_sample": true,
  "eos_token_id": 151643,
  "max_new_tokens": 512,
  "max_window_size": 6144,
  "pad_token_id": 151643,
  "repetition_penalty": 1.1,
  "top_k": 0,
  "top_p": 0.8
}

01/11/2024 08:51:03 - INFO - llmtuner.model.adapter - Fine-tuning method: LoRA
01/11/2024 08:51:34 - INFO - llmtuner.model.adapter - Merged 1 adapter(s).
01/11/2024 08:51:34 - INFO - llmtuner.model.adapter - Loaded adapter(s): saves/Qwen-14B-Chat/lora/checkpoint
01/11/2024 08:51:34 - INFO - llmtuner.model.loader - trainable params: 0 || all params: 14167290880 || trainable%: 0.0000
01/11/2024 08:51:34 - INFO - llmtuner.model.loader - This IS expected that the trainable params is 0 if you are using model for inference only.
Traceback (most recent call last):
  File "/mnt/d/LLaMA-Factory/src/web_demo.py", line 11, in <module>
    main()
  File "/mnt/d/LLaMA-Factory/src/web_demo.py", line 5, in main
    demo = create_web_demo()
           ^^^^^^^^^^^^^^^^^
  File "/mnt/d/LLaMA-Factory/src/llmtuner/webui/interface.py", line 59, in create_web_demo
    engine = Engine(pure_chat=True)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/LLaMA-Factory/src/llmtuner/webui/engine.py", line 20, in __init__
    self.chatter = WebChatModel(manager=self.manager, demo_mode=demo_mode, lazy_init=(not pure_chat))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/LLaMA-Factory/src/llmtuner/webui/chatter.py", line 30, in __init__
    super().__init__()
  File "/mnt/d/LLaMA-Factory/src/llmtuner/chat/chat_model.py", line 31, in __init__
    self.model = dispatch_model(self.model)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/LLaMA-Factory/src/llmtuner/model/utils.py", line 46, in dispatch_model
    return model.to(device=get_current_device())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wxy/anaconda3/envs/llmfactory/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2460, in to
    return super().to(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wxy/anaconda3/envs/llmfactory/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1160, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/wxy/anaconda3/envs/llmfactory/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
    module._apply(fn)
  File "/home/wxy/anaconda3/envs/llmfactory/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
    module._apply(fn)
  File "/home/wxy/anaconda3/envs/llmfactory/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "/home/wxy/anaconda3/envs/llmfactory/lib/python3.11/site-packages/torch/nn/modules/module.py", line 833, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "/home/wxy/anaconda3/envs/llmfactory/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1158, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 50.00 MiB. GPU 0 has a total capacty of 47.99 GiB of which 23.25 GiB is free. Including non-PyTorch memory, 
this process has 17179869184.00 GiB memory in use. Of the allocated memory 22.77 GiB is allocated by PyTorch, and 1.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF


### Versions

Collecting environment information...
PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA RTX A6000
Nvidia driver version: 537.70
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.6
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             16
On-line CPU(s) list:                0-15
Vendor ID:                          GenuineIntel
Model name:                         13th Gen Intel(R) Core(TM) i9-13900K
CPU family:                         6
Model:                              183
Thread(s) per core:                 2
Core(s) per socket:                 8
Socket(s):                          1
Stepping:                           1
BogoMIPS:                           5990.39
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm consta
nt_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand h
ypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm serialize flush_l1d arch_capabilities
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.2
[pip3] torch==2.1.2
[pip3] torchaudio==2.1.2
[pip3] torchvision==0.16.2
[pip3] triton==2.1.0
[conda] numpy                     1.26.2                   pypi_0    pypi
[conda] torch                     2.1.2                    pypi_0    pypi
[conda] torchaudio                2.1.2                    pypi_0    pypi
[conda] torchvision               0.16.2                   pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypi


cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @ptrblck