Repository: pytorch
Issue: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled cuda error (run with NCCL_DEBUG=INFO for details)
Labels: oncall: distributed,triaged,module: wsl

### 🐛 Describe the bug

When I try to finetune with ddp([LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)) in wsl2(win10 host), I get this error:
```
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO Bootstrap : Using eth0:172.23.125.43<0>
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO cudaDriverVersion 11080
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO cudaDriverVersion 11080
NCCL version 2.18.5+cuda11.8
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO Bootstrap : Using eth0:172.23.125.43<0>
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO NET/IB : No device found.
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO NET/Socket : Using [0]eth0:172.23.125.43<0>
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO Using network Socket
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO NET/IB : No device found.
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO NET/Socket : Using [0]eth0:172.23.125.43<0>
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO Using network Socket
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO comm 0x7aef8cc0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 4000 commId 0xfab41a312f23304f - Init START
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO comm 0x7e10b200 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 1000 commId 0xfab41a312f23304f - Init START
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO NCCL_IGNORE_DISABLED_P2P set by environment to 1.
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO NCCL_IGNORE_DISABLED_P2P set by environment to 1.
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO Channel 00/02 :    0   1
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO Channel 01/02 :    0   1
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO P2P Chunksize set to 131072
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO P2P Chunksize set to 131072
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct

DESKTOP-VMBL43V:1354:1395 [0] transport.cc:154 NCCL WARN Cuda failure 'invalid argument'
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO init.cc:1079 -> 1
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO init.cc:1358 -> 1
DESKTOP-VMBL43V:1354:1395 [0] NCCL INFO group.cc:65 -> 1 [Async thread]
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO group.cc:406 -> 1
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO group.cc:96 -> 1

DESKTOP-VMBL43V:1355:1396 [1] transport.cc:154 NCCL WARN Cuda failure 'invalid argument'
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO init.cc:1079 -> 1
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO init.cc:1358 -> 1
DESKTOP-VMBL43V:1355:1396 [1] NCCL INFO group.cc:65 -> 1 [Async thread]
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO group.cc:406 -> 1
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO group.cc:96 -> 1
Traceback (most recent call last):
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/train_bash.py", line 14, in <module>
    main()
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/train_bash.py", line 5, in main
    run_exp()
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/llmtuner/tuner/tune.py", line 26, in run_exp
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/llmtuner/tuner/sft/workflow.py", line 29, in run_sft
    dataset = preprocess_dataset(dataset, tokenizer, data_args, training_args, stage="sft")
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/llmtuner/dsets/preprocess.py", line 216, in preprocess_dataset
    with training_args.main_process_first(desc="dataset map pre-processing"):
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/transformers/training_args.py", line 2084, in main_process_first
    dist.barrier()
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
    work = default_pg.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 'invalid argument'
Traceback (most recent call last):
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/train_bash.py", line 14, in <module>
    main()
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/train_bash.py", line 5, in main
    run_exp()
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/llmtuner/tuner/tune.py", line 26, in run_exp
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/llmtuner/tuner/sft/workflow.py", line 29, in run_sft
    dataset = preprocess_dataset(dataset, tokenizer, data_args, training_args, stage="sft")
  File "/mnt/c/Users/marco/Desktop/finetunegpl/LLaMA-Efficient-Tuning/src/llmtuner/dsets/preprocess.py", line 216, in preprocess_dataset
    with training_args.main_process_first(desc="dataset map pre-processing"):
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/transformers/training_args.py", line 2093, in main_process_first
    dist.barrier()
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
    work = default_pg.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 'invalid argument'
DESKTOP-VMBL43V:1355:1355 [1] NCCL INFO comm 0x7aef8cc0 rank 1 nranks 2 cudaDev 1 busId 4000 - Abort COMPLETE
DESKTOP-VMBL43V:1354:1354 [0] NCCL INFO comm 0x7e10b200 rank 0 nranks 2 cudaDev 0 busId 1000 - Abort COMPLETE
[2023-10-13 07:17:04,991] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1354) of binary: /home/marco/miniconda3/envs/lama/bin/python
Traceback (most recent call last):
  File "/home/marco/miniconda3/envs/lama/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
    multi_gpu_launcher(args)
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/marco/miniconda3/envs/lama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
src/train_bash.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-10-13_07:17:04
  host      : DESKTOP-VMBL43V.
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 1355)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-10-13_07:17:04
  host      : DESKTOP-VMBL43V.
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1354)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

```
the command I use is `accelerate launch src/train_bash.py --stage sft --model_name_or_path /mnt/c/Users/marco/Desktop/finetunegpl/sheep-duck-llama-2-13b/ --do_train --finetuning_type lora --template default --flash_attn  --shift_attn  --dataset_dir data --dataset lima --cutoff_len 1024 --learning_rate 5e-05 --num_train_epochs 3.0 --max_samples 100000 --per_device_train_batch_size 4 --gradient_accumulation_steps 4 --lr_scheduler_type cosine --max_grad_norm 1.0 --logging_steps 5 --save_steps 100 --warmup_steps 0 --lora_rank 128 --lora_dropout 0.1 --lora_target q_proj,v_proj --resume_lora_training  --output_dir saves\Custom\lora\2023-10-13-02-59-59 --fp16  --plot_loss`

### Versions

PyTorch version: 2.1.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090 Ti
GPU 1: NVIDIA GeForce RTX 3090 Ti

Nvidia driver version: 522.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          12
On-line CPU(s) list:             0-11
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen 5 5500
CPU family:                      25
Model:                           80
Thread(s) per core:              2
Core(s) per socket:              6
Socket(s):                       1
Stepping:                        0
BogoMIPS:                        7599.71
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat umip vaes vpclmulqdq rdpid fsrm
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       192 KiB (6 instances)
L1i cache:                       192 KiB (6 instances)
L2 cache:                        3 MiB (6 instances)
L3 cache:                        16 MiB (1 instance)
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.1.0+cu118
[pip3] torchaudio==2.1.0+cu118
[pip3] torchvision==0.16.0+cu118
[pip3] triton==2.1.0
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] torch                     2.1.0+cu118              pypi_0    pypi
[conda] torchaudio                2.1.0+cu118              pypi_0    pypi
[conda] torchvision               0.16.0+cu118             pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypi

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite