Repository: pytorch
Issue: torch.distributed.DistBackendError: NCCL error
Labels: oncall: distributed

### 🐛 Describe the bug

https://huggingface.co/docs/transformers/run_scripts#distributed-training-and-mixed-precision

```
(hfenv) isharma@VMNUVOCINPOC:~/workspace/ground0/repos/transformers/examples$ torchrun     --nproc_per_node 4 pytorch/summarization/run_summarization.py     --fp16     --model_name_or_path google-t5/t5-small     --do_train     --do_eval     --dataset_name cnn_dailymail     --dataset_config "3.0.0"     --source_prefix "summarize: "     --output_dir /tmp/tst-summarization     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir     --predict_with_generate
[2024-03-05 07:01:31,834] torch.distributed.run: [WARNING] 
[2024-03-05 07:01:31,834] torch.distributed.run: [WARNING] *****************************************
[2024-03-05 07:01:31,834] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-03-05 07:01:31,834] torch.distributed.run: [WARNING] *****************************************
03/05/2024 07:01:37 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: True
03/05/2024 07:01:37 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_steps=None,
evaluation_strategy=no,
fp16=True,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
generation_config=None,
generation_max_length=None,
generation_num_beams=None,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/tmp/tst-summarization/runs/Mar05_07-01-35_VMNUVOCINPOC,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=linear,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
optim=adamw_torch,
optim_args=None,
output_dir=/tmp/tst-summarization,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=4,
per_device_train_batch_size=4,
predict_with_generate=True,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard', 'wandb'],
resume_from_checkpoint=None,
run_name=/tmp/tst-summarization,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
sortish_sampler=False,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
03/05/2024 07:01:37 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, 16-bits training: True
03/05/2024 07:01:37 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1, distributed training: True, 16-bits training: True
03/05/2024 07:01:37 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1, distributed training: True, 16-bits training: True
Overwrite dataset info from restored data version if exists.
03/05/2024 07:01:46 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
Loading Dataset info from /home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d
03/05/2024 07:01:46 - INFO - datasets.info - Loading Dataset info from /home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d
Found cached dataset cnn_dailymail (/home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d)
03/05/2024 07:01:46 - INFO - datasets.builder - Found cached dataset cnn_dailymail (/home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d)
Loading Dataset info from /home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d
03/05/2024 07:01:46 - INFO - datasets.info - Loading Dataset info from /home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d
[INFO|configuration_utils.py:726] 2024-03-05 07:01:46,678 >> loading configuration file config.json from cache at /home/isharma/.cache/huggingface/hub/models--google-t5--t5-small/snapshots/df1b051c49625cf57a3d0d8d3863ed4d13564fe4/config.json
[INFO|configuration_utils.py:789] 2024-03-05 07:01:46,681 >> Model config T5Config {
  "_name_or_path": "google-t5/t5-small",
  "architectures": [
    "T5ForConditionalGeneration"
  ],
  "classifier_dropout": 0.0,
  "d_ff": 2048,
  "d_kv": 64,
  "d_model": 512,
  "decoder_start_token_id": 0,
  "dense_act_fn": "relu",
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "feed_forward_proj": "relu",
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "is_gated_act": false,
  "layer_norm_epsilon": 1e-06,
  "model_type": "t5",
  "n_positions": 512,
  "num_decoder_layers": 6,
  "num_heads": 8,
  "num_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "relative_attention_max_distance": 128,
  "relative_attention_num_buckets": 32,
  "task_specific_params": {
    "summarization": {
      "early_stopping": true,
      "length_penalty": 2.0,
      "max_length": 200,
      "min_length": 30,
      "no_repeat_ngram_size": 3,
      "num_beams": 4,
      "prefix": "summarize: "
    },
    "translation_en_to_de": {
      "early_stopping": true,
      "max_length": 300,
      "num_beams": 4,
      "prefix": "translate English to German: "
    },
    "translation_en_to_fr": {
      "early_stopping": true,
      "max_length": 300,
      "num_beams": 4,
      "prefix": "translate English to French: "
    },
    "translation_en_to_ro": {
      "early_stopping": true,
      "max_length": 300,
      "num_beams": 4,
      "prefix": "translate English to Romanian: "
    }
  },
  "transformers_version": "4.39.0.dev0",
  "use_cache": true,
  "vocab_size": 32128
}

[INFO|tokenization_utils_base.py:2057] 2024-03-05 07:01:46,907 >> loading file spiece.model from cache at /home/isharma/.cache/huggingface/hub/models--google-t5--t5-small/snapshots/df1b051c49625cf57a3d0d8d3863ed4d13564fe4/spiece.model
[INFO|tokenization_utils_base.py:2057] 2024-03-05 07:01:46,908 >> loading file tokenizer.json from cache at /home/isharma/.cache/huggingface/hub/models--google-t5--t5-small/snapshots/df1b051c49625cf57a3d0d8d3863ed4d13564fe4/tokenizer.json
[INFO|tokenization_utils_base.py:2057] 2024-03-05 07:01:46,908 >> loading file added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:2057] 2024-03-05 07:01:46,908 >> loading file special_tokens_map.json from cache at None
[INFO|tokenization_utils_base.py:2057] 2024-03-05 07:01:46,908 >> loading file tokenizer_config.json from cache at /home/isharma/.cache/huggingface/hub/models--google-t5--t5-small/snapshots/df1b051c49625cf57a3d0d8d3863ed4d13564fe4/tokenizer_config.json
[INFO|modeling_utils.py:3262] 2024-03-05 07:01:47,126 >> loading weights file model.safetensors from cache at /home/isharma/.cache/huggingface/hub/models--google-t5--t5-small/snapshots/df1b051c49625cf57a3d0d8d3863ed4d13564fe4/model.safetensors
[INFO|configuration_utils.py:845] 2024-03-05 07:01:47,136 >> Generate config GenerationConfig {
  "decoder_start_token_id": 0,
  "eos_token_id": 1,
  "pad_token_id": 0
}

[INFO|modeling_utils.py:3998] 2024-03-05 07:01:47,322 >> All model checkpoint weights were used when initializing T5ForConditionalGeneration.

[INFO|modeling_utils.py:4006] 2024-03-05 07:01:47,322 >> All the weights of T5ForConditionalGeneration were initialized from the model checkpoint at google-t5/t5-small.
If your task is similar to the task the model of the checkpoint was trained on, you can already use T5ForConditionalGeneration for predictions without further training.
[INFO|configuration_utils.py:800] 2024-03-05 07:01:47,544 >> loading configuration file generation_config.json from cache at /home/isharma/.cache/huggingface/hub/models--google-t5--t5-small/snapshots/df1b051c49625cf57a3d0d8d3863ed4d13564fe4/generation_config.json
[INFO|configuration_utils.py:845] 2024-03-05 07:01:47,544 >> Generate config GenerationConfig {
  "decoder_start_token_id": 0,
  "eos_token_id": 1,
  "pad_token_id": 0
}

Loading cached processed dataset at /home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d/cache-abe0fb5ccd835b8e.arrow
03/05/2024 07:01:47 - INFO - datasets.arrow_dataset - Loading cached processed dataset at /home/isharma/.cache/huggingface/datasets/cnn_dailymail/3.0.0/0.0.0/96df5e686bee6baa90b8bee7c28b81fa3fa6223d/cache-abe0fb5ccd835b8e.arrow
Traceback (most recent call last):
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 788, in <module>
    main()
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 596, in main
    with training_args.main_process_first(desc="train dataset map pre-processing"):
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/transformers/training_args.py", line 2189, in main_process_first
    dist.barrier()
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3439, in barrier
    work = default_pg.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: /opt/conda/conda-bld/pytorch_1708025847130/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, internal error - please report this issue to the NCCL developers, NCCL version 2.19.3
ncclInternalError: Internal check failed.
Last error:
Attribute busid of node nic not found
Traceback (most recent call last):
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 788, in <module>
    main()
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 596, in main
    with training_args.main_process_first(desc="train dataset map pre-processing"):
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/transformers/training_args.py", line 2180, in main_process_first
    dist.barrier()
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3439, in barrier
    work = default_pg.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: /opt/conda/conda-bld/pytorch_1708025847130/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, internal error - please report this issue to the NCCL developers, NCCL version 2.19.3
ncclInternalError: Internal check failed.
Last error:
Attribute busid of node nic not found
Traceback (most recent call last):
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 788, in <module>
    main()
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 596, in main
    with training_args.main_process_first(desc="train dataset map pre-processing"):
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/transformers/training_args.py", line 2180, in main_process_first
    dist.barrier()
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3439, in barrier
    work = default_pg.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: /opt/conda/conda-bld/pytorch_1708025847130/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, internal error - please report this issue to the NCCL developers, NCCL version 2.19.3
ncclInternalError: Internal check failed.
Last error:
Attribute busid of node nic not found
Traceback (most recent call last):
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 788, in <module>
    main()
  File "/home/isharma/workspace/ground0/repos/transformers/examples/pytorch/summarization/run_summarization.py", line 596, in main
    with training_args.main_process_first(desc="train dataset map pre-processing"):
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/transformers/training_args.py", line 2180, in main_process_first
    dist.barrier()
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3439, in barrier
    work = default_pg.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: /opt/conda/conda-bld/pytorch_1708025847130/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, internal error - please report this issue to the NCCL developers, NCCL version 2.19.3
ncclInternalError: Internal check failed.
Last error:
Attribute busid of node nic not found
[2024-03-05 07:01:51,863] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 69811) of binary: /home/isharma/miniconda3/envs/hfenv/bin/python
Traceback (most recent call last):
  File "/home/isharma/miniconda3/envs/hfenv/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.2.1', 'console_scripts', 'torchrun')())
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/isharma/miniconda3/envs/hfenv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
pytorch/summarization/run_summarization.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-03-05_07:01:51
  host      : vmnuvocinpoc.internal.cloudapp.net
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 69812)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-03-05_07:01:51
  host      : vmnuvocinpoc.internal.cloudapp.net
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 69813)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-03-05_07:01:51
  host      : vmnuvocinpoc.internal.cloudapp.net
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 69814)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-03-05_07:01:51
  host      : vmnuvocinpoc.internal.cloudapp.net
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 69811)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
```

### Versions

```
Collecting environment information...
PyTorch version: 2.2.1
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-1015-azure-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe

Nvidia driver version: 550.54.14
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             96
On-line CPU(s) list:                0-95
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7V13 64-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 1
Core(s) per socket:                 48
Socket(s):                          2
Stepping:                           1
BogoMIPS:                           4890.87
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          3 MiB (96 instances)
L1i cache:                          3 MiB (96 instances)
L2 cache:                           48 MiB (96 instances)
L3 cache:                           384 MiB (12 instances)
NUMA node(s):                       4
NUMA node0 CPU(s):                  0-23
NUMA node1 CPU(s):                  24-47
NUMA node2 CPU(s):                  48-71
NUMA node3 CPU(s):                  72-95
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pytorch-triton==3.0.0+901819d2b6
[pip3] torch==2.2.1
[pip3] torchaudio==2.2.1
[pip3] torchvision==0.17.1
[pip3] triton==2.2.0
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
[conda] mkl                       2023.1.0         h213fc3f_46344  
[conda] mkl-service               2.4.0           py310h5eee18b_1  
[conda] mkl_fft                   1.3.8           py310h5eee18b_0  
[conda] mkl_random                1.2.4           py310hdb19cb5_0  
[conda] numpy                     1.26.4          py310h5f9d8c6_0  
[conda] numpy-base                1.26.4          py310hb5e798b_0  
[conda] pytorch                   2.2.1           py3.10_cuda12.1_cudnn8.9.2_0    pytorch
[conda] pytorch-cuda              12.1                 ha16c6d3_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-triton            3.0.0+901819d2b6          pypi_0    pypi
[conda] torchaudio                2.2.1               py310_cu121    pytorch
[conda] torchtriton               2.2.0                     py310    pytorch
[conda] torchvision               0.17.1              py310_cu121    pytorch
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang