Repository: pytorch
Issue: Tensor Parallel Inference Performance Issue with Llama-7b-chat-hf Model caused by straggler problem
Labels: oncall: distributed

### 🐛 Describe the bug

Hello PyTorch Team,

I am currently attempting to leverage Tensor Parallelism for inferencing the Llama-7b-chat-hf model. However, I've encountered a significant bottleneck related to server straggler issues, leading to an absence of expected speedup from tensor parallelization. The core of the problem seems to lie in the AllReduce operation, where all ranks are forced to wait for the completion of the slowest rank. Consequently, each AllReduce operation incurs a delay exceeding 1ms.

My experimental environment consists of a server equipped with 8 V100 GPUs, interconnected via NVLink. To illustrate the issue, I am providing two nsight profile results comparing the performance of parallel inference across 4 V100 GPUs, with and without compilation. The profiles indicate a consistent issue where one rank experiences significant latency in kernel launching, thereby causing delays in the AllReduce operation for other ranks. The AllReduce operators are highlighted for ease of reference.

Below is the code snippet employed for this setup. Could you please advise if there are any misconfigurations or suggest potential optimizations to mitigate this issue?
![image](https://github.com/pytorch/pytorch/assets/50972467/9363b35d-ed8a-437e-97f5-105d1d553395)
![image](https://github.com/pytorch/pytorch/assets/50972467/1438fd88-38dc-4b20-a2e8-f1aece6c3da8)
Following is the code I use, Is there any wrong configuration of it?

```
import fire
import time
from transformers import LlamaForCausalLM, AutoTokenizer, LlamaTokenizer
import torch
from torch.distributed._tensor import DeviceMesh, DTensor
from torch.distributed.tensor.parallel import (
    parallelize_module,
    ColwiseParallel,
    RowwiseParallel,
)
import torch.distributed as dist
import os
from torch._dynamo.utils import CompileProfiler
import torch.distributed.checkpoint as dist_cp

#from utils import print_submodules, parallelize_llama_MLP_block, tp_llama, _load_tp_checkpoints


def parallelize_llama_MLP_block(model, module_path, mesh):
    block = model.get_submodule(module_path)
    parallelized_block = parallelize_module(
        module=block,
        device_mesh=mesh,
        parallelize_plan={
            "gate_proj": ColwiseParallel(),
            "up_proj": ColwiseParallel(),
            "down_proj": RowwiseParallel(),
        },
        # tp_mesh_dim=0,
    )
    return parallelized_block
def tp_llama(model, mesh):
    for i in range(model.config.num_hidden_layers):
        block = parallelize_llama_MLP_block(model, f"model.layers.{i}.mlp", mesh)


def _load_tp_checkpoints(tp_model, CHECKPOINT_DIR):
    local_rank = int(os.environ.get("LOCAL_RANK", -1))
    if local_rank == -1:
        raise RuntimeError("Expected local_rank to be set, but it is not!")
    tp_state_dict = tp_model.state_dict()
    dist_cp.load_state_dict(
        state_dict=tp_state_dict,
        storage_reader=dist_cp.FileSystemReader(CHECKPOINT_DIR),
    )
    tp_model.load_state_dict(tp_state_dict)


def main(model_name: str = "meta-llama/Llama-2-7b-chat-hf",
         checkpoint_dir: str = None,
         compile: bool = False,
         meta_device: bool = False,
         iteration: int = 1):
    backend = "nccl"
    dist.init_process_group("nccl")
    _rank = int(os.environ["RANK"])
    _local_rank = int(os.environ["LOCAL_RANK"])

    world_size = int(os.environ["WORLD_SIZE"])  # total number of training processes
    device = f"cuda:{_local_rank}"

    torch.cuda.set_device(device)

    # with torch.device("meta"):
    model = LlamaForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, use_cache=True)
    model.to(device)
    tokenizer = LlamaTokenizer.from_pretrained(model_name)
    # model = BetterTransformer.transform(model)
    mesh = (
        DeviceMesh(
            device_type="cuda",
            mesh=list(range(dist.get_world_size())),
        ))

    tp_llama(model, mesh)
    if meta_device:
        model.to_empty(device='cuda')
        _load_tp_checkpoints(model, checkpoint_dir)
    dummy_input = "what is the recipe of ketchup?"
    dummy_input = tokenizer(dummy_input, return_tensors="pt").to(device)

    if compile:
        with CompileProfiler() as prof:
            compiled_model = torch.compile(model, backend="inductor", mode="max-autotune")
            for i in range(iteration):
                t0 = time.perf_counter()
                output = compiled_model.generate(
                    **dummy_input,
                    max_new_tokens=200,
                    do_sample=True,
                    top_p=0.9,
                    temperature=0.6)
                output = tokenizer.decode(output[0], skip_special_tokens=True)
                print(output)
                t = time.perf_counter() - t0
                print(f'Iteration {i}: output token num: {len(output)}, token/s: {len(output) / t}')
                print("====================================================")
                print(prof.report())
    else:
        for i in range(iteration):
            t0 = time.perf_counter()
            output = model.generate(
                **dummy_input,
                max_new_tokens=200,
                do_sample=True,
                top_p=0.9,
                temperature=0.6)
            output = tokenizer.decode(output[0], skip_special_tokens=True)
            print(output)
            t = time.perf_counter() - t0
            print(f'Iteration {i}: output token num: {len(output)}, token/s: {len(output) / t}')
            print("====================================================")
if __name__ == "__main__":
    fire.Fire(main)
```



You can run the code by following command:
```
torchrun --nnodes 1 --nproc_per_node 4 llama-simple.py --model_name=YOUR_MODEL # without compile
torchrun --nnodes 1 --nproc_per_node 4 llama-simple.py --model_name=YOUR_MODEL --compile #with compile 
```


### Versions
```
Collecting environment information...
PyTorch version: 2.3.0.dev20240226
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-2ubuntu1~20.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 520.61.05
cuDNN version: Probably one of the following:
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn.so.8.9.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.6
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   42 bits physical, 48 bits virtual
CPU(s):                          64
On-line CPU(s) list:             0-63
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
Stepping:                        4
CPU MHz:                         3000.000
BogoMIPS:                        6000.00
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       1 MiB
L1i cache:                       1 MiB
L2 cache:                        32 MiB
L3 cache:                        49.5 MiB
NUMA node0 CPU(s):               0-31
NUMA node1 CPU(s):               32-63
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:          Vulnerable
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT Host state unknown
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat md_clear flush_l1d

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.3.0.dev20240226
[pip3] torchaudio==2.2.0.dev20240226
[pip3] torchvision==0.18.0.dev20240226
[pip3] triton==3.0.0
[conda] blas                      1.0                         mkl  
[conda] brotlipy                  0.7.0           py311h9bf148f_1002    pytorch-nightly
[conda] cffi                      1.15.1          py311h9bf148f_3    pytorch-nightly
[conda] cryptography              38.0.4          py311h46ebde7_0    pytorch-nightly
[conda] filelock                  3.9.0                   py311_0    pytorch-nightly
[conda] libjpeg-turbo             2.0.0                h9bf148f_0    pytorch-nightly
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py311h9bf148f_0    pytorch-nightly
[conda] mkl_fft                   1.3.1           py311hc796f24_0    pytorch-nightly
[conda] mkl_random                1.2.2           py311hbba84a0_0    pytorch-nightly
[conda] mpmath                    1.2.1                   py311_0    pytorch-nightly
[conda] numpy                     1.24.3          py311hc206e33_0  
[conda] numpy-base                1.24.3          py311hfd5febd_0  
[conda] pillow                    9.3.0           py311h3fd9d12_2    pytorch-nightly
[conda] pysocks                   1.7.1                   py311_0    pytorch-nightly
[conda] pytorch                   2.3.0.dev20240226 py3.11_cuda11.8_cudnn8.7.0_0    pytorch-nightly
[conda] pytorch-cuda              11.8                 h7e8668a_5    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] requests                  2.28.1                  py311_0    pytorch-nightly
[conda] torchaudio                2.2.0.dev20240226     py311_cu118    pytorch-nightly
[conda] torchtriton               3.0.0+901819d2b6           py311    pytorch-nightly
[conda] torchvision               0.18.0.dev20240226     py311_cu118    pytorch-nightly
[conda] urllib3                   1.26.14                 py311_0    pytorch-nightly
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225