Repository: pytorch
Issue: batch_isend_irecv: the receiving end cannot receive large tensors from the sending end correctly
Labels: oncall: distributed,triaged

## 🐛 Bug

I am using the latest pytorch(1.8 unstable), and found the problem of abnormal data sending/receiving with `batch_isend_irecv` in a simple scenario： two processes act as sender and receiver respectively，when the data is small（tensor_sizes = [10] * 7，7 tensors([0,1,2,...9]) as a batch）, it ok, but when the data reaches a certain large value（tensor_sizes = [100000] * 7,  7 tensor([0,1,2,3, ..., 99999]) as a batch）, it does not work as expectations

## To Reproduce

The simple demo.py

```
import time
import torch
import torch.distributed as dist

dist.init_process_group("nccl")

num_iterations = 10

# small tensor ok
# tensor_sizes = [10] * 7

# large tensor do not work
tensor_sizes = [100000] * 7

rank = dist.get_rank()

groups = []
for i in range(len(tensor_sizes)):
    groups.append(dist.new_group([0, 1]))

if rank == 0:
    torch.cuda.set_device(0)
    
    for j in range(num_iterations):
        tensors = []
        for tensor_size in tensor_sizes:
            tensors.append(torch.tensor(range(tensor_size), dtype=torch.float32).cuda(0))

        start = time.time()
        p2p_op_list = []
        for i, tensor in enumerate(tensors):
            p2p_op_list.append(dist.P2POp(dist.isend, tensor, 1, groups[i]))
        reqs =  dist.batch_isend_irecv(p2p_op_list)
        for req in reqs:
            req.wait()
        print(f"{j} - Send[batch] - Total: {time.time() - start}")


else: # rank = 1
    torch.cuda.set_device(1)

    for j in range(num_iterations):
        tensors = []
        for tensor_size in tensor_sizes:
            tensors.append(torch.zeros(tensor_size).cuda(1))

        start = time.time()
        p2p_op_list = []
        for i, tensor in enumerate(tensors):
            p2p_op_list.append(dist.P2POp(dist.irecv, tensor, 0, groups[i]))

        reqs =  dist.batch_isend_irecv(p2p_op_list)
        for req in reqs:
            req.wait()
        print(f"{j} - Recv[batch] - Total: {time.time() - start}")
        print(tensors)
```

Run as:
```
python -m torch.distributed.launch --nproc_per_node 2   demo.py 
```

We get the abnormal received tensors like below:

```
[tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:1'), tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:1'), tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:1'), tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:1'), tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:1'), tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:1'), tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:1')]
```
## Expected behavior

The receiving end can receive large tensors correctly, received tensors as below:

```
[tensor([0.0000e+00, 1.0000e+00, 2.0000e+00,  ..., 9.9997e+04, 9.9998e+04,
        9.9999e+04], device='cuda:1'), tensor([0.0000e+00, 1.0000e+00, 2.0000e+00,  ..., 9.9997e+04, 9.9998e+04,
        9.9999e+04], device='cuda:1'), tensor([0.0000e+00, 1.0000e+00, 2.0000e+00,  ..., 9.9997e+04, 9.9998e+04,
        9.9999e+04], device='cuda:1'), tensor([0.0000e+00, 1.0000e+00, 2.0000e+00,  ..., 9.9997e+04, 9.9998e+04,
        9.9999e+04], device='cuda:1'), tensor([0.0000e+00, 1.0000e+00, 2.0000e+00,  ..., 9.9997e+04, 9.9998e+04,
        9.9999e+04], device='cuda:1'), tensor([0.0000e+00, 1.0000e+00, 2.0000e+00,  ..., 9.9997e+04, 9.9998e+04,
        9.9999e+04], device='cuda:1'), tensor([0.0000e+00, 1.0000e+00, 2.0000e+00,  ..., 9.9997e+04, 9.9998e+04,
        9.9999e+04], device='cuda:1')]
```

## Environment

Collecting environment information...
PyTorch version: 1.8.0a0+a5b27d7
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.3 LTS (x86_64)
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Clang version: Could not collect
CMake version: version 3.18.2

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 418.67
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] numpy==1.17.4
[pip] numpydoc==1.0.0
[pip] pytorch-pretrained-bert==0.6.2
[pip] pytorch-transformers==1.1.0
[pip] torch==1.8.0a0
[pip] torchvision==0.5.0
[pip] ztorch==1.9.1
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.1.243 h6bb024c_0
[conda] magma-cuda110 2.5.2 1 pytorch
[conda] mkl 2019.4 243
[conda] mkl-include 2020.2 256
[conda] mkl-service 2.3.0 py37he904b0f_0
[conda] mkl_fft 1.0.15 py37ha843d7b_0
[conda] mkl_random 1.1.0 py37hd6b4f25_0
[conda] numpy 1.17.4 py37hc1035e2_0
[conda] numpy-base 1.17.4 py37hde5b4d6_0
[conda] numpydoc 1.0.0 pypi_0 pypi
[conda] pytorch 1.4.0 py3.7_cuda10.1.243_cudnn7.6.3_0 pytorch
[conda] pytorch-pretrained-bert 0.6.2 pypi_0 pypi
[conda] pytorch-transformers 1.1.0 pypi_0 pypi
[conda] torch 1.8.0a0 dev_0
[conda] torchvision 0.5.0 py37_cu101 pytorch
[conda] ztorch 1.9.1 pypi_0 pypi



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu