Repository: pytorch
Issue: Data type mismatch in `batch_isend_irecv` docstring example
Labels: oncall: distributed

### 📚 The doc issue

In the example usage of [`batch_isend_irecv`](https://pytorch.org/docs/stable/distributed.html#torch.distributed.batch_isend_irecv):

https://github.com/pytorch/pytorch/blob/6c9b94dcda696984552769da6cf578ccca0cfbdd/torch/distributed/distributed_c10d.py#L1709-L1718

`send_tensor` is `int64` while `recv_tensor` is `float32`. The resulting `recv_tensor` shows incorrect data.

To reproduce:

```python
import os
import torch
import torch.distributed as dist


def main():
    device = "cuda"
    rank = int(os.environ["RANK"])
    local_rank = int(os.environ["LOCAL_RANK"])
    world_size = int(os.environ["WORLD_SIZE"])

    torch.cuda.set_device(local_rank)
    dist.init_process_group(backend="nccl")

    send_tensor = (torch.arange(2) + 2 * rank).to(device)
    recv_tensor = torch.randn(2).to(device)

    print(
        f"(rank {rank}) send_tensor before: {send_tensor}\n"
        f"         recv_tensor before: {recv_tensor}\n",
        end=""
    )

    send_op = dist.P2POp(dist.isend, send_tensor, (rank + 1) % world_size)
    recv_op = dist.P2POp(dist.irecv, recv_tensor, (rank - 1 + world_size) % world_size)
    
    reqs = dist.batch_isend_irecv([send_op, recv_op])
    for req in reqs:
        req.wait()

    torch.cuda.synchronize()
    print(f"(rank {rank}) recv_tensor after: {recv_tensor}\n", end="")

if __name__ == "__main__":
    main()
```

(the extra `synchronize` is to fix #82450 prior to PyTorch 1.13)

`torchrun --nproc_per_node 2 ./batch_isend_irecv_gpu.py` shows

```
(rank 0) recv_tensor after: tensor([2.8026e-45, 0.0000e+00], device='cuda:0')
(rank 1) recv_tensor after: tensor([0., 0.], device='cuda:1')
```

`torchrun --nproc_per_node 8 ./batch_isend_irecv_gpu.py` shows

```
(rank 4) recv_tensor after: tensor([8.4078e-45, 0.0000e+00], device='cuda:4')
(rank 1) recv_tensor after: tensor([0., 0.], device='cuda:1')
(rank 2) recv_tensor after: tensor([2.8026e-45, 0.0000e+00], device='cuda:2')
(rank 3) recv_tensor after: tensor([5.6052e-45, 0.0000e+00], device='cuda:3')
(rank 6) recv_tensor after: tensor([1.4013e-44, 0.0000e+00], device='cuda:6')
(rank 7) recv_tensor after: tensor([1.6816e-44, 0.0000e+00], device='cuda:7')
(rank 5) recv_tensor after: tensor([1.1210e-44, 0.0000e+00], device='cuda:5')
(rank 0) recv_tensor after: tensor([1.9618e-44, 0.0000e+00], device='cuda:0')
```

due to mis-interprete int as float.  NCCL doesn't check datatype mismatch, [unlike MPI](https://clang.llvm.org/extra/clang-tidy/checks/mpi/type-mismatch.html)

### Suggest a potential alternative/fix

Simply change data type to:

```python
send_tensor = (torch.arange(2, dtype=torch.float32) + 2.0 * rank).to(device)
recv_tensor = torch.randn(2, dtype=torch.float32).to(device)
```

Then the result becomes correct:

```
(rank 0) recv_tensor after: tensor([2., 3.], device='cuda:0')
(rank 1) recv_tensor after: tensor([0., 1.], device='cuda:1')
```

```
(rank 1) recv_tensor after: tensor([0., 1.], device='cuda:1')
(rank 0) recv_tensor after: tensor([14., 15.], device='cuda:0')
(rank 7) recv_tensor after: tensor([12., 13.], device='cuda:7')
(rank 5) recv_tensor after: tensor([8., 9.], device='cuda:5')
(rank 3) recv_tensor after: tensor([4., 5.], device='cuda:3')
(rank 6) recv_tensor after: tensor([10., 11.], device='cuda:6')
(rank 4) recv_tensor after: tensor([6., 7.], device='cuda:4')
(rank 2) recv_tensor after: tensor([2., 3.], device='cuda:2')
```


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu