Repository: pytorch
Issue: When dist.broadcast float32 to int64, it will silently generate wrong results
Labels: oncall: distributed

### 🐛 Describe the bug

I find that if we use torch.distributed.broadcast, if the tensor on another GPU has a different dtype, it will silently generate wrong results.
```python
def run(rank, world_size=2):
    dist.init_process_group('gloo', rank=rank, world_size=world_size)
    if rank == 0:
        n_id = torch.arange(6.).to(rank)
        print(n_id.dtype)
    else:
        n_id = torch.zeros(14, dtype=torch.int64).to(rank)
    dist.broadcast(n_id, src=0)
    print(f'rank {rank} {n_id} ')
```

Outputs:
```
torch.float32
rank 1 tensor([4575657221408423936, 4629700418010611712, 4656722015783223296,
                          0,                   0,                   0,
                          0,                   0,                   0,
                          0,                   0,                   0,
                          0,                   0], device='cuda:1')
rank 0 tensor([0., 1., 2., 3., 4., 5.], device='cuda:0')
```

Maybe we could add a dtype sanity check on dist.broadcast and other distribution functions.

### Versions

GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[pip3] torch-geometric==2.1.0.post1
[pip3] torch-quiver==0.1.0
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.13
[pip3] torchelastic==0.2.0
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               ha36c431_9    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.3.0           h06a4308_520
[conda] mkl-service               2.4.0            py37h7f8727e_0
[conda] mkl_fft                   1.3.1            py37hd3c417c_0
[conda] mkl_random                1.2.2            py37h51133e4_0
[conda] numpy                     1.21.2           py37h20f2e39_0
[conda] numpy-base                1.21.2           py37h79a1101_0
[conda] pytorch                   1.10.0          py3.7_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch-geometric           2.1.0.post1              pypi_0    pypi
[conda] torch-quiver              0.1.0                    pypi_0    pypi
[conda] torch-scatter             2.0.9                    pypi_0    pypi
[conda] torch-sparse              0.6.13                   pypi_0    pypi
[conda] torchelastic              0.2.0                    pypi_0    pypi
[conda] torchtext                 0.11.0                     py37    pytorch
[conda] torchvision               0.11.0               py37_cu113    pytorch

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu