Repository: pytorch
Issue: Distributed broadcast fails with simple GPU tensor on Windows + GLOO
Labels: oncall: distributed,module: c10d

### 🐛 Describe the bug

OS: Windows 10
Environment created using conda, CUDA 11.4 is installed system-wide, torch 1.10.1 is installed using pip.
Here are the repro steps:
```python
Python 3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> import torch
>>> import os
>>> import torch.distributed as dist
>>> torch.__version__
'1.10.1+cu113'

>>> os.environ['MASTER_ADDR'] = '127.0.0.1'
>>> os.environ['MASTER_PORT'] = '27501'
>>> os.environ['LOCAL_RANK'] = '0'
>>> os.environ['RANK'] = '0'
>>> os.environ['WORLD_SIZE'] = '1'

>>> dist.init_process_group(backend='gloo')
>>> g = dist.new_group([0])

>>> m = torch.nn.Linear(16, 16)
>>> m.cuda()
Linear(in_features=16, out_features=16, bias=True)

>>> p = list(m.parameters())[0]
>>> p.device
device(type='cuda', index=0)

>>> dist.broadcast(p, 0, group=g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "___\miniconda3\envs\torch\lib\site-packages\torch\distributed\distributed_c10d.py", line 1167, in broadcast
    work.wait()
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
```

The error does not happen with CPU tensors (e.g. if you skip `m.cuda()`)

If this is not a bug, is there something special to do on Windows for Gloo's `broadcast` to work with GPU tensors?

The same issue reproduces on a similar environment with PyTorch 1.8.2+cu111 and on Windows 11.

### Versions

PyTorch version: 1.10.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: 13.0.0 (https://github.com/llvm/llvm-project.git d7b669b3a30345cfcdb2fde2af6f48aa4b94845d)
CMake version: version 3.21.21080301-MSVC_2
Libc version: N/A

Python version: 3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19044-SP0
Is CUDA available: True
CUDA runtime version: 11.4.100
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090

Nvidia driver version: 497.29
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] torch==1.10.1+cu113
[pip3] torch-tb-profiler==0.2.1
[pip3] torchaudio==0.10.1+cu113
[pip3] torchvision==0.11.2+cu113
[conda] libblas                   3.9.0              11_win64_mkl    conda-forge
[conda] libcblas                  3.9.0              11_win64_mkl    conda-forge
[conda] liblapack                 3.9.0              11_win64_mkl    conda-forge
[conda] mkl                       2021.3.0           hb70f87d_564    conda-forge
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.21.0                   pypi_0    pypi
[conda] torch                     1.10.1+cu113             pypi_0    pypi
[conda] torch-tb-profiler         0.2.1                    pypi_0    pypi
[conda] torchaudio                0.10.1+cu113             pypi_0    pypi
[conda] torchvision               0.11.2+cu113             pypi_0    pypi

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang