Repository: pytorch
Issue: 1.11.0 distribution train different with 1.8.1
Labels: oncall: distributed

If you have a question or would like help and support, please ask at our
[forums](https://discuss.pytorch.org/).

If you are submitting a feature request, please preface the title with [feature request].
If you are submitting a bug report, please fill in the following details.

## Issue description
**I use 1.11.0 run distribution and gpu use like**
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     23723      C   .../hubo/install/bin/python3    22143MiB |
|    0   N/A  N/A     23724      C   .../hubo/install/bin/python3     1411MiB |
|    0   N/A  N/A     23725      C   .../hubo/install/bin/python3     1409MiB |
|    0   N/A  N/A     23726      C   .../hubo/install/bin/python3     1409MiB |
|    0   N/A  N/A     23727      C   .../hubo/install/bin/python3     1411MiB |
|    0   N/A  N/A     23728      C   .../hubo/install/bin/python3     1409MiB |
|    0   N/A  N/A     23729      C   .../hubo/install/bin/python3     1409MiB |
|    0   N/A  N/A     23730      C   .../hubo/install/bin/python3     1409MiB |
|    1   N/A  N/A     23724      C   .../hubo/install/bin/python3    20841MiB |
|    2   N/A  N/A     23725      C   .../hubo/install/bin/python3    22977MiB |
|    3   N/A  N/A     23726      C   .../hubo/install/bin/python3    23971MiB |
|    4   N/A  N/A     23727      C   .../hubo/install/bin/python3    13683MiB |
|    5   N/A  N/A     23728      C   .../hubo/install/bin/python3    29723MiB |
|    6   N/A  N/A     23729      C   .../hubo/install/bin/python3    16557MiB |
|    7   N/A  N/A     23730      C   .../hubo/install/bin/python3    19963MiB |

**but I use 1.8.1 gpu use like**
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     36861      C   /data/hubo/install/bin/python3             27377MiB |
|    1     36862      C   /data/hubo/install/bin/python3             23477MiB |
|    2     36863      C   /data/hubo/install/bin/python3             24645MiB |
|    3     36864      C   /data/hubo/install/bin/python3             30633MiB |
|    4     36865      C   /data/hubo/install/bin/python3             21605MiB |
|    5     36866      C   /data/hubo/install/bin/python3             23303MiB |
|    6     36867      C   /data/hubo/install/bin/python3             28221MiB |
|    7     36868      C   /data/hubo/install/bin/python3             24285MiB |

I want to know how to set 1.11.0 make other gpu don't run in gpu 0
Provide a short description.

## Code example

Please try to provide a minimal example to repro the bug.
Error messages and stack traces are also helpful.

## System Info
Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

- PyTorch or Caffe2:
- How you installed PyTorch (conda, pip, source):
- Build command you used (if compiling from source):
- OS:
- PyTorch version:3.9.12
- Python version:1.11.0
- CUDA/cuDNN version:11.3
- GPU models and configuration:v100
- GCC version (if compiling from source):10.1.0
- CMake version: None
- Versions of any other relevant libraries:


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang