Repository: pytorch
Issue: Increasing RTT when using rpc_async in a PS architecture setup
Labels: oncall: distributed,triaged

## 🐛 Bug

When using a PS architecture setup for aggregating tensors, I notice that the time interval between sending a rpc_async request and when the callback is triggered increases. Given that there is only 1 worker and the PS is essentially just sending back the tensor as soon as it receives it, I would expect the RTT to be the same for all rpc_async calls.

Am I missing something? Why do I see an increase in the RTT?

## To Reproduce
Run the following gist with --num_buckets --use_cuda_tensors --bucket_size(in MB)
https://gist.github.com/anj-s/87fcd2b504b29642e0fc00639e776d8e

Timing info with --num_buckets=20 --bucket_size=10 --use_cuda_tensors
https://gist.github.com/anj-s/abd393afce896476ed626cfa4dc699da

## Expected behavior

RTT of all requests should be around some interval of time.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
  python collect_env.py
```
Collecting environment information...
PyTorch version: 1.9.0.dev20210428+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.9.0.dev20210428+cu111
[pip3] torchaudio==0.9.0.dev20210428
[pip3] torchtext==0.10.0a0+34ccda8
[pip3] torchvision==0.10.0.dev20210428+cu111
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.0.221             h6bb024c_0
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py38he904b0f_0
[conda] mkl_fft                   1.2.0            py38h23d657b_0
[conda] mkl_random                1.1.1            py38h0573a6f_0
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] torch                     1.9.0.dev20210428+cu111           <pip>
[conda] torchaudio                0.9.0.dev20210428           <pip>
[conda] torchtext                 0.10.0a0+34ccda8           <pip>
[conda] torchvision               0.10.0.dev20210428+cu111           <pip>

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23