Repository: pytorch
Issue: processes hang when executing cross-machine asynchronous P2P communication on NCCL backend
Labels: oncall: distributed,module: nccl,module: c10d

### 🐛 Describe the bug

I try to use the NCCL backend for cross-machine asynchronous P2P communication and find a problem with it.  Here is how the communication is executed for each process (and you can just choose to transfer any data you want):

```python
# send & recv in a logical ring (send_dict and recv_dict are used to retrieve corresponding data and construct buffer)
def data_transmission(local_data, send_dict, recv_dict, data_dim, device, dtype):
    rank, world_size = dist.get_rank(), dist.get_world_size()
    buffer_dict = {}
    for pid, ids in recv_dict.items():
        number_data = len(ids)
        buffer = torch.zeros(size=(number_data, data_dim), device=device, dtype=dtype)
        buffer_dict[pid] = buffer
    for i in range(1, world_size):
        ops = []
        left = (rank - i + world_size) % world_size
        right = (rank + i) % world_size
        send_data = local_data[send_dict[right]]
        buffer = buffer_dict[left]
        ops.append(dist.P2POp(op=dist.isend, tensor=send_data, peer=right))
        ops.append(dist.P2POp(op=dist.irecv, tensor=buffer, peer=left))
        reqs = dist.batch_isend_irecv(ops)
        for req in reqs:
            req.wait()
        torch.cuda.synchronize()
     return buffer_dict

```
I try two settings, and the first one is: each machine just starts one process(which means the world_size is 2). Here is the script and terminal message for this setting:

On machine 0
```python
export NCCL_DEBUG=INFO
torchrun --nproc_per_node=1 --nnodes=2 --node_rank=0 --master_addr="net-g5" --master_port="1234" main.py
```

On machine 1
```python
export NCCL_DEBUG=INFO
torchrun --nproc_per_node=1 --nnodes=2 --node_rank=1 --master_addr="net-g5" --master_port="1234" main.py
```

Terminal message on machine 0
 ```python
worker 0 device: cuda:0
net-g5:26896:26896 [0] NCCL INFO Bootstrap : Using p1p1:10.28.1.20<0>
net-g5:26896:26896 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
net-g5:26896:26896 [0] NCCL INFO NET/IB : No device found.
net-g5:26896:26896 [0] NCCL INFO NET/Socket : Using [0]p1p1:10.28.1.20<0> [1]em1.9:202.45.128.225<0> [2]em1:fe80::ae1f:6bff:fe17:b94e%em1<0>
net-g5:26896:26896 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda10.2
net-g5:26896:26946 [0] NCCL INFO Channel 00/02 :    0   1
net-g5:26896:26946 [0] NCCL INFO Channel 01/02 :    0   1
net-g5:26896:26946 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
net-g5:26896:26946 [0] NCCL INFO Setting affinity for GPU 0 to ffff
net-g5:26896:26946 [0] NCCL INFO Channel 00 : 1[3000] -> 0[2000] [receive] via NET/Socket/0
net-g5:26896:26946 [0] NCCL INFO Channel 01 : 1[3000] -> 0[2000] [receive] via NET/Socket/0
net-g5:26896:26946 [0] NCCL INFO Channel 00 : 0[2000] -> 1[3000] [send] via NET/Socket/0
net-g5:26896:26946 [0] NCCL INFO Channel 01 : 0[2000] -> 1[3000] [send] via NET/Socket/0
net-g5:26896:26946 [0] NCCL INFO Connected all rings
net-g5:26896:26946 [0] NCCL INFO Connected all trees
net-g5:26896:26946 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
net-g5:26896:26946 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g5:26896:26946 [0] NCCL INFO comm 0x7fd77c001200 rank 0 nranks 2 cudaDev 0 busId 2000 - Init COMPLETE
net-g5:26896:26896 [0] NCCL INFO Launch mode Parallel
worker 0 begins transfer...
net-g5:26896:26954 [0] NCCL INFO Channel 00/02 :    0   1
net-g5:26896:26954 [0] NCCL INFO Channel 01/02 :    0   1
net-g5:26896:26954 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
net-g5:26896:26954 [0] NCCL INFO Setting affinity for GPU 0 to ffff
net-g5:26896:26954 [0] NCCL INFO Channel 00 : 1[2000] -> 0[2000] [receive] via NET/Socket/0
net-g5:26896:26954 [0] NCCL INFO Channel 01 : 1[2000] -> 0[2000] [receive] via NET/Socket/0
net-g5:26896:26954 [0] NCCL INFO Channel 00 : 0[2000] -> 1[2000] [send] via NET/Socket/0
net-g5:26896:26954 [0] NCCL INFO Channel 01 : 0[2000] -> 1[2000] [send] via NET/Socket/0
net-g5:26896:26954 [0] NCCL INFO Connected all rings
net-g5:26896:26954 [0] NCCL INFO Connected all trees
net-g5:26896:26954 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
net-g5:26896:26954 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g5:26896:26954 [0] NCCL INFO comm 0x7fd780001200 rank 0 nranks 2 cudaDev 0 busId 2000 - Init COMPLETE
net-g5:26896:26896 [0] NCCL INFO Launch mode Parallel
worker 0 data exchange done.
```

Terminal message on machine 1
```python
worker 1 device: cuda:0
net-g6:19369:19369 [1] NCCL INFO Bootstrap : Using p1p1:10.28.1.21<0>
net-g6:19369:19369 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
net-g6:19369:19369 [1] NCCL INFO NET/IB : No device found.
net-g6:19369:19369 [1] NCCL INFO NET/Socket : Using [0]p1p1:10.28.1.21<0> [1]em1.9:202.45.128.226<0> [2]em1:fe80::ae1f:6bff:fe17:b976%em1<0> [3]veth99c26f7:fe80::e475:2ff:fea8:9808%veth99c26f7<0>
net-g6:19369:19369 [1] NCCL INFO Using network Socket
net-g6:19369:19415 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
net-g6:19369:19415 [1] NCCL INFO Setting affinity for GPU 1 to ffff
net-g6:19369:19415 [1] NCCL INFO Channel 00 : 0[2000] -> 1[3000] [receive] via NET/Socket/0
net-g6:19369:19415 [1] NCCL INFO Channel 01 : 0[2000] -> 1[3000] [receive] via NET/Socket/0
net-g6:19369:19415 [1] NCCL INFO Channel 00 : 1[3000] -> 0[2000] [send] via NET/Socket/0
net-g6:19369:19415 [1] NCCL INFO Channel 01 : 1[3000] -> 0[2000] [send] via NET/Socket/0
net-g6:19369:19415 [1] NCCL INFO Connected all rings
net-g6:19369:19415 [1] NCCL INFO Connected all trees
net-g6:19369:19415 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
net-g6:19369:19415 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g6:19369:19415 [1] NCCL INFO comm 0x7f0b48001200 rank 1 nranks 2 cudaDev 1 busId 3000 - Init COMPLETE
worker 1 begins transfer...
net-g6:19369:19418 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
net-g6:19369:19418 [0] NCCL INFO Setting affinity for GPU 0 to ffff
net-g6:19369:19418 [0] NCCL INFO Channel 00 : 0[2000] -> 1[2000] [receive] via NET/Socket/0
net-g6:19369:19418 [0] NCCL INFO Channel 01 : 0[2000] -> 1[2000] [receive] via NET/Socket/0
net-g6:19369:19418 [0] NCCL INFO Channel 00 : 1[2000] -> 0[2000] [send] via NET/Socket/0
net-g6:19369:19418 [0] NCCL INFO Channel 01 : 1[2000] -> 0[2000] [send] via NET/Socket/0
net-g6:19369:19418 [0] NCCL INFO Connected all rings
net-g6:19369:19418 [0] NCCL INFO Connected all trees
net-g6:19369:19418 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
net-g6:19369:19418 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g6:19369:19418 [0] NCCL INFO comm 0x7f0b48005bc0 rank 1 nranks 2 cudaDev 0 busId 2000 - Init COMPLETE
worker 1 data exchange done.
```

All the processes go well with this setting. However, when I change the setting to: each machine starts two processes(which means the world_size is 4), all the processes hang when execute above asynchronous P2P communication. Here is the script and terminal message for this setting:

On machine 0
```python
export NCCL_DEBUG=INFO
torchrun --nproc_per_node=2 --nnodes=2 --node_rank=0 --master_addr="net-g5" --master_port="1234" main.py
```

On machine 1
```python
export NCCL_DEBUG=INFO
torchrun --nproc_per_node=2 --nnodes=2 --node_rank=1 --master_addr="net-g5" --master_port="1234" main.py
```

Terminal message on machine 0
 ```python
worker 1 device: cuda:1
worker 0 device: cuda:0
net-g5:27098:27098 [0] NCCL INFO Bootstrap : Using p1p1:10.28.1.20<0>
net-g5:27098:27098 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
net-g5:27098:27098 [0] NCCL INFO NET/IB : No device found.
net-g5:27098:27098 [0] NCCL INFO NET/Socket : Using [0]p1p1:10.28.1.20<0> [1]em1.9:202.45.128.225<0> [2]em1:fe80::ae1f:6bff:fe17:b94e%em1<0>
net-g5:27098:27098 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda10.2
net-g5:27099:27099 [1] NCCL INFO Bootstrap : Using p1p1:10.28.1.20<0>
net-g5:27099:27099 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
net-g5:27099:27099 [1] NCCL INFO NET/IB : No device found.
net-g5:27099:27099 [1] NCCL INFO NET/Socket : Using [0]p1p1:10.28.1.20<0> [1]em1.9:202.45.128.225<0> [2]em1:fe80::ae1f:6bff:fe17:b94e%em1<0>
net-g5:27099:27099 [1] NCCL INFO Using network Socket
net-g5:27098:27147 [0] NCCL INFO Channel 00/02 :    0   1   2   3
net-g5:27099:27148 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
net-g5:27098:27147 [0] NCCL INFO Channel 01/02 :    0   1   2   3
net-g5:27099:27148 [1] NCCL INFO Setting affinity for GPU 1 to ffff
net-g5:27098:27147 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2
net-g5:27098:27147 [0] NCCL INFO Setting affinity for GPU 0 to ffff
net-g5:27098:27147 [0] NCCL INFO Channel 00 : 3[3000] -> 0[2000] [receive] via NET/Socket/0
net-g5:27098:27147 [0] NCCL INFO Channel 01 : 3[3000] -> 0[2000] [receive] via NET/Socket/0
net-g5:27098:27147 [0] NCCL INFO Channel 00 : 0[2000] -> 1[3000] via direct shared memory
net-g5:27098:27147 [0] NCCL INFO Channel 01 : 0[2000] -> 1[3000] via direct shared memory
net-g5:27099:27148 [1] NCCL INFO Channel 00 : 1[3000] -> 2[2000] [send] via NET/Socket/0
net-g5:27099:27148 [1] NCCL INFO Channel 01 : 1[3000] -> 2[2000] [send] via NET/Socket/0
net-g5:27099:27148 [1] NCCL INFO Connected all rings
net-g5:27099:27148 [1] NCCL INFO Channel 00 : 1[3000] -> 0[2000] via direct shared memory
net-g5:27099:27148 [1] NCCL INFO Channel 01 : 1[3000] -> 0[2000] via direct shared memory
net-g5:27098:27147 [0] NCCL INFO Connected all rings
net-g5:27098:27147 [0] NCCL INFO Channel 00 : 2[2000] -> 0[2000] [receive] via NET/Socket/0
net-g5:27098:27147 [0] NCCL INFO Channel 01 : 2[2000] -> 0[2000] [receive] via NET/Socket/0
net-g5:27098:27147 [0] NCCL INFO Channel 00 : 0[2000] -> 2[2000] [send] via NET/Socket/0
net-g5:27098:27147 [0] NCCL INFO Channel 01 : 0[2000] -> 2[2000] [send] via NET/Socket/0
net-g5:27098:27147 [0] NCCL INFO Connected all trees
net-g5:27098:27147 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
net-g5:27098:27147 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g5:27099:27148 [1] NCCL INFO Connected all trees
net-g5:27099:27148 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
net-g5:27099:27148 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g5:27098:27147 [0] NCCL INFO comm 0x7fac40001200 rank 0 nranks 4 cudaDev 0 busId 2000 - Init COMPLETE
net-g5:27099:27148 [1] NCCL INFO comm 0x7f928c001200 rank 1 nranks 4 cudaDev 1 busId 3000 - Init COMPLETE
net-g5:27098:27098 [0] NCCL INFO Launch mode Parallel
worker 1 begins transfer...
worker 0 begins transfer...
NCCL version 2.10.3+cuda10.2
```

Terminal message on machine 1
 ```python
worker 2 device: cuda:1
worker 3 device: cuda:0
net-g6:19527:19527 [1] NCCL INFO Bootstrap : Using p1p1:10.28.1.21<0>
net-g6:19527:19527 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
net-g6:19527:19527 [1] NCCL INFO NET/IB : No device found.
net-g6:19527:19527 [1] NCCL INFO NET/Socket : Using [0]p1p1:10.28.1.21<0> [1]em1.9:202.45.128.226<0> [2]em1:fe80::ae1f:6bff:fe17:b976%em1<0> [3]veth99c26f7:fe80::e475:2ff:fea8:9808%veth99c26f7<0>
net-g6:19527:19527 [1] NCCL INFO Using network Socket
net-g6:19526:19526 [0] NCCL INFO Bootstrap : Using p1p1:10.28.1.21<0>
net-g6:19526:19526 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
net-g6:19526:19526 [0] NCCL INFO NET/IB : No device found.
net-g6:19526:19526 [0] NCCL INFO NET/Socket : Using [0]p1p1:10.28.1.21<0> [1]em1.9:202.45.128.226<0> [2]em1:fe80::ae1f:6bff:fe17:b976%em1<0> [3]veth99c26f7:fe80::e475:2ff:fea8:9808%veth99c26f7<0>
net-g6:19526:19526 [0] NCCL INFO Using network Socket
net-g6:19527:19566 [1] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2
net-g6:19526:19567 [0] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/0/-1->2->-1
net-g6:19527:19566 [1] NCCL INFO Setting affinity for GPU 1 to ffff
net-g6:19526:19567 [0] NCCL INFO Setting affinity for GPU 0 to ffff
net-g6:19526:19567 [0] NCCL INFO Channel 00 : 1[3000] -> 2[2000] [receive] via NET/Socket/0
net-g6:19526:19567 [0] NCCL INFO Channel 01 : 1[3000] -> 2[2000] [receive] via NET/Socket/0
net-g6:19526:19567 [0] NCCL INFO Channel 00 : 2[2000] -> 3[3000] via direct shared memory
net-g6:19526:19567 [0] NCCL INFO Channel 01 : 2[2000] -> 3[3000] via direct shared memory
net-g6:19527:19566 [1] NCCL INFO Channel 00 : 3[3000] -> 0[2000] [send] via NET/Socket/0
net-g6:19527:19566 [1] NCCL INFO Channel 01 : 3[3000] -> 0[2000] [send] via NET/Socket/0
net-g6:19527:19566 [1] NCCL INFO Connected all rings
net-g6:19527:19566 [1] NCCL INFO Channel 00 : 3[3000] -> 2[2000] via direct shared memory
net-g6:19527:19566 [1] NCCL INFO Channel 01 : 3[3000] -> 2[2000] via direct shared memory
net-g6:19526:19567 [0] NCCL INFO Connected all rings
net-g6:19526:19567 [0] NCCL INFO Channel 00 : 0[2000] -> 2[2000] [receive] via NET/Socket/0
net-g6:19526:19567 [0] NCCL INFO Channel 01 : 0[2000] -> 2[2000] [receive] via NET/Socket/0
net-g6:19526:19567 [0] NCCL INFO Channel 00 : 2[2000] -> 0[2000] [send] via NET/Socket/0
net-g6:19526:19567 [0] NCCL INFO Channel 01 : 2[2000] -> 0[2000] [send] via NET/Socket/0
net-g6:19526:19567 [0] NCCL INFO Connected all trees
net-g6:19526:19567 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
net-g6:19526:19567 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g6:19527:19566 [1] NCCL INFO Connected all trees
net-g6:19527:19566 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
net-g6:19527:19566 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
net-g6:19526:19567 [0] NCCL INFO comm 0x7faa4c001200 rank 2 nranks 4 cudaDev 0 busId 2000 - Init COMPLETE
net-g6:19527:19566 [1] NCCL INFO comm 0x7f2f40001200 rank 3 nranks 4 cudaDev 1 busId 3000 - Init COMPLETE
worker 2 begins transfer...
worker 3 begins transfer...
NCCL version 2.10.3+cuda10.2
```

When I change the backend to GLOO, all processes go well in the above two settings, correct data can be sent and received for each process. I guess the problem may be relevant to cudaStream used by NCCL for P2P communication. the message from two machines' terminals indicates that all the processes fail to use Socket(for remote access) or direct Shared Memory(for local access) for communication. So is that mean PyTorch with NCCL backend cannot deal with multiple isend/irecv requests on a single machine?





### Versions

configuration on machine 0
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.22.0
Libc version: glibc-2.17

Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-77-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 455.45.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.21
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==2.0.3
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.12
[pip3] torch-spline-conv==1.2.1
[pip3] torch-tb-profiler==0.3.1
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py37h7f8727e_0  
[conda] mkl_fft                   1.3.1            py37hd3c417c_0  
[conda] mkl_random                1.2.2            py37h51133e4_0  
[conda] numpy                     1.21.2           py37h20f2e39_0  
[conda] numpy-base                1.21.2           py37h79a1101_0  
[conda] pyg                       2.0.3           py37_torch_1.10.0_cu102    pyg
[conda] pytorch                   1.10.0          py3.7_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-cluster           1.5.9           py37_torch_1.10.0_cu102    pyg
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-scatter           2.0.9           py37_torch_1.10.0_cu102    pyg
[conda] pytorch-sparse            0.6.12          py37_torch_1.10.0_cu102    pyg
[conda] pytorch-spline-conv       1.2.1           py37_torch_1.10.0_cu102    pyg
[conda] torch-tb-profiler         0.3.1                    pypi_0    pypi
[conda] torchvision               0.11.1               py37_cu102    pytorch

configuration on machine 1
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.22.0
Libc version: glibc-2.17

Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-77-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 455.32.00
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.21
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==2.0.3
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.12
[pip3] torch-spline-conv==1.2.1
[pip3] torch-tb-profiler==0.3.1
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py37h7f8727e_0  
[conda] mkl_fft                   1.3.1            py37hd3c417c_0  
[conda] mkl_random                1.2.2            py37h51133e4_0  
[conda] numpy                     1.21.2           py37h20f2e39_0  
[conda] numpy-base                1.21.2           py37h79a1101_0  
[conda] pyg                       2.0.3           py37_torch_1.10.0_cu102    pyg
[conda] pytorch                   1.10.0          py3.7_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-cluster           1.5.9           py37_torch_1.10.0_cu102    pyg
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-scatter           2.0.9           py37_torch_1.10.0_cu102    pyg
[conda] pytorch-sparse            0.6.12          py37_torch_1.10.0_cu102    pyg
[conda] pytorch-spline-conv       1.2.1           py37_torch_1.10.0_cu102    pyg
[conda] torch-tb-profiler         0.3.1                    pypi_0    pypi
[conda] torchvision               0.11.1               py37_cu102    pytorch

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang