Repository: pytorch
Issue: Some test failed when running in parallel.
Labels: oncall: distributed

### 🐛 Describe the bug

Running `ctest -j36 --output-on-failure` on Ubuntu 18.04

```
 96/141 Test  #71: workspace_test ............................   Passed    0.73 sec
        Start  10: c10_Bitset_test
 97/141 Test  #12: c10_ConstexprCrc_test .....................   Passed    0.01 sec
        Start   9: c10_Array_test
 98/141 Test  #15: c10_Metaprogramming_test ..................   Passed    0.01 sec
        Start   8: c10_SizesAndStrides_test
 99/141 Test  #16: c10_SmallVectorTest .......................   Passed    0.02 sec
        Start   7: c10_InlineStreamGuard_test
100/141 Test #135: FileStoreTest .............................   Passed    0.66 sec
        Start   6: c10_InlineDeviceGuard_test
101/141 Test  #73: fixed_divisor_test ........................   Passed    0.75 sec
        Start   5: c10_StreamGuard_test
102/141 Test  #11: c10_C++17_test ............................   Passed    0.02 sec
        Start   4: c10_DispatchKeySet_test
103/141 Test  #10: c10_Bitset_test ...........................   Passed    0.02 sec
        Start   3: c10_Device_test
104/141 Test   #9: c10_Array_test ............................   Passed    0.01 sec
        Start   2: c10_DeviceGuard_test
105/141 Test   #8: c10_SizesAndStrides_test ..................   Passed    0.01 sec
        Start   1: c10_CompileTimeFunctionPointer_test
106/141 Test  #65: parallel_net_test .........................   Passed    7.54 sec
107/141 Test  #63: operator_schema_test ......................   Passed    0.79 sec
108/141 Test   #7: c10_InlineStreamGuard_test ................   Passed    0.01 sec
109/141 Test   #6: c10_InlineDeviceGuard_test ................   Passed    0.01 sec
110/141 Test   #5: c10_StreamGuard_test ......................   Passed    0.01 sec
111/141 Test   #4: c10_DispatchKeySet_test ...................   Passed    0.01 sec
112/141 Test   #2: c10_DeviceGuard_test ......................   Passed    0.01 sec
113/141 Test   #1: c10_CompileTimeFunctionPointer_test .......   Passed    0.00 sec
114/141 Test   #3: c10_Device_test ...........................   Passed    0.01 sec
115/141 Test  #14: c10_LeftRight_test ........................   Passed    0.16 sec
116/141 Test #103: utility_ops_test ..........................   Passed    0.78 sec
117/141 Test #137: HashStoreTest .............................   Passed    0.81 sec
118/141 Test  #72: inline_container_test .....................   Passed    0.83 sec
119/141 Test #139: ProcessGroupGlooAsyncTest .................   Passed    0.81 sec
120/141 Test  #37: c10_cuda_CUDATest .........................   Passed    0.94 sec
121/141 Test #134: utility_ops_gpu_test ......................   Passed    9.77 sec
122/141 Test  #64: operator_test .............................   Passed    9.82 sec
123/141 Test #121: event_gpu_test ............................   Passed    9.93 sec
124/141 Test #114: nnpack_test ...............................***Failed   10.09 sec
125/141 Test #132: reshape_op_gpu_test .......................   Passed   10.27 sec
126/141 Test #128: elementwise_op_gpu_test ...................   Passed   10.73 sec
127/141 Test #131: operator_fallback_gpu_test ................   Passed   10.76 sec
128/141 Test #123: operator_gpu_test .........................   Passed   10.77 sec
129/141 Test #129: generate_proposals_op_gpu_test ............   Passed   10.79 sec
130/141 Test  #66: plan_executor_test ........................   Passed   10.81 sec
131/141 Test #127: batch_permutation_op_gpu_test .............   Passed   10.91 sec
132/141 Test #133: roi_align_op_gpu_test .....................   Passed   11.13 sec
133/141 Test #126: batch_matmul_op_gpu_test ..................   Passed   11.35 sec
134/141 Test #124: math_gpu_test .............................   Passed   11.35 sec
135/141 Test #119: blob_gpu_test .............................   Passed   12.27 sec
136/141 Test #141: ProcessGroupNCCLErrorsTest ................   Passed   14.95 sec
137/141 Test #120: context_gpu_test ..........................   Passed   16.02 sec
138/141 Test  #51: blob_test .................................   Passed   22.06 sec
139/141 Test #130: generate_proposals_op_util_nms_gpu_test ...   Passed   22.63 sec
140/141 Test  #75: fatal_signal_asan_no_sig_test .............   Passed   29.40 sec
141/141 Test #140: ProcessGroupNCCLTest ......................Bus error***Exception:  45.71 sec

98% tests passed, 3 tests failed out of 141

Total Test time (real) =  46.09 sec

The following tests FAILED:
         50 - op_registration_test (Failed)
        114 - nnpack_test (Failed)
        140 - ProcessGroupNCCLTest (Bus error)
```

Log of bus error:
```
141/141 Test #140: ProcessGroupNCCLTest ......................Bus error***Exception:  39.07 sec
Running main() from ../third_party/googletest/googletest/src/gtest_main.cc
[==========] Running 11 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 11 tests from ProcessGroupNCCLTest
[ RUN      ] ProcessGroupNCCLTest.testAllreduce
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0420 13:18:49.426566 233385 ProcessGroupNCCLTest.cpp:585] Multi-node world size: 1 rank: 0
I0420 13:19:01.975904 233385 ProcessGroupNCCL.cpp:632] [Rank 0] will wait up to 1800000 msec for NCCL health check to complete.
I0420 13:19:03.251540 233385 ProcessGroupNCCL.cpp:583] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
NCCL_DEBUG: UNSET
I0420 13:19:03.251551 238663 ProcessGroupNCCL.cpp:728] [Rank 0] NCCL watchdog thread started!
```

- op_registration_test is probably https://github.com/pytorch/pytorch/issues/70160
- nnpack_test occasionally failed.
- ProcessGroupNCCLTest consistently failed, but running the executable directly is fine.

Sequential ctest is fine except for op reg.

Regarding NCCL, this is a dual-GPU system with Volta + Pascal and running inside docker.

### Versions

Commit 025cd69a86b4cce951c1fccb5b7160d8174e5ea9, the last one supporting stock python 3.6 on Ubuntu 18.04.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang