Repository: pytorch
Issue: Test failure in test_shared_allgather_nccl: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:537
Labels: oncall: distributed,triaged,module: POWER

## 🐛 Bug

For the past couple weeks, the pytorch CPU-based build on platform ppc64le has been failing (and subsequently hanging) in the test_shared_allgather_nccl testcase, as seen in recent nightly runs observed here:
https://powerci.osuosl.org/job/pytorch-master-nightly-py3-linux-ppc64le/
For example, to see error messages at the end of a test run, one example is seen at the end of this output log:
https://powerci.osuosl.org/job/pytorch-master-nightly-py3-linux-ppc64le/1137/console
(After the failure, the test run hangs and the final abort seen here is a kill of the run after timing out; the failure
is immediately proceeding it).

## To Reproduce

Steps to reproduce the behavior:
You'd need a ppc64le Power hardware system as seen above (presumably; I don't know if it occurs on any other platforms.)
You can see the execution as in the log at the CI link above.

I can reproduce it locally in a docker container within my lab, using the same OSU CUDA 10.2 docker image that is used in the CI environment.
In my local environment, I set NCCL_DEBUG=TRACE and found that there is a "No space left on device" while doing a posix_fallocate, as seen here:
```8950d9e0cb8d:42345:44067 [0] NCCL INFO Ring 02 : 0[0] -> 1[1] via P2P/direct pointer
8950d9e0cb8d:42345:44070 [3] NCCL INFO Ring 02 : 3[3] -> 0[0] via direct shared memory

8950d9e0cb8d:42345:44069 [2] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
8950d9e0cb8d:42345:44069 [2] NCCL INFO include/shm.h:41 -> 2

8950d9e0cb8d:42345:44069 [2] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ff91c9f3656b19f9-3-1-2 (size 4460544)

8950d9e0cb8d:42345:44069 [2] NCCL INFO transport/shm.cc:173 -> 2
8950d9e0cb8d:42345:44069 [2] NCCL INFO init.cc:340 -> 2
8950d9e0cb8d:42345:44069 [2] NCCL INFO init.cc:650 -> 2
8950d9e0cb8d:42345:44069 [2] NCCL INFO init.cc:815 -> 2
8950d9e0cb8d:42345:44069 [2] NCCL INFO init.cc:951 -> 2
8950d9e0cb8d:42345:44069 [2] NCCL INFO misc/group.cc:69 -> 2 [Async thread]

8950d9e0cb8d:42345:44067 [0] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
8950d9e0cb8d:42345:44067 [0] NCCL INFO include/shm.h:41 -> 2

8950d9e0cb8d:42345:44067 [0] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ff91c9f3656b19f9-3-3-0 (size 4460544)

```
However, monitoring of both the container and the host (with "df") show that we never get close to running out of space on any physical device.  And this happens at the same point in both the official OSULAB CI environment and my local lab system, so it seems it's something impacting this point of the testing rather than an actual space issue coincidental at the same point in both environments.

## Expected behavior
Test should pass without error.

## Environment
ppc64le, CUDA 10.2, docker container as used in the pytorch CI at the link farther above.

I've been trying to figure this out myself without any luck.  **I'm opening this as a bug, due to the test failure -- any ideas? I'll be happy to troubleshoot with any ideas offered.**  (Of course, the build of pytorch itself completes fine, and numerous other tests run prior to this failure point.)



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski