Repository: pytorch
Issue: test_backward_deadlock fails with "Directory not empty"
Labels: oncall: jit,module: POWER

## 🐛 Bug

I'm running a build &  test on a node without a GPU (installation for a cluster)
One test (`TestCppExtensionJIT`) seems to fail with what looks like a race condition. The error is "OSError: [Errno 39] Directory not empty: 'test_backward_deadlock'"   
See output below

## To Reproduce

Steps to reproduce the behavior:

1. python setup.py install
1. python tests/run_tests.py

Part of the output before the failure:
```
test_jit_cuda_archflags (__main__.TestCppExtensionJIT) ... skipped 'CUDA not found'
test_jit_cuda_extension (__main__.TestCppExtensionJIT) ... skipped 'CUDA not found'
test_jit_cudnn_extension (__main__.TestCppExtensionJIT) ... skipped 'CuDNN not found'
test_lenient_flag_handling_in_jit_extensions (__main__.TestCppExtensionJIT) ... Loading extension module jit_extension...
Using /home/h3/s3248973/.cache/torch_extensions as PyTorch extensions root...
Creating extension directory /home/h3/s3248973/.cache/torch_extensions/lenient_flag_handling_extension...
Emitting ninja build file /home/h3/s3248973/.cache/torch_extensions/lenient_flag_handling_extension/build.ninja...
Building extension module lenient_flag_handling_extension...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] g++ -MMD -MF main.o.d -DTORCH_EXTENSION_NAME=lenient_flag_handling_extension -DTORCH_API_INCLUDE_EXTENSION_H -I/dev/shm/s3248973-EasyBuild/PyTorch/1.6.0/fosscuda-2019b-Python-3.7.4/pytorch-1.6.0-rc2/test/cpp_extensions -isystem /tmp
/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/
lib/python3.7/site-packages/torch/include/TH -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include/THC -isystem /sw/installed/Python/3.7.4-GCCcore-8.3.0/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fP
IC -std=c++14 -g -O0 -Wall -c /home/h3/s3248973/.cache/torch_extensions/lenient_flag_handling_extension/main.cpp -o main.o 
[2/2] g++ main.o -shared -L/tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o lenient_flag_handling_extension.so
ok
test_reload_jit_extension (__main__.TestCppExtensionJIT) ... Loading extension module lenient_flag_handling_extension...
Using /home/h3/s3248973/.cache/torch_extensions as PyTorch extensions root...
Creating extension directory /home/h3/s3248973/.cache/torch_extensions/reloaded_jit_extension...
Emitting ninja build file /home/h3/s3248973/.cache/torch_extensions/reloaded_jit_extension/build.ninja...
Building extension module reloaded_jit_extension...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] g++ -MMD -MF main.o.d -DTORCH_EXTENSION_NAME=reloaded_jit_extension -DTORCH_API_INCLUDE_EXTENSION_H -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpN
peQu4/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include/TH -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-package
s/torch/include/THC -isystem /sw/installed/Python/3.7.4-GCCcore-8.3.0/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/h3/s3248973/.cache/torch_extensions/reloaded_jit_extension/main.cpp -o main.o 
[2/2] g++ main.o -shared -L/tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o reloaded_jit_extension.so
Loading extension module reloaded_jit_extension...
Using /home/h3/s3248973/.cache/torch_extensions as PyTorch extensions root...
The input conditions for extension module reloaded_jit_extension have changed. Bumping to version 1 and re-building as reloaded_jit_extension_v1...
Emitting ninja build file /home/h3/s3248973/.cache/torch_extensions/reloaded_jit_extension/build.ninja...
Building extension module reloaded_jit_extension_v1...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] g++ -MMD -MF main.o.d -DTORCH_EXTENSION_NAME=reloaded_jit_extension_v1 -DTORCH_API_INCLUDE_EXTENSION_H -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/t
mpNpeQu4/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include/TH -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-pack
ages/torch/include/THC -isystem /sw/installed/Python/3.7.4-GCCcore-8.3.0/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/h3/s3248973/.cache/torch_extensions/reloaded_jit_extension/main.cpp -o main.o 
[2/2] g++ main.o -shared -L/tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o reloaded_jit_extension_v1.so
Loading extension module reloaded_jit_extension_v1...
Using /home/h3/s3248973/.cache/torch_extensions as PyTorch extensions root...
No modifications detected for re-loaded extension module reloaded_jit_extension_v1, skipping build step...
Loading extension module reloaded_jit_extension_v1...
Using /home/h3/s3248973/.cache/torch_extensions as PyTorch extensions root...
The input conditions for extension module reloaded_jit_extension have changed. Bumping to version 2 and re-building as reloaded_jit_extension_v2...
Emitting ninja build file /home/h3/s3248973/.cache/torch_extensions/reloaded_jit_extension/build.ninja...
Building extension module reloaded_jit_extension_v2...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[...]
test_set_default_type_also_changes_aten_default_type (__main__.TestCppExtensionJIT) ... Loading extension module is_python_module_v1...
Using /home/h3/s3248973/.cache/torch_extensions as PyTorch extensions root...
Creating extension directory /home/h3/s3248973/.cache/torch_extensions/test_set_default_type...
Emitting ninja build file /home/h3/s3248973/.cache/torch_extensions/test_set_default_type/build.ninja...
Building extension module test_set_default_type...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] g++ -MMD -MF main.o.d -DTORCH_EXTENSION_NAME=test_set_default_type -DTORCH_API_INCLUDE_EXTENSION_H -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include/TH -isystem /tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/include/THC -isystem /sw/installed/Python/3.7.4-GCCcore-8.3.0/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/h3/s3248973/.cache/torch_extensions/test_set_default_type/main.cpp -o main.o 
[2/2] g++ main.o -shared -L/tmp/easybuild-tmp/eb-RgtlFs/tmpNpeQu4/lib/python3.7/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o test_set_default_type.so
ok
test_warning (__main__.TestCppExtensionJIT) ... Loading extension module test_set_default_type...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0710 12:47:03.169983 24274 main.cpp:12] Warning: Error with CPUDoubleType (function foo)
W0710 12:47:03.170961 24274 main.cpp:12] Warning: Error with CPUDoubleType (function foo)
W0710 12:47:03.171689 24274 main.cpp:12] Warning: Error with CPUDoubleType (function foo)
W0710 12:47:03.172047 24274 main.cpp:12] Warning: Error with CPUDoubleType (function foo)
W0710 12:47:16.688925 24274 main.cpp:12] Warning: Error with torch.DoubleTensor (function foo)
W0710 12:47:16.689333 24274 main.cpp:12] Warning: Error with torch.DoubleTensor (function foo)
W0710 12:47:16.689601 24274 main.cpp:12] Warning: Error with torch.DoubleTensor (function foo)
W0710 12:47:16.690302 24274 main.cpp:12] Warning: Error with torch.DoubleTensor (function foo)
ok
ERROR

======================================================================
ERROR: tearDownClass (__main__.TestCppExtensionJIT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_cpp_extensions_jit.py", line 60, in tearDownClass
    remove_build_path()
  File "test_cpp_extensions_jit.py", line 36, in remove_build_path
    shutil.rmtree(default_build_root)
  File "/sw/installed/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/shutil.py", line 494, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/sw/installed/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/shutil.py", line 436, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/sw/installed/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/shutil.py", line 434, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'test_backward_deadlock'
```

## Expected behavior

Tests succeed

## Environment

```
PyTorch version: 1.6.0-rc2
Is debug build: N/A
CUDA used to build PyTorch: N/A

OS: Red Hat Enterprise Linux Server release 7.8 (Maipo)
GCC version: (GCC) 8.3.0
CMake version: version 3.15.3

Python version: 3.7
Is CUDA available: N/A
CUDA runtime version: 10.1.243
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: 7.6.4.38

Versions of relevant libraries:
[pip3] numpy==1.17.3
```


cc @suo @gmagogsfm