Repository: pytorch
Issue: Custom ops get stuck in multiprocess data loader under certain environments
Labels: module: dataloader,triaged,module: deadlock,module: custom-operators

## 🐛 Bug

`torchaudio` has custom ops which are written on top of `libsox` and bound via `TORCH_LIBRARY`.
Some ops get stuck when **all of the following conditions are met**;
 - Called in `DataLoader` with `num_workers > 0`
 - Python `3.6` or `3.7`
 - `macOS`.

They *work fine* on `Linux`, `Python 3.8 of macOS`, or `in the main process of Python 3.5/3.6 of macOS`.

There are two functions get stuck. `torchaudio.sox_effects.apply_effects_file` and `torchaudio.sox_effects.apply_effects_tensor`. What they do is, receive either file path or Tensor object, fetch signal data in `libsox`'s signal format (contiguous memory of `int32_t`), apply various effects, then convert the result into Tensor.

The functions are 
 - Implemented in C++ [here](https://github.com/pytorch/audio/blob/8e8c5277557d781c70ddad4da3e95d9ad1313f27/torchaudio/csrc/sox_effects.h#L14-L22)
 - Registered [here](https://github.com/pytorch/audio/blob/master/torchaudio/csrc/register.cpp#L67-L72)
 - Python front end is found [here](https://github.com/pytorch/audio/blob/master/torchaudio/sox_effects/sox_effects.py#L60-L259)

The test code is
 - [apply_effects_file](https://github.com/pytorch/audio/blob/master/test/torchaudio_unittest/sox_effect/dataset_test.py#L85-L94)
 - [apply_effects_tensor](https://github.com/pytorch/audio/blob/master/test/torchaudio_unittest/sox_effect/dataset_test.py#L105-L114)

1. When I replace the call to `torchaudio.sox_effects.apply_effects_file` on Python side to something like `torch.zeros`, then this stuck does not happen, so I think this is related to custom ops.

1. With bunch of debug print added, I located the line of stuck [here in the function implementation](https://github.com/pytorch/audio/blob/master/torchaudio/csrc/sox_effects.cpp#L79-L85) and specifically [here in the helper function](https://github.com/pytorch/audio/blob/master/torchaudio/csrc/sox_utils.cpp#L165), which is when the last step of both functions, and where the `libsox`'s resulting memory data are converted to Tensor of float32.

1. Interestingly, I simply commented out the whole part after the call to [`torch::from_blob`](https://github.com/pytorch/audio/blob/master/torchaudio/csrc/sox_utils.cpp#L160-L161), but it still gets stuck.

1. When I replace the entire content of [`apply_effects_file` in C++](https://github.com/pytorch/audio/blob/8e8c5277557d781c70ddad4da3e95d9ad1313f27/torchaudio/csrc/sox_effects.cpp#L91-L134) with `return torch::zeros({2, 44100});`, it still happens, so it seems not related to the implementation detail of the `apply_effects_file` function.

1. Also when I run the said test, I get the following warning, 

    ```
    [W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
    ```

    but `torch.set_num_threads` is not used anywhere.

## To Reproduce

Steps to reproduce the behavior:

1. On macOS, install `miniconda` and create environments with either Python 3.6 or 3.7. (Python 3.8 should not suffer this)
1. Install PyTorch 1.7.0 (or nightly).
1. Clone torchaudio. I am working on `release/0.7` branch but `master` branch should exhibit the same behavior.
1. Build torchaudio `BUILD_SOX=0 python setup.py clean develop`
1. Run the tests, they will be stuck 
    ```
    pytest test/torchaudio_unittest/sox_effect/dataset_test.py::TestSoxEffectsDataset::test_apply_effects_file -v -s
    pytest test/torchaudio_unittest/sox_effect/dataset_test.py::TestSoxEffectsDataset::test_apply_effects_tensor -v -s
    ```

## Expected behavior

These ops `apply_effects_file` and `apply_effects_tensor` should not get stuck.

## Environment

<details>

```
PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Mac OSX 10.15.6 (x86_64)
GCC version: Could not collect
Clang version: 11.0.0 (clang-1100.0.33.16)
CMake version: version 3.18.2

Python version: 3.6 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.7.0
[pip3] torchaudio==0.7.0a0+53643bf
[conda] blas                      1.0                         mkl
[conda] libblas                   3.8.0                    14_mkl    conda-forge
[conda] libcblas                  3.8.0                    14_mkl    conda-forge
[conda] liblapack                 3.8.0                    14_mkl    conda-forge
[conda] mkl                       2019.4                      233
[conda] mkl-service               2.3.0            py36hfbe908c_0
[conda] mkl_fft                   1.2.0            py36hc64f4ea_0
[conda] mkl_random                1.1.1            py36h959d312_0
[conda] numpy                     1.19.1           py36h3b9f5b6_0
[conda] numpy-base                1.19.1           py36hcfb5961_0
[conda] pytorch                   1.7.0                   py3.6_0    pytorch-test
[conda] torchaudio                0.7.0a0+53643bf           dev_0    <develop>
```

</details>

## Additional context

**This is also happening on torchaudio release branch**.

https://github.com/pytorch/audio/pull/946 -> [CircleCI workflow](https://app.circleci.com/pipelines/github/pytorch/audio/3597/workflows/36d5bc2f-ad1f-49fa-90b6-71a170f14dc3)



cc @SsnL @VitalyFedyunin @ejguan