Repository: pytorch
Issue: Calling benchmark.Timer with default `num_threads=1` disables parallelism permanently
Labels: module: docs,triaged,module: macos,module: multithreading

## 🐛 Bug

For those using native parallel backend (now the default in pre-built Mac OS binaries from pytorch and pytorch-nightly channels after #43067), the number of intraop threads can only be set once over the lifetime of a python script or notebook. An unintended consequence is that, when using the new `torch.utils.benchmark.Timer` tool with default num_threads=1, parallelism is disabled for the remainder of the python session, including any future calls to Timer with different num_threads arguments (the num_threads parameter is basically useless for Mac users). There is no documentation on this in the `torch.utils.benchmark` or `torch.set_num_threads` docs pages. Furthermore, the phenomenon is easily unnoticed: although a warning message appears, for those using Jupyter Notebooks, the message is printed into terminal rather than into a notebook cell.

## To Reproduce

code:
```python
import torch
import torch.utils.benchmark as benchmark

t = benchmark.Timer(
    setup="import torch; x = torch.randn(1024, 1024); y = torch.randn(1024, 1024)", 
    stmt="torch.mm(x, y)",
)
print(t.blocked_autorange())

print('\n---- After Timer call ----')

print('num_threads (initial): %d' % torch.get_num_threads())
torch.set_num_threads(4)
print('num_threads (after set->4): %d' % torch.get_num_threads())
```

out:
```
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
<torch.utils.benchmark.utils.common.Measurement object at 0x7fa950b93460>
torch.mm(x, y)
setup: import torch; x = torch.randn(1024, 1024); y = torch.randn(1024, 1024)
  Median: 17.81 ms
  IQR:    0.69 ms (17.43 to 18.13)
  12 measurements, 1 runs per measurement, 1 thread

---- After Timer call ----
num_threads (initial): 1
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
num_threads (after set->4): 1
```

## Expected behavior

I would expect to be able to call Timer with different `num_threads` arguments. I would also expect that the number of threads return back to its pytorch default (e.g. 6 for me) after using Timer. The tool is meant to augment pytorch performance, not harm it.

## Environment

Result from collect_env.py:

```
PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 11.2.3 (x86_64)
GCC version: Could not collect
Clang version: 12.0.5 (clang-1205.0.22.9)
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] numpydoc==1.1.0
[pip3] torch==1.8.1
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      233
[conda] mkl-service               2.3.0            py38h9ed2024_0
[conda] mkl_fft                   1.3.0            py38ha059aab_0
[conda] mkl_random                1.1.1            py38h959d312_0
[conda] numpy                     1.19.2           py38h456fd55_0
[conda] numpy-base                1.19.2           py38hcfb5961_0
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1
[conda] pytorch                   1.8.1                   py3.8_0    pytorch
[conda] torchvision               0.9.1                  py38_cpu    pytorch
```

Parallel info

```python
print(torch.__config__.parallel_info())
```
out:
```
ATen/Parallel:
	at::get_num_threads() : 6
	at::get_num_interop_threads() : 6
OpenMP not found
Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
	mkl_get_max_threads() : 1
Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
std::thread::hardware_concurrency() : 12
Environment variables:
	OMP_NUM_THREADS : [not set]
	MKL_NUM_THREADS : [not set]
ATen parallel backend: native thread pool
```

## Additional context

Many of the PyTorch tutorials involving multithreading, such as the tutorial on [tuning thread counts](https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html#tuning-the-number-of-threads) or the tutorial on [comparing benchmark results](https://pytorch.org/tutorials/recipes/recipes/benchmark.html#comparing-benchmark-results), are not applicable to those using native parallel backend (e.g. all Mac users). I suggest making a note of this in the tutorials/docs, or perhaps fixing the Mac binary builds to correctly find OpenMP (see #43036).

On a related note: It's unclear to me why I am seeing `mkl_get_max_threads() : 1` in my parallel_info result (see output block above). When I check the same MKL distribution from C++ (e.g. by linking the library and include directories from my pytorch conda environment), I see that `mkl_get_max_threads() == 6`, the expected result. Furthermore, with an equivalent pytorch binary for Linux—which seems to have successfully found OpenMP—I see that mkl_get_max_threads matches at::get_num_threads. If there are any performance implications for BLAS/LAPACK ops, they would be good to understand.


cc @brianjo @mruberry