Repository: pytorch
Issue: Importing numpy interacts with `tensor.sum` perf
Labels: module: performance,triaged,module: numpy

## 🐛 Bug

It seems as if importing `numpy` interacts with PyTorch on CPU workloads, which is causing a speedup when `numpy` is imported before `torch`.
This behavior was isolated to the import by @mwawrzos (thanks a lot for the great debugging and forwarding it!) while debugging a regression in GNMT.

## To Reproduce

```python
#import numpy as np # <- uncomment this for speedup
import torch
import matplotlib.pyplot as plt
import time

# warmup
time.sleep(1)
src_length = torch.arange(128)
for _ in range(3000):
    x = src_length.sum()
time.sleep(1)

res = []
for _ in range(3000):
    translate_timer = time.time_ns()
    x = src_length.sum()
    elapsed = time.time_ns() - translate_timer
    #print(elapsed)
    res.append(elapsed)

# save results
import numpy as np
res = np.array(res)
np.save('no_np_02', res)

# filter slow outliers to see the difference better
# threshold = 3000
# res = res[res < threshold]

print('mean {}, std {}, min {}, max {}'.format(
    np.mean(res), np.std(res), np.min(res), np.max(res)))
```
Running this code would save the results for the numpy/no-numpy runs and you could plot the `hist` using:
```python
import numpy as np
import matplotlib.pyplot as plt

with_np = np.load('./with_np_02.npy')
no_np = np.load('./no_np_02.npy')

# remove slow "outliers"
threshold = 3000
with_np = with_np[with_np < threshold]
no_np = no_np[no_np < threshold]

f = plt.figure()
plt.hist(with_np, bins=100, label='with numpy')
plt.hist(no_np, bins=100, label='no numpy')
plt.legend()
```
which would create:
![image](https://user-images.githubusercontent.com/11662379/138248649-e2efab40-b19a-4aa0-ad83-5aa617ac793e.png)

My best guess is that numpy could be setting some multi-threading options, which could be beneficial.
After a discussion with @t-vi we've used `top -H` to check the created threads and are seeing that the "no-numpy run" creates a lot of threads while the "numpy run" does not.
![image](https://user-images.githubusercontent.com/11662379/138249314-b44426d5-6de1-4d99-8809-93eac1607ee9.png)

We've tried to use `OMP_NUM_THREADS=128` and are seeing that the "no-numpy run" comes closer to the "numpy run", but isn't equal. Changing `set_num_interop_threads` also changes the behavior, but doesn't yield the same perf.
![image](https://user-images.githubusercontent.com/11662379/138250166-9644f54f-5e7d-413e-a45d-32e1ea41c390.png)

## Environment
```
PyTorch version: 1.11.0a0+gitb6df043
Is debug build: False
CUDA used to build PyTorch: 11.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.21.3
Libc version: glibc-2.31

Python version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.4.120
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
[...]

Nvidia driver version: 470.57.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.4
[...]
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] nvidia-dlprof-pytorch-nvtx==1.6.0
[pip3] pytorch-quantization==2.1.2
[pip3] torch==1.11.0a0+gitb6df043
[pip3] torchtext==0.11.0a0
[pip3] torchvision==0.11.0a0
[conda] magma-cuda110             2.5.2                         5    local
[conda] mkl                       2019.5                      281    conda-forge
[conda] mkl-include               2019.5                      281    conda-forge
[conda] numpy                     1.21.2           py38he2449b9_0    conda-forge
[conda] nvidia-dlprof-pytorch-nvtx 1.6.0                    pypi_0    pypi
[conda] pytorch-quantization      2.1.2                    pypi_0    pypi
[conda] torch                     1.11.0a0+gitb6df043           dev_0    <develop>
[conda] torchtext                 0.11.0a0                 pypi_0    pypi
[conda] torchvision               0.11.0a0                 pypi_0    pypi
```
Reproduced on an `AMD EPYC 7742 64-Core Processor`


cc @VitalyFedyunin @ngimel @mruberry @rgommers