Repository: pytorch
Issue: Performance regression on CPU from 0.4.1 to 1.0.0 on ResNet inference
Labels: module: performance,module: cpu,triaged,module: single threaded

## 🐛 Bug

ResNet50 inference is 20 - 65% slower on pytorch 1.0.0 and 1.0.1 vs 0.4.1 on CPU with `OMP_NUM_THREADS=1`, and  ~~2800% slower without `OMP_NUM_THREADS=1` (8 core CPU), although these are probably two separate issues and I'm personally more interested in `OMP_NUM_THREADS=1` behaviour because it gets maximal use of available cores regardless of the version used~~ (edited: updated for 1.0.1, all timings below are very similar between 1.0.0 and 1.0.1).

## To Reproduce

Steps to reproduce the behavior:

- install pytorch from PyPI on Linux with Python 3.6, specifically these wheels were installed: `https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl` and `https://files.pythonhosted.org/packages/49/0e/e382bcf1a6ae8225f50b99cc26effa2d4cc6d66975ccf3fa9590efcbedce/torch-0.4.1-cp36-cp36m-manylinux1_x86_64.whl`
- run `CUDA_VISIBLE_DEVICES= OMP_NUM_THREADS=1 ipython`
- in ipython, run:
```
import torch, torchvision
model = torchvision.models.resnet50(num_classes=10)
with torch.no_grad(): model(torch.zeros(1, 3, 224, 224))`
```
- still in ipython, run:
`%timeit with torch.no_grad(): model(torch.zeros(1, 3, 224, 224))`

## Expected behavior

I expected pytorch 1.0.0 to be in the same ballpark as pytorch 0.4.1, or faster. Instead it is from 20% to 65% slower. I tested on two different servers, one old `Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz` and another newer `Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz`.

Results of `%timeit` on `Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz`:

pytorch 0.4.1:
```
In [4]: %timeit with torch.no_grad(): model(torch.zeros(1, 3, 224, 224))
362 ms ± 40.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

pytorch 1.0.0:
```
In [4]: %timeit with torch.no_grad(): model(torch.zeros(1, 3, 224, 224))
596 ms ± 57.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Results on `Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz`:

pytorch 0.4.1:
```
In [6]: %timeit with torch.no_grad(): model(torch.zeros(1, 3, 224, 224))
174 ms ± 786 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

pytorch 1.0.0:
```
In [5]: %timeit with torch.no_grad(): model(torch.zeros(1, 3, 224, 224))
210 ms ± 593 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

~~Also on the same server (8 virtual cores), without ``OMP_NUM_THREADS=1`` I get ``143 ms ± 4.14 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)`` for 0.4.1, and ``4.05 s ± 126 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`` for 1.0.0 (so 28 times slower).~~ (edited: this bit is resolved in 1.0.1)

## Environment

For pytorch 0.4.1:
```
# python3 collect_env.py
Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 18.04.1 LTS
GCC version: (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: 9.2.148
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```


Same for 1.0.0:

```
# python3 collect_env.py
Collecting environment information...
PyTorch version: 1.0.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 18.04.1 LTS
GCC version: (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: 9.2.148
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```