Repository: pytorch
Issue: Performance difference between 0.4.1 and 1.1.0
Labels: module: performance,triaged

## ❓ Questions and Help

The avearge runtime of my model under the environment of PyTorch 0.4.1.post2 is about 16ms, while changing to PyTorch 1.1.0.dev20190411, the time is increased to 20ms.

runtime in 0.4.1
![QQ截图20190430161427](https://user-images.githubusercontent.com/11815604/56949065-8132c300-6b64-11e9-960a-84f207c98bda.png)

runtime in 1.1.0
![QQ截图20190430161534](https://user-images.githubusercontent.com/11815604/56949109-97d91a00-6b64-11e9-8bb9-9d7dd570b5d0.png)


Moreover, the performance drops ~1.0 AP (object detection) using PyTorch 1.1.0 compared with 0.4.1.

I wonder whether some improvements in 0.4.1 are removed in 1.1.0 or their implementations have been changed?

## Environment
### 1.1.0
PyTorch version: 1.1.0.dev20190411
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080

Nvidia driver version: 410.73
cuDNN version: Probably one of the following:
/usr/local/MATLAB/R2016b/bin/glnxa64/libcudnn.so.4.0.7
/usr/local/cuda-8.0/lib64/libcudnn.so.7.0.2
/usr/local/cuda-8.0/lib64/libcudnn_static.a
/usr/local/cuda-9.0/cuda/lib64/libcudnn.so.7.0.5
/usr/local/cuda-9.0/cuda/lib64/libcudnn_static.a
/usr/local/cuda-9.0/lib64/libcudnn.so
/usr/local/cuda-9.0/lib64/libcudnn.so.7
/usr/local/cuda-9.0/lib64/libcudnn.so.7.0.5
/usr/local/cuda-9.0/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] Could not collect
[conda] blas                      1.0                         mkl  
[conda] mkl                       2018.0.3                      1  
[conda] mkl_fft                   1.0.6            py37h7dd41cf_0  
[conda] mkl_random                1.0.1            py37h4414c95_1  
[conda] pytorch-nightly           1.1.0.dev20190411 py3.7_cuda10.0.130_cudnn7.4.2_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torch                     1.0.0a0+54e8623           <pip>

### 0.4.1
PyTorch version: 0.4.1.post2
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080

Nvidia driver version: 410.73
cuDNN version: Probably one of the following:
/usr/local/MATLAB/R2016b/bin/glnxa64/libcudnn.so.4.0.7
/usr/local/cuda-8.0/lib64/libcudnn.so.7.0.2
/usr/local/cuda-8.0/lib64/libcudnn_static.a
/usr/local/cuda-9.0/cuda/lib64/libcudnn.so.7.0.5
/usr/local/cuda-9.0/cuda/lib64/libcudnn_static.a
/usr/local/cuda-9.0/lib64/libcudnn.so
/usr/local/cuda-9.0/lib64/libcudnn.so.7
/usr/local/cuda-9.0/lib64/libcudnn.so.7.0.5
/usr/local/cuda-9.0/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] numpy (1.16.2)
[pip] torch (0.4.1.post2)
[pip] torch-encoding (0.5.2)
[pip] torchfile (0.1.0)
[pip] torchnet (0.0.4)
[pip] torchvision (0.2.1)
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.0                      118  
[conda] mkl_fft                   1.0.6            py36h7dd41cf_0  
[conda] mkl_random                1.0.1            py36h4414c95_1  
[conda] pytorch                   0.4.1           py36_py35_py27__9.0.176_7.1.2_2    pytorch
[conda] torch-encoding            0.4.5+69f6e1c             <pip>
[conda] torch-encoding            0.5.2                     <pip>
[conda] torchfile                 0.1.0                     <pip>
[conda] torchnet                  0.0.4                     <pip>
[conda] torchvision               0.2.1                    py36_1    pytorch