Repository: pytorch
Issue: [JIT] Huge delay (1274s vs 0.031s) when running scripted model
Labels: high priority,oncall: jit,triaged

## 🐛 Bug

When a model that worked properely on a pure python is scripted, it works with a huge delay (1274s vs 0.031s before scripting), while the result at last is the same.

Below are timings for processing 2 copies of the same image. Note that sometimes it is the 1st image that is processed very slowly, while sometimes it is the 2nd one. I've tested 2 models: the simple, wherer delay is about 10s vs 0.008s and more complex where delay is about 1000s vs 0.056s, tests were done on 2 environments: latest nightly pytorch 1.5.0a0+2dc2933 on 1080Ti and on pytorch 1.4.0 on NVIDIA 1050.

Results:

1.5.0a0+2dc2933 on 1080Ti
cfg/yolov3-tiny.cfg weights/yolov3-tiny.pt (more simple model)
Model Summary: 37 layers, 8.85237e+06 parameters, 8.85237e+06 gradients
before scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (0.748s)
image 2/2 data\samples\TEST_AO_2_2_4300002.jpeg: Done. (0.008s) _(...to compare with...)_
after scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (0.170s)
image 2/2 data\samples\TEST_AO_2_2_4300002.jpeg: Done. (11.391s) **(not good...)**

cfg/yolov3.cfg weights/yolov3.pt (more complex model)
Model Summary: 222 layers, 6.19491e+07 parameters, 6.19491e+07 gradients
before scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (0.768s)
image 2/2 data\samples\TEST_AO_2_2_4300001copy.jpeg: Done. (0.031s) _(...to compare with...)_
after scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (1.392s)
image 2/2 data\samples\TEST_AO_2_2_4300001copy.jpeg: Done. (579.728s) **(... terrible)**

1.4.0 on 1050:
cfg/yolov3-tiny.cfg weights/yolov3-tiny.pt (more simple)
Model Summary: 37 layers, 8.85237e+06 parameters, 8.85237e+06 gradients
before scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (0.862s) _(...to compare with...)_
image 2/2 data\samples\TEST_AO_2_2_4300001copy.jpeg: Done. (0.014s)
after scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (5.997s)  **(not good...)**
image 2/2 data\samples\TEST_AO_2_2_4300001copy.jpeg: Done. (0.012s)

cfg/yolov3.cfg weights/yolov3.pt (more complex model)
Model Summary: 222 layers, 6.19491e+07 parameters, 6.19491e+07 gradients
before scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (1.009s) _(...to compare with...)_
image 2/2 data\samples\TEST_AO_2_2_4300001copy.jpeg: Done. (0.056s)
after scripting:
image 1/2 data\samples\TEST_AO_2_2_4300001.jpeg: Done. (1274.444s)  **(... terrible)**
image 2/2 data\samples\TEST_AO_2_2_4300001copy.jpeg: Done. (0.053s)

## To Reproduce
Sorry that I didn't produce more localized example. 
Steps to reproduce the behavior:
1. git clone https://github.com/IlyaOvodov/yolov3.git
2. git checkout performance_problem_test
3. Download weights yolov3-tiny.pt (for simple model) and yolov3.pt (for complex one) from https://drive.google.com/drive/folders/1LezFG5g3BCW6iYaV89B2i64cqEUZD7e0
and place them into weights directory
4. python test_timing.py
This run a simple model test (with has a 5-10s delay)
For a complex model test uncomment line 26 in test_timing.py: `cfg, weights = ('cfg/yolov3.cfg', 'weights/yolov3.pt')`

## Environment
1.
PyTorch version: 1.5.0a0+2dc2933
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: version 3.12.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 1080 Ti
Nvidia driver version: 441.87
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.2
[pip3] pytorch-ignite==0.2.0
[pip3] pytorch-toolbelt==0.2.1
[pip3] torch==1.5.0a0+2dc2933
[pip3] torchvision==0.4.1a0+d94043a
[conda] blas                      1.0                         mkl
[conda] cuda92                    1.0                           0    pytorch
[conda] mkl                       2019.0                   pypi_0    pypi
[conda] mkl-include               2019.0                   pypi_0    pypi
[conda] mkl_fft                   1.0.6            py36hdbbee80_0
[conda] mkl_random                1.0.1            py36h9258bd6_0
[conda] numpy                     1.17.2                   pypi_0    pypi
[conda] pytorch-ignite            0.2.0                     dev_0    <develop>
[conda] pytorch-toolbelt          0.2.1                     dev_0    <develop>
[conda] torch                     1.5.0a0+2dc2933           dev_0    <develop>
[conda] torchvision               0.4.1a0+d94043a           dev_0    <develop>

2.
Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 7 Џа®дҐббЁ®­ «м­ п
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 1050
Nvidia driver version: 441.22
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.6.3
[pip3] numpy==1.18.0
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] efficientnet-pytorch      0.6.3                     <pip>
[conda] mkl                       2019.0                    <pip>
[conda] mkl-include               2019.0                    <pip>
[conda] numpy                     1.18.0                    <pip>
[conda] torch                     1.4.0                     <pip>
[conda] torchvision               0.5.0                     <pip>



cc @ezyang @gchanan @zou3519 @suo