Repository: pytorch
Issue: Significantly slower in latest version than in 0.4.0
Labels: needs reproduction,module: performance,module: cuda,triaged

Hi,

if I run the following script to train a model, one can aware significantly performance drawbacks in 1.2.0 than in 0.4.0.

```
import torch 
import torch.nn as nn
import torch.optim as optim
from torch.optim.lr_scheduler import MultiStepLR
import time 

class DummyModel(nn.Module):
    def __init__(self): 
        super(DummyModel, self).__init__() 
        
        self.conv1Ds= nn.Sequential(
            nn.Conv1d(60, 512, 5, padding=2), 
            nn.ReLU(),
            nn.BatchNorm1d(512),

            nn.Conv1d(512, 512, 3, padding=1),
            nn.ReLU(),
            nn.BatchNorm1d(512),

            nn.Conv1d(512, 512, 3, dilation=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm1d(512),

            nn.Conv1d(512, 512, 3, dilation=4, padding=4),
            nn.ReLU(),
            nn.BatchNorm1d(512),

            nn.Conv1d(512, 512, 3, dilation=6, padding=6),
            nn.ReLU(),
            nn.BatchNorm1d(512),

            nn.Conv1d(512, 512, 1),
            nn.ReLU(),
            nn.BatchNorm1d(512)).cuda()

        self.linears = nn.Sequential(
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.BatchNorm1d(512),

            nn.Linear(512, 512),
            nn.ReLU(),
            nn.BatchNorm1d(512),

            nn.Linear(512, 30)).cuda()

    def forward(self, x):
        x = x.permute(0, 2, 1)
        x = self.conv1Ds(x)
        x = x.permute(0, 2, 1)
        x = self.linears(x)
        return x


def train():
    batch_size = 150
    feature = torch.rand(batch_size, 512, 60).cuda()
    label = torch.rand(batch_size, 30).long().cuda()

    criterion = nn.CrossEntropyLoss(ignore_index=-1).cuda()
    optimizer = optim.Adam(net.parameters(), lr=0.0005)

    for epoch in range(20):
        net.zero_grad()
        predict = net(feature)
        loss = criterion(predict, label)
        loss.backward()
        optimizer.step()


if __name__ == '__main__':
    print('torch version: {}'.format(torch.__version__))                                                                                                                                                                                         
    from_time = time.time()                                                                                                                                                                                                                      
    net = DummyModel()                                                                                                                                                                                                                           
    train()                                                                                                                                                                                                                                      
    running_time = time.time() - from_time                                                                                                                                                                                                       
    print('running time: {}'.format(running_time))
```
I'm not sure if this is a duplicate as [this one](https://github.com/pytorch/pytorch/issues/14456), since according to [soumith](https://github.com/pytorch/pytorch/issues/15054#issuecomment-488482160), the newly installed pytorch should have the correct cudnn that fixed this issure. 

Config:
OS: Ubuntu 16.04.6 LTS
Cuda: 10.1
NVIDIA Driver Version: 418.56 
GPU: 1080Ti