Repository: pytorch
Issue: Code becomes more than x20 slower after upgrading torch version
Labels: module: performance,triaged

## Problem
My code runs very fast in the environment with torch 1.5.1 and cudatoolkit 10.2.89.
But in 2 other environments with torch and cudatoolkit versions (1.6.0, 10.1.243) and (1.7.0, 10.2.89) the same code becomes more than 20 times slower.

## To Reproduce
here's the code:
```py
import numpy as np
import time

import torch
import torch.nn as nn

class DownSampleBlock(nn.Module):
    def __init__(self, in_c, out_c, kernel_size=3, dilation=1, dropout_rate=0.1):
        super().__init__()
        padding = (dilation - 1) + kernel_size // 2
        self.block = nn.Sequential(
            nn.Conv2d(in_c, out_c, kernel_size=kernel_size, stride=2, padding=kernel_size // 2, bias=False),
            nn.BatchNorm2d(out_c),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_c, out_c, kernel_size=kernel_size, stride=1, dilation=dilation, padding=padding, bias=False),
            nn.BatchNorm2d(out_c),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_c, out_c, kernel_size=kernel_size, stride=1, dilation=dilation, padding=padding, bias=False),
            nn.BatchNorm2d(out_c),
            nn.ReLU(inplace=True),
            nn.Dropout2d(p=dropout_rate)
        )

    def forward(self, inp):
        return self.block(inp)


class TransformBlock(nn.Module):
    def __init__(self, in_c, out_c, kernel_size=3, dilation=1, dropout_rate=0.1):
        super().__init__()
        padding = (dilation - 1) + kernel_size // 2
        self.block = nn.Sequential(
            nn.Conv2d(in_c, out_c, kernel_size=kernel_size, stride=1, dilation=dilation, padding=padding, bias=False),
            nn.BatchNorm2d(out_c),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_c, out_c, kernel_size=kernel_size, stride=1, dilation=dilation, padding=padding, bias=False),
            nn.BatchNorm2d(out_c),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_c, out_c, kernel_size=kernel_size, stride=1, dilation=dilation, padding=padding, bias=False),
            nn.BatchNorm2d(out_c),
            nn.ReLU(inplace=True),
            nn.Dropout2d(p=dropout_rate)
        )

    def forward(self, inp):
        return self.block(inp)


class Net(nn.Module):
    def __init__(self):
        super().__init__()
        C=64
        self.drop=0.1
        self.embed = nn.Embedding(num_embeddings=100, embedding_dim=10, padding_idx=0, sparse=True)
        self.block1 = DownSampleBlock(in_c=10, out_c=C, dilation=1, dropout_rate=0.1)
        self.block2 = DownSampleBlock(in_c=C, out_c=2 * C, dilation=1, dropout_rate=self.drop)
        self.block3 = DownSampleBlock(in_c=2 * C, out_c=4 * C, dilation=2, dropout_rate=self.drop)
        self.block4 = TransformBlock(in_c=4 * C, out_c=8 * C, dilation=4, dropout_rate=self.drop)
        
    def forward(self, x):
        emb = self.embed(x).permute(0, 3, 1, 2) # bring embedding dim to channels position
        o1 = self.block1(emb)
        o2 = self.block2(o1)
        o3 = self.block3(o2)
        o4 = self.block4(o3)
        return o4


if __name__ == '__main__':
	seed= 1
	torch.backends.cudnn.enabled = True
	torch.backends.cudnn.benchmark = True
	torch.backends.cudnn.deterministic = True
	np.random.seed(seed)
	torch.manual_seed(seed)
	torch.cuda.manual_seed(seed)
	torch.cuda.manual_seed_all(seed)

	device = 'cuda'

	net = Net()
	net.to(device)

	cc = torch.randint(0, 100, [4,300,200]).to(device)

	for i in range(5):	
		sta = time.perf_counter()
		aa = net(cc)
		print(aa.sum().item())
		print("Forward {}: {}\n".format(i, time.perf_counter() - sta))
```

## Expected behavior

I've noticed the problem when simply tried to move my code from one conda environment to another one with newer torch version.
Expected behavior is that updating torch version shouldn't make the code slower.
It is worth mentioning that if the last TransformBlock or the very first nn.Embedding layer is removed (and the input shape is changed accordingly), then the code becomes very fast again in all environments.
Also, in the environments with newer torch versions the code runs faster on cpu than on cuda.

## Environment

I've created several conda environments manually the main difference of which are the torch versions (see above). This issue was reproduced by my colleague on his computer as well. My GPU is 1080, Nvidia driver version is 450.80.02.
OS is Ubuntu 18.04

cc @VitalyFedyunin @ngimel