Repository: pytorch
Issue: version1.7.0 is ~1.3x slower than 1.4.0 on ResNet18
Labels: module: performance,triaged

## 🐛 Bug
If we run the same program, the calculation time to learn a batch with version 1.7.0 is longer than with version 1.4.0. The bigger the batch size is , the larger the difference is. The average time and its standard deviation measured 5 times are as below. It takes up to 1.3 times.

## To Reproduce
https://colab.research.google.com/drive/1rkNRgPbv8_aC1_5SwH62CCGFbaUz5S1p?usp=sharing

```
import torch
import torch.nn as nn
import torchvision
from torch.utils.data import DataLoader
from torchvision import transforms
import torchvision.models as models
import torch.optim as optim
from sklearn.metrics import classification_report
import time
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])
transform_test = transforms.Compose([
  transforms.ToTensor(),
  transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])
train_set = torchvision.datasets.CIFAR10(root="./data", train=True, download=True, transform=transform_train)
test_set = torchvision.datasets.CIFAR10(root="./data", train=False, download=True, transform=transform_test)
class_names = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
device = 'cuda'
all_train_time = []
for i in range(1, 14):
  print(i)
  train_time_list = []
  batch_size = pow(2, i)
  for j in range(5):
    model = models.resnet18().to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=5e-4)
    train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=0)
    start = time.time()
    for epoch in range(1):
      output_list = []
      target_list = []
      running_loss = 0.0
      for batch_idx, (inputs, targets) in enumerate(train_loader):
        inputs, targets = inputs.to(device), targets.to(device)
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        output_list += [int(o.argmax()) for o in outputs]
        target_list += [int(t) for t in targets]
        running_loss += loss.item()
        result = classification_report(output_list, target_list, output_dict=True)
        train_acc = round(result['weighted avg']['f1-score'], 6)
        train_loss = round(running_loss / len(train_loader), 6)
        break
    elapsed_time = time.time() - start
    train_time_list.append(elapsed_time)
  all_train_time.append(train_time_list)
import math
for i in range(13):
  sum = 0
  for j in range(5):
    sum+=all_train_time[i][j]
  average = sum/5
  square_division = 0
  for j in range(5):
    square_division+= pow((average-all_train_time[i][j]), 2)
  square_division/=5
  stdev = math.sqrt(square_division)
  print("batchsize:{0}, average_time:{1:6.3f}s, standard deviation:{2:1.5f}".format(str(pow(2, i+1)).rjust(5),average,stdev))
print("\nwithout_BatchNorm")
```
```
version1.7.0
batchsize:    2, average_time: 0.021s, standard deviation:0.00356
batchsize:    4, average_time: 0.026s, standard deviation:0.00064
batchsize:    8, average_time: 0.035s, standard deviation:0.00093
batchsize:   16, average_time: 0.036s, standard deviation:0.00108
batchsize:   32, average_time: 0.050s, standard deviation:0.00181
batchsize:   64, average_time: 0.076s, standard deviation:0.00170
batchsize:  128, average_time: 0.132s, standard deviation:0.00920
batchsize:  256, average_time: 0.202s, standard deviation:0.01761
batchsize:  512, average_time: 0.351s, standard deviation:0.01826
batchsize: 1024, average_time: 0.750s, standard deviation:0.03474
batchsize: 2048, average_time: 1.463s, standard deviation:0.01464
batchsize: 4096, average_time: 3.052s, standard deviation:0.08125
batchsize: 8192, average_time: 5.963s, standard deviation:0.07113
```
```
version1.4.0
batchsize:    2, average_time: 0.022s, standard deviation:0.00279
batchsize:    4, average_time: 0.024s, standard deviation:0.00145
batchsize:    8, average_time: 0.027s, standard deviation:0.00018
batchsize:   16, average_time: 0.082s, standard deviation:0.00168
batchsize:   32, average_time: 0.045s, standard deviation:0.00167
batchsize:   64, average_time: 0.055s, standard deviation:0.00055
batchsize:  128, average_time: 0.083s, standard deviation:0.00111
batchsize:  256, average_time: 0.156s, standard deviation:0.00458
batchsize:  512, average_time: 0.297s, standard deviation:0.00548
batchsize: 1024, average_time: 0.580s, standard deviation:0.00620
batchsize: 2048, average_time: 1.152s, standard deviation:0.01654
batchsize: 4096, average_time: 2.271s, standard deviation:0.03547
batchsize: 8192, average_time: 4.525s, standard deviation:0.02307
```

## Expected behavior
Version1.7.0 and 1.4.0 have the same execution time.

## Environment
PyTorch version: 1.4.0, 1.7.0
CUDA used to build PyTorch: 10.1
Python version: 3.6.9
Versions of relevant libraries:
[pip3] torchvision==0.8.1

cc @VitalyFedyunin @ngimel