Repository: pytorch
Issue: Conv2D 2x~20x slower than Tensorflow when channel count is small
Labels: module: performance,module: cudnn,module: cuda,module: convolution,triaged

## 🐛 Bug

When I run my benchmark code with `channel_count = 64`  on both TensorFlow and PyTorch, the PyTorch version shows ~2x slower speed than TensorFlow version.
But when I set `channel_count = 256`, Tensorflow and PyTorch perform similar speed.
Is it a 🐛?

Tested on CPU(Ryzen 3700X), Tesla K80(up to 20x slower), Tesla T4(~2x slower), RTX 2070 Super.

## To Reproduce
Run these benchmark code with Python 3
Or run it in Colab: https://colab.research.google.com/drive/13hvxoV-pe1E5fYLiF9BQhzZcS5910y9V

### Tensorflow version:
````python3
import tensorflow as tf
import time

channel_count = 64
# Model
print("* Create model")
p_input = tf.random.uniform(shape=(1, 3, 64, 64), minval=0.0, maxval=1.0, dtype=tf.float32)
p_target = tf.random.uniform(shape=(1, 3, 64, 64), minval=0.0, maxval=1.0, dtype=tf.float32)

v = p_input
v = tf.nn.relu(tf.keras.layers.Conv2D(channel_count, (3, 3), padding="same", data_format="channels_first").apply(v))
v = tf.nn.relu(tf.keras.layers.Conv2D(channel_count, (3, 3), padding="same", data_format="channels_first").apply(v))
v = tf.nn.relu(tf.keras.layers.Conv2D(channel_count, (3, 3), padding="same", data_format="channels_first").apply(v))
v = tf.nn.relu(tf.keras.layers.Conv2D(channel_count, (3, 3), padding="same", data_format="channels_first").apply(v))
v = tf.keras.layers.Conv2D(3, (3, 3), padding="same", data_format="channels_first").apply(v)

loss_fn = tf.reduce_mean(tf.abs(v - p_target))
optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(loss_fn)

with tf.Session() as sess:
  # Initialized, Load state
  sess.run(tf.global_variables_initializer())

  t = time.time()
  for i_step in range(100000):
    loss_value, _ = sess.run([loss_fn, optimizer])
    if i_step % 5000 == 0:
      print(time.time() - t)
      t = time.time()
````

### Torch version:
````python3
import torch
import time

# Model
print("* Create model")

channel_count = 64
net = torch.nn.Sequential(
  torch.nn.Conv2d(3, channel_count, (3, 3), padding=(1, 1)),
  torch.nn.ReLU(),
  torch.nn.Conv2d(channel_count, channel_count, (3, 3), padding=(1, 1)),
  torch.nn.ReLU(),
  torch.nn.Conv2d(channel_count, channel_count, (3, 3), padding=(1, 1)),
  torch.nn.ReLU(),
  torch.nn.Conv2d(channel_count, channel_count, (3, 3), padding=(1, 1)),
  torch.nn.ReLU(),
  torch.nn.Conv2d(channel_count, 3, (3, 3), padding=(1, 1)),
).train().cuda()
#net = torch.jit.trace_module(net, {"forward": torch.empty((1, 3, 64, 64)).uniform_(-1.0, 1.0).cuda()})
train_param_list = net.parameters()

opt = torch.optim.Adam(train_param_list, lr=1e-4, eps=1e-8, weight_decay=0.0)

t = time.time()
for i_step in range(100000):
  opt.zero_grad()
  input_data = torch.empty((1, 3, 64, 64)).uniform_(-1.0, 1.0).cuda()
  target_data = torch.empty((1, 3, 64, 64)).uniform_(-1.0, 1.0).cuda()

  pred = net(input_data)
  loss = (target_data - pred).abs().mean()
  
  loss.backward()
  opt.step()
  if i_step % 5000 == 0:
    print(time.time() - t)
    t = time.time()
````

## Environment

````
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Arch Linux
GCC version: (GCC) 9.1.0
CMake version: version 3.15.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce RTX 2070 SUPER
Nvidia driver version: 435.21
cuDNN version: /usr/lib/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip3] numpy==1.17.1
[pip3] torch==1.2.0
[conda] Could not collect

Tensorflow: 1.14.0
````