Repository: pytorch
Issue: RTX3090 performs no better than 1080ti
Labels: module: performance,triaged

# Questions

I want to compare the performance of Pytorch on 3090 and 1080ti graphics cards. I find that 3090 is no better than 1080ti when training a network and 1x faster than 1080ti when testing a network. Why?

# Hardware and software configuration

- GPU: ``RTX3090`` (MSI VENTUS 3X 24G OC), ``GTX1080ti``
- CPU: ``E5-2696v3``
- Memory: ``64G``
- PCI version: 3.0
- Power Supply: 1000W
- CUDA: ``11.1``
- CUDNN: ``8005``
- Pytorch: ``1.8.0.dev20210106+cu110``


# Results

## MNIST classification with a CNN

```bash
Torch Version:  1.8.0.dev20210106+cu110
Torch CUDA Version:  11.0
CUDNN Version:  8005
GPU Device:  GeForce RTX 3090
CUDNN Benchmark:  False
CUDNN Deterministic:  True
CUDA TF32:  True
CUDNN TF32:  True
Workers:  4
Batch Size:  64
Epochs:  10
Training Loss:  0.0009187350056997093
Testing  Loss:  0.03026894662413977
Time: 63.0998



Torch Version:  1.8.0.dev20210106+cu110
Torch CUDA Version:  11.0
CUDNN Version:  8005
GPU Device:  GeForce GTX 1080 Ti
CUDNN Benchmark:  False
CUDNN Deterministic:  True
CUDA TF32:  True
CUDNN TF32:  True
Workers:  4
Batch Size:  64
Epochs:  10
Training Loss:  0.0008879667648342487
Testing  Loss:  0.030606915746741615
Time: 56.9057
```

## Convolution

```bash
Torch version:  1.8.0.dev20210106+cu110
Torch CUDA version:  11.0
CUDNN Version:  8005
GeForce RTX 3090
Conv1d model, benchmark=False, deterministic=False, 0.687ms/iter
Conv1d model, benchmark=True, deterministic=False, 0.511ms/iter
Conv1d model, benchmark=False, deterministic=True, 0.540ms/iter
Conv1d model, benchmark=True, deterministic=True, 0.484ms/iter
Conv2d model, benchmark=False, deterministic=False, 1.327ms/iter
Conv2d model, benchmark=True, deterministic=False, 1.335ms/iter
Conv2d model, benchmark=False, deterministic=True, 1.474ms/iter
Conv2d model, benchmark=True, deterministic=True, 1.480ms/iter
Conv2d1 model, benchmark=False, deterministic=False, 3.278ms/iter
Conv2d1 model, benchmark=True, deterministic=False, 3.280ms/iter
Conv2d1 model, benchmark=False, deterministic=True, 3.286ms/iter
Conv2d1 model, benchmark=True, deterministic=True, 3.286ms/iter

Torch version:  1.8.0.dev20210106+cu110
Torch CUDA version:  11.0
CUDNN Version:  8005
GeForce GTX 1080 Ti
Conv1d model, benchmark=False, deterministic=False, 0.709ms/iter
Conv1d model, benchmark=True, deterministic=False, 0.711ms/iter
Conv1d model, benchmark=False, deterministic=True, 0.844ms/iter
Conv1d model, benchmark=True, deterministic=True, 0.711ms/iter
Conv2d model, benchmark=False, deterministic=False, 2.684ms/iter
Conv2d model, benchmark=True, deterministic=False, 2.883ms/iter
Conv2d model, benchmark=False, deterministic=True, 2.212ms/iter
Conv2d model, benchmark=True, deterministic=True, 2.195ms/iter
Conv2d1 model, benchmark=False, deterministic=False, 5.583ms/iter
Conv2d1 model, benchmark=True, deterministic=False, 6.077ms/iter
Conv2d1 model, benchmark=False, deterministic=True, 6.097ms/iter
Conv2d1 model, benchmark=True, deterministic=True, 6.120ms/iter
```

# Code
 
## MNIST classification with a CNN

The code can be downloaded from [here](https://github.com/antsfamily/thlib/blob/master/tests/device/th_mnist_cnn_device.py).

## Convolution

The code can be downloaded from [here](https://github.com/antsfamily/thlib/blob/master/tests/device/th_conv_device.py).


cc @VitalyFedyunin @ngimel