Repository: pytorch
Issue: FP16 is much slower than FP32 on GTX1660 SUPER
Labels: module: performance,module: cuda,triaged,module: half

### 🐛 Describe the bug

I tested the performance of my GTX1660 SUPER with the following script.
```python
from torch.utils import benchmark
import pandas as pd
from collections import defaultdict
import inspect
import torch

print('Pytorch version\t:', torch.__version__)
print('CUDA version\t:', torch.version.cuda)
print('GPU\t\t:', torch.cuda.get_device_name())

pd.options.display.precision = 3


def var_dict(*args):
    callers_local_vars = inspect.currentframe().f_back.f_locals.items()
    return dict([(name, val) for name, val in callers_local_vars if val is arg][0]
                for arg in args)


def walltime(stmt, arg_dict, duration=3):
    return benchmark.Timer(stmt=stmt, globals=arg_dict).blocked_autorange(
        min_run_time=duration).median


matmul_tflops = defaultdict(lambda: {})
for n in [1024, 2048, 4096, 8192]:
    for dtype in (torch.float32, torch.float16):
        a = torch.randn(n, n, dtype=dtype).cuda()
        b = torch.randn(n, n, dtype=dtype).cuda()
        t = walltime('a @ b', var_dict(a, b))
        matmul_tflops[f'n={n}'][dtype] = 2*n**3 / t / 1e12
        del a, b

print(pd.DataFrame(matmul_tflops))
```
And the result shows that the peformance of FP16 is extremely low, only about 0.5TFLOPS.
```
Pytorch version : 2.2.1+cu121
CUDA version    : 12.1
GPU             : NVIDIA GeForce GTX 1660 SUPER
               n=1024  n=2048  n=4096  n=8192
torch.float32   2.839   3.364   4.859   4.626
torch.float16   0.560   0.555   0.573   0.569
```

### Versions

PyTorch version: 2.2.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.28.3
Libc version: N/A

Python version: 3.10.13 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:24:38) [MSC v.1916 64 bit 
(AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 SUPER
Nvidia driver version: 537.58
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cudnn_ops_train64_8.dll     
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2904
DeviceID=CPU0
Family=205
L2CacheSize=1536
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2904
Name=Intel(R) Core(TM) i5-10400F CPU @ 2.90GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.26.2
[pip3] torch==2.2.1+cu121
[pip3] torchaudio==2.2.1+cu121
[pip3] torchvision==0.17.1+cu121
[pip3] triton==2.0.0
[conda] numpy                     1.26.2                   pypi_0    pypi
[conda] torch                     2.2.1+cu121              pypi_0    pypi
[conda] torchaudio                2.2.1+cu121              pypi_0    pypi
[conda] torchvision               0.17.1+cu121             pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi

cc @ptrblck