Repository: pytorch
Issue: AMP does not improve performance
Labels: triaged,module: amp (automated mixed precision)

## 🐛 Bug

Using `torch.autocast` does not improve my performance.

## To Reproduce

My model is a GPT2 transformer from Huggingface's `transformers`.

```python
for epoch in trange(epochs):
    # calculate loss
    with torch.autocast("cuda", enabled=USE_AMP):
        outputs = model(**batch)
        loss = loss_criterion(outputs.logits, labels)
     scaler.scale(loss).backward()
```
Regardless of the value of `USE_AMP`, I get about 1.1it/s. 

## Expected behavior

I would expect some meaningful speed up (10-20%).

## Environment

```
Collecting environment information...
PyTorch version: 1.10.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux Server release 7.9 (Maipo) (x86_64)
GCC version: (GCC) 10.3.0
Clang version: Could not collect
CMake version: version 2.8.12.2
Libc version: glibc-2.17

Python version: 3.8.11 (default, Aug  3 2021, 15:09:35)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-PCIE-16GB
Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.19.2
[pip3] torch==1.10.0+cu113
[pip3] torchaudio==0.10.0+cu113
[pip3] torchvision==0.11.1+cu113
[conda] _pytorch_select           0.1                       cpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libmklml                  2019.0.5                      0
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py38he904b0f_0
[conda] mkl_fft                   1.3.0            py38h54f3939_0
[conda] mkl_random                1.1.1            py38h0573a6f_0
[conda] mypy                      0.910              pyhd3eb1b0_0
[conda] mypy_extensions           0.4.3                    py38_0    conda-forge
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] torch                     1.10.0+cu113             pypi_0    pypi
[conda] torchaudio                0.10.0+cu113             pypi_0    pypi
[conda] torchvision               0.11.1+cu113             pypi_0    pypi
```

The Tesla V100 supports fp16 but is not Ampere, so I don't think I'm using `tf32` by default.

## Additional context

I am worried about the multiple of 8 constraint. Does literally every number have to be a multiple of 8? My batches are of size 1 due to GPU memory constraints, and I have exactly 1 example that I am trying to overfit.

I also have a screenshot of my GPU usage. I am confident that I am saturating the GPU during training.

<img width="662" alt="Screen Shot 2021-11-04 at 6 03 52 PM" src="https://user-images.githubusercontent.com/26638161/140427903-fc99eb36-9123-42cf-aee3-aa1691ece206.png">

cc @mcarilli @ptrblck