Repository: pytorch
Issue: RuntimeError in Scaled Dot Product Attention Tutorial Code
Labels: module: cpu,triaged

### 🐛 Describe the bug

When running the [Explicit Dispatcher Control tutorial code](https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html#explicit-dispatcher-control), encountered this error on a CPU only device.
```
Traceback (most recent call last):
  File "/opt/project/sandbox/flash_attention/flash_attention_testbed.py", line 32, in <module>
    print(f"The default implementation runs in {benchmark_torch_function_in_microseconds(F.scaled_dot_product_attention, query, key, value):,.3f} microseconds")
  File "/opt/project/sandbox/flash_attention/flash_attention_testbed.py", line 18, in benchmark_torch_function_in_microseconds
    return t0.blocked_autorange().mean * 1e6
  File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/utils/benchmark/utils/timer.py", line 394, in blocked_autorange
    number = self._estimate_block_size(min_run_time)
  File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/utils/benchmark/utils/timer.py", line 311, in _estimate_block_size
    time_taken = self._timeit(number)
  File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/utils/benchmark/utils/timer.py", line 256, in _timeit
    return max(self._timer.timeit(number), 1e-9)
  File "/home/ray/anaconda3/lib/python3.8/timeit.py", line 177, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 6, in inner
RuntimeError: "baddbmm_with_gemm" not implemented for 'Half'
```

Code in question is this
```
import torch
import torch.nn as nn
import torch.nn.functional as F

# Lets define a helpful benchmarking function:
import torch.utils.benchmark as benchmark
def benchmark_torch_function_in_microseconds(f, *args, **kwargs):
    t0 = benchmark.Timer(
        stmt="f(*args, **kwargs)", globals={"args": args, "kwargs": kwargs, "f": f}
    )
    return t0.blocked_autorange().mean * 1e6

device = "cuda" if torch.cuda.is_available() else "cpu"

# Lets define the hyper-parameters of our input
batch_size = 32
max_sequence_len = 1024
num_heads = 32
embed_dimension = 32

dtype = torch.float16

query = torch.rand(batch_size, num_heads, max_sequence_len, embed_dimension, device=device, dtype=dtype)
key = torch.rand(batch_size, num_heads, max_sequence_len, embed_dimension, device=device, dtype=dtype)
value = torch.rand(batch_size, num_heads, max_sequence_len, embed_dimension, device=device, dtype=dtype)

print(f"The default implementation runs in {benchmark_torch_function_in_microseconds(F.scaled_dot_product_attention, query, key, value):.3f} microseconds")

# Lets explore the speed of each of the 3 implementations
from torch.backends.cuda import sdp_kernel, SDPBackend

# Helpful arguments mapper
backend_map = {
    SDPBackend.MATH: {"enable_math": True, "enable_flash": False, "enable_mem_efficient": False},
    SDPBackend.FLASH_ATTENTION: {"enable_math": False, "enable_flash": True, "enable_mem_efficient": False},
    SDPBackend.EFFICIENT_ATTENTION: {
        "enable_math": False, "enable_flash": False, "enable_mem_efficient": True}
}

with sdp_kernel(**backend_map[SDPBackend.MATH]):
    print(f"The math implementation runs in {benchmark_torch_function_in_microseconds(F.scaled_dot_product_attention, query, key, value):.3f} microseconds")


with sdp_kernel(**backend_map[SDPBackend.FLASH_ATTENTION]):
    try:
        print(f"The flash attention implementation runs in {benchmark_torch_function_in_microseconds(F.scaled_dot_product_attention, query, key, value):.3f} microseconds")
    except RuntimeError:
        print("FlashAttention is not supported. See warnings for reasons.")

with sdp_kernel(**backend_map[SDPBackend.EFFICIENT_ATTENTION]):
    try:
        print(f"The memory efficient implementation runs in {benchmark_torch_function_in_microseconds(F.scaled_dot_product_attention, query, key, value):.3f} microseconds")
    except RuntimeError:
        print("EfficientAttention is not supported. See warnings for reasons.")
```

**Note**:  

* Changing  `dtype` from `torch.float16` to `torch.float32` does not generate the error.
* The ["Hardware Dependence" section](https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html#hardware-dependence) of the tutorial does not mention any limitation of hardware other than run-time duration.
* As I understand [this page](https://pytorch.org/docs/stable/tensors.html#data-types)  `dtype` `torch.float16` is supported for the `cpu` device.

I with expecting the tutorial code would run successfully on a cpu only device.


### Versions

The physical runtime environment is a 
* MacBook Pro (16-inch, 2019)
*  MacOS: 12.6.1
* Docker Desktop for Mac: 4.9.1

The failing code is executed in a Docker container...
```
Collecting environment information...
PyTorch version: 2.0.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.13 (default, Oct 21 2022, 23:50:54)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.10.104-linuxkit-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s):                       8
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           158
Model name:                      Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Stepping:                        13
CPU MHz:                         2300.000
BogoMIPS:                        4600.00
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        2 MiB
L3 cache:                        128 MiB
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable; SMT Host state unknown
Vulnerability Meltdown:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, STIBP: disabled
Vulnerability Srbds:             Unknown: Dependent on hypervisor status
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht pbe syscall nx pdpe1gb lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid pni pclmulqdq dtes64 ds_cpl ssse3 sdbg fma cx16 xtpr pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 avx2 bmi2 erms xsaveopt arat

Versions of relevant libraries:
[pip3] botorch==0.8.5
[pip3] gpytorch==1.10
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.2
[pip3] torch==2.0.0+cpu
[pip3] torchaudio==2.0.1+cpu
[pip3] torchdata==0.6.0
[pip3] torchinfo==1.7.2
[pip3] torchmetrics==0.11.4
[pip3] torchtest==0.5
[pip3] torchtext==0.15.1+cpu
[pip3] torchvision==0.15.1+cpu
[pip3] torchviz==0.0.2
[conda] botorch                   0.8.5                    pypi_0    pypi
[conda] gpytorch                  1.10                     pypi_0    pypi
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     2.0.0+cpu                pypi_0    pypi
[conda] torchaudio                2.0.1+cpu                pypi_0    pypi
[conda] torchdata                 0.6.0                    pypi_0    pypi
[conda] torchinfo                 1.7.2                    pypi_0    pypi
[conda] torchmetrics              0.11.4                   pypi_0    pypi
[conda] torchtest                 0.5                      pypi_0    pypi
[conda] torchtext                 0.15.1+cpu               pypi_0    pypi
[conda] torchvision               0.15.1+cpu               pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi

```

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10