Repository: pytorch
Issue: FakeTensor consolidated strategy for in_kernel_invocation and dispatch keys
Labels: triaged,module: fakeTensor

### 🐛 Describe the bug

@eellison has done a lot of good work fixing segfaults related to meta tensors incorrectly reporting themselves as cpu tensors (due to fake tensor) and then causing in c++ code that is subsequently expecting the tensor in question to actually be a cpu tensor and have real data. This issue is an attempt to explicate the underlying conceptual strategy for how to decide whether or not to treat a fake tensor as a cpu tensor or not.

The fundamental problem is we have two opposing requirements:

* We wish fake tensors to mimic CPU/CUDA tensors as much as possible, so that a fake tensor behaves much the same way as their real brethren. This applies to both to device tests (e.g., looking at device, or is_cpu) as well as to dispatch key dispatch (relevant for any functionality keys, esp autocast)
* We don't want any code that directly accesses pointer data to operate on fake tensors, as there isn't any data and you will segfault.

Here is a simple heuristic for how to distinguish:

* In Python, direct data access never happens, we advertise as CPU/CUDA 
* In C++, where direct data access can happen, we advertise as Meta

But this is inaccurate on a number of fronts:

* When we autocast, we want to get the CPU/CUDA autocast logic on meta tensors, otherwise the dtype is not accurate. But we don't set CPU/CUDA dispatch key on fake tensor, so we don't get the corresponding autocast logic. Repro:

```
import torch
from torch._subclasses.fake_tensor import FakeTensorMode

def f():
    x = torch.randn(2, 3, requires_grad=True)
    y = torch.randn(3, 4)
    with torch.autocast('cpu'):
        r = x @ y 
        return r 

with FakeTensorMode():
    r = f()
print(r)
# prints FakeTensor(cpu, torch.Size([2, 4]), torch.float32)
# should be bfloat32
```

* When we query for device on a FakeTensor, this unconditionally dispatches to `torch.ops.aten.device`, no matter if subclass dispatch is disabled. This means that without some other logic, we will report the wrong device type in C++. This is why `in_kernel_invocation_manager` exists in the fake tensor implementation today

It would be good to have a clearer delineation of the boundary here. This is also intimately related to https://github.com/pytorch/pytorch/pull/81471 where FunctionalWrapperTensors don't have full implementation and also must avoid hitting Dense kernels.

It seems to me that the right way to do this is to focus on "direct data access". Our primary problem is avoiding direct access to data pointers. This happens only at a very specific part of the dispatch key order (Dense and similar). So we just need to make sure these keys are not runnable, either by removing the Dense from the functionality dispatch key set, or by adding another dispatch key in front of all of these keys to "block" execution of those keys (this is how Python key and wrapper subclass works).

PS What's the difference between wrapper subclass and fake tensor? At this point, it seems primarily because fake tensor "is a" meta tensor, which makes it marginally more efficient (one dynamic allocation rather than two) and means in C++ we can go straight to meta tensor implementation in the back... it's kind of weak, sorry @eellison for making you suffer with is-a haha.

cc @bdhirsh @Chillee 

### Versions

master