Repository: pytorch
Issue: Improve meta tensor testing
Labels: module: tests,triaged,module: structured kernels

https://github.com/pytorch/pytorch/pull/53682 is cool but now that we've built out more support for structured kernels there are some things we can do to improve. Some of these items are more effort than they are worth but we should at least investigate it a little.

Most common reasons why tests get skipped:

```
aten::_solve_helper: 16
aten::frexp.Tensor_out: 16
aten::sort.stable: 18
aten::logspace.out: 18
aten::sort: 19
aten::bmm: 20
aten::linalg_vector_norm: 20
aten::linalg_solve: 20
aten::nonzero: 22
aten::flip: 23
aten::segment_reduce: 24
aten::_fft_r2c: 26
aten::_linalg_qr_helper: 27
aten::index.Tensor: 32
aten::max: 32
aten::linspace.out: 34
aten::_lu_with_info: 35
aten::logit: 47
aten::to_sparse: 50
aten::nan_to_num.out: 50
aten::min: 51
aten::logical_not.out: 52
aten::angle: 56
aten::arange.start_out: 66
aten::randperm.generator_out: 69
aten::min.dim: 87
aten::eye.m_out: 104
aten::logical_or.out: 144
aten::logical_xor.out: 145
aten::logical_and.out: 153
aten::view_as_real: 237
aten::_local_scalar_dense: 246
aten::_cat: 271
aten::_svd_helper: 288
aten::all: 375
aten::repeat_interleave.Tensor: 418
aten::sum: 734
aten::abs.out: 2265
aten::complex.out: 2427
```

This was computed by running `( GLOBIGNORE="test/test_cpp_api_parity.py:test/test_cuda.py:test/test_torch.py:test/test_nn.py:test/test_cpp_extensions_aot.py:test/test_bundled_images.py"; pytest -v --disable-warnings --tb=native -k meta --junitxml=test_all.xml test/*.py)` and then processing it with

```
from junitparser import JUnitXml
import re
from collections import defaultdict
import sys

xml = JUnitXml.fromfile(sys.argv[1])

result = defaultdict(int)

for suite in xml:
    for case in suite:
        if case.result:
            r = case.result[0]
            if r.type == "pytest.skip" and r.message.startswith("not implemented"):
                m = re.search(r"Could not run '([^']+)'", r.text)
                if m:
                    op = m.group(1)
                elif "torch.tensor" in r.text:
                    op = "torch.tensor"
                else:
                    continue
                result[op] += 1 

for k, v in sorted(result.items(), key=lambda item: item[1]):
    print(f"{k}: {v}")
```

Some stuff:

* We can make individual tests more meta tensor friendly by changing some operations to be hard failure as opposed to skip, in any case where it's not possible that adding more meta tensor support will "fix" the issue. In particular, any time we run `foo(out=...)` with a CPU tensor as the output tensor, this should hard error. This might be annoying to actually do as we need to fix all the sites not to use this idiom and/or add skips to them. Speaking of which, we should figure out why complex.out is being called so much!
  * Some of this may involve taking tests that are fine for meta tensors except for one part, and factoring them into two separate parts. Worth spending all the time on it? Maybe not...
* Maybe use Python mode https://github.com/pytorch/pytorch/pull/63496 to force allocations onto meta device? IDK how good of an idea this is; device-generic tests are supposed to take device explicitly anyway, so maybe there's just some refactoring to do
* It would be nice for the statistics above to be captured and then rendered as part of the normal master build process so we can easily track it over time

cc @mruberry @ysiraichi