Repository: pytorch
Issue: Subclassed Tensors Decrease Training GPU Throughput by ~40% 
Labels: module: performance,triaged,module: __torch_function__,tensor subclass,oncall: pt2,module: pt2-dispatcher

### 🐛 Describe the bug

Reopening #79321 per @lezcano's [comment](https://github.com/pytorch/pytorch/issues/79321#issuecomment-1383178277) as the ~40% reduction in training performance is still repeatable in PyTorch 1.13 and PyTorch 2.0.

I created a [minimal reproduction script](https://github.com/warner-benjamin/subclassed_tensors) with no external dependencies outside of a standard PyTorch install. I benchmarked  PyTorch 1.13 with Cuda 11.7 and PyTorch 2.0 (Feb 5) with Cuda 11.8 on my local GPU, using a ResNet50, 224px image size, batch size of 96, and mixed precision. 

In all cases, using subclassed tensors results in up to ~40% worse performance, undoing both the expected channels last and `torch.compile` speedup.

I am a contributor to [fastai](https://docs.fast.ai) which uses subclassed tensors for multiple features, including preserving metadata, built in display methods, and dispatching based on the subclassed tensor type. I created a workaround where fastai casts any subclassed tensor back to `torch.Tensor` before passing a batch to the model.

If this performance bug can't be fixed or a subclassed tensor should always be cast back to a `torch.Tensor` before use, I think the [subclassing tensor documentation](https://pytorch.org/docs/stable/notes/extending.html#subclassing-torch-tensor) needs to be updated to warn about performance degradation, as right now it reads as if subclassed tensors should behave the same as normal tensors.

## PyTorch 1.13: Samples/Second
| Subclass | Channels Last | Mean | Std Dev |
|----------|---------------|-----------------------|--------------------------|
| False    | False         | 865.11                | 46.73                    |
| False    | True          | 1033.73               | 57.12                    |
| True     | False         | 683.34                | 36.24                    |
| True     | True          | 627.90                | 32.99                    |

## PyTorch 2.0: Samples/Second
| Subclass | Channels Last | Compile | Mean | Std Dev |
|----------|---------------|---------|-----------------------|--------------------------|
| False    | False         | False   | 850.30                | 48.07                    |
| False    | True          | False   | 1050.69               | 59.87                    |
| False    | True          | True    | 1074.00               | 63.65                    |
| True     | False         | False   | 684.66                | 36.84                    |
| True     | True          | False   | 640.14                | 34.48                    |
| True     | True          | True    | 637.26                | 34.90                    |

### Versions

PyTorch version: 2.0.0.dev20230205
Is debug build: False
CUDA used to build PyTorch: 11.8

GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-6.0.12-76060006-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[conda] blas                      2.16                        mkl    conda-forge
[conda] libblas                   3.8.0                    16_mkl    conda-forge
[conda] libcblas                  3.8.0                    16_mkl    conda-forge
[conda] liblapack                 3.8.0                    16_mkl    conda-forge
[conda] liblapacke                3.8.0                    16_mkl    conda-forge
[conda] mkl                       2020.2                      256  
[conda] numpy                     1.22.4           py39hc58783e_0    conda-forge
[conda] pytorch                   2.0.0.dev20230205 py3.9_cuda11.8_cudnn8.7.0_0    pytorch-nightly
[conda] pytorch-cuda              11.8                 h8dd9ede_2    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchaudio                2.0.0.dev20230205      py39_cu118    pytorch-nightly
[conda] torchtriton               2.0.0+0d7e753227            py39    pytorch-nightly
[conda] torchvision               0.15.0.dev20230205      py39_cu118    pytorch-nightly

cc @hameerabbasi @rgommers @peterbell10 @ezyang @msaroufim @albanD @wconstab @bdhirsh @anijain2305 @zou3519 @gchanan @soumith @ngimel