Repository: pytorch
Issue: Class-based structured kernels instruction count regression
Labels: module: performance,module: internals,triaged,module: structured kernels

@robieta's benchmark suite found #48718 regressed TensorIterator based operations around 2% instruction count.

I diffed the before and after with also @bdhirsh's #49123 and got (cleaned up)

```
    5600  build/../c10/core/TensorOptions.h:at::Tensor::options() const
    3600  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIterator::set_output ... ng>, c10::ArrayRef<long>, c10::TensorOptions, c10::ArrayRef<at::Dimname>)
    3300  build/aten/src/ATen/core/TensorMethods.cpp:at::Tensor::options() const
    2500  build/aten/src/ATen/core/TensorMethods.cpp:at::Tensor::device() const
    1400  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIteratorBase::allocate_or_resize_outputs()
    1000  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIteratorBase::populate_operands(at::TensorIteratorConfig&)
     800  build/../c10/core/DispatchKeySet.h:at::Tensor::is_meta() const
     600  build/../c10/core/TensorImpl.h:at::Tensor::layout() const
     500  build/../c10/util/SmallVector.h:at::TensorIteratorBase::allocate_or_resize_outputs()
     400  build/aten/src/ATen/core/TensorMethods.cpp:at::Tensor::is_meta() const
     400  build/../c10/util/SmallVector.h:c10::TensorImpl::sizes() const
     400  build/../c10/util/SmallVector.h:at::TensorIterator::set_output(long, c10 ... ng>, c10::ArrayRef<long>, c10::TensorOptions, c10::ArrayRef<at::Dimname>)
     400  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIteratorBase::build(at::TensorIteratorConfig&)
     300  build/aten/src/ATen/core/TensorBody.h:at::TensorIteratorBase::allocate_or_resize_outputs()
     300  build/../c10/util/ArrayRef.h:at::TensorIteratorBase::allocate_or_resize_outputs()
     200  build/aten/src/ATen/core/TensorMethods.cpp:at::Tensor::layout() const
     200  build/aten/src/ATen/core/TensorMethods.cpp:at::Tensor::dtype() const
     200  build/../c10/core/TensorImpl.h:at::Tensor::dtype() const
     200  build/../c10/core/TensorImpl.h:at::Tensor::device() const
     200  build/../c10/core/TensorImpl.cpp:c10::TensorImpl::sizes() const
     200  build/../c10/core/DispatchKeySet.h:at::Tensor::layout() const
     100  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIteratorBase::TensorIteratorBase()
     -26  /build/glibc-ZN95T4/glibc-2.31/malloc/malloc.c:_int_memalign
    -100  /usr/include/c++/9/bits/stl_algobase.h:at::TensorIteratorBase::build(at::TensorIteratorConfig&)
    -194  /build/glibc-ZN95T4/glibc-2.31/malloc/malloc.c:_int_free
    -200  build/../c10/util/SmallVector.h:at::TensorIteratorBase::populate_operands(at::TensorIteratorConfig&)
    -200  build/../aten/src/ATen/native/TypeProperties.cpp:at::native::is_complex(at::Tensor const&)
    -300  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&)

Total: 21780
```

What we can see are that (1) there are a lot more metadata calls (options, device, dtype, etc) accounting for maybe (10k instrs); (2) set_output, allocate_or_resize_outputs and populate_operands are more expensive (5k instrs). We can see that more metadata calls occur in a number of locations in the PR:

* https://github.com/pytorch/pytorch/pull/48718/files#diff-9d753eedf6d7efa0049a1d17ece53ccb89ea0ea0485d2680fae489812484bf8cL494 - options, sizes
* https://github.com/pytorch/pytorch/pull/48718/files#diff-9d753eedf6d7efa0049a1d17ece53ccb89ea0ea0485d2680fae489812484bf8cR953 - is_meta

The cost of all the extra metadata invocations is almost certainly the first bullet point.

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh @ailzhang @VitalyFedyunin @ngimel