Repository: pytorch
Issue: Structured kernels have increased TensorIterator overhead.
Labels: module: performance,module: bootcamp,triaged

## 🐛 Bug

Structured kernels are an attempt to rationalize PyTorch kernels with common semantics by codegen-ing much of the intermediate wrapper code. This has, however, increased the overhead of these kernels as a result of some of the internals refactoring. This issue concerns the effect of structuring on our main mathematical helper class. (TensorIterator) This is a good bootcamp task if you're interested in C++ kernels and the machinery that we use to write them.

## To Reproduce

[This script](https://gist.github.com/robieta/2b5d5b4286d29ef5f09870a924cb0f77) will collect instruction counts and wall times; substitute the following to measure this regression:

```
# =============================================================================
# == PR specific code =========================================================
# =============================================================================
COMMIT_0 = "a6fa3b26825fef139aecf7b931aefc7a4c1822ac"
COMMIT_1 = "16b8e6ab011c56ccc6a58f7764cd0592f89f35ac"
make_timer = lambda: Timer(
    "x + 1.0;",
    "auto x = torch::ones({1});",
    language="cpp",
)
```

```
$ python bootcamp.py
40 / 40
<torch.utils.benchmark.utils.common.Measurement object at 0x7f6b754f9c50>
x + 1.0;
setup: auto x = torch::ones({1});
  Median: 5.13 us
  IQR:    0.46 us (5.01 to 5.47)
  8943 measurements, 1 runs per measurement, 1 thread

<torch.utils.benchmark.utils.common.Measurement object at 0x7f6b754f9a90>
x + 1.0;
setup: auto x = torch::ones({1});
  Median: 5.23 us
  IQR:    0.49 us (5.12 to 5.62)
  10247 measurements, 1 runs per measurement, 1 thread

1703718
1708067

<torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.FunctionCounts object at 0x7f6b6c6d4f90>
   10400  build/aten/src/ATen/RegisterCPU.cpp:at::(anonymous namespace)::structured_add_out_functional ... ng, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::TensorOptions, c10::ArrayRef<at::Dimname>)
    5900  build/../aten/src/ATen/native/cpu/Loops.h:void at::native::(anonymous namespace)::vectorized ... t::native::(anonymous namespace)::add_kernel(at::TensorIteratorBase&, c10::Scalar)::{lambda()
    5300  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIteratorBase::build_binary_op(at::Tensor const&, at::Tensor const&, at::Tensor const&)
    4400  build/aten/src/ATen/RegisterCPU.cpp:at::(anonymous namespace)::add_Tensor(at::Tensor const&, at::Tensor const&, c10::Scalar)
    4000  build/../c10/core/DispatchKeySet.h:c10::DispatchKeySet::has(c10::DispatchKey) const
    4000  build/../aten/src/ATen/core/dispatch/Dispatcher.h:at::Tensor c10::Dispatcher::call<at::Tenso ... Tensor const&, c10::Scalar)> const&, at::Tensor const&, at::Tensor const&, c10::Scalar) const
    3700  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIteratorBase::set_output(long, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::TensorOptions, c10::ArrayRef<at::Dimname>)
    3300  build/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp.AVX2.cpp:at::native::(anonymous namespace)::add_kernel(at::TensorIteratorBase&, c10::Scalar)::{lambda()
    3000  build/../aten/src/ATen/native/BinaryOps.cpp:at::native::structured_add_out::impl(at::Tensor&, at::Tensor const&, at::Tensor const&, c10::Scalar)
     ...
   -3300  build/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp.AVX2.cpp:at::native::(anonymous namespace)::add_kernel(at::TensorIterator&, c10::Scalar)::{lambda()
   -3400  build/../aten/src/ATen/record_function.cpp:at::RecordFunction::RecordFunction(at::RecordScope)
   -3500  build/aten/src/ATen/RegisterBackendSelect.cpp:at::(anonymous namespace)::empty_memory_format ... ::Layout>, c10::optional<c10::Device>, c10::optional<bool>, c10::optional<c10::MemoryFormat>)
   -3700  build/../aten/src/ATen/core/boxing/KernelFunction_impl.h:at::empty(c10::ArrayRef<long>, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>)
   -3900  build/../c10/core/TensorOptions.h:c10::computeDispatchKey(c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>)
   -3900  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIterator::set_output(long, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::TensorOptions, c10::ArrayRef<at::Dimname>)
   -4000  build/../aten/src/ATen/native/BinaryOps.cpp:at::native::add(at::Tensor const&, at::Tensor const&, c10::Scalar)
   -5300  build/../aten/src/ATen/TensorIterator.cpp:at::TensorIterator::binary_op(at::Tensor&, at::Tensor const&, at::Tensor const&)
   -5900  build/../aten/src/ATen/native/cpu/Loops.h:void at::native::(anonymous namespace)::vectorized_loop<at::native::(anonymous namespace)::add_kernel(at::TensorIterator&, c10::Scalar)::{lambda()

Total: 4349
```

## Environment

 - PyTorch Version (e.g., 1.0): Nightly. (See commits above.)
 - OS (e.g., Linux): CentOS 8
 - How you installed PyTorch (`conda`, `pip`, source): Built from source
 - Build command you used (if compiling from source): Script provided above
 - Python version: 3.7
 - CUDA/cuDNN version: N/A
 - GPU models and configuration: N/A
 - Any other relevant information: N/A

## Additional context

This is a bootcamp task. If you have any questions, please reach out to @robieta or @orionr 
CC @bitfort, @xuzhao9, @ezyang, @bdhirsh 


cc @VitalyFedyunin @ngimel