Repository: pytorch
Issue: Optimizations to TORCH_CHECK change inlining behavior.
Labels: module: performance,module: bootcamp,triaged

## 🐛 Bug

A number of recent micro-optimizations have been aimed at refining small C++ implementation details of the PyTorch runtime. One such optimization is https://github.com/pytorch/pytorch/pull/52221, which changes how the compiler inlines kernel checks.  The commit message describes some of the mixed results (https://github.com/pytorch/pytorch/commit/efbb854ed8a9df35c0ea896c2d216ab3da10d677), and a PR later in the stack addresses some of them. (https://github.com/pytorch/pytorch/commit/7cd9892f8352c541ebdeea590f6cacb1557e61a5)

The exact impact will vary based on system specifics; it initially manifested as a small increase in instruction counts during a historic sweep, however it is a small reduction when I tested it on my devserver. However the wall time clearly shows a 5-10% regression, which does not appear to be noise. So Callgrind will only provide starting points. This is a good bootcamp task if you're interested in the fine details of the PyTorch binary.

## To Reproduce

[This script](https://gist.github.com/robieta/2b5d5b4286d29ef5f09870a924cb0f77) will collect instruction counts and wall times; substitute the following to measure this regression:

```
# =============================================================================
# == PR specific code =========================================================
# =============================================================================
COMMIT_0 = "ba77b8d84eefd594f571ddd7fbdfe68ae194bb01"
COMMIT_1 = "efbb854ed8a9df35c0ea896c2d216ab3da10d677"
# Set `COMMIT_1 = "7cd9892f8352c541ebdeea590f6cacb1557e61a5"` to compare with the mitigated version.
make_timer = lambda: Timer("torch::empty({1});", language="cpp")
```

This assumes a conda environment for each commit. [This bash function](https://gist.github.com/robieta/cdfc9137522dd6cc3f80452159e20cb1) can automate the creation of these environments. (However feel free to use a different workflow if you prefer.)

```
40 / 40
<torch.utils.benchmark.utils.common.Measurement object at 0x7fcfd5c0fb50>
torch::empty({1});
  Median: 517.36 ns
  IQR:    58.04 ns (507.71 to 565.76)
  10211 measurements, 1 runs per measurement, 1 thread
  WARNING: Interquartile range is 11.2% of the median measurement.
           This could indicate system fluctuation.

<torch.utils.benchmark.utils.common.Measurement object at 0x7fcfd5c0f990>
torch::empty({1});
  Median: 549.57 ns
  IQR:    157.65 ns (528.54 to 686.18)
  9186 measurements, 1 runs per measurement, 1 thread
  WARNING: Interquartile range is 28.7% of the median measurement.
           This suggests significant environmental influence.

249470
247770

<torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.FunctionCounts object at 0x7fcfd5c00cd0>
     916  /tmp/benchmark_utils_jit_build_dfe9397c083f4bc2bff4d37cdbbe915b/timer_cpp_8606808577451618519/timer_src.cpp:main
     600  ???:0x000000000b9c2ae0
     600  ???:0x000000000b9c23f0
     600  ???:0x000000000b9c2220
     400  build/../aten/src/ATen/core/op_registration/hacky_wrapper_for_legacy_signatures.h:at::empty(c10::ArrayRef<long>, c10::TensorOptions, c10::optional<c10::MemoryFormat>)
     400  build/../aten/src/ATen/core/boxing/KernelFunction_impl.h:at::empty(c10::ArrayRef<long>, c10::TensorOptions, c10::optional<c10::MemoryFormat>)
     300  build/../c10/util/Optional.h:at::empty(c10::ArrayRef<long>, c10::TensorOptions, c10::optional<c10::MemoryFormat>)
     200  lib/python3.7/site-packages/torch/include/c10/util/intrusive_ptr.h:torch::autograd::make_variable(at::Tensor, bool, bool)
     200  build/../c10/core/TensorOptions.h:at::empty(c10::ArrayRef<long>, c10::TensorOptions, c10::optional<c10::MemoryFormat>)
     ...
    -200  /usr/include/c++/8/bits/atomic_base.h:c10::intrusive_ptr<c10::TensorImpl, c10::UndefinedTensorImpl>::reset_()
    -300  build/../c10/core/TensorOptions.h:c10::impl::check_tensor_options_and_extract_memory_format(c10::TensorOptions const&, c10::optional<c10::MemoryFormat>)
    -400  build/aten/src/ATen/Functions.cpp:at::empty(c10::ArrayRef<long>, c10::TensorOptions, c10::optional<c10::MemoryFormat>)
    -400  build/../c10/core/TensorOptions.h:c10::computeDispatchKey(c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>)
    -600  ???:0x000000000bbaeae0
    -600  ???:0x000000000bbae3f0
    -600  ???:0x000000000bbae220
    -916  /tmp/benchmark_utils_jit_build_8d03910179d74801bd0da6fca7c64644/timer_cpp_5970178969148357901/timer_src.cpp:main
   -1900  build/../aten/src/ATen/core/op_registration/hacky_wrapper_for_legacy_signatures.h:c10::impl: ... ptions_and_extract_memory_format(c10::TensorOptions const&, c10::optional<c10::MemoryFormat>)

Total: -1700
```

## Environment

 - PyTorch Version (e.g., 1.0): Nightly. (See commits above.)
 - OS (e.g., Linux): CentOS 8
 - How you installed PyTorch (`conda`, `pip`, source): Built from source
 - Build command you used (if compiling from source): Script provided above
 - Python version: 3.7
 - CUDA/cuDNN version: N/A
 - GPU models and configuration: N/A
 - Any other relevant information: N/A

## Additional context

This is a bootcamp task. If you have any questions, please reach out to @robieta or @orionr 
CC @bitfort, @xuzhao9, @swolchok 


cc @VitalyFedyunin @ngimel