Repository: pytorch
Issue: Reduce number of stack frames used up by dispatcher
Labels: module: internals,module: bootcamp,triaged

Make a debug build of PyTorch. Load up `build/bin/atest` in gdb. Breakpoint on `at::native::add` (don't worry about the symbol not being defined; it's in a dynamic library that will be loaded later). You will break inside the implementation of add kernel on CPU, and you will get a backtrace that looks like this:

```
Breakpoint 1, 0x00007fffeb4bef60 in at::native::add(at::Tensor const&, c10::Scalar, c10::Scalar)@plt ()                                                                                        
   from /data/users/ezyang/pytorch-tmp/build/lib/libtorch_cpu.so                                                                                                                               
(gdb) bt                                                                                                                                                                                       
#0  0x00007fffeb4bef60 in at::native::add(at::Tensor const&, c10::Scalar, c10::Scalar)@plt ()                                                                                                  
   from /data/users/ezyang/pytorch-tmp/build/lib/libtorch_cpu.so                                                                                                                               
#1  0x00007fffec192866 in at::TypeDefault::add_Scalar (self=..., other=..., alpha=...)                                                                                                         
    at aten/src/ATen/TypeDefault.cpp:260                                                                                                                                                       
#2  0x00007fffeb6f65f8 in c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, c10::Scalar, c10::Scalar), at::Tensor, c10::guts::typelist::typelist<at::Tensor 
const&, c10::Scalar, c10::Scalar> >::operator() (                                                                                                                                              
    this=0x97ab10, args#0=..., args#1=..., args#2=...)                                         
    at ../aten/src/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18                                                                                                                   
#3  0x00007fffeb705e73 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, c10::Scalar, c10::Scalar), at::Tensor, c
10::guts::typelist::typelist<at::Tensor const&, c10::Scalar, c10::Scalar> >, at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)>::call(c10::OperatorKernel*, at::Tensor const&, c10::Scal
ar, c10::Scalar) (functor=0x97ab10, args#0=..., args#1=..., args#2=...)                                                                                                                        
    at ../aten/src/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:416                                                                                                                 
#4  0x00007fffec0dc56b in c10::callUnboxedKernelFunction<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar> (                                                                            
    unboxed_kernel_func=0x7fffeb705df9 <c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, c10::Scalar, c10::Scalar),
 at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::Scalar, c10::Scalar> >, at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)>::call(c10::OperatorKernel*, at::Tensor con
st&, c10::Scalar, c10::Scalar)>, functor=0x97ab10)                                             
    at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:43                                                                                                                                   
#5  0x00007fffec069d97 in c10::KernelFunction::call<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar> (                                                                                 
    this=0x5addb0, opHandle=...) at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:53                                                                                                      
#6  0x00007fffebff68ec in c10::Dispatcher::callWithDispatchKey<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, c10::Scalar, c1
0::Scalar)> const&, c10::DispatchKey, at::Tensor const&, c10::Scalar, c10::Scalar) const (this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, op=...,                              
    dispatchKey=c10::DispatchKey::CPU) at ../aten/src/ATen/core/dispatch/Dispatcher.h:370                                                                                                      
#7  0x00007fffebf778cd in c10::Dispatcher::call<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)> con
st&, at::Tensor const&, c10::Scalar, c10::Scalar) const (this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, op=...)                                                               
    at ../aten/src/ATen/core/dispatch/Dispatcher.h:381                                                                                                                                         
#8  0x00007fffebf52784 in c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)>::call(at::Tensor const&, c10::Scalar, c10::Scalar) const (                        
    this=0x7ffff7d6f3e0 <at::add(at::Tensor const&, c10::Scalar, c10::Scalar)::op>, args#0=..., args#1=..., args#2=...)                                                                        
    at ../aten/src/ATen/core/dispatch/Dispatcher.h:323                                                                                                                                         
#9  0x00007fffebf02809 in at::add (self=..., other=..., alpha=...) at aten/src/ATen/Functions.cpp:653                                                                                          
#10 0x00007fffedcc0897 in torch::autograd::VariableType::(anonymous namespace)::<lambda()>::operator()(void) const (                                                                           
    __closure=0x7fffffffbde0) at ../torch/csrc/autograd/generated/VariableType_2.cpp:831                                                                                                       #11 0x00007fffedcc0ad1 in torch::autograd::VariableType::(anonymous namespace)::add_Scalar (self=..., other=...,                                                                               
    alpha=...) at ../torch/csrc/autograd/generated/VariableType_2.cpp:832                                                                                                                      
#12 0x00007fffeb6f65f8 in c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, c10::Scalar, c10::Scalar), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::Scalar, c10::Scalar> >::operator() (                                                                                                                                                  this=0x1090bb0, args#0=..., args#1=..., args#2=...)                                                                                                                                        
    at ../aten/src/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18                                                                                                                   
#13 0x00007fffeb705e73 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, c10::Scalar, c10::Scalar), at::Tensor, c
10::guts::typelist::typelist<at::Tensor const&, c10::Scalar, c10::Scalar> >, at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)>::call(c10::OperatorKernel*, at::Tensor const&, c10::Scalar, c10::Scalar) (functor=0x1090bb0, args#0=..., args#1=..., args#2=...)                                                                                                                           at ../aten/src/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:416                                                                                                                 
--Type <RET> for more, q to quit, c to continue without paging--                                                                                                                               
#14 0x00007fffec0dc56b in c10::callUnboxedKernelFunction<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar> ( 
    unboxed_kernel_func=0x7fffeb705df9 <c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, c10::Scalar, c10::Scalar),
 at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::Scalar, c10::Scalar> >, at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)>::call(c10::OperatorKernel*, at::Tensor con
st&, c10::Scalar, c10::Scalar)>, functor=0x1090bb0) at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:43
#15 0x00007fffec069d97 in c10::KernelFunction::call<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar> (this=0x5ae0f0, opHandle=...)
    at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:53
#16 0x00007fffebff68ec in c10::Dispatcher::callWithDispatchKey<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, c10::Scalar, c1
0::Scalar)> const&, c10::DispatchKey, at::Tensor const&, c10::Scalar, c10::Scalar) const (this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, op=..., 
    dispatchKey=c10::DispatchKey::Autograd) at ../aten/src/ATen/core/dispatch/Dispatcher.h:370
#17 0x00007fffebf778cd in c10::Dispatcher::call<at::Tensor, at::Tensor const&, c10::Scalar, c10::Scalar>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)> con
st&, at::Tensor const&, c10::Scalar, c10::Scalar) const (this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, op=...) at ../aten/src/ATen/core/dispatch/Dispatcher.h:381
#18 0x00007fffebf52784 in c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, c10::Scalar, c10::Scalar)>::call(at::Tensor const&, c10::Scalar, c10::Scalar) const (
    this=0x7ffff7d747a0 <at::Tensor::add(c10::Scalar, c10::Scalar) const::op>, args#0=..., args#1=..., args#2=...) at ../aten/src/ATen/core/dispatch/Dispatcher.h:323
#19 0x00007fffec26c831 in at::Tensor::add (this=0x7fffffffc810, other=..., alpha=...) at aten/src/ATen/core/TensorMethods.cpp:330
#20 0x00007fffebb3aee0 in at::operator+ (x=..., y=...) at ../aten/src/ATen/TensorOperators.h:110
#21 0x00007fffebb3ca9d in at::native::isclose (self=..., other=..., rtol=1.0000000000000001e-05, atol=1e-08, equal_nan=false) at ../aten/src/ATen/native/TensorCompare.cpp:71
#22 0x00007fffec19acf1 in at::TypeDefault::isclose (self=..., other=..., rtol=1.0000000000000001e-05, atol=1e-08, equal_nan=false) at aten/src/ATen/TypeDefault.cpp:1208
#23 0x00007fffec21a503 in c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, at::Tensor const&, double, double, bool), at::Tensor, c10::guts::typelist::typel
ist<at::Tensor const&, at::Tensor const&, double, double, bool> >::operator() (this=0xa7af90, args#0=..., args#1=..., args#2=1.0000000000000001e-05, args#3=1e-08, args#4=false)
    at ../aten/src/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
#24 0x00007fffec243666 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, at::Tensor const&, double, double, bool)
, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, double, double, bool> >, at::Tensor (at::Tensor const&, at::Tensor const&, double, double, bool)>::call(c10::
OperatorKernel*, at::Tensor const&, at::Tensor const&, double, double, bool) (functor=0xa7af90, args#0=..., args#1=..., args#2=1.0000000000000001e-05, args#3=1e-08, args#4=false)
    at ../aten/src/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:416
#25 0x00007fffec0fa67d in c10::callUnboxedKernelFunction<at::Tensor, at::Tensor const&, at::Tensor const&, double, double, bool> (
    unboxed_kernel_func=0x7fffec2435b9 <c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, at::Tensor const&, double,
 double, bool), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, double, double, bool> >, at::Tensor (at::Tensor const&, at::Tensor const&, double, double, bool
)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&, double, double, bool)>, functor=0xa7af90) at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:43
#26 0x00007fffec0875ef in c10::KernelFunction::call<at::Tensor, at::Tensor const&, at::Tensor const&, double, double, bool> (this=0xa7c240, opHandle=...)
    at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:53
#27 0x00007fffec01421f in c10::Dispatcher::callWithDispatchKey<at::Tensor, at::Tensor const&, at::Tensor const&, double, double, bool>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, 
at::Tensor const&, double, double, bool)> const&, c10::DispatchKey, at::Tensor const&, at::Tensor const&, double, double, bool) const (
    this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, op=..., dispatchKey=c10::DispatchKey::CPU) at ../aten/src/ATen/core/dispatch/Dispatcher.h:370
#28 0x00007fffebf988fb in c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&, double, double, bool>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor cons
t&, double, double, bool)> const&, at::Tensor const&, at::Tensor const&, double, double, bool) const (this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, op=...)
    at ../aten/src/ATen/core/dispatch/Dispatcher.h:381
#29 0x00007fffebf5af20 in c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&, double, double, bool)>::call(at::Tensor const&, at::Tensor const&, double, double, bool) c
onst (this=0x7ffff7d705e0 <at::isclose(at::Tensor const&, at::Tensor const&, double, double, bool)::op>, args#0=..., args#1=..., args#2=1.0000000000000001e-05, args#3=1e-08, args#4=false)
    at ../aten/src/ATen/core/dispatch/Dispatcher.h:323
#30 0x00007fffebf12f2e in at::isclose (self=..., other=..., rtol=1.0000000000000001e-05, atol=1e-08, equal_nan=false) at aten/src/ATen/Functions.cpp:3821
#31 0x00007fffebb3c2a8 in at::native::allclose (self=..., other=..., rtol=1.0000000000000001e-05, atol=1e-08, equal_nan=false) at ../aten/src/ATen/native/TensorCompare.cpp:20
#32 0x00007fffec1930d6 in at::TypeDefault::allclose (self=..., other=..., rtol=1.0000000000000001e-05, atol=1e-08, equal_nan=false) at aten/src/ATen/TypeDefault.cpp:316
#33 0x00007fffec211c63 in c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<bool (*)(at::Tensor const&, at::Tensor const&, double, double, bool), bool, c10::guts::typelist::typelist<at::Tens
or const&, at::Tensor const&, double, double, bool> >::operator() (this=0x989090, args#0=..., args#1=..., args#2=1.0000000000000001e-05, args#3=1e-08, args#4=false)
    at ../aten/src/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
#34 0x00007fffec23cba2 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<bool (*)(at::Tensor const&, at::Tensor const&, double, double, bool), bool
, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, double, double, bool> >, bool (at::Tensor const&, at::Tensor const&, double, double, bool)>::call(c10::OperatorKernel*, a
t::Tensor const&, at::Tensor const&, double, double, bool) (functor=0x989090, args#0=..., args#1=..., args#2=1.0000000000000001e-05, args#3=1e-08, args#4=false)
    at ../aten/src/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:416
#35 0x00007fffec0de0f6 in c10::callUnboxedKernelFunction<bool, at::Tensor const&, at::Tensor const&, double, double, bool> (
    unboxed_kernel_func=0x7fffec23caff <c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<bool (*)(at::Tensor const&, at::Tensor const&, double, doubl
e, bool), bool, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, double, double, bool> >, bool (at::Tensor const&, at::Tensor const&, double, double, bool)>::call(c10::Oper
atorKernel*, at::Tensor const&, at::Tensor const&, double, double, bool)>, functor=0x989090) at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:43
#36 0x00007fffec06bd82 in c10::KernelFunction::call<bool, at::Tensor const&, at::Tensor const&, double, double, bool> (this=0x989f20, opHandle=...)
    at ../aten/src/ATen/core/boxing/KernelFunction_impl.h:53
#37 0x00007fffebff8a7b in c10::Dispatcher::callWithDispatchKey<bool, at::Tensor const&, at::Tensor const&, double, double, bool>(c10::TypedOperatorHandle<bool (at::Tensor const&, at::Tensor c
onst&, double, double, bool)> const&, c10::DispatchKey, at::Tensor const&, at::Tensor const&, double, double, bool) const (this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, 
    op=..., dispatchKey=c10::DispatchKey::CPU) at ../aten/src/ATen/core/dispatch/Dispatcher.h:370
#38 0x00007fffebf79e2b in c10::Dispatcher::call<bool, at::Tensor const&, at::Tensor const&, double, double, bool>(c10::TypedOperatorHandle<bool (at::Tensor const&, at::Tensor const&, double, 
double, bool)> const&, at::Tensor const&, at::Tensor const&, double, double, bool) const (this=0x7ffff7d68ae0 <c10::Dispatcher::singleton()::_singleton>, op=...)
    at ../aten/src/ATen/core/dispatch/Dispatcher.h:381
#39 0x00007fffebf52f65 in c10::TypedOperatorHandle<bool (at::Tensor const&, at::Tensor const&, double, double, bool)>::call(at::Tensor const&, at::Tensor const&, double, double, bool) const
    (this=0x7ffff7d74820 <at::Tensor::allclose(at::Tensor const&, double, double, bool) const::op>, args#0=..., args#1=..., args#2=1.0000000000000001e-05, args#3=1e-08, args#4=false)
    at ../aten/src/ATen/core/dispatch/Dispatcher.h:323
#40 0x00007fffec26cf36 in at::Tensor::allclose (this=0x7fffffffd5c0, other=..., rtol=1.0000000000000001e-05, atol=1e-08, equal_nan=false) at aten/src/ATen/core/TensorMethods.cpp:418
#41 0x000000000045f80e in unit_binary_ops_test<at::Tensor& (*)(at::Tensor&, at::Tensor const&, at::Tensor const&)> (
    func=0x7fffebf381e0 <at::max_out(at::Tensor&, at::Tensor const&, at::Tensor const&)>, x_tensor=..., y_tensor=..., exp=..., dtype=c10::ScalarType::Float)
    at ../aten/src/ATen/test/atest.cpp:48
#42 0x000000000045d03d in run_binary_ops_test<at::Tensor& (*)(at::Tensor&, at::Tensor const&, at::Tensor const&)> (
    func=0x7fffebf381e0 <at::max_out(at::Tensor&, at::Tensor const&, at::Tensor const&)>, x_tensor=..., y_tensor=..., exp=..., option=7) at ../aten/src/ATen/test/atest.cpp:81
#43 0x0000000000450a3b in atest_max_operators_Test::TestBody (this=0x14a5160) at ../aten/src/ATen/test/atest.cpp:185
#44 0x0000000000490f0b in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x14a5160, method=&virtual testing::Test::TestBody(), 
    location=0x4a446b "the test body") at ../third_party/googletest/googletest/src/gtest.cc:2443
#45 0x000000000048b7b5 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x14a5160, method=&virtual testing::Test::TestBody(), 
    location=0x4a446b "the test body") at ../third_party/googletest/googletest/src/gtest.cc:2479
#46 0x000000000046ca66 in testing::Test::Run (this=0x14a5160) at ../third_party/googletest/googletest/src/gtest.cc:2517
#47 0x000000000046d2e9 in testing::TestInfo::Run (this=0x14ab890) at ../third_party/googletest/googletest/src/gtest.cc:2693
#48 0x000000000046d909 in testing::TestCase::Run (this=0x14abe50) at ../third_party/googletest/googletest/src/gtest.cc:2811
#49 0x0000000000477f25 in testing::internal::UnitTestImpl::RunAllTests (this=0xe23510) at ../third_party/googletest/googletest/src/gtest.cc:5177
#50 0x0000000000491ecc in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0xe23510, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x477c94 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x4a4e20 "auxiliary test code (environments or event listeners)") at ../third_party/googletest/googletest/src/gtest.cc:2443
#51 0x000000000048c57f in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0xe23510, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x477c94 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x4a4e20 "auxiliary test code (environments or event listeners)") at ../third_party/googletest/googletest/src/gtest.cc:2479
#52 0x0000000000476b10 in testing::UnitTest::Run (this=0x4d0be0 <testing::UnitTest::GetInstance()::instance>) at ../third_party/googletest/googletest/src/gtest.cc:4786
#53 0x00000000004658ef in RUN_ALL_TESTS () at ../third_party/googletest/googletest/include/gtest/gtest.h:2341
#54 0x0000000000465888 in main (argc=1, argv=0x7fffffffdbb8) at ../third_party/googletest/googletest/src/gtest_main.cc:36
```

Monstrous. Frame #41 is the actual call into ATen, and then it takes almost ten stack frames to do every redispatch into the dispatcher. (If all of this is greek to you, go read http://blog.ezyang.com/2019/05/pytorch-internals/ )

It would make our stack traces more interpretable if there was a way to reduce the number of stack frames each call through the dispatcher makes. Unfortunately, there doesn't seem to be a way to directly force compilers to omit frames for inline function on -O0, so that means we are forced to inline functions. Inlining functions can be detrimental to DRY (don't repeat yourself) and code readability, but in some cases, we have relatively cost free opportunity to improve the situation.

Task:
* Reproduce this stack trace on your own
* Find a spot in the dispatcher chain where you can inline a function (without causing code duplication) to remove one stack frame
* Look and see if there are any other opportunities to reduce the number of stack frames without having to write a lot of code

I intend to bootcamp this task

cc @ezyang @bhosmer @smessmer @ljk53