Repository: pytorch
Issue: Custom backend not called for compiling backward graph
Labels: feature,triaged,module: backend,oncall: pt2

### 🐛 Describe the bug

Given the code below, the custom backend is only invoked during forward graph compilation and not backwards. "MyBackend called" is print only once (logs below)

```
import logging
import torch

class MyBackend(torch._TorchCompileInductorWrapper):

    def __init__(self, mode):
        super().__init__(mode, None, False)

    def __call__(self, model_, inputs_):
        logging.info("MyBackend called")

        return super().__call__(model_, inputs_)

if __name__ == "__main__":
  logging.basicConfig(format='%(asctime)s %(filename)s %(levelname)s:%(message)s', level=logging.DEBUG)

  model = torch.compile(torch.nn.Linear(10, 10).cuda(), backend=MyBackend("default"))
  inp = torch.rand(10, 10).cuda()
  model(inp).sum().backward()
````

### Error logs

[2023-11-20 15:52:36,934] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing inner torch/_dynamo/external_utils.py:15
[2023-11-20 15:52:36,934] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2023-11-20 15:52:36,938] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line torch/_dynamo/external_utils.py:15 in inner (wrap_inline)
[2023-11-20 15:52:36,938] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         @functools.wraps(fn)
[2023-11-20 15:52:36,938] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line torch/_dynamo/external_utils.py:17 in inner (wrap_inline.inner)
[2023-11-20 15:52:36,938] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             return fn(*args, **kwargs)
[2023-11-20 15:52:36,939] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF fn []
[2023-11-20 15:52:36,939] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [LazyVariableTracker()]
[2023-11-20 15:52:36,939] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [LazyVariableTracker(), LazyVariableTracker()]
[2023-11-20 15:52:36,939] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [LazyVariableTracker(), LazyVariableTracker(), ConstDictVariable()]
[2023-11-20 15:52:36,939] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [LazyVariableTracker(), LazyVariableTracker(), ConstDictVariable(), LazyVariableTracker()]
[2023-11-20 15:52:38,024] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [LazyVariableTracker(), LazyVariableTracker(), ConstDictVariable()]
[2023-11-20 15:52:38,025] [0/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_args_0_ L['args'][0]
[2023-11-20 15:52:38,026] [0/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['args'][0] (10, 10) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-11-20 15:52:38,033] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-11-20 15:52:38,033] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing inner (RETURN_VALUE)
[2023-11-20 15:52:38,033] [0/0] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-11-20 15:52:38,033] [0/0] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file torch/_dynamo/external_utils.py, line 17 in inner>], graph_break=False)
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]  ===== __compiled_fn_0 =====
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]  <eval_with_key>.0 class GraphModule(torch.nn.Module):
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]     def forward(self, L_args_0_ : torch.Tensor):
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]         l_args_0_ = L_args_0_
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]         
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]         # File: torch/_dynamo/external_utils.py:17, code: return fn(*args, **kwargs)
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]         fn = self.fn(l_args_0_);  l_args_0_ = None
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]         return (fn,)
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG]         
[2023-11-20 15:52:38,034] [0/0] torch._dynamo.output_graph.__graph_code: [DEBUG] 
[2023-11-20 15:52:38,035] [0/0] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
[2023-11-20 15:52:38,035] [0/0] torch._dynamo.output_graph.__graph: [DEBUG]  __compiled_fn_0 <eval_with_key>.0 opcode       name       target     args          kwargs
[2023-11-20 15:52:38,035] [0/0] torch._dynamo.output_graph.__graph: [DEBUG] -----------  ---------  ---------  ------------  --------
[2023-11-20 15:52:38,035] [0/0] torch._dynamo.output_graph.__graph: [DEBUG] placeholder  l_args_0_  L_args_0_  ()            {}
[2023-11-20 15:52:38,035] [0/0] torch._dynamo.output_graph.__graph: [DEBUG] call_module  fn         fn         (l_args_0_,)  {}
[2023-11-20 15:52:38,035] [0/0] torch._dynamo.output_graph.__graph: [DEBUG] output       output     output     ((fn,),)      {}
[2023-11-20 15:52:38,035] [0/0] torch._dynamo.output_graph.__graph: [DEBUG] 
[2023-11-20 15:52:38,039] [0/0] torch._dynamo.output_graph.__graph_sizes: [DEBUG] TRACED GRAPH TENSOR SIZES
[2023-11-20 15:52:38,039] [0/0] torch._dynamo.output_graph.__graph_sizes: [DEBUG] ===== __compiled_fn_0 =====
[2023-11-20 15:52:38,039] [0/0] torch._dynamo.output_graph.__graph_sizes: [DEBUG] l_args_0_: (10, 10)
[2023-11-20 15:52:38,039] [0/0] torch._dynamo.output_graph.__graph_sizes: [DEBUG] fn: (10, 10)
[2023-11-20 15:52:38,039] [0/0] torch._dynamo.output_graph.__graph_sizes: [DEBUG] 
[2023-11-20 15:52:38,039] [0/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function <__main__.MyBackend object at 0x155554a2d990>
2023-11-20 15:52:38,040 testcompile.py INFO:MyBackend called
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO] TRACED GRAPH
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]  ===== Joint graph 0 =====
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]  <eval_with_key>.4 from torch/fx/experimental/proxy_tensor.py:509 in wrapped class joint_helper(torch.nn.Module):
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]     def forward(self, primals, tangents):
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         primals_1: "f32[10, 10]"; primals_2: "f32[10]"; primals_3: "f32[10, 10]"; tangents_1: "f32[10, 10]"; 
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]     
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         primals_1, primals_2, primals_3, tangents_1, = fx_pytree.tree_flatten_spec([primals, tangents], self._in_spec)
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         # File: torch/_dynamo/external_utils.py:17, code: return fn(*args, **kwargs)
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         permute: "f32[10, 10]" = torch.ops.aten.permute.default(primals_1, [1, 0]);  primals_1 = None
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         addmm: "f32[10, 10]" = torch.ops.aten.addmm.default(primals_2, primals_3, permute);  primals_2 = permute = None
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         permute_1: "f32[10, 10]" = torch.ops.aten.permute.default(tangents_1, [1, 0])
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         mm: "f32[10, 10]" = torch.ops.aten.mm.default(permute_1, primals_3);  permute_1 = primals_3 = None
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         permute_2: "f32[10, 10]" = torch.ops.aten.permute.default(mm, [1, 0]);  mm = None
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         sum_1: "f32[1, 10]" = torch.ops.aten.sum.dim_IntList(tangents_1, [0], True);  tangents_1 = None
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         view: "f32[10]" = torch.ops.aten.view.default(sum_1, [10]);  sum_1 = None
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         permute_3: "f32[10, 10]" = torch.ops.aten.permute.default(permute_2, [1, 0]);  permute_2 = None
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         return pytree.tree_unflatten([addmm, permute_3, view, None], self._out_spec)
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO]         
[2023-11-20 15:52:38,797] [0/0] torch._functorch.aot_autograd.__aot_joint_graph: [INFO] 
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]  ===== Forward graph 0 =====
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]  <eval_with_key>.40 class GraphModule(torch.nn.Module):
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]     def forward(self, primals_1: "f32[10, 10]", primals_2: "f32[10]", primals_3: "f32[10, 10]"):
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         # File: torch/_dynamo/external_utils.py:17, code: return fn(*args, **kwargs)
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         permute: "f32[10, 10]" = torch.ops.aten.permute.default(primals_1, [1, 0]);  primals_1 = None
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         addmm: "f32[10, 10]" = torch.ops.aten.addmm.default(primals_2, primals_3, permute);  primals_2 = permute = None
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         return [addmm, primals_3]
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO] 
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]  ===== Backward graph 0 =====
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]  <eval_with_key>.41 class GraphModule(torch.nn.Module):
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]     def forward(self, primals_3: "f32[10, 10]", tangents_1: "f32[10, 10]"):
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         # File: torch/_dynamo/external_utils.py:17, code: return fn(*args, **kwargs)
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         permute_1: "f32[10, 10]" = torch.ops.aten.permute.default(tangents_1, [1, 0])
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         mm: "f32[10, 10]" = torch.ops.aten.mm.default(permute_1, primals_3);  permute_1 = primals_3 = None
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         permute_2: "f32[10, 10]" = torch.ops.aten.permute.default(mm, [1, 0]);  mm = None
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         sum_1: "f32[1, 10]" = torch.ops.aten.sum.dim_IntList(tangents_1, [0], True);  tangents_1 = None
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         view: "f32[10]" = torch.ops.aten.view.default(sum_1, [10]);  sum_1 = None
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         permute_3: "f32[10, 10]" = torch.ops.aten.permute.default(permute_2, [1, 0]);  permute_2 = None
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         return [permute_3, view, None]
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO]         
[2023-11-20 15:52:39,277] [0/0] torch._functorch.aot_autograd.__aot_graphs: [INFO] 
torch/_inductor/compile_fx.py:140: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
[2023-11-20 15:52:39,280] [0/0] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO] TRACED GRAPH
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]  ===== AFTER POST GRAD =====
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]  <eval_with_key>.42 class GraphModule(torch.nn.Module):
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]     def forward(self, primals_1: "f32[10, 10]", primals_2: "f32[10]", primals_3: "f32[10, 10]"):
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         # File: torch/_dynamo/external_utils.py:17, code: return fn(*args, **kwargs)
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         permute: "f32[10, 10]" = torch.ops.aten.permute.default(primals_1, [1, 0]);  primals_1 = None
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         addmm: "f32[10, 10]" = torch.ops.aten.addmm.default(primals_2, primals_3, permute);  primals_2 = permute = None
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         return [addmm, primals_3]
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         
[2023-11-20 15:52:39,304] [0/0] torch._inductor.compile_fx.__post_grad_graphs: [INFO] 
[2023-11-20 15:52:39,307] [0/0] torch._inductor.graph: [DEBUG] lowering %primals_1 : [num_users=1] = placeholder[target=primals_1] 
[2023-11-20 15:52:39,334] [0/0] torch._inductor.graph: [DEBUG] lowering %primals_2 : [num_users=1] = placeholder[target=primals_2] 
[2023-11-20 15:52:39,334] [0/0] torch._inductor.graph: [DEBUG] lowering %primals_3 : [num_users=2] = placeholder[target=primals_3] 
[2023-11-20 15:52:39,335] [0/0] torch._inductor.graph: [DEBUG] lowering %permute : [num_users=1] = call_function[target=torch.ops.aten.permute.default](args = (%primals_1, [1, 0]), kwargs = {}) 
[2023-11-20 15:52:39,335] [0/0] torch._inductor.graph: [DEBUG]   via <function permute at 0x1554948b1d80>
[2023-11-20 15:52:39,335] [0/0] torch._inductor.graph: [DEBUG] lowering %addmm : [num_users=1] = call_function[target=torch.ops.aten.addmm.default](args = (%primals_2, %primals_3, %permute), kwargs = {}) 
[2023-11-20 15:52:39,335] [0/0] torch._inductor.graph: [DEBUG]   via <function tuned_addmm at 0x1554943452d0>
[2023-11-20 15:52:39,336] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] eval True == True [statically known]
[2023-11-20 15:52:39,337] [0/0] torch._inductor.select_algorithm: [INFO] Max autotune selects from 1 choices.
[2023-11-20 15:52:39,337] [0/0] torch._inductor.graph: [DEBUG] lowering return [addmm, primals_3] 
[2023-11-20 15:52:39,337] [0/0] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 0
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG] scheduling ExternKernelOut(
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   kernel='extern_kernels.addmm',
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   name=buf0,
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   layout=FixedLayout('cuda', torch.float32, size=[10, 10], stride=[10, 1]),
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   inputs=[InputBuffer(name='primals_2', layout=FixedLayout('cuda', torch.float32, size=[10], stride=[1])), InputBuffer(name='primals_3', layout=FixedLayout('cuda', torch.float32, size=[10, 10], stride=[10, 1])), ReinterpretView(
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]     StorageBox(
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]       InputBuffer(name='primals_1', layout=FixedLayout('cuda', torch.float32, size=[10, 10], stride=[10, 1]))
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]     ),
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]     FixedLayout('cuda', torch.float32, size=[10, 10], stride=[1, 10]),
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]     origins={permute}
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   )],
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   constant_args=(),
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   kwargs={'alpha': 1, 'beta': 1},
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   output_view=None,
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   ordered_kwargs_for_cpp_kernel=('beta', 'alpha'),
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   origin_node=addmm,
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG]   origins={addmm}
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG] )
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG] scheduling output buf0
[2023-11-20 15:52:39,338] [0/0] torch._inductor.scheduler: [DEBUG] scheduling output primals_3
[2023-11-20 15:52:39,339] [0/0] torch._inductor.debug: [INFO] Writing debug ir to  torch_compile_debug/run_2023_11_20_15_52_36_931032-pid_1060761/torchinductor/model__0_forward_10.0/ir_pre_fusion.txt
[2023-11-20 15:52:39,341] [0/0] torch._inductor.debug: [INFO] Writing debug ir to  torch_compile_debug/run_2023_11_20_15_52_36_931032-pid_1060761/torchinductor/model__0_forward_10.0/ir_post_fusion.txt
[2023-11-20 15:52:39,348] [0/0] torch._inductor.scheduler: [DEBUG] Generating code for node buf0 with estimated runtime 0.608129
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph: [DEBUG] Output code written to: /tmp/6t/c6tvzoej4bmmjsidh7d54c3ih63z52blkodhj42z3usf67nt3c26.py
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] Output code: 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from ctypes import c_void_p, c_long
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] import torch
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] import math
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] import random
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] import os
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] import tempfile
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from math import inf, nan
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.hooks import run_intermediate_hooks
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.utils import maybe_profile
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.codegen.memory_planning import _align as align
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from torch import device, empty, empty_strided
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.codecache import AsyncCompile
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.select_algorithm import extern_kernels
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] aten = torch.ops.aten
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] inductor_ops = torch.ops.inductor
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] assert_size_stride = torch._C._dynamo.guards.assert_size_stride
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] alloc_from_pool = torch.ops.inductor._alloc_from_pool
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] reinterpret_tensor = torch.ops.inductor._reinterpret_tensor
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] async_compile = AsyncCompile()
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] async_compile.wait(globals())
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] del async_compile
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] def call(args):
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     primals_1, primals_2, primals_3 = args
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     args.clear()
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     assert_size_stride(primals_1, (10, 10), (10, 1))
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     assert_size_stride(primals_2, (10, ), (1, ))
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     assert_size_stride(primals_3, (10, 10), (10, 1))
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     with torch.cuda._DeviceGuard(0):
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]         torch.cuda.set_device(0) # no-op to ensure context
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]         buf0 = empty((10, 10), device='cuda', dtype=torch.float32)
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]         # Source Nodes: [fn], Original ATen: [aten.addmm]
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]         extern_kernels.addmm(primals_2, primals_3, reinterpret_tensor(primals_1, (10, 10), (1, 10), 0), alpha=1, beta=1, out=buf0)
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]         del primals_1
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]         del primals_2
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]         return (buf0, primals_3, )
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] def benchmark_compiled_module(times=10, repeat=10):
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     from torch._dynamo.testing import rand_strided
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     from torch._inductor.utils import print_performance
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     primals_1 = rand_strided((10, 10), (10, 1), device='cuda:0', dtype=torch.float32)
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     primals_2 = rand_strided((10, ), (1, ), device='cuda:0', dtype=torch.float32)
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     primals_3 = rand_strided((10, 10), (10, 1), device='cuda:0', dtype=torch.float32)
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     return print_performance(lambda: call([primals_1, primals_2, primals_3]), times=times, repeat=repeat)
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] if __name__ == "__main__":
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     from torch._inductor.wrapper_benchmark import compiled_module_main
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG]     compiled_module_main('None', benchmark_compiled_module)
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,354] [0/0] torch._inductor.graph.__output_code: [INFO] Output code written to: /tmp/6t/c6tvzoej4bmmjsidh7d54c3ih63z52blkodhj42z3usf67nt3c26.py
[2023-11-20 15:52:39,360] [0/0] torch._inductor.compile_fx: [DEBUG] FX codegen and compilation took 0.080s
[2023-11-20 15:52:39,360] [0/0] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 0
[2023-11-20 15:52:39,360] [0/0] torch._inductor.debug: [WARNING] model__0_forward_10 debug trace: /tmp/6t/c6tvzoej4bmmjsidh7d54c3ih63z52blkodhj42z3usf67nt3c26.debug
[2023-11-20 15:52:39,362] [0/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function <__main__.MyBackend object at 0x155554a2d990>
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] track_symint L['args'][0].size()[0] 10 None
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] track_symint L['args'][0].size()[1] 10 None
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] track_symint L['args'][0].stride()[0] 10 None
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] track_symint L['args'][0].stride()[1] 1 None
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] track_symint L['args'][0].storage_offset() 0 None
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] Skipping guard L['args'][0].size()[0] == 10
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] Skipping guard L['args'][0].size()[1] == 10
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] Skipping guard L['args'][0].stride()[0] == 10
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] Skipping guard L['args'][0].stride()[1] == 1
[2023-11-20 15:52:39,365] [0/0] torch.fx.experimental.symbolic_shapes: [DEBUG] Skipping guard L['args'][0].storage_offset() == 0
[2023-11-20 15:52:39,365] [0/0] torch._dynamo.guards.__guards: [DEBUG] GUARDS:
[2023-11-20 15:52:39,365] [0/0] torch._dynamo.guards.__guards: [DEBUG] ___check_obj_id(L['fn'], 23456236359680)                    
[2023-11-20 15:52:39,367] [0/0] torch._dynamo.guards.__guards: [DEBUG] L['fn'].training == True                                    
[2023-11-20 15:52:39,367] [0/0] torch._dynamo.guards.__guards: [DEBUG] ___check_type_id(L['args'], 93824995909056)                 
[2023-11-20 15:52:39,367] [0/0] torch._dynamo.guards.__guards: [DEBUG] len(L['args']) == 1                                         
[2023-11-20 15:52:39,367] [0/0] torch._dynamo.guards.__guards: [DEBUG] ___check_type_id(L['kwargs'], 93824995929696)               
[2023-11-20 15:52:39,367] [0/0] torch._dynamo.guards.__guards: [DEBUG] set(L['kwargs'].keys()) == set()                            
[2023-11-20 15:52:39,368] [0/0] torch._dynamo.guards.__guards: [DEBUG] hasattr(L['args'][0], '_dynamo_dynamic_indices') == False   
[2023-11-20 15:52:39,368] [0/0] torch._dynamo.guards.__guards: [DEBUG] utils_device.CURRENT_DEVICE == None                           # _dynamo/output_graph.py:365 in init_ambient_guards
[2023-11-20 15:52:39,368] [0/0] torch._dynamo.guards.__guards: [DEBUG] (___skip_backend_check() or ___current_backend() == ___lookup_backend(23453531104704))  # _dynamo/output_graph.py:371 in init_ambient_guards
[2023-11-20 15:52:39,368] [0/0] torch._dynamo.guards.__guards: [DEBUG] ___compile_config_hash() == '9275ab63dc5d0539a089c0d4cec725df'  # _dynamo/output_graph.py:373 in init_ambient_guards
[2023-11-20 15:52:39,368] [0/0] torch._dynamo.guards.__guards: [DEBUG] check_tensor(L['args'][0], Tensor, DispatchKeySet(CUDA, BackendSelect, ADInplaceOrView, AutogradCUDA), torch.float32, device=0, requires_grad=False, size=[10, 10], stride=[10, 1])
[2023-11-20 15:52:39,386] torch._dynamo.eval_frame: [DEBUG] Unsetting top-level compile config hash: 9275ab63dc5d0539a089c0d4cec725df
[2023-11-20 15:52:39,404] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling BACKWARDS graph 0
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO] TRACED GRAPH
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]  ===== AFTER POST GRAD =====
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]  <eval_with_key>.44 class GraphModule(torch.nn.Module):
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]     def forward(self, primals_3: "f32[10, 10]", tangents_1: "f32[10, 10]"):
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         # File: torch/_dynamo/external_utils.py:17, code: return fn(*args, **kwargs)
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         permute_1: "f32[10, 10]" = torch.ops.aten.permute.default(tangents_1, [1, 0])
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         mm: "f32[10, 10]" = torch.ops.aten.mm.default(permute_1, primals_3);  permute_1 = primals_3 = None
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         permute_2: "f32[10, 10]" = torch.ops.aten.permute.default(mm, [1, 0]);  mm = None
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         sum_1: "f32[1, 10]" = torch.ops.aten.sum.dim_IntList(tangents_1, [0], True);  tangents_1 = None
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         view: "f32[10]" = torch.ops.aten.reshape.default(sum_1, [10]);  sum_1 = None
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         permute_3: "f32[10, 10]" = torch.ops.aten.permute.default(permute_2, [1, 0]);  permute_2 = None
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         return [permute_3, view, None]
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO]         
[2023-11-20 15:52:39,416] torch._inductor.compile_fx.__post_grad_graphs: [INFO] 
[2023-11-20 15:52:39,417] torch._inductor.graph: [DEBUG] lowering %primals_3 : [num_users=1] = placeholder[target=primals_3] 
[2023-11-20 15:52:39,418] torch._inductor.graph: [DEBUG] lowering %tangents_1 : [num_users=2] = placeholder[target=tangents_1] 
[2023-11-20 15:52:39,418] torch._inductor.graph: [DEBUG] lowering %permute_1 : [num_users=1] = call_function[target=torch.ops.aten.permute.default](args = (%tangents_1, [1, 0]), kwargs = {}) 
[2023-11-20 15:52:39,418] torch._inductor.graph: [DEBUG]   via <function permute at 0x1554948b1d80>
[2023-11-20 15:52:39,419] torch._inductor.graph: [DEBUG] lowering %mm : [num_users=1] = call_function[target=torch.ops.aten.mm.default](args = (%permute_1, %primals_3), kwargs = {}) 
[2023-11-20 15:52:39,419] torch._inductor.graph: [DEBUG]   via <function tuned_mm at 0x155494345090>
[2023-11-20 15:52:39,419] torch._inductor.select_algorithm: [INFO] Max autotune selects from 1 choices.
[2023-11-20 15:52:39,419] torch._inductor.graph: [DEBUG] lowering %permute_2 : [num_users=1] = call_function[target=torch.ops.aten.permute.default](args = (%mm, [1, 0]), kwargs = {}) 
[2023-11-20 15:52:39,419] torch._inductor.graph: [DEBUG]   via <function permute at 0x1554948b1d80>
[2023-11-20 15:52:39,420] torch._inductor.graph: [DEBUG] lowering %sum_1 : [num_users=1] = call_function[target=torch.ops.aten.sum.dim_IntList](args = (%tangents_1, [0], True), kwargs = {}) 
[2023-11-20 15:52:39,420] torch._inductor.graph: [DEBUG]   via <function sum_ at 0x155494785900>
[2023-11-20 15:52:39,420] torch._inductor.graph: [DEBUG] lowering %view : [num_users=1] = call_function[target=torch.ops.aten.reshape.default](args = (%sum_1, [10]), kwargs = {}) 
[2023-11-20 15:52:39,420] torch._inductor.graph: [DEBUG]   via <function view at 0x1554948b1b40>
[2023-11-20 15:52:39,421] torch._inductor.graph: [DEBUG] lowering %permute_3 : [num_users=1] = call_function[target=torch.ops.aten.permute.default](args = (%permute_2, [1, 0]), kwargs = {}) 
[2023-11-20 15:52:39,421] torch._inductor.graph: [DEBUG]   via <function permute at 0x1554948b1d80>
[2023-11-20 15:52:39,421] torch._inductor.graph: [DEBUG] lowering return [permute_3, view, None] 
[2023-11-20 15:52:39,421] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 0
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG] scheduling ExternKernelOut(
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   kernel='extern_kernels.mm',
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   name=buf0,
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   layout=FixedLayout('cuda', torch.float32, size=[10, 10], stride=[10, 1]),
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   inputs=[ReinterpretView(
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]     StorageBox(
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]       InputBuffer(name='tangents_1', layout=FixedLayout('cuda', torch.float32, size=[10, 10], stride=[10, 1]))
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]     ),
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]     FixedLayout('cuda', torch.float32, size=[10, 10], stride=[1, 10]),
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]     origins={permute_1}
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   ), InputBuffer(name='primals_3', layout=FixedLayout('cuda', torch.float32, size=[10, 10], stride=[10, 1]))],
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   constant_args=(),
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   kwargs={},
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   output_view=None,
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   ordered_kwargs_for_cpp_kernel=(),
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   origin_node=mm,
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   origins={mm}
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG] )
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG] scheduling ComputedBuffer(name='buf1', layout=FixedLayout('cuda', torch.float32, size=[1, 10], stride=[10, 1]), data=Reduction(
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   'cuda',
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   torch.float32,
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   def inner_fn(index, rindex):
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]       _, i1 = index
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]       r0 = rindex
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]       tmp0 = ops.load(tangents_1, i1 + 10 * r0)
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]       return tmp0
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   ,
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   ranges=[1, 10],
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   reduction_ranges=[10],
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   reduction_type=sum,
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   origin_node=sum_1,
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG]   origins={sum_1}
[2023-11-20 15:52:39,429] torch._inductor.scheduler: [DEBUG] ))
[2023-11-20 15:52:39,430] torch._inductor.scheduler: [DEBUG] scheduling output buf0
[2023-11-20 15:52:39,430] torch._inductor.scheduler: [DEBUG] scheduling output buf1
[2023-11-20 15:52:39,431] torch._inductor.debug: [INFO] Writing debug ir to  torch_compile_debug/run_2023_11_20_15_52_36_931032-pid_1060761/torchinductor/model__0_backward_12.1/ir_pre_fusion.txt
[2023-11-20 15:52:39,434] torch._inductor.debug: [INFO] Writing debug ir to  torch_compile_debug/run_2023_11_20_15_52_36_931032-pid_1060761/torchinductor/model__0_backward_12.1/ir_post_fusion.txt
[2023-11-20 15:52:39,438] torch._inductor.scheduler: [DEBUG] Generating code for node buf0 with estimated runtime 0.588512
[2023-11-20 15:52:39,438] torch._inductor.scheduler: [DEBUG] Generating code for node buf1 with estimated runtime 0.215788
[2023-11-20 15:52:39,443] torch._inductor.codegen.triton: [DEBUG] Generating kernel code with kernel_name: triton_per_fused_sum_0
[2023-11-20 15:52:39,467] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-11-20 15:52:39,467] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_ctas: 1, num_stages: 1, enable_warp_specialization: False, enable_persistent: False
[2023-11-20 15:52:39,472] torch._inductor.graph: [DEBUG] Output code written to: /tmp/yi/cyinfgu7auaorubjnnbuxzbcm727a6nuaugj7jtghbhkksy7yemb.py
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] Output code: 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from ctypes import c_void_p, c_long
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import torch
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import math
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import random
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import os
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import tempfile
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from math import inf, nan
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.hooks import run_intermediate_hooks
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.utils import maybe_profile
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.codegen.memory_planning import _align as align
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch import device, empty, empty_strided
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.codecache import AsyncCompile
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.select_algorithm import extern_kernels
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] aten = torch.ops.aten
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] inductor_ops = torch.ops.inductor
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] assert_size_stride = torch._C._dynamo.guards.assert_size_stride
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] alloc_from_pool = torch.ops.inductor._alloc_from_pool
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] reinterpret_tensor = torch.ops.inductor._reinterpret_tensor
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] async_compile = AsyncCompile()
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] # kernel path: /tmp/ji/cjifagllonkqhk3vyveoqxakqaeesjuehqx72vwfxx3xqxhredf6.py
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] # Source Nodes: [], Original ATen: [aten.sum]
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] triton_per_fused_sum_0 = async_compile.triton('triton_', '''
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import triton
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import triton.language as tl
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.ir import ReductionHint
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.ir import TileHint
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.triton_heuristics import AutotuneHint, persistent_reduction
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.utils import instance_descriptor
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor import triton_helpers
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] @persistent_reduction(
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     size_hints=[16, 16],
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     reduction_hint=ReductionHint.DEFAULT,
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     filename=__file__,
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     triton_meta={'signature': {0: '*fp32', 1: '*fp32', 2: 'i32', 3: 'i32'}, 'device': 0, 'device_type': 'cuda', 'constants': {}, 'configs': [instance_descriptor(divisible_by_16=(0, 1), equal_to_1=(), ids_of_folded_args=(), divisible_by_8=())]},
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     inductor_meta={'autotune_hints': set(), 'kernel_name': 'triton_per_fused_sum_0', 'mutated_arg_names': []}
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] )
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] @triton.jit
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] def triton_(in_ptr0, out_ptr0, xnumel, rnumel, XBLOCK : tl.constexpr):
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     xnumel = 10
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     rnumel = 10
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     RBLOCK: tl.constexpr = 16
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     xoffset = tl.program_id(0) * XBLOCK
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     xindex = xoffset + tl.arange(0, XBLOCK)[:, None]
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     xmask = xindex < xnumel
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     rindex = tl.arange(0, RBLOCK)[None, :]
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     rmask = rindex < rnumel
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     r1 = rindex
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     x0 = xindex
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     tmp0 = tl.load(in_ptr0 + (x0 + (10*r1)), rmask & xmask, other=0.0)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     tmp1 = tl.broadcast_to(tmp0, [XBLOCK, RBLOCK])
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     tmp3 = tl.where(rmask & xmask, tmp1, 0)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     tmp4 = tl.sum(tmp3, 1)[:, None]
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     tl.store(out_ptr0 + (x0), tmp4, xmask)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] ''')
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import triton
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] import triton.language as tl
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._inductor.triton_heuristics import grid, start_graph, end_graph
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] from torch._C import _cuda_getCurrentRawStream as get_cuda_stream
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] async_compile.wait(globals())
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] del async_compile
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] def call(args):
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     primals_3, tangents_1 = args
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     args.clear()
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     assert_size_stride(primals_3, (10, 10), (10, 1))
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     assert_size_stride(tangents_1, (10, 10), (10, 1))
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     with torch.cuda._DeviceGuard(0):
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         torch.cuda.set_device(0) # no-op to ensure context
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         buf0 = empty((10, 10), device='cuda', dtype=torch.float32)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         # Source Nodes: [], Original ATen: [aten.mm]
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         extern_kernels.mm(reinterpret_tensor(tangents_1, (10, 10), (1, 10), 0), primals_3, out=buf0)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         del primals_3
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         buf1 = empty((1, 10), device='cuda', dtype=torch.float32)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         # Source Nodes: [], Original ATen: [aten.sum]
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         stream0 = get_cuda_stream(0)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         triton_per_fused_sum_0.run(tangents_1, buf1, 10, 10, grid=grid(10), stream=stream0)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         del tangents_1
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]         return (reinterpret_tensor(buf0, (10, 10), (10, 1), 0), reinterpret_tensor(buf1, (10, ), (1, ), 0), None, )
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] def benchmark_compiled_module(times=10, repeat=10):
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     from torch._dynamo.testing import rand_strided
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     from torch._inductor.utils import print_performance
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     primals_3 = rand_strided((10, 10), (10, 1), device='cuda:0', dtype=torch.float32)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     tangents_1 = rand_strided((10, 10), (10, 1), device='cuda:0', dtype=torch.float32)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     return print_performance(lambda: call([primals_3, tangents_1]), times=times, repeat=repeat)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] if __name__ == "__main__":
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     from torch._inductor.wrapper_benchmark import compiled_module_main
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG]     compiled_module_main('None', benchmark_compiled_module)
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [DEBUG] 
[2023-11-20 15:52:39,472] torch._inductor.graph.__output_code: [INFO] Output code written to: /tmp/yi/cyinfgu7auaorubjnnbuxzbcm727a6nuaugj7jtghbhkksy7yemb.py
[2023-11-20 15:52:39,478] torch._inductor.compile_fx: [DEBUG] FX codegen and compilation took 0.074s
[2023-11-20 15:52:39,478] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling BACKWARDS graph 0
[2023-11-20 15:52:39,478] torch._inductor.debug: [WARNING] model__0_backward_12 debug trace: /tmp/yi/cyinfgu7auaorubjnnbuxzbcm727a6nuaugj7jtghbhkksy7yemb.debug
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] Function                                Runtimes (s)
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] ------------------------------------  --------------
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner               2.4412
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] OutputGraph.call_user_compiler                1.3229
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] create_aot_dispatcher_function                0.6577
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] compile_fx.<locals>.fw_compiler_base          0.0817
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] GraphLowering.run                             0.0344
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] GraphLowering.compile_to_module               0.0792
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] Scheduler.__init__                            0.0182
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] Scheduler.codegen                             0.0135
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] WrapperCodeGen.generate                       0.0013
[2023-11-20 15:52:39,480] torch._dynamo.utils: [INFO] compile_fx.<locals>.bw_compiler               0.0757

### Minified repro

_No response_

### Versions

PyTorch Nightly

cc @bdhirsh @ezyang @msaroufim @anijain2305 @zou3519 @chauhang @wconstab