Repository: pytorch
Issue: [Inductor test failure] torch:inductor/test_select_algorithm TestSelectAlgorithm.test_convolution1 with cuda 12.6.3
Labels: needs reproduction,triaged,oncall: pt2,module: inductor

### 🐛 Describe the bug

This is Qianli from Google. Our GPU team is testing the release of cuda 12.6.3 and noticed a test failure from inductor in our internal infra. I don't have a dev box setup to fully mirror this issue in a OSS env, but please see the error log and see if anything in the recent cuda change could cause this error. 

The current internal cuda version used by us is 12.0, and from https://pytorch.org/, it seems that pytorch support 12.4. This potentially indicate that the behavior change happens between 12.4 and 12.6.3.


In the passing run, the info log looks like below, somehow the pass of triton_convolution2d_48 failed.

```
[test/inductor/test_select_algorithm](http://test/inductor/test_select_algorithm).py::TestSelectAlgorithm::test_convolution1 AUTOTUNE convolution(2x33x34x41, 34x33x3x3)
  convolution 0.0436 ms 100.0% 
  triton_convolution2d_46 0.0592 ms 73.7% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=2, num_warps=4
  triton_convolution2d_50 0.0749 ms 58.2% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=2, num_warps=4
  triton_convolution2d_51 0.1000 ms 43.6% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=2, num_warps=8
  triton_convolution2d_49 0.1172 ms 37.2% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=2, num_warps=8
  triton_convolution2d_52 0.1977 ms 22.1% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=2, num_warps=8
  triton_convolution2d_47 0.2401 ms 18.2% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=2, num_warps=4
  triton_convolution2d_48 0.2537 ms 17.2% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=1, num_warps=8

frames [('total', 1), ('ok', 1)]
stats [('calls_captured', 2), ('unique_graphs', 1)]
aot_autograd [('total', 1), ('ok', 1)]
```

### Error logs

Traceback (most recent call last):
  File "torch/test/inductor/test_select_algorithm.py", line 264, in test_convolution1
    foo(
  File "torch/_dynamo/eval_frame.py", line 556, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/convert_frame.py", line 1404, in __call__
    return self._torchdynamo_orig_callable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/convert_frame.py", line 1188, in __call__
    result = self._inner_convert(
             ^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/convert_frame.py", line 549, in __call__
    return _compile(
           ^^^^^^^^^
  File "torch/_dynamo/convert_frame.py", line 985, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/convert_frame.py", line 712, in compile_inner
    return _compile_inner(code, one_graph, hooks, transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_utils_internal.py", line 95, in wrapper_function
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/convert_frame.py", line 747, in _compile_inner
    out_code = transform_code_object(code, transform)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/bytecode_transformation.py", line 1348, in transform_code_object
    transformations(instructions, code_options)
  File "torch/_dynamo/convert_frame.py", line 233, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/convert_frame.py", line 664, in transform
    tracer.run()
  File "torch/_dynamo/symbolic_convert.py", line 2841, in run
    super().run()
  File "torch/_dynamo/symbolic_convert.py", line 1032, in run
    while self.step():
          ^^^^^^^^^^^
  File "torch/_dynamo/symbolic_convert.py", line 944, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "torch/_dynamo/symbolic_convert.py", line 3021, in RETURN_VALUE
    self._return(inst)
  File "torch/_dynamo/symbolic_convert.py", line 3006, in _return
    self.output.compile_subgraph(
  File "torch/_dynamo/output_graph.py", line 1077, in compile_subgraph
    self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
  File "torch/_dynamo/output_graph.py", line 1349, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/output_graph.py", line 1399, in call_user_compiler
    return self._call_user_compiler(gm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/output_graph.py", line 1448, in _call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "torch/_dynamo/output_graph.py", line 1429, in _call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
    compiled_gm = compiler_fn(gm, example_inputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/__init__.py", line 2304, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/compile_fx.py", line 1733, in compile_fx
    return aot_autograd(
           ^^^^^^^^^^^^^
  File "torch/_dynamo/backends/common.py", line 72, in __call__
    cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_functorch/aot_autograd.py", line 1103, in aot_module_simplified
    compiled_fn = dispatch_and_compile()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_functorch/aot_autograd.py", line 1079, in dispatch_and_compile
    compiled_fn, _ = create_aot_dispatcher_function(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_functorch/aot_autograd.py", line 527, in create_aot_dispatcher_function
    return _create_aot_dispatcher_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_functorch/aot_autograd.py", line 778, in _create_aot_dispatcher_function
    compiled_fn, fw_metadata = compiler_fn(
                               ^^^^^^^^^^^^
  File "torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 197, in aot_dispatch_base
    compiled_fw = compiler(fw_module, updated_flat_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/compile_fx.py", line 1546, in fw_compiler_base
    return _fw_compiler_base(model, example_inputs, is_inference)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/compile_fx.py", line 1615, in _fw_compiler_base
    return inner_compile(
           ^^^^^^^^^^^^^^
  File "torch/_inductor/compile_fx.py", line 599, in compile_fx_inner
    return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_dynamo/repro/after_aot.py", line 102, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/compile_fx.py", line 756, in _compile_fx_inner
    compiled_graph = FxGraphCache.load(
                     ^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/codecache.py", line 1577, in load
    compiled_graph = compile_fx_fn(
                     ^^^^^^^^^^^^^^
  File "torch/_inductor/compile_fx.py", line 663, in codegen_and_compile
    compiled_graph = fx_codegen_and_compile(gm, example_inputs, **fx_kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/compile_fx.py", line 974, in fx_codegen_and_compile
    compiled_fn = graph.compile_to_fn()
                  ^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/graph.py", line 2019, in compile_to_fn
    return self._compile_to_fn()
           ^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/graph.py", line 2051, in _compile_to_fn
    return self.compile_to_module().call
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/graph.py", line 1957, in compile_to_module
    return self._compile_to_module()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/graph.py", line 1963, in _compile_to_module
    self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
                                                             ^^^^^^^^^^^^^^
  File "torch/_inductor/graph.py", line 1894, in codegen
    self.scheduler = Scheduler(self.operations)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/scheduler.py", line 1763, in __init__
    self._init(nodes)
  File "torch/_inductor/scheduler.py", line 1833, in _init
    self.nodes = self.fuse_nodes(self.nodes)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/scheduler.py", line 2326, in fuse_nodes
    nodes = self.fuse_nodes_once(nodes)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/scheduler.py", line 2622, in fuse_nodes_once
    if not self.speedup_by_fusion(node1, node2):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/scheduler.py", line 2519, in speedup_by_fusion
    choice_timings = multi_node.choice_timings
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/ir.py", line 4252, in choice_timings
    self._choice_timings = self._choice_timings_fn()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/select_algorithm.py", line 1468, in get_timings
    timings = do_autotuning(precompile_fn)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/select_algorithm.py", line 1433, in do_autotuning
    timings = self.lookup(
              ^^^^^^^^^^^^
  File "torch/test/inductor/test_select_algorithm.py", line 26, in skip_cache
    return benchmark(choices)
           ^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/select_algorithm.py", line 1418, in autotune
    return make_benchmark_fn()(choices)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "torch/_inductor/select_algorithm.py", line 1611, in benchmark_in_current_process
    raise AssertionError(  # noqa: B904
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
AssertionError: Incorrect result from choice TritonTemplateCaller(/tmp/tmpojnhspqd/lu/cluvzbjsxripzemg5ihr3rxsnjpiin5vvwg3ejwznf7th2yegd42.py, ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=4, PADDING_W=5, STRIDE_H=2, STRIDE_W=3, UNROLL=False, num_stages=1, num_warps=8)

Tensor-likes are not close!

Mismatched elements: 19584 / 23120 (84.7%)
Greatest absolute difference: 132.32015991210938 at index (0, 22, 4, 13) (up to 0.0001 allowed)
Greatest relative difference: inf at index (0, 0, 1, 0) (up to 0.0001 allowed)


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True


To execute this test, run the following from the base repo dir:
    PYTORCH_TEST_WITH_ASAN=1 PYTORCH_TEST_WITH_UBSAN=1 python test/inductor/test_select_algorithm.py TestSelectAlgorithm.test_convolution1

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0


### Versions

Torch version:
commit hash f4ce9ac29d524fae51b6d8e300c4cab016fc8f18
closest_version: "ciflow/periodic/0b13bdd877f7b612ab2990e327ab3b40242945bf"

cc @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @aakhundov