Repository: pytorch
Issue: inductor.test_unbacked_symints.TestUnbackedSymintsCPU isn't run on OSS CI
Labels: module: ci,triaged,oncall: pt2,module: dynamic shapes,module: inductor

https://github.com/pytorch/pytorch/pull/124021 was green, but its imported diff D57313221 has a failed test.

```
buck2 test '@fbcode//mode/opt' fbcode//caffe2/test/inductor:unbacked_symints -- --exact 'caffe2/test/inductor:unbacked_symints - test_autotuning_cpu (caffe2.test.inductor.test_unbacked_symints.TestUnbackedSymintsCPU)'
```

It turns out that OSS CI on the PR didn't run this test at all and missed the failure.  Specifically, `python -bb inductor/test_unbacked_symints.py -m serial --shard-id=1 --num-shards=1 -v -vv -rfEX -p no:xdist --use-pytest -x --reruns=2 --import-slow-tests --import-disabled-tests` ran 0 tests.  This is due to the `is_big_gpu(0)` check https://github.com/pytorch/pytorch/blob/main/test/inductor/test_unbacked_symints.py#L199-L200 which short-circuits the whole test on OSS CI.  IMO, `inductor.test_unbacked_symints.TestUnbackedSymintsCPU` should still be run because it's a CPU, not a CUDA tests.

If I bypass `run_test` and uses pytest directly `pytest -v test/inductor/test_unbacked_symints.py`, the failure can be reproduced:

```
pytest -v test/inductor/test_unbacked_symints.py
=================================================================== test session starts ====================================================================
platform linux -- Python 3.11.8, pytest-7.3.2, pluggy-1.5.0 -- /data/users/huydo/conda/py3.11/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/data/users/huydo/github/pytorch/.hypothesis/examples')
rootdir: /data/users/huydo/github/pytorch
configfile: pytest.ini
plugins: xdoctest-1.1.0, hypothesis-5.35.1, xdist-3.3.1, rerunfailures-14.0, flakefinder-1.1.0, cpp-2.3.0
collected 18 items
Running 18 items in this shard

test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_autotuning_cpu FAILED [2.7126s]                                                 [  5%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_broadcast_tensors_cpu PASSED [16.3224s]                                         [ 11%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_equivalent_backed_unbacked_cpu PASSED [4.0925s]                                 [ 16%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_expand_cpu PASSED [0.2888s]                                                     [ 22%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_expand_ok_with_runtime_assert_cpu PASSED [0.2045s]                              [ 27%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_nonzero_in_inference_mode_cpu PASSED [0.1760s]                                  [ 33%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_split_with_sizes_cpu PASSED [4.2065s]                                           [ 38%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_triton_kernel_grid_cpu SKIPPED [0.0033s] (Triton kernel requires GPU)           [ 44%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_view_of_slice_cpu PASSED [2.9532s]                                              [ 50%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_autotuning_cuda PASSED [44.0452s]                                              [ 55%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_broadcast_tensors_cuda PASSED [2.2866s]                                        [ 61%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_equivalent_backed_unbacked_cuda PASSED [8.9798s]                               [ 66%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_expand_cuda PASSED [0.1728s]                                                   [ 72%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_expand_ok_with_runtime_assert_cuda PASSED [0.1306s]                            [ 77%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_nonzero_in_inference_mode_cuda PASSED [0.3753s]                                [ 83%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_split_with_sizes_cuda PASSED [4.4997s]                                         [ 88%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_triton_kernel_grid_cuda PASSED [6.3483s]                                       [ 94%]
test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCUDA::test_view_of_slice_cuda PASSED [2.8973s]                                            [100%]

========================================================================= FAILURES =========================================================================
________________________________________________________ TestUnbackedSymintsCPU.test_autotuning_cpu ________________________________________________________
Traceback (most recent call last):
  File "/data/users/huydo/github/pytorch/test/inductor/test_unbacked_symints.py", line 83, in test_autotuning
    actual = torch.compile(fn, fullgraph=True)(*example_inputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 420, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 1069, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state, skip=1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 464, in _convert_frame_assert
    return _compile(
           ^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_utils_internal.py", line 70, in wrapper_function
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 791, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 626, in compile_inner
    out_code = transform_code_object(code, transform)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1167, in transform_code_object
    transformations(instructions, code_options)
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 178, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 572, in transform
    tracer.run()
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2234, in run
    super().run()
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 884, in run
    while self.step():
          ^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 799, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2423, in RETURN_VALUE
    self._return(inst)
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2408, in _return
    self.output.compile_subgraph(
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1084, in compile_subgraph
    self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
  File "/data/users/huydo/conda/py3.11/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1301, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1392, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1373, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/__init__.py", line 1747, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1492, in compile_fx
    return aot_autograd(
           ^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/backends/common.py", line 65, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 962, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 683, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(
                  ^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 177, in aot_dispatch_base
    compiled_fw = compiler(fw_module, updated_flat_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1397, in fw_compiler_base
    return inner_compile(
           ^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/debug.py", line 304, in inner
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 513, in compile_fx_inner
    compiled_graph = FxGraphCache.load(
                     ^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 991, in load
    compiled_graph = compile_fx_fn(gm, example_inputs, **fx_kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 799, in fx_codegen_and_compile
    graph.run(*example_inputs)
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 732, in run
    return super().run(*args)
           ^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/fx/interpreter.py", line 145, in run
    self.env[node] = self.run_node(node)
                     ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1206, in run_node
    result = super().run_node(n)
             ^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/fx/interpreter.py", line 202, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 979, in call_function
    raise LoweringException(e, target, args, kwargs).with_traceback(
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 976, in call_function
    out = lowerings[target](*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/lowering.py", line 303, in wrapped
    out = decomp_fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/kernel/mm.py", line 161, in tuned_mm
    if use_cpp_packed_gemm_template(layout, mat1, mat2):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/utils.py", line 1010, in use_cpp_packed_gemm_template
    micro_gemm = create_micro_gemm(
                 ^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/codegen/cpp_micro_gemm.py", line 347, in create_micro_gemm
    m = V.graph.sizevars.size_hint(m) if isinstance(m, sympy.Expr) else m
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/torch/_inductor/sizevars.py", line 434, in size_hint
    return int(out)
           ^^^^^^^^
  File "/data/users/huydo/conda/py3.11/lib/python3.11/site-packages/sympy/core/expr.py", line 320, in __int__
    raise TypeError("Cannot convert symbols to int")
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
LoweringException: TypeError: Cannot convert symbols to int
  target: aten.mm.default
  args[0]: TensorBox(StorageBox(
    ComputedBuffer(name='buf4', layout=FlexibleLayout('cpu', torch.float32, size=[u0, 32], stride=[32, 1]), data=Pointwise(
      'cpu',
      torch.float32,
      def inner_fn(index):
          i0, i1 = index
          tmp0 = ops.constant(1, torch.float32)
          return tmp0
      ,
      ranges=[u0, 32],
      origin_node=full,
      origins={full}
    ))
  ))
  args[1]: TensorBox(StorageBox(
    InputBuffer(name='arg1_1', layout=FixedLayout('cpu', torch.float32, size=[32, 16], stride=[16, 1]))
  ))

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True


To execute this test, run the following from the base repo dir:
     python test/inductor/test_unbacked_symints.py -k TestUnbackedSymintsCPU.test_autotuning_cpu

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------
stats [('calls_captured', 4)]
inductor [('fxgraph_cache_miss', 1)]
aot_autograd [('total', 1), ('not_ok', 1)]
================================================================= short test summary info ==================================================================
FAILED [2.7126s] test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU::test_autotuning_cpu - torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
=================================================== 1 failed, 16 passed, 1 skipped in 110.86s (0:01:50) ====================================================
```
 

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire