Repository: pytorch
Issue: [dynamic shapes] torch.fx.experimental.symbolic_shapes.ConstraintViolationError from demucs with dynamic shapes
Labels: triaged,oncall: pt2,module: dynamic shapes,module: inductor,module: dynamo

This is possibly related to #107200 which has a similar error.

Repro
```
./benchmarks/dynamo/torchbench.py --training  --accuracy --no-translation-validatio --amp --backend=inductor --only demucs --dynamic-shapes --dynamic-batch-only --disable-cudagraphs
```

Results in:
```
loading model: 0it [00:03, ?it/s]
cuda train demucs                             
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR] Error while creating guard:
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR] Name: ''
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     Source: shape_env
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     Create Function: SHAPE_ENV
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     Guard Types: None
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     Code List: None
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     Object Weakref: None
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     Guarded Class Weakref: None
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR] Created at:
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]   File "/home/jansel/pytorch/torch/_dynamo/convert_frame.py", line 509, in transform
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     tracer = InstructionTranslator(
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]   File "/home/jansel/pytorch/torch/_dynamo/symbolic_convert.py", line 2075, in __init__
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     output=OutputGraph(
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]   File "/home/jansel/pytorch/torch/_dynamo/output_graph.py", line 297, in __init__
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     self.init_ambient_guards()
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]   File "/home/jansel/pytorch/torch/_dynamo/output_graph.py", line 371, in init_ambient_guards
[2023-12-09 15:28:05,619] [5/0_1] torch._guards: [ERROR]     self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV))
ERROR:common:Constraints violated (L['streams'].size()[0])! For more information, run with TORCH_LOGS=dynamic.
  - Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4).


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True
Traceback (most recent call last):
  File "/home/jansel/pytorch/benchmarks/dynamo/common.py", line 2241, in check_accuracy
    new_result = optimized_model_iter_fn(model_copy, example_inputs)
  File "/home/jansel/pytorch/torch/_dynamo/eval_frame.py", line 488, in _fn
    return fn(*args, **kwargs)
  File "/home/jansel/pytorch/benchmarks/dynamo/common.py", line 1980, in run_n_iterations
    self.model_iter_fn(mod, inputs, collect_outputs=False)
  File "/home/jansel/pytorch/./benchmarks/dynamo/torchbench.py", line 535, in forward_and_backward_pass
    cloned_inputs = clone_inputs(inputs)
  File "/home/jansel/pytorch/./benchmarks/dynamo/torchbench.py", line 536, in resume_in_forward_and_backward_pass
    self.optimizer_zero_grad(mod)
  File "/home/jansel/pytorch/./benchmarks/dynamo/torchbench.py", line 538, in resume_in_forward_and_backward_pass
    pred = mod(*cloned_inputs)
  File "/home/jansel/pytorch/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jansel/pytorch/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jansel/pytorch/torch/_dynamo/eval_frame.py", line 654, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/jansel/pytorch/torch/_dynamo/convert_frame.py", line 727, in _convert_frame
    result = inner_convert(frame, cache_entry, hooks, frame_state)
  File "/home/jansel/pytorch/torch/_dynamo/convert_frame.py", line 383, in _convert_frame_assert
    compiled_product = _compile(
  File "/home/jansel/pytorch/torch/_dynamo/convert_frame.py", line 646, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/jansel/pytorch/torch/_dynamo/utils.py", line 244, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/jansel/pytorch/torch/_dynamo/convert_frame.py", line 626, in compile_inner
    check_fn = CheckFunctionManager(
  File "/home/jansel/pytorch/torch/_dynamo/guards.py", line 1011, in __init__
    guard.create(builder)
  File "/home/jansel/pytorch/torch/_guards.py", line 246, in create
    return self.create_fn(builder, self)
  File "/home/jansel/pytorch/torch/_dynamo/guards.py", line 670, in SHAPE_ENV
    guards = output_graph.shape_env.produce_guards(
  File "/home/jansel/pytorch/torch/fx/experimental/symbolic_shapes.py", line 3000, in produce_guards
    raise ConstraintViolationError(
torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['streams'].size()[0])! For more information, run with TORCH_LOGS=dynamic.
  - Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4).


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

TorchDynamo optimized model failed to run because of following error
fail_to_run
```

This could possibly be a result of duck-sizing, because if I apply this patch:
```patch
diff --git a/benchmarks/dynamo/torchbench.py b/benchmarks/dynamo/torchbench.py
index 633b9f60805..939fa56ff77 100755
--- a/benchmarks/dynamo/torchbench.py
+++ b/benchmarks/dynamo/torchbench.py
@@ -226,7 +226,7 @@ TRT_NOT_YET_WORKING = {
 }
 
 DONT_CHANGE_BATCH_SIZE = {
-    "demucs",
+    # "demucs",
     "pytorch_struct",
     "pyhpc_turbulent_kinetic_energy",
     "vision_maskrcnn",  # https://github.com/pytorch/benchmark/pull/1656
```

Then adding `--batch-size 2`:
```
./benchmarks/dynamo/torchbench.py --training  --accuracy --no-translation-validatio --amp --backend=inductor --only demucs --dynamic-shapes --dynamic-batch-only --disable-cudagraphs --stats --batch-size 2
...
  File "/home/jansel/torchbenchmark/torchbenchmark/models/demucs/demucs/augment.py", line 86, in forward
    raise ValueError(f"Batch size {batch} must be divisible by group size {group_size}")
ValueError: Batch size 2 must be divisible by group size 4
```
So the value of 4 is clearly special for this model.

Adding `--batch-size 8`:
```
./benchmarks/dynamo/torchbench.py --training  --accuracy --no-translation-validatio --amp --backend=inductor --only demucs --dynamic-shapes --dynamic-batch-only --disable-cudagraphs --stats --batch-size 8
loading model: 0it [00:03, ?it/s]
cuda train demucs                             
[2023-12-09 15:37:58,821] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.06715, (ref-fp64): 0.01621 and shape=torch.Size([])
fail_accuracy
```

So the ConstraintViolationError goes away with a different batch size.

Maybe we are over-specializing modulo guards.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler