Repository: pytorch
Issue: Deferred runtime asserts triggered during backwards should not trigger assert not self.runtime_asserts_frozen
Labels: triaged,module: dynamic shapes

### 🐛 Describe the bug

Internal xref: https://fb.workplace.com/groups/6829516587176185/posts/7506045909523246/

If a deferred runtime assert is being added after we have already produced guards for a shape environment, you can end up with:

```
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/_subclasses/fake_tensor.py", line 1078, in __torch_dispatch__
    return self.dispatch(func, types, args, kwargs)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/_subclasses/fake_tensor.py", line 1467, in dispatch
    return self._cached_dispatch_impl(func, types, args, kwargs)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/_subclasses/fake_tensor.py", line 1170, in _cached_dispatch_impl
    output = self._dispatch_impl(func, types, args, kwargs)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/_subclasses/fake_tensor.py", line 1747, in _dispatch_impl
    op_impl_out = op_impl(self, func, *args, **kwargs)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/_subclasses/fake_impls.py", line 179, in constructors
    r = func(*args, **new_kwargs)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/_ops.py", line 670, in __call__
    return self_._op(*args, **kwargs)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/fx/experimental/sym_node.py", line 438, in expect_size
    r = b.expect_true(file, line)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/fx/experimental/sym_node.py", line 429, in expect_true
    return self.shape_env.defer_runtime_assert(
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/fx/experimental/recording.py", line 245, in wrapper
    return fn(*args, **kwargs)
  File "/data/users/ivankobzarev/fbsource/buck-out/v2/gen/fbcode/e387b76d131edcc7/torchrec/distributed/tests/__test_pt2_multiprocess__/test_pt2_multiprocess#link-tree/torch/fx/experimental/symbolic_shapes.py", line 5381, in defer_runtime_assert
    assert not self.runtime_asserts_frozen, expr
torch._inductor.exc.LoweringException: AssertionError: 12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21 - Min(0, 12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21) + Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16 + 12*u18, 12*u16 + 12*u18 + 12*u20)) + Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16 + 12*u18 + 12*u20, 12*u16 + 12*u17 + 12*u18 + 12*u20)) + Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16 + 12*u17 + 12*u18 + 12*u20, 12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20)) - Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16 + 12*u18)) - Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16 + 12*u18 + 12*u20)) - Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16 + 12*u17 + 12*u18 + 12*u20)) - Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20)) + Min(12*u16 + 12*u17 + 12*u18 + 12*u19 + 12*u20 + 12*u21, Max(0, 12*u16, 12*u16 + 12*u18)) >= 0
```

Specifically, the deferred runtime assert is being added during Inductor code generation of the backwards (as can be seen in the backtrace).

However, this is a bit specious. Unlike guards, deferred runtime asserts can happily be run in backwards. So there isn't really any reason to freak out because an extra runtime assert is added. However, we do have to make sure we actually codegen the runtime assert, which I suspect isn't happening right now.

Not high priority, because in this case the guard is spurious and we can suppress it directly.

### Versions

main

cc @penguinwu