Repository: pytorch
Issue: Catalogue of flaky tests under `test/dynamo`
Labels: triaged,oncall: pt2,module: dynamo

### 🐛 Describe the bug

1. predispatch + export + out_dtype fails when running twice. Somewhat mysterious, on second invocation, `make_fx` is baking in a constant as a fake tensor. `pytest test_export.py test_export.py -k test_predispatch_with_for_out_dtype`. They are run twice due to `test_dynamic`
```
 test/dynamo/test_export.py::ExportTests::test_predispatch_with_for_out_dtype - torch._subclasses.fake_tensor.DataDependentOutputException: aten.allclose.default
 test/dynamo/test_export.py::ExportTests::test_predispatch_with_for_out_dtype_nested - torch._subclasses.fake_tensor.DataDependentOutputException: aten.allclose.default
```

2. library registration fails when done twice `pytest  test_misc.py test_misc.py -k test_non_pt2_compliant_ops_graph_break`
```
 test/dynamo/test_misc.py::MiscTests::test_non_pt2_compliant_ops_graph_break - AttributeError: '_OpNamespace' object has no attribute 'bar2'
```

3. Not sure for this failure yet, but repro is `pytest test_logging.py test_recompile_ux.py`.
```
 test/dynamo/test_recompile_ux.py::RecompileUxTests::test_drop_cache_on_skip - AssertionError: False is not true
```

4. This test also fails if `test_logging` is run beforehand `pytest test_logging.py test_repros.py`
```
 test/dynamo/test_repros.py::ReproTests::test_optim_state_references_cleared - AssertionError: tensor([[0.1000, 0.1000, 0.1000,  ..., 0.1000, 0.1000, 0.1000],
```

5. This test fails if `test_modules` is run beforehand `pytest  test_modules.py test_repros.py `
```
 test/dynamo/test_repros.py::ReproTests::test_reformer_train - AssertionError: '3' != '1'
```


Non-flaky failing local tests (i.e. fails when run individually)
```
 test/dynamo/test_dynamic_shapes.py::DynamicShapesFunctionTests::test_numpy_random_dynamic_shapes - torch._dynamo.exc.Unsupported: call_function UserDefinedObjectVariable(randn) [ConstantVariable(int), ConstantVariable(int)] {}
 test/dynamo/test_functions.py::FunctionTests::test_numpy_random - torch._dynamo.exc.Unsupported: call_function UserDefinedObjectVariable(randn) [ConstantVariable(int), ConstantVariable(int)] {}
 test/dynamo/test_replay_record.py::ReplayRecordTests::test_fn_call_args - AssertionError: no logs of level ERROR or higher triggered on torch._dynamo
 test/dynamo/test_replay_record.py::ReplayRecordTests::test_local_module - AssertionError: no logs of level ERROR or higher triggered on torch._dynamo
 test/dynamo/test_replay_record.py::ReplayRecordTests::test_nonlocal_fn_call - AssertionError: no logs of level ERROR or higher triggered on torch._dynamo
 test/dynamo/test_replay_record.py::ReplayRecordTests::test_nonlocal_module_class - AssertionError: no logs of level ERROR or higher triggered on torch._dynamo
 test/dynamo/test_replay_record.py::ReplayRecordTests::test_nonlocal_module_fn_call - AssertionError: no logs of level ERROR or higher triggered on torch._dynamo
 test/dynamo/test_replay_record.py::ReplayRecordTests::test_successful_inline - AssertionError: no logs of level ERROR or higher triggered on torch._dynamo
 test/dynamo/test_replay_record.py::ReplayRecordTests::test_unsuccessful_inline - AssertionError: no logs of level ERROR or higher triggered on torch._dynamo
```

Uncatalogued failing local tests
```
 test/dynamo/test_repros.py::ReproTests::test_graph_break_on_jit_isinstance - KeyError: '140089192996896\n\nTo execute this test, run the following from the base repo dir:\n     python test/dynamo/test_repros.py -k test_graph_break_on_ji...
 test/dynamo/test_repros.py::ReproTests::test_sigmoid_out - KeyError: '140089192996896\n\nTo execute this test, run the following from the base repo dir:\n     python test/dynamo/test_repros.py -k test_sigmoid_out\n\nTh...
 test/dynamo/test_repros.py::ReproTests::test_sort_out - KeyError: '140089192996896\n\nTo execute this test, run the following from the base repo dir:\n     python test/dynamo/test_repros.py -k test_sort_out\n\nThis ...
 test/dynamo/test_repros.py::ReproTests::test_tokenization - KeyError: '140089192996896\n\nTo execute this test, run the following from the base repo dir:\n     python test/dynamo/test_repros.py -k test_tokenization\n\nT...
 test/dynamo/test_subclasses.py::SubclassTests::test_recompile_with_symbool_inputs - AssertionError
 test/dynamo/test_trace_rules.py::TraceRuleTests::test_torch_name_rule_map - AssertionError: False is not true : New torch objects: {<class 'torch._decomp.decompositions_for_rng.PhiloxStateTracker'>, <class 'torch.utils.tensorboard.writ...
 ```

cc: @voznesenskym @zou3519 @ezyang 

### Versions

N/A

cc @ezyang @msaroufim @bdhirsh @anijain2305 @zou3519 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @wconstab