Repository: pytorch
Issue: More comprehensive debug logging of intermediate FX graphs for Inductor passes
Labels: triaged,oncall: pt2,module: inductor,internal ramp-up task

### 🐛 Describe the bug

Today, with aot and inductor logging, you get the graph produced by AOTAutograd, and then the post grad graph. However, there are a lot of intermediate passes scattered throughout the code which cannot be easily be discovered. I recently littered my checkout with this to pin down who was messing up my graph:

```
diff --git a/torch/_inductor/compile_fx.py b/torch/_inductor/compile_fx.py
index 4c76bccb778..c1d194696df 100644
--- a/torch/_inductor/compile_fx.py
+++ b/torch/_inductor/compile_fx.py
@@ -278,6 +278,8 @@ def compile_fx_inner(
     If you change the argument list for this function, make sure you
     also update the call to save_args_for_compile_fx_inner below accordingly.
     """
+    post_grad_graphs_log.debug("%s", lazy_format_graph_code("COMPILE_FX_INNER START", gm))
+
     if dynamo_utils.count_calls(gm.graph) == 0 and not aot_mode:
         return make_boxed_func(gm.forward)
 
@@ -514,7 +516,9 @@ def fx_codegen_and_compile(
         f"{'BACKWARDS' if is_backward else 'FORWARDS'} "
         f"graph {graph_id}",
     )
+    post_grad_graphs_log.debug("%s", lazy_format_graph_code("START", gm))
     V.debug.fx_graph(gm, example_inputs)
+    post_grad_graphs_log.debug("%s", lazy_format_graph_code("FX_GRAPH", gm))
 
     shape_env = _shape_env_from_inputs(example_inputs)
 
@@ -535,6 +539,7 @@ def fx_codegen_and_compile(
     # Also this has to be done before FakeTensorProp below to avoid the failed
     # .view() call.
     view_to_reshape(gm)
+    post_grad_graphs_log.debug("%s", lazy_format_graph_code("VIEW_TO_RESHAPE", gm))
 
     # It is safe to run FakeTensorProp under no_grad because by the time
     # we're in inductor, we assume that AOTAutograd has already "taken care"
@@ -542,6 +547,7 @@ def fx_codegen_and_compile(
     # graph.
     with torch.no_grad():
         fake_mode = fake_tensor_prop(gm, example_inputs)
+    post_grad_graphs_log.debug("%s", lazy_format_graph_code("FAKE_TENSOR_PROP", gm))
 
     # pattern matcher passes might not preserve striding information
     # on node.meta["val"]. if in the future we rely on these being
@@ -549,6 +555,7 @@ def fx_codegen_and_compile(
 
     with V.set_fake_mode(fake_mode):
         # has some issues with memory in training
+        post_grad_graphs_log.debug("%s", lazy_format_graph_code("PRE POST GRAD", gm))
         post_grad_passes(gm, is_inference=is_inference)
         V.debug.fx_graph_transformed(gm, example_inputs)
         post_grad_graphs_log.debug("%s", lazy_format_graph_code("AFTER POST GRAD", gm))
@@ -984,6 +991,7 @@ def compile_fx(
     config_patches: Optional[Dict[str, Any]] = None,
     decompositions: Optional[Dict[OpOverload, Callable[..., Any]]] = None,
 ):
+    post_grad_graphs_log.debug("%s", lazy_format_graph_code("COMPILE_FX START", model_))
     """Main entrypoint to a compile given FX graph"""
     if config_patches:
         with config.patch(config_patches):
```

Also most of the passes in torch/_inductor/fx_passes/joint_graph.py

There should be a way to dump all the intermediates.

This could be spammy. Some other suggestions from WC:

* Dump to files @aakhundov 
* Just dump everything in debug mode
* A way to control which FX passes are on/off @bdhirsh 
* A gas mechanism for how much optimization to do before you stop
* Log only the diff from the previous graph @eellison 

### Versions

main

cc @msaroufim @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler