Repository: pytorch
Issue: [dynamo] incorrect error traceback for runtime errors when executing dynamo codegen
Labels: low priority,triaged,oncall: pt2,module: dynamo

### 🐛 Describe the bug

When runtime errors happen from executing the dynamo codegen, the error still points to the user program.

Real repro is too big. Toy repro: https://github.com/pytorch/pytorch/commit/eb370e0ed6dc8d87d5a804acabc7c9fb643365cc
In this toy example, we intentionally insert bad bytecode (simulating a real bytecode generation problem) at the end of graph.

```                                                                
Traceback (most recent call last):                                                                                                                                                                                                        
  File "/home/xmfan/core/pytorch/issue.py", line 6, in <module>                                                      
    torch.compile(foo)(torch.ones(10))                                                                                                                                                                                                    
  File "/home/xmfan/core/pytorch/torch/_dynamo/eval_frame.py", line 410, in _fn                                      
    return fn(*args, **kwargs)                                                                                       
  File "/home/xmfan/core/pytorch/issue.py", line 3, in foo                                                           
    def foo(x):                                                                                                      
IndexError: index 100 is out of bounds for dimension 0 with size 10 
```

The trackback here is not very helpful in finding out the real issue: incorrect bytecode in the graph epilogue. AFAIK, the only way to debug this currently is to think about using TORCH_LOGS="bytecode" and inspecting manually (See ERROR marker comment):
```
DEBUG:torch._dynamo.convert_frame.__bytecode:ORIGINAL BYTECODE foo /home/xmfan/core/pytorch/issue.py line 3          
  5           0 LOAD_FAST                0 (x)                                                                                                                                                                                            
              2 LOAD_METHOD              0 (sum)                                                                                                                                                                                          
              4 CALL_METHOD              0                                                                           
              6 RETURN_VALUE                                                                                                                                                                                                              
                                                                                                                     
                                                          
DEBUG:torch._dynamo.convert_frame.__bytecode:MODIFIED BYTECODE foo /home/xmfan/core/pytorch/issue.py line 3 
  3           0 LOAD_GLOBAL              1 (__compiled_fn_1)
              2 LOAD_FAST                0 (x)                                                                       
              4 CALL_FUNCTION            1                                                                                                                                                                                                
              6 UNPACK_SEQUENCE          1
              8 LOAD_FAST                0 (x)    
             10 LOAD_CONST               1 (100)  <------------ ERROR                                                                  
             12 BINARY_SUBSCR                                        
             14 RETURN_VALUE                                                                                         
                                                          
                                                                                                                                                                                                                                                                                                                    
Traceback (most recent call last):                                                                                                                                                                                                        
  File "/home/xmfan/core/pytorch/issue.py", line 6, in <module>                                                      
    torch.compile(foo)(torch.ones(10))                                                                                                                                                                                                    
  File "/home/xmfan/core/pytorch/torch/_dynamo/eval_frame.py", line 410, in _fn                                      
    return fn(*args, **kwargs)                                                                                       
  File "/home/xmfan/core/pytorch/issue.py", line 3, in foo                                                           
    def foo(x):                                                                                                      
IndexError: index 100 is out of bounds for dimension 0 with size 10 
```

We should at the very least be catching these runtime errors as InternalTorchDynamoError. And if possible provide a better error trace.

I ran across this when adding support for list stealing in dynamo for compiled autograd. But the repro wasn't minimal, adding the logs here to convey the experience:
<details>
  <summary>Non-toy error log for reference</summary>

```
Traceback (most recent call last):                                                                                                                                                         
  File "/home/xmfan/core/pytorch/torch/fx/graph_module.py", line 304, in __call__                                                                                                          
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]                                                                                                            
  File "/home/xmfan/core/pytorch/torch/nn/modules/module.py", line 1527, in _wrapped_call_impl                                                                                             
    return self._call_impl(*args, **kwargs)                                                                                                                                                
  File "/home/xmfan/core/pytorch/torch/nn/modules/module.py", line 1536, in _call_impl                                                                                                     
    return forward_call(*args, **kwargs)                                                                                                                                                   
  File "<eval_with_key>.0", line 4, in forward                                                                                                                                             
    def forward(self, inputs, sizes, hooks):                                                                                                                                               
IndexError: list index out of range                                                                                                                                                        
                                                                                                                                                                                           
Call using an FX-traced Module, line 4 of the traced Module's generated forward function:                                                                                                  
                                                                                                                                                                                           
def forward(self, inputs, sizes, hooks):

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    getitem = inputs[0]

    getitem_1 = inputs[1];  inputs = None
```

In this case, we need to inspect the actual error via TORCH_LOGS="bytecode". We should catch this as a DynamoInternalError and use this in the traceback error when possible
```
0 LOAD_GLOBAL             17 (__compiled_fn_0)                                                                                                                               
2 LOAD_FAST                1 (inputs)                                                                                                                                        
4 LOAD_CONST               2 (1)                                                                                                                                             
6 BINARY_SUBSCR                                                                                                                                                              
8 STORE_FAST              19 (inputs_ref_1)                                                                                                                                  
10 LOAD_FAST                1 (inputs)                                                                                                                                        
12 CALL_FUNCTION            1                         <-- call __compiled_fn_0 a.k.a. forward(self, inputs, sizes, hooks)                                                                                                                                               
14 STORE_FAST              18 (graph_out_0)                                                                                                                                   
16 BUILD_LIST               0                                                                                                                                                 
18 LOAD_FAST               18 (graph_out_0)                                                                                                                                   
20 LOAD_CONST               1 (0)                                                                                                                                             
22 BINARY_SUBSCR                                                                                                                                                              
24 LOAD_FAST               19 (inputs_ref_1)                                                                                                                                  
26 LOAD_FAST                1 (inputs)                <--- inputs is already empty at this point                                                                                                              
28 LOAD_CONST               1 (0)                                                                                                                                             
30 BINARY_SUBSCR                                                                                                                                                              
32 BUILD_LIST               1                                                                                                                                                 
34 LOAD_FAST                3 (hooks)                                                                                                                                         
36 LOAD_CONST               1 (0)                                                                                                                                             
38 BINARY_SUBSCR                                                                                                                                                              
40 LOAD_ATTR               15 (__closure__)                                                                                                                                   
42 LOAD_CONST               1 (0)                                                                                                                                             
44 BINARY_SUBSCR                                                                                                                                                              
46 LOAD_ATTR               16 (cell_contents)                                                                                                                                 
48 LOAD_CONST               0 (None)                                                                                                                                          
50 LOAD_CONST               0 (None)                                                                                                                                          
52 BUILD_SLICE              2                                                                                                                                                 
54 STORE_SUBSCR                                                                                                                                                               
56 STORE_ATTR              14 (grad)                                                                                                                                          
58 DELETE_FAST             18 (graph_out_0)                                                                                                                                   
60 RETURN_VALUE
```

</details>

### Versions

main

cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng