Repository: pytorch
Issue: 2nd compile of deepcopy(model) fails on multiple ubuntu-pc (fatal error: Python.h: file not found)
Labels: triaged,topic: build,bug,oncall: pt2,module: dynamo

### 🐛 Describe the bug

I use different Systems: `Python 3.11, torch 2.2.2, Ubuntu 22.04 or 23.10 (x64), all Cuda 12.x, all Intel CPUs (11..13 gen), all using pip not conda` system language setting english or german.
First torch.compile(model) works fine on all 5 systems, and numeric results are not affected by compile.
But 2nd compile (as we use two model instances in parallel) fails on some systems (since some weeks - see below).

As far as I remember: the error started with torch 2.3, but downgrade to 2.2 did not recover. (not sure if this true for all 5 pc)
So I cannot afford to check on another system doing a torch version change at the moment (I'm afraid of more issues).

But anyway, I have no idea what could be the reason for suddenly not finding a header file, that was useable shortly before, again and again .. (what is different for 2nd compile regarding system setup?)
Perhaps some post-compile cleanup is removing the header file after first compile, and it is not prepared a 2nd time (as most users only compile one model per run, it could be invisible issue usually, just speculation here ..)

```
class Net(torch.nn.Module): # 10+10, N= 7960
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = torch.nn.Linear(784, 10)
        self.relu = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(10, 10)

    def forward(self, x): # line 59 from stderr log below
        z1 = self.fc1(torch.flatten(x, start_dim=1))
        z1 = self.fc2(self.relu(z1))
        return F.log_softmax(z1, dim=1)

# model is trivial Net above or ResNet (18 or 34 or Wide28 or 50)
model = torch.compile(model) # always ok
# here 10 epochs training (good progess, no issues)
model2 = deepcopy(model) # ok
model2 = torch.compile(model2) # fails (system dependend)
```

parts of stderr log:
```
/tmp/tmpmiq1y2mh/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden # german: cannot find file
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpj15_8vtb/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated. # repeated for 8 C-files.
....
torch._dynamo.convert_frame: [WARNING] CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp5o0_7ce5/main.c', '-O3', '-I/home/aleks/myvenv3.11/lib/python3.11/site-packages/triton/common/../third_party/cuda/include', '-I/usr/include/python3.11', '-I/tmp/tmp5o0_7ce5', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmp5o0_7ce5/triton_.cpython-311-x86_64-linux-gnu.so', '-L/lib/x86_64-linux-gnu', '-L/lib/i386-linux-gnu', '-L/lib/x86_64-linux-gnu', '-L/lib/i386-linux-gnu']' returned non-zero exit status 1.
```


### Error logs

/tmp/tmpmiq1y2mh/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden # german: cannot find file
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpj15_8vtb/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpqemdf8mu/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpzbktfctd/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmp32faforh/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmplpa8sknu/main.c:5:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    5 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmp5o0_7ce5/main.c:4:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] WON'T CONVERT forward /data/Sourcen/git/numericspublication/main_elra.py line 59 
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] due to: 
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] Traceback (most recent call last):
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 727, in _convert_frame
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     result = inner_convert(frame, cache_entry, hooks, frame_state)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 383, in _convert_frame_assert
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_product = _compile(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                        ^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 646, in _compile
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     guarded_code = compile_inner(code, one_graph, hooks, transform)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 562, in compile_inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     out_code = transform_code_object(code, transform)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     transformations(instructions, code_options)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return fn(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 527, in transform
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     tracer.run()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2128, in run
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     super().run()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     and self.step()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     getattr(self, inst.opname)(inst)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2243, in RETURN_VALUE
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     self.output.compile_subgraph(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 919, in compile_subgraph
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/contextlib.py", line 81, in inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return func(*args, **kwds)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1087, in compile_and_call_fx_graph
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = self.call_user_compiler(gm)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1159, in call_user_compiler
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1140, in call_user_compiler
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = compiler_fn(gm, self.example_inputs())
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_gm = compiler_fn(gm, example_inputs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/__init__.py", line 1668, in __call__
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return compile_fx(model_, inputs_, config_patches=self.config)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1168, in compile_fx
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return aot_autograd(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     cg = aot_module_simplified(gm, example_inputs, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 887, in aot_module_simplified
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = create_aot_dispatcher_function(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 600, in create_aot_dispatcher_function
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 425, in aot_wrapper_dedupe
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 630, in aot_wrapper_synthetic_base
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 97, in aot_dispatch_base
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fw = compiler(fw_module, updated_flat_args)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1100, in fw_compiler_base
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return inner_compile(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     inner_compiled_fn = compiler_fn(gm, example_inputs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/debug.py", line 305, in inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return fn(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/contextlib.py", line 81, in inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return func(*args, **kwds)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 320, in compile_fx_inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_graph = fx_codegen_and_compile(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                      ^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 550, in fx_codegen_and_compile
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = graph.compile_to_fn()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1116, in compile_to_fn
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return self.compile_to_module().call
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1070, in compile_to_module
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     mod = PyCodeCache.load_by_key_path(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 1892, in load_by_key_path
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     exec(code, mod.__dict__, mod.__dict__)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/tmp/torchinductor_aleks/si/csivrh32cwzjdjoq6hzozmar3c3khh7gta6rjyou63fgxeyso6zl.py", line 115, in <module>
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     async_compile.wait(globals())
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2486, in wait
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     scope[key] = result.result()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                  ^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2329, in result
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     self.future.result()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return self.__get_result()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     raise self._exception
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp5o0_7ce5/main.c', '-O3', '-I/home/aleks/myvenv3.11/lib/python3.11/site-packages/triton/common/../third_party/cuda/include', '-I/usr/include/python3.11', '-I/tmp/tmp5o0_7ce5', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmp5o0_7ce5/triton_.cpython-311-x86_64-linux-gnu.so', '-L/lib/x86_64-linux-gnu', '-L/lib/i386-linux-gnu', '-L/lib/x86_64-linux-gnu', '-L/lib/i386-linux-gnu']' returned non-zero exit status 1.
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] 
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] 
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] Traceback (most recent call last):
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 727, in _convert_frame
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     result = inner_convert(frame, cache_entry, hooks, frame_state)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 383, in _convert_frame_assert
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_product = _compile(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                        ^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 646, in _compile
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     guarded_code = compile_inner(code, one_graph, hooks, transform)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 562, in compile_inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     out_code = transform_code_object(code, transform)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     transformations(instructions, code_options)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return fn(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 527, in transform
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     tracer.run()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2128, in run
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     super().run()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     and self.step()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     getattr(self, inst.opname)(inst)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2243, in RETURN_VALUE
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     self.output.compile_subgraph(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 919, in compile_subgraph
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/contextlib.py", line 81, in inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return func(*args, **kwds)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1087, in compile_and_call_fx_graph
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = self.call_user_compiler(gm)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1159, in call_user_compiler
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1140, in call_user_compiler
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = compiler_fn(gm, self.example_inputs())
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_gm = compiler_fn(gm, example_inputs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/__init__.py", line 1668, in __call__
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return compile_fx(model_, inputs_, config_patches=self.config)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1168, in compile_fx
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return aot_autograd(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     cg = aot_module_simplified(gm, example_inputs, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 887, in aot_module_simplified
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = create_aot_dispatcher_function(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 600, in create_aot_dispatcher_function
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 425, in aot_wrapper_dedupe
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 630, in aot_wrapper_synthetic_base
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 97, in aot_dispatch_base
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fw = compiler(fw_module, updated_flat_args)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1100, in fw_compiler_base
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return inner_compile(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     inner_compiled_fn = compiler_fn(gm, example_inputs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/debug.py", line 305, in inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return fn(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/contextlib.py", line 81, in inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return func(*args, **kwds)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 320, in compile_fx_inner
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_graph = fx_codegen_and_compile(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                      ^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 550, in fx_codegen_and_compile
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     compiled_fn = graph.compile_to_fn()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                   ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1116, in compile_to_fn
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return self.compile_to_module().call
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     r = func(*args, **kwargs)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]         ^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1070, in compile_to_module
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     mod = PyCodeCache.load_by_key_path(
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 1892, in load_by_key_path
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     exec(code, mod.__dict__, mod.__dict__)
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/tmp/torchinductor_aleks/si/csivrh32cwzjdjoq6hzozmar3c3khh7gta6rjyou63fgxeyso6zl.py", line 115, in <module>
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     async_compile.wait(globals())
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2486, in wait
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     scope[key] = result.result()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]                  ^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2329, in result
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     self.future.result()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     return self.__get_result()
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]            ^^^^^^^^^^^^^^^^^^^
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING]     raise self._exception
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp5o0_7ce5/main.c', '-O3', '-I/home/aleks/myvenv3.11/lib/python3.11/site-packages/triton/common/../third_party/cuda/include', '-I/usr/include/python3.11', '-I/tmp/tmp5o0_7ce5', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmp5o0_7ce5/triton_.cpython-311-x86_64-linux-gnu.so', '-L/lib/x86_64-linux-gnu', '-L/lib/i386-linux-gnu', '-L/lib/x86_64-linux-gnu', '-L/lib/i386-linux-gnu']' returned non-zero exit status 1.
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] 
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
[2024-06-06 07:46:58,066] torch._dynamo.convert_frame: [WARNING] 
/tmp/tmp4aexwpzz/main.c:4:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
Exception ignored in atexit callback: <function _MultiProcessingDataLoaderIter._clean_up_worker at 0x7511dbf958a0>
Traceback (most recent call last):
  File "/home/aleks/myvenv3.11/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1473, in _clean_up_worker
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.11/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 40, in wait
    if not wait([self.sentinel], timeout):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 948, in wait
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt: 

### Minified repro

A short snipset is above.
It takes significant effort to shrink the project (and I cannot do it this week). But not finding a header inside compile should not depend on our optimizer training code. Please read the above first. If needed, I could provide a hidden private repo next week.

### Versions

Ubuntu 22.04 (x64) + Python 3.11.9

torch                    2.2.2
torchaudio               2.2.2
torchvideo               0.0.0
torchvision              0.17.2

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
|   0  NVIDIA GeForce RTX 4060 Ti     Off | 00000000:01:00.0  On |                  N/A |
+-----------------------------------------+----------------------+----------------------+

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0


cc @ezyang @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @msaroufim