Repository: pytorch
Issue: Torch compile stochastically fails with FileNotFoundError
Labels: triaged,oncall: pt2,module: inductor

### 🐛 Describe the bug

Torch compile stochastically fails during multinode training with FileNotFoundError in torch._dynamo. Full stack trace below. This is a difficult bug to provide a minimal reproduction for - it randomly occurs in torch.compile (a quite slow operation) and shows up often only in large-scale distributed multinode training, while being very rare in single-node training. The use of concurrent.futures in conjunction with the FileNotFoundError makes me strongly suspect a race condition. 

[rank60]: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/torchinductor_ubuntu/triton/4/5eba80077c5d5c6def3f812b8cccbc7d/triton_.cubin.tmp.pid_93828_100098'

```
torchrun --nproc_per_node 8 --nnodes 8 --rdzv-backend c10d --rdzv-endpoint foundry-1:29500 exp/codebook_collapse_trace.py --stage 1 --latent-type 'kl' --load-from quart_maze_vacation_120000 --step 158000
W0824 00:04:25.161000 140224365408704 torch/distributed/run.py:757]
W0824 00:04:25.161000 140224365408704 torch/distributed/run.py:757] *****************************************
W0824 00:04:25.161000 140224365408704 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0824 00:04:25.161000 140224365408704 torch/distributed/run.py:757] *****************************************
All clear!
Rank 56 of 64.
Rank 63 of 64.
Rank 57 of 64.
Rank 60 of 64.
Rank 59 of 64.
Rank 62 of 64.
Rank 58 of 64.
Rank 61 of 64.


[rank60]: Traceback (most recent call last):
[rank60]:   File "/home/ubuntu/si/exp/codebook_collapse_trace.py", line 435, in <module>
[rank60]:     trainer_config(default(tokenizer, None)).train()
[rank60]:   File "/home/ubuntu/si/si/contact/hertz.py", line 588, in train
[rank60]:     self.do_step(data)
[rank60]:   File "/home/ubuntu/si/si/contact/hertz.py", line 575, in do_step
[rank60]:     self.eval_step()
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank60]:     return func(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/si/contact/hertz.py", line 395, in eval_step
[rank60]:     eval_recons, *_ = self.vae(eval_audio)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank60]:     return self._call_impl(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank60]:     return forward_call(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1593, in forward
[rank60]:     else self._run_ddp_forward(*inputs, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1411, in _run_ddp_forward
[rank60]:     return self.module(*inputs, **kwargs)  # type: ignore[index]
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank60]:     return self._call_impl(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank60]:     return forward_call(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
[rank60]:     return fn(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank60]:     return self._call_impl(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank60]:     return forward_call(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 917, in catch_errors
[rank60]:     return hijacked_callback(frame, cache_entry, hooks, frame_state)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 786, in _convert_frame
[rank60]:     result = inner_convert(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert
[rank60]:     return _compile(
[rank60]:   File "/usr/lib/python3.10/contextlib.py", line 79, in inner
[rank60]:     return func(*args, **kwds)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 676, in _compile
[rank60]:     guarded_code = compile_inner(code, one_graph, hooks, transform)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
[rank60]:     r = func(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 535, in compile_inner
[rank60]:     out_code = transform_code_object(code, transform)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1036, in transform_code_object
[rank60]:     transformations(instructions, code_options)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 165, in _fn
[rank60]:     return fn(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 500, in transform
[rank60]:     tracer.run()
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2149, in run
[rank60]:     super().run()
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 810, in run
[rank60]:     and self.step()
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 773, in step
[rank60]:     getattr(self, inst.opname)(inst)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2268, in RETURN_VALUE
[rank60]:     self.output.compile_subgraph(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1001, in compile_subgraph
[rank60]:     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
[rank60]:   File "/usr/lib/python3.10/contextlib.py", line 79, in inner
[rank60]:     return func(*args, **kwds)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1178, in compile_and_call_fx_graph
[rank60]:     compiled_fn = self.call_user_compiler(gm)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
[rank60]:     r = func(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1251, in call_user_compiler
[rank60]:     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1232, in call_user_compiler
[rank60]:     compiled_fn = compiler_fn(gm, self.example_inputs())
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/backends/distributed.py", line 606, in compile_fn
[rank60]:     submod_compiler.run(*example_inputs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 145, in run
[rank60]:     self.env[node] = self.run_node(node)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/backends/distributed.py", line 348, in run_node
[rank60]:     compiled_submod_real = self.compile_submod(real_mod, new_args, kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/backends/distributed.py", line 263, in compile_submod
[rank60]:     self.compiler(input_mod, args),
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
[rank60]:     compiled_gm = compiler_fn(gm, example_inputs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/__init__.py", line 1731, in __call__
[rank60]:     return compile_fx(model_, inputs_, config_patches=self.config)
[rank60]:   File "/usr/lib/python3.10/contextlib.py", line 79, in inner
[rank60]:     return func(*args, **kwds)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1330, in compile_fx
[rank60]:     return aot_autograd(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 58, in compiler_fn
[rank60]:     cg = aot_module_simplified(gm, example_inputs, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 903, in aot_module_simplified
[rank60]:     compiled_fn = create_aot_dispatcher_function(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
[rank60]:     r = func(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 628, in create_aot_dispatcher_function
[rank60]:     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 443, in aot_wrapper_dedupe
[rank60]:     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 648, in aot_wrapper_synthetic_base
[rank60]:     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 119, in aot_dispatch_base
[rank60]:     compiled_fw = compiler(fw_module, updated_flat_args)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
[rank60]:     r = func(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1257, in fw_compiler_base
[rank60]:     return inner_compile(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
[rank60]:     inner_compiled_fn = compiler_fn(gm, example_inputs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/debug.py", line 304, in inner
[rank60]:     return fn(*args, **kwargs)
[rank60]:   File "/usr/lib/python3.10/contextlib.py", line 79, in inner
[rank60]:     return func(*args, **kwds)
[rank60]:   File "/usr/lib/python3.10/contextlib.py", line 79, in inner
[rank60]:     return func(*args, **kwds)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
[rank60]:     r = func(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 438, in compile_fx_inner
[rank60]:     compiled_graph = fx_codegen_and_compile(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 714, in fx_codegen_and_compile
[rank60]:     compiled_fn = graph.compile_to_fn()
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1307, in compile_to_fn
[rank60]:     return self.compile_to_module().call
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
[rank60]:     r = func(*args, **kwargs)
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1254, in compile_to_module
[rank60]:     mod = PyCodeCache.load_by_key_path(
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2160, in load_by_key_path
[rank60]:     exec(code, mod.__dict__, mod.__dict__)
[rank60]:   File "/tmp/torchinductor_ubuntu/32/c32zlm4gweniydec2oxyilmjnaayry5azbluy5dcokzx735gtcre.py", line 1806, in <module>
[rank60]:     async_compile.wait(globals())
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2715, in wait
[rank60]:     scope[key] = result.result()
[rank60]:   File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2522, in result
[rank60]:     self.future.result()
[rank60]:   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
[rank60]:     return self.__get_result()
[rank60]:   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[rank60]:     raise self._exception
[rank60]: torch._dynamo.exc.BackendCompilerFailed: backend='compile_fn' raised:
[rank60]: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/torchinductor_ubuntu/triton/4/5eba80077c5d5c6def3f812b8cccbc7d/triton_.cubin.tmp.pid_93828_100098'

[rank60]: Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


[rank60]: You can suppress this exception and fall back to eager by setting:
[rank60]:     import torch._dynamo
[rank60]:     torch._dynamo.config.suppress_errors = True

W0824 00:16:38.381000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 92391 closing signal SIGTERM
W0824 00:16:38.398000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 92392 closing signal SIGTERM
W0824 00:16:38.405000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 92393 closing signal SIGTERM
W0824 00:16:38.423000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 92394 closing signal SIGTERM
W0824 00:16:38.460000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 92396 closing signal SIGTERM
W0824 00:16:38.480000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 92397 closing signal SIGTERM
W0824 00:16:38.500000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 92398 closing signal SIGTERM
E0824 00:16:44.556000 140224365408704 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 4 (pid: 92395) of binary: /home/ubuntu/si/.venv/bin/python3
Traceback (most recent call last):
  File "/home/ubuntu/si/.venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ubuntu/si/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
exp/codebook_collapse_trace.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-08-24_00:16:38
  host      : foundry-8
  rank      : 60 (local_rank: 4)
  exitcode  : 1 (pid: 92395)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================`
```

### Versions

Collecting environment information...

PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-1023-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             192
On-line CPU(s) list:                0-191
Vendor ID:                          GenuineIntel
Model name:                         Intel Xeon Processor (SapphireRapids)
CPU family:                         6
Model:                              143
Thread(s) per core:                 1
Core(s) per socket:                 192
Socket(s):                          1
Stepping:                           4
BogoMIPS:                           4200.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk avx512_fp16 arch_capabilities
Virtualization:                     VT-x
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          6 MiB (192 instances)
L1i cache:                          6 MiB (192 instances)
L2 cache:                           768 MiB (192 instances)
L3 cache:                           16 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-191
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Unknown: No mitigations
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI Syscall hardening, KVM SW loop
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] numpy==2.0.1
[pip3] torch==2.3.1
[pip3] torchaudio==2.3.1
[pip3] torchvision==0.18.1
[pip3] triton==2.3.1
[conda] Could not collect

cc @ezyang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire