Repository: pytorch
Issue: [H100][CUDA] test_runtime_checks_fp8_abi_compatible_cuda fails with `Unsupported conversion from f8E5M2FNUZ to f16`
Labels: triaged,oncall: pt2,module: inductor,module: float8

### 🐛 Describe the bug

test_runtime_checks_fp8_abi_compatible_cuda fails with `Unsupported conversion from f8E5M2FNUZ to f16`

Reproduce:
```python
root@b7d054b7e5d9:/opt/pytorch/pytorch# python test/inductor/test_aot_inductor.py -v -k AOTInductorTestABICompatibleCuda.test_runtime_checks_fp8_abi_compatible_cuda
test_runtime_checks_fp8_abi_compatible_cuda (__main__.AOTInductorTestABICompatibleCuda) ... Unsupported conversion from f8E5M2FNUZ to f16
LLVM ERROR: Unsupported rounding mode for conversion.
stats [('calls_captured', 7), ('unique_graphs', 1)]
inductor []
ERROR

======================================================================
ERROR: test_runtime_checks_fp8_abi_compatible_cuda (__main__.AOTInductorTestABICompatibleCuda)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 2756, in wrapper
    method(*args, **kwargs)
  File "/opt/pytorch/pytorch/test/inductor/test_torchinductor.py", line 10233, in new_test
    return value(self)
  File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor.py", line 2526, in test_runtime_checks_fp8
    self.check_model(
  File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor.py", line 108, in check_model
    actual = AOTIRunnerUtil.run(
  File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor_utils.py", line 100, in run
    so_path = AOTIRunnerUtil.compile(
  File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor_utils.py", line 53, in compile
    so_path = torch._inductor.aot_compile(gm, example_inputs, options=options)  # type: ignore[arg-type]
  File "/opt/pytorch/pytorch/torch/_inductor/__init__.py", line 104, in aot_compile
    return compile_fx_aot(
  File "/opt/pytorch/pytorch/torch/_inductor/compile_fx.py", line 1143, in compile_fx_aot
    compiled_lib_path = compile_fx(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/pytorch/pytorch/torch/_inductor/compile_fx.py", line 1249, in compile_fx
    return compile_fx(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/pytorch/pytorch/torch/_inductor/compile_fx.py", line 1283, in compile_fx
    return compile_fx(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/pytorch/pytorch/torch/_inductor/compile_fx.py", line 1487, in compile_fx
    return inference_compiler(unlifted_gm, example_inputs_)
  File "/opt/pytorch/pytorch/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
  File "/opt/pytorch/pytorch/torch/_inductor/compile_fx.py", line 1397, in fw_compiler_base
    return inner_compile(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/pytorch/pytorch/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
  File "/opt/pytorch/pytorch/torch/_inductor/debug.py", line 304, in inner
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/pytorch/pytorch/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
  File "/opt/pytorch/pytorch/torch/_inductor/compile_fx.py", line 522, in compile_fx_inner
    compiled_graph = fx_codegen_and_compile(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/pytorch/pytorch/torch/_inductor/compile_fx.py", line 818, in fx_codegen_and_compile
    compiled_fn = graph.compile_to_fn()
  File "/opt/pytorch/pytorch/torch/_inductor/graph.py", line 1716, in compile_to_fn
    code, linemap = self.codegen_with_cpp_wrapper()
  File "/opt/pytorch/pytorch/torch/_inductor/graph.py", line 1548, in codegen_with_cpp_wrapper
    compiled = self.compile_to_module().call
  File "/opt/pytorch/pytorch/torch/_dynamo/utils.py", line 210, in time_wrapper
    r = func(*args, **kwargs)
  File "/opt/pytorch/pytorch/torch/_inductor/graph.py", line 1682, in compile_to_module
    mod = PyCodeCache.load_by_key_path(
  File "/opt/pytorch/pytorch/torch/_inductor/codecache.py", line 2584, in load_by_key_path
    mod = _reload_python_module(key, path)
  File "/opt/pytorch/pytorch/torch/_inductor/runtime/compile_tasks.py", line 44, in _reload_python_module
    exec(code, mod.__dict__, mod.__dict__)
  File "/tmp/tmptz6yfzoa/7m/c7mq2cpk6kbgjhhn5xknlsb72uzvfuha7v5eeqh4dot6hgnko6zn.py", line 81, in <module>
  File "/opt/pytorch/pytorch/torch/_inductor/codecache.py", line 3145, in wait
    scope[key] = result.result()
  File "/opt/pytorch/pytorch/torch/_inductor/codecache.py", line 2947, in result
    self.future.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

To execute this test, run the following from the base repo dir:
     python test/inductor/test_aot_inductor.py -k AOTInductorTestABICompatibleCuda.test_runtime_checks_fp8_abi_compatible_cuda

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
Ran 1 test in 3.494s

FAILED (errors=1)
```

### Versions

H100, pytorch build from source

- torch_commit: https://github.com/pytorch/pytorch/commit/53f73cdeb6fe53155f91064ca15b722e80dec2f3
- openai_triton_commit (from .ci pin): https://github.com/openai/triton/commit/45fff310c891f5a92d55445adf8cc9d29df5841e

For this test, we had a different error message before https://github.com/pytorch/pytorch/pull/124329 and the PR fixed that. After the PR, we saw the current error message.

cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @yanbing-j @vkuzo @albanD @eqy @nWEIdia @malfet