Repository: tensorflow
Issue: Cannot disable XLA and/or JIT
Labels: stat:awaiting tensorflower,type:bug,comp:xla,TF 2.15

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.15.0

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 22.04.3

### Mobile device

_No response_

### Python version

3.10.12 (Unrelated)

### Bazel version

6.3.2

### GCC/compiler version

Clang 18

### CUDA/cuDNN version

12.2

### GPU model and memory

A100 and A10

### Current behavior?

After upgrading from `2.12.1` to `2.15.0` we observed a lot **new** logs when starting a **C++** service using Tensorflow, that it calls `ptxas` to compile some generated PTX, a snippet is attached below.

We tried a bunch of options in TF_XLA_FLAGS such as `--tf_xla_auto_jit=-1`, `--tf_mlir_enable_mlir_bridge=0`, `--tf_xla_cpu_global_jit=0`, `--tf_xla_clustering_fuel=0`, etc. but it still compiles those ops in the pass of `CreateGpuKernelToBlobPass` anyway.

I wonder if anything related to XLA / JIT changed between `2.12.1` and `2.15.0`? And is there a way to simply disable all XLA and JIT?

BTW: Both Tensorflow versions were built with XLA and CUDA supports, with `TF_CUDA_COMPUTE_CAPABILITIES="7.0,7.5,8.0,8.6,9.0"`.

### Standalone code to reproduce the issue

```shell
It is a C++ service which loads model and do inference, we do not currently have a minimal example at this time. However, we can try providing detailed context as much as possible if we can narrow the scenario down to a considerable small portion of the service.
```


### Relevant log output

```shell
2024-01-29 15:14:43.203921: I external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:565] Compile module main_kernel_0 time: 7.33 ms (cumulative: 82.4 ms, max: 9.23 ms, #called: 15)2024-01-29 15:14:43.204021: I external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:565] Compile module main_kernel time: 7.43 ms (cumulative: 89.9 ms, max: 9.23 ms, #called: 16)2024-01-29 15:14:43.204084: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:263] ptx written to: /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-d460ae06-1673-61010665dbba92024-01-29 15:14:43.204112: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:295] /usr/local/cuda-12.2/bin/ptxas /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-d460ae06-1673-61010665dbba9 -o /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-d460ae06-1673-61010665dbc05 -arch=sm_86 --warn-on-spills -v 2024-01-29 15:14:43.204182: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:263] ptx written to: /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-b79066f7-1673-61010665dbc0d2024-01-29 15:14:43.204209: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:295] /usr/local/cuda-12.2/bin/ptxas /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-b79066f7-1673-61010665dbc0d -o /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-b79066f7-1673-61010665dbc66 -arch=sm_86 --warn-on-spills -v2024-01-29 15:14:43.204334: I external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:565] Compile module main_kernel_1 time: 7.71 ms (cumulative: 97.6 ms, max: 9.23 ms, #called: 17)2024-01-29 15:14:43.204499: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:263] ptx written to: /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-7941e65c-1673-61010665dbd48
2024-01-29 15:14:43.204528: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:295] /usr/local/cuda-12.2/bin/ptxas /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-7941e65c-1673-61010665dbd48 -o /tmp/tempfile-jscs02-ai-deep-dev-4a10-01-7941e65c-1673-61010665dbda4 -arch=sm_86 --warn-on-spills -v2024-01-29 15:14:43.236003: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'ptxas info    : Function properties for main_kernel    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loadsptxas info    : Used 12 registers, 452 bytes cmem[0]
2024-01-29 15:14:43.236589: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'ptxas info    : Function properties for main_kernel    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 14 registers, 476 bytes cmem[0]

2024-01-29 15:14:43.237263: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'
ptxas info    : Function properties for main_kernel
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 14 registers, 508 bytes cmem[0]

2024-01-29 15:14:43.238685: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'
ptxas info    : Function properties for main_kernel
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 14 registers, 532 bytes cmem[0]

2024-01-29 15:14:43.240855: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'
ptxas info    : Function properties for main_kernel
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 15 registers, 564 bytes cmem[0]

2024-01-29 15:14:43.241720: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'
ptxas info    : Function properties for main_kernel
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 18 registers, 444 bytes cmem[0]

2024-01-29 15:14:43.241731: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'
ptxas info    : Function properties for main_kernel
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 16 registers, 444 bytes cmem[0]

2024-01-29 15:14:43.242423: I external/local_xla/xla/stream_executor/gpu/asm_compiler.cc:333] ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function 'main_kernel' for 'sm_86'
ptxas info    : Function properties for main_kernel
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 18 registers, 452 bytes cmem[0]

libunwind: __unw_add_dynamic_fde: bad fde: FDE is really a CIE
2024-01-29 15:14:43.416923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:812] GpuDevice::ComputeHelper scheduled dense/clip_by_value_65 op Maximum on GPU 0 stream[0]
```