Repository: tensorflow
Issue: Bug: `Compiling upb/upb.c failed` due to `Clang` Version Mismatch in Tensorflow's Docker Build Image
Labels: stat:awaiting tensorflower,type:build/install,subtype: ubuntu/linux,TF 2.18

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf 2.18

### Custom code

No

### OS platform and distribution

Linux Ubuntu 22.04.3 LTS

### Mobile device

_No response_

### Python version

3.9

### Bazel version

6.5.0

### GCC/compiler version

11.4.0

### CUDA/cuDNN version

CUDA 12.5.1 / cuDNN 9.3.0

### GPU model and memory

NVIDIA RTX 3090 24GB DDR6

### Current behavior?

I expected the `tensorflow-gpu` wheel to compile successfully. However I received the error detailed below. I believe it's because the required `Clang` version (`18.1.8`) installed in the docker image is incorrect. [According to the documentation](https://www.tensorflow.org/install/source#gpu), the required version of `Clang` is `17.0.6` for `tf2.18`. I pulled [`tensorflow/build:2.18-python3.9`](https://hub.docker.com/layers/tensorflow/build/2.18-python3.9/images/sha256-843f48fe24727cdef4d76ae2724edc8385b2b2b44e3e4da0752e84b9ca142a81?context=explore) from DockerHub.

Do I need to manually roll back to `Clang 17.0.6` within the container, or should I be pulling a different image?
```bash
ERROR: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/upb/BUILD:57:11: Compiling upb/upb.c failed: (Exit 1): clang failed: error executing command (from target @upb//:upb) /usr/lib/llvm-18/bin/clang -MD -MF bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.pic.o' '-DBAZEL_CURRENT_REPOSITORY="upb"' -iquote ... (remaining 44 arguments skipped)
external/upb/upb/upb.c:192:10: error: defining a type within 'offsetof' is a Clang extension [-Werror,-Wgnu-offsetof-extensions]
  192 |   n &= ~(upb_alignof(upb_arena) - 1);
      |          ^~~~~~~~~~~~~~~~~~~~~~
external/upb/upb/upb.c:183:37: note: expanded from macro 'upb_alignof'
  183 | #define upb_alignof(type) offsetof (struct { char c; type member; }, member)
      |                                     ^~~~~~
/usr/lib/llvm-18/lib/clang/18/include/__stddef_offsetof.h:16:43: note: expanded from macro 'offsetof'
   16 | #define offsetof(t, d) __builtin_offsetof(t, d)
      |                                           ^
1 error generated.
Target //tensorflow/tools/pip_package:wheel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 524.930s, Critical Path: 5.67s
INFO: 1437 processes: 820 internal, 617 local.
FAILED: Build did NOT complete successfully
```

### Standalone code to reproduce the issue

```shell
# Pull the docker image to build tf.
docker pull tensorflow/build:2.18-python3.9

# Pull the tensorflow repo.
git pull https://github.com/tensorflow/tensorflow.git

# Change to the tensorflow directory.
cd tensorflow

# Checkout and switch to the r2.18 branch.
git fetch origin && git checkout -b r2.18 origin/r2.18

# Run a container to build tensorflow.
docker run \
        -it \
	--name tf-build \
	-h tf-build \
	-u root \
	-e HOST_PERMS="$(id -u):$(id -g)" \
	--runtime=nvidia \
	--gpus=all \
	--rm \
	--shm-size=2g \
	--ulimit memlock=-1 \
	--ulimit stack=67108864 \
	-v $PWD:/mnt \
	-w /mnt \
	tensorflow/build:2.18-python3.9 \
	bash

# From within the container, configure the build.
./configure

# Compile the wheel.
bazel build //tensorflow/tools/pip_package:wheel --repo_env=WHEEL_NAME=tensorflow --config=cuda --config=cuda_wheel --config=opt
```


### Relevant log output

```shell
tf-docker /mnt > ./configure
You have bazel 6.5.0 installed.
Please specify the location of python. [Default is /usr/bin/python3]:


Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.9/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Please specify the hermetic CUDA version you want to use or leave empty to use the default version. 12.5.1


Please specify the hermetic cuDNN version you want to use or leave empty to use the default version. 9.3.0


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 8.6


Please specify the local CUDA path you want to use or leave empty to use the default version.


Please specify the local CUDNN path you want to use or leave empty to use the default version.


Please specify the local NCCL path you want to use or leave empty to use the default version.


Do you want to use clang as CUDA compiler? [Y/n]: y
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /usr/lib/llvm-18/bin/clang]:


You have Clang 18.1.8 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: -march=native


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
	--config=nogcp       	# Disable GCP support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished
tf-docker /mnt > bazel build //tensorflow/tools/pip_package:wheel --repo_env=WHEEL_NAME=tensorflow --config=cuda --config=cuda_wheel --config=opt
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: The following configs were expanded more than once: [cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Reading 'startup' options from /mnt/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=173
INFO: Reading rc options for 'build' from /mnt/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /etc/bazel.bazelrc:
  'build' options: --action_env=DOCKER_CACHEBUSTER=1726964088092166976 --host_action_env=DOCKER_HOST_CACHEBUSTER=1726964088140283903
INFO: Reading rc options for 'build' from /mnt/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /mnt/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3/dist-packages --python_path=/usr/bin/python3 --action_env LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 --config=cuda_clang --action_env CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-18/bin/clang --config=cuda_clang
INFO: Found applicable config definition build:short_logs in file /mnt/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /mnt/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda_clang in file /mnt/.bazelrc: --config=cuda --@local_config_cuda//:cuda_compiler=clang --copt=-Qunused-arguments --repo_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES=sm_60,sm_70,sm_80,sm_89,compute_90 --host_linkopt=-fuse-ld=lld --host_linkopt=-lm --linkopt=-fuse-ld=lld --linkopt=-lm
INFO: Found applicable config definition build:cuda in file /mnt/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --repo_env=HERMETIC_CUDA_VERSION=12.5.1 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda in file /mnt/.tf_configure.bazelrc: --repo_env HERMETIC_CUDA_VERSION=12.5.1 --repo_env HERMETIC_CUDNN_VERSION=9.3.0 --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=8.6
INFO: Found applicable config definition build:cuda_clang in file /mnt/.bazelrc: --config=cuda --@local_config_cuda//:cuda_compiler=clang --copt=-Qunused-arguments --repo_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES=sm_60,sm_70,sm_80,sm_89,compute_90 --host_linkopt=-fuse-ld=lld --host_linkopt=-lm --linkopt=-fuse-ld=lld --linkopt=-lm
INFO: Found applicable config definition build:cuda in file /mnt/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --repo_env=HERMETIC_CUDA_VERSION=12.5.1 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda in file /mnt/.tf_configure.bazelrc: --repo_env HERMETIC_CUDA_VERSION=12.5.1 --repo_env HERMETIC_CUDNN_VERSION=9.3.0 --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=8.6
INFO: Found applicable config definition build:cuda in file /mnt/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --repo_env=HERMETIC_CUDA_VERSION=12.5.1 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda in file /mnt/.tf_configure.bazelrc: --repo_env HERMETIC_CUDA_VERSION=12.5.1 --repo_env HERMETIC_CUDNN_VERSION=9.3.0 --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=8.6
INFO: Found applicable config definition build:cuda_wheel in file /mnt/.bazelrc: --@local_config_cuda//cuda:include_cuda_libs=false
INFO: Found applicable config definition build:opt in file /mnt/.tf_configure.bazelrc: --copt=-march=native --host_copt=-march=native
INFO: Found applicable config definition build:linux in file /mnt/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /mnt/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_xla/third_party/py/python_repo.bzl:96:14:
HERMETIC_PYTHON_VERSION variable was not set correctly, using default version.
Python 3.9 will be used.
To select Python version, either set HERMETIC_PYTHON_VERSION env variable in
your shell:
  export HERMETIC_PYTHON_VERSION=3.12
OR pass it as an argument to bazel command directly or inside your .bazelrc
file:
  --repo_env=HERMETIC_PYTHON_VERSION=3.12
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_xla/third_party/py/python_repo.bzl:107:10: Using hermetic Python 3.9
WARNING: The following configs were expanded more than once: [cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_cupti/linux-x86_64/cuda_cupti-linux-x86_64-12.5.82-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvtx/linux-x86_64/cuda_nvtx-linux-x86_64-12.5.82-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/libcusolver/linux-x86_64/libcusolver-linux-x86_64-11.6.3.83-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/libnvjitlink/linux-x86_64/libnvjitlink-linux-x86_64-12.5.82-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/libcurand/linux-x86_64/libcurand-linux-x86_64-10.3.6.82-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/linux-x86_64/cuda_cudart-linux-x86_64-12.5.82-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cudnn/redist/cudnn/linux-x86_64/cudnn-linux-x86_64-9.3.0.75_cuda12-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/libcusparse/linux-x86_64/libcusparse-linux-x86_64-12.5.1.3-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/linux-x86_64/cuda_nvml_dev-linux-x86_64-12.5.82-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/libcublas/linux-x86_64/libcublas-linux-x86_64-12.5.3.2-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/libcufft/linux-x86_64/libcufft-linux-x86_64-11.2.3.61-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/linux-x86_64/cuda_nvcc-linux-x86_64-12.5.82-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/linux-x86_64/cuda_cccl-linux-x86_64-12.5.39-archive.tar.xz
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/nccl/hermetic/nccl_redist_init_repository.bzl:73:10: Downloading and extracting https://files.pythonhosted.org/packages/df/99/12cd266d6233f47d00daf3a72739872bdc10267d0383508b0b9c84a18bb6/nvidia_nccl_cu12-2.21.5-py3-none-manylinux2014_x86_64.whl
DEBUG: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/local_tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl:269:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvprune/linux-x86_64/cuda_nvprune-linux-x86_64-12.5.82-archive.tar.xz
INFO: Analyzed target //tensorflow/tools/pip_package:wheel (759 packages loaded, 56246 targets configured).
INFO: Found 1 target...
ERROR: /root/.cache/bazel/_bazel_root/39de0dbcfb68c8735bd088c62fa061a4/external/upb/BUILD:57:11: Compiling upb/upb.c failed: (Exit 1): clang failed: error executing command (from target @upb//:upb) /usr/lib/llvm-18/bin/clang -MD -MF bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.pic.o' '-DBAZEL_CURRENT_REPOSITORY="upb"' -iquote ... (remaining 44 arguments skipped)
external/upb/upb/upb.c:192:10: error: defining a type within 'offsetof' is a Clang extension [-Werror,-Wgnu-offsetof-extensions]
  192 |   n &= ~(upb_alignof(upb_arena) - 1);
      |          ^~~~~~~~~~~~~~~~~~~~~~
external/upb/upb/upb.c:183:37: note: expanded from macro 'upb_alignof'
  183 | #define upb_alignof(type) offsetof (struct { char c; type member; }, member)
      |                                     ^~~~~~
/usr/lib/llvm-18/lib/clang/18/include/__stddef_offsetof.h:16:43: note: expanded from macro 'offsetof'
   16 | #define offsetof(t, d) __builtin_offsetof(t, d)
      |                                           ^
1 error generated.
Target //tensorflow/tools/pip_package:wheel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 524.930s, Critical Path: 5.67s
INFO: 1437 processes: 820 internal, 617 local.
FAILED: Build did NOT complete successfully
```