Repository: tensorflow
Issue: Failure to compile TF 2.15.0/2.16.1 (with Cuda support) using clang in Ubuntu 22.04
Labels: stat:awaiting tensorflower,type:build/install,subtype: ubuntu/linux,TF 2.15

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.15.0/2.16.1

### Custom code

No

### OS platform and distribution

Linux Ubuntu 16.04

### Mobile device

_No response_

### Python version

3.10.12

### Bazel version

6.1.0

### GCC/compiler version

Clang 14.0.0-1ubuntu1.1

### CUDA/cuDNN version

11.8-12.3

### GPU model and memory

Quadro RTX 6000 24GB

### Current behavior?

Compiling TF 2.15.0/2.16.1 in Ubuntu 22.04 using clang fails. Log is attached. 

### Standalone code to reproduce the issue

```shell
Using this configuration: 

./configure
You have bazel 6.1.0 installed. (6.5 for TF 2.16.1)
Please specify the location of python. [Default is /usr/bin/python3]: 


Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: 
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.

Found CUDA 11.8 in:
    /usr/local/cuda-11.8/targets/x86_64-linux/lib
    /usr/local/cuda-11.8/targets/x86_64-linux/include
Found cuDNN 8 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include
Found TensorRT 8.5.3 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include/x86_64-linux-gnu



Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 7.5,7.5]: 

Do you want to use clang as CUDA compiler? [Y/n]: 
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /usr/bin/clang]: 


You have Clang 14.0.0-1ubuntu1.1 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: -Wno-sign-compare


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: 
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
	--config=nogcp       	# Disable GCP support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished
```


### Relevant log output

```shell
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: The following configs were expanded more than once: [tensorrt, cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Reading 'startup' options from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=100
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3/dist-packages --python_path=/usr/bin/python3 --config=tensorrt --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.8 --action_env TF_CUDA_COMPUTE_CAPABILITIES=7.5,7.5 --action_env LD_LIBRARY_PATH=/usr/local/cuda-11.8/include:/usr/local/cuda-11.8/targets/x86_64-linux/lib: --config=cuda_clang --action_env CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-14/bin/clang --config=cuda_clang
INFO: Found applicable config definition build:short_logs in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:tensorrt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:cuda_clang in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --config=cuda --config=tensorrt --action_env=TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang --repo_env=TF_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_75,compute_80
INFO: Found applicable config definition build:cuda in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:tensorrt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:cuda_clang in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --config=cuda --config=tensorrt --action_env=TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang --repo_env=TF_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_75,compute_80
INFO: Found applicable config definition build:cuda in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:tensorrt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:opt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:cuda in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
DEBUG: /home/nicola/Software/tensorflow/gpu/tensorflow/tensorflow/tools/toolchains/python/python_repo.bzl:21:14: 
TF_PYTHON_VERSION variable was not set correctly, using default version. 3.10 Python
will be used.

To set Python version, run
export TF_PYTHON_VERSION=3.9
WARNING: The following configs were expanded more than once: [tensorrt, cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (699 packages loaded, 49725 targets configured).
INFO: Found 1 target...
ERROR: /home/nicola/.cache/bazel/_bazel_nicola/c53ed0be17816f9e0970b1ba234e403c/external/com_google_protobuf/BUILD.bazel:27:11: Compiling src/google/protobuf/arenaz_sampler.cc [for tool] failed: (Exit 1): clang failed: error executing command (from target @com_google_protobuf//:protobuf_lite) 
  (cd /home/nicola/.cache/bazel/_bazel_nicola/c53ed0be17816f9e0970b1ba234e403c/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-11.8/include:/usr/local/cuda-11.8/targets/x86_64-linux/lib: \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/snap/bin \
    PWD=/proc/self/cwd \
  /usr/lib/llvm-14/bin/clang -MD -MF bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf_lite/arenaz_sampler.d '-frandom-seed=bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf_lite/arenaz_sampler.o' '-DBAZEL_CURRENT_REPOSITORY="com_google_protobuf"' -iquote external/com_google_protobuf -iquote bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/src -fmerge-all-constants -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIE -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wno-invalid-partial-specialization -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=/usr/local/cuda-11.8' -g0 -w -Wno-sign-compare -g0 '-std=c++17' -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -c external/com_google_protobuf/src/google/protobuf/arenaz_sampler.cc -o bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf_lite/arenaz_sampler.o)
# Configuration: fccf36a8de6040562b99997447208dc7348eb48df9f6a445d4378df7e13a876c
# Execution platform: @local_execution_config_platform//:platform
In file included from external/com_google_protobuf/src/google/protobuf/arenaz_sampler.cc:31:
external/com_google_protobuf/src/google/protobuf/arenaz_sampler.h:34:10: fatal error: 'atomic' file not found
#include <atomic>
         ^~~~~~~~
1 error generated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 84.960s, Critical Path: 0.43s
INFO: 60 processes: 59 internal, 1 local.
FAILED: Build did NOT complete successfully
```