Repository: tensorflow
Issue: [TF-2.18] Protoc-related Segmentation Fault on GH200 when Building from Source 
Labels: stat:awaiting tensorflower,type:build/install,subtype: ubuntu/linux,TF 2.18

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

TF 2.18

### Custom code

No

### OS platform and distribution

Rocky Linux 9.3 

### Mobile device

_No response_

### Python version

3.9

### Bazel version

6.5.0

### GCC/compiler version

17.0.6 (LLVM)

### CUDA/cuDNN version

CUDA-12.4.1/cuDNN-8.9.6

### GPU model and memory

Grace Hopper GH200 

### Current behavior?

Since there is no pip package for GH200, and Amazon only provides CPU wheel for `aarch64`, we are building `tf-2.18` from src. 

```
$ git branch 
master 
* r2.18
```
Both LLVM and Bazel versions are according to recommendation
```
$ clang --version 
clang version 17.0.6
Target: aarch64-unknown-linux-gnu
Thread model: posix
``` 
``` 
$ bazel --version 
bazel 6.5.0- (@non-git)
```

Bazel build failed with protoc-related segmentation fault. 

### Standalone code to reproduce the issue

```shell
./configure 
Please specify the location of python. [Default is /scratch/optpar01/.conda/tf2_src/bin/python3]: 

Found possible Python library paths:
  /scratch/optpar01/.conda/tf2_src/lib/python3.9/site-packages
Please input the desired Python library path to use.  Default is [/scratch/optpar01/.conda/tf2_src/lib/python3.9/site-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: 
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Please specify the hermetic CUDA version you want to use or leave empty to use the default version. 12.4.1

Please specify the hermetic cuDNN version you want to use or leave empty to use the default version. 8.9.6

Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 7.0,8.0,9.0

Please specify the local CUDA path you want to use or leave empty to use the default version. 

Please specify the local CUDNN path you want to use or leave empty to use the default version. 

Please specify the local NCCL path you want to use or leave empty to use the default version. 


Do you want to use clang as CUDA compiler? [Y/n]: 
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/llvm-17.0.6-52ysnwfyyhc6tchulc5wlen7h6yc3fju/bin/clang]: 

You have Clang 17.0.6 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: --copt=-Wno-error=unused-command-line-argument

Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: 
Not configuring the WORKSPACE for Android builds.
```


### Relevant log output

```shell
$ bazel build //tensorflow/tools/pip_package:wheel --repo_env=WHEEL_NAME=tensorflow --config=cuda --config=cuda_wheel --verbose_failures
ERROR: /scratch/optpar01/.cache/_bazel_optpar01/1b4d76cef27e0cbebd5791dfb99ed3a8/external/local_tsl/tsl/profiler/protobuf/BUILD:36:17: Action external/local_tsl/tsl/profiler/protobuf/profiler_service.grpc.pb.h failed: (Segmentation fault): protoc failed: error executing command (from target @local_tsl//tsl/profiler/protobuf:_profiler_service_cc_grpc_proto_grpc_codegen) 
  (cd /scratch/optpar01/.cache/_bazel_optpar01/1b4d76cef27e0cbebd5791dfb99ed3a8/execroot/org_tensorflow && \
  exec env - \
    CLANG_CUDA_COMPILER_PATH=/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/llvm-17.0.6-52ysnwfyyhc6tchulc5wlen7h6yc3fju/bin/clang-17 \
    LD_LIBRARY_PATH=/apps/ARM_node/ARM_applications/python/3.12.4/lib:/apps/ARM_node/ARM_applications/python/3.12.4/lib/python3.12:/usr/lib64:/usr/lib64/slurm:/usr/local/lib \
    PATH=/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/bazel-6.5.0-es6l5xjprjwauy22iuwlxoj4uztdcxsu/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/zip-3.0-pxcrfxbv4w274tczkoz7t4a54rv3r5de/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/clang-17.0.6/openjdk-11.0.23_9-s6cgcclfuzjttl5cdh3evsp3rrmuuvo6/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/bazel-6.5.0-es6l5xjprjwauy22iuwlxoj4uztdcxsu/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/zip-3.0-pxcrfxbv4w274tczkoz7t4a54rv3r5de/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/bzip2-1.0.8-jvd2b77w6emrwhj2g7mquhr7oteeb2uk/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/clang-17.0.6/openjdk-11.0.23_9-s6cgcclfuzjttl5cdh3evsp3rrmuuvo6/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/llvm-17.0.6-52ysnwfyyhc6tchulc5wlen7h6yc3fju/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/unzip-6.0-fpkt734eot7dakwydys5mgo5yomxwac3/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/curl-8.7.1-ca7uqkpgoorf7yzvajfyab2dfty7eitr/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/llvm-17.0.6-52ysnwfyyhc6tchulc5wlen7h6yc3fju/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/swig-4.1.1-gpwxntk2voqulme22h55ngp5ztgzyg6x/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/pcre2-10.43-bzcntkq3ul5umao7fcke5w4vrlyssbk7/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/lua-5.3.6-lec5qldurljnfjg4jo2fvsekfughzwru/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/unzip-6.0-fpkt734eot7dakwydys5mgo5yomxwac3/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/readline-8.2-47up4u4jjwf4io45zeioeyrh3wwwg4tg/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/curl-8.7.1-ca7uqkpgoorf7yzvajfyab2dfty7eitr/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/openssl-3.3.1-7uxchckf5qrcojfgntn77hvtoxjjkrq6/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/nghttp2-1.62.0-x5y5gbq2qybl4iiqvjv7bflc2nlx35zu/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/hwloc-2.9.3-onlxonz6ibms2b6fxhk7wqcdpo5l5myn/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/ncurses-6.5-fvbf5zfdj2bcwylse6cv6mztrwadhgyv/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-13.3.0/binutils-2.43.1-t3kxibwizkzbb7e3boxm3qkpngjuzzns/bin:/scratch/optpar01/spack/opt/spack/linux-rocky9-neoverse_v2/gcc-11.4.1/zstd-1.5.6-miqeogpdx7z7lcovzh3bsfwr2tzihxw2/bin:/scratch/optpar01/.conda/tf2_src/bin:/apps/ARM_node/ARM_applications/Miniconda/24.5.0/condabin:/apps/ARM_node/ARM_applications/python/3.12.4/bin:/home01/optpar01/.cargo/bin:/scratch/optpar01/spack/bin:/home01/optpar01/apps/build/gv/3.7.4/bin:/apps/applications/htop/3.0.5:/apps/applications/nvtop/1.1.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home01/optpar01/bin \
    PYTHON_BIN_PATH=/scratch/optpar01/.conda/tf2_src/bin/python3 \
    PYTHON_LIB_PATH=/scratch/optpar01/.conda/tf2_src/lib/python3.9/site-packages \
    SPACK_LOADED_HASHES=es6l5xjprjwauy22iuwlxoj4uztdcxsu:52ysnwfyyhc6tchulc5wlen7h6yc3fju \
    SPACK_PYTHON=/usr/bin/python3 \
    SPACK_ROOT=/scratch/optpar01/spack \
    TF2_BEHAVIOR=1 \
  bazel-out/aarch64-opt-exec-50AE0418/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/aarch64-opt-exec-50AE0418/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=services_namespace=grpc,generate_mock_code=true:bazel-out/aarch64-opt/bin/external/local_tsl' '--proto_path=external/local_tsl' '--proto_path=external/local_tsl' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/any_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/api_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/source_context_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/type_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/compiler_plugin_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/duration_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/empty_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/field_mask_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/struct_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto' '--proto_path=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto' '--proto_path=external/local_tsl' '--proto_path=bazel-out/aarch64-opt/bin/external/local_tsl/external/local_tsl' external/local_tsl/tsl/profiler/protobuf/profiler_service.proto)
```
```