Repository: tensorflow
Issue: Compile tensorflow with static cuda libs
Labels: stat:awaiting tensorflower,type:build/install,subtype:centos,TF 2.11

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Build/Install

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.11.0

### Custom Code

No

### OS Platform and Distribution

Centos &

### Mobile device

_No response_

### Python version

Python-3.9.16

### Bazel version

bazel-5.3.0

### GCC/Compiler version

9.3.1 

### CUDA/cuDNN version

11.2

### GPU model and memory

Quadro RTX 4000

### Current Behaviour?

I am trying to link TF with cuda static libs (cublas, cufft, cusparse etc). I have made some changes in the following files:
1. tensorflow/lite/toco/BUILD b/tensorflow/lite/toco/BUILD
2. tensorflow/tensorflow.bzl
3. tensorflow/compiler/mlir/tools/kernel_gen/BUILD
4. third_party/gpus/cuda_configure.bzl
In the first 3 files I have added inkopts = if_not_windows(["-lm", "-Wl,-ldl"]) + lrt_if_needed() + ["-L/usr/local/cuda/lib64", "-L/usr/local/cuda/extras/CUPTI/lib64", "-lcuda", "-lcudart", "-lcublas","-lcublasLt","-lculibos","-lcufft", "-lcudnn", "-lcurand", "-lcupti", "-lcusolver", "-lcusparse"],
In the fourth I have changed static = False to static = True. 

### Standalone code to reproduce the issue

```shell
https://drive.google.com/file/d/1vGyJD-1OWWlE2tYu4qbglMklSfep6x4T/view?usp=share_link
```


### Relevant log output

```shell
bazel build //tensorflow/tools/pip_package:build_pip_package 
$TEST_TMPDIR defined: output root default is '/tmp/bazel_manospavl' and max_idle_secs default is '15'.
$TEST_TMPDIR defined: output root default is '/tmp/bazel_manospavl' and max_idle_secs default is '15'.
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=104
INFO: Reading rc options for 'build' from /tmp/manospavl/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /tmp/manospavl/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false
INFO: Reading rc options for 'build' from /tmp/manospavl/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/local/bin/python3.9 --action_env PYTHON_LIB_PATH=/opt/rh/llvm-toolset-7/root/usr/lib/python2.7/site-packages --python_path=/usr/local/bin/python3.9 --action_env PYTHONPATH=/opt/rh/devtoolset-8/root/usr/lib64/python2.7/site-packages:/opt/rh/devtoolset-8/root/usr/lib/python2.7/site-packages:/opt/rh/llvm-toolset-7/root/usr/lib/python2.7/site-packages --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.2 --action_env TF_CUDA_COMPUTE_CAPABILITIES=7.5 --action_env LD_LIBRARY_PATH=/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/archive/users/gxanth/gcc-12.1.0/lib64:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/linux64/lib:/opt/altera/intelFPGA/20.3/hld/host/linux64/lib:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/tests/extlibs/lib:/usr/local/cuda-11.2/extras/CUPTI/lib64:/usr/local/cuda-11.2/lib64:/usr/lib64/atlas:/usr/local/lib:/lib64:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/archive/users/gxanth/gcc-12.1.0/lib64:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/linux64/lib:/opt/altera/intelFPGA/20.3/hld/host/linux64/lib:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/tests/extlibs/lib:/usr/local/cuda-11.2/extras/CUPTI/lib64:/usr/local/cuda-11.2/lib64:/usr/lib64/atlas:/usr/local/lib:/lib64:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/archive/users/gxanth/gcc-12.1.0/lib64:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/linux64/lib:/opt/altera/intelFPGA/20.3/hld/host/linux64/lib:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/tests/extlibs/lib:/usr/local/cuda-11.2/lib64:/usr/lib64/atlas:/usr/local/lib:/lib64:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/archive/users/gxanth/gcc-12.1.0/lib64:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/linux64/lib:/opt/altera/intelFPGA/20.3/hld/host/linux64/lib:/opt/altera/intelFPGA/20.3/hld/board/de5a_net_ddr4/tests/extlibs/lib:/usr/local/cuda-11.2/lib64:/usr/lib64/atlas:/usr/local/lib:/lib64:/opt/rh/llvm-toolset-7/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/usr/mpi/gcc/openmpi-4.0.3rc4/lib64 --action_env GCC_HOST_COMPILER_PATH=/opt/rh/devtoolset-9/root/usr/bin/gcc --config=cuda
INFO: Reading rc options for 'build' from /tmp/manospavl/tensorflow/.bazelrc:
  'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /tmp/manospavl/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /tmp/manospavl/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file /tmp/manospavl/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /tmp/manospavl/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-unknown-warning --copt=-Wno-array-parameter --copt=-Wno-stringop-overflow --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /tmp/manospavl/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (558 packages loaded, 38371 targets configured).
INFO: Found 1 target...
ERROR: /tmp/manospavl/tensorflow/tensorflow/BUILD:1426:19: Executing genrule //tensorflow:tf_python_api_gen_v2 failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)
2023-04-20 16:53:52.189168: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Traceback (most recent call last):
  File "/tmp/bazel_manospavl/_bazel_manospavl/3b86cb968a28ba76b0db07c39e753f80/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 62, in <module>
    from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: /tmp/bazel_manospavl/_bazel_manospavl/3b86cb968a28ba76b0db07c39e753f80/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/../libtensorflow_framework.so.2: undefined symbol: cublasLtMatmulDescDestroy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/bazel_manospavl/_bazel_manospavl/3b86cb968a28ba76b0db07c39e753f80/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 22, in <module>
    from tensorflow.python.tools.api.generator import doc_srcs
  File "/tmp/bazel_manospavl/_bazel_manospavl/3b86cb968a28ba76b0db07c39e753f80/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 36, in <module>
    from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
  File "/tmp/bazel_manospavl/_bazel_manospavl/3b86cb968a28ba76b0db07c39e753f80/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 77, in <module>
    raise ImportError(
ImportError: Traceback (most recent call last):
  File "/tmp/bazel_manospavl/_bazel_manospavl/3b86cb968a28ba76b0db07c39e753f80/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 62, in <module>
    from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: /tmp/bazel_manospavl/_bazel_manospavl/3b86cb968a28ba76b0db07c39e753f80/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/../libtensorflow_framework.so.2: undefined symbol: cublasLtMatmulDescDestroy


Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors for some common causes and solutions.
If you need help, create an issue at https://github.com/tensorflow/tensorflow/issues and include the entire stack trace above this error message.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /tmp/manospavl/tensorflow/tensorflow/python/tools/BUILD:98:10 Middleman _middlemen/tensorflow_Spython_Stools_Simport_Upb_Uto_Utensorboard-runfiles failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)
INFO: Elapsed time: 29.342s, Critical Path: 6.86s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
```
</details>