Repository: tensorflow
Issue: Unable to retrieve BLAS factory in Tensorflow 2.16.1
Labels: stat:awaiting tensorflower,type:build/install,subtype: ubuntu/linux,TF 2.16

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.16.1

### Custom code

Yes

### OS platform and distribution

Linux, Ubuntu 20.04

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

cuda:12.2.2-cudnn8

### GPU model and memory

NVIDIA T4

### Current behavior?

We tried to upgrade our current 2.13.0 `libtensorflow` binary in our Golang application to the latest 2.16.1. However, the new `libtensorflow` does not work and starts to produce errors during the model inference:

```
2024-03-14 20:31:02.259695: E external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:837] Unable to retrieve BLAS factory: No suitable BLAS plugin registered. Have you linked in a BLAS-providing plugin?
```

The only change we made is to replace the Google-distributed `libtensorflow` 2.13.0  with `libtensorflow` 2.16.1 from [here](https://github.com/tensorflow/tensorflow/issues/63384#issuecomment-1991794920) and change the base docker image:

```diff
- ARG CUDA_IMAGE=nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04
+ ARG CUDA_IMAGE=nvcr.io/nvidia/cuda:12.2.2-cudnn8-devel-ubuntu20.04
```

### Standalone code to reproduce the issue

```shell
-
```


### Relevant log output

```shell
2024-03-14 20:31:02.259695: E external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:837] Unable to retrieve BLAS factory: No suitable BLAS plugin registered. Have you linked in a BLAS-providing plugin?
2024-03-14 20:31:02.259770: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INTERNAL: No blas support for stream
	 [[{{function_node __inference__wrapped_model_57676}}{{node model/cross_0/dense/MatMul}}]]
2024-03-14 20:31:02.259787: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INTERNAL: No blas support for stream
	 [[{{function_node __inference__wrapped_model_57676}}{{node model/cross_0/dense/MatMul}}]]
	 [[StatefulPartitionedCall_41/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall_2/StatefulPartitionedCall/model/mean/Sigmoid/_55]]
2024-03-14 20:31:02.259876: I tensorflow/core/framework/local_rendezvous.cc:422] Local rendezvous recv item cancelled. Key hash: 14129393403982148508
2024-03-14 20:31:02.259917: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: ABORTED: Stopping remaining executors.
{"level":"error","model_name":"cpi-model","model_version":20240314181330,"error":"failed to infer request during warmup: failed to run: failed to run session: 2 root error(s) found.\n  (0) INTERNAL: No blas support for stream\n\t [[{{function_node __inference__wrapped_model_57676}}{{node model/cross_0/dense/MatMul}}]]\n\t [[StatefulPartitionedCall_41/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall_2/StatefulPartitionedCall/model/mean/Sigmoid/_55]]\n  (1) INTERNAL: No blas support for stream\n\t [[{{function_node __inference__wrapped_model_57676}}{{node model/cross_0/dense/MatMul}}]]\n0 successful operations.\n0 derived errors ignored.","time":"2024-03-14T20:31:02Z","message":"failed to warmup model"}
{"level":"info","model_name":"cpi-model","model_version":20240314181330,"time":"2024-03-14T20:31:02Z","message":"model warmup finished"}
```