Repository: tensorflow
Issue: Higher Memory Usage with model.predict in Recent TF Versions (TF 2.10, 2.11 etc)
Labels: comp:runtime,comp:keras,type:performance,TF 2.11

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Source

binary 
- we use `pip install` to reproduce the issue although we use `poetry` in production. 

### Tensorflow Version

2.10 and 2.11 and maybe others after 2.8.2 all have this issue

### Custom Code

No (can reproduce the issue with simple code below)

### OS Platform and Distribution

```
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
```x86_64```

### Mobile device

_No response_

### Python version

3.8.15

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

We use TF to serve multiple models in production and recently we tried updating from TF 2.8.2 to TF 2.10. We noticed on many of our services/workers there was a memory increase/leak with some services even crashing. We had to revert back to 2.8.2. I also see the issue in TF 2.11. With this deploy in particular the only thing we upgraded was Tensorflow. No application code changes.

We monitor the memory with DataDog and all our host graphs showed major spikes. These are mostly `tf.keras` models. I thought originally it was a `ddtrace` issue because from TF 2.9 and up `model.predict` started printing progress bars and I thought all those were getting collected in  DataDog traces. But I set `verbose=0` on all model.predict calls and turned off ddtrace an the issue was still there.

Some screenshots from two different services

<img width="1876" alt="Screenshot 2022-11-24 at 5 17 04 PM" src="https://user-images.githubusercontent.com/16509365/203864156-2faece9b-98b1-4cb0-bb9f-6c86faed633c.png">
<img width="1872" alt="Screenshot 2022-11-24 at 5 17 42 PM" src="https://user-images.githubusercontent.com/16509365/203864162-8ddfca83-add9-4040-93d9-eee2bb59501f.png">



### Standalone code to reproduce the issue

Run this code on a machine with 8 cores and 16GB of ram for example.
Here is also a [colab](https://colab.research.google.com/drive/1fZ5I72KMPuoJf9Gqxd6XBo-294MM9jP5#scrollTo=RhJF-f7atWI6) with the same code as below although not sure you can replicate the memory leak issue in colab. I am replicating the issue on a standard aws ec2 instance `c5.2xlarge`. Running in a single ipython session for an hour or two will show the leak. You can speed it up by running 5 ipython sessions at once in 5 shells.

**Minimal Dockerfile**
```
FROM python:3.8-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
        wget \
        htop \
        vim

COPY . /app

RUN pip install tensorflow #  pip install tensorflow==2.8.2 does not have MEM leak
RUN pip install ipython
RUN pip install Pillow
EXPOSE 5000
```

```
docker build . -t tf
docker run -it tf /bin/bash
```

Now in the docker container
```
ipython
```

```
# download an image
!wget --output-document=my_image.jpg https://www.wwf.org.uk/sites/default/files/styles/social_share_image/public/2018-10/Large_WW1113482.jpg?itok=Bluh496C
```

```
import tensorflow
from tensorflow import keras
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.resnet50 import preprocess_input
import numpy as np

model = ResNet50(weights='imagenet', include_top=False, pooling='avg')
img_path = 'my_image.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
for _ in range(100000):
    embedding = model.predict(x)
```


Look at the memory usage with `htop` with TF 2.8.2 and the memory will not grow.
Now repeat the same thing with TF 2.10, 2.11 etc (maybe even 2.9 But I never checked it)
and you will see much higher memory usage and a linear growth.

### Relevant log output

#### TF 2.8.2

Here is the `pip freeze` inside the Docker ENV for TF 2.8.2 version
```
absl-py==1.3.0
asttokens==2.1.0
astunparse==1.6.3
backcall==0.2.0
cachetools==5.2.0
certifi==2022.9.24
charset-normalizer==2.1.1
decorator==5.1.1
executing==1.2.0
flatbuffers==22.11.23
gast==0.5.3
google-auth==2.14.1
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.50.0
h5py==3.7.0
idna==3.4
importlib-metadata==5.1.0
ipython==8.6.0
jedi==0.18.2
keras==2.8.0
Keras-Preprocessing==1.1.2
libclang==14.0.6
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
numpy==1.23.5
oauthlib==3.2.2
opt-einsum==3.3.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
prompt-toolkit==3.0.33
protobuf==3.19.6
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.13.0
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
six==1.16.0
stack-data==0.6.1
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.2
tensorflow-estimator==2.8.0
tensorflow-io-gcs-filesystem==0.28.0
termcolor==2.1.1
traitlets==5.5.0
typing_extensions==4.4.0
urllib3==1.26.13
wcwidth==0.2.5
Werkzeug==2.2.2
wrapt==1.14.1
zipp==3.10.0
```
Memory Stays at Like 1GB
<img width="1464" alt="Screenshot 2022-11-25 at 11 14 13 AM" src="https://user-images.githubusercontent.com/16509365/204013870-260bf5b0-27e9-48cd-bd3a-400e0fe68e0c.png">

Here is the log output when you `import tensorflow as tf` on TF 2.8.2

```
2022-11-25 15:44:46.912225: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-11-25 15:44:46.912258: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
```


#### TF 2.11
And here is the `pip freeze` for a higher version of TF 2.11

```
absl-py==1.3.0
asttokens==2.1.0
astunparse==1.6.3
backcall==0.2.0
cachetools==5.2.0
certifi==2022.9.24
charset-normalizer==2.1.1
decorator==5.1.1
executing==1.2.0
flatbuffers==22.11.23
gast==0.4.0
google-auth==2.14.1
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.50.0
h5py==3.7.0
idna==3.4
importlib-metadata==5.1.0
ipython==8.6.0
jedi==0.18.2
keras==2.11.0
libclang==14.0.6
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
numpy==1.23.5
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==21.3
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
prompt-toolkit==3.0.33
protobuf==3.19.6
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.13.0
pyparsing==3.0.9
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
six==1.16.0
stack-data==0.6.1
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.28.0
termcolor==2.1.1
traitlets==5.5.0
typing_extensions==4.4.0
urllib3==1.26.13
wcwidth==0.2.5
Werkzeug==2.2.2
wrapt==1.14.1
zipp==3.10.0
```

The memory will keep growing to even over 15GB.
Here is a screen shot of it on its way growing over time
<img width="1825" alt="Screenshot 2022-11-25 at 11 42 00 AM" src="https://user-images.githubusercontent.com/16509365/204018443-2a4a2468-77ee-422a-affa-56a1d9c56f4a.png">


Here is the log output when `import tensorfow` on most recent version TF 2.11
```
2022-11-25 15:37:47.360066: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-25 15:37:47.498679: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-11-25 15:37:47.498726: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-11-25 15:37:48.351066: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2022-11-25 15:37:48.351130: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2022-11-25 15:37:48.351143: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
```