Repository: tensorflow
Issue: Very serious! Using this method will definitely result in memory leaks, I hope you can provide support
Labels: stat:awaiting tensorflower,comp:runtime,type:performance,TF 2.18

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.18

### Custom code

Yes

### OS platform and distribution

ubuntu 2.2 or mac m1

### Mobile device

_No response_

### Python version

3.11

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have tried various methods, but the memory is definitely leaking, it seems that the release of memory cannot keep up. Through the logs, it can be found that there is periodic memory recycling, but with the increase of time, there is still a clear upward trend

### Standalone code to reproduce the issue

```shell
import gc

import keras
import numpy as np
import psutil
from keras.optimizers import Adam
from keras.layers import Dense, Dropout, Input, LSTM
import tensorflow as tf
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Sequential, load_model
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
import time
import json


num_samples = 6
num_features = 3
num_classes = 4
epochs = 50
batch_size = 2
identifier = "test_model"
num_iterations = 500  

def build_model(X, num_classes):
    model = Sequential()
    model.add(Input(shape=(X.shape[1], X.shape[2])))
    model.add(LSTM(16, return_sequences=True))
    model.add(LSTM(16))
    model.add(Dropout(0.4))
    model.add(Dense(8, activation='tanh'))
    model.add(Dense(num_classes, activation='softmax'))

    model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['categorical_accuracy'])
    return model


data_X = np.random.rand(num_samples, num_features)
data_Y = np.random.randint(0, num_classes, size=(num_samples, 1))  


data_Y = np.eye(num_classes)[data_Y.flatten()]  
print(type(data_X))

scaler = MinMaxScaler()
data_X_scaled = scaler.fit_transform(data_X)


train_X, test_X, train_Y, test_Y = train_test_split(data_X_scaled, data_Y, train_size=0.6, random_state=42)


train_X = np.expand_dims(train_X, axis=1)
test_X = np.expand_dims(test_X, axis=1)

for iteration in range(num_iterations):
   
    tf.keras.backend.clear_session()
    tf.compat.v1.reset_default_graph()
    model = build_model(train_X, num_classes)
 
    model_name = f"model_{iteration}"
    early_stopping = EarlyStopping(monitor='loss', patience=10, verbose=0, restore_best_weights=True)
    print(f"Iteration {iteration + 1}/{num_iterations}")
    process = psutil.Process()
    mem_info = process.memory_info()
    print(f"start Current memory usage: {mem_info.rss / (1024 * 1024):.2f} MB")  # RSS - Resident Set Size
    try:
        history = model.fit(train_X, train_Y, epochs=epochs, batch_size=batch_size, shuffle=True,
                            validation_data=(test_X, test_Y), verbose=0)
        print(f"Training model: {model.name}")
        del model
        tf.keras.backend.clear_session()
        gc.collect()
    except Exception as e:
        print("err:", e)
    finally:
        process = psutil.Process()
        mem_info = process.memory_info()
        print(f"end Current memory usage: {mem_info.rss / (1024 * 1024):.2f} MB")  # RSS - Resident Set Size

print("end！")
```


### Relevant log output

```shell
Iteration 1/500
start Current memory usage: 450.69 MB
Training model: sequential
end Current memory usage: 524.41 MB
Iteration 2/500
start Current memory usage: 524.52 MB
Training model: sequential
end Current memory usage: 564.97 MB
Iteration 3/500
start Current memory usage: 564.98 MB
Training model: sequential
end Current memory usage: 598.00 MB
Iteration 4/500
start Current memory usage: 598.03 MB
Training model: sequential
end Current memory usage: 624.69 MB
Iteration 5/500
start Current memory usage: 624.69 MB
Training model: sequential
end Current memory usage: 653.89 MB
Iteration 6/500
start Current memory usage: 653.91 MB
Training model: sequential
end Current memory usage: 679.45 MB
Iteration 7/500
start Current memory usage: 679.45 MB
Training model: sequential
end Current memory usage: 701.59 MB
Iteration 8/500
start Current memory usage: 701.59 MB
Training model: sequential
end Current memory usage: 726.83 MB
Iteration 9/500
start Current memory usage: 726.84 MB
Training model: sequential
end Current memory usage: 749.56 MB
Iteration 10/500
start Current memory usage: 749.56 MB
Training model: sequential
end Current memory usage: 782.56 MB
Iteration 11/500
start Current memory usage: 782.56 MB
Training model: sequential
end Current memory usage: 805.92 MB
Iteration 12/500
start Current memory usage: 805.92 MB
Training model: sequential
end Current memory usage: 833.17 MB
Iteration 13/500
start Current memory usage: 833.17 MB
Training model: sequential
end Current memory usage: 852.84 MB
Iteration 14/500
start Current memory usage: 852.84 MB
Training model: sequential
end Current memory usage: 875.05 MB
Iteration 15/500
start Current memory usage: 875.06 MB
Training model: sequential
end Current memory usage: 901.56 MB
Iteration 16/500
start Current memory usage: 901.56 MB
Training model: sequential
end Current memory usage: 930.62 MB
Iteration 17/500
start Current memory usage: 705.70 MB
Training model: sequential
end Current memory usage: 762.64 MB
Iteration 18/500
start Current memory usage: 762.70 MB
Training model: sequential
end Current memory usage: 798.06 MB
Iteration 19/500
start Current memory usage: 798.17 MB
Training model: sequential
end Current memory usage: 824.98 MB
Iteration 20/500
start Current memory usage: 824.98 MB
Training model: sequential
end Current memory usage: 850.34 MB
Iteration 21/500
start Current memory usage: 850.42 MB
Training model: sequential
end Current memory usage: 876.81 MB
Iteration 22/500
start Current memory usage: 876.81 MB
Training model: sequential
end Current memory usage: 904.02 MB
Iteration 23/500
start Current memory usage: 904.08 MB
Training model: sequential
end Current memory usage: 929.70 MB
Iteration 24/500
start Current memory usage: 929.73 MB
Training model: sequential
end Current memory usage: 952.33 MB
Iteration 25/500
start Current memory usage: 952.34 MB
Training model: sequential
end Current memory usage: 952.28 MB
Iteration 26/500
start Current memory usage: 952.47 MB
Training model: sequential
end Current memory usage: 980.39 MB
Iteration 27/500
start Current memory usage: 978.78 MB
Training model: sequential
end Current memory usage: 999.02 MB
Iteration 28/500
start Current memory usage: 999.05 MB
Training model: sequential
end Current memory usage: 1023.50 MB
Iteration 29/500
start Current memory usage: 1023.53 MB
Training model: sequential
end Current memory usage: 1047.80 MB
Iteration 30/500
start Current memory usage: 1047.83 MB
Training model: sequential
end Current memory usage: 1068.88 MB
Iteration 31/500
start Current memory usage: 1068.94 MB
Training model: sequential
end Current memory usage: 1095.78 MB
Iteration 32/500
start Current memory usage: 1095.78 MB
Training model: sequential
end Current memory usage: 1119.03 MB
Iteration 33/500
start Current memory usage: 1119.03 MB
Training model: sequential
end Current memory usage: 1039.41 MB
Iteration 34/500
start Current memory usage: 1022.78 MB
Training model: sequential
end Current memory usage: 1040.88 MB
Iteration 35/500
start Current memory usage: 1040.70 MB
Training model: sequential
end Current memory usage: 1054.58 MB
Iteration 36/500
start Current memory usage: 1054.58 MB
Training model: sequential
end Current memory usage: 1076.16 MB
Iteration 37/500
start Current memory usage: 1076.19 MB
Training model: sequential
end Current memory usage: 1097.02 MB
Iteration 38/500
start Current memory usage: 1097.03 MB
Training model: sequential
end Current memory usage: 1113.70 MB
Iteration 39/500
start Current memory usage: 1114.12 MB
Training model: sequential
end Current memory usage: 1140.30 MB
Iteration 40/500
start Current memory usage: 1140.33 MB
Training model: sequential
end Current memory usage: 1163.81 MB
Iteration 41/500
start Current memory usage: 1163.86 MB
Training model: sequential
end Current memory usage: 1195.83 MB
Iteration 42/500
start Current memory usage: 1195.83 MB
Training model: sequential
end Current memory usage: 1221.53 MB
Iteration 43/500
start Current memory usage: 1221.55 MB
Training model: sequential
end Current memory usage: 1231.09 MB
Iteration 44/500
start Current memory usage: 1231.14 MB
Training model: sequential
end Current memory usage: 1245.78 MB
Iteration 45/500
start Current memory usage: 1199.55 MB
Training model: sequential
end Current memory usage: 1221.59 MB
Iteration 46/500
start Current memory usage: 1221.59 MB
Training model: sequential
end Current memory usage: 1249.11 MB
Iteration 47/500
start Current memory usage: 1249.22 MB
Training model: sequential
end Current memory usage: 1275.50 MB
Iteration 48/500
start Current memory usage: 1259.83 MB
Training model: sequential
end Current memory usage: 1290.91 MB
Iteration 49/500
start Current memory usage: 1285.67 MB
Training model: sequential
end Current memory usage: 1296.75 MB
Iteration 50/500
start Current memory usage: 1296.75 MB
Training model: sequential
end Current memory usage: 1306.59 MB
Iteration 51/500
start Current memory usage: 1306.59 MB
Training model: sequential
end Current memory usage: 1287.53 MB
Iteration 52/500
start Current memory usage: 1287.53 MB
Training model: sequential
end Current memory usage: 1297.23 MB
Iteration 53/500
start Current memory usage: 1297.25 MB
Training model: sequential
end Current memory usage: 1285.45 MB
Iteration 54/500
start Current memory usage: 1285.45 MB
Training model: sequential
end Current memory usage: 1290.36 MB
Iteration 55/500
start Current memory usage: 1282.14 MB
Training model: sequential
end Current memory usage: 1302.14 MB
Iteration 56/500
start Current memory usage: 1302.14 MB
Training model: sequential
end Current memory usage: 1287.70 MB
Iteration 57/500
start Current memory usage: 1287.75 MB
Training model: sequential
end Current memory usage: 1282.77 MB
Iteration 58/500
start Current memory usage: 1271.38 MB
Training model: sequential
end Current memory usage: 1232.14 MB
Iteration 59/500
start Current memory usage: 1212.70 MB
Training model: sequential
end Current memory usage: 1201.16 MB
Iteration 60/500
start Current memory usage: 1200.53 MB
Training model: sequential
end Current memory usage: 1169.45 MB
Iteration 61/500
start Current memory usage: 1169.45 MB
Training model: sequential
end Current memory usage: 1209.73 MB
Iteration 62/500
start Current memory usage: 1207.19 MB
Training model: sequential
end Current memory usage: 1226.28 MB
Iteration 63/500
start Current memory usage: 1226.28 MB
Training model: sequential
end Current memory usage: 1231.45 MB
Iteration 64/500
start Current memory usage: 1210.11 MB
Training model: sequential
end Current memory usage: 1176.00 MB
Iteration 65/500
start Current memory usage: 1173.97 MB
Training model: sequential
end Current memory usage: 1201.42 MB
Iteration 66/500
start Current memory usage: 1201.42 MB
Training model: sequential
end Current memory usage: 1223.94 MB
Iteration 67/500
start Current memory usage: 1222.50 MB
Training model: sequential
end Current memory usage: 1229.80 MB
Iteration 68/500
start Current memory usage: 1227.14 MB
Training model: sequential
end Current memory usage: 1219.02 MB
Iteration 69/500
start Current memory usage: 1210.48 MB
Training model: sequential
end Current memory usage: 1247.17 MB
Iteration 70/500
start Current memory usage: 1245.94 MB
Training model: sequential
end Current memory usage: 1259.84 MB
Iteration 71/500
start Current memory usage: 1259.86 MB
Training model: sequential
end Current memory usage: 1286.39 MB
Iteration 72/500
start Current memory usage: 1286.53 MB
Training model: sequential
end Current memory usage: 1316.52 MB
Iteration 73/500
start Current memory usage: 1311.53 MB
Training model: sequential
end Current memory usage: 1338.72 MB
Iteration 74/500
start Current memory usage: 1338.75 MB
Training model: sequential
end Current memory usage: 1348.45 MB
Iteration 75/500
start Current memory usage: 1338.30 MB
Training model: sequential
end Current memory usage: 1354.97 MB
Iteration 76/500
start Current memory usage: 1353.83 MB
Training model: sequential
end Current memory usage: 1385.67 MB
Iteration 77/500
start Current memory usage: 1385.69 MB
Training model: sequential
end Current memory usage: 1408.83 MB
Iteration 78/500
start Current memory usage: 1408.88 MB
Training model: sequential
end Current memory usage: 1430.91 MB
Iteration 79/500
start Current memory usage: 1430.94 MB
Training model: sequential
end Current memory usage: 1443.62 MB
Iteration 80/500
start Current memory usage: 1428.00 MB
Training model: sequential
end Current memory usage: 1436.50 MB
Iteration 81/500
start Current memory usage: 1436.64 MB
Training model: sequential
end Current memory usage: 1454.66 MB
Iteration 82/500
start Current memory usage: 1440.91 MB
Training model: sequential
end Current memory usage: 1461.81 MB
Iteration 83/500
start Current memory usage: 1460.47 MB
Training model: sequential
end Current memory usage: 1481.19 MB
Iteration 84/500
start Current memory usage: 1481.19 MB
Training model: sequential
end Current memory usage: 1477.84 MB
Iteration 85/500
start Current memory usage: 1477.84 MB
Training model: sequential
end Current memory usage: 1493.55 MB
Iteration 86/500
start Current memory usage: 1493.58 MB
Training model: sequential
end Current memory usage: 1509.50 MB
Iteration 87/500
start Current memory usage: 1509.50 MB
Training model: sequential
end Current memory usage: 1543.94 MB
Iteration 88/500
start Current memory usage: 1542.83 MB
Training model: sequential
end Current memory usage: 1516.17 MB
Iteration 89/500
start Current memory usage: 1516.20 MB
Training model: sequential
end Current memory usage: 1470.17 MB
Iteration 90/500
start Current memory usage: 1470.22 MB
Training model: sequential
end Current memory usage: 1443.72 MB
Iteration 91/500
start Current memory usage: 1444.36 MB
Training model: sequential
end Current memory usage: 1486.23 MB
Iteration 92/500
start Current memory usage: 1476.41 MB
Training model: sequential
end Current memory usage: 1524.97 MB
Iteration 93/500
start Current memory usage: 1524.97 MB
Training model: sequential
end Current memory usage: 1534.94 MB
Iteration 94/500
start Current memory usage: 1551.98 MB
Training model: sequential
end Current memory usage: 1853.48 MB
Iteration 95/500
start Current memory usage: 1853.48 MB
Training model: sequential
end Current memory usage: 1790.12 MB
Iteration 96/500
start Current memory usage: 1792.27 MB
Training model: sequential
end Current memory usage: 1883.20 MB
Iteration 97/500
start Current memory usage: 1879.05 MB
Training model: sequential
end Current memory usage: 1759.69 MB
Iteration 98/500
start Current memory usage: 1669.66 MB
Training model: sequential
end Current memory usage: 1596.77 MB
Iteration 99/500
start Current memory usage: 1597.12 MB
Training model: sequential
end Current memory usage: 1568.83 MB
Iteration 100/500
start Current memory usage: 1532.98 MB
Training model: sequential
end Current memory usage: 1516.75 MB
Iteration 101/500
start Current memory usage: 1465.98 MB
Training model: sequential
end Current memory usage: 1486.66 MB
Iteration 102/500
start Current memory usage: 1483.34 MB
Training model: sequential
end Current memory usage: 1523.19 MB
Iteration 103/500
start Current memory usage: 1523.14 MB
Training model: sequential
end Current memory usage: 1532.77 MB
Iteration 104/500
start Current memory usage: 1531.14 MB
Training model: sequential
end Current memory usage: 1561.78 MB
Iteration 105/500
start Current memory usage: 1555.67 MB
Training model: sequential
end Current memory usage: 1586.70 MB
Iteration 106/500
start Current memory usage: 1586.75 MB
Training model: sequential
end Current memory usage: 1608.41 MB
Iteration 107/500
start Current memory usage: 1603.81 MB
Training model: sequential
end Current memory usage: 1629.00 MB
Iteration 108/500
start Current memory usage: 1629.05 MB
Training model: sequential
end Current memory usage: 1609.25 MB
Iteration 109/500
start Current memory usage: 1609.31 MB
Training model: sequential
end Current memory usage: 1630.09 MB
Iteration 110/500
start Current memory usage: 1629.20 MB
Training model: sequential
end Current memory usage: 1638.66 MB
Iteration 111/500
start Current memory usage: 1620.30 MB
Training model: sequential
end Current memory usage: 1642.81 MB
Iteration 112/500
start Current memory usage: 1642.94 MB
Training model: sequential
end Current memory usage: 1659.45 MB
Iteration 113/500
start Current memory usage: 1655.17 MB
Training model: sequential
end Current memory usage: 1687.80 MB
Iteration 114/500
start Current memory usage: 1673.33 MB
Training model: sequential
end Current memory usage: 1705.94 MB
Iteration 115/500
start Current memory usage: 1699.95 MB
Training model: sequential
end Current memory usage: 1708.22 MB
Iteration 116/500
start Current memory usage: 1707.88 MB
Training model: sequential
end Current memory usage: 1648.23 MB
Iteration 117/500
start Current memory usage: 1634.03 MB
Training model: sequential
end Current memory usage: 1670.97 MB
Iteration 118/500
start Current memory usage: 1671.97 MB
Training model: sequential
end Current memory usage: 1649.69 MB
Iteration 119/500
start Current memory usage: 1645.14 MB
Training model: sequential
end Current memory usage: 1698.64 MB
Iteration 120/500
start Current memory usage: 1699.69 MB
Training model: sequential
end Current memory usage: 1737.67 MB
Iteration 121/500
start Current memory usage: 1737.67 MB
Training model: sequential
end Current memory usage: 1738.05 MB
Iteration 122/500
start Current memory usage: 1721.47 MB
Training model: sequential
end Current memory usage: 1730.64 MB
Iteration 123/500
start Current memory usage: 1729.53 MB
Training model: sequential
end Current memory usage: 1766.12 MB
Iteration 124/500
start Current memory usage: 1761.22 MB
Training model: sequential
end Current memory usage: 1796.58 MB
Iteration 125/500
start Current memory usage: 1796.73 MB
Training model: sequential
end Current memory usage: 1709.02 MB
Iteration 126/500
start Current memory usage: 1721.67 MB
Training model: sequential
end Current memory usage: 1771.50 MB
Iteration 127/500
start Current memory usage: 1771.50 MB
Training model: sequential
end Current memory usage: 1777.38 MB
Iteration 128/500
start Current memory usage: 1757.58 MB
Training model: sequential
end Current memory usage: 1806.50 MB
Iteration 129/500
start Current memory usage: 1758.81 MB
Training model: sequential
end Current memory usage: 1812.45 MB
Iteration 130/500
start Current memory usage: 1812.86 MB
Training model: sequential
end Current memory usage: 1811.14 MB
Iteration 131/500
start Current memory usage: 1799.61 MB
Training model: sequential
end Current memory usage: 1835.33 MB
Iteration 132/500
start Current memory usage: 1716.38 MB
Training model: sequential
end Current memory usage: 1759.75 MB
Iteration 133/500
start Current memory usage: 1752.44 MB
Training model: sequential
end Current memory usage: 1818.41 MB
Iteration 134/500
start Current memory usage: 1811.42 MB
Training model: sequential
end Current memory usage: 1853.58 MB
Iteration 135/500
start Current memory usage: 1853.70 MB
Training model: sequential
end Current memory usage: 1858.50 MB
Iteration 136/500
start Current memory usage: 1858.56 MB
Training model: sequential
end Current memory usage: 1874.84 MB
Iteration 137/500
start Current memory usage: 1862.92 MB
Training model: sequential
end Current memory usage: 1768.23 MB
Iteration 138/500
start Current memory usage: 1762.73 MB
Training model: sequential
end Current memory usage: 1843.39 MB
Iteration 139/500
start Current memory usage: 1843.52 MB
Training model: sequential
end Current memory usage: 1885.88 MB
Iteration 140/500
start Current memory usage: 1885.95 MB
Training model: sequential
end Current memory usage: 1924.86 MB
Iteration 141/500
start Current memory usage: 1925.05 MB
Training model: sequential
end Current memory usage: 1946.80 MB
Iteration 142/500
start Current memory usage: 1946.69 MB
Training model: sequential
end Current memory usage: 1977.53 MB
Iteration 143/500
start Current memory usage: 1974.27 MB
Training model: sequential
end Current memory usage: 1995.17 MB
Iteration 144/500
start Current memory usage: 1992.41 MB
Training model: sequential
end Current memory usage: 1984.45 MB
Iteration 145/500
start Current memory usage: 1963.42 MB
Training model: sequential
end Current memory usage: 1947.31 MB
Iteration 146/500
start Current memory usage: 1944.47 MB
Training model: sequential
end Current memory usage: 1996.00 MB
Iteration 147/500
start Current memory usage: 1996.08 MB
Training model: sequential
end Current memory usage: 2008.41 MB
Iteration 148/500
start Current memory usage: 1999.69 MB
Training model: sequential
end Current memory usage: 1951.30 MB
Iteration 149/500
start Current memory usage: 1942.98 MB
Training model: sequential
end Current memory usage: 1992.28 MB
Iteration 150/500
start Current memory usage: 1982.86 MB
Training model: sequential
end Current memory usage: 2008.83 MB
Iteration 151/500
start Current memory usage: 2008.83 MB
Training model: sequential
end Current memory usage: 1946.42 MB
Iteration 152/500
start Current memory usage: 1946.92 MB
Training model: sequential
end Current memory usage: 1992.48 MB
Iteration 153/500
start Current memory usage: 1979.52 MB
Training model: sequential
end Current memory usage: 2035.66 MB
Iteration 154/500
start Current memory usage: 2023.91 MB
Training model: sequential
end Current memory usage: 2030.31 MB
Iteration 155/500
start Current memory usage: 1974.39 MB
Training model: sequential
end Current memory usage: 2029.30 MB
Iteration 156/500
start Current memory usage: 1997.42 MB
Training model: sequential
end Current memory usage: 2000.31 MB
Iteration 157/500
start Current memory usage: 1964.38 MB
Training model: sequential
end Current memory usage: 1979.45 MB
Iteration 158/500
start Current memory usage: 1973.12 MB
Training model: sequential
```