Repository: tensorflow
Issue: Memory leak in training
Labels: stat:awaiting tensorflower,type:bug,2.17

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.16.1 and 2.17

### Custom code

Yes

### OS platform and distribution

Debian 11

### Mobile device

_No response_

### Python version

Python 3.10.14

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

cuda/12.0.0_gcc-10.4.0 and cudnn/8.9.7.29-12_gcc-10.4.0

### GPU model and memory

different GPU, among which Tesla V100-SXM2-32GB

### Current behavior?

I have a memory leak (GPU memory and RAM constantly increase) during my training. 

**This does not happen with Tensorflow 2.11.1**. 

[Here is the project](https://github.com/deep-finder/tirfm-deepfinder).
Unfortunately, this is not my code and I do not have time to create a minimal standalone code.

### Standalone code to reproduce the issue

```shell
def printMemoryUsage(self):
        gpus = tf.config.list_physical_devices('GPU')             
        for gpu in gpus:
            gpuNameRoot = gpu.name.split(':')[0] + ':'
            memory_info = tf.config.experimental.get_memory_info(gpu.name.replace(gpuNameRoot, ''))
            self.display(f'Memory info of GPU {gpu.name}: current: {memory_info["current"]/1e9:.2f}, peak: {memory_info["peak"]/1e9:.2f}')
            try:
                import psutil
                virtual_memory = psutil.virtual_memory()
                print(f'Memory info of CPU: total:{virtual_memory[0]/1e9:.2f}Gb, available: {virtual_memory[1]/1e9:.2f}Gb, percent: {virtual_memory[2]}%')
            except:
                pass

[...]
        # Training loop:
        for e in range(self.epochs):
            # TRAINING:
            start = time.time()
            list_loss_train = []
            list_acc_train = []
            for it in range(self.steps_per_epoch):
                if self.flag_direct_read:
                    batch_data, batch_target = self.generate_batch_direct_read(path_data, path_target, self.batch_size, objlist_train)
                else:
                    batch_data, batch_target, idx_list = self.generate_batch_from_array(data_list, target_list, self.batch_size, objlist_train)

                if self.sample_weights is not None:
                    sample_weight = self.sample_weights[idx_list]
                else:
                    sample_weight = None

                loss_train = self.net.train_on_batch(batch_data, batch_target,
                                                     class_weight=self.class_weight,
                                                     sample_weight=sample_weight)

                self.display('epoch %d/%d - it %d/%d - loss: %0.3f - acc: %0.3f' % (e + 1, self.epochs, it + 1, self.steps_per_epoch, loss_train[0], loss_train[1]))

                self.printMemoryUsage()

                list_loss_train.append(loss_train[0])
                list_acc_train.append(loss_train[1])
                del batch_data
                del batch_target
                if idx_list is not None:
                    del idx_list
                gc.collect()
```


### Relevant log output

```shell
With Tensorflow 2.11.1:


=============================================================
epoch 3/50 - it 1/100 - loss: 2.012 - acc: 0.976
Memory info of GPU /physical_device:GPU:0: current: 0.02, peak: 2.90
Memory info of CPU: total:201.37Gb, available: 191.38Gb, percent: 5.0%
epoch 3/50 - it 2/100 - loss: 2.008 - acc: 0.985
Memory info of GPU /physical_device:GPU:0: current: 0.02, peak: 2.90
Memory info of CPU: total:201.37Gb, available: 191.38Gb, percent: 5.0%
epoch 3/50 - it 3/100 - loss: 2.004 - acc: 0.992
Memory info of GPU /physical_device:GPU:0: current: 0.02, peak: 2.90
Memory info of CPU: total:201.37Gb, available: 191.38Gb, percent: 5.0%
epoch 3/50 - it 4/100 - loss: 2.006 - acc: 0.987
Memory info of GPU /physical_device:GPU:0: current: 0.02, peak: 2.90
Memory info of CPU: total:201.37Gb, available: 191.38Gb, percent: 5.0%
epoch 3/50 - it 5/100 - loss: 2.006 - acc: 0.989
```


With Tensorflow 2.16.1 and 2.17:

```
epoch 1/50 - it 6/100 - loss: 2.473 - acc: 0.305
Memory info of GPU /physical_device:GPU:0: current: 0.18, peak: 2.55
Memory info of CPU: total:201.37Gb, available: 190.78Gb, percent: 5.3%
epoch 1/50 - it 7/100 - loss: 2.470 - acc: 0.394
Memory info of GPU /physical_device:GPU:0: current: 0.21, peak: 2.58
Memory info of CPU: total:201.37Gb, available: 190.58Gb, percent: 5.4%
epoch 1/50 - it 8/100 - loss: 2.466 - acc: 0.463
Memory info of GPU /physical_device:GPU:0: current: 0.24, peak: 2.61
Memory info of CPU: total:201.37Gb, available: 190.38Gb, percent: 5.5%
epoch 1/50 - it 9/100 - loss: 2.461 - acc: 0.519
Memory info of GPU /physical_device:GPU:0: current: 0.27, peak: 2.64
Memory info of CPU: total:201.37Gb, available: 190.11Gb, percent: 5.6%
epoch 1/50 - it 10/100 - loss: 2.455 - acc: 0.564
Memory info of GPU /physical_device:GPU:0: current: 0.29, peak: 2.67
Memory info of CPU: total:201.37Gb, available: 189.94Gb, percent: 5.7%
epoch 1/50 - it 11/100 - loss: 2.448 - acc: 0.603
Memory info of GPU /physical_device:GPU:0: current: 0.32, peak: 2.70
Memory info of CPU: total:201.37Gb, available: 189.67Gb, percent: 5.8%
epoch 1/50 - it 12/100 - loss: 2.437 - acc: 0.634
Memory info of GPU /physical_device:GPU:0: current: 0.35, peak: 2.72
Memory info of CPU: total:201.37Gb, available: 189.48Gb, percent: 5.9%
epoch 1/50 - it 13/100 - loss: 2.425 - acc: 0.662
Memory info of GPU /physical_device:GPU:0: current: 0.38, peak: 2.75
Memory info of CPU: total:201.37Gb, available: 189.21Gb, percent: 6.0%
epoch 1/50 - it 14/100 - loss: 2.408 - acc: 0.685
Memory info of GPU /physical_device:GPU:0: current: 0.41, peak: 2.78
Memory info of CPU: total:201.37Gb, available: 189.03Gb, percent: 6.1%
epoch 1/50 - it 15/100 - loss: 2.389 - acc: 0.705
Memory info of GPU /physical_device:GPU:0: current: 0.44, peak: 2.81
Memory info of CPU: total:201.37Gb, available: 188.79Gb, percent: 6.2%
epoch 1/50 - it 16/100 - loss: 2.369 - acc: 0.723
Memory info of GPU /physical_device:GPU:0: current: 0.46, peak: 2.84
Memory info of CPU: total:201.37Gb, available: 188.54Gb, percent: 6.4%
epoch 1/50 - it 17/100 - loss: 2.350 - acc: 0.739
Memory info of GPU /physical_device:GPU:0: current: 0.49, peak: 2.87
Memory info of CPU: total:201.37Gb, available: 188.37Gb, percent: 6.5%
epoch 1/50 - it 18/100 - loss: 2.332 - acc: 0.753
Memory info of GPU /physical_device:GPU:0: current: 0.52, peak: 2.89
Memory info of CPU: total:201.37Gb, available: 188.10Gb, percent: 6.6%
epoch 1/50 - it 19/100 - loss: 2.315 - acc: 0.765
Memory info of GPU /physical_device:GPU:0: current: 0.55, peak: 2.92
Memory info of CPU: total:201.37Gb, available: 187.87Gb, percent: 6.7%
epoch 1/50 - it 20/100 - loss: 2.300 - acc: 0.776
Memory info of GPU /physical_device:GPU:0: current: 0.58, peak: 2.95
Memory info of CPU: total:201.37Gb, available: 187.64Gb, percent: 6.8%
epoch 1/50 - it 21/100 - loss: 2.286 - acc: 0.786
Memory info of GPU /physical_device:GPU:0: current: 0.61, peak: 2.98
Memory info of CPU: total:201.37Gb, available: 187.49Gb, percent: 6.9%
epoch 1/50 - it 22/100 - loss: 2.274 - acc: 0.795
Memory info of GPU /physical_device:GPU:0: current: 0.63, peak: 3.01
Memory info of CPU: total:201.37Gb, available: 187.25Gb, percent: 7.0%
epoch 1/50 - it 23/100 - loss: 2.262 - acc: 0.804
Memory info of GPU /physical_device:GPU:0: current: 0.66, peak: 3.04
Memory info of CPU: total:201.37Gb, available: 186.97Gb, percent: 7.1%
epoch 1/50 - it 24/100 - loss: 2.251 - acc: 0.811
Memory info of GPU /physical_device:GPU:0: current: 0.69, peak: 3.06
Memory info of CPU: total:201.37Gb, available: 186.81Gb, percent: 7.2%
epoch 1/50 - it 25/100 - loss: 2.241 - acc: 0.818
Memory info of GPU /physical_device:GPU:0: current: 0.72, peak: 3.09
Memory info of CPU: total:201.37Gb, available: 186.59Gb, percent: 7.3%
epoch 1/50 - it 26/100 - loss: 2.232 - acc: 0.825
Memory info of GPU /physical_device:GPU:0: current: 0.75, peak: 3.12
Memory info of CPU: total:201.37Gb, available: 186.36Gb, percent: 7.5%
epoch 1/50 - it 27/100 - loss: 2.224 - acc: 0.831
Memory info of GPU /physical_device:GPU:0: current: 0.78, peak: 3.15
Memory info of CPU: total:201.37Gb, available: 186.09Gb, percent: 7.6%
epoch 1/50 - it 28/100 - loss: 2.216 - acc: 0.836
Memory info of GPU /physical_device:GPU:0: current: 0.80, peak: 3.18
Memory info of CPU: total:201.37Gb, available: 185.90Gb, percent: 7.7%
epoch 1/50 - it 29/100 - loss: 2.209 - acc: 0.841
Memory info of GPU /physical_device:GPU:0: current: 0.83, peak: 3.21
Memory info of CPU: total:201.37Gb, available: 185.65Gb, percent: 7.8%
epoch 1/50 - it 30/100 - loss: 2.202 - acc: 0.846
Memory info of GPU /physical_device:GPU:0: current: 0.86, peak: 3.23
Memory info of CPU: total:201.37Gb, available: 185.46Gb, percent: 7.9%
epoch 1/50 - it 31/100 - loss: 2.196 - acc: 0.851
Memory info of GPU /physical_device:GPU:0: current: 0.89, peak: 3.26
Memory info of CPU: total:201.37Gb, available: 185.24Gb, percent: 8.0%
epoch 1/50 - it 32/100 - loss: 2.190 - acc: 0.855
Memory info of GPU /physical_device:GPU:0: current: 0.92, peak: 3.29
Memory info of CPU: total:201.37Gb, available: 184.99Gb, percent: 8.1%
epoch 1/50 - it 33/100 - loss: 2.185 - acc: 0.859
Memory info of GPU /physical_device:GPU:0: current: 0.95, peak: 3.32
Memory info of CPU: total:201.37Gb, available: 184.75Gb, percent: 8.3%
epoch 1/50 - it 34/100 - loss: 2.180 - acc: 0.863
Memory info of GPU /physical_device:GPU:0: current: 0.97, peak: 3.35
Memory info of CPU: total:201.37Gb, available: 184.51Gb, percent: 8.4%
epoch 1/50 - it 35/100 - loss: 2.174 - acc: 0.866
Memory info of GPU /physical_device:GPU:0: current: 1.00, peak: 3.38
Memory info of CPU: total:201.37Gb, available: 184.36Gb, percent: 8.4%
epoch 1/50 - it 36/100 - loss: 2.170 - acc: 0.870
Memory info of GPU /physical_device:GPU:0: current: 1.03, peak: 3.40
Memory info of CPU: total:201.37Gb, available: 184.06Gb, percent: 8.6%
epoch 1/50 - it 37/100 - loss: 2.165 - acc: 0.873
Memory info of GPU /physical_device:GPU:0: current: 1.06, peak: 3.43
Memory info of CPU: total:201.37Gb, available: 183.89Gb, percent: 8.7%
epoch 1/50 - it 38/100 - loss: 2.161 - acc: 0.876
Memory info of GPU /physical_device:GPU:0: current: 1.09, peak: 3.46
Memory info of CPU: total:201.37Gb, available: 183.66Gb, percent: 8.8%
epoch 1/50 - it 39/100 - loss: 2.157 - acc: 0.879
Memory info of GPU /physical_device:GPU:0: current: 1.11, peak: 3.49
Memory info of CPU: total:201.37Gb, available: 183.42Gb, percent: 8.9%
epoch 1/50 - it 40/100 - loss: 2.153 - acc: 0.881
Memory info of GPU /physical_device:GPU:0: current: 1.14, peak: 3.52
Memory info of CPU: total:201.37Gb, available: 183.26Gb, percent: 9.0%
epoch 1/50 - it 41/100 - loss: 2.150 - acc: 0.884
Memory info of GPU /physical_device:GPU:0: current: 1.17, peak: 3.54
Memory info of CPU: total:201.37Gb, available: 183.02Gb, percent: 9.1%
```
```