Repository: tensorflow
Issue: unbounded memory leak in tf.io.gfile.isdir()
Labels: comp:tensorboard,stat:awaiting tensorflower

This was discovered in debugging of https://github.com/tensorflow/tensorboard/issues/766 by a combination of @psybuzz, @wchargin, and myself.  From empirical evidence from TensorBoard users it appears that this grows without bound, so in practical usage it only takes a day or so to consume dozens of GB of memory.

Calling `tf.io.gfile.isdir()` leaks memory at a rate of approximately 1 MB per 20,000 calls, and this is reproducible at TF 2.0.0 and latest tf-nightly (`tf-nightly-2.1.0.dev20191219`), on macOS, Ubuntu 16.04, and Linux Debian (a Google workstation), and with python 2.7, 3.5, and 3.7.

Here's our repro script:
```python
import gc
import os
import resource
import time

import tensorflow as tf

print("PID: %d\n" % (os.getpid(),))
prev = 0
while True:
  peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
  print("peak memory = %d (+%d) in kb (Linux) or b (macOS)" % (peak, peak-prev))
  prev = peak
  for _ in range(20000):
    tf.io.gfile.isdir(b"/tmp/nonexistent-file-for-tf-memory-leak")
  gc.collect()
  time.sleep(1.0)
```

Sample output of `python repro.py`:
```
PID: 153611

peak memory = 226796 (+226796) in kb (Linux) or b (macOS)
peak memory = 228108 (+1312) in kb (Linux) or b (macOS)
peak memory = 229132 (+1024) in kb (Linux) or b (macOS)
peak memory = 229900 (+768) in kb (Linux) or b (macOS)
peak memory = 230924 (+1024) in kb (Linux) or b (macOS)
peak memory = 231948 (+1024) in kb (Linux) or b (macOS)
peak memory = 232716 (+768) in kb (Linux) or b (macOS)
peak memory = 233740 (+1024) in kb (Linux) or b (macOS)
peak memory = 234764 (+1024) in kb (Linux) or b (macOS)
peak memory = 235788 (+1024) in kb (Linux) or b (macOS)
peak memory = 236556 (+768) in kb (Linux) or b (macOS)
peak memory = 237580 (+1024) in kb (Linux) or b (macOS)
...
```

Our initial attempt to find a root cause led us to suspect the fact that `is_directory_v2` uses ScopedTFStatus while the rest of the `gfile` API does not (we spot-checked a few other APIs, including `tf.io.gfile.stat()`, and did not see the same issue).

Here's the code from v2.0.0 (file_io.py was just converted to PyBind11 today so it's possible this actually fixes the issue, but there is not yet a nightly with the change to check):
https://github.com/tensorflow/tensorflow/blob/v2.0.0/tensorflow/python/lib/io/file_io.py#L585-L596

We attempted to debug further by deconstructing the calls to `isdir()` into the two lines, one that creates `ScopedTFStatus` and one that calls `pywrap_tensorflow.IsDirectory()`, and it seemed to be the case that the memory leak is proportional to the number of times `IsDirectory()` is called with a *distinct* `ScopedTFStatus` pointer (calling it over and over with the same status doesn't seem to leak at a proportional rate; reusing the status here seemed fine for testing this because `IsDirectory()` does not actually touch the status in the codepath for a nonexistent file).  So we suspect maybe there's a weird interaction at the SWIG boundary that results in the leak.

Furthermore, it also seems to leak when the argument is an existing filename; the repro uses a nonexistent one for simplicity and because that makes the codepath slightly simpler (since then `IsDirectory()` exits early on file nonexistence via the `access()` syscall and never even calls `stat()`).  Also, the leak still occurs when the `gc.collect()` is omitted; it's also just there to isolate possible causes of the leak.