Repository: mlcommons/inference
Issue: BERT: tf_SUT runs SingleStream forever?
Root cause: Architectural Constraint

Running BERT with TensorFlow in the SingleStream scenario:
```
$ time python3 run.py --backend=tf --scenario=SingleStream
```
seems to take forever. (With `bert.SingleStream.target_latency = 55` (ms), the ONNX runtime finishes in just over 1 minute.)

Weirdly, it prints:
```
2020-05-02 00:25:38.505377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:02:00.0
2020-05-02 00:25:38.505409: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-05-02 00:25:38.505440: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-05-02 00:25:38.505458: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-05-02 00:25:38.505474: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-05-02 00:25:38.505490: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-05-02 00:25:38.505506: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-05-02 00:25:38.505528: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-05-02 00:25:38.505630: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-02 00:25:38.506097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-02 00:25:38.506466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-05-02 00:25:38.506493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-02 00:25:38.506502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2020-05-02 00:25:38.506509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2020-05-02 00:25:38.506632: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-02 00:25:38.507101: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-02 00:25:38.507498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7533 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
```
again and again (and again).

I thought this could be related to my PR https://github.com/mlperf/inference/pull/593. I tried, however, [moving the new code into `__init__`](https://github.com/mlperf/inference/pull/593#issuecomment-622614462) and nothing seemed to have changed.

When `bert.SingleStream.target_latency = 55555` (ms), ONNX finishes very quickly, but TF keeps going and going...

@psyhtest This is because we are creating a new TF session every time IssueQuery() is called. Maybe I should have somehow kept the session "open", but I need help from TF experts for how to do that.

Hm, tf_SUT.py looks ok to me, don't see yet why it would close the session.

I've managed to run only two queries by placing:
```
bert.SingleStream.target_latency = 120000
bert.SingleStream.min_query_count = 1
```
into `user.conf`. (Interestingly, no matter how much I increase `target_latency` beyond 120 seconds, LoadGen still issues 2 queries.)

Unlike ONNX and PyTorch, TF does run abysmally slow because of repeated init:
```
(mlperf) anton@mlperf-inference-bert-anton:/workspace$ time python3 run.py --backend=tf --scenario=SingleStream
...
================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 0.07
QPS w/o loadgen overhead        : 0.08
```

@guschmue  please help! 

this is because tf.estimator.predict() will always create the model, than takes the batch and yields the results. Since we call this on issue_query() this will happen for each issue_query() and is going to be extra slow.
The fix would be to export an inference interface into a saved_model, than use this one with a regular tf.Session().

@guschmue Could you give me some pointers or docs about that? Thanks

The doc is here
https://www.tensorflow.org/api_docs/python/tf/estimator/Estimator
but the real truth is here:
https://github.com/tensorflow/estimator/blob/master/tensorflow_estimator/python/estimator/estimator.py#L612

Creating a new interface for inference should be straight forward but we'd need to upload the new saved-model.
Switching over use a good old session should not be too much work.

@guschmue  will share with @nvpohanh on a script to covert to a new saved-model. Loadgen does not work with the estimator interface. The decision is to update the reference model to the new format. 

@nvpohanh, I think we can export to a saved_model with 
```run_squad.py --export_triton```

https://github.com/NVIDIA/DeepLearningExamples/blob/master/TensorFlow/LanguageModeling/BERT/run_squad.py#L1098


@guschmue Ah, that's great! I am OOTO on 7/9-10, but I will try this next Monday.

TF model is frozen with batch size 1.  The submitter is free to other batch sizes with the provided script.