Repository: mlcommons/inference
Issue: Question about inference/cloud/translation/gnmt/tensorflow/
Root cause: Algorithmic Error

I'm using tensorflow-gpu version ot Turing T4 for inferencing benchmarking.

 Is the benchmark really using GPU for inferencing?

I got the code from mlperf/inference/cloud/translation/gnmt/tensorflow/.

I increased the # of iterations for 1 to 500000 in run_task.py.
It seems that the GPU is not being used for inferencing. Initially for first few seconds GPU utilization is 30%.  Then nvidia-smi reports GPU utilization to be zero for most of the time for 500000 iterations.

It takes about 525 seconds to complete the run. For the first 125 or seconds, nvidia-smi shows any GPU utilization. After that nvidia-smi shows 0 utilization. Is the benchmark really using GPU for inferencing?

Here is is excerpt form the log of the run:

  dynamic_seq2seq/decoder/output_projection/kernel:0, (1024, 32316), 
2019-03-08 14:08:50.113076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-03-08 14:08:50.113130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-08 14:08:50.113139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2019-03-08 14:08:50.113145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2019-03-08 14:08:50.113335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3318 MB memory) -> physical GPU (device: 0, name: GRID T4-4Q, pci bus id: 0000:02:01.0, compute capability: 7.5)
  loaded infer model parameters from /home/ml/inference-master/cloud/translation/gnmt/tensorflow/ende_gnmt_model_4_layer/translate.ckpt, time 1.01s
# Start decoding
  decoding to output /home/ml/inference-master/cloud/translation/gnmt/tensorflow/nmt/data/output/g_nmt-out
Iterations  500000
  done, num sentences 3003, num translations per input 1, time 35s, Fri Mar  8 14:09:27 2019.
  done, num sentences 3003, num translations per input 1, time 35s, Fri Mar  8 14:09:27 2019.
  done, num sentences 3003, num translations per input 1, time 35s, Fri Mar  8 14:09:27 2019.
  done, num sentences 3003, num translations per input 1, time 35s, Fri Mar  8 14:09:27 2019.
  done, num sentences 3003, num translations per input 1, time 35s, Fri Mar  8 14:09:27 2019.
  done, num sentences 3003, num translations per input 1, time 35s, Fri Mar  8 14:09:27 2019.


Excerpt from nvidia-smi log: GPU utilization drops from 26% to 0 and then stays 0 for the rest of the run.
Fri Mar  8 14:09:27 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.92       Driver Version: 410.92       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID T4-4Q          On   | 00000000:02:01.0 Off |                  N/A |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     26%      Default |
+-------------------------------+----------------------+----------------------+

| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     26%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     26%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     25%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     24%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     24%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     23%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     22%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     21%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     21%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     20%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     19%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     18%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     18%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     17%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     16%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     15%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     14%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     14%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     13%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     12%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     11%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     11%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |     10%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      9%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      8%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      8%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      7%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      6%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      5%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      5%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      4%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      3%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      2%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      2%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      1%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |
| N/A   N/A    P0    N/A /  N/A |   3851MiB /  4096MiB |      0%      Default |



Thanks,

-Uday




@ukurkure I'd like to wait for the MLPerf load generators to integrated then we will see if this issue still exist. I expect the code to change dramatically with the load generator.

Hi @christ1ne when is expected the release of the MLPerf load generator?

please test the latest GNMT code with the LoadGen & re-open the issue if needed. Thanks @ukurkure