Repository: mlcommons/inference
Issue: mlperf_loadgen incompatible function arguments
Root cause: External Dependency Breakage

I use docker to run mlperf:
#run_and_time.sh tf resnet50 gpu

DATA_DIR=mlperf/inference/vision/classification_and_detection/dataset/ILSVRC2012_img_val
MODEL_DIR=mlperf/inference/vision/classification_and_detection/model

#pip list | grep -i mlperf
mlperf -loadgen 1.1

#nvidia-smi 
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A40          Off  | 00000000:A0:00.0 Off |                    0 |
|  0%   67C    P0   103W / 300W |   8587MiB / 46068MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+


I got error:
STARTING RUN AT 2022-05-11 02:24:02 AM
INFO:main:Namespace(accuracy=False, audit_conf='audit.config', backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/opt/mlperf/inference/vision/classification_and_detection/dataset/ILSVRC2012_img_val', debug=False, find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=32, max_latency=None, mlperf_conf='./mlperf.conf', model='/opt/mlperf/inference/vision/classification_and_detection/model/resnet50_v1.pb', model_name='resnet50', output='/output', outputs=['ArgMax:0'], performance_sample_count=None, profile='resnet50-tf', qps=None, samples_per_query=8, scenario='SingleStream', threads=128, time=None, user_conf='user.conf')
INFO:imagenet:loaded 50000 images, cache=0, took=1.6sec
WARNING:tensorflow:From /mlperf/python/backend_tf.py:47: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/tools/strip_unused_lib.py:85: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/tools/optimize_for_inference_lib.py:113: remove_training_nodes (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.remove_training_nodes`
2022-05-11 02:25:09.964820: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-11 02:25:11.091793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 34887 MB memory:  -> device: 0, name: NVIDIA A40, pci bus id: 0000:a0:00.0, compute capability: 8.6
2022-05-11 02:25:12.757393: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8100
2022-05-11 02:25:16.174342: I tensorflow/stream_executor/cuda/cuda_blas.cc:1786] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.

Traceback (most recent call last):
  File "python/main.py", line 571, in <module>
    main()
  File "python/main.py", line 541, in main
    sut = lg.ConstructSUT(issue_queries, flush_queries, process_latencies)
TypeError: ConstructSUT(): incompatible function arguments. The following argument types are supported:
    1. (arg0: Callable[[List[mlperf_loadgen.QuerySample]], None], arg1: Callable[[], None]) -> int

Invoked with: <function main.<locals>.issue_queries at 0x7f9a069a8ea0>, <function main.<locals>.flush_queries at 0x7f9a069a8bf8>, <function main.<locals>.process_latencies at 0x7f9a069a8f28>
ENDING RUN AT 2022-05-11 02:25:17 AM

Ah, I was about to open the issue for this.
After recent [PR 1138](https://github.com/mlcommons/inference/pull/1138), the API is broken and likely *ALL* the reference models we have in this repo got affected. @tjablin FYI.
Fix is to find the line like this and remove the third argument, until official fixes are done:
```
sut = lg.ConstructSUT(issue_queries, flush_queries, process_latencies)
=>
sut = lg.ConstructSUT(issue_queries, flush_queries)
```