Repository: mlcommons/inference
Issue: null backend fails at offline mode
Root cause: Validation Gap

$ ./run_local.sh tf resnet50 gpu --scenario Offline --backend null


INFO:main:Namespace(accuracy=False, backend='null', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/home/t/CK-TOOLS/dataset-imagenet-ilsvrc2012-val/', inputs=['input_tensor:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], model='/home/t/GitHub/inference/v0.5/classification_and_detection//resnet50_v1.pb', output='/home/t/GitHub/inference/v0.5/classification_and_detection/output/resnet50-tf-gpu/results.json', outputs=['ArgMax:0'], profile='resnet50-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=12, time=None)
INFO:imagenet:loaded 200 images, cache=0, took=0.0sec
INFO:main:starting TestScenario.Offline

]
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "python/main.py", line 327, in handle_tasks
    self.run_one_item(qitem)
  File "python/main.py", line 286, in run_one_item
    bi = array.array("B", np.array(processed_results[idx], np.float32).tobytes()).buffer_info()
IndexError: list index out of range

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "python/main.py", line 327, in handle_tasks
    self.run_one_item(qitem)
  File "python/main.py", line 286, in run_one_item
    bi = array.array("B", np.array(processed_results[idx], np.float32).tobytes()).buffer_info()
IndexError: list index out of range

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "python/main.py", line 327, in handle_tasks
    self.run_one_item(qitem)
  File "python/main.py", line 286, in run_one_item
    bi = array.array("B", np.array(processed_results[idx], np.float32).tobytes()).buffer_info()
IndexError: list index out of range

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/t/anaconda3/envs/tf_gpu/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "python/main.py", line 327, in handle_tasks
    self.run_one_item(qitem)
  File "python/main.py", line 286, in run_one_item
    bi = array.array("B", np.array(processed_results[idx], np.float32).tobytes()).buffer_info()
IndexError: list index out of range

.....
similar error messages.