Repository: mlcommons/inference
Issue: ValueError: no images in image list found
Root cause: Validation Gap

Trying to run inferencing benchmark using ResNet34 model for COCO dataset.
Have successfully set up the environment, downloaded COCO dataset (both ways, manually and through ck tool), transformed the data into 1200x1200 images using the tool given, downloaded model implementation and set path for data and model directories.
When I finally run the docker container, it is build up successfully but is not able to run the model as it doesnt find any images.
The complete log is as follows:

root@ml:/tmp/inference/v0.5/classification_and_detection# ./run_and_time.sh tf ssd-re                                                                                                                                                         snet34 gpu --accuracy --scenario Offline
Sending build context to Docker daemon  557.1kB
Step 1/17 : FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu16.04
 ---> 7bb1f0b039e1
Step 2/17 : ENV PYTHON_VERSION=3.7
 ---> Using cache
 ---> 0a6f4fbb143a
Step 3/17 : ENV LANG C.UTF-8
 ---> Using cache
 ---> 08676d9e2c08
Step 4/17 : ENV LC_ALL C.UTF-8
 ---> Using cache
 ---> d6761198a019
Step 5/17 : ENV PATH /opt/anaconda3/bin:$PATH
 ---> Using cache
 ---> 653e27f1b372
Step 6/17 : WORKDIR /root
 ---> Using cache
 ---> 703b890e28f1
Step 7/17 : ENV HOME /root
 ---> Using cache
 ---> 22bf809d6bbc
Step 8/17 : RUN apt-get update
 ---> Using cache
 ---> 5c7feb51b499
Step 9/17 : RUN apt-get install -y --no-install-recommends       git       build-esse                                                                                                                                                         ntial       software-properties-common       ca-certificates       wget       curl                                                                                                                                                                htop       zip       unzip
 ---> Using cache
 ---> db6efa0f6404
Step 10/17 : RUN cd /opt &&     wget --quiet https://repo.anaconda.com/miniconda/Mini                                                                                                                                                         conda3-4.6.14-Linux-x86_64.sh -O miniconda.sh &&     /bin/bash ./miniconda.sh -b -p /                                                                                                                                                         opt/anaconda3 &&     rm miniconda.sh &&     /opt/anaconda3/bin/conda clean -tipsy &&                                                                                                                                                              ln -s /opt/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh &&     echo "                                                                                                                                                         . /opt/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc &&     echo "conda activate bas                                                                                                                                                         e" >> ~/.bashrc &&     conda config --set always_yes yes --set changeps1 no
 ---> Using cache
 ---> 28a05c88c67b
Step 11/17 : RUN pip install --upgrade pip &&     pip install cython future pillow on                                                                                                                                                         nx opencv-python-headless &&     ln -s /usr/local/cuda/lib64 /usr/local/cuda/lib &&                                                                                                                                                              cp /usr/lib/x86_64-linux-gnu/libnccl* /usr/local/cuda/lib &&     ldconfig
 ---> Using cache
 ---> 9d668ed2e793
Step 12/17 : RUN conda install pytorch torchvision -c pytorch
 ---> Using cache
 ---> df421d4b485d
Step 13/17 : RUN pip install tensorflow-gpu==1.14 onnxruntime-gpu
 ---> Using cache
 ---> 7b278f991336
Step 14/17 : RUN pip install Cython && pip install pycocotools
 ---> Using cache
 ---> cec0a1f46601
Step 15/17 : RUN cd /tmp &&     git clone https://github.com/mlperf/inference &&                                                                                                                                                              cd inference/loadgen &&     pip install pybind11 &&     CFLAGS="-std=c++14" python se                                                                                                                                                         tup.py install &&     rm -rf mlperf
 ---> Using cache
 ---> ac4f13936c0e
Step 16/17 : RUN  echo "/usr/local/cuda/compat" >> /etc/ld.so.conf.d/cuda-10-0.conf &                                                                                                                                                         &     ldconfig
 ---> Using cache
 ---> fa54e52d1524
Step 17/17 : ENTRYPOINT ["/bin/bash"]
 ---> Using cache
 ---> 1fd64525112e
Successfully built 1fd64525112e
Successfully tagged mlperf-infer-imgclassify-gpu:latest
Clearing caches.
3
STARTING RUN AT 2020-05-20 11:59:18 AM
INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, config='./mlperf.co                                                                                                                                                         nf', count=None, data_format='NCHW', dataset='coco-1200-tf', dataset_list=None, datas                                                                                                                                                         et_path='/data/coco-1200', find_peak_performance=False, inputs=['image:0'], max_batch                                                                                                                                                         size=32, max_latency=None, model='/tmp/model/tf_ssd_resnet34_22.1/resnet34_tf.22.1.pb                                                                                                                                                         ', model_name='ssd-resnet34', output='/output', outputs=['detection_bboxes:0', 'detec                                                                                                                                                         tion_classes:0', 'detection_scores:0'], profile='ssd-resnet34-tf', qps=None, samples_                                                                                                                                                         per_query=None, scenario='Offline', threads=48, time=None)
/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516:                                                                                                                                                          FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a                                                                                                                                                          future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517:                                                                                                                                                          FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a                                                                                                                                                          future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518:                                                                                                                                                          FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a                                                                                                                                                          future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519:                                                                                                                                                          FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a                                                                                                                                                          future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520:                                                                                                                                                          FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a                                                                                                                                                          future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525:                                                                                                                                                          FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a                                                                                                                                                          future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.                                                                                                                                                         py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecate                                                                                                                                                         d; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.                                                                                                                                                         py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecate                                                                                                                                                         d; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.                                                                                                                                                         py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecate                                                                                                                                                         d; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.                                                                                                                                                         py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecate                                                                                                                                                         d; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.                                                                                                                                                         py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecate                                                                                                                                                         d; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.                                                                                                                                                         py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecate                                                                                                                                                         d; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
ERROR:coco:no images in image list found
Traceback (most recent call last):
  File "python/main.py", line 537, in <module>
    main()
  File "python/main.py", line 428, in main
    count=count, **kwargs)
  File "/mlperf/python/coco.py", line 106, in __init__
    raise ValueError("no images in image list found")
ValueError: no images in image list found
ENDING RUN AT 2020-05-20 11:59:27 AM


I have in the past successfully run training benchmarks, so all the prerequisites are successfully installed, like docker and drivers.
I have tried with pytorch and tensorflow versions, both, but both of them are failing with the same error. So the issue is with the data somehow and not the model and the setup environment.

Have you solved it? I got the same problem

It was a long time back, so I dont remember exactly what worked.
Did you check if you have the data in the exact format thats expected? like inside coco folder you have:

1. val2017 subfolder containing all the images
2. annotations subfolder containing annotations

Also, check if you have preprocessed the images right, in 1200x1200 format or 300x300 format, according to the model you are trying to benchmark.
Hope this helps.

> 2\. 
> 
> Also, check if you have preprocessed the images right, i

> It was a long time back, so I dont remember exactly what worked.
> Did you check if you have the data in the exact format thats expected? like inside coco folder you have:
> 
> 1. val2017 subfolder containing all the images
> 2. annotations subfolder containing annotations
> 
> Also, check if you have preprocessed the images right, in 1200x1200 format or 300x300 format, according to the model you are trying to benchmark.
> Hope this helps.

I solved it by change DATA_DIR, but I don't know why
At first ,my DATA_DIR=/data/coco-300
but I got that error.
and
DATA_DIR=/data/coco is right


Oh yes, now I remember!
The issue is that all the models made available through links preprocess the data intrinsically and we dont have to provide preprocessed data!
The scripts are given only if someone tries there own implementation of the models!
So silly of me to not remember that. I moved onto submitted results by others for benchmark so didnt remember "vanilla" mlperf issue. I forgot to add the solution back then only. Good that you found it on your own.
Closing this issue then.