Repository: mlcommons/inference
Issue: `val_map.txt` not found in ImageNet2012 validation dataset when using the MLPerf Benchmark
Root cause: Documentation Deficiency

I am a complete beginner in running or using ML workloads. I am just following the ReadMe instructions in the corresponding benchmark git pages.

I want to run the [MLPerf Inference benchmark](https://github.com/mlcommons/inference) suite on [resnet50-v1.5](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection) [onnx](https://zenodo.org/record/2592612/files/resnet50_v1.onnx) model, using the [ImageNet-2012 validation dataset](https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar).

![image](https://github.com/mlcommons/inference/assets/43134757/d8b09a13-b7cd-456a-909c-5df9f2485458)

I extract the `.tar` file downloaded from the link above. The .tar file contains no directory structure; it just contains many images. (I thought it be grouped in a specific directory structure to represent the classes).

```
$ tar -xf  ILSVRC2012_img_val.tar 
$ ls
ILSVRC2012_val_00000001.JPEG  ILSVRC2012_val_00012501.JPEG  ILSVRC2012_val_00025001.JPEG  ILSVRC2012_val_00037501.JPEG
ILSVRC2012_val_00000002.JPEG  ILSVRC2012_val_00012502.JPEG  ILSVRC2012_val_00025002.JPEG  ILSVRC2012_val_00037502.JPEG
ILSVRC2012_val_00000003.JPEG  ILSVRC2012_val_00012503.JPEG  ILSVRC2012_val_00025003.JPEG  ILSVRC2012_val_00037503.JPEG
ILSVRC2012_val_00000004.JPEG  ILSVRC2012_val_00012504.JPEG  ILSVRC2012_val_00025004.JPEG  ILSVRC2012_val_00037504.JPEG
...
```

I exported the data directory and model directory and then decided to use the Docker container for experimentation.

```
$ ./run_and_time.sh onnxruntime resnet50 gpu
```
The above command successfully builds the docker image and tries to launch the application inside the container, but the application complains about a missing `val_map.txt`. 

```
[+] Building 0.9s (16/16) FINISHED                                                                                             
 => [internal] load build definition from Dockerfile.gpu
...
             TRUNCATED DOCKER BUILD OUTPUT
...                                                               
 => => naming to docker.io/library/mlperf-infer-imgclassify-gpu                                                           0.0s 
Clearing caches.                                                                                                               
3
STARTING RUN AT 2023-07-27 07:23:05 PM
INFO:main:Namespace(accuracy=False, audit_conf='audit.config', backend='onnxruntime', cache=0, cache_dir=None, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/home/abhishek/shweta_machine/DGSF_workloads/Image_Classification/ImageNet_2012_validation', debug=False, find_peak_performance=False, inputs=None, max_batchsize=32, max_latency=None, mlperf_conf='./mlperf.conf', model='/home/abhishek/shweta_machine/DGSF_workloads/Image_Classification/resnet50_v1.onnx', model_name='resnet50', output='/output', outputs=['ArgMax:0'], performance_sample_count=None, preprocessed_dir=None, profile='resnet50-onnxruntime', qps=None, samples_per_query=8, scenario='SingleStream', threads=16, time=None, use_preprocessed_dataset=False, user_conf='user.conf')
Traceback (most recent call last):
  File "python/main.py", line 624, in <module>
    main()
  File "python/main.py", line 501, in main
    **kwargs)
  File "/mlperf/python/imagenet.py", line 53, in __init__
    with open(image_list, 'r') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/abhishek/shweta_machine/DGSF_workloads/Image_Classification/ImageNet_2012_validation/val_map.txt'
ENDING RUN AT 2023-07-27 07:23:05 PM
abhishek@abhishek:classification_and_detection$
```

Looking into the specific section of the codebase I found the following:

```lang-python
        ...
        # input images are in HWC
        self.need_transpose = True if image_format == "NCHW" else False

        self.not_found = 0
        if image_list is None:
            # by default look for val_map.txt
            image_list = os.path.join(data_path, "val_map.txt")  
        with open(image_list, 'r') as fp:                           # <--- line 53
            for count, line in enumerate(fp):
                pass
        count = count + 1
        if not self.count:
            CNT = count
        else:
            CNT = count if count <= self.count else self.count

        os.makedirs(self.cache_dir, exist_ok=True)
        ...
```

I assume the `val_map.txt` is required for getting the mapping of class, which is kind of mentioned in [here](https://github.com/microsoft/UniCL/blob/main/DATA.md).

But I do not have much knowledge regarding this. Please, can anyone help me?


---------------------------------------------------

A previous issue #567 sort of mentioned this. But the solution mentioned there refers to a section which no longer works.

@arjunsuresh Hello, sir. The world is a small place, and I found you providing valuable solutions to other git issues here. Any help or pointers shall be highly appreciated. 

#567 Mentions using the collective minds framework. But that link is broken, apparently.

I tried the following steps, guided by intuition:

```
$ python3 -m pip install ck --user
$ ck --help
WARNING: this framework was discontinued in favor of the new CK2 framework aka CM being developed by the open taskforce on education and reproducibility at MLCommons:
         https://bit.ly/mlperf-edu-wg
 ....
               OUTPUT OF HELP TRUNCATED
 ....
 $ ck pull repo:ck-env
 $ ck install package --tags=image-classification,dataset,imagenet,val

WARNING: this framework was discontinued in favor of the new CK2 framework aka CM being developed by the open taskforce on education and reproducibility at MLCommons:
         https://bit.ly/mlperf-edu-wg


More than one package or version found:

 0) imagenet-2012-val-min-resized  Version 2012  (476be5741f52384f)
 1) imagenet-2012-val-min  Version 2012  (60cb8b2218445c36)
 2) imagenet-2012-val  Version 2012  (14db79a136d98dd4)
 3) dataset-imagenet-preprocessed-using-tensorflow (fac1d0d5f4e69a85)
 4) dataset-imagenet-preprocessed-using-pillow (a6a4613ba6dfd570)
 5) dataset-imagenet-preprocessed-using-opencv (4932bbdd2ac7a17b)

Please select the package to install [ Enter -1 to use CK packages or hit return for "0" ]: 2


  Package found: imagenet-2012-val (14db79a136d98dd4)

=============================================================================================
ImageNet 2012 validation set is no longer publicly available
and this CK meta-package cannot automatically download it!

If you already have it installed on your machine, you can detect
and register it to work with CK workflows using this command:

$ ck detect soft:dataset.imagenet.val --search_dir={directory where the dataset is installed}

You can also download it via Academic Torrents 
at https://academictorrents.com/details/5d6d0df7ed81efd49ca99ea4737e0ae5e3a5f2e5
and then register in the CK using above command
=============================================================================================

CK error: [package] This package has a known problem - see above how to solve it!
```
This `ck` solution seems outdated. And the academic torrents link does not seem to work. 

I guess `cm` is the substitute for `ck`. Might be wrong through. I need to check that.

Nice to meet you here :) 

Yes, CM is the new version of CK and CK is no longer supported. 

Actually, most of the commands you tried are old and no longer working/supported. 

For resnet50 benchmarking you can follow the README given [here](https://github.com/mlcommons/ck/blob/master/docs/mlperf/inference/resnet50/README_reference.md). It is also added as a Github action [here](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-resnet50.yml) (using a smaller sized dataset) and so should be working out of the box. Also feel free to join [here](https://discord.gg/y7hupJsUNb) for live discussions regarding MLPerf and even officially submit the obtained performance numbers.  

What is the `val_map.txt`, and why is it required?
The `val_map.txt` is supposed to store the mapping of each image to the corresponding class.
Since the `val_map.txt` is not provided, why not try to make it?
So, first, let us have a look at the classes in the ImageNet dataset:
![image](https://github.com/mlcommons/inference/assets/43134757/639c962f-cb02-47a7-9ffa-f57df242128a)
Note that the class indices start from `0`.

Now, how to understand the kind of `val_map.txt` the Resnet-50 model in MLPerf wants?
Note that in this [tutorial ipynb](https://github.com/mlperf/inference/blob/master/vision/classification_and_detection/GettingStarted.ipynb), they make use of a fake imagenet dataset using `tools/make_fake_imagenet.sh` script. I tried that and tried to figure out the format of the `val_map.txt`:
```
val/800px-Porsche_991_silver_IAA.jpg 817
val/512px-Cacatua_moluccensis_-Cincinnati_Zoo-8a.jpg 89
val/800px-Sardinian_Warbler.jpg 13
val/800px-7weeks_old.JPG 207
val/800px-20180630_Tesla_Model_S_70D_2015_midnight_blue_left_front.jpg 817
val/800px-Welsh_Springer_Spaniel.jpg 156
val/800px-Jammlich_crop.jpg 233
val/782px-Pumiforme.JPG 285
```
The contents look like the above— image <space> class.
For the ImageNet validation data to class mapping, I used this [script](https://raw.githubusercontent.com/Abhishekghosh1998/MLPerf_ImageNet_val_vap_map_txt/main/valprep.sh)

And then used the following Python [script](https://github.com/Abhishekghosh1998/MLPerf_ImageNet_val_vap_map_txt/blob/main/create_val_map.py) to get our required `val_map.txt`

After that, I got the required `val_map.txt` as shown [here](https://github.com/Abhishekghosh1998/MLPerf_ImageNet_val_vap_map_txt/blob/main/val_map.txt).