Repository: mlcommons/inference
Issue: Benchmarking image classification on RPi failing with this error: FileNotFoundError: [Errno 2] No such file or directory: 'tmp-ck-timer.json'
Root cause: Documentation Deficiency

Hi,

I am trying to get image classification numbers for TFLite on RPi 4, and I am following instructions from here:
https://github.com/ARM-software/armnn-mlperf#preprocess-on-an-x86-machine-and-detect-on-an-arm-dev-board

I am able to run successfully with the 500 imagenet dataset, but when I try the full dataset (50000 images), with this command:

ck benchmark program:image-classification-tflite --repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=50000 --record --record_repo=local --record_uoa=mlperf-mobilenet-tflite-accuracy-50000 --tags=image-classification,mlperf,mobilenet,tflite,accuracy,50000 --skip_print_timers --skip_stat_analysis --process_multi_keys

it fails with the following error:

  File "/home/pi/CK/ck-mlperf/script/image-classification/postprocess.py", line 198, in ck_postprocess
    with open('tmp-ck-timer.json', 'r') as o:
FileNotFoundError: [Errno 2] No such file or directory: 'tmp-ck-timer.json'

I have the full error attached in this issue.
[errorSummary.txt](https://github.com/mlperf/inference/files/5418970/errorSummary.txt)

I faced another error before this while registering the full imagenet dataset (after preprocessing on an x86 machine and copying it over to RPi). I used this command to register:

ck detect soft --tags=dataset,imagenet,preprocessed,rgb8 --extra_tags=using-opencv \
--full_path=<dataset-imagenet-preprocessed-using-opencv-dir>/ILSVRC2012_val_00000001.JPEG

Could you please help me navigate these errors?

Thank you for your time,
Sindhuja

Hi @sindhujapr, the error you are seeing is only a symptom of the `ck benchmark` command failing. We need to understand the cause. Mostly likely, it's because you haven't registered the full preprocessed dataset properly. 

Is your copy of preprocessed ImageNet really at the root directory (as suggested by `/ILSVRC2012_val_00000001.JPEG`), or is it under some directory? For example, if it's under e.g. `/datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224`, the command should be:

```bash
$ ck detect soft --tags=dataset,imagenet,preprocessed,rgb8 --extra_tags=using-opencv \
--full_path=/datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224/ILSVRC2012_val_00000001.rgb8
```
(Also, note that the extension is `.rgb8`, not `.JPEG`.)

Hi @psyhtest ,
Thanks for your quick response. You are correct, I did get errors when I tried to register the full imagenet dataset.

To try using files in JPEG format directly, I tried this:

ck detect soft --tags=dataset.imagenet,preprocessed,JPEG --extra_tags=using-opencv --full_path=/home/pi/preprocessedImagenet/local/imn_usb_disk_03/imagenet/raw_data/validation/ILSVRC20212_val_00000001.JPEG
CK error: [soft] software entry was not found!

I am guessing I have to convert JPG to RGB8 to register on RPi?

ck detect soft --tags=dataset,imagenet,preprocessed,rgb8 --extra_tags=using-opencv \
> --full_path=/home/pi/preprocessedImagenet/local/imn_usb_disk_03/imagenet/raw_data/validation/ILSVRC2012_val_00000001.rgb8

 Searching for ImageNet dataset (preprocessed uint8) (ILSVRC2012_val_00000001.rgb8) to automatically register in the CK - it may take some time, please wait ...


  Detecting and sorting versions (ignore some work output) ...

    * /home/pi/preprocessedImagenet/local/imn_usb_disk_03/imagenet/raw_data/validation/ILSVRC2012_val_00000001.rgb8
        WARNING: version was not detected


  Registering in the CK (/home/pi/preprocessedImagenet/local/imn_usb_disk_03/imagenet/raw_data/validation/ILSVRC2012_val_00000001.rgb8) ...

  Software entry found: dataset.imagenet.preprocessed (c2afe53b5d17f38c)
CK error: [soft] software not found in a specified path (/home/pi/preprocessedImagenet/local/imn_usb_disk_03/imagenet/raw_data/validation/ILSVRC2012_val_00000001.rgb8)!


Thank you,
Sindhuja

Currently, the images I have are in JPEG format (images stored as uint8). Do you have a suggested way of converting it to .rgb8 (quantized and normalized?)

Hi @sindhujapr , I thought you had successfully resolved this [before](https://github.com/mlperf/inference/issues/622#issuecomment-651999050)?

The steps are still the same:

1. [Detect](https://github.com/mlperf/inference/tree/master/vision/classification_and_detection/optional_harness_ck/classification#install-the-full-dataset-50000-images) e.g.:
```bash
x86-host $ ck detect soft:dataset.imagenet.val --full_path=<imagenet-dir>/ILSVRC2012_val_00000001.JPEG
```

2. [Preprocess](https://github.com/ARM-software/armnn-mlperf#full-50000-images) e.g.:
```bash
x86-host $ ck install package --tags=dataset,imagenet,val,preprocessed,full
x86-host $ ck locate env --tags=dataset,imagenet,val,preprocessed,full
```

3. [Copy and detect](https://github.com/ARM-software/armnn-mlperf#preprocess-on-an-x86-machine-and-detect-on-an-arm-dev-board)
```bash
arm-device $ ck detect soft --tags=dataset,imagenet,preprocessed,rgb8 --extra_tags=using-opencv \
--full_path=<imagenet-preprocessed-using-opencv-dir>/ILSVRC2012_val_00000001.rgb8
```

Hi @psyhtest,

I had it resolved when I worked with a subset of the imagenet dataset (500 images). It still works for that scenario. However, it is failing with the above mentioned errors when I work with the full imagenet dataset.

When I do the prerocessing step and transfer thef iles in batches from an x86 machine to RPi, the images are in JPEG format, not .rbg8. That is probably what is causing issues. Maybe the preprocessing was not done correctly?

Let me clear up the existing format, redo these steps to look at it again from a clean slate, and get back to you.

The preprocessing step should take in `*.JPEG` images and generate `.rgb8` files. Please double check you archive the right directory.

Is this still an issue? @sindhujapr 

I haven't checked on the pre-processing step yet, but I think I have sufficient information for the error. I will close this issue.

Thank you for all you help!