Repository: mlcommons/inference
Issue: Running into "FileNotFoundError: [Errno 2] No such file or directory: 'tmp-ck-timer.json'" error on running "benchmarking accuracy" command.
Root cause: Environment Incompatibility

Hi!

As a part of benchmarking TFLite performance in MathWorks' products, I am attempting to get the TFLite MLPerfbenchmarks running on a Raspberry Pi 4. I am following instructions from this [page](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/optional_harness_ck/classification/tflite/README.md): 

When I reach the "benchmarking accuracy" section and run this command:

ck benchmark program:image-classification-tflite --speed \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=50000 \
--skip_print_timers --skip_stat_analysis --process_multi_keys --record --record_repo=local \
--record_uoa=mlperf-image-classification-tflite-accuracy-using-opencv-mobilenet-quantized \
--tags=mlperf,image-classification,tflite,accuracy,using-opencv,mobilenet,quantized \
--dep_add_tags.images=preprocessed,using-opencv --dep_add_tags.weights=mobilenet,quantized


I get this error: 

  File "/home/pi/CK/ck-mlperf/script/image-classification/postprocess.py", line                                                                                                                      198, in ck_postprocess
    with open('tmp-ck-timer.json', 'r') as o:
FileNotFoundError: [Errno 2] No such file or directory: 'tmp-ck-timer.json'

"tmp-ck-timer.json" was used in the "benchmarking performance" section just above this section, and that command ran smoothly with no errors. Can you help me get through this error? Is this a path issue?

I have attached a .txt file with the sequence of commands I executed and the full  error message.

Thank you!
[command sequence and error message.txt](https://github.com/mlperf/inference/files/4787407/command.sequence.and.error.message.txt)




Hi @sindhujapr, sorry you've encountered this issue. I suspect a couple of things here can be wrong. 

First, the full ImageNet is [not available](https://github.com/mlperf/inference/issues/542) for download. But if you have a copy, you could have registered it yourself if you followed [this advice](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification#install-the-full-dataset-50000-images). Please confirm.

Second, OpenCV preprocessing is not available on Arm platforms. There's [only a placeholder](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification#detect-datasets-preprocessed-on-a-different-machine) in the instructions you followed, but luckily the instructions [do exist](https://github.com/ARM-software/armnn-mlperf#preprocess-on-an-x86-machine-and-detect-on-an-arm-dev-board) in the Arm repo.

Please let me know how you get on. If it's neither of the issues, please record a complete session with `typescript` or similar.

A bit of advice. When building TFLite for RPi4, please [use](https://github.com/ctuning/ck-tensorflow/tree/master/package/lib-tflite-1.15.0-src-static):
```bash
$ ck install package:lib-tflite-1.15.0-src-static \
--env.EXTRA_CXXFLAGS="-march=armv7-a+neon+vfpv4 -mfpu=neon-vfpv4" 
```

With these flags, we see a 3x execution time improvement with ResNet50 e.g.
```
v1.15: Average classification time: 0.679943s
v1.13: Average classification time: 2.018279s
```

Hi @sindhujapr, has your issue been resolved? We now have an even easier way to build for RPi4 (with 32-bit Debian):
```bash
$ ck pull repo:ck-tensorflow
$ ck install package --tags=lib,tflite,rel.1.15.3,rpi4
```

Hi @psyhtest, sorry about the delay, I have been preoccupied with some other work tasks. No, the issue has not been solved. I tried to use the mini-imagenet dataset with only 500 images (env.CK_BATCH_COUNT=50000 ), and changed the command to do so, and the error still occurred.

I will try the other two steps today and get back to you by tomorrow:
1. Build TFLite using the new command.
2. Process Open CV on another Linux device and copy over the files to the RPi.


HI @psyhtest, 
The accuracy script ran through without any errors. 

As recommended by you, I preprocessed the images on an x86 machine and copied over the files to the RPi. I limited it to 20 images to test out the accuracy script. I will continue working with the 500 imagenet dataset and then the 50,000 imagenet dataset. But it is clear that the error stemmed from ARM not directly supporting OpenCV preprocessing.

I also used the latest TFLite installation command for RPi:
$ ck pull repo:ck-tensorflow
$ ck install package --tags=lib,tflite,rel.1.15.3,rpi4

It is a much clearer command, specifying the target!


Thank you for all your help, and I can close this issue if you don't have any further comments.

Hi @sindhujapr, Glad to hear it's all working for you without a glitch! (as it should be with CK) I'm going to re-haul the instructions and put everything together soon.

That's great! I look forward to the revised documentation, thank you.