Repository: mlcommons/inference
Issue: Object detection: postprocessing issue 
Root cause: Environment Incompatibility

Hi,
I'm trying to run the object detection on a Linux18.04 using the following command:
```
ck benchmark program:object-detection-tf-py \
--repetitions=10 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=2 --env.CK_METRIC_TYPE=COCO \
--record --record_repo=local --record_uoa=mlperf-object-detection-ssd-mobilenet-tf-py-performance \
--tags=mlperf,object-detection,ssd-mobilenet,tf-py,performance \
--skip_print_timers --skip_stat_analysis --process_multi_keys
```

Everything is running in a correct way until the post processing where i get the following error: 

```
 ***************************************************************************************
Attempting to set CPU frequency to ondemand to "calm" system (if supported) ...


CMD to set CPU frequency:
  ck-set-cpu-ondemand
sh: 1: ck-set-cpu-ondemand: not found

  Warning: setting frequency possibly failed - return code 32512
***************************************************************************************
Attempting to set GPU frequency to ondemand to "calm" system (if supported) ...


CMD to set GPU frequency:
  ck-set-gpu-ondemand
sh: 1: ck-set-gpu-ondemand: not found

  Warning: setting frequency possibly failed - return code 32512

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   CK detected a PROBLEM in the third-party CK program workflow:

   Failed(?) CK program: object-detection-tf-py
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Please, submit the log to the community of this external CK program workflow at "https://groups.google.com/forum/#!forum/collective-knowledge" to collaboratively fix this problem!

***************************************************************************************
Pipeline failed (execution failed - problem loading python code: can't load module code (path=/home/guillouh/CK/ck-tensorflow/program/object-detection-tf-py/, name=postprocess, err=No module named 'pycocotools._mask'))!
***************************************************************************************
Done!
***************************************************************************************
Some statistics:

* Failed: yes
* Reason: execution failed - problem loading python code: can't load module code (path=/home/guillouh/CK/ck-tensorflow/program/object-detection-tf-py/, name=postprocess, err=No module named 'pycocotools._mask')

* Binary size: 0
* Object size: 0

* Kernel repeat: 0

* Normalized time in sec. (min .. max): 0 .. 0

* Total time in us (min .. max): 0 .. 0
```


Hi @guilo0, thank you for your interest in MLPerf Inference and (unofficial) [Collective Knowledge](http://cknowledge.org) workflows! Now, we haven't paid much attention to this [TensorFlow-Python workflow](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/detection/tf-py) since @guschmue created the [official reference Python application](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection), but still have used it relatively recently to [omni-benchmark object detection models](http://bit.ly/ob-od). 

My first hunch is that something wrong with your Cython installation. I'll try to reproduce on a clean platform and get back to you ASAP.

Some undocumented things I had to do on TX1 with Jetpack 4.2.2:
```
$ sudo apt-get install libfreetype6-dev
```

By the way, did you know you could build or download a [Docker image](https://github.com/ctuning/ck-object-detection/tree/master/docker/object-detection-tf-py.tensorrt.ubuntu-18.04) with all the system dependencies unambiguously resolved?

@guilo0 Please try to reinstall the COCO Python API with Cython installed:
```
$ ck clean env --tags=tool,coco,api --force
$ ck install package --tags=lib,python-package,cython
$  ck install package --tags=tool,coco,api
...
# install pycocotools locally                         
/usr/bin/python3.6 setup.py build_ext --inplace              
running build_ext                        
cythoning pycocotools/_mask.pyx to pycocotools/_mask.c               
/home/anton/CK_TOOLS/lib-python-cython-compiler.python-3.6.8-linux-64/build/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change i
n a later release! File: /home/anton/CK_TOOLS/tool-coco-master-gcc-7.4.0-compiler.python-3.6.8-linux-64/src/PythonAPI/pycocotools/_mask.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)                                                       
building 'pycocotools._mask' extension
creating build                                                                                                                  
creating build/common                
creating build/temp.linux-aarch64-3.6                                                            
creating build/temp.linux-aarch64-3.6/pycocotools
gcc-7 -DNDEBUG -g -fwrapv -O2 -Wall -Wno-misleading-indentation -Wno-maybe-uninitialized -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/anton/CK_TOOLS/lib-python-matplotlib-compiler.python-3.6.8-linux-64/b
uild/numpy/core/include -I../common -I/usr/include/python3.6m -c ../common/maskApi.c -o build/temp.linux-aarch64-3.6/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
gcc-7 -DNDEBUG -g -fwrapv -O2 -Wall -Wno-misleading-indentation -Wno-maybe-uninitialized -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/anton/CK_TOOLS/lib-python-matplotlib-compiler.python-3.6.8-linux-64/b
uild/numpy/core/include -I../common -I/usr/include/python3.6m -c pycocotools/_mask.c -o build/temp.linux-aarch64-3.6/pycocotools/_mask.o -Wno-cpp -Wno-unused-function -std=c99
creating build/lib.linux-aarch64-3.6                                 
creating build/lib.linux-aarch64-3.6/pycocotools
aarch64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -Wno-misleading-indentation -Wno-maybe-uninitialized -Wd
ate-time -D_FORTIFY_SOURCE=2 build/temp.linux-aarch64-3.6/../common/maskApi.o build/temp.linux-aarch64-3.6/pycocotools/_mask.o -o build/lib.linux-aarch64-3.6/pycocotools/_mask.cpython-36m-aarch64-linux-gn
u.so                                                         
copying build/lib.linux-aarch64-3.6/pycocotools/_mask.cpython-36m-aarch64-linux-gnu.so -> pycocotools
rm -rf build                                                        
                                                                                                                                                                                                            
Installed 'tool-coco' to '/home/anton/CK_TOOLS/tool-coco-master-gcc-7.4.0-compiler.python-3.6.8-linux-64'.
```

Thank you fr your fast answering.
I follow all your steps and i'm coming out with such error:

```````
$ ck install package --tags=tool,coco,api
# install pycocotools locally
/usr/bin/python3.7 setup.py build_ext --inplace
running build_ext
cythoning pycocotools/_mask.pyx to pycocotools/_mask.c
/usr/local/lib/python3.7/dist-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/guillouh/CK-TOOLS/tool-coco-master-gcc-7.4.0-compiler.python-3.7.3-linux-64/src/PythonAPI/pycocotools/_mask.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
building 'pycocotools._mask' extension
creating build
creating build/common
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/pycocotools
gcc-7 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -flto -fuse-linker-plugin -ffat-lto-objects -Wno-misleading-indentation -Wno-maybe-uninitialized -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.7/dist-packages/numpy/core/include -I../common -I/usr/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-3.7/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
gcc-7 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -flto -fuse-linker-plugin -ffat-lto-objects -Wno-misleading-indentation -Wno-maybe-uninitialized -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.7/dist-packages/numpy/core/include -I../common -I/usr/include/python3.7m -c pycocotools/_mask.c -o build/temp.linux-x86_64-3.7/pycocotools/_mask.o -Wno-cpp -Wno-unused-function -std=c99
pycocotools/_mask.c:4:10: fatal error: Python.h: No such file or directory
 #include "Python.h"
          ^~~~~~~~~~
compilation terminated.
error: command 'gcc-7' failed with exit status 1
Makefile:2: recipe for target 'all' failed
make: *** [all] Error 1
Error: make failed!

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   CK detected a PROBLEM in the third-party CK package:

   CK package:           tool-coco
   Native package URL:   https://github.com/cocodataset/cocoapi.git
   CK repo:              ck-env
   CK repo URL:          https://github.com/ctuning/ck-env
   CK package URL:       https://github.com/ctuning/ck-env/tree/master/package/tool-coco
   Issues URL:           https://github.com/ctuning/ck-env/issues
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Please, submit the log to the authors of this external CK package at "https://github.com/ctuning/ck-env/issues" to collaboratively fix this problem!

CK error: [package] package installation failed!
```````

Forget my previous message, Error was due to a bad installation of python3-dev and multiple python version setted on my computer. 

Thank you for your fast help : 

````
Summary:
-------------------------------
Graph loaded in 1.211498s
All images loaded in 0.019801s
All images detected in 0.071618s
Average detection time: 0.071618s
mAP: 0.15445544554455443
Recall: 0.15363636363636363
--------------------------------


  (reading fine grain timers from tmp-ck-timer.json ...)

{
  "avg_fps": 13.9630010619634,
  "avg_time_ms": 71.61784172058105,
  "detection_time_avg_s": 0.07161784172058105,
  "detection_time_total_s": 0.07161784172058105,
  "graph_load_time_s": 1.2114982604980469,
  "images_load_time_avg_s": 0.009900569915771484,
  "images_load_time_total_s": 0.01980113983154297,
  "mAP": 0.15445544554455443,
  "metrics": {
    "DetectionBoxes_Precision/mAP": 0.15445544554455443,
    "DetectionBoxes_Precision/mAP (large)": 0.9499999999999998,
    "DetectionBoxes_Precision/mAP (medium)": 0.0665016501650165,
    "DetectionBoxes_Precision/mAP (small)": 0.0,
    "DetectionBoxes_Precision/mAP@.50IOU": 0.17371737173717366,
    "DetectionBoxes_Precision/mAP@.75IOU": 0.15571557155715565,
    "DetectionBoxes_Recall/AR@1": 0.14454545454545456,
    "DetectionBoxes_Recall/AR@10": 0.15363636363636363,
    "DetectionBoxes_Recall/AR@100": 0.15363636363636363,
    "DetectionBoxes_Recall/AR@100 (large)": 0.95,
    "DetectionBoxes_Recall/AR@100 (medium)": 0.06666666666666667,
    "DetectionBoxes_Recall/AR@100 (small)": 0.0
  },
  "recall": 0.15363636363636363,
  "setup_time_s": 1.2304043769836426,
  "test_time_s": 3.6746273040771484
}
````

You are very welcome @guilo0!

Yes,`python3-dev` is one of the dependencies. In general, I would recommend to take a look at the [Dockerfile](https://github.com/ctuning/ck-object-detection/blob/master/docker/object-detection-tf-py.tensorrt.ubuntu-18.04/Dockerfile) for any further dependencies (as Dockerfiles don't lie!)

It's just because i have a lot of different python version leaving together: python 3.7.3; python 3.6.8 and others older version of python3. So, python3-dev was not installed for version of python used to run the installation.
I am currently working with @gfursin on reproducibility, so I try not to use  dockerfile!

Thank you for your fast help @psyhtest