Repository: mlcommons/inference
Issue: LoadGen ERROR: mlperf_loadgen-0.5a0-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform.
Root cause: Environment Incompatibility

Hi all, 

Is the load generator ready to use to run automatically the MLPerf inferences? If so, what are the models already integrated to it? 

Also, I have tried to build the LoadGen as a python module and as a C++ library but I got errors with both methods:


**To build as a python module:**
$ /mlperf_inference/loadgen# sudo -H pip install dist/mlperf_loadgen-0.5a0-cp27-cp27mu-linux_x86_64.whl
`ERROR: mlperf_loadgen-0.5a0-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform.
`



**To build as a C++ library:**
$ /mlperf_inference# make mlperf_loadgen
```
cd third_party/ninja && python configure.py --bootstrap
bootstrapping ninja...
warning: A compatible version of re2c (>= 0.11.3) was not found; changes to src/*.in.cc will not affect your build.
wrote build.ninja.
bootstrap complete.  rebuilding...
[25/25] LINK ninja
# Generate gn's ninja build file
python third_party/gn/build/gen.py
# Build gn
third_party/ninja/ninja -C third_party/gn/out
ninja: Entering directory `third_party/gn/out'
[1/246] CXX tools/gn/inherited_libraries.o
FAILED: tools/gn/inherited_libraries.o
clang++ -MMD -MF tools/gn/inherited_libraries.o.d  -I/home/mlperf_inference/third_party/gn -I/home/mlperf_inference/third_party/gn/out -DNDEBUG -O3 -fdata-sections -ffunction-sections -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -pthread -pipe -fno-exceptions -fno-rtti -fdiagnostics-color -std=c++14 -Wno-c++11-narrowing -c /home/mlperf_inference/third_party/gn/tools/gn/inherited_libraries.cc -o tools/gn/inherited_libraries.o
/bin/sh: 1: clang++: not found
[2/246] CXX base/callback_internal.o
..... a bunch of FAILED messages
```


Some help how to fix the issue?


right now all code should use python3 by default. Can you try python3? Thanks!

hi @christ1ne. Yes I have tried with python3 too, but I don't have the file mlperf_loadgen-0.5a0-cp3X-cp3Xmu-linux_x86_64.whl which uses python3

I don't follow the instructions and instead do ```python3 setup.py install``` which has been working well for me.

Thanks @christ1ne , @guschmue I ran again the instructions replacing with python3, it said that mlperf-loadgen-0.5a0 was installed successfully. However, I don't see how to run the inference benchmarks with the LoadGen, the command ` python3 demos/py_demo.py` doesn't produce any output, does it run all the models automatically? 

```
$ CFLAGS="-std=c++14" python3 setup.py bdist_wheel
$ pip3 install dist/mlperf_loadgen-0.5a0-cp35-cp35m-linux_x86_64.whl
$ python3 demos/py_demo.py
```



https://github.com/mlperf/inference/tree/master/cloud/image_classification
is using loadgen, single stream mode for now.

I have successfully installed it before. This problem occurred after the code was updated and recompiled for installation.
mlperf_loadgen-0.5a0-cp37-cp37m-linux_x86_64.whl is not a supported wheel on this platform.
how to fix the issue?


My problem has been solved. The python version causes this problem.

> https://github.com/mlperf/inference/tree/master/cloud/image_classification
> is using loadgen, single stream mode for now.

Thanks @guschmue, is the image_classification the only benchmark merged with latest loadgen?

I belief so. We are in the process to add ssd to image_classification so maybe we can cover those with the same app.

> https://github.com/mlperf/inference/tree/master/cloud/image_classification
> is using loadgen, single stream mode for now.

thanks @guschmue, I ran the test `python3 demos/py_demo_single_stream.py` and got the below output:
```
Average latency:
1181.7272925547734
Median latency:
1170.0
90 percentile latency:
1224.0
```

Later on, I ran the test `/mlperf_inference/cloud/image_classification# ./run_and_time.sh tf gpu` and got this error:
```
STARTING RUN AT 2019-05-08 03:52:34 PM
Traceback (most recent call last):
  File "python/main.py", line 22, in <module>
    import coco
  File "/mlperf/python/coco.py", line 14, in <module>
    from pycocotools.cocoeval import COCOeval
ModuleNotFoundError: No module named 'pycocotools'
ENDING RUN AT 2019-05-08 03:52:35 PM
```

Some help on how to fix it?



I need to update the readme - ```pip install pycocotools``` will fix it for now.

@guschmue, pycocotools issue resolved with its installation, now I am getting this error: 


```
STARTING RUN AT 2019-05-08 07:07:01 PM
INFO:main:Namespace(backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/data', inputs=['input_tensor:0'], max_latency=[0.01, 0.05, 0.1, 0.2, 0.4], model='/data/resnet50_v1.pb', output='/output/results.json', outputs=['ArgMax:0'], profile='resnet50-tf', qps=10, threads=112, time=128)
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/opt/anaconda3/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/opt/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "python/main.py", line 355, in <module>
    main()
  File "python/main.py", line 262, in main
    backend = get_backend(args.backend)
  File "python/main.py", line 141, in get_backend
    from backend_tf import BackendTensorflow
  File "/mlperf/python/backend_tf.py", line 7, in <module>
    import tensorflow as tf
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/opt/anaconda3/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/opt/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
ENDING RUN AT 2019-05-08 07:07:04 PM
```

If I run  `./run_and_time.sh tf gpu` without `/mlperf/run_helper.sh 2>&1 | tee $OUTPUT_DIR/output.txt` it takes me inside the docker but doesn't execute any command after that.
some help please?




I think this means the tensorflow build installed on your box does not find all cuda components it needs ... something wrong with your tensorflow install.

For ```./run_and_time.sh tf gpu``` I forgot to add the ```pip install pycocotools``` to Dockerfile.gpu and will fix it today. With that fix ```./run_and_time.sh tf gpu``` is happy on my box:
```
Step 1/15 : FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04
 ---> dbb8e0883fc2
Step 2/15 : ENV PYTHON_VERSION=3.6
...
Step 16/16 : ENTRYPOINT ["/bin/bash"]
 ---> Running in 96cf7e793ff7
Removing intermediate container 96cf7e793ff7
 ---> 95cbed3c88f2
Successfully built 95cbed3c88f2
Successfully tagged mlperf-infer-imgclassify-gpu:latest
Clearing caches.
3
STARTING RUN AT 2019-05-08 08:09:01 PM
INFO:main:Namespace(backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/data', inputs=['input_tensor:0'], max_latency=[0.01, 0.05, 0.1, 0.2, 0.4], model='/data/resnet50_v1.pb', output='/output/results.json', outputs=['ArgMax:0'], profile='resnet50-tf', qps=10, threads=8, time=128)
...
```


@guschmue, in the run_and_time.sh file for some reason the variable `dockercmd=nvidia-docker` wasn't working, so I typed it explicitly when running the GPU docker image and it worked, see below:
```
nvidia-docker run -e opts="$opts" \
    -v $DATA_DIR:/data -v $MODEL_DIR:/model -v `pwd`:/mlperf \
    -v $OUTPUT_DIR:/output -v /proc:/host_proc \
    -t $image:latest /mlperf/run_helper.sh 2>&1 | tee $OUTPUT_DIR/output.txt
```

On another hand, once inside the docker image (running the benchmark) the system threw the error: 
```
Traceback (most recent call last):
  File "python/main.py", line 355, in <module>
    main()
  File "python/main.py", line 278, in main
    model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
  File "/mlperf/python/backend_tf.py", line 38, in load
    with open(model_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/data/resnet50_v1.pb'
```

So, in the same run_and_time.sh file I updated the variable to `model_path_in_container="/model/$(basename $model_path)"` and it worked. I think it should be updated since the volume mounted for the model was ` -v $MODEL_DIR:/model`. Is it ok?


Hi @guschmue, the image_classification test using Loadgen was completed successfully. Where can I find the description of the produced metrics qps, mean and percentiles 50.0-99.9?. Thanks again, Vilmara



yeah, the docker images had some issues. https://github.com/mlperf/inference/pull/115 should fix this and updates to newer versions at the same time.

The qps and percentiles we will be reporting is still under discussion and not on paper yet. What we log right now is what I'd normally use: queries/sec, mean 50, 90, 99, 99.9 percentiles (in sec).

Hi @guschmue thanks for the update, the commands in the docker files should be using python3 by default, right?

Also,  I have tried to run the benchmark with CPU only `./run_and_time.sh tf cpu` but got the below error, some tip on how to fix it?:
```
Successfully built f20c51de5d5f
Successfully tagged mlperf-infer-imgclassify_cu9loadgen-cpu:latest
Clearing caches.
3
STARTING RUN AT 2019-05-10 02:53:34 PM
Traceback (most recent call last):
  File "python/main.py", line 17, in <module>
    import mlperf_loadgen as lg
ImportError: dynamic module does not define module export function (PyInit_mlperf_loadgen)
ENDING RUN AT 2019-05-10 02:53:34 PM

```



loadgen changed api - https://github.com/mlperf/inference/pull/120 reflects those changes.
Should work again.

hi @guschmue< I am trying again with the new API but getting this error, some tip?

`CFLAGS="-std=c++14" python3 setup.py bdist_wheel`

Error:
```
build/temp.linux-x86_64-3.5/logging.o:/home/mlperf_inference/loadgen/logging.cc:315: first defined here
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-g++' failed with exit status 1
```

one workaround:
pip install pybind11 -> use the offical pybind rather than the 3rd party

 in setup.py
 
sources = [
    "bindings/python_api.cc",
    "generated/version.cc",
    "loadgen.cc",
    "logging.cc",
]
 
should become:
 
sources = [
    "bindings/python_api.cc",
]

hi @christ1ne, thanks for your prompt reply. It worked for one of the servers; however, I got this new error with the next server, some tip on how to solve this one:

```
$ pip install pybind11
Requirement already satisfied: pybind11 in /usr/local/lib/python3.6/dist-packages (2.2.4)
```

`$ CFLAGS="-std=c++14" python3 setup.py bdist_wheel`

**error:**
```
running bdist_wheel
running build
running build_ext
building 'mlperf_loadgen' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c loadgen.cc -o build/temp.linux-x86_64-3.6/loadgen.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c logging.cc -o build/temp.linux-x86_64-3.6/logging.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c generated/version.cc -o build/temp.linux-x86_64-3.6/generated/version.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c bindings/python_api.cc -o build/temp.linux-x86_64-3.6/bindings/python_api.o
In file included from ../third_party/pybind/include/pybind11/pytypes.h:12:0,
                 from ../third_party/pybind/include/pybind11/cast.h:13,
                 from ../third_party/pybind/include/pybind11/attr.h:13,
                 from ../third_party/pybind/include/pybind11/pybind11.h:44,
                 from ../third_party/pybind/include/pybind11/functional.h:12,
                 from bindings/python_api.cc:11:
`../third_party/pybind/include/pybind11/detail/common.h:112:20: fatal error: Python.h: No such file or directory`
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
```



Hi @guschmue / @christ1ne I am getting the below error building loadgen for pyhon module:

```
x86_64-linux-gnu-gcc: error: test_settings_internal.ccversion.cc: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
```

Thanks, 

V.S.

can you try the commit which I tested? please see https://github.com/mlperf/inference/issues/127


Hi @christ1ne, thanks for your reply. I see this commit is for edge/object_classification and I am using the cloud/image_classification, does it work for both benchmarks?

That commit is just for showing the version of the master which I tested, not for bug fixing. Hope it helps!

hi @christ1ne / @guschmue, I have tried several options in several machines but still getting the same error:

```
x86_64-linux-gnu-gcc: error: test_settings_internal.ccversion.cc: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
```

Possible that there was some issue in the past.
In master I have not seen issues for some time now, should work.
If find it easiest to do:
```
cd loadgen
pip install pybind11
python setup.py develop
````


hi @christ1ne / @guschmue, thanks for your recommendation. I ran the test with accuracy=true and it dropped to zero in the second iteration, do you have an idea why is this happening?

```
STARTING RUN AT 2019-05-30 06:26:23 PM
INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/home/user1/CK-TOOLS/dataset-im                     agenet-ilsvrc2012-val/', inputs=['input_tensor:0'], max_latency=[0.01, 0.05, 0.1, 0.2, 0.4], model='/home/user1/tensorflow_models/resnet50_v1.pb', output='/output/results.json', ou                     tputs=['ArgMax:0'], profile='resnet50-tf', qps=10, scenario=[TestScenario.SingleStream], threads=80, time=128)
INFO:imagenet:loaded 50000 images, cache=0, took=1.4sec
2019-05-30 18:26:26.338539: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FM                     A
2019-05-30 18:26:26.392210: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500000000 Hz
2019-05-30 18:26:26.400339: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55830ae1cda0 executing computations on platform Host. Devices:
2019-05-30 18:26:26.400389: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
INFO:main:warmup ...
INFO:main:starting accuracy pass on 50000 items
INFO:main:starting accuracy pass on 50000 items
Accuracy qps=96.36, mean=0.829631, time=518.88, acc=73.87, queries=50000, tiles=50.0:0.8235,80.0:0.8993,90.0:0.9426,95.0:0.9814,99.0:1.0664,99.9:1.2125
INFO:main:starting TestScenario.SingleStream, latency=0.01
TestScenario.SingleStream-0.01 qps=18.95, mean=0.052680, time=67.56, acc=0.00, queries=1280, tiles=50.0:0.0526,80.0:0.0562,90.0:0.0577,95.0:0.0588,99.0:0.0608,99.9:0.0622
INFO:main:starting TestScenario.SingleStream, latency=0.05
TestScenario.SingleStream-0.05 qps=18.97, mean=0.052619, time=67.46, acc=0.00, queries=1280, tiles=50.0:0.0526,80.0:0.0562,90.0:0.0576,95.0:0.0588,99.0:0.0610,99.9:0.0624
INFO:main:starting TestScenario.SingleStream, latency=0.1
TestScenario.SingleStream-0.1 qps=18.89, mean=0.052847, time=67.75, acc=0.00, queries=1280, tiles=50.0:0.0528,80.0:0.0563,90.0:0.0579,95.0:0.0591,99.0:0.0611,99.9:0.0722
INFO:main:starting TestScenario.SingleStream, latency=0.2
TestScenario.SingleStream-0.2 qps=18.94, mean=0.052703, time=67.57, acc=0.00, queries=1280, tiles=50.0:0.0525,80.0:0.0563,90.0:0.0579,95.0:0.0591,99.0:0.0608,99.9:0.0691
INFO:main:starting TestScenario.SingleStream, latency=0.4
TestScenario.SingleStream-0.4 qps=18.96, mean=0.052669, time=67.52, acc=0.00, queries=1280, tiles=50.0:0.0526,80.0:0.0562,90.0:0.0577,95.0:0.0588,99.0:0.0607,99.9:0.0622
ENDING RUN AT 2019-05-30 06:40:48 PM
user1@r740xd:~/mlperf_inference/cloud/image_classification$ sudo ./run_and_time.sh tf cpu --max-latency 0.5
```