Repository: mlcommons/inference
Issue: Error installing Loadgen: "command 'x86_64-linux-gnu-gcc' failed with exit status 1"
Root cause: Environment Incompatibility

Hi all, 

I'm trying to install MLPerf Inference Benchmarks locally without docker and I get some issue while trying to install Loadgen: 

I'm currently running this on: 
- Ubuntu 18.04 ( with all latest upgraded installed )
- Python 3.7.5

And I get the following output when running the command `CFLAGS="-std=c++14" python setup.py develop --user` : 

``````
running develop
running egg_info
writing mlperf_loadgen.egg-info/PKG-INFO
writing dependency_links to mlperf_loadgen.egg-info/dependency_links.txt
writing top-level names to mlperf_loadgen.egg-info/top_level.txt
reading manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
writing manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
running build_ext
building 'mlperf_loadgen' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c loadgen.cc -o build/temp.linux-x86_64-3.7/loadgen.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c logging.cc -o build/temp.linux-x86_64-3.7/logging.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c test_settings_internal.cc -o build/temp.linux-x86_64-3.7/test_settings_internal.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c utils.cc -o build/temp.linux-x86_64-3.7/utils.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c version.cc -o build/temp.linux-x86_64-3.7/version.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c bindings/python_api.cc -o build/temp.linux-x86_64-3.7/bindings/python_api.o
bindings/python_api.cc:26:10: fatal error: pybind11/functional.h: No such file or directory
 #include "pybind11/functional.h"

compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
``````

Do you have any ideas for me ?
Herve

Hi @guilo0, you probably need to install `pybind` first: 
```
$ sudo apt install pybind11-dev
```
and try again.

But do you know you can build LoadGen C++ with CK as follows:
```
$ ck install package --tags=mlperf,loadgen,static
```

See this thread: https://github.com/mlperf/inference/issues/230

The conclusion was that pybind is not needed if you install with CMake (which is the way CK uses).

I tried both solution but unfortunatly both failed :

after pybind reinstallation 
``````
running develop
running egg_info
writing mlperf_loadgen.egg-info/PKG-INFO
writing dependency_links to mlperf_loadgen.egg-info/dependency_links.txt
writing top-level names to mlperf_loadgen.egg-info/top_level.txt
reading manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
writing manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
running build_ext
building 'mlperf_loadgen' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c loadgen.cc -o build/temp.linux-x86_64-3.7/loadgen.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c logging.cc -o build/temp.linux-x86_64-3.7/logging.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c test_settings_internal.cc -o build/temp.linux-x86_64-3.7/test_settings_internal.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c utils.cc -o build/temp.linux-x86_64-3.7/utils.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c version.cc -o build/temp.linux-x86_64-3.7/version.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.7m -c bindings/python_api.cc -o build/temp.linux-x86_64-3.7/bindings/python_api.o
bindings/python_api.cc:179:16: error: expected constructor, destructor, or type conversion before ‘(’ token
 PYBIND11_MODULE(mlperf_loadgen, m) {
                ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
``````
using CK:

``````
Installing to /home/guillouh/CK-TOOLS/lib-mlperf-loadgen-static-gcc-7.4.0-compiler.python-3.7.5-linux-64

LoadGen source directory: /home/guillouh/loadgen
LoadGen build directory: /home/guillouh/CK-TOOLS/lib-mlperf-loadgen-static-gcc-7.4.0-compiler.python-3.7.5-linux-64/build
/usr/bin/cmake "/home/guillouh/loadgen" -DCMAKE_C_COMPILER="/usr/bin/gcc-7" -DCMAKE_C_FLAGS=" " -DCMAKE_CXX_COMPILER="/usr/bin/g++-7" -DCMAKE_CXX_FLAGS=" -std=c++14 " -DCMAKE_AR="/usr/bin/ar" -DCMAKE_RANLIB="" -DCMAKE_LINKER="" -DPYTHON_EXECUTABLE="/usr/bin/python3.7" -DCMAKE_INSTALL_PREFIX="/home/guillouh/CK-TOOLS/lib-mlperf-loadgen-static-gcc-7.4.0-compiler.python-3.7.5-linux-64"

CMake Error: The source directory "/home/guillouh/loadgen" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   CK detected a PROBLEM in the third-party CK package:

   CK package:           lib-mlperf-loadgen-static
   CK repo:              ck-mlperf
   CK repo URL:          https://github.com/ctuning/ck-mlperf
   CK package URL:       https://github.com/ctuning/ck-mlperf/tree/master/package/lib-mlperf-loadgen-static
   Issues URL:           https://github.com/ctuning/ck-mlperf/issues
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Please, submit the log to the authors of this external CK package at "https://github.com/ctuning/ck-mlperf/issues" to collaboratively fix this problem!

CK error: [package] package installation failed!
``````

We have followed up the CK issue offline (via the CodeReef Slack), and it turned out to be the case of CK detecting a previously cloning repository. Without it, everything works as expected.

Thank you @psyhtest for your support!

You are welcome!