Repository: mlcommons/inference
Issue: pybind adventures in the LoadGen land
Root cause: Documentation Deficiency

I've been trying the ["Quick Start"](https://github.com/mlperf/inference/tree/master/loadgen#quick-start) instructions for LoadGen, but got somewhat stuck.

First, I installed the documented Python dependencies in my user space as follows:
```
$ python3 -m pip install absl-py numpy --user
Requirement already satisfied: absl-py in /home/anton/.local/lib/python3.6/site-packages (0.7.1)
Requirement already satisfied: numpy in /home/anton/.local/lib/python3.6/site-packages (1.16.4)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from absl-py) (1.11.0)
```
But the build failed due to not finding a `pybind` C header:
```
$ CFLAGS="-std=c++14" python setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
building 'mlperf_loadgen' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c loadgen.cc -o build/temp.linux-x86_64-3.6/loadgen.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c logging.cc -o build/temp.linux-x86_64-3.6/logging.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c test_settings_internal.cc -o build/temp.linux-x86_64-3.6/test_settings_internal.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c utils.cc -o build/temp.linux-x86_64-3.6/utils.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c version.cc -o build/temp.linux-x86_64-3.6/version.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c bindings/python_api.cc -o build/temp.linux-x86_64-3.6/bindings/python_api.o
bindings/python_api.cc:23:10: fatal error: pybind11/functional.h: No such file or directory
 #include "pybind11/functional.h"
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
```
But the problem remained even after installing `pybind11`:
```
$ python3 -m pip install pybind11 --user
Collecting pybind11
  Downloading https://files.pythonhosted.org/packages/5d/85/c7a8dffda52ce25a8bcfe9a28b6861bdd52da59ae001fdd4173e054b7d9b/pybind11-2.3.0-py2.py3-none-any.whl (147kB)
     |████████████████████████████████| 153kB 2.0MB/s 
Installing collected packages: pybind11
Successfully installed pybind11-2.3.0

$ CFLAGS="-std=c++14" python3 setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
building 'mlperf_loadgen' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c loadgen.cc -o build/temp.linux-x86_64-3.6/loadgen.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c logging.cc -o build/temp.linux-x86_64-3.6/logging.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c test_settings_internal.cc -o build/temp.linux-x86_64-3.6/test_settings_internal.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c utils.cc -o build/temp.linux-x86_64-3.6/utils.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c version.cc -o build/temp.linux-x86_64-3.6/version.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c bindings/python_api.cc -o build/temp.linux-x86_64-3.6/bindings/python_api.o
bindings/python_api.cc:23:10: fatal error: pybind11/functional.h: No such file or directory
 #include "pybind11/functional.h"
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
```
I got a bit further when install the headers via `apt`:
```
$ sudo apt install pybind11-dev

CFLAGS="-std=c++14" python3 setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
building 'mlperf_loadgen' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c loadgen.cc -o build/temp.linux-x86_64-3.6/loadgen.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c logging.cc -o build/temp.linux-x86_64-3.6/logging.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c test_settings_internal.cc -o build/temp.linux-x86_64-3.6/test_settings_internal.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c utils.cc -o build/temp.linux-x86_64-3.6/utils.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c version.cc -o build/temp.linux-x86_64-3.6/version.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.6m -c bindings/python_api.cc -o build/temp.linux-x86_64-3.6/bindings/python_api.o
bindings/python_api.cc:168:16: error: expected constructor, destructor, or type conversion before ‘(’ token
 PYBIND11_MODULE(mlperf_loadgen, m) {
                ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
```
but there was perhaps a mismatch in the header versions.

It's peculiar that when I try to uninstall `pybind11`, I see that header in the list of ones being removed:
```
python3 -m pip uninstall pybind11
Uninstalling pybind11-2.3.0:
  Would remove:
    /home/anton/.local/include/python3.6m/pybind11/attr.h
    /home/anton/.local/include/python3.6m/pybind11/buffer_info.h
    /home/anton/.local/include/python3.6m/pybind11/cast.h
    /home/anton/.local/include/python3.6m/pybind11/chrono.h
    /home/anton/.local/include/python3.6m/pybind11/common.h
    /home/anton/.local/include/python3.6m/pybind11/complex.h
    /home/anton/.local/include/python3.6m/pybind11/detail/class.h
    /home/anton/.local/include/python3.6m/pybind11/detail/common.h
    /home/anton/.local/include/python3.6m/pybind11/detail/descr.h
    /home/anton/.local/include/python3.6m/pybind11/detail/init.h
    /home/anton/.local/include/python3.6m/pybind11/detail/internals.h
    /home/anton/.local/include/python3.6m/pybind11/detail/typeid.h
    /home/anton/.local/include/python3.6m/pybind11/eigen.h
    /home/anton/.local/include/python3.6m/pybind11/embed.h
    /home/anton/.local/include/python3.6m/pybind11/eval.h
    /home/anton/.local/include/python3.6m/pybind11/functional.h
    /home/anton/.local/include/python3.6m/pybind11/iostream.h
    /home/anton/.local/include/python3.6m/pybind11/numpy.h
    /home/anton/.local/include/python3.6m/pybind11/operators.h
    /home/anton/.local/include/python3.6m/pybind11/options.h
    /home/anton/.local/include/python3.6m/pybind11/pybind11.h
    /home/anton/.local/include/python3.6m/pybind11/pytypes.h
    /home/anton/.local/include/python3.6m/pybind11/stl.h
    /home/anton/.local/include/python3.6m/pybind11/stl_bind.h
    /home/anton/.local/lib/python3.6/site-packages/pybind11-2.3.0.dist-info/*
    /home/anton/.local/lib/python3.6/site-packages/pybind11/*
Proceed (y/n)?
```

I must admit that I was lazy and didn't try `git clone --recurse-submodules`. It works with recursion.

Out of curiosity, why can't we use system headers? (I've found this [commit](https://github.com/mlperf/inference/commit/dcce465aa07e589be9510961c07aed105a8bbff6).)

Yes, would be so much easier to use pybind11 from the python install, than setup.py can just take care of installing this stuff. That also would make sure that we are not using pybind11 master.

@guschmue Just to clarify, are you advocating to keep or to change the current approach? 

I'm advocating for changing the current approach because using the pybind11 distribution is more predictable than using pybind11 master.

@briandersn To be fair, isn't it what `build @ ce8824f` is trying to achieve - to freeze "last known good" versions, rather than relying on `master`?

Sorry I missed this discussion earlier.

Both the DEPS approach and the submodule approach to checking out the code will freeze pybind11 to a specific commit. DEPS does it explictly in the DEPS file. Submodules are confusing and do it implicitly somewhere in the .git metadata. It looks like it will checkout the latest master, but it doesn't actually do that.

Looking into it more closely, the DEPS file checks out the latest version of the pybind v2.2 branch, whereas submodules is frozen on a specific commit in the pybind v2.2 branch. To keep things consistent, I'm just going to remove the DEPS approach to checking out the code. I don't even use that any more and dealing with bugs due to pybind version differences would be annoying.

```git submodule add -b branch repo_url``` is all it takes. 

The problem is that DEPS is a gn thing and gn is not really used by any nn framework so I'd expect that submitters are using cmake or python to build. And because the version is not attached to the submodule they will end up with the master which is not good.

submitter can use any pybind version. pybind is NOT needed if you use cmake.

Hi guys, this problem costs my half an hour. And I found this need pybind11 with version 2.2. 
First, I remove pybind11 by pip uninstall pybind11
Then, I install pybind11 by pip install pybind11=2.2
Problem solved!
Good luck!