Repository: mlcommons/inference
Issue: Segmentation fault (core dumped) with --scenario MultiStream
Root cause: Environment Incompatibility

Hi all, I am running image_classification in several configurations `./run_and_time.sh <tf,pytorch, onnxruntime> resnet50 <cpu,gpu>` but I am getting the below error with MultiStream scenario, some tips to solve it?
Error:
` /mlperf/run_helper.sh: line 22: 10 Segmentation fault (core dumped) python python/main.py $opts --output /output/results.json --scenario MultiStream
`

known issue - @briandersn 

Half this issue (crashes during IssueQuery) should be addressed by:
https://github.com/mlperf/inference/commit/1d740be2ca14ed688152849d1325dd4c51dd0139

The other half (crashes at the end during a call to UnloadSamplesFromRam) should be addressed by:
PR #203 

The latter still needs to land.

@vilmara can you please confirm and close?

Hi all, sorry I am still getting segmentation errors with  the updated master branch and MultiStream scenario

Hi @vilmara, can you attach the resulting "mlperf_log_detail.txt" file and verify the git log in "mlperf_log_detail.txt" contains the patches referenced in this bug?

I've had issues re-installing the mlperf_loadgen python module in the past.

Hi @briandersn, here is the file, I have cloned the updated master branch with the command `git clone --recurse-submodules https://github.com/mlperf/inference.git mlperf_inference`. I don't see reflected the PR #203 in the log file

[mlperf_log_detail.txt](https://github.com/mlperf/inference/files/3312012/mlperf_log_detail.txt)




How are you building and installing the mlperf_loadgen module?
Are you using:
1> CFLAGS="-std=c++14" python setup.py develop
Or:
2> CFLAGS="-std=c++14" python setup.py bdist_wheel && pip install --force-reinstall dist/mlperf_loadgen-0.5a0-cp36-cp36m-linux_x86_64.whl

I've had better luck with 2 than 1 in the past. We may need to update the README files.

@briandersn  here are the instructions I am following:
```
git clone --recurse-submodules https://github.com/mlperf/inference.git mlperf_inference
cd mlperf_inference/loadgen
CFLAGS="-std=c++14" python3 setup.py bdist_wheel 
pip3 install dist/mlperf_loadgen-0.5a0-cp35-cp35m-linux_x86_64.whl
```


It looks like you have a system that defaults to python 2 like I do.

Did you also have to change run_local.sh to call python3?

I just double checked that it works for me if I run the commands you mentioned.


@briandersn, I have updated the script `run_helper.sh` with python3 but still getting the same error

Wonder what's going on. Does running the following help?
In loadgen > CFLAGS="-std=c++14" python setup.py develop
In v0.5/classification_and_detection> sudo python3 setup.py develop

@guschmue : Do you have any recommendations for @vilmara 


@vilmara : As a sanity check, can you attach your loadgen/generated/version_generated.cc file?

Hi @briandersn, I will try with your instructions, meanwhile here is the file you have requested (I have changed it to txt extension since github didn't allow to upload cc files)
[version_generated.txt](https://github.com/mlperf/inference/files/3314686/version_generated.txt)

@briandersn I have tried several instructions to rebuild loadgen but got errors:

**With python3 instead of python in both commands:**
In loadgen > `CFLAGS="-std=c++14" python3 setup.py develop`
In v0.5/classification_and_detection> `sudo python3 setup.py develop`
Error:
```
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support \
```

**With python and later python3 as stated in your instructions:**
In loadgen > `CFLAGS="-std=c++14" python setup.py develop`
In v0.5/classification_and_detection> `sudo python3 setup.py develop`
Error:
`ImportError: No module named 'Cython'`

**With the updated instructions in README.md** (python is downloading mlperf_loadgen-0.5a0-cp27-cp27mu-linux_x86_64.whl instead of cp35 version): 
In loadgen > 
`CFLAGS="-std=c++14" python setup.py bdist_wheel`
`pip install --force-reinstall dist/mlperf_loadgen-0.5a0-cp36-cp36m-linux_x86_64.whl`
Error:
```
WARNING: Requirement 'dist/mlperf_loadgen-0.5a0-cp36-cp36m-linux_x86_64.whl' looks like a filename, but the file does not exist
ERROR: mlperf_loadgen-0.5a0-cp36-cp36m-linux_x86_64.whl is not a supported wheel on this platform.
```


hm, this works for me:
```
(base):~/inference/loadgen$ CFLAGS="-std=c++14" python3 setup.py develop
running develop
running egg_info
writing mlperf_loadgen.egg-info/PKG-INFO
writing dependency_links to mlperf_loadgen.egg-info/dependency_links.txt
writing top-level names to mlperf_loadgen.egg-info/top_level.txt
reading manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
writing manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
running build_ext
building 'mlperf_loadgen' extension
gcc -pthread -B /opt/anaconda3/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++14 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/opt/anaconda3/include/python3.7m -c loadgen.cc -o build/temp.linux-x86_64-3.7/loadgen.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /opt/anaconda3/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++14 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/opt/anaconda3/include/python3.7m -c logging.cc -o build/temp.linux-x86_64-3.7/l
...
```
In the gcc command line you see ``` -std=c++14``` ?

And the Cython issue - ```python setup.py develop``` will install it for you if it is not there, for example:
```
# I remove  Cython
(base) ~/inference/v0.5/classification_and_detection$ pip uninstall Cython
Uninstalling Cython-0.29.6:
  Would remove:
    /opt/anaconda3/bin/cygdb
    /opt/anaconda3/bin/cython
    /opt/anaconda3/bin/cythonize
    /opt/anaconda3/lib/python3.7/site-packages/Cython-0.29.6.dist-info/*
    /opt/anaconda3/lib/python3.7/site-packages/Cython/*
    /opt/anaconda3/lib/python3.7/site-packages/cython.py
    /opt/anaconda3/lib/python3.7/site-packages/pyximport/*
Proceed (y/n)? y
  Successfully uninstalled Cython-0.29.6

# and python setup.py develop installs it:
(base) ~/inference/v0.5/classification_and_detection$ python setup.py develop
running develop
running create_version
running build
running build_py
creating build
creating build/lib
creating build/lib/python
copying python/imagenet.py -> build/lib/python
copying python/main.py -> build/lib/python
copying python/backend_onnxruntime.py -> build/lib/python
copying python/backend_pytorch_native.py -> build/lib/python
copying python/coco.py -> build/lib/python
copying python/backend.py -> build/lib/python
copying python/pycoco.py -> build/lib/python
copying python/dataset.py -> build/lib/python
copying python/backend_null.py -> build/lib/python
copying python/backend_tflite.py -> build/lib/python
copying python/backend_tf.py -> build/lib/python
copying python/backend_pytorch.py -> build/lib/python
copying python/__init__.py -> build/lib/python
copying python/version.py -> build/lib/python
running egg_info
creating mlperf_inference.egg-info
writing mlperf_inference.egg-info/PKG-INFO
writing dependency_links to mlperf_inference.egg-info/dependency_links.txt
writing requirements to mlperf_inference.egg-info/requires.txt
writing top-level names to mlperf_inference.egg-info/top_level.txt
writing manifest file 'mlperf_inference.egg-info/SOURCES.txt'
reading manifest file 'mlperf_inference.egg-info/SOURCES.txt'
writing manifest file 'mlperf_inference.egg-info/SOURCES.txt'
running build_ext
Creating /opt/anaconda3/lib/python3.7/site-packages/mlperf-inference.egg-link (link to .)
Adding mlperf-inference 0.1.0 to easy-install.pth file

Installed /home/gs/inference/v0.5/classification_and_detection
Processing dependencies for mlperf-inference==0.1.0
Searching for Cython
Reading https://pypi.org/simple/Cython/
Downloading https://files.pythonhosted.org/packages/b8/c0/500afce93187b9d94d5c6e80a3032f90bf559bd4c8d12b45a10bc9a67b92/Cython-0.29.10-cp37-cp37m-manylinux1_x86_64.whl#sha256=700ccf921b2fdc9b23910e95b5caae4b35767685e0812343fa7172409f1b5830
Best match: Cython 0.29.10
Processing Cython-0.29.10-cp37-cp37m-manylinux1_x86_64.whl
Installing Cython-0.29.10-cp37-cp37m-manylinux1_x86_64.whl to /opt/anaconda3/lib/python3.7/site-packages
Adding Cython 0.29.10 to easy-install.pth file
Installing cygdb script to /opt/anaconda3/bin
Installing cython script to /opt/anaconda3/bin
Installing cythonize script to /opt/anaconda3/bin
...
```


Hi @guschmue, thanks for your feedback, here are the outputs I am getting:

**Step 1:** 
~/mlperf_inference/loadgen# `CFLAGS="-std=c++14" python3 setup.py develop`
```
running develop
Checking .pth file support in /usr/local/lib/python3.5/dist-packages/
/usr/bin/python3 -E -c pass
TEST PASSED: /usr/local/lib/python3.5/dist-packages/ appears to support .pth files
running egg_info
creating mlperf_loadgen.egg-info
writing mlperf_loadgen.egg-info/PKG-INFO
writing dependency_links to mlperf_loadgen.egg-info/dependency_links.txt
writing top-level names to mlperf_loadgen.egg-info/top_level.txt
writing manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
reading manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
writing manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'
running build_ext
building 'mlperf_loadgen' extension
creating build
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/bindings
creating build/temp.linux-x86_64-3.5/generated
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.5m -c loadgen.cc -o build/temp.linux-x86_64-3.5/loadgen.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.5m -c logging.cc -o build/temp.linux-x86_64-3.5/logging.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
logging.cc: In destructor ‘mlperf::Logger::~Logger()’:
logging.cc:160:14: warning: variable ‘tls_logger_prev’ set but not used [-Wunused-but-set-variable]
   TlsLogger* tls_logger_prev = nullptr;
              ^
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.5m -c test_settings_internal.cc -o build/temp.linux-x86_64-3.5/test_settings_internal.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.5m -c utils.cc -o build/temp.linux-x86_64-3.5/utils.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.5m -c version.cc -o build/temp.linux-x86_64-3.5/version.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.5m -c bindings/python_api.cc -o build/temp.linux-x86_64-3.5/bindings/python_api.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/usr/include/python3.5m -c generated/version_generated.cc -o build/temp.linux-x86_64-3.5/generated/version_generated.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
creating build/lib.linux-x86_64-3.5
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -std=c++14 -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/loadgen.o build/temp.linux-x86_64-3.5/logging.o build/temp.linux-x86_64-3.5/test_settings_internal.o build/temp.linux-x86_64-3.5/utils.o build/temp.linux-x86_64-3.5/version.o build/temp.linux-x86_64-3.5/bindings/python_api.o build/temp.linux-x86_64-3.5/generated/version_generated.o -o build/lib.linux-x86_64-3.5/mlperf_loadgen.cpython-35m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.5/mlperf_loadgen.cpython-35m-x86_64-linux-gnu.so ->
Creating /usr/local/lib/python3.5/dist-packages/mlperf-loadgen.egg-link (link to .)
mlperf-loadgen 0.5a0 is already the active version in easy-install.pth

Installed /home/mlperf_inference/loadgen
Processing dependencies for mlperf-loadgen==0.5a0
Finished processing dependencies for mlperf-loadgen==0.5a0

```

**Step 2:** 
~/mlperf_inference/v0.5/classification_and_detection# `python setup.py develop`
```
running develop
running create_version
running build
running build_py
copying python/version.py -> build/lib.linux-x86_64-2.7/python
running egg_info
writing requirements to mlperf_inference.egg-info/requires.txt
writing mlperf_inference.egg-info/PKG-INFO
writing top-level names to mlperf_inference.egg-info/top_level.txt
writing dependency_links to mlperf_inference.egg-info/dependency_links.txt
reading manifest file 'mlperf_inference.egg-info/SOURCES.txt'
writing manifest file 'mlperf_inference.egg-info/SOURCES.txt'
running build_ext
Creating /usr/local/lib/python2.7/dist-packages/mlperf-inference.egg-link (link to .)
mlperf-inference 0.1.0 is already the active version in easy-install.pth

Installed /home/mlperf_inference/v0.5/classification_and_detection
Processing dependencies for mlperf-inference==0.1.0
Searching for pycocotools
Reading https://pypi.python.org/simple/pycocotools/
Best match: pycocotools 2.0.0
Downloading https://files.pythonhosted.org/packages/96/84/9a07b1095fd8555ba3f3d519517c8743c2554a245f9476e5e39869f948d2/pycocotools-2.0.0.tar.gz#sha256=cbb8c2fbab80450a67ee9879c63b0bc8a69e58dd9a0153d55de404c0d383a94b
Processing pycocotools-2.0.0.tar.gz
Writing /tmp/easy_install-rzgwzm/pycocotools-2.0.0/setup.cfg
Running pycocotools-2.0.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-rzgwzm/pycocotools-2.0.0/egg-dist-tmp-98AHv6
Traceback (most recent call last):
  File "setup.py", line 83, in <module>
    install_requires=['numpy>=1.14.1', 'onnx>=1.5', 'pybind11', 'Cython', 'pycocotools', 'mlperf_loadgen']
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "setup.py", line 62, in run
    setuptools.command.develop.develop.run(self)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/develop.py", line 34, in run
    self.install_for_development()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/develop.py", line 133, in install_for_development
    self.process_distribution(None, self.dist, not self.no_deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 740, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 826, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1092, in best_match
    return self.obtain(req, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1104, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 663, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 693, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 873, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1101, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1087, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 246, in run_setup
    raise
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 243, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 273, in run
    return func()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 242, in runner
    _execfile(setup_script, ns)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 46, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-rzgwzm/pycocotools-2.0.0/setup.py", line 2, in <module>
    # Licensed under the MIT license.
ImportError: No module named Cython.Build
```



classification_and_detection does not support python2.7

Note: PR #207 should fix the unused variable error that shows up in release builds for systems whose python installs have strict error flags when compiling modules. It looks like your system is one of the strict ones.

thanks @briandersn / @guschmue . Does the PR #207 will fix the issue running classification_and_detection with system that default to python 2?. 

PR #207 will not fix python 2.

In general, we are not targeting or supporting python 2, so you'll have to make sure the always use python3 and pip3.

Thanks it was solved, closing it!. I have updated the python version to 3.7, edited the files ( run_helper and dockerfile) to use python3, and rebuild the docker image with `./run_and_time.sh`