Cannot run nemo asr on Nvidia Jetson Xavier (segemntation fault)
State: closed | Pull request: no

I am running a xavier with JetPack 4.4.
NeMo installs flawessly with pip3 install nemo_toolkit[all]. Also reinstall.sh does not throw an error and ends with success message although it comments that nemo asr, nlp and tts will not be removed since they are not installed (?!). 
Again nemo_toolkit[asr], nemo_toolkit[nlp], nemo_toolkit[tts] can all be installed with pip3 install. Onnxruntime_gpu, Apex and sentencepiece also installed without error - anyhow when I try any library of nemo asr I get a segmentation fault:


python3
Python 3.6.9 (default, Oct  8 2020, 12:12:24) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytorch_lightning as pl
>>> from omegaconf import OmegaConf
>>> from nemo.collections.asr.models import EncDecCTCModel
[NeMo W 2020-12-13 14:51:09 experimental:28] Module <class 'nemo.collections.asr.losses.ctc.CTCLoss'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2020-12-13 14:51:12 experimental:28] Module nemo.collections.asr.data.audio_to_text.AudioToCharDataset is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2020-12-13 14:51:12 experimental:28] Module nemo.collections.asr.data.audio_to_text.AudioToBPEDataset is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2020-12-13 14:51:12 experimental:28] Module nemo.collections.asr.data.audio_to_text.AudioLabelDataset is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2020-12-13 14:51:12 experimental:28] Module nemo.collections.asr.data.audio_to_text.TarredAudioToTextDataset is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2020-12-13 14:51:12 experimental:28] Module nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2020-12-13 14:51:12 experimental:28] Module nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset is experimental, not ready for production and is not fully supported. Use at your own risk.
Segmentation fault (core dumped)

When I do unittests for the tests directory -  the same.
 Any Idea, why? Meanwhile I am lost in the posts stating how NeMo can run on a Jetson Xavier.


P.S.: I do not want to use docker since I have to implement modified training and inference into an app running locally on the Xavier...


"pip install nemo_toolkit[all]" will install an OLD version without PTL (0.11).

Try: "pip install nemo_toolkit[all]=1.0.0b3"

At the moment we do not do testing on Jetson.
Also, from the error description above it is hard to see why segfault happened.

I guess, it's ok to see few tests fail on Jetson - for example, why would you do onnx export on Jetson.
If you only need ASR, try installing "pip install nemo_toolkit[asr]==1.0.0b3" 



1.0.0b3 installed flawlessly without error.
Anyhow calling "from nemo.collections.asr.models import EncDecCTCModel" still gives "segmentation fault (core dumped)"

This seg fault seems to be related to a pytorch dependency (detectron2).  I eventually found that using torch 1.6.0 helped resolve the issue:
```
pip install torch==1.6.0 torchvision==0.7.0
```

deinstalled torch and torchvision libraries and installed the 1.6.0. and 0.7.0 from nvidia prebuilt wheels.

....sorry still exactly the same behavior.

pip3 list shows following python libraries installed - do you recognize any other incompatibilities?

Package                       Version
----------------------------- -------------------
absl-py                       0.10.0
alabaster                     0.7.12
antlr4-python3-runtime        4.8
apex                          0.1
appdirs                       1.4.4
apturl                        0.5.2
argon2-cffi                   20.1.0
asn1crypto                    0.24.0
attrdict                      2.0.1
attrs                         20.1.0
audioread                     2.1.8
Babel                         2.8.0
backcall                      0.2.0
beautifulsoup4                4.6.0
black                         19.10b0
bleach                        3.1.5
blinker                       1.4
boto3                         1.14.54
botocore                      1.17.54
braceexpand                   0.1.5
Brlapi                        0.6.6
cachetools                    4.1.1
certifi                       2020.6.20
cffi                          1.14.2
chardet                       3.0.4
click                         7.1.2
configparser                  5.0.1
cryptography                  2.1.4
cupshelpers                   1.0
cycler                        0.10.0
Cython                        0.29.21
dataclasses                   0.7
decorator                     4.1.2
defer                         1.0.6
defusedxml                    0.6.0
Distance                      0.1.3
distro-info                   0.18ubuntu0.18.04.1
docker-pycreds                0.4.0
docopt                        0.6.2
docutils                      0.15
editdistance                  0.5.3
entrypoints                   0.3
feedparser                    5.2.1
filelock                      3.0.12
frozendict                    1.2
fsspec                        0.8.4
future                        0.18.2
g2p-en                        2.1.0
gdown                         3.12.2
gitdb                         4.0.5
GitPython                     3.1.11
google-auth                   1.21.0
google-auth-oauthlib          0.4.1
graphsurgeon                  0.4.5
grpcio                        1.31.0
h5py                          2.10.0
html2text                     2020.1.16
html5lib                      0.999999999
httplib2                      0.9.2
hydra-core                    1.0.4
idna                          2.6
imagesize                     1.2.0
importlib-metadata            1.7.0
importlib-resources           3.3.0
inflect                       4.1.0
iniconfig                     1.0.1
ipdb                          0.13.3
ipykernel                     5.3.4
ipyparallel                   6.3.0
ipython                       7.16.1
ipython-genutils              0.2.0
ipywidgets                    7.5.1
isort                         4.3.21
jedi                          0.17.2
Jetson.GPIO                   2.0.11
Jinja2                        2.11.2
jmespath                      0.10.0
joblib                        0.16.0
json5                         0.9.5
jsonschema                    3.2.0
jupyter-client                6.1.7
jupyter-core                  4.6.3
jupyterlab                    2.2.6
jupyterlab-server             1.2.0
kaldi-io                      0.9.4
kaldi-python-io               1.1.2
keyring                       10.6.0
keyrings.alt                  3.0
kiwisolver                    1.2.0
language-selector             0.1
latexcodec                    2.0.1
launchpadlib                  1.10.6
lazr.restfulclient            0.13.5
lazr.uri                      1.0.3
librosa                       0.7.2
linecache2                    1.0.0
llvmlite                      0.31.0
louis                         3.5.0
lxml                          4.2.1
macaroonbakery                1.1.3
Mako                          1.0.7
Markdown                      3.2.2
MarkupSafe                    1.0
marshmallow                   3.7.1
matplotlib                    3.3.3
megatron-lm                   1.1.5
mistune                       0.8.4
more-itertools                8.5.0
msgpack                       1.0.0
mypy-extensions               0.4.3
nbconvert                     5.6.1
nbformat                      5.0.7
nemo-toolkit                  1.0.0b3
nltk                          3.5
nose                          1.3.7
notebook                      6.1.3
num2words                     0.5.10
numba                         0.48.0
numpy                         1.19.1
oauth                         1.0.1
oauthlib                      3.1.0
objectio                      0.2.29
omegaconf                     2.0.5
onboard                       1.4.1
onnx                          1.7.0
onnxruntime-gpu               1.4.0
oset                          0.1.3
packaging                     20.4
pandas                        0.22.0
pandocfilters                 1.4.2
parameterized                 0.7.4
parso                         0.7.1
pathspec                      0.8.0
pathtools                     0.1.2
pbr                           3.1.1
pesq                          0.0.1
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        7.2.0
pip                           20.3.1
pip-api                       0.0.14
pipreqs                       0.4.10
pluggy                        0.13.1
progressbar                   2.5
prometheus-client             0.8.0
promise                       2.3
prompt-toolkit                3.0.7
protobuf                      3.14.0
psutil                        5.7.3
ptyprocess                    0.6.0
py                            1.9.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pybind11                      2.5.0
pybtex                        0.22.2
pybtex-docutils               0.2.2
pycairo                       1.16.2
pycparser                     2.20
pycrypto                      2.6.1
pycups                        1.9.73
Pygments                      2.6.1
pygobject                     3.26.1
PyJWT                         1.5.3
pymacaroons                   0.13.0
PyNaCl                        1.1.2
pyparsing                     2.4.7
pypinyin                      0.39.0
pyRFC3339                     1.0
pyrsistent                    0.16.0
PySocks                       1.7.1
pystoi                        0.3.3
pytest                        6.0.1
pytest-runner                 5.2
python-apt                    1.6.5+ubuntu0.4
python-dateutil               2.6.1
python-debian                 0.1.32
pytorch-lightning             1.0.8
pytz                          2018.3
pyxdg                         0.25
PyYAML                        5.3.1
pyzmq                         19.0.2
qtconsole                     4.7.7
QtPy                          1.9.0
rapidfuzz                     0.11.1
regex                         2020.7.14
requests                      2.24.0
requests-oauthlib             1.3.0
requests-unixsocket           0.1.5
resampy                       0.2.2
rouge-score                   0.0.4
rsa                           4.6
ruamel.yaml                   0.16.10
ruamel.yaml.clib              0.2.0
s3transfer                    0.3.3
sacremoses                    0.0.43
scikit-learn                  0.23.2
scipy                         1.5.4
SecretStorage                 2.3.1
semantic-version              2.8.5
Send2Trash                    1.5.0
sentencepiece                 0.1.91
sentry-sdk                    0.19.4
setuptools                    51.0.0
setuptools-rust               0.11.3
shortuuid                     1.0.1
simplejson                    3.13.2
six                           1.15.0
smmap                         3.0.4
snowballstemmer               2.0.0
SoundFile                     0.10.3.post1
sox                           1.4.0
Sphinx                        3.2.1
sphinx-rtd-theme              0.5.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-bibtex          1.0.0
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
ssh-import-id                 5.7
subprocess32                  3.5.4
system-service                0.3
systemd-python                234
tensorboard                   2.3.0
tensorboard-plugin-wit        1.7.0
tensorboardX                  2.1
tensorrt                      7.1.3.0
terminado                     0.8.3
testpath                      0.4.4
testresources                 2.0.0
threadpoolctl                 2.1.0
tokenizers                    0.9.3
toml                          0.10.1
torch                         1.6.0
torch-stft                    0.1.4
torchtext                     0.6.0
torchvision                   0.7.0a0+78ed10c
tornado                       6.0.4
tqdm                          4.48.2
traceback2                    1.4.0
traitlets                     4.3.3
transformers                  3.5.1
typed-ast                     1.4.1
typer                         0.3.2
typing-extensions             3.7.4.2
ubuntu-drivers-common         0.0.0
uff                           0.6.9
Unidecode                     1.1.1
unittest2                     1.1.0
unity-scope-calculator        0.1
unity-scope-chromiumbookmarks 0.1
unity-scope-colourlovers      0.1
unity-scope-devhelp           0.1
unity-scope-firefoxbookmarks  0.1
unity-scope-manpages          0.1
unity-scope-openclipart       0.1
unity-scope-texdoc            0.1
unity-scope-tomboy            0.1
unity-scope-virtualbox        0.1
unity-scope-yelp              0.1
unity-scope-zotero            0.1
urllib3                       1.22
urwid                         2.0.1
virtualenv                    15.1.0
wadllib                       1.3.2
wandb                         0.10.12
watchdog                      0.10.4
wcwidth                       0.2.5
webdataset                    0.1.37
webencodings                  0.5
Werkzeug                      1.0.1
wget                          3.2
wheel                         0.30.0
widgetsnbextension            3.5.1
wrapt                         1.12.1
xkit                          0.0.0
yarg                          0.1.9
youtokentome                  1.0.6
zipp                          3.1.0
zope.interface                4.3.2


nothing pops out, but do you have cudatoolkit installed? other than that I don't know.

calling nvcc --version gives:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_21:14:42_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

which nvcc gives:
/usr/local/cuda-10.2/bin/nvcc

so cuda-toolkit seems to be installed.
Nevertheless nvidia-smi is not installed.



I got the same error , when I " import nemo" it works well, and use nemo.core works too;  But when I  "import nemo.collections.asr " or "import nemo.collections.nlp "  raise un error "segmentation fault (core dumped)" ,  Seems like only collections Class maybe have something wrong ?

Python 3.6.9
pytorch 1.6.0
cuda 10.2
nemo 1.0.0b3

proposed solution is to use these versions
llvmlite-0.32.1
#numba-0.49.1
does it resolve your issue?
https://forums.developer.nvidia.com/t/nemo-installation-done-performance-to-be-evaluated/112012/38

> proposed solution is to use these versions
> llvmlite-0.32.1
> #numba-0.49.1
> does it resolve your issue?
> https://forums.developer.nvidia.com/t/nemo-installation-done-performance-to-be-evaluated/112012/38

@AndreV84 many thanks for your advice , but it dosen't solve my issue . follow your advice I try reinstall  llvmlite-0.32.1 and numba-0.49.1 , got an error **from numba.utils import total_ordering ImportError: cannot import name 'total_ordering’** seems like still version's problem , then i changed numba version to be 0.48.0 with this method https://github.com/librosa/librosa/issues/1103solved error **from numba.utils import total_ordering ImportError: cannot import name 'total_ordering’**  but unfortunately return back to old problem **"segmentation fault (core dumped)"**again when i **import nemo.collection.asr as nemo_asr** .

I used faulthandler catch the error feedback like this:
------------------------------------------------------------------------------------
```
Current thread 0x0000007fa63d6010 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 922 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 658 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "/home/yipeng/.local/lib/python3.6/site-packages/sentencepiece.py", line 15 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "/home/yipeng/.local/lib/python3.6/site-packages/nemo/collections/common/tokenizers/sentencepiece_tokenizer.py", line 19 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "/home/yipeng/.local/lib/python3.6/site-packages/nemo/collections/common/tokenizers/__init__.py", line 17 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1023 in _handle_fromlist
  File "/home/yipeng/.local/lib/python3.6/site-packages/nemo/collections/common/__init__.py", line 16 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 941 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "/home/yipeng/.local/lib/python3.6/site-packages/nemo/collections/asr/models/classification_models.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "/home/yipeng/.local/lib/python3.6/site-packages/nemo/collections/asr/models/__init__.py", line 16 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1023 in _handle_fromlist
  File "/home/yipeng/.local/lib/python3.6/site-packages/nemo/collections/asr/__init__.py", line 15 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<stdin>", line 1 in <module>
Segmentation fault (core dumped)

```
--------------------------------------------------------------------------------------------------------------
@okuchaiev Is there some successful exeperience test for running nemo on jetson nano/nx with https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/01_ASR_with_NeMo.ipynb ?
looking forward to your solution really appreciate !


yes, but the numba version needs to be 47
I suggest to close the issue as it seems got resolved
@yipenglinoe do you need  the issue to remain open?

> yes, but the numba version needs to be 47
> I suggest to close the issue as it seems got resolved
> @yipenglinoe do you need the issue to remain open?
yes ,  for me no problem we can close it . thank you again @AndreV84 

Is there a documented way to use NeMo in full on Jetsons? This is something I'm interested in if it exists. I'm using the L4T base Docker image and virtualenvs within the image, so dependency conflicts with the base L4T image aren't a problem.

@znmeb 
there is nemo documentation 
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/
 however historically it was not targeted onto jetson devices but rather x86_64 as it seems to me

specifically for juetson there were released for voice:
https://github.com/NVIDIA-AI-IOT/jetson-cloudnative-demo
https://ngc.nvidia.com/catalog/containers/nvidia:jetson-voice

@AndreV84 voice is quite close to my intended applications (algorithmic composition - digital sound synthesis) so I'll test those out. I have an RTX 3090 I can use for training. ;-)

>> "Is there a documented way to use NeMo in full on Jetsons? "
- This is a frequent ask, we don't have it at this point.