Segmentation fault when importing nemo.collections.asr (protobuf?)
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Importing `nemo.collections.asr` throws a segmentation fault.

**Steps/Code to reproduce bug**

Install NeMo in a fresh conda environment as per the instructions on the README.md: https://github.com/NVIDIA/NeMo#installation

<details>

<summary> Packages </summary>

```
# packages in environment at /opt/conda/envs/hf:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   1.2.0                    pypi_0    pypi
aiohttp                   3.8.1                    pypi_0    pypi
aiosignal                 1.2.0                    pypi_0    pypi
alabaster                 0.7.12                   pypi_0    pypi
aniso8601                 9.0.1                    pypi_0    pypi
antlr4-python3-runtime    4.8                      pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
asttokens                 2.0.8                    pypi_0    pypi
async-timeout             4.0.2                    pypi_0    pypi
attrdict                  2.0.1                    pypi_0    pypi
attrs                     22.1.0                   pypi_0    pypi
audioread                 3.0.0                    pypi_0    pypi
babel                     2.10.3                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
beautifulsoup4            4.11.1                   pypi_0    pypi
black                     19.10b0                  pypi_0    pypi
blas                      1.0                         mkl  
boto3                     1.24.58                  pypi_0    pypi
botocore                  1.27.58                  pypi_0    pypi
braceexpand               0.1.7                    pypi_0    pypi
brotlipy                  0.7.0           py38h0a891b7_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.6.15            ha878542_0    conda-forge
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.6.15        py38h578d9bd_0    conda-forge
cffi                      1.15.1           py38h4a40e3a_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3                    pypi_0    pypi
colorama                  0.4.5                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
cryptography              37.0.1           py38h9ce1e76_0  
cudatoolkit               11.6.0              hecad31d_10    conda-forge
cycler                    0.11.0                   pypi_0    pypi
cython                    0.29.32                  pypi_0    pypi
debugpy                   1.6.3                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
distance                  0.1.3                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
docutils                  0.19                     pypi_0    pypi
editdistance              0.6.0                    pypi_0    pypi
einops                    0.4.1                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
executing                 0.10.0                   pypi_0    pypi
faiss-cpu                 1.7.2                    pypi_0    pypi
fasttext                  0.9.2                    pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.8.0                    pypi_0    pypi
flask                     2.2.2                    pypi_0    pypi
flask-restful             0.3.9                    pypi_0    pypi
fonttools                 4.37.0                   pypi_0    pypi
freetype                  2.12.1               hca18f0e_0    conda-forge
frozendict                2.3.4                    pypi_0    pypi
frozenlist                1.3.1                    pypi_0    pypi
fsspec                    2022.7.1                 pypi_0    pypi
ftfy                      6.1.1                    pypi_0    pypi
g2p-en                    2.1.0                    pypi_0    pypi
gdown                     4.5.1                    pypi_0    pypi
gitdb                     4.0.9                    pypi_0    pypi
gitpython                 3.1.27                   pypi_0    pypi
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               2.11.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
grpcio                    1.47.0                   pypi_0    pypi
h5py                      3.7.0                    pypi_0    pypi
huggingface-hub           0.9.0                    pypi_0    pypi
hydra-core                1.1.2                    pypi_0    pypi
idna                      3.3                pyhd8ed1ab_0    conda-forge
ijson                     3.1.4                    pypi_0    pypi
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        4.12.0                   pypi_0    pypi
importlib-resources       5.2.3                    pypi_0    pypi
inflect                   6.0.0                    pypi_0    pypi
iniconfig                 1.1.1                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561  
ipadic                    1.0.0                    pypi_0    pypi
ipykernel                 6.15.1                   pypi_0    pypi
ipython                   8.4.0                    pypi_0    pypi
ipywidgets                8.0.1                    pypi_0    pypi
isort                     4.3.21                   pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jarowinkler               1.2.1                    pypi_0    pypi
jedi                      0.18.1                   pypi_0    pypi
jieba                     0.42.1                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9e                   h166bdaf_2    conda-forge
jupyter-client            7.3.4                    pypi_0    pypi
jupyter-core              4.11.1                   pypi_0    pypi
jupyterlab-widgets        3.0.2                    pypi_0    pypi
kaldi-python-io           1.2.2                    pypi_0    pypi
kaldiio                   2.17.2                   pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
lame                      3.100             h7f98852_1001    conda-forge
latexcodec                2.0.1                    pypi_0    pypi
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libdeflate                1.13                 h166bdaf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libpng                    1.6.37               h753d276_4    conda-forge
librosa                   0.9.2                    pypi_0    pypi
libsqlite                 3.39.2               h753d276_1    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libtiff                   4.4.0                h0e0dad5_3    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libzlib                   1.2.12               h166bdaf_2    conda-forge
llvmlite                  0.39.0                   pypi_0    pypi
lxml                      4.9.1                    pypi_0    pypi
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
marshmallow               3.17.1                   pypi_0    pypi
matplotlib                3.5.3                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mecab-python3             1.0.5                    pypi_0    pypi
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py38h95df7f1_0    conda-forge
mkl_fft                   1.3.1            py38h8666266_1    conda-forge
mkl_random                1.2.2            py38h1abd341_0    conda-forge
mpmath                    1.2.1                    pypi_0    pypi
multidict                 6.0.2                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
nemo-toolkit              1.11.0rc0                 dev_0    <develop>
nest-asyncio              1.5.5                    pypi_0    pypi
nettle                    3.6                  he412f7d_0    conda-forge
nltk                      3.7                      pypi_0    pypi
numba                     0.56.0                   pypi_0    pypi
numpy                     1.22.4                   pypi_0    pypi
oauthlib                  3.2.0                    pypi_0    pypi
omegaconf                 2.1.2                    pypi_0    pypi
onnx                      1.12.0                   pypi_0    pypi
opencc                    1.1.4                    pypi_0    pypi
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   3.0.5                h166bdaf_1    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    1.4.3                    pypi_0    pypi
pangu                     4.0.6.1                  pypi_0    pypi
parameterized             0.8.1                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pathspec                  0.9.0                    pypi_0    pypi
pathtools                 0.1.2                    pypi_0    pypi
pesq                      0.0.4                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.2.0            py38ha3b2c9c_2    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
pip-api                   0.0.30                   pypi_0    pypi
pipreqs                   0.4.11                   pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
pooch                     1.6.0                    pypi_0    pypi
portalocker               2.5.1                    pypi_0    pypi
promise                   2.3                      pypi_0    pypi
prompt-toolkit            3.0.30                   pypi_0    pypi
protobuf                  3.19.4                   pypi_0    pypi
psutil                    5.9.1                    pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pyannote-core             4.5                      pypi_0    pypi
pyannote-database         4.1.3                    pypi_0    pypi
pyannote-metrics          3.2.1                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pybind11                  2.10.0                   pypi_0    pypi
pybtex                    0.24.0                   pypi_0    pypi
pybtex-docutils           1.0.2                    pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.9.2                    pypi_0    pypi
pydeprecate               0.3.2                    pypi_0    pypi
pydub                     0.25.1                   pypi_0    pypi
pygments                  2.13.0                   pypi_0    pypi
pynini                    2.1.5                    pypi_0    pypi
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9                    pypi_0    pypi
pypinyin                  0.47.1                   pypi_0    pypi
pysocks                   1.7.1            py38h578d9bd_5    conda-forge
pystoi                    0.3.3                    pypi_0    pypi
pytest                    7.1.2                    pypi_0    pypi
pytest-runner             6.0.0                    pypi_0    pypi
python                    3.8.13          ha86cf86_0_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python_abi                3.8                      2_cp38    conda-forge
pytorch                   1.12.1          py3.8_cuda11.6_cudnn8.3.2_0    pytorch
pytorch-lightning         1.7.2                    pypi_0    pypi
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2022.2.1                 pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
pyzmq                     23.2.1                   pypi_0    pypi
rapidfuzz                 2.6.0                    pypi_0    pypi
readline                  8.1.2                h0f457ee_0    conda-forge
regex                     2022.8.17                pypi_0    pypi
requests                  2.28.1             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1                    pypi_0    pypi
resampy                   0.4.0                    pypi_0    pypi
rich                      12.5.1                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
ruamel-yaml               0.17.21                  pypi_0    pypi
ruamel-yaml-clib          0.2.6                    pypi_0    pypi
s3transfer                0.6.0                    pypi_0    pypi
sacrebleu                 2.2.0                    pypi_0    pypi
sacremoses                0.0.53                   pypi_0    pypi
scikit-learn              1.1.2                    pypi_0    pypi
scipy                     1.9.0                    pypi_0    pypi
sentence-transformers     2.2.2                    pypi_0    pypi
sentencepiece             0.1.97                   pypi_0    pypi
sentry-sdk                1.9.5                    pypi_0    pypi
setproctitle              1.3.2                    pypi_0    pypi
setuptools                59.5.0                   pypi_0    pypi
shellingham               1.5.0                    pypi_0    pypi
shortuuid                 1.0.9                    pypi_0    pypi
simplejson                3.17.6                   pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0                    pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
soundfile                 0.10.3.post1             pypi_0    pypi
soupsieve                 2.3.2.post1              pypi_0    pypi
sox                       1.4.1                    pypi_0    pypi
sphinx                    5.1.1                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-bibtex      2.5.0                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
sqlite                    3.39.2               h4ff8645_1    conda-forge
stack-data                0.4.0                    pypi_0    pypi
sympy                     1.11                     pypi_0    pypi
tabulate                  0.8.10                   pypi_0    pypi
tensorboard               2.10.0                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tk                        8.6.12               h27826a3_0    conda-forge
tokenizers                0.12.1                   pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
torchaudio                0.12.1               py38_cu116    pytorch
torchmetrics              0.9.3                    pypi_0    pypi
torchvision               0.13.1               py38_cu116    pytorch
tornado                   6.2                      pypi_0    pypi
tqdm                      4.64.0                   pypi_0    pypi
traitlets                 5.3.0                    pypi_0    pypi
transformers              4.21.1                   pypi_0    pypi
typed-ast                 1.5.4                    pypi_0    pypi
typer                     0.6.1                    pypi_0    pypi
typing_extensions         4.3.0              pyha770c72_0    conda-forge
unidecode                 1.3.4                    pypi_0    pypi
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wandb                     0.13.2                   pypi_0    pypi
wcwidth                   0.2.5                    pypi_0    pypi
webdataset                0.1.62                   pypi_0    pypi
werkzeug                  2.2.2                    pypi_0    pypi
wget                      3.2                      pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.2                    pypi_0    pypi
wrapt                     1.14.1                   pypi_0    pypi
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yarg                      0.1.9                    pypi_0    pypi
yarl                      1.8.1                    pypi_0    pypi
youtokentome              1.0.6                    pypi_0    pypi
zipp                      3.8.1                    pypi_0    pypi
zlib                      1.2.12               h166bdaf_2    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge
```

</details>

Now try and import `nemo.collections.asr`:

```python
import nemo.collections.asr
```

```
Segmentation Fault
```

Debugging with [gdb](https://wiki.python.org/moin/DebuggingWithGdb):
<details>

<summary> Traceback </summary>

```
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 10897]
[New Thread 0x7fff6f483700 (LWP 10898)]
[New Thread 0x7fff6ec82700 (LWP 10899)]
[New Thread 0x7fff6a481700 (LWP 10900)]
[New Thread 0x7fff67c80700 (LWP 10901)]
[New Thread 0x7fff6547f700 (LWP 10902)]
[New Thread 0x7fff64c7e700 (LWP 10903)]
[New Thread 0x7fff6247d700 (LWP 10904)]
[New Thread 0x7fff5fc7c700 (LWP 10905)]
[New Thread 0x7fff5d47b700 (LWP 10906)]
[New Thread 0x7fff5ac7a700 (LWP 10907)]
[New Thread 0x7fff56479700 (LWP 10908)]
[New Thread 0x7fff53c78700 (LWP 10909)]
[New Thread 0x7fff51477700 (LWP 10910)]
[New Thread 0x7fff4ec76700 (LWP 10911)]
[New Thread 0x7fff4c475700 (LWP 10912)]
[New Thread 0x7fff49c74700 (LWP 10913)]
[New Thread 0x7fff47473700 (LWP 10914)]
[New Thread 0x7fff44c72700 (LWP 10915)]
[New Thread 0x7fff42471700 (LWP 10916)]
[New Thread 0x7fff3fc70700 (LWP 10917)]
[New Thread 0x7fff3d46f700 (LWP 10918)]
[New Thread 0x7fff3cc6e700 (LWP 10919)]
[New Thread 0x7fff3a46d700 (LWP 10920)]
[New Thread 0x7fff2cd3c700 (LWP 10921)]
[New Thread 0x7fff2c53b700 (LWP 10922)]
[New Thread 0x7fff29d3a700 (LWP 10923)]
[New Thread 0x7fff27539700 (LWP 10924)]
[New Thread 0x7fff24d38700 (LWP 10925)]
[New Thread 0x7fff22537700 (LWP 10926)]
[New Thread 0x7fff1fd36700 (LWP 10927)]
[New Thread 0x7fff1d535700 (LWP 10928)]
[New Thread 0x7fff1ad34700 (LWP 10929)]
[New Thread 0x7fff18533700 (LWP 10930)]
[New Thread 0x7fff15d32700 (LWP 10931)]
[New Thread 0x7fff13531700 (LWP 10932)]
[New Thread 0x7fff0ed30700 (LWP 10933)]
[New Thread 0x7fff0c52f700 (LWP 10934)]
[New Thread 0x7fff09d2e700 (LWP 10935)]
[New Thread 0x7fff0752d700 (LWP 10936)]
[New Thread 0x7fff04d2c700 (LWP 10937)]
[New Thread 0x7fff0252b700 (LWP 10938)]
[New Thread 0x7ffeffd2a700 (LWP 10939)]
[New Thread 0x7ffefd529700 (LWP 10940)]
[New Thread 0x7ffefcd28700 (LWP 10941)]
[New Thread 0x7ffef8527700 (LWP 10942)]
[New Thread 0x7ffef5d26700 (LWP 10943)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffeefe711e1 in google::protobuf::internal::ReflectionOps::FindInitializationErrors(google::protobuf::Message const&, std::string const&, std::vector<std::string, std::allocator<std::string> >*) ()
   from /opt/conda/envs/hf/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
(gdb) 
```
</details>

Is this possibly related to protobuf? Note: uninstalling and re-installing protobuf gives the same error:
```
pip uninstall protobuf
conda uninstall protobuf
conda install -c anaconda protobuf 

python -c "import nemo.collections.asr"
```

```
Segmentation Fault
```

**Expected behavior**

Import of the module!

**Environment overview (please complete the following information)**

 - Environment location: GCP. GPU: a2-highgpu-2g. CPU: Intel Cascade Lake
 - Method of NeMo install: from source (as per https://github.com/NVIDIA/NeMo#from-source)

**Environment details**
- OS version: Google, Debian 10 based Deep Learning VM with CUDA 11.0, M95, Base CUDA 11.0, Deep Learning VM Image with CUDA 11.0 preinstalled.
- PyTorch version: 1.12.1
- Python version: 3.8