nemo2riva tts_en_fastpitch fails if not using nemo:1.5.1
State: closed | Pull request: no
Labels: name: bug

Bear with me, this is going to be a lengthy one :)
 
Riva: 1.8.0-beta
Nemo: 1.5.1, 1.6.2, head
Model: [tts_en_fastpitch_align.nemo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_fastpitch)

Related to #3613, could also be relevant for #3609, #3605 and #3567

Working example
- start off of `nvcr.io/nvidia/nemo:1.5.1` 
- install the Riva toolset
- call `nemo2riva` to convert `tts_en_fastpitch_align.nemo` to `riva` format 

Non working example 1:
- start off of `nvcr.io/nvidia/nemo:1.5.1`
- clone the NeMo repository, checkout tag `v1.6.2` or `1.7.0` and upgrade with `./reinstall.sh` 
- install Riva toolset
- call `nemo2riva` to convert `tts_en_fastpitch_align.nemo` to `riva` format 
crashes with
```
E0206 16:37:38.504896 140528981497664 cookbook.py:71] Export failed. Please make sure your NeMo model class (nemo.collections.tts.models.fastpitch.FastPitchModel) has working export() and that you have the latest NeMo package installed with [all] dependencies.
Traceback (most recent call last):
  File "/opt/conda/bin/nemo2riva", line 8, in <module>
    sys.exit(nemo2riva())
  File "/opt/conda/lib/python3.8/site-packages/nemo2riva/cli/nemo2riva.py", line 50, in nemo2riva
    Nemo2Riva(args)
  File "/opt/conda/lib/python3.8/site-packages/nemo2riva/convert.py", line 65, in Nemo2Riva
    save_archive(obj=model, save_path=riva_out, cfg=cfg, artifacts=artifacts, metadata=manifest['metadata'])
  File "/opt/conda/lib/python3.8/site-packages/nemo2riva/cookbook.py", line 76, in save_archive
    raise e
  File "/opt/conda/lib/python3.8/site-packages/nemo2riva/cookbook.py", line 69, in save_archive
    _, descriptions = obj.export(export_file, check_trace=cfg.args.runtime_check)
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/exportable.py", line 179, in export
    output_example = tuple(self.forward(*input_list, **input_dict))
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/tts/models/fastpitch.py", line 497, in forward_for_export
    return self.fastpitch.infer(text=text, pitch=pitch, pace=pace, speaker=speaker)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/tts/modules/fastpitch.py", line 300, in infer
    dec_out, _ = self.decoder(input=len_regulated, seq_lens=dec_lens)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/tts/modules/transformer.py", line 215, in forward
    return self._forward(input, mask_from_lens(seq_lens).unsqueeze(2), conditioning)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/tts/modules/transformer.py", line 223, in _forward
    out = layer(out, mask=mask)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/tts/modules/transformer.py", line 171, in forward
    output = self.dec_attn(dec_inp, attn_mask=~mask.squeeze(2))
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/tts/modules/transformer.py", line 113, in forward
    return self._forward(inp, attn_mask)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/tts/modules/transformer.py", line 142, in _forward
    attn_prob = F.softmax(attn_score, dim=2)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1779, in softmax
    ret = input.softmax(dim)
RuntimeError: CUDA out of memory. Tried to allocate 9.30 GiB (GPU 0; 15.78 GiB total capacity; 12.18 GiB already allocated; 2.05 GiB free; 12.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
``` 

Non working example 2:
- clone the NeMo repository, checkout tag `v1.6.2`
- use nemo Dockerfile to create container
- install Riva toolset in container
- call `nemo2riva` to convert `tts_en_fastpitch_align.nemo` to `riva` format 
crashes with the same error as example 1

Non working example 3:
- clone the NeMo repository
- use nemo Dockerfile to create container from head (as a difference to previous examples this time the container is based on pytorch:22.01)
- install Riva toolset in container (requires an upgrade to Numba=0.55)
- call `nemo2riva` to convert `tts_en_fastpitch_align.nemo` to `riva` format 
crashes with
```
/opt/conda/lib/python3.8/site-packages/torch/_masked/__init__.py:223: UserWarning: Failed to initialize NumPy: module compiled against API version 0xf but this version of numpy is 0xe (Triggered internally at  /opt/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:68.)
  example_input = torch.tensor([[-3, -2, -1], [0, 1, 2]])
Traceback (most recent call last):
  File "/opt/conda/bin/nemo2riva", line 5, in <module>
    from nemo2riva.cli import nemo2riva
  File "/opt/conda/lib/python3.8/site-packages/nemo2riva/__init__.py", line 12, in <module>
    from nemo2riva.convert import Nemo2Riva
  File "/opt/conda/lib/python3.8/site-packages/nemo2riva/convert.py", line 18, in <module>
    from nemo2riva.cookbook import save_archive
  File "/opt/conda/lib/python3.8/site-packages/nemo2riva/cookbook.py", line 10, in <module>
    from nemo.core import Exportable, ModelPT
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/__init__.py", line 16, in <module>
    from nemo.core.classes import *
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/__init__.py", line 18, in <module>
    import pytorch_lightning
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/__init__.py", line 20, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.base import Callback
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/callbacks/base.py", line 26, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/utilities/types.py", line 25, in <module>
    from torchmetrics import Metric
  File "/opt/conda/lib/python3.8/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/opt/conda/lib/python3.8/site-packages/torchmetrics/functional/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio.pit import permutation_invariant_training, pit, pit_permutate
  File "/opt/conda/lib/python3.8/site-packages/torchmetrics/functional/audio/__init__.py", line 26, in <module>
    from torchmetrics.functional.audio.pesq import perceptual_evaluation_speech_quality  # noqa: F401
  File "/opt/conda/lib/python3.8/site-packages/torchmetrics/functional/audio/pesq.py", line 20, in <module>
    import pesq as pesq_backend
  File "/opt/conda/lib/python3.8/site-packages/pesq/__init__.py", line 6, in <module>
    from .cypesq import cypesq, cypesq_retvals, cypesq_error_message as pesq_error_message
  File "pesq/cypesq.pyx", line 1, in init cypesq
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
```
Googling for the UserWarning leads to https://github.com/numba/numba/issues/7754, but the installed numpy is 1.21.5.

What is behind all this ... We added support to `fastpitch.py` (based on the soon to be deprecated `fastpitch_align.yml` not `1.05`) for a non-English language that we are working with. Our starting point was `NeMo:1.6.0`, but after finishing and successfully training a model we discovered that if we want to deploy it using Riva, we first have to back port all of our changes. Since the old `yml` will become deprecated only in `NeMo:1.8.0`, it perhaps should still be supported for conversion to riva format.