Error when exporting from nemo to riva
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I'm following the tutorial to finetune the Fastpitch and hifigan models through the NeMo 23.02 container. After updating paths, following the tutorial and fine tuning both models, I couldn't make inferences from the models or export nemo models to riva.

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

Download docker image by: docker pull nvcr.io/nvidia/nemo:23.02
Running the NeMo 23.02 docker container by
sudo docker run --runtime=nvidia -it --name nemo -v /dev/shm:/dev/shm --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 - -ulimit stack=67108864 nvcr.io/nvidia/nemo:23.02
Download the NeMo-main folder from github inside the container
Download the fine-tuning notebook tutorial


**Expected behavior**
when executing this cell:
hifigan_nemo_file_path = '/workspace/nemo/finetuning/NeMo/results_dir/HifiGan/2023-04-25_03-04-02/checkpoints/HifiGan.nemo'
hifigan_riva_file_path = hifigan_nemo_file_path[:-5]+".riva"
fastpitch_nemo_file_path = '/workspace/nemo/finetuning/NeMo/results_dir/FastPitch/2023-04-25_01-27-33/checkpoints/FastPitch.nemo'
fastpitch_riva_file_path = fastpitch_nemo_file_path[:-5]+".riva"

!nemo2riva --out {fastpitch_riva_file_path} --key=nemotoriva {fastpitch_nemo_file_path}
!nemo2riva --out {hifigan_riva_file_path} --key=nemotoriva {hifigan_nemo_file_path}

However i got this error:

INFO: Created a temporary directory at /tmp/tmp04gd3x52
INFO: Writing /tmp/tmp04gd3x52/_remote_module_non_scriptable.py
[NeMo W 2023-04-25 03:39:33 experimental:27] Module <class 'nemo.collections.tts.models.fastpitch_ssl.FastPitchModel_SSL'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:39:33 experimental:27] Module <class 'nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.IPATokenizer'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:39:33 experimental:27] Module <class 'nemo.collections.tts.models.radtts.RadTTSModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:39:33 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:39:33 experimental:27] Module <class 'nemo.collections.tts.models.ssl_tts.SSLDisentangler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:39:33 experimental:27] Module <class 'nemo.collections.tts.models.vits.VitsModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo I 2023-04-25 03:39:33 nemo2riva:44] Logging level set to 20
[NeMo I 2023-04-25 03:39:33 convert:41] Restoring NeMo model from '/workspace/nemo/finetuning/NeMo/results_dir/FastPitch/2023-04-25_01-27-33/checkpoints/FastPitch.nemo'
WARNING: Logging before flag parsing goes to stderr.
I0425 03:39:33.679497 140719000115008 setup.py:163] GPU available: True (cuda), used: False
I0425 03:39:33.679834 140719000115008 setup.py:166] TPU available: False, using: 0 TPU cores
I0425 03:39:33.679970 140719000115008 setup.py:169] IPU available: False, using: 0 IPUs
I0425 03:39:33.680056 140719000115008 setup.py:172] HPU available: False, using: 0 HPUs
[NeMo W 2023-04-25 03:39:33 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/setup.py:176: PossibleUserWarning: GPU available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='gpu', devices=1)`.
      rank_zero_warn(
    
[NeMo W 2023-04-25 03:40:05 fastpitch:209] This checkpoint support will be dropped after r1.18.0.
[NeMo E 2023-04-25 03:40:05 common:506] Model instantiation failed!
    Target class:	nemo.collections.tts.models.fastpitch.FastPitchModel
    Error(s):	Error locating target 'nemo.collections.tts.g2p.models.en_us_arpabet.EnglishG2p', see chained exception above.
    full_key: text_tokenizer.g2p
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 639, in _locate
        obj = getattr(obj, part)
    AttributeError: module 'nemo.collections.tts.g2p' has no attribute 'models'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 645, in _locate
        obj = import_module(mod)
      File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
    ModuleNotFoundError: No module named 'nemo.collections.tts.g2p.models'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
        target = _locate(target)
      File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 648, in _locate
        raise ImportError(
    ImportError: Error loading 'nemo.collections.tts.g2p.models.en_us_arpabet.EnglishG2p':
    ModuleNotFoundError("No module named 'nemo.collections.tts.g2p.models'")
    Are you sure that 'models' is importable from module 'nemo.collections.tts.g2p'?
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 485, in from_config_dict
        instance = imported_cls(cfg=config, trainer=trainer)
      File "/usr/local/lib/python3.8/dist-packages/nemo/collections/tts/models/fastpitch.py", line 100, in __init__
        self._setup_tokenizer(cfg)
      File "/usr/local/lib/python3.8/dist-packages/nemo/collections/tts/models/fastpitch.py", line 224, in _setup_tokenizer
        text_tokenizer_kwargs["g2p"] = instantiate(cfg.text_tokenizer.g2p, **g2p_kwargs)
      File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 222, in instantiate
        return instantiate_node(
      File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 325, in instantiate_node
        _target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
      File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
        raise InstantiationException(msg) from e
    hydra.errors.InstantiationException: Error locating target 'nemo.collections.tts.g2p.models.en_us_arpabet.EnglishG2p', see chained exception above.
    full_key: text_tokenizer.g2p
    
[NeMo E 2023-04-25 03:40:05 convert:56] Failed to restore model from NeMo file : /workspace/nemo/finetuning/NeMo/results_dir/FastPitch/2023-04-25_01-27-33/checkpoints/FastPitch.nemo. Please make sure you have the latest NeMo package installed with [all] dependencies.
Traceback (most recent call last):
  File "/usr/local/bin/nemo2riva", line 8, in <module>
    sys.exit(nemo2riva())
  File "/usr/local/lib/python3.8/dist-packages/nemo2riva/cli/nemo2riva.py", line 55, in nemo2riva
    Nemo2Riva(args)
  File "/usr/local/lib/python3.8/dist-packages/nemo2riva/convert.py", line 61, in Nemo2Riva
    raise e
  File "/usr/local/lib/python3.8/dist-packages/nemo2riva/convert.py", line 54, in Nemo2Riva
    model = ModelPT.restore_from(restore_path=nemo_in, trainer=trainer)
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/modelPT.py", line 436, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/connectors/save_restore_connector.py", line 239, in restore_from
    loaded_params = self.load_config_and_state_dict(
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/connectors/save_restore_connector.py", line 162, in load_config_and_state_dict
    instance = calling_cls.from_config_dict(config=conf, trainer=trainer)
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 507, in from_config_dict
    raise e
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 499, in from_config_dict
    instance = cls(cfg=config, trainer=trainer)
TypeError: Can't instantiate abstract class ModelPT with abstract methods list_available_models, setup_training_data, setup_validation_data
INFO: Created a temporary directory at /tmp/tmprpbstve7
INFO: Writing /tmp/tmprpbstve7/_remote_module_non_scriptable.py
[NeMo W 2023-04-25 03:40:12 experimental:27] Module <class 'nemo.collections.tts.models.fastpitch_ssl.FastPitchModel_SSL'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:40:12 experimental:27] Module <class 'nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.IPATokenizer'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:40:12 experimental:27] Module <class 'nemo.collections.tts.models.radtts.RadTTSModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:40:13 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:40:13 experimental:27] Module <class 'nemo.collections.tts.models.ssl_tts.SSLDisentangler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-25 03:40:13 experimental:27] Module <class 'nemo.collections.tts.models.vits.VitsModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo I 2023-04-25 03:40:13 nemo2riva:44] Logging level set to 20
[NeMo I 2023-04-25 03:40:13 convert:41] Restoring NeMo model from '/workspace/nemo/finetuning/NeMo/results_dir/HifiGan/2023-04-25_03-04-02/checkpoints/HifiGan.nemo'
WARNING: Logging before flag parsing goes to stderr.
I0425 03:40:14.087827 140341964519232 setup.py:163] GPU available: True (cuda), used: False
I0425 03:40:14.088244 140341964519232 setup.py:166] TPU available: False, using: 0 TPU cores
I0425 03:40:14.088342 140341964519232 setup.py:169] IPU available: False, using: 0 IPUs
I0425 03:40:14.088403 140341964519232 setup.py:172] HPU available: False, using: 0 HPUs
[NeMo W 2023-04-25 03:40:14 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/setup.py:176: PossibleUserWarning: GPU available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='gpu', devices=1)`.
      rank_zero_warn(
    
[NeMo W 2023-04-25 03:40:14 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    dataset:
      _target_: nemo.collections.tts.data.tts_dataset.VocoderDataset
      manifest_filepath: data_dir/es_cl_female/mels/merged_train_mel.json
      sample_rate: 22050
      n_segments: 8192
      max_duration: null
      min_duration: 0.75
      load_precomputed_mel: true
      hop_length: 256
    dataloader_params:
      drop_last: false
      shuffle: true
      batch_size: 32
      num_workers: 4
      pin_memory: true
    
[NeMo W 2023-04-25 03:40:14 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    dataset:
      _target_: nemo.collections.tts.data.tts_dataset.VocoderDataset
      manifest_filepath: data_dir/es_cl_female/mels/merged_val_mel.json
      sample_rate: 22050
      n_segments: 66048
      max_duration: null
      min_duration: 3
      load_precomputed_mel: true
      hop_length: 256
    dataloader_params:
      drop_last: false
      shuffle: false
      batch_size: 16
      num_workers: 4
      pin_memory: true
    
[NeMo I 2023-04-25 03:40:14 features:287] PADDING: 0
[NeMo I 2023-04-25 03:40:14 features:295] STFT using exact pad
[NeMo I 2023-04-25 03:40:14 features:287] PADDING: 0
[NeMo I 2023-04-25 03:40:14 features:295] STFT using exact pad
[NeMo I 2023-04-25 03:40:16 save_restore_connector:247] Model HifiGanModel was successfully restored from /workspace/nemo/finetuning/NeMo/results_dir/HifiGan/2023-04-25_03-04-02/checkpoints/HifiGan.nemo.
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/nlp-tkc-exported-bert.yaml for nemo.collections.nlp.models.TokenClassificationModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/nlp-pc-exported-bert.yaml for nemo.collections.nlp.models.PunctuationCapitalizationModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/asr-stt-exported-encdecctcmodel.yaml for nemo.collections.asr.models.EncDecCTCModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/tts-exported-hifiganmodel.yaml for nemo.collections.tts.models.HifiGanModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/nlp-isc-exported-bert.yaml for nemo.collections.nlp.models.IntentSlotClassificationModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/tts-exported-fastpitchmodel.yaml for nemo.collections.tts.models.FastPitchModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/nlp-mt-exported-encdecmtmodel.yaml for nemo.collections.nlp.models.MTEncDecModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/nlp-qa-exported-bert.yaml for nemo.collections.nlp.models.QAModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/nlp-tc-exported-bert.yaml for nemo.collections.nlp.models.TextClassificationModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/asr-scr-exported-encdecclsmodel.yaml for nemo.collections.asr.models.classification_models.EncDecClassificationModel
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/asr-stt-exported-encdectcmodelbpe.yaml for nemo.collections.asr.models.EncDecCTCModelBPE
[NeMo I 2023-04-25 03:40:16 schema:154] Loaded schema file /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/tts-exported-radttsmodel.yaml for nemo.collections.tts.models.RadTTSModel
[NeMo I 2023-04-25 03:40:16 schema:193] Found validation schema for nemo.collections.tts.models.HifiGanModel at /usr/local/lib/python3.8/dist-packages/nemo2riva/validation_schemas/tts-exported-hifiganmodel.yaml
[NeMo I 2023-04-25 03:40:16 schema:222] Checking installed NeMo version ... 1.17.0 OK (>=1.1)
[NeMo I 2023-04-25 03:40:16 artifacts:65] Found model at ./model_weights.ckpt
[NeMo E 2023-04-25 03:40:16 artifacts:112] nemo2riva's get_artifacts() was called but was unable to continue due to missing modules. Please ensure that nemo_toolkit and it's dependencies are all installed before re-running nemo2riva.
Traceback (most recent call last):
  File "/usr/local/bin/nemo2riva", line 8, in <module>
    sys.exit(nemo2riva())
  File "/usr/local/lib/python3.8/dist-packages/nemo2riva/cli/nemo2riva.py", line 55, in nemo2riva
    Nemo2Riva(args)
  File "/usr/local/lib/python3.8/dist-packages/nemo2riva/convert.py", line 85, in Nemo2Riva
    artifacts, manifest = get_artifacts(restore_path=nemo_in, model=model, passphrase=key, **patch_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/nemo2riva/artifacts.py", line 117, in get_artifacts
    if isinstance(_HAVE_PATCHES_ERROR_MSG, exception):
TypeError: isinstance() arg 2 must be a type or tuple of types

**Additional context**

I'm using a T4 in AWS