nemo2riva fails on stt_en_conformer_ctc_large_1.6.0
State: closed | Pull request: no
Labels: name: bug

Start off with `nemo:22.01` container, install `riva_quickstart_v2.0.0` (`riva_api` and `nemo2riva`), download `stt_en_conformer_ctc_large_1.6.0` from ngc and unzip it. 

Running `nemo2riva --out stt_en_conformer_ctc_large.riva stt_en_conformer_ctc_large.nemo` gives the following output
```
INFO: Logging level set to 20                                                                                                                                                                                                                          
INFO: Restoring NeMo model from 'stt_en_conformer_ctc_large.nemo'                                                                                                                                                                                      
################################################################################                                                                                                                                                                       
### WARNING, path does not exist: KALDI_ROOT=/mnt/matylda5/iveselyk/Tools/kaldi-trunk                                                                                                                                                                  
###          (please add 'export KALDI_ROOT=<your_path>' in your $HOME/.profile)                                                                                                                                                                       
###          (or run as: KALDI_ROOT=<your_path> python <your_script>.py)                                                                                                                                                                               
################################################################################                                                                                                                                                                       
                                                                                                                                                                                                                                                       
[NeMo I 2022-04-11 20:32:55 mixins:165] Tokenizer SentencePieceTokenizer initialized with 128 tokens                                                                                                                                                   
[NeMo W 2022-04-11 20:32:56 modelPT:148] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.                                      
    Train config :                                                                                                                                                                                                                                     
    manifest_filepath: /data/NeMo_ASR_SET/English/v2.0/train/tarred_audio_manifest.json                                                                                                                                                                
    sample_rate: 16000                                                                                                                                                                                                                                 
    batch_size: 32                                                                                                                                                                                                                                     
    shuffle: true                                                                                                                                                                                                                                      
    num_workers: 8                                                                                                                                                                                                                                     
    pin_memory: true                                                                                                                                                                                                                                   
    use_start_end_token: false                                                                                                                                                                                                                         
    trim_silence: false                                                                                                                                                                                                                                
    max_duration: 20.0                                                                                                                                                                                                                                 
    min_duration: 0.1                                                                                                                                                                                                                                  
    shuffle_n: 2048                                                                                                                                                                                                                                    
    is_tarred: true                                                                                                                                                                                                                                    
    tarred_audio_filepaths: /data/NeMo_ASR_SET/English/v2.0/train/audio__OP_0..4095_CL_.tar                                                                                                                                                            
                                                                                                                                                                                                                                                       
[NeMo W 2022-04-11 20:32:56 modelPT:155] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(
s).                                                                                                                                                                                                                                                    
    Validation config :                                                                                                                                                                                                                                
    manifest_filepath:                                                                                                                                                                                                                                 
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-other.json                                                                                                                                                                      
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-clean.json                                                                                                                                                                      
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-other.json                                                                                                                                                                     
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-clean.json                                                                                                                                                                     
    sample_rate: 16000                                                                                                                                                                                                                                 
    batch_size: 16                                                                                                                                                                                                                                     
    shuffle: false                                                                                                                                                                                                                                     
    num_workers: 8                                                                                                                                                                                                                                     
    pin_memory: true                                                                                                                                                                                                                                   
    use_start_end_token: false                                                                                                                                                                                                                         
    is_tarred: false                                                                                                                                                                                                                                   
    tarred_audio_filepaths: na                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                       
[NeMo W 2022-04-11 20:32:56 modelPT:161] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).                                               
    Test config :                                                                                                                                                                                                                                      
    manifest_filepath:                                                                                                                                                                                                                                 
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-other.json                                                                                                                                                                     
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-clean.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-other.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-clean.json
    sample_rate: 16000
    batch_size: 16
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: false
    is_tarred: false
    tarred_audio_filepaths: na
     
[NeMo I 2022-04-11 20:32:56 features:259] PADDING: 0
[NeMo I 2022-04-11 20:32:56 features:276] STFT using torch
[NeMo I 2022-04-11 20:32:59 save_restore_connector:158] Model EncDecCTCModelBPE was successfully restored from /data/stt_en_conformer_ctc_large.nemo.
[NeMo I 2022-04-11 20:33:03 export_utils:261] Swapped 108 modules
[NeMo W 2022-04-11 20:33:04 nemo_logging:349] /workspace/nemo/nemo/collections/asr/modules/conformer_encoder.py:240: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if max_audio_length > self.max_audio_length:
     
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
...
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
[W] 'Shape tensor cast elision' routine failed with: None
```

Running `nemo2riva --validate stt_en_conformer_ctc_large.nemo` gives the same output, but at the end appends the following lines
```
WARNING: Logging before flag parsing goes to stderr.
E0411 20:34:31.553215 140467938121536 <frozen eff.validator.validator>:59] Condition for key 'min_nemo_version' (1.3  <built-in function eq> 1.1) is not fulfilled
E0411 20:34:31.553421 140467938121536 schema.py:198] Exported model at stt_en_conformer_ctc_large.riva failed Riva compliance, using schema at /opt/conda/lib/python3.8/site-packages/nemo2riva/validation_schemas/asr-stt-exported-encdectcmodelbpe.yaml !
```

Using `reinstall.sh` to install `nemo:1.9.0rc0` makes no difference.