Nemo Citrinet model fails to deploy to Riva 1.10.0-beta
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
This Nemo model:
https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_citrinet_1024

When build and deploy with Riva 1.10.0-beta (offline config) fails with:
```
2022-04-05 05:08:47,705 [INFO] Writing Riva model repository to '/data/models'...
2022-04-05 05:08:47,705 [INFO] The riva model repo target directory is /data/models
2022-04-05 05:09:02,391 [INFO] Using tensorrt
2022-04-05 05:09:02,396 [INFO] Extract_binaries for nn -> /data/models/riva-trt-my_speech_service-offline-2-am-streaming-offline/1
2022-04-05 05:09:02,396 [INFO] extracting {'onnx': ('nemo.collections.asr.models.ctc_bpe_models.EncDecCTCModelBPE', 'model_graph.onnx')} -> /data/models/riva-trt-my_speech_service-offline-2-am-streaming-offline/1
2022-04-05 05:09:16,155 [INFO] Printing copied artifacts:
2022-04-05 05:09:16,155 [INFO] {'onnx': '/data/models/riva-trt-my_speech_service-offline-2-am-streaming-offline/1/model_graph.onnx'}
2022-04-05 05:09:16,155 [INFO] Building TRT engine from ONNX file
[W] colored module is not installed, will not use colors when logging. To enable colors, please install the colored module: python3 -m pip install colored
[W] 'Shape tensor cast elision' routine failed with: None
[libprotobuf WARNING /home/jenkins/agent/workspace/OSS/OSS_L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING /home/jenkins/agent/workspace/OSS/OSS_L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 572097855
[04/05/2022-05:09:19] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:02] [TRT] [W] Output type must be INT32 for shape outputs
[04/05/2022-05:10:06] [TRT] [E] 4: [shapeCompiler.cpp::evaluateShapeChecks::832] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: condition '==' violated. Where_35: dimensions not compatible for select)
2022-04-05 05:10:06,610 [ERROR] Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/servicemaker/cli/deploy.py", line 100, in deploy_from_rmir
    generator.serialize_to_disk(
  File "/usr/local/lib/python3.8/dist-packages/servicemaker/triton/triton.py", line 398, in serialize_to_disk
    module.serialize_to_disk(repo_dir, rmir, config_only, verbose, overwrite)
  File "/usr/local/lib/python3.8/dist-packages/servicemaker/triton/triton.py", line 282, in serialize_to_disk
    self.update_binary(version_dir, rmir, verbose)
  File "/usr/local/lib/python3.8/dist-packages/servicemaker/triton/asr.py", line 133, in update_binary
    RivaTRTConfigGenerator.update_binary_from_copied(self, version_dir, rmir, copied, verbose)
  File "/usr/local/lib/python3.8/dist-packages/servicemaker/triton/triton.py", line 688, in update_binary_from_copied
    with self.build_trt_engine_from_onnx(model_weights) as engine, open(
AttributeError: __enter__
```
**Steps/Code to reproduce bug**

1. Download the above model from NGC. Convert to Riva with:
`nemo2riva --out my_speech_service-2.riva stt_en_citrinet_1024.nemo`
Using docker container: nvcr.io/nvidia/nemo:22.01

2. Build for offline deployment using docker container: nvcr.io/nvidia/riva/riva-speech:1.10.0-beta-servicemaker
```
riva-build speech_recognition -f \
   /servicemaker-dev/stt_en_citrinet_1024_v1.0.0rc1/my_speech_service-offline-2.rmir    /servicemaker-dev/stt_en_citrinet_1024_v1.0.0rc1/my_speech_service-2.riva \
   --offline \
   --name=my_speech_service-offline-2 \
   --ms_per_timestep=80 \
   --featurizer.use_utterance_norm_params=False \
   --featurizer.precalc_norm_time_steps=0 \
   --featurizer.precalc_norm_params=False \
   --chunk_size=900 \
   --left_padding_size=0. \
   --right_padding_size=0. \
   --decoder_type=flashlight \
   --flashlight_decoder.asr_model_delay=-1 \
--decoding_language_model_binary=/myworkspace/speechtotext_en_us_lm_vdeployable_v1.1/riva_asr_train_datasets_3gram.binary \
--decoding_vocab=/myworkspace/speechtotext_en_us_lm_vdeployable_v1.1/flashlight_decoder_vocab.txt \
   --flashlight_decoder.lm_weight=0.2 \
   --flashlight_decoder.word_insertion_score=0.2 \
   --flashlight_decoder.beam_threshold=20. \
   --language_code=en-US 
```

3. Riva deploy using docker container: nvcr.io/nvidia/riva/riva-speech:1.10.0-beta-servicemaker:
```
riva-deploy -f /servicemaker-dev/stt_en_citrinet_1024_v1.0.0rc1/my_speech_service-offline-2.rmir /data/models
```

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
GPU: V100 32GB