onnx export for G2P models
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Adds support for ONNX export to G2P models (CTC/byT5)

**Collection**: tts


# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
  
**PR Type**:
- [x] New Feature

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Hello,

i am trying to make G2P models exportable, but i am running into issues with ONNX export.
I hope someone here can guide me to the correct solution.




both (t5 and ctc) return this error:
i am guessing i am doing something wrong with the output types ?

```
Traceback (most recent call last):
  File "/workspace/6_export.py", line 38, in <module>
    model.export('mymodel.onnx')
  File "/usr/local/lib/python3.10/dist-packages/nemo/core/classes/exportable.py", line 113, in export
    out, descr, out_example = model._export(
  File "/usr/local/lib/python3.10/dist-packages/nemo/core/classes/exportable.py", line 220, in _export
    torch.onnx.export(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 1582, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 1182, in _model_to_graph
    _C._jit_pass_onnx_assign_output_shape(
RuntimeError: ArrayRef: invalid index Index = 0; Length = 0

```

i was able to get onnx export to finish (with a lot of warnings) for CTC model, if i changed the last lines to :

```
        preds_str, _ = self.decoding.ctc_decoder_predictions_tensor(
            log_probs, decoder_lengths=encoded_len, return_hypotheses=True
        )
        results = [h.y_sequence for h in preds_str]
        return tuple(results)
```

as it seems that returning a string was an issue ?

however when i try to use this onnx model, it tells me it has 0 inputs:

```
ort_session = onnxruntime.InferenceSession("mymodel.onnx", providers=["CPUExecutionProvider"])
print(ort_session.get_inputs())
```

returns []

in the latest commit https://github.com/NVIDIA/NeMo/pull/8177/commits/e93ffeca5c57fd88e4a00d8fd522052731f07a68 ctc decoding is skipped, in this way export works succesfully for CTC model as well.

is it not possible to make ctc decoding part of the onnx model ?

i verified the generated onnx models and they seem to work correctly.

still i would really appreciate if someone can guide to towards making ctc decoding part of the nemo model as well as without it its quite hard to consume it.

@rlangman can you take a look ?

@borisfom could you take a look at this?

@ppisljar @rlangman : For ASR models. it's typical to exclude CTC decoder and tokenizer from ONNX export - as they normally could not be exported. So only DNN part (EncDecModel) is usually represented by ONNX graph.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins

@mgrafu , @borisfom are you OK with the changes?

jenkins

ping

jenkins

jenkins

jenkins