Repository: mlcommons/inference
Issue: RNNT: Conversion to ONNX
Root cause: External Dependency Breakage

Hi Daniel,

I've grabbed your PR branch to have a play with.

In the `inference.py` script there's commented out code that exports the model to ONNX. I tried uncommneting to see if I can export, but it falls over. (Running with the comments in place, all is well)

Is this something that you've had working or is this still a work in progress?

I'm using PyTorch 1.5.

Thanks.

cc: @galv 

it's not something I'm planning to do anymore, unfortunately. There are too many errors associated pytorch's onnx export path. I acutally commented on this earlier:

https://github.com/mlperf/inference/pull/610#issuecomment-637095346

To elaborate, the tuple-lowering pass does not work (it doesn't support the `Optional[Tuple[]]` type and it requires all tuples to have been created a `TupleConstrust` SSA node, if I remember correctly, which doesn't hold true for loops) and I ran into a segfault afterward I fixed that, which i chose not to investigate since it was becoming a bit of a rabbithole.

So getting an onnx support would require making some fixes to Torchscript's passes, and would probably open a list to unknown unknowns when actually trying to run the onnx model. I just don't really have the time for that unfortunately.

Apologies, I missed your comment on the other thread. Thanks for the explanation. Getting this to work sounds non trivial!




@galv Maybe it's a stupid question but is it possible to bypass Torchscript when converting to ONNX?

Well, I don't see why you couldn't manually write a model using the onnx
opset, but it may be error prone. Were I to do this, I would tackle this by
improving the pytorch optimization and lowering passes, so that other
models could benefit.

On Wed, Jun 10, 2020 at 3:25 PM Anton Lokhmotov <notifications@github.com>
wrote:

> @galv <https://github.com/galv> Maybe it's a stupid question but is it
> possible to bypass Torchscript when converting to ONNX?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/617#issuecomment-642299593>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABEL6UEVYLXXX4JEEUKADXLRWAB7HANCNFSM4N2KYMVA>
> .
>


-- 
Daniel Galvez
http://danielgalvez.me
https://github.com/galv


@galv I'm a compiler guy, so I'd do the same :).

@galv is it the PyTorch passes that are the problem or is it something specific to TorchScript? If it's the latter then the PyTorch rather than TorchScriptified model may be exportable to ONNX?



Torchscript can represent the RNN-T model just fine. I rewrote the original RNN-T code such that this would be possible, in fact. You can see that the reference model actually uses torchscript here for the featurization as well as the RNN-T model here: https://github.com/mlperf/inference/pull/610/files#diff-3f1217a1a54110ac31a4dd0f6d778730R62
The ScriptGreedyDecoder can also be converted to Torchscript, although I don't do that there.

So suppose I did `torch.onnx.export(torch.jit.script(ScriptGreedyDecoder(len(rnnt_vocab) - 1, model)), <other-onnx-export-arguments>)` The torch.onnx.export function would be the cause of errors. If you wanted to look into it, I would recommend installing pytorch from source with debug symbols.

I've had some success. The model exports to ONNX by reverting back to using the RNNTGreedyDecoder and removing all the calls to jit. Running on CPU, performance seems to be unaffected with this config.

To be clear, are you sure you are not just exporting the encoder?

In RNN-T, there are four components:

encoder network
prediction network
joint network
the greedy decoder

The greedy decoder has the logic to decide when to execute which subnetwork (it should be small enough for you to read yourself and understand, even if you are new to this). I tried to export the greedy decoder (ScriptGreedyDecoder) as an ONNX model. This failed. However, it is conceivable that you could export the other three as ONNX models, and then rewrite the ScriptGreedyDecoder to use onnxruntime, in C++, python, or whatever. Of course, for me, such a state of affairs is disappointing, because the onnx model will still depend on that external code.

Yes, you're right. I had just exported the encoder.

I switched back to using ScriptGreedyDecoder and can export the entirety to an ONNX model if I massage the outputs, but when viewed in Netron the graph that's exported is very incorrect. Guessing I'm hitting the issue with the tuple-lowering pass.

I've taken the route you suggest and exported each of the three networks individually to ONNX. These all look reasonable viewed in Netron.

Is there any update on this? Shall I close this issue for now? Please re-open if needed