Repository: mlcommons/inference
Issue: Tracking RNN-T ONNX work 
GitHub labels: inference v2.1 and backlog

@nvpohanh @psyhtest @galv 

@guschmue you mentioned that you would like to take a look. Do you need a unit test or something to get you started at looking?

For anyone interested in this, I will summarize the ONNX implementation:

RNN-T has four components:
1. Encoder network
1. Prediction network
1. Joint network
1. Decoder (This is not a neural network)

The first three can be exported to ONNX successfully, as far as I know. (I am not directly leading this effort.)

Pytorch 1.5 does *not* work here. You need the current HEAD version, for multiple reasons.

What I would like is three unit tests, one for each of the three neural network components. The unit tests should check that onnxruntime and pytorch match (within reason) on one or a few inputs.

At that point, someone could easily rewrite the decoder to use the three onnx components instead of the three original pytorch components. (The decoder is fairly straightforward and not performance critical.)

I am attaching some observations I made regarding the problems with exporting the full model (all four components):

> The problems I found were in torchscript, not in ONNX, from what I remember.
> There were two things, both having to do with LSTMs' hidden states:
> 1) Tuple-lowering pass is unable to handle an Optional[Tuple[Tensor, Tensor]], IIRC. The optional throws it off somehow, even though the compiler should be able to deduce that the type is Tuple[Tensor, Tensor] in that case.
> 2) Since LSTM acts using a for-loop, it passes a tuple of hidden states between each iteration. Tuple-lowering expects that the SSA node producing the input to a given tuple-using operation be produced by a TupleConstruct node (source: https://github.com/pytorch/pytorch/blob/3ed96e465cbbb8c7f9ea4fea2abe3cb84dac535d/torch/csrc/jit/passes/lower_tuples.cpp#L33), not by an assignment node, or whatever. Should be pretty easy to fix by simply searching back through your assignments until you find the TupleConstruct. But I may be missing something.
> 3) In addition, the output of the decoder is a variable-length tensor. The best way to implement this is via a std::vector<> that you append to, or a buffer with a fixed size larger than you would ever expect to use (as well as a boolean to indicate if you overflowed the buffer). I'm not entirely sure which is better for ONNX.
> I would certainly appreciate help with this sort of thing. I do want to be respectful of people's time, though. I can make github issues in pytorch about these things with enough time (I have to manage a release this week). The main question is whether we have someone who can answer them effectively and apply resources to get them solved. (I totally understand if that is not the case. Everyone is busy.)


@guschmue  @galv still working on this.

I've been playing with this, and as @galv says, the first three components do export, but I've not been in a position to see whether they work as expected.

I've also been attempting to get the model to export as a whole. Doing this, I end up with either a model that has the prediction and joint networks missing, or they are unrolled. I.e. one lstm per logit. This makes for quite a sizeable (and inflexible) model :)

I'm not sure whether the exporter has the capability to support the LSTMs in the model. It can't handle batching as far as I can tell from the error message (below), but as reference is made to passing back in the hidden state then it makes me think that the message actually refers to the sequence length. This may be related to the Tuple lowering problem @galv was seeing?

`UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.`

@guschmue don't suppose you could clarify the bounds of support for LSTM export?

Rather than attempting to export the model as a whole, it may make more sense to drop back to looking at the encoder network to see if that piece holds together.


Forgot to add, I took the 'naive' route of converting all python variables to tensors which was sufficient to allow the model to export , but several  warnings still remain about use of python variables even though i've converted all to tensors. Doing this the output from the pytorch model remains correct but on export the model produced is incorrect. I'm sure my approach is not very efficient either.

As well as the general question of whether the LSTM export should be able to support this model, can anyone (@guschmue @galv) comment on whether the code within this forward declaration and _greedy_decode, should export successfully without too much modification -

https://github.com/mlperf/inference/blob/master/v0.7/speech_recognition/rnnt/pytorch/decoders.py#L49

Another thing I've noticed - identifying which of the inputs are of variable length in the call to export, doesn't seem to meaningfully affect the output.

Attached is a zip containing an svg of one of the attempts to export.
[bigmodel.zip](https://github.com/mlperf/inference/files/4889700/bigmodel.zip)



First thing:

I missed something. RNN-T actually has five components:

1. Encoder network
1. Prediction network
1. Joint network
1. Decoder (This is not a neural network)
1. An audio preprocessor (It transforms the waveform into a stream of feature vectors, basically)

Anyway, @G4V I would like to take a more active role in looking at this. Can I ask you to create a unit-test to reproduce your work? I will upload my own unit test tonight showing how I wrote a unit-test for the audio preprocessor (that unit-test is currently failing to do export successfully). Then hopefully you can base your work on that.

> comment on whether the code within this forward declaration and _greedy_decode, should export successfully without too much modification -
> https://github.com/mlperf/inference/blob/master/v0.7/speech_recognition/rnnt/pytorch/decoders.py#L49

I expect export of that code to be hard, based on my experience. It's using dynamically resized std::vectors (aka, python arrays). I'm not sure what kind of support onnx has for that sort of thing. I did actually write a version of this that used a fixed size output array (which would also return a boolean saying whether the fixed size output array had overflown) a while ago, but I got rid of it. My suggestion for now is simple: Convert the other modules into onnx, and then rewrite the decoder to use the onnx modules.

FYI @G4V if you can make unit tests like I did here: https://github.com/mlperf/inference/pull/651, that would make it a lot easier to help out.

Okay, the audio preprocessor requires a short-time-fourier transform.

While aten supports this with the aten::stft op, that is not a supported op for export at this time: https://pytorch.org/docs/stable/onnx.html#supported-operators

I looked at the onnx repo, and there is some discussion about FFT functions, which has culminated in this issue: https://github.com/onnx/onnx/pull/2625 This presumably means onnxruntime won't support it either.

(Note that STFT is implemented in terms of many individual FFT operations.)

In addition, this is a bit more technical and easier to fix, but there is no way to export the op aten::list right now to ONNX, which the STFT implementation in pytorch depends on. This is probably fixable with either a code rewrite of STFT or a new compiler pass in pytorch.

You can implement a DFT instead of FFT pretty easily, but you may as well do this the right way and do some open source work.

People are free to disagree with me, but I just don't think ONNX is ready for prime-time for a model like RNN-T without someone being designated to go and make code changes to onnxruntime and pytorch. @G4V @psyhtest @guschmue 

Sounds like the audio preprocessor might be a show stopper, but instead can it be removed from the timed benchmark? For image classification, preprocessing isn't timed, so there's precedent.

To test my concerns re the LSTM I exported just the pre LSTM and set the sequence length variable and what came out looks good, so the only limitation is with the batch size, I think. And if I understand correctly @galv, the batch size is always one?

So I was hoping that for the whole of the model to export successfully (barring the audio preprocessor) then the problem to iron out is to make the loop over the logits through the third LSTM, ONNX exporter friendly.

Certainly can have a look at creating a test, time permitting. Trying to visualize how it would look as it's effectively the whole model.

The batch size is not necessarily one in this benchmark. For example, in
the offline scenario, a way to get about a 2x speedup is to sort your data
by length, and then run with minibatches. However, I did not upload the
code to do this, because I was concerned about maintenance.

However, it's conceivable that a submission could be locked into using only
a single sample, if that's what you wanted.

Unfortunately, we made the decision towards the start of RNN-T benchmark
development that featurization would be part of the benchmark:
https://github.com/mlperf/inference/pull/582/files#r407876804

The preprocessing permitted for vision models (reshaping, etc.,
https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#71-pre--and-post-processing)
is
really not comparable to the featurization done in speech models, which
involves actual meaningful computation.

Re: tests, okay, but just understand that I cannot provide very meaningful
support if I cannot reproduce your exact issues.

On Thu, Jul 9, 2020 at 1:46 AM Gavin <notifications@github.com> wrote:

> Sounds like the audio preprocessor might be a show stopper, but instead
> can it be removed from the timed benchmark? For image classification,
> preprocessing isn't timed, so there's precedent.
>
> To test my concerns re the LSTM I exported just the pre LSTM and set the
> sequence length variable and what came out looks good, so the only
> limitation is with the batch size, I think. And if I understand correctly
> @galv <https://github.com/galv>, the batch size is always one?
>
> So I was hoping that for the whole of the model to export successfully
> (barring the audio preprocessor) then the problem to iron out is to make
> the loop over the logits through the third LSTM, ONNX exporter friendly.
>
> Certainly can have a look at creating a test, time permitting. Trying to
> visualize how it would look as it's effectively the whole model.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/637#issuecomment-655997411>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABEL6UC32YGNKSCIQM3B4OLR2V7XJANCNFSM4OMUDQPQ>
> .
>


-- 
Daniel Galvez
http://danielgalvez.me
https://github.com/galv


I'm all for having an ONNX model for RNN-T.

But the goal of the model selection is to make submissions reflective of production inference. We don't want to make decisions about models and submissions based on constraints of an interchange format or its framework exporter (especially given that ONNX is a nice-to-have under the current rules.) 

Running at large batch in datacenter will be common, and multistream scenario strongly implies that you'll do that, both in scenario design and practicality of run length. And audio preprocessing, as Anton mentioned, involves meaningful work and (unlike images, where the samples may be coming from camera so may not need JPEG/resize/crop) is always there.

A reasonable compromise until the format and exporter reach adequate capability could be to export the DNN part of the model at batch 1, and let submitters convert the batchsize and STFT manually.



@galv - apologies, my comment wasn't meant to come across as uncooperative. As I say, happy to help, and further, respectful of your time.

Would it be useful for me to share a branch of the benchmark where I've done enough to export the DNN part of the model as a whole (even if what's produced is incorrect)? Viewing in Netron, the LSTMs look fine, it's in the decoder block, something goes awry.


@G4V Sure, I would be happy to take a look at that.

Apologies again @galv, it's taken me a while to get the branch together that allows the model to at least export, and looking at it, there are very few changes so not sure it will give you any great insights.

https://github.com/G4V/inference/tree/onnxexport

To get the model to export I made changes to return a format the exporter is happy with, which was changing the use of ints to tensors, and secondly, removed all calls to convert to torchscript. I now realise that torchscript is required to accommodate the loop in the decoder.

Although all datatypes have been converted to tensors, their comparison to each other still throws up warnings from the exporter. This may be something that conversion to torchscript resolves as it looks like it's in and around control flow?

I've tried also to get the decoder into a position where on it's own it converts to torchscript, but as yet been unsuccessful.

Another thing I've attempted is to make the output of fixed size (maximum of all sample lengths) and return the result as a single tensor + actual length (something you've mentioned previously), but for me this gave a model that was just the hard coded output representing the example sample passed into the exporter.



will continue for v0.8

Hi, Is there RNN-T onnx export script useful? This problem bother me a bit long time!!!! @galv @G4V

Hi, would you have any update on this issue??? @christ1ne 

Hi. Not that I know of. The conversion of the model to ONNX is non trivial
as I understand.

On Mon, 24 May 2021 at 10:48, RobotTim ***@***.***> wrote:

> Hi, would you have any update on this issue??? @christ1ne
> <https://github.com/christ1ne>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlcommons/inference/issues/637#issuecomment-846924474>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AI4GCTBP3GG57VZC5N47NVDTPIOIRANCNFSM4OMUDQPQ>
> .
>


@RobotTim please take alook at this comment for my most recent update on the issues with ONNX export: https://github.com/mlcommons/inference/issues/637#issuecomment-655818489

Yes, my understanding too that there are parts missing. If I remember correctly @psyhtest had some of it work but not all.