Repository: mlcommons/inference
Issue: [RNN-T] question about _max_symbols_per_step

In RNN-T decoder, there is a check against _max_symbols_per_step.  My understanding is this check ensures decoder will stop in definite steps during decoding.  Thus its a stop condition.
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/decoders.py#L84
However, in a batched decoder, determine whether symbols added is exactly _max_symbols_per_step seems not very efficient. Like the batch decoder below.
https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/Intel/code/rnnt/pytorch/decoders.py#L280
After all, it's a condition to ensure decoder would stop thus it also make sense if we only check whether no one in the batch met _blank_id for _max_symbols_per_step.  Does it conform to the intention of the author?   This condition would never be true for test case and check it in a relaxed way seems a valid optimization for real life application.

Hi @rnaidu02 can we check with author of RNN-T for comments?

+ @galv 

@delock I'm sorry, can you state more straightforwardly what you are asking? I've reread your message a few times, but I don't get the question. What would you like to change? I think the most helpful thing is if you could show code with your proposed change. If it is in fact equivalent to the reference code in outcome, it seems like it would be permitted by MLPerf inference rules.

Hi Galv, the pseudo code of our proposed check would be like the following.  In our batch greedy decoder implementation, symbols_added in step 1 is a vector, time_idxs in step 2 is also a vector.  We propose to use these two to prevent deadloop in greedy decoder.
1. run the decoder loop for _max_symbols_per_step without check whether symbols_added > _max_symbols_per_step
2. Check if there is no update on time_idx during these 30 steps.  If it is true, advance time_idx by 1 for all its elements (except those already exceed input length), otherwise, do nothing.

With this implementation, we check whether the sequence generated for current input is too long only once in 30 iterations.  This would make effective _max_symbols_per_step >30, but it could also effectively prevent deadloop in decoder, if this is what this check intend to be.   It is also more friendly to a vectorized batch greedy decoder.

@galv @delock  is proposing an alternative approach to test the "**symbols_added > self._max_symbols_per_step**" condition in batch greedy implementation and would like to check with you to see if this is acceptable. Can you please look?

for galv's comments.  the output is exactly the same because the condition
that max symbols per step is hit never be true for test dataset.

rameshchukka ***@***.***> 于 2021年7月20日周二 上午8:48写道：

> @galv <https://github.com/galv> @delock <https://github.com/delock> is
> proposing an alternative approach to test the "*symbols_added >
> self._max_symbols_per_step*" condition in batch greedy implementation and
> would like to check with you to see if this is acceptable. Can you please
> look?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlcommons/inference/issues/970#issuecomment-882960090>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AETZ4U5LRPXW22BDRYVBMO3TYTBXBANCNFSM47YZMQLA>
> .
>


Is there conclusion?   Could we use the alternative stop condition as mentioned below?

> Hi Galv, the pseudo code of our proposed check would be like the following. In our batch greedy decoder implementation, symbols_added in step 1 is a vector, time_idxs in step 2 is also a vector. We propose to use these two to prevent deadloop in greedy decoder.
> 
> 1. run the decoder loop for _max_symbols_per_step without check whether symbols_added > _max_symbols_per_step
> 2. Check if there is no update on time_idx during these 30 steps.  If it is true, advance time_idx by 1 for all its elements (except those already exceed input length), otherwise, do nothing.
> 
> With this implementation, we check whether the sequence generated for current input is too long only once in 30 iterations. This would make effective _max_symbols_per_step >30, but it could also effectively prevent deadloop in decoder, if this is what this check intend to be. It is also more friendly to a vectorized batch greedy decoder.



Does the proposed implementation always yield the same result as the reference implementation?

The result is the same if max symbols per step is not exceeded.  that is,
for each input feature, no more than 30 characters are generated.  in the
test suite, this limit is never exceeded.

per our understanding, max symbols per step is a stopping criteria that
prevent deadloop, otherwise a random initialized model may never stop
during evaluation.  our proposed method is an alternative stopping criteria
that prevents deadloop, which is more friendly to vectorized batch decoder,
and yield identical result for test set.

tjablin ***@***.***> 于 2021年8月6日周五 上午1:33写道：

> Does the proposed implementation always yield the same result as the
> reference implementation?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlcommons/inference/issues/970#issuecomment-893652227>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AETZ4U5G4LAYAQMNKSNEF6DT3LDNVANCNFSM47YZMQLA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
> .
>