RNNT Decoder outputting blank only
State: closed | Pull request: no
Labels: name: bug

name: stale

I am trying to train a Conformer-Encoder based ASR model on a relatively small and bad quality dataset. So far, the model trains via a CTC loss to about 58% WER and I tried to extend it to use a hybrid CTC/RNNT loss as it can also be done in NeMo. 
Due to the necessity to adapt to an existing codebase, I am not running the provided NeMo scripts/models for training, but rather I create an instance of the required classes for RNNT-models (RNNTLoss, RNNTJoiner, RNNTDecoder) and then include those into my own PyTorch Lightning module. To get all tensors right and fed to the correct modules, I went through the nemo code step by step with a debugger while running the default _speech_to_text_hybrid_rnnt_ctc_bpe.py_ script.  

I feed the zero-padded and tokenized transcripts to the decoder and the output goes to the joiner along with the conformer encoding. I then use the resulting tensors to obtain the RNNT-Loss:
```
decoder, target_length, states = self.decoder(targets=transcript, target_length=transcript_len)
joint = self.joint(encoder_outputs=encoded, decoder_outputs=decoder)
loss_value = self.loss(log_probs=joint, targets=transcript, input_lengths=encoded_len, target_lengths=target_length)
```
(which is basically how it is done in the training step of _nemo/collections/asr/models/rnnt_models.py_ )

So far this works well and I can see the RNNT-Loss dropping (from around 180 to 60 for 'mean_batch' reduction), both when using it alone and when using it together with a CTC loss.
I observed that the performance (in terms of WER) when decoding the CTC output layer's output stays about constant. I am not really sure whether using the RNNT-Loss along with the CTC-Loss will also improve the quality of the CTC decoding, so I decided to try the RNNT decoding. 

Same principle, I instantiated the RNNTDecoding object directly by passing the configuration and the decoder network and joiner network of my model to create a RNNTDecoding object.
However, passing my conformer encoding outputs to _RNNTDecoding.rnnt_decoder_predictions_tensor(...)_  (of the RNNTDecoding instantiated with my joiner and decoder) always results in an empty output/hypothesis with score 0. 
Taking a deeper look into the decoding process (in _ _greedy_decode_blank_as_pad()_ ) shows that in every step the log_probability (logp) of the blank token is extremely high (around 10-12) while lying around -2 .. 2 for all other tokens, which in the end results in an empty hypothesis. This seems especially weird to me since a log_probability is not meant to be positive. 
To get more insight, I ran a pretrained RNNT model (large conformer transducer) and took a look a the same place in the code. While my encoder and decoder outputs look similar (and also f and g) in terms of the value range, logp will always be around -100 .. -20 for this model, but never positive. 
Going into the "joint" step showed that there is some form of log_softmax calculation, but only if the signal is on the cpu. I noticed a similar thing in the RNNT loss already, where (as far as I understood) the log-softmax calculation is handled by numba in case of a GPU tensor.

Formulating an actual question seems hard (and I hope that this is "suited" to be an issue anyway, sorry if it isn't .. ) .. 
But why does the usage of the RNNT decoding fail, although the loss seems to work fine?

Is there any general misconception that I have using the RNNT elements of the library? Are they not meant to be used outside of the NeMo world? 

Why does the logp-signal in   _ _greedy_decode_blank_as_pad_ seems to be log-softmaxed in the pretrained model but this operation is obviously not performed when the class is used in an isolated way? If there is a specific reason, is there a way around it?