  We present our transducer model on Librispeech. We study variants to include
an external language model (LM) with shallow fusion and subtract an estimated
internal LM. This is justified by a Bayesian interpretation where the
transducer model prior is given by the estimated internal LM. The subtraction
of the internal LM gives us over 14% relative improvement over normal shallow
fusion. Our transducer has a separate probability distribution for the
non-blank labels which allows for easier combination with the external LM, and
easier estimation of the internal LM. We additionally take care of including
the end-of-sentence (EOS) probability of the external LM in the last blank
probability which further improves the performance. All our code and setups are
published.
