  Streaming end-to-end automatic speech recognition (ASR) systems are widely
used in everyday applications that require transcribing speech to text in
real-time. Their minimal latency makes them suitable for such tasks. Unlike
their non-streaming counterparts, streaming models are constrained to be causal
with no future context and suffer from higher word error rates (WER). To
improve streaming models, a recent study [1] proposed to distill a
non-streaming teacher model on unsupervised utterances, and then train a
streaming student using the teachers' predictions. However, the performance gap
between teacher and student WERs remains high. In this paper, we aim to close
this gap by using a diversified set of non-streaming teacher models and
combining them using Recognizer Output Voting Error Reduction (ROVER). In
particular, we show that, despite being weaker than RNN-T models, CTC models
are remarkable teachers. Further, by fusing RNN-T and CTC models together, we
build the strongest teachers. The resulting student models drastically improve
upon streaming models of previous work [1]: the WER decreases by 41% on
Spanish, 27% on Portuguese, and 13% on French.
