  The two most common paradigms for end-to-end speech recognition are
connectionist temporal classification (CTC) and attention-based encoder-decoder
(AED) models. It has been argued that the latter is better suited for learning
an implicit language model. We test this hypothesis by measuring temporal
context sensitivity and evaluate how the models perform when we constrain the
amount of contextual information in the audio input. We find that the AED model
is indeed more context sensitive, but that the gap can be closed by adding
self-attention to the CTC model. Furthermore, the two models perform similarly
when contextual information is constrained. Finally, in contrast to previous
research, our results show that the CTC model is highly competitive on WSJ and
LibriSpeech without the help of an external language model.
