Paper:
In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model. The acoustic model requires only audio transcription for training -no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -no phonetic word lexicon is needed. Key ingredients for the acoustic model are Gated Linear Units and high dropout. We show near state-of-the-art results in word error rate on the LibriSpeech corpus (Panayotov et al., 2015) with MFSC features, both on the CLEAN and OTHER configurations.

Review:
This paper applies gated convolutional neural networks [1] to speech recognition, using the training criterion ASG [2]. It is fair to say that this paper contains almost no novelty.
This paper starts by bashing the complexity of conventional HMM systems, and states the benefits of their approach. However, all of the other grapheme-based end-to-end systems enjoy the same benefit as CTC and ASG. Prior work along this line includes [3, 4, 5, 6, 7].
Using MFSC, or more commonly known as log mel filter bank outputs, has been pretty common since [8]. Having a separate subsection (2.1) discussing this seems unnecessary.
Arguments in section 2.3 are weak because, again, all other grapheme-based end-to-end systems have the same benefit as CTC and ASG. It is unclear why discriminative training, such as MMI, sMBR, and lattice-free MMI, is mentioned in section 2.3. Discriminative training is not invented to overcome the lack of manual segmentations, and is equally applicable to the case where we have manual segmentations.
The authors argue that ASG is better than CTC in section 2.3.1 because it does not use the blank symbol and can be faster during decoding. However, once the transition scores are introduced in ASG, the search space becomes quadratic in the number of characters, while CTC is still linear in the number characters. In addition, ASG requires additional forward-backward computation for computing the partition function (second term in eq 3). There is no reason to believe that ASG can be faster than CTC in both training and decoding.
The connection between ASG, CTC, and marginal log loss has been addressed in [9], and it does make sense to train ASG with the partition function. Otherwise, the objective won't be a proper probability distribution.
The citation style in section 2.4 seems off. Also see [4] for a great description of how beam search is done in CTC.
Details about training, such as the optimizer, step size, and batch size, are missing. Does no batching (in section 3.2) means a batch size of one utterance?
In the last paragraph of section 3.2, why is there a huge difference in real-time factors between the clean and other set? Something is wrong unless the authors are using different beam widths in the two settings.
The paper can be significantly improved if the authors compare the performance and decoding speed against CTC with the same gated convnet. It would be even better to compare CTC and ASG to seq2seq-based models with the same gated convnet. Similar experiments should be conducted on switchboard and wsj because librespeech is several times larger than switchboard and wsj. None of the comparison in table 4 is really meaningful, because none of the other systems have parameters as many as 19 layers of convolution. Why does CTC fail when trained without the blanks? Is there a way to fix it besides using ASG? It is also unclear why speaker-adaptive training is not needed. At which layer do the features become speaker invariant? Can the system improve further if speaker-adaptive features are used instead of log mels? This paper would be much stronger if the authors can include these experiments and analyses.
[1] R Collobert, C Puhrsch, G Synnaeve, Wav2letter: an end-to-end convnet-based speech recognition system, 2016
[2] Y Dauphin, A Fan, M Auli, D Grangier, Language modeling with gated convolutional nets, 2017
[3] A Graves and N Jaitly, Towards End-to-End Speech Recognition with Recurrent Neural Networks, 2014
[4] A Maas, Z Xie, D Jurafsky, A Ng, Lexicon-Free Conversational Speech Recognition with Neural Networks, 2015
[5] Y Miao, M Gowayyed, F Metze, EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding, 2015
[6] D Bahdanau, J Chorowski, D Serdyuk, P Brakel, Y Bengio, End-to-end attention-based large vocabulary speech recognition, 2016
[7] W Chan, N Jaitly, Q Le, O Vinyals, Listen, attend and spell, 2015
[8] A Graves, A Mohamed, G Hinton, Speech recognition with deep recurrent neural networks, 2013
[9] H Tang, L Lu, L Kong, K Gimpel, K Livescu, C Dyer, N Smith, S Renals, End-to-End Neural Segmental Models for Speech Recognition, 2017