Paper:
This work presents a scalable solution to continuous visual speech recognition. To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video). In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words. The proposed system achieves a word error rate (WER) of 40.9% as measured on a held-out set. In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset when having access to additional types of contextual information. Our approach significantly improves on previous lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 89.8% and 76.8% WER respectively.

Review:
The paper presents a large-scale lipreading system - no surprises there. This is good work and probably the strongest general purpose lip-reading system out there at this time, but i don't see both the work and the paper as a good fit for ICLR.
The authors take a large corpus of YouTube videos (on which Google has already trained direct acoustics-to-word speech recognizers, and which is manually transcribed), filter it, and extract regions that can be used for lipreading. They then describe a scalable preprocessing, and train a phone-based acoustic model using CTC. They seem to be using the (Miao et al., 2015) and Google WFST based decoding framework, and achieve a word error rate of ca 40%. That is impressive, but I don't see any novelty here, and the paper is full of contradictions, and leaves some important open questions:
- the authors argue for "phonemes and ctc", and no speech person would disagree with them; in fact (Miao et al., 2015) and many other papers show that the WERs with a good phoneme based dictionary in English are lower than with a character based model. it's just easier if one does not need a dictionary.
- why are the authors not using a viseme dictionary, or map their phoneme dictionary to a viseme dictionary. In visual space, their own "homonym" argument applies, too, and "mop" (or "mom") and "pop" should be mapped to the same "viseme" sequence - and the resulting uncertainty should be handled by the decoder, and not the classifier.
- how did the authors generate the one million word phoneme vocabulary? even google used around 100,000k words in their whole-word experiments, if i remember correctly? what happens if the authros reduce the vocabulary? could you provide some error analysis or at least deletions/ insertions/ substitutiosn, and compare them against an audio system?
- LipNet and the proposed architecture seem to be very similar - maybe you could provide some insight into which changes made the biggest difference?
- is the data going to be available?
- what is a "production-level speech decoder"? how come your model "is the first to combine a deep learning-based phoneme recognition model with production-grade word-level decoding techniques" if Google does essentially the same ("in production")?
- in Section 1, you say that "by design, the trained model only performs well when videos are shot at specific angles when a subject is facing the camera, [...] It does not perform well in other contexts". in Section 5, you demonstrate the "generalization power of our V2P approach"and find that it "is able to generalize well" - please clarify
- "speech impaired patients" often have non-canonical articulation, the proposed system may not work well for them
- it would be interesting to also know the absolute levels of insertions/ deletions/ substitutions for words and/ or phonemes, and for the audio only and visual systems, to be able to diagnose what the problems are. 
- finally, Figure 10 is really hard to view - i'd be happy to be shown fewer faces, the main message is that the quality of the face detection is really good?