Paper:
Published as a conference paper at ICLR 2018 DEEP VOICE 3: SCALING TEXT-TO-SPEECH WITH CONVOLUTIONAL SEQUENCE LEARNING
We present Deep Voice 3, a fully-convolutional attention-based neural textto-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training an order of magnitude faster. We scale Deep Voice 3 to dataset sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on a single GPU server.

Review:
This paper provides an overview of the Deep Voice 3 text-to-speech system. It describes the system in a fair amount of detail and discusses some trade-offs w.r.t. audio quality and computational constraints. Some experimental validation of certain architectural choices is also provided.
My main concern with this work is that it reads more like a tech report: it describes the workings and design choices behind one particular system in great detail, but often these choices are simply stated as fact and not really motivated, or compared to alternatives. This makes it difficult to tell which of these aspects are crucial to get good performance, and which are just arbitrary choices that happen to work okay.
As this system was clearly developed with actual deployment in mind (and not purely as an academic pursuit), all of these choices must have been well-deliberated. It is unfortunate that the paper doesn't demonstrate this. I think this makes the work less interesting overall to an ICLR audience. That said, it is perhaps useful to get some insight into what types of models are actually used in practice.
An exception to this is the comparison of "converters", model components that convert the model's internal representation of speech into waveforms. This comparison is particularly interesting because some of the results are remarkable, i.e. Griffin-Lim spectrogram inversion and the WORLD vocoder achieving very similar MOS scores in some cases (Table 2). I wish there would be more of that kind of thing in the paper. The comparison of attention mechanisms is also useful.
I'm on the fence as I think it is nice to get some insight into a practical pipeline which benefits from many current trends in deep learning research (autoregressive models, monotonic attention, ...), but I also feel that the paper is a bit meager when it comes to motivating all the architectural aspects. I think the paper is well written so I've tentatively recommended acceptance.
Other comments:
- The separation of the "decoder" and "converter" stage is not entirely clear to me. It seems that the decoder is trained to predict spectrograms autoregressively, but its final layer is then discarded and its hidden representation is then used as input to the converter stage instead? The motivation for doing this is unclear to me, surely it would be better to train everything end-to-end, including the converter? This seems like an unnecessary detour, what's the reasoning behind this?
- At the bottom of page 2 it is said that "the whole model is trained end-to-end, excluding the vocoder", which I think is an unfortunate turn of phrase. It's either end-to-end, or it isn't.
- In Section 3.3, the point of mixing of h_k and h_e is unclear to me. Why is this done?
- The gated linear unit in Figure 2a shows that speaker embedding information is only injected in the linear part. Has this been experimentally validated to work better than simpler mechanisms such as adding conditioning-dependent biases/gains?
- When the decoder is trained to do autoregressive prediction of spectrograms, is it autoregressive only in time, or also in frequency? I'm guessing it's the former, but this means there is an implicit independence assumption (the intensities in different frequency bins are conditionally independent, given all past timesteps). Has this been taken into consideration? Maybe it doesn't matter because the decoder is never used directly anyway, and this is only a "feature learning" stage of sorts?
- Why use the L1 loss on spectrograms?
- The recent work on Parallel WaveNet may allow for speeding up WaveNet when used as a vocoder, this could be worth looking into seeing as inference speed is used as an argument to choose different vocoder strategies (with poorer audio quality as a result).
- The title heavily emphasizes that this model can do multi-speaker TTS with many (2000) speakers, but that seems to be only a minor aspect that is only discussed briefly in the paper. And it is also something that preceding systems were already capable of (although maybe it hasn't been tested with a dataset of this size before). It might make sense to rethink the title to emphasize some of the more relevant and novel aspects of this work.
----
Revision: the authors have adequately addressed quite a few instances where I feel motivations / explanations were lacking, so I'm happy to increase my rating from 6 to 7. I think the proposed title change would also be a good idea.