Paper:
VOICELOOP: VOICE FITTING AND SYNTHESIS VIA A PHONOLOGICAL LOOP
We present a new neural text to speech (TTS) method that is able to transform text to speech in voices that are sampled in the wild. Unlike other systems, our solution is able to deal with unconstrained voice samples and without requiring aligned phonemes or linguistic features. The network architecture is simpler than those in the existing literature and is based on a novel shifting buffer working memory. The same buffer is used for estimating the attention, computing the output audio, and for updating the buffer itself. The input sentence is encoded using a context-free lookup table that contains one entry per character or phoneme. The speakers are similarly represented by a short vector that can also be fitted to new identities, even with only a few samples. Variability in the generated speech is achieved by priming the buffer prior to generating the audio. Experimental results on several datasets demonstrate convincing capabilities, making TTS accessible to a wider range of applications. In order to promote reproducibility, we release our source code and models 1 .

Review:
This paper present the application of the memory buffer concept to speech synthesis, and additionally learns a "speaker vector" that makes the system adaptive and work reasonably well on "in-the-wild" speech data. This is a relevant problem, and a novel solution, but synthesis is a wicked problem to evaluate, so I am not sure if ICLR is the best venue for this paper. I see two competing goals:
- If the focus is on showing that the presented approach outperforms other approaches under given conditions, a different task would be better (for example recognition, or some sort of trajectory reconstruction)
- If the focus is on showing that the system outperforms other synthesis systems, then a speech oriented venue might be best (and it is unfortunate that optimized hyper-parameters for the other systems are not available for a fair comparsion)
- If fair comparisons with the other appraoches cannot be made, my sense is that the multi-speaker (post-training fitting) option is really the most interesting and novel contribution here, which could be discussed in mroe detail
Still, the approach is creative and interesting and deserves to be presented. I have a few questions/ suggestions:
Introduction
- The link to Baddeley's "phonological loop" concept seems weak at best. There is nothing phonological about the features that this model stores and retrieves, and no evidence that the model behaves in a way consistent with "phonologcial" (or articulatory) assumptions or models - maybe best to avoid distracting the reader with this concept and strengthen the speaker adaptation aspect?
- The memory model is not an RNN, but it is a recurrently called structure (as the name "phonological loop" also implies) - so I would also not highlight this point much
- Why would the four properties of the proposed method (mid of p. 2, end of introduction: memory buffer, shared memory, shallow fully connected networks, and simple reader mechanism) lead to better robustness and improve performance on noisy and limited training data? Maybe the proposed approach works better for any speech synthesis task? Why specifically for "in-the-wild" data? The results in Table 2 show that the proposed system outperforms other systems on Blizzard 2013, but not Blizzard 2011 - does this support the previous argument?
- Why not also evaluate MCD scores? This should be a quick and automatic way to diagnose what the system is doing? Or is this not meaningful with the noisy training data?
Previous work
- Please introduce abbreviations the first time they are used ("CBHG" for example)
- There is other work on using "in-the-wild" speech as well: Pallavi Baljekar and Alan W Black. Utterance Selection Techniques for TTS Systems using Found Speech, SSW 2016, Sunnyvale, USA Sept 2016
The architecture
- Please explain the "GMM" (Gaussian Mixture Model?) attention mechanism in a bit more detail, how does back-propagation work in this case?
- Why was this approach chosen? Does it promise to be robust or good for low data situations specifically?
- The fonts in Figure 2 are very small, please make them bigger, and the Figure may not print well in b/w. Why does the mean of the absolute weights go up for high buffer positions? Is there some "leaking" from even longer contexts?
- I don't understand "However, human speech is not deterministic and one cannot expect [...] truth". You are saying that the model cannot be excepted to reproduce the input exactly? Or does this apply only to the temporal distribution of the sequence (but not the spectral characteristics)? The previous sentence implies that it does. And how does teacher-forcing help in this case?
- what type of speed is "x5"? Five times slower or faster than real-time?
Experiments
- Table 2: maybe mention how these results were computed, i.e. which systems use optimized hyper parameters, and which don't? How do these results support the interpretation of hte results in the introruction re in-the-wild data and found data?
- I am not sure how to read Figure 4. Maybe it would be easier to plot the different phone sequences against each other and show how the timings are off, i.e. plot the time of the center of panel one vs the time of the center of panel 2 for the corresponding phone, and show how this is different from a straight line. Or maybe plot phones as rectangles that get deformed from square shape as durations get learned?
- Figure 5: maybe provide spectrograms and add pitch contours to better show the effect of the dfifferent intonations? 
- Figure 4 uses a lot of space, could be reduced, if needed
Discussion
- I think the first claim is a bit to broad - nowhere is it shown that the method is inherently more robust to clapping and laughs, and variable prosody. The authors will know the relevant data-sets better than I do, maybe they can simply extend the discussion to show that this is what happens. 
- Efficiency: I think Wavenet has also gotten much faster and runs in less than real-time now - can you expand that discussion a bit, or maybe give estimates in times of FLOPS required, rather than anecdotal evidence for systems that may or may not be comparable?
Conclusion
- Now the advantage of the proposed model is with the number of parameters, rather than the computation required. Can you clarify? Are your models smaller than competing models?