Paper:
CLARINET: PARALLEL WAVE GENERATION IN END-TO-END TEXT-TO-SPEECH
In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.

Review:
This paper proposes some modifications to established procedures for neural speech synthesis and investigates their effect experimentally. The proposed modifications are mostly fairly straightforward conceptually, but appear to work well, and this reviewer feels the paper has huge value in its experimental contributions extending and clarifying certain aspects of WaveNet training and distillation. The paper is well-written and fairly concise, with a short-and-sweet experimental results section.
Major comments:
The conceptual novelty seems a little overstated in the abstract. For example, the value seems to not really be in the "proposing" a text-to-wave neural architecture for speech synthesis (which, aside from important experimental tweaks, is essentially Tacotron 2 training all parameters from scratch) but in showing that it works well experimentally. Conceptually the paper is extremely close to the parallel wavenet paper, the main differences being slightly different component distributions (Gaussian instead of logistic), a different set of loss terms in addition to the reverse KL, and joint training of the spectral synthesis and waveform synthesis parts of the model.
It would be super insightful to include log probabilities on the test set (everywhere MOS results have been reported) in the experimental results. This would help tease apart the effects of architecture inductive bias, different divergences, distillation, etc. One of the really nice things about flow-based models is the ability to compute the log probability tractably.
Minor comments:
Perhaps mention that teacher forcing is maximum likelihood in the introduction? Currently it almost sounds like the paper is contrasting teacher forcing for WaveNet (paragraph 2) and MLE (list item 1).
At the end of paragraph 3 in the introduction, it would be helpful to mention that the intractable KL divergence being referred to is the frame-level one-step-ahead predictions, not the entire sequence-level prediction. Also, for 1D distributions isn't taking a large number of samples quite effective in practice?
In introduction list item 3, suggest mentioning Tacotron 2 (Shen et al) and contrasting with the present work for clarity.
In section 3.1, it surprises me slightly that clipping at -7 is essential. It would be helpful to state what exactly goes wrong if this is not done. Does it lead to overfitting and so bad test log likelihoods? What effect is noticeable in the generated samples?
Equation (6) is incorrect. It should be conditioned on < t, not <= t. Conditioning on z <= t would make x_t deterministic.
Equation (7) is technically true as written, but only because all the distributions involved are deterministic. If <= t is replaced with < t (which based on the mistake in (6) is what I suspect the authors intended) then it is no longer true. This equation is not used anywhere as far as I can tell. It seems to me like the property that enables non-recursive-over-time ("parallel") sampling is (5), not (7). Incidentally, when multiple one-step-ahead samples are taken per frame for parallel wavenet, the samples viewed at the sequence level are highly correlated, and do not obey anything like (7), but it doesn't affect the correctness of the expected value.
The IAF doesn't really "infers its output x at all time steps". Maybe "models" instead of "infers"?
Learning an "IAF directly through maximum likelihood" doesn't seem all that impractical. People train networks with recursive dependence such as RNNs (which is essentially what would be required to train certain forms of IAF with MLE) as opposed to non-recursive dependence such as CNNs all the time, after all. It seems like this claim depends on the details of the transform $f$.
Out of interest, did the authors consider reversing the sequence being generated in time between successive IAF blocks? This would limit the ability to do low latency synthesis but might improve performance considerably.
The first paragraph in section 3.3 seems like it should probably be part of section 3.3.1 (it's not related to other losses such as spectrogram frame loss, for example). It would be helpful to state explicitly that: (a) the goal is to minimize the sequence-level reverse KL; (b) this can be approximated by taking a single sample z, but this may have high variance; (c) the variance of this estimate can be reduced by marginalizing over the one-step-ahead predictions for each frame; (d) parallel wavenet's mixture of logistics means it has to use a separate Monte Carlo sampling at the frame-level, whereas the proposed Gaussian allows this one-step-ahead marginalization to be performed analytically. This one-step-ahead marginalization is an example of Rao-Blackwellization.
It didn't seem clear from section 3.3 and 3.3.1 that parallel wavenet also uses the one-step-ahead marginalization trick to reduce the variance.
It might be helpful to mention that using the reverse KL would be expected to have mode-fitting behavior, making samples sound better but log probability on the test set worse.
It was not clear to me what difference or similarity was being demonstrated in Figure 1.
Small point, but "Oord et al" should be "van Oord et al" throughout (it's a surname).
In section 3.3.2, can the authors give any insight as to why training with reverse KL alone leads to whispering, and why adding the STFT term fixes this? (If it's only something that's been noticed empirically, "will lead" -> "empirically we found"?)
I noticed quite a large qualitative perceptual difference between the student and teacher samples, particularly in the speech synthesis case (experiment 3), even though I think I'd rate the quality on a linear scale as fairly similar (in line with the MOS results). The teacher sounds noticeably "harsher" but "clearer" Do the authors have any insight as to why this perceptual difference occurs (if they also perceive a qualitative difference)? Is it probably a difference in inductive bias between an AF (which WaveNet can be seen as) and IAF?
I found it fascinating that reverse KL and forward KL lead to roughly the same MOS for spectrum-to-waveform. I assumed reverse KL would be better due to its preference for high-quality samples due to mode fitting.
Out of curiosity, what is responsible for the pops at the start of the spectrogram-conditioned distilled models? Also why are the synthesized samples shorter than the ground truth (less initial silence)?