Paper:
This paper proposes a novel generative model called PUGAN, which progressively synthesizes high-quality audio in a raw waveform. PUGAN leverages on the recently proposed idea of progressive generation of higher-resolution images by stacking multiple encode-decoder architectures. To effectively apply it to raw audio generation, we propose two novel modules: (1) a neural upsampling layer and (2) a sinc convolutional layer. Compared to the existing state-of-the-art model called WaveGAN, which uses a single decoder architecture, our model generates audio signals and converts them in a higher resolution in a progressive manner, while using a significantly smaller number of parameters, e.g., 20x smaller for 44.1 kHz output, than an existing technique called WaveGAN. Our experiments show that the audio signals can be generated in real-time with the comparable quality to that of WaveGAN with respect to the inception scores and the human evaluation.

Review:
The paper presents an approach based on generative adversarial models for the unconditional  generation of audio. The authors take inspiration from WaveGAN, to which they add more sophisticated upsampling blocks (called the bandwidth extension module) instead of transposed convolutions. They also propose to add a sinc convolution layer to the discriminators to improve training. Finally, they propose a progressive training scheme similar in spirit to the progressive training of GANs in images. Experiments are performed on generating audio pronunciation of digits, and the authors compare their work in terms of inception score, human evaluation and computation cost to WaveGAN.
As stated by the authors, one of the main motivation oof their approach is the reduction of computation cost. The motivation, expressed in terms of the 10ms "interactive rate" follows a good story. The measurements performed at the end of the paper show a ~ x2 performance gains compared to WaveGAN on CPU and about same running times on GPU, which is significant but not compelling.
Overall, I liked the story of the paper, but the paper lacks clarity and details. An important aspect of the paper is the progressive training, which is detailed nowhere (e.g., what is the stopping criterion to get to the next stage), should there be a special initialization of the last block of a new stage, etc.). The "Bandwidth extension module", which from my understanding is one of thee main contribution of the paper, is detailed in the appendix and comes essentially without justification. One of the main motivations of the paper is to be able to generate 44kHz audio, but the only results available at this resolution are inception scores that are below those of the 16kHz generation, which leaves open the question of whether the goal is effectively achieved.
I found the different versions of PUGAN difficult to read. The picture uses 2 blocks of bandwidth extension to generate 16kHz, whereas the evaluations are done with PUGAN-1 (1 block), which if I understand correctly is based on the lightweight WaveGAN that generates at 8kHz (whereas Section 4.1 suggests that 16kHz is generated from 4kHz generations by WaveGAN and two blocks). Also, the fact that evaluations are carried out with PUGAN-1 suggests that the progressive training does not really works well past a single block.
In the text it is suggested that spectral normalization and sinc conv on the discriminator is an "improvement" of WaveGAN, and an independent contribution of the article. While Table 2 clearly shows an improvement in terms of inception score, the human evaluation is not that clear: the accuracy of human labelers drops to 0.52 from 0.63 and the quality seems totally within the variance (2.7 +/- 1.5 vs 2.6 +/- 1.3). While Table 3 also shows clear improvement over the basic WaveGAN with the changes made by the authors (in particular in terms of win ratio vs PUGAN-1), the loss of accuracy should be discussed.
The performance obtained by PUGAN-1 is nonetheless noticeable -- +1 quality score over WaveGAN, and much better pairwise win ratios. Nonetheless, the paper lacks clarity and motivation for the exact form of the bandwidth extraction module, and does not fulfill its promises (importance of progressive training, high quality generation at 44kHz), so I am leaning towards rejection.