Paper: Masked Audio Generation using a Single Non-Autoregressive Transformer (ICLR-2024)

Reviewer comment (Novelty):
A problem with the work is related to the title and the overall tone, in which the authors claim proposing a new type of audio/music model. In a paper titled "Masked Audio Generative Modeling," it is supposed that the authors proposed for the first time a masked model for audio. Nonetheless, the papers introducing such an idea in audio are SoundStorm (which the authors cite correctly) and VampNet https://arxiv.org/abs/2307.04686 (which the authors should cite as concurrent work). The authors could change the title of the article to better delineate the paper as a set of improvements of such models (providing at least one of the two as baselines).

Author response:
**Regarding the evaluated baselines:** We would like to first emphasize the main differences between SoundStorm and VampNet to the proposed method. 

SoundStrom was proposed to speed up inference time for the AudioLM [1] model, while the authors presented results for the task of text-to-speech only. As it is based on AudioLM, SoundStorm first models semantic tokens in an autoregressive manner, specifically using w2v-bert tokens [2]. Hence, it is not clear how to convert such a model to text-to-music generation considering the semantic tokens needed for the task. Nevertheless, following the reviewer's request, we trained a version of SoundStorm, using the exact setup as described in the original soundstorm paper while we consider the first EnCodec codebook as semantic tokens. Results are reported below. As can be seen MAGNeT performs better across all metrics. Notice, the number of steps of the SoundStorm model is significantly higher than MAGNeT due to its autoregressive modeling of the first codebook. 

As for the VampNet model, the authors presented results for music inpainting and super-resolution and did not explore full-blown text-to-music generation without an additional **audio-prompting**. Moreover, unlike the proposed method, VampNet is based on two different models, one for the "coarse” tokens and one for the “fine” tokens. MAGNeT uses a single model to generate all tokens. We updated the manuscript to include VampNet in the related work. 

|Model  | Fad$_{vgg} \downarrow$ | Kl $\downarrow$ | Clap$_{scr} \uparrow$ | # Steps | Latency (s) |
|-----------------|----------------------------------------|--------------------------|---------------------------------------|--------------------------|--------------------------|
| Mousai          | 7.5                                    | 1.59               | 0.23                            | 200               | 44.0                 |
| MusicLM         | 4.0                                    | -                        | -                                     | -                 | -                    |
| AudioLDM 2      | 3.1                                    | 1.20               | 0.31                                    | 208               | 18.1                 |
| MusicGen-small | 3.1                                    | 1.29               | 0.31                              | 1500              | 17.6                 |
| Musicgen-large | 3.4                                    | 1.23               | 0.32                               | 1500              | 41.3                 |
| MAGNeT-small   | 3.3                                    | 1.12              | 0.31                           | 180               | 4.0                  |
| MAGNeT-large   | 4.0                                    | 1.15            | 0.29                          | 180               | 12.6                 |
| *SoundStorm-small | 7.5                     | 1.63           | 0.18                         | 1518               | 18.1                 |

Reviewer comment (Novelty):
Except for the masking strategy, which I find important, the other introduced techniques seem relatively not critical or novel, especially the rescorer model, given that it improves the FAD only marginally.

Author response:
**Regarding the novelty of the proposed method:** We would like to state our contribution: (i) We present a **single** non-autoregressive model for the task of audio modeling and generation (this is in contrast to prior work which uses a cascade of models). We empirically demonstrate the proposed method is able to generate relatively long sequences, **30 seconds long**; (ii) We leverage an external pre-trained model during inference to improve generation quality via a rescoring method; and (iii) We show how the proposed method can be combined with autoregressive modeling to reach a single model that performs joint optimization (Hybrid-MAGNeT). 

**Regarding the impact of the rescorer:** In Table 3, we presented an ablation study to better understand the effect of the rescorer. Results suggest that the rescoring method does improve model performance across all metrics, both objective and (more importantly) human study. 

Moreover, we believe the usage of external models to guide and constrain the generation process (e.g., the rescoring method), especially in the non-autoregressive setup, opens up a new and exciting research direction that we hope the community will find interesting and valuable.

Reviewer comment (Theory):
The scheduler $\gamma(i; s) = \cos( \pi(i−1)/ 2s )$ is badly defined because computed on $i=1$ returns still $1$.

Author response:
**Regarding the scheduler:** We believe there is a misunderstanding as at the beginning of iterative decoding we would like to have a 100% masking rate. Which is exactly what we get when setting i=1. We would be happy to provide more details if we misunderstand the reviewer.

Reviewer comment (Evaluation):
I do not understand why the new CFG anneal performs better.

Author response:
**Regarding CFG annealing ablation:** We follow a similar approach to the one proposed by [3] for textually guided 3D shape generation. In preliminary studies, we found such an approach to perform better than a CFG without annealing. 

In the Table below we provide results of the proposed method considering different settings of CFG. We report FAD, KL, and CLAPScores. Results suggest that while some settings provide comparable performance considering one metric, following CFG annealing using 10->1 provides the best results considering all metrics.


$\lambda_0 \rightarrow \lambda_1$      | 1 $\rightarrow$ 1 | 3 $\rightarrow$ 3 | 10 $\rightarrow$ 10 | 10 $\rightarrow$ 1 | 20 $\rightarrow$ 20 | 20 $\rightarrow$ 1 |
|----------------------------------------|-------------------|-------------------|---------------------|--------------------|---------------------|--------------------|
| Fad$_{\text{vgg}} \downarrow$ |  3.95  | 0.99  | 0.63  | **0.61**   | 0.80   | 0.68 |
| Kl $\downarrow$               | 0.79 | 0.60 | 0.56   | 0.56  | 0.57   | 0.56 |
| Clap$_{\text{scr}} \uparrow$   | 0.13 | 0.28 | 0.28   | 0.31  | 0.31   | 0.31 |

Reviewer comment (Evaluation):
Additionally, can the authors explain better why the restricted context improves the metrics? Intuitively, the only advantage should be training/inference time improvement.

Author response:
**Regarding the impact and motivation of the restricted context:** The EnCodec model was optimized using residual vector quantization. Meaning that codebooks greater than one, are mainly influenced by the previous codebooks at the same time-step. Following that modeling paradigm, MAGNeT should not learn dependencies between the different timesteps outside the EnCodec receptive field window for codebooks above one. 

In Table 2, we presented an ablation study of the proposed method with and without a restricted context constraint. Results suggest that restricting the context of codebooks higher than 1 constantly improves results. As stated before, this method highlights a specific modeling approach designed for the chosen encoding method. From listening tests, we additionally observe that when we do not follow the restricted context method, the model discards the harmony/music texture and mainly focuses on rhythmic patterns. We updated the demo page on the supplemental material with several music samples demonstrating such an effect.