Paper: SoundStorm: Efficient Parallel Audio Generation (ICLR-2024)

Reviewer comment (Novelty):
Technical novelty is quite limited. The masking and decoding algorithms are adapted from MaskGIT. Probably the main difference is that SoundStorm enables the mask on multiple RVQ levels.

Author response:
While some of the building blocks of our model come from previous works, we would like to emphasize that, in contrast to previous works, we show for the first time that it is possible to generate all tokens of a residual vector-quantized neural audio codec non-autoregressively based on conditioning signals such as the semantic tokens with no quality loss.

Reviewer comment (Experiments):
It is unclear why Conformer is being used instead of the same architecture as AudioLM; no comparison is shown.

Author response:
We are planning to add a comparison to a Transformer backbone.

Reviewer comment (Reproducibility):
Reproducibility: most of the components in the proposed approach and even baseline AudioLM are not open-source. While I understand this is not controllable for certain reasons, more implementation details, e.g., optimizers, learning rate, batch size, the cross-entropy loss when convergence, etc., should be included in the paper so that there is a higher chance of reproducing the results and have a fair comparison for future work.

Author response:
Thank you for pointing this out, we added these details for the revision.

Reviewer comment (Experiments):
The number of RVQ levels affects the number of heads. Usually, more heads take more time to converge. Are there any insights into the relationship between the number of RVQ levels and the convergence in the training?

Author response:
The number of RVQ will not only impact the training time, but also the generation quality and inference decoding steps – is an interesting avenue we are planning to investigate.

Reviewer comment (Experiments):
How important are the conditional semantic tokens in the proposed method? Would it still be possible to train with other types of semantic tokens?

Author response:
The conditioning information is very important for our method. Increasing the amount of information makes the synthesis task easier, while, on the other hand, we could not achieve promising results when replacing the semantic tokens with the transcripts.

Reviewer comment (Experiments):
The experiments are not convincing. The authors only compare the proposed SoundStorm with the AudioLM baseline. However, there are many works such as VALL-E [1] and NaturalSpeech 2 [2] that have successfully addressed the high-quality zero-shot TTS task. More comparison and discussion should be included to demonstrate the generation performance. Furthermore, prosody is also an important aspect of generation quality. Evaluation of prosody should also be conducted.

Author response:
Our main claim is that we can speed up the AudioLM framework by 100x without loss of quality. Accordingly, all our experimental evaluations are designed for verifying this claim and ablating the design choices. Our purpose with this framing is that our method is directly applicable without loss of quality to all models relying on the AudioLM framework, including SPEAR-TTS [1] which has been shown to significantly improve over VALL-E. We agree regarding the importance of prosody, which is also captured by our MOS study — Appendix A shows that the naturalness of the prosody is one of the criteria for rating.

Reviewer comment (Evaluation):
The human evaluation should be more detailed. In this paper, only MOS is reported. To measure the generation similarity, SMOS results should be reported.

Author response:
Our subjective listening study does capture the generation similarity —  Fig. 6 in Appendix A shows that the noticeable drifts in the speakers’ voices is one of the criteria for rating.

Reviewer comment (Experiments):
In the human evaluation test (Table 2), the author uses ground-truth semantic tokens, which is not fair for baselines.

Author response:
We are unsure what is meant by “uses ground-truth semantic tokens, which is not fair for baselines”. The gold baseline (“Ground Truth”) in Table 2 is the SoundStream reconstruction of the original audio, whereas the other methods are evaluated for the same task they are trained for, that is, audio synthesis based on the semantic tokens extracted from the original audio.

Reviewer comment (Reproducibility):
Some details are missing to reproduce the results easily. Specifically:  
a) What are your temperature and top-k values for sampling masked positions and sampling tokens in a confidence-based sampling scheme?  
b) Do the “mask” tokens in different RVQ layers share the same embedding?

Author response:
Thank you for pointing these out, we added these details to the revised version.

Reviewer comment (Novelty):
Related work coverage: Some works such as NaturalSpeech 2 [2] also use an efficient framework (latent diffusion) to model the tokens of neural audio codecs.

Author response:
We added the suggested reference, although we note that NaturalSpeech 2 models the quantized continuous latents, not the discrete tokens.

Reviewer comment (Evaluation):
Can SoundStorm maintain the features of the prompt such as the acoustic environment and the speaker’s emotion?

Author response:
a) SoundStorm is conditioned on the semantic tokens. Hence the answer to this question is highly dependent on the type of information the semantic tokens contain. For example, the semantic tokens used in our paper do not fully capture the speaker id, but as shown in the MOS study, SoundStorm manages to maintain the speaker id of the prompt. The speaker's emotion is mostly contained in the semantic tokens.

Reviewer comment (Experiments):
Since SoundStorm has a sampling mechanism in the acoustic token generation, have you tried the diversity of generated speeches? For example, an AR model can generate different prosody while preserving similar timbre, content, etc.

Author response:
b) Given the same semantic tokens, similar to AudioLM’s original acoustic generator, SoundStorm will sample the same semantic content but with different speakers and prosody variations – we provided samples from the same semantic tokens in the supplementary materials.

Reviewer comment (Experiments):
How does the prompt length affect the generation quality?

Author response:
c) We have not conducted a study on the effect of the prompt length, but it is an interesting direction for future work.

Reviewer comment (Evaluation):
The WavLM similarity for Codec reconstruction is 0.63 – 0.66, which is low. Does it mean the Codec reconstruction quality is not good?

Author response:
a) The SoundStream codec used in the paper (6kbps) provides close-to-perceptually lossless reconstruction for speech, however, the similarity score model is not perfectly robust to imperceptible codec artifacts, resulting in lower speaker similarity scores. Nevertheless, we find that the scores correlate strongly with subjective listening test ratings.

Reviewer comment (Evaluation):
For Voice preservation and Acoustic consistency results with a speaker prompt setting, do you use a prompt reconstructed by audio codec as an evaluation reference or an original prompt (which means the reference prompt is not constructed by codec)?

Author response:
b) We use the reconstructed prompts.

Reviewer comment (Writing):
The main weakness of the paper is the writing part which lacks some details sometimes, or the description is not clear without reading the MaskGIT reference paper.

Author response:
1-3. Thank you for pointing these out, we addressed them in the revised version.

Reviewer comment (Writing):
In the 2nd paragraph of Sec. 3.3, when you mention “the conditional independence assumption in finer levels”, you should add that this assumption is made along the time dimension. Otherwise, it could confuse the readers that such conditional independence exists along the RVQ level dimension which conflicts with the fact that inference is performed from lower level to higher level.

Author response:
1-3. Thank you for pointing these out, we addressed them in the revised version.

Reviewer comment (Experiments):
In Sec. 3, for the ablation study of number of decoding steps, did you also perform such experiment to measure the effect of the number of decoding steps on the WER/CER performance?

Author response:
Our MOS study captures this aspect partially, where the ratings (which also capture speech intelligibility) do not increase by using more iterations. We will also add the automated metrics.

Reviewer comment (Experiments):
In Sec. 3.3, iterative parallel decoding, have you tried to replace the unmasked tokens from previous inference stage with the estimation from current inference stage with a higher confidence score? The question here applies to both current RVQ layer and previous RVQ layers.

Author response:
We did not try this, but we agree that this is a promising future direction.

Reviewer comment (Evaluation):
The original MaskGIT paper describes the limitation and failure cases of the MaskGIT method, such as semantic and color shifts, ignore and modify objects on the boundary when applied to outpainting and inpainting, oversmoothing or creates undesired artifacts on complex structure, etc. Are these limitations also appliable to speech generation task? Could you comment the technique limitations of the SoundStorm method?

Author response:
In contrast to MaskGIT, which was proposed with direct text conditioning, we proposed SoundStorm with semantic token conditioning. Arguably, our setup is “easier” in the sense that our semantic tokens already contain a significant amount of information (phonetic content, partial prosody information, some speaker information), which leaves less potential for failures when used as conditioning. This also suggests the limitation of SoundStorm: extending it to weaker conditioning signals such as text could be non-trivial.