Paper: Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR-2024)

Reviewer comment (Novelty):
The idea of using a generative way to model audio and solve multiple audio tasks in one model is not a new idea. For example, we have a unified VAE approach to do source separation and transcription [1]. Of course the current work is novel but it would be better to include those studies and compare the approaches in related work.

Author response:
Thank you for pointing this out. We will definitely include and discuss the mentioned paper in our section on generative separation.

Reviewer comment (Novelty):
What is the novelty compared to NCSN-BASIS? Any intuition for MSDM Dirac? Is it a marginal improvement from existing methods, or not?

Author response:
The main difference between NCSN-BASIS and ISDM is the proposed DIract likelihood, which allows us to consistently improve generation/separation results with respect to the Gaussian one, used in NCSN-BASIS. This better performance may be attributed to Dirac's enforcement of mixture consistency during inference. Moreover, we propose the MSDM model, able to generate consistent mixtures composed of multiple tracks of individual instruments, which is not possible with NCSN-BASIS.

Reviewer comment (Writing):
The introduction of ISDM method is not clear. Is it a baseline method where the assumptions are obviously wrong? Is it another valid approach? Compared to MSDM, is there a trade-off in terms of generation quality and separation performance?

Author response:
Thank you for pointing out this lack of clarity in the explanation of the role of ISDM. Essentially the method allows us to demonstrate the effectiveness of generative source separation when combined with Dirac likelihood. It is indeed **not** a valid approach for total and partial generation because the interdependencies between sources are not modeled, and thus it is unable to generate the constituents of a full track, contrary to MSDM (while able to generate a single type of source). This explanation has been added to the section where ISDM is introduced in the revised paper.

Reviewer comment (Evaluation):
The separation result of ISDM and MSDM in Table 2 suggests a trade-off in terms of generation quality and separation performance. Is it confirmed? If so, why using a unified approach for source separation and generation? I understand on one hand, ISDM helps to show the diffusion method is comparable to the SOTA method. However, since MSDM is always lower than ISDM, I doubt the fundamental assumption in this paper is not valid that it is superior to use joint distribution to model generation and separation at the same time.

Author response:
We thank the reviewer for the question. Actually, we do not claim that modeling the joint distribution (MSDM) brings a significant benefit to the separation task, but it enables the generalization of consistent instrumental stems, which is not possible with ISDM. 

As for the tradeoff, a direct comparison of the quality of the two models is not possible only through the results contained in Table 2 because the total number of parameters in the two methods is different. Indeed, each model in ISDM uses the same number of parameters as the full MSDM, resulting in the ISDM being able to exploit 4 times parameters to model all 4 instruments. The goal of Table 2 was to show the effectiveness of generative source separation when combined with the proposed Dirac likelihood (in both ISDM and MSDM).

In general, having a single model, like MSDM, for multiple tasks is more versatile, not requiring specialized solutions for different individual tasks, similar to how LLMs can handle different tasks with a single model.

Reviewer comment (Reproducibility):
Could you provide some demos for ISDM method (Now or after publication)?

Author response:
Thanks for your interest, we will certainly provide demos for ISDM separation on the GitHub page after publication.

Reviewer comment (Reproducibility):
Adding details on the correction step could make the paper even more self-contained.

Author response:
We thank the reviewer for pointing out this improvement. In the revised paper, we have moved the section on the sampler to the main paper and clarified some passages. Let us know if this makes it clearer.

Reviewer comment (Evaluation):
It would have been great to compare the inference time between the different methods, as the diffusion-based methods are likely to be order of magnitudes bigger.

Author response:
We thank the reviewer for the question. We reported a table with the requested inference times in the supplementary material. According to Table 4, inference times for 12 seconds of audio on a single NVIDIA RTX A6000 are approximately 0.1s for Demucs, 5s for MSDM, 20s for ISDM, 30 seconds for Demucs + Gibbs (256 steps) and 1 minute for Demucs + Gibbs (512 steps). Demucs + Gibbs (256 steps) was added because 256 is the minimum number of steps that makes the SI-SDRi over all instruments (17.59) greater than the one of ISDM. While ISDM and MSDM are not time-competitive to Demucs, as the reviewer pointed out, they are more time-efficient compared to Demucs + Gibbs (256 and 512 steps).

Reviewer comment (Evaluation):
The results of the paper would be made stronger with more discussion of the computational footprint and details for training and inference. How does the footprint compare to Demucs or other methods? How does computation scale with the amount of data?

Author response:
We provided a table (Table 4) showing inference times for Demucs, Demucs + Gibbs, ISDM, and MSDM in the appendix of the revised paper. According to Table 4, inference times for 12 seconds of audio on a single GPU are approximately 0.1s for Demucs, 5s for MSDM, 20s for ISDM, 30 seconds for Demucs + Gibbs (256 steps) and 1 minute for Demucs + Gibbs (512 steps), making ISDM time-competitive compared to Demucs + Gibbs (256 and 512 steps). Gibbs (256 steps) was added because 256 is the minimum number of steps that makes the SI-SDRi over all instruments (17.59) greater than the one of ISDM.

As regards training times, our models (MSDM and the four stems of ISDM) on Slakh were trained for around 400 epochs each, employing around 46 minutes per epoch, while MSDM on MusDB was trained for 1700 epochs, each one lasting around 11 minutes. The only comparative training time available is for Demucs on MusDB, because training times for the models trained on Slakh are not reported in [1]. Traning Demucs v2 on MusDB employs 360 epochs, each one lasting around 4 minutes. All the timing is intended on a single GPU, an NVIDIA V100 with 16GB of RAM for Demucs and an NVIDIA Quadro RTX 6000 with 24GB of RAM for our models.

[1] Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, and Jesse Engel. Improving source separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

Reviewer comment (Evaluation):
Some brief qualitative analysis of the results might be warranted. Do the authors have an explanation for the relative performance on certain stems relative to Demucs? What are the high-level takeaways from the qualitative results in Table 3 beyond those that the reader could intuit or speculate about?

Author response:
For Demucs, MSDM, and ISDM, separating Guitar and Piano presents a notable challenge due to their similar sound profiles in Slakh2100. This similarity often leads to source spilling between the two stems during separation, potentially harming performance. Interestingly, Gibbs sampling seems to enhance the performance of Demucs + Gibbs in these cases. The performance boost likely stems from Gibbs sampling's property of fixing one source while generating the other.

Looking at Table 3, we notice that increasing the number of conditioning sources (thereby reducing the number of generated sources) consistently improves the FAD score in a near-monotonic manner. This improvement is logical, considering that more conditioning data simplifies the generation task, and more ground truth sources are included in the evaluated mix. However, when generating a single source (Drums) while conditioning on the other three (Bass, Guitar, Piano), we observe a decline in FAD score. This suggests that the strong rhythmic structure provided by the drums makes it easier to generate the other sources, in line with reviewer K9Yj's observations. In future work, we plan to explore the integration of additional conditioning elements, like the controls in the study found at [2], which could complement the conditioning sources in our model.

[2] Shih-Lun Wu, Chris Donahue, Shinji Watanabe, & Nicholas J. Bryan.  (2023). Music ControlNet: Multiple Time-varying Controls for Music Generation.

Reviewer comment (Experiments):
Is there an understanding of the approach's data efficiency? How much does Slakh2100 versus MusDB alone? What about compared to other settings in Demucs?

Author response:
In Table 6 in the appendix of the revised paper, we added SISDR metrics for a model trained on MusdBD alone. As we can see, the performances are not comparable with Demucs, being the latter specifically trained for the separation task. Generative properties are notoriously data-hungry (for example, Mousai [3] has been trained on more than 2500 hours of audio).

As for the Demucs settings, we are unsure of what the reviewer is asking; could the reviewer please clarify it?

[3] Schneider, F., Jin, Z., & Scholkopf, B. (2023). Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion*. arXiv preprint arXiv:2301.11757*.