Paper: Separate and Diffuse: Using a Pretrained Diffusion Model for Better Source Separation (ICLR-2024)

Reviewer comment (Experiments):
The authors' choice of utilizing only one diffusion-based generative model to validate the performance enhancement brought by noise introduction appears to be limiting. Although the theoretical incapacity of deterministic generative models to achieve performance enhancement has been demonstrated, the naturalness in generation by HiFiGAN is inherently lower than that of DiffWave. This raises concerns about whether this disparity is the reason for HiFiGAN's lack of improvement. I would recommend the authors consider incorporating other diffusion-based generative models, such as FastDiff [1], or superior deterministic models like UnivNet [2], to bolster the robustness of their results.

Author response:
We thank the reviewer for the comprehensive feedback.

*Generative model selection:*

Our proposed method relies on conditioning of the generative signal based on deterministic estimation of the signal. 

FastDiff’s conditioning is based on text and thus is not in the scope of this work. 

UnivNet is a non-deterministic GAN, which has a similar input to that of the diffusion-based model that we use (DiffWave). It is indeed very interesting to evaluate since the GAN we employed in the paper (HiFiGAN) is deterministic and was employed to show that deterministic de-noising methods do not improve performance.

Following the review, we have run UnivNet in combination with Gated LSTM and our phase alignment network. The results for Libri2Mix are reported in the table below (and added to Table 1 of the manuscript). Evidently, UnivNet and DiffWaveresults are within negligible differences of around 0.1dB. This strongly reinforces our main claim.

Reviewer comment (Experiments):
Regarding the alignment network, the authors utilize the relative phase difference between $V_g$ and $V_d$ as well as the phase of $V_d$ as inputs to align the phase of the fused output with $V_d$. This poses a question: Is the observed enhancement in model performance a result of the phase alignment between $V_g$ and $V_d$, or is it due to the introduction of additional parameters, i.e., the alignment network? Another hypothesis worth considering is if the alignment network, when directly using the phase of $V_g$ and $V_d$ as inputs, would produce a similar effect.

Author response:
*Alignment Network Concerns*

Following the review, we have conducted three additional ablations that are added to the revised transcript:
Using the alignment network with $v_d$ twice.
Using  the alignment network with $v_d$ and phase-shifted $v_d$
Retraining the alignment network with only the absolute phase without the difference of the phase. Then running as in our original method.

In all tests, the results of the ablation were worse,
| Ablation Type                       | Libri2Mix | WSJ2MIX | Libri5Mix |
|-------------------------------------|-----------|---------|-----------|
| (a) Only $v_d$                            | 20.2      | 22.0    | 12.4      |
| (b) Only $v_d$ shifted                    | 20.3      | 22.1    | 12.6      |
| (c) Re-trained without phase difference | 21.3      | 23.3    | 13.7      |
| Our Method                          | 21.5      | 23.9    | 14.2      |

As can be seen, the phase alignment network does not improve performance when the generative model’s output is left out.

The alternative network architecture in (c) is somewhat inferior. In this case, the phase difference information exists only implicitly (i.e., it can be inferred from the two absolute phases but is not given explicitly).

Reviewer comment (Experiments):
One notable observation from the manuscript is its primary focus on clean datasets for evaluation. It would be beneficial to see how the proposed combined model performs on noisy datasets, such as WHAM! or the noisy version of Librimix. Evaluating on these datasets can provide insights into the model's robustness in more realistic scenarios, where environmental noise might significantly impact the performance of the generative model. Such an evaluation will offer a more comprehensive understanding of the model's real-world applicability and its ability to tackle inherent challenges posed by noisy environments.

Author response:
*Testing on Noisy Datasets:*

Following the review, we evaluated our method on the WHAM! benchmark. We used a pretrained version of SepFormer over WHAM found in hugging faces. 

| Method                      | WSJ2Mix + WHAM |
|-----------------------------|----------------|
| Sepformer                   | 16.3           |
| SepFormer + DiffWave (ours) | 16.8           |

As a side note: the additional independent noise in WHAM can be modeled, and a tighter upper bound can be computed for this dataset, using the insights of Fig. 1.

Reviewer comment (Reproducibility):
The manuscript should clearly specify whether the separation and generative models were involved in the training of the alignment network $F$.

Author response:
*Training Concerns:*

Both pretrained networks are not fine-tuned and are not part of the training of the alignment network.

Reviewer comment (Writing):
Please ensure a consistent and standardized representation of symbols throughout the paper. Conventionally, vectors are denoted in boldface.

Author response:
*Symbol Representation:*

We thank the reviewer and we will make sure that the symbols are consistent.

Reviewer comment (Experiments):
I would like to see an ablation where only the generative output is used after conditioning on the discriminative output (not a simple average like the current ablation). Have you done those experiments and how did they perform?

Author response:
> I would like to see an ablation where only the generative output is used after conditioning on the discriminative output (not a simple average like the current ablation). Have you done those experiments and how did they perform?

In the paper, we demonstrate the need to average the output of the generative model with the deterministic method’s output in Fig. 3, 4. 

Following the review, we have added to the appendix the results for Libr2iMix and Libri5Mix obtained without mixing the generative model output. These are provided twice:

Without aligning the phase to the source signal. 
Linearly aligning of the phase.

In both cases, the performance of the reconstruction with the generative part alone (of course, conditioned on the deterministic part) is worse than the reconstruction using the combination of both the deterministic and generative samples.

| Method            | Libri2Mix | Libri5Mix |
|-------------------|-----------|-----------|
|$\bar v_d$ (baseline)|             |              |
| (a) $\bar v_g$          | 19.8      | 10.2      |
| (b) $\bar v_g$ phase aligned | 20.0      | 11.3      |
| Our full method        | 21.9      | 14.2      |


Evidently, the generative method’s output is not as effective as the deterministic one (reinforcing figures 3 and 4) but improves the performance when combined with it. We note that the theoretical analysis assumes this combination. $\bar v_g$ by itself has the same bound as $\bar v_d$. Only their sum is improved.

Reviewer comment (Theory):
I'm uncertain on how the two inequalities in (11) are derived. For the first inequality, how is $I(v_r; v_{gr})$ related to $I(v_r; v_{dr},v_{gr})$ in (9)? The necessary condition for the second inequality is $I(v_r; v_{gr}, v_{dr}) \leq I(v_r; m_r)$, but this is not implied from (7) or (9). It would be helpful if the authors could clarify the steps. Also, it'll be helpful to explain how the "3.0" db is obtained in (21).

Author response:
To answer the reviewer’s questions:

1. $I(v_r;v_{gr})$ is the mutual information between the source at the r-th chunk and the generative sample in the same chunk. $I(v_r;v_{dr},v_{gr})$ is the mutual information between the source and both the deterministic estimation and the generative sample. The first term is upper bounded by the second part by definition since additional signals can add information and not remove information. 

2. In Eq. 9 we show that the mutual information between the generative and source is upper bound by the sum of the mutual information of the source and mixture (first term) and the mutual information between the (1) source, and (2) the tuple of (generative sample, deterministic estimation). Since the generative sample is from approximately the same distribution as the deterministic distribution, the same data-processing theorem bound holds for it. Using Eq. 7 we can upper bound again the second term in Eq. 9  to obtain the second inequality.
The 3dB is due to the factor of 2 over the previous bound which translates to a 3dB addition in the logarithmic scale.

Reviewer comment (Presentation):
Any reason for transposing the horizontal and vertical axes for the visualizations in Figure 5? It is conventional to display the spectral information in the vertical axis and temporal dimension horizontally.

Author response:
In the final version, we will recreate Fig. 5 of the appendix, replacing the two axes as requested.