Paper: Which pre-trained model is effective for speech separation ? (ICLR-2024)

Reviewer comment (Novelty):
While the authors claim to explore the influence of pre-trained models on the separation task, the paper appears more akin to a two-stage training paradigm leveraging contrastive learning. Conventionally, pre-trained models refer to general-purpose speech models, such as Wav2vec and WavLM.

Author response:
The main argument of the paper is that these general-purpose speech models have failed to generate features that improve speech separation based on [1]. We are investigating whether a standalone pre-trained model tailored for speech separation will result in better features. We investigate these with four variations of pre-trained models trained with the following varying input features.
•	DFT features +augmentations
•	DFT features without augmentations
•	Raw waveform features +Augmentations
•	Raw waveform features without augmentations

Reviewer comment (Experiments):
The two core contributions of this paper are the pre-trained model using contrastive learning and the deep modularization network. Firstly, I believe the authors should demonstrate the significance of the pre-trained model for the separation task. The authors displayed in Table 1 the impact of different pre-trained inputs on performance when using a deep modularization network. This perspective might be narrow, as it may be specifically tailored for this network structure. Comparative evaluations with other separation networks like Conv-TasNet and DPRNN are warranted. Secondly, the authors should contrast the use of existing pre-trained models combined with the deep modularization network to underscore the network's significance.

Author response:
We invite the reviewer to look at the table 4 in the appendix, where we compare the results with the existing technique. Since deep modularization network clusters data based on  what we refer as frame i.e., spectrogram datapoints, it is difficult  to fit this input to existing pre-trained model since their input were not structured this way. However, we acknowledge this is a valid concern. But we rely on justification from [1] that their effectiveness in  speech separation is limited.

Reviewer comment (Writing):
The paper contains some typographical errors and figure reference inaccuracies. For instance, all equations should be followed by punctuation, "equation 3" in Figure 1 should be "equation 2", and in Section 4, "$A_{ij}$ iff" should be corrected to "if". I hope the authors can meticulously address these mistakes.

Author response:
This well noted. We have corrected based on this observation.

Reviewer comment (Experiments):
Given that the authors utilized speech signals with noise and reverberation as inputs for the pre-trained model, can their method separate mixed signals that contain noise or reverberation? For instance, have experiments been conducted on datasets like WHAM! or WHAMR!?

Author response:
We use the common speech separation dataset wsj0-2mix, wsj0-3mix ,wsj0-4mix, ws0-5mix,  Libri5Mix and Libri10Mix which has both training, validation, and testing. The training and validation have overlap but testing has no overlap with training and validation I.e., the training and validation sets share common speakers, which is not the case for test set. This explained in pg 8. Furthermore, the models are trained using WSJO dataset but are able to perform well in Libri5Mix and Libri10Mix( see table 4 in the appendix )  without any data adaptation technique. The dataset have been used by our baseline tools so  allows for direct comparison. We did not evaluate the tool in the dataset that have reverberation. However, the performance of the proposed tool in the mixtures which have  reverberation and noise can be explored further.

Reviewer comment (Experiments):
I perceive the clustering parameter 'k' as a pivotal hyperparameter. Is it possible to evaluate the model's performance under various 'k' values?

Author response:
Since our dataset contained mixtures having 10 speakers  we evaluated with k=10,k=15 and k=20 and k=20 provided optimum results so k was set experimentally.

Reviewer comment (Novelty):
Considering that, for example, TF-GridNet can achieve an SI-SNRi=23.4 on WSJ0-2mix, the authors' method of pre-training followed by separation is inherently more intricate. How do the authors rationalize this complexity?

Author response:
Our work was not entirely to propose a new speech separation tool, however our key goal was to 
1.	Establish whether  there is need for a standalone pre-trained model for speech separation where the training dataset is contaminated by noise and reverberation, or the existing general audio pre-trained model suffice.

2.	Establish the  best type of input ( raw waveform or Fourier based features) are effective for training the pre-trained models tailored for speech separation. The proposed deep modularization is used to avoid the problem of permutation ambiguity during speech separation

3.	Most tools use supervised technique of speech separation which suffer from the problem of permutation ambiguity. They solve this by using permutation invariant training (PIT).  PIT has a computation complexity of $O(S!)$ with $S$ being number of speakers. The proposed method has a complexity of  (O)(d2n)( see pg 6).

Reviewer comment (Novelty):
Missing Key References: The paper overlooks crucial literature related to the use of noise and reverberation in pre-training, particularly the WavLM-based pre-trained model as discussed in Chen et al. ("Wavlm: Large-scale self-supervised pre-training for full stack speech processing." IEEE Journal of Selected Topics in Signal Processing 16.6 (2022): 1505-1518.). Additionally, relevant studies on speech separation using WavLM, such as Chen et al.'s work ("Speech separation with large-scale self-supervised learning." ICASSP 2023), are not cited. These omissions are significant as these works also concluded the beneficial impact of noise and reverberation.

Author response:
While the papers quoted definitely are related to our work since  WavLM was trained with contaminated speech. However,  our work primary focus is to evaluate  if such training( where input is contaminated) makes it generate better features for speech separation as  compared to pre-trained model such as [1] which are trained using clean speech. We investigate this by training four pre-trained models using constrative learning. The four variations of pre-trained models trained with the following varying input features.
•	DFT features +augmentations
•	DFT features without augmentations
•	Raw waveform features +Augmentations
•	Raw waveform features without augmentations

Work in IEEE Journal of Selected Topics in Signal Processing 16.6 (2022): 1505-1518.) investigates the various techniques to efficiently integrate the WavLM model in speech separation when faced with limited computation budget, including a low frame rate SSL model training setup and a fine-tuning scheme using only the part of the pre-trained model. It does not look at the quality of features generated based on how the model was trained .

Reviewer comment (Experiments):
Lack of Baseline System: The paper is not able to provide baseline performance for the comparable downstream speech separation model. This absence hinders readers' ability to effectively assess and compare the proposed model's effectiveness.

Author response:
We think you missed the appendix section where we compare the results with other baseline tools. We invite you to kindly  look at  table 4 in the appendix, where we discuss baseline systems.

Reviewer comment (Evaluation):
Incorporating an analysis of run-time factors would significantly enrich the paper, considering the typically high computational costs associated with pre-trained models.

Author response:
Most tools use supervised technique of speech separation which suffer from the problem of permutation ambiguity. They solve this by using permutation invariant training (PIT). PIT has a computation complexity of $O(S!)$ with  $S$ being number of speakers. The proposed method has a complexity of (O)(d^2n)( see pg 6) which is much faster.

Reviewer comment (Experiments):
The rationale behind selecting CONDEEPMOD as the pre-trained model remains ambiguous. The paper would benefit from a more detailed justification of this choice.

Author response:
In introduction section pg 2 we justify why we use deep modularization

-to avoid permutation ambiguity problem.

-to avoid challenges of data annotation of supervised learning.

However the key goal of the paper was to evaluate 
1. Whether contamination of input data makes pre-trained model more suitable for speech separation as compared to that trained without contamination
2. Which input features are the best for training the models i.e raw waveform or DFT transform features ?

Reviewer comment (Writing):
In terms of clarity, I find the paper difficult to understand. One major point of confusion is the use of the term "frame" to, I believe, refer to individual spectrogram points. A frame of a spectrogram is an entire column, so it is not clear whether it is spectrogram points or spectrogram columns that are being clustered. Based on the results, I assume it is points, because clustering columns would not produce sufficient speech enhancement, but, for example, section 6.1 states "we divide a given speech in the training set into frames (chunks) of size 250 with 125 overlap between two subsequent frames." No units are provided for these numbers, but they appear to be columns of spectrograms. This critical detail is very unclear.

Author response:
Note that we pre-train the models( in our case four models) using raw waveform and DFT transformed raw waveform therefore we adopt a common name frame to refer to datapoints. For raw waveform input, given a time-domain speech signal x ∈ R^T , the signal is processed by an encoder to generate representation h ∈ R^{F ×T} . This is then chunked
into frames along the time axis to generate L ∈ R^{F ×S×N} . Here N represents the number of frames
generated.The representation h is  chunked  into frames of size 250 with 125 overlaps between two subsequent frames.. For DFT transformed waveform, from the resulting frequency-domain representations we extract frames of spectrogram (T-F bins). These frames of spectrograms serve as our datapoints.

Reviewer comment (Experiments):
The title of the paper suggests that existing pre-trained models (e.g., wav2vec 2.0) will be compared for speech separation. The abstract further argues that existing pre-trained models are trained on clean speech and so have trouble representing mixed speech. This line of reasoning is quickly abandoned, however, and a new model trained on a relatively small corpus is introduced instead of a large self-supervised model.

Author response:
Our argument is that  the effectiveness of  general audio pre-trained model such as wave2vec have been investigated in [1] where they conclude they do not generate quality features for speech separation.  Their experiments establish that the 13 pre-trained models used do not significantly improve feature representations as compared to those of baselines in speech separation.  We therefore investigate if:
1.	Adding noise will make pre-trained model develop robust features for speech separation.
2.	 Adding noise+reverberation will make pre-trained model develop robust features for speech separation.
3.	Does input (DFT based vs raw waveform) features used to train pre-trained model significantly  affect the quality of representation generated by the pre-trained model. Where we have four variants of pre-trained models
•	DFT features +augmentations
•	DFT features without augmentations
•	Raw waveform features +Augmenations
•	Raw waveform features without augmentations
With regard to data size, based on our technique of using  spectrogram frames, a 30 hr speech signals generates over 15 millions frames (datapoints)

Reviewer comment (Writing):
Several different loss functions are introduced, and it is not clear when each one is used.

Author response:
We use  only two loss functions one for training pre-trained model(equation2) and one for clustering data points(equation12). Confusion is maybe generated  based on equation 11 which we expand to generate 12.

Reviewer comment (Presentation):
The results tables from the appendix should be included in the main text because they include comparisons to baseline systems. The current tables need not be included and could be replaced by these.

Author response:
We tried this, but the strict page limit does not allow it to fit even after removing section 6.2.

Reviewer comment (Presentation):
The paper contains no paragraph breaks, making it unfriendly to read. The paper would be improved by removing some of the less relevant literature review to make space for proper spacing. Additionally, citations are all \cite{}, which are rendered as "Isik et al. (2016)" even when the entire citation should be in the parentheses. This effect can be achieved by instead using \citep{}, which will be rendered as "(Isik et al., 2016)".

Author response:
We have corrected citation. Due to strict page limitation, we resorted to exploit as much space as possible.

Reviewer comment (Writing):
What do you mean by "frame"? And if it is indeed a traditional frame (i.e., spectrogram column), then is separation performing within frames or each frame is assigned to a single source entirely?

Author response:
Note that we pre-train the models( in our case four models) using raw waveform and DFT transformed raw waveform therefore we adopt a common name frame to refer to datapoints. For raw waveform input, given a time-domain speech signal x ∈ R^T , the signal is processed by an encoder to generate representation h ∈ R^{F ×T} . This is then chunked
into frames along the time axis to generate L ∈ R^{F ×S×N} . Here N represents the number of frames
generated.The representation h is  chunked  into frames of size 250 with 125 overlaps between two subsequent frames.. For DFT transformed waveform, from the resulting frequency-domain representations we extract frames of spectrogram (T-F bins). These frames of spectrograms serve as our datapoints.