Paper: RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (ICLR-2024)

Reviewer comment (Novelty):
As the authors mention, the dual-path RNN idea is not new to this problem. I understand that the authors chose SRU for their complexity-related considerations, but I also wonder if the audio processing module could benefit from its own self-attention mechanism, such as in the SepFormer model.

Author response:
We experimented with different sequence processing architectures in Appendix D, which has now been updated to include several transformer designs. The self-attention mechanism leads to a decrease of 1 dB SI-SNRi, while significantly increasing the memory requirements of the model. Other transformer approaches resulted in extremely high parameter counts, which contradicts our aim of constructing a lightweight AVSS model.

Reviewer comment (Novelty):
The spectral source separation module might be the weakest contribution, because complex masks have been extensively studied in the audio-only source separation literature.

Author response:
Some research has been conducted in the AOSS field using complex masks, but such studies are relatively scarce in the context of audio-visual speech separation. Additionally, to our knowledge, current state-of-the-art audio-only source separation methods, such as Sepformer, still employ real-valued masks. Indeed, TF-GridNet, the current AOSS SOTA method, does not use masks at all and directly computes the separated sources. We provided a study on all these different methods in Table 4. It is important to note that results in AOSS do not necessarily translate to the AVSS field.

Reviewer comment (Reproducibility):
I wish the paper provides more details on the TDANet block for video processing, which is relegated to the reference in the current version.

Author response:
We have included an explanation of the video preprocessing block (VP BLOCK) in Appendix A.

Reviewer comment (Experiments):
The authors chose to "add" f_1 and f_2 (eq 11) after the CAF processing. I think it's a little abrupt in the sense that there might be other choices that preserve the unique information that each vector learns, such as concatenation. Have the authors considered other ways to combine the two vectors?

Author response:
In Table 2 of our paper we compare the CAF block with pure concatenation (CTCNet Fusion (adapted)) of the audio and visual features. We observe that despite our CAF Block using only 3.6% ofthe parameters and 1.3% of the MACs, it outperformed pure concatenation by a large margin.

The purpose for the CAF Block is to produce a low-parameter and MAC alternative to CTCNet’s fusion approach. If we directly concatenated $\mathbf{f}_1$ and $\mathbf{f}_2$, and then downscaled to the correct channel dimension using a convolution, this would be the same parameter and MAC count as the pure concatenation discussed above and in Table 2 of our paper. However, in order to fully answer your question, we ran an experiment where we concatenated $\mathbf{f}_1$ and $\mathbf{f}_2$ and then downscaled using a group convolution. Using a group convolution does not avoid the memory increase that comes from the concatenation, but it does avoid increasing the parameter and MAC counts significantly. The results show that this is an ineffective operation, and that a simple summation should be preferred.

|  Method       |   Params  (K)  |  MACs  (M)  |  LRS2-2Mix SI-SNRi      |  LRS2-2Mix SDRi       |
| --------------- | ------------- | --------------- | ------------- | ------------- |
|  Add  (ours)  |  14.1       |  14.3  |  739  |  21,896  |
|  Concat       |  13.7       |  13.9  |  741  |  21,913  |

Reviewer comment (Presentation):
The baseline methods listed in table 1 should include their references.

Author response:
A1: Thank you for spotting that, we have updated the table.

Reviewer comment (Evaluation):
RTFS-Net-12 is only about 10% more efficient that CTCNet. How much difference does that make in practical applications?

Author response:
A2: From the 10% figure we assume that the efficiency you mention is RTFS-Net's inference time of 109.9ms vs CTCNet's inference time of 122.7ms. The inference time is the time taken to process 2 seconds of audio. Scaling up to 1000 hours of audio, it would take 54.95 hours for RTFS-Net-12 and 61.35 hours for CTCNet. Since the separation performance of RTFS-Net is higher, and the time taken is shorter, the logical solution would be to use RTFS-Net as it would save around 6 and a half hours of GPU processing, and the associated electricity cost. Further, the performance drop from switching to RTFS-Net-4 is pretty minimal. Using this model, 1000 hours of audio would only take 28.9 hours.

The total parameters and MACs are also important factors to be considered in practical applications. Compared to CTCNet, RTFS-Net-12 has reduced computational costs by three times, while the number of parameters utilized is only one-tenth.

Reviewer comment (Evaluation):
Some of the comparison examples are not distinguishable to this reviewer. This makes me wonder how to interpret the relative SNR gains.

Author response:
A3: You are correct. The purpose of the demo, and indeed this paper, is to show that SOTA performance can be replicated using a fraction of the parameters and a greatly reduced model complexity. For example, in Appendix C we showed that if we do not compress the frequency and time resolutions of the audio features (i.e. q=1), we can outperform the SOTA method by a significant margin using only R=4 repetitions. Using 16 or more layers with q=1 would greatly increase the SNR further. However, the large memory requirement and computational complexity associated with this approach defeats the purpose of our paper.

Reviewer comment (Experiments):
Since the best performance is achieved with R=12, why not explore a higher R?

Author response:
A4: A higher value of R leads to increased computational demands, which contradicts our intention to propose a lightweight model. Nonetheless, we still present a result as shown below. The results indicate that a higher R achieves better separation quality, but it comes with a more significant computational cost. Note that because we share parameters, increasing R only increases the MACs and not the parameter count.

RTFS-Net with 16 and 20 layers on LRS2 dataset

|  Model        |  SI-SNRi  |  SDRi  |  Params (K)  |  MACs (G)  |
| --------------- | ----------- | -------- | -------------- | ------------ |
|  RTFS-Net-12  |  14.9     |  15.1  |  739         |  56.4      |
|  RTFS-Net-16  |  15.2     |  15.5  |  739         |  73.7      |
|  RTFS-Net-20  |  15.4     |  15.6  |  739         |  90.9      |

RTFS-Net-20 outperforms CTCNet by over 1dB in both SDRi and SI-SNRi, while still utilizing only 91G MACs – just over half the MACs utilized by CTCNet. However, 20 RTFS Blocks requires a much larger GPU memory for training. As a result, we had to use NVIDIA 3090/4090s in order to obtain these results in a short period of time.

Reviewer comment (Evaluation):
Have the authors considered conducting studies with human listeners? If the target application is ASR, would it be helpful to measure WER in a recognition task?

Author response:
A5: We tested the Automatic Speech Recognition (ASR) results on the LRS2 test set. The results show that RTFS-Net generalizes well to downstream tasks, achieving almost identical performance to CTCNet – the previous SOTA method. For details, please refer to our response to the third question from Reviewer #1FXL.

We have shared a demo in the supplementary material that can be listened to at your leisure, and we have shared our results with other members of our lab. However, conducting a formal study with large quantities of human participants or professional stenographers lies beyond the scope of our current research and is not feasible within the limited timeframe given for responding to reviews and questions.

Reviewer comment (Experiments):
Testing is limited to only 2 speaker mixtures. Performance with more speakers is uncertain.

Author response:
**A1:**   We tested our model on two-speaker mixtures because most existing methods are verified in this manner. Furthermore, it is important to note that our model incorporates visual information from a single speaker to isolate and extract the audio of this target speaker. The integration of visual cues plays a crucial role in enhancing the accuracy of our model, as well as other AVSS models. Hence, we are not limited by the number of speakers.

We randomly sampled speakers from the LRS2-2Mix test set to obtain a dataset with three speakers. The results indicate that the presence of multiple speakers somewhat impacts the model's performance. However, it is important to note that these models were not trained on the dataset with three speakers, only tested on the dataset with three speakers. In addition, RTFS-Net-12 outperformed CTCNet by a significant margin, scaling much better to the harder task.

|  Model          |  SI-SNRi  |  SDRi  |
| ----------------- | ----------- | -------- |
|  AV-ConvTasNet  |  10.0     |  10.3  |
|  AVLIT-8        |  10.4     |  10.8  |
|  CTCNet         |  12.5     |  12.8  |
|  RTFS-Net-12    |  13.7     |  13.9  |

Reviewer comment (Evaluation):
Missing PESQ evaluation in results which most other target speech extraction papers provide.

Author response:
**A2:** We have added the PESQ evaluation metrics to Table 1 in the updated paper. As with SI-SNRi and SDRi, the PESQ scores show that RTFS-Net obtains comparable accuracy to the previous SOTA method (CTCNet) across all three datasets, while utilizing fewer parameters and MACs.

Reviewer comment (Experiments):
Doesn't include latest SOTA model comparison: Dual-Path Cross-Modal Attention for Better Audio-Visual Speech Extraction, ICASSP 2023. https://arxiv.org/pdf/2207.04213.pdf. This gives superior performance for SI-SNRi and provides PESQ results as well. It does not provide MACs analysis.

Author response:
**A3:** The model code for this ICASSP publication is not open source. Its results on the LRS3 dataset for the model “AV-TASNET” are different from ours, indicating that either their mixing method is different, or that their train/eval/test splits might be different. For similar reasons, we also did not include AV-Sepformer ([https://arxiv.org/abs/2306.14170](https://arxiv.org/abs/2306.14170)) in our paper. Both of these methods are Sepformer-like, meaning the parameter count for each model is at least 20 million. However, by comparing our paper with AV-Sepformer’s PESQ values, this method also does not outperform CTCNet or RTFS-Net (AV-Sepformer: 2.31, CTCNet: 3.00, RTFS-Net: 3.00).

We would also like to emphasize that the purpose of RTFS-Net is not to obtain SOTA performance. As can now be seen in the updated Appendix C, we can easily outperform CTCNet by directly applying TF-GridNet to the audio-visual setting with only 4 RTFS-Net block layers. The focus of RTFS-Net is to obtain comparable performance to the SOTA method using a very small model footprint. Our contribution is a fast, efficient and quick-to-train model.