Paper: CRAFT: Cross-Representation modeling on Audio waveForms and specTrograms (ICLR-2024)

Reviewer comment (Writing):
The introduction is poorly written, with too many terms introduced (e.g., PSWaCL, SWaB, MSAE) without details of what to expect in the paper or what the real contributions are.

Author response:
Thank you for the valuable suggestion. We have carefully revised our introduction section to give a brief explanation about our proposed MSAE, SWaPT, PSWaCL and SWaB.

Reviewer comment (Writing):
Challenges in feature fusion are not clear, and many statements are loose or vague.

Author response:
Regarding the challenges of feature fusion, we aim to differentiate the focus of spectrogram and waveform representations. Spectrograms and waveforms are more focused on time-frequency and time-amplitude features, respectively. While acknowledging that we can absolutely apply a learned filterbank to the waveforms, our primary goal is not to answer the questions, such as whether waveform can replace spectrogram, or how to determine the best representation from spectrogram or waveform. Instead, our unique contribution is a novel contrast learning based method to combine complementary information from spectrogram and waveform.

Reviewer comment (Writing):
The claim that waveform-based features concentrate more on capturing common patterns is unclear—what are these common patterns?

Author response:
Regarding the challenges of feature fusion, we aim to differentiate the focus of spectrogram and waveform representations. Spectrograms and waveforms are more focused on time-frequency and time-amplitude features, respectively. While acknowledging that we can absolutely apply a learned filterbank to the waveforms, our primary goal is not to answer the questions, such as whether waveform can replace spectrogram, or how to determine the best representation from spectrogram or waveform. Instead, our unique contribution is a novel contrast learning based method to combine complementary information from spectrogram and waveform.

Reviewer comment (Theory):
The claim that spectrograms predominantly emphasize time-frequency responses is unclear—what are these responses? Are we assuming a system here? Why can't waveform learning using a learned filterbank do the same?

Author response:
Regarding the challenges of feature fusion, we aim to differentiate the focus of spectrogram and waveform representations. Spectrograms and waveforms are more focused on time-frequency and time-amplitude features, respectively. While acknowledging that we can absolutely apply a learned filterbank to the waveforms, our primary goal is not to answer the questions, such as whether waveform can replace spectrogram, or how to determine the best representation from spectrogram or waveform. Instead, our unique contribution is a novel contrast learning based method to combine complementary information from spectrogram and waveform.

Reviewer comment (Writing):
The suggestion to enhance the comprehensiveness of the feature set is vague—comprehensive in what sense?

Author response:
It has been established in various audio papers ([5-9]) that, waveforms and spectrograms convey complementary information. By combining waveform and spectrogram, the model classifies the audio input according to a combined feature set, instead of spectrogram only or waveform only.

Reviewer comment (Writing):
None of the above-mentioned statements are actually related to Semantic Misalignment; "Semantic" has a different meaning.

Author response:
By the discussion of semantic and temporal misalignment, we aim to differentiate the design of CRAFT during pretraining and finetuning, respectively. During pretraining, we designed the model to contrast spectrogram-waveform representations. To do so, the transformer backbone first generates global representations on spectrogram and waveform, on which the contrastive loss is conducted. Only final outputs from different representations are contrasted, which helps alleviate the global semantics misalignment. During finetuning, we intentionally designed the model to exchange cross-representation information at each transformer layer. Due to the different temporal granularities, fusion bottlenecks were utilized to restrict most of attention flow within each modality. In contrast to pretraining, the active information exchange at each layer helps mitigate the temporal misalignment. 

While acknowledging that we can always do late fusion on spectrogram and waveform, as demonstrated by SWaPT-SWAST in Table. 1, the extension of simply concatenating waveform and spectrogram by late fusion in both pretraining and finetuning, will actually hurt the performances, from 29.0 mAP to 27.7 mAP. In addition, in ablation Table. 5, the optimal performance of CRAFT is not achieved by late fusion, neither. Therefore, to enhance the cross-representation modeling, we tailored our design to contrast multi-representations in pretraining and restrict cross-representation calculations in finetuning.

Reviewer comment (Theory):
The claim about Temporal Misalignment is wrong; both linear and non-linear processing methods are available, and temporal alignment can be easily achieved using basic DSP. Moreover, late fusion after learning complementary features is always possible.

Author response:
By the discussion of semantic and temporal misalignment, we aim to differentiate the design of CRAFT during pretraining and finetuning, respectively. During pretraining, we designed the model to contrast spectrogram-waveform representations. To do so, the transformer backbone first generates global representations on spectrogram and waveform, on which the contrastive loss is conducted. Only final outputs from different representations are contrasted, which helps alleviate the global semantics misalignment. During finetuning, we intentionally designed the model to exchange cross-representation information at each transformer layer. Due to the different temporal granularities, fusion bottlenecks were utilized to restrict most of attention flow within each modality. In contrast to pretraining, the active information exchange at each layer helps mitigate the temporal misalignment. 

While acknowledging that we can always do late fusion on spectrogram and waveform, as demonstrated by SWaPT-SWAST in Table. 1, the extension of simply concatenating waveform and spectrogram by late fusion in both pretraining and finetuning, will actually hurt the performances, from 29.0 mAP to 27.7 mAP. In addition, in ablation Table. 5, the optimal performance of CRAFT is not achieved by late fusion, neither. Therefore, to enhance the cross-representation modeling, we tailored our design to contrast multi-representations in pretraining and restrict cross-representation calculations in finetuning.

Reviewer comment (Novelty):
The proposed method (Sec 3) is essentially a combination of well-known existing methods and a small extension of the method by Gong et al., 2022a. Novelty is limited and not well highlighted in the context of the problem addressed.

Author response:
While acknowledging that many techniques used in CRAFT were proposed before, we brought unique insights into the cross-representation audio classification. That is, by treating spectrograms and waveforms as data augmentations to each other in contrastive learning, existing single representation learning can be further enhanced. We believe this is a non-trivial finding and have rephrased our main novelty in the paper.

Reviewer comment (Writing):
The paper states "Our work is built upon SSAST" but does not explain what SSAST is.

Author response:
We added a brief introduction of SSAST in Appendix. A.2.

Reviewer comment (Novelty):
Contrastive learning is widely used in multimodal generative models, so the method is not novel in itself. The idea of "natural or unnatural pairing" is not defined.

Author response:
While acknowledging that contrastive learning (CL) is a widely used approach, we argue that applying CL on multi-representations is an non-trivial finding. Typical contrastive learning approaches require tailored data augmentations. For example, in SimCLR paper, authors showed that "composition of data augmentations plays a critical role in defining effective predictive tasks". Our main novelty is that **without tailored designs, spectrograms and waveforms can naturally serve as data augmentations to each other in contrastive learning**. We believe we are the first to do so, and natural spectrogram-waveform is already an additional contrastive pair, which eliminates the necessity of hand-crafted techniques as discussed in SimCLR, such as Cutout, Sobel filtering, etc.

Reviewer comment (Novelty):
MSAE is a known technique for designing adaptively learned filter banks. It is unclear what is novel here, and existing works should be referred to.

Author response:
Thank you for the valuable suggestion. We do not claim MSAE as our main contribution, and just borrow the idea for waveform embedding in the era of transformer. We have added the existing works in Section. 3.1.

Reviewer comment (Presentation):
The "Patichyfy" operation is not explained. A diagram would help readers. Pooling will reduce information for short kernel-based convolution outputs with bigger dimensions; instead, zero padding, dilation, adaptive strides, or deformed convolution could be used to learn multiscale features, as pooling is established to be a poor choice in current practice.

Author response:
To keep our main contribution more clear and reduce the difference between SSAST and CRAFT, our Patchfy function on both spectrograms and waveforms are the same as SSAST as a simple convolution operation. We have revised the explanations accordingly in Section. 3. 

We would like to thank you for all the possible solutions. As reviewer aBV8 has mentioned, all these solutions might have superior performances over simple pooling. We would like to explain that we do not intentionally design the downsampling techniques, and just follow the same pooling operations as in [1]. On the other hand, we only claim the waveform embedding as a minor contribution, and will leave more advanced waveform embeddings as future research.

Reviewer comment (Reproducibility):
There is no description of how spectrogram and waveform feature inputs are processed in the transformer frontend—is it a single transformer with shared weights or individual ones? These crucial details are superficially treated.

Author response:
The function "Specify" denotes the conversion from waveforms to spectrograms, and has been further explained in the revised Section. 3.1. 
There are multiple reasons why we intentionally use the same transformer on both spectrograms and spectrograms. Firstly, by keeping other factors the same, this makes our contributions more distinct. Secondly, according to our responses to reviewer Cffs, the performance difference between sharing the same backbone (i.e. CRAFT) and having 2 individual backbones (i.e. SWa2-AST), is quite small. At last, sharing the same backbone also reduces the memory consumption.

Reviewer comment (Theory):
The claim that spectrogram and waveform patches can naturally serve as contrasting pairs is not substantiated. It is unclear how this happens, and no reference to existing work is given.

Author response:
There is no existing work to establish this. Instead, **we claim the contrast between spectrogram and waveform patches as our unique contribution**. As explained before, without tailor-designed data augmentations, both representations themselves already form a natural contrastive pair.

Reviewer comment (Reproducibility):
The meaning of t_spec and t_wav in equations (5a, 5b) is unclear—what are the dimensions, on which axis is concatenation happening, and from where do bottleneck features come? Why are they required? Could the features instead be projected to the same dimensional space? The design of a multi-stream bottleneck transformer is not explained.

Author response:
As discussed in Section. 3.4, $t_{spec}$ and $t_{wav}$ are input tokens of one transformer layer. And they have dimension of $[b, l, d]$, where $b$ is the batch size, $l$ is the number of $t_{spec}$ or $t_{wav}$ tokens, and $d$ is the hidden dimension. The concatenation is happening along the token dimension, i.e. $l$. We mainly followed the idea of bottleneck fusion from MBT paper [2], where a few extra bottleneck tokens are added into each modality on each layer. 
The necessity of fusion bottlenecks are:
1. While naively concatenating spectrogram and waveform tokens indeed improved the results, we found that it only brought very limited benefits. As in Table. 3, comparing SWaPT-AST and SWaPT-SWAST, we kept pretraining as the same but only changed the finetuning as joint training on spectrogram and waveform instead of training on spectrogram-only. As a result, SWaPT-SWAST has only 0.3 higher mAP compared with SWaPT-AST. 
2. Due to the largely diversified misalignment between spectrogram and waveform, they can be treated as multi-modal inputs, which naturally brings the usage of fusion bottlenecks as introduced in MBT to solve multi-modal fusion problem. 

Again to keep our main contribution more distinct from SSAST, we designed the multi-stream bottleneck transformer the same as the original SSAST. The only difference is that bottleneck transformer takes all spectrogram tokens, waveform tokens and bottleneck tokens as inputs.

Reviewer comment (Writing):
Authors use Mel-spectrograms, which are non-linear features, but in the introduction they argue about misalignment due to fixed resolution. This presentation is misleading for a wider audience.

Author response:
We understand the reviewer's concern about the data augmentation. The reason why augmentation is not applied on waveform representations is because **waveform itself is treated as data augmentation against spectrograms**. The reason we introduced waveform into SSAST is because we believe contrastive learning on same domain representations can be utilized to further enhance the performances. 
Therefore, treated as one kind of data augmentation, waveform might not need extra augmentation techniques. And we will leave it as a future work. However, as established in CRAFT, contrastive learning on multi-representations can already enhance the audio performances **even if waveform is still un-augmented**.

Reviewer comment (Experiments):
Experimental results are not state-of-the-art, and strong baselines are not considered. Existing works have achieved over 98% on the ESC-50 benchmark, and references are provided. Similar observations apply for the Audioset dataset.

Author response:
Thank you for pointing out several SOTA results. We agree that CRAFT is SOTA comparable on AudioSet and ESC-50. As explained above, we kept many details the same as our baseline, such as the spectrogram calculations and the multi-stream transformer backbone architecture in SSAST. However, we note that our proposed techniques are especially unique in the sense that we are the first to contrast different representations from the same data, and intentionally treat waveform as data augmentation against spectrogram. Our method can be possibly incorporated into many audio modeling approaches, such as Audio-MAE [3], CAV-MAE [4].

Reviewer comment (Novelty):
Novelty is limited considering the ICRL standards. It might be a better fit for speech-related conferences (e.g. Interspeech)

Author response:
While acknowledging that many techniques in our paper are present in the current deep learning community, we would like to highlight that proposed in CRAFT, the **contrastive learning between different representations from same data** is a non-trivial contribution. In order to learn complementary information from different representations, we selected other techniques and carefully designed each component in our framework. 

We would like to note that our contrastive learning on multi-representations can be possibly extended to other domains as well, and we mainly tested it on audio tasks as a showcase. Thus we believe CRAFT might be a good fit for audio representation learning in ICLR.

Reviewer comment (Writing):
The writing of this paper needs to be improved. Too many acronyms to make it less readable and hard to follow the idea, especially in section 3.2.

Author response:
Thank you for your suggestions on writings. We have revised the acronyms, and added a recap of SSAST in our revision for better presentation.

Reviewer comment (Evaluation):
In table 1, it shows that the performance of AST (spectrum-only) is still better than all the proposed methods in the paper. How to explain it?

Author response:
This is a good finding that AST on spectrum-only can beat our results. We would like to mention that AST is a supervised learning method with labels available, and our approach is based on self-supervised setting where labels are missing. Generally in audio classification, supervised learning approaches have a clear advantage over self-supervised learning. For example, AST and SSAST have 34.7% and 29.0% mAP on AudioSet-bal set, which means a gap of 5.7%. However, our proposed CRAFT achieved a mAP of 33.4%, **significantly reduced the gap from 5.7% mAP to 1.3% mAP**.

Reviewer comment (Novelty):
In introduction paragraph 1, the authors try to illustrate the difference between spectrogram and waveform representation by differentiating the tasks based on them. However, many of the audio/speech tasks can build on both spectrogram and waveform representation and both achieve good results. Actually, in areas like audio signal processing and ASR, both spectrogram [1,2] and waveform [3,4] are frequently used.

Author response:
While acknowledging that many audio / speech solutions are built upon both spectrogram and waveform, our goal is not to differentiate them or select the best of them. Instead, we would like to simultaneously combine them and draw complementary presentations from each other. In addition, while there are a bunch of spectrogram-waveform fusion / joint modeling works, we would like to note that CRAFT is the first work to contrast spectrogram and waveform and learn joint representations from them.

Reviewer comment (Theory):
In introduction paragraph 2, it is quite confusing why car engine sound is more clear in the time-frequency domain, while successive breezes is clear in waveform domain. Need clarification. Moreover, waveform representation can also present time-frequency patterns. Take an example of conv-TasNet [3] in audio signal processing domain, the waveform filters spontaneously converge to a frequency response similar to that of a log-mel filter bank.

Author response:
By elaborating car engine sound and successive breezes, we are explaining the different focus of spectrogram and waveform representations, which are more concentrated on time-frequency and time-amplitude features, respectively. It might be easier to locate the explosive car engine sound by detecting frequency changes, and successive breezes might be more accurately reflected on amplitude changes. 

We understand that the reviewer's concern is the redundancy of using spectrograms and waveforms at the same time. However, it has been demonstrated in many works that spectrograms and waveforms contain complementary information. By tailored design of joint-learning on both representations, the performances can be improved compared with single-representation learning, for example [1-3]. 

Thank you for the constructive comment of one possible waveform augmentation. The main contribution of this paper is to consider the encoded **waveform itself as one way of data augmentation against spectrogram**, and contrast them. The design of data augmentation in self-supervised learning is an active research question. We would like to thank reviewer Cffs for pointing out conv-TasNet, which is intriguing for us to design other audio contrastive pairs in the future.

Reviewer comment (Experiments):
To sufficiently prove the effectiveness of spectrogram-waveform representation combination, the authors should show the comparison between spectrogram-only, waveform-only, and joint spectrogram-waveform representations while **keeping other factors the same**. However, the waveform-only results come from very old research, where the SSL and audio transformer techniques are not well established. Since waveform includes more information than the spectrogram, maybe using WaPT will result in better performance than SSaPT and comparable performance of PSWaCL. If this is true, spectrogram can be completely substituted by waveform in audio classification task. Here, WaPT denotes "Waveform modeling in PreTraining" and the only difference from SWaPT is to remove the spectrogram input and the corresponding training loss.

Author response:
Thank you for the valuable suggestion, i.e. the ablations of modeling on spectrogram-only, waveform-only, and joint spectrogram-waveform representations, while keeping other factors the same. Accordingly, we have added these results to a new **Appendix A.3** in our revision. Note that as shown in Appendix A.3, modeling on spectrogram have clear advantage over modeling on waveform directly, i.e. **4.6% higher mAP** on AudioSet-bal set. After carefully grid-searched hyper-parameters, **WaPT still has a 9.0\% smaller mAP than CRAFT**.

Reviewer comment (Experiments):
The result of "SWaPT is worse than SSaPT" is confusing. Why adding additional representation degrades performance? If the reason is the conflict between spectrogram and waveform representation, why not use different parameters in waveform branch and spectrogram branch? If this simple way can solve the misalignment between spectrogram and waveform feature, the necessity of PSWaCL is quite doubtful.

Author response:
Thank you for the valuable suggestion. The question is if we use different parameters in waveform branch and spectrogram branch, i.e. they have 2 individual backbones, will SWaPT be better than SSAST? To answer it, we did a quick experiment on AudioSet-bal pretraining. Let's use SWa2-AST to denote 2 backbone joint-modeling on spectrogram and waveform. The results are given in the following table:


| SSAST    | SWa2-AST | SWaPT-AST |  CRAFT |
| -------- | -------- | -------- | -------- |
| 13.68    | 14.656     | 14.80     | 14.664 |


In this experiment, pretrained on a small dataset, both SWa2-AST and SWaPT-AST successfully perform better than SSAST. However, there is **no significant difference** between them. In particular, sharing same backbone even has higher mAP than having 2 individual backbones. In addition, **2 individual backbones doubles the model parameters** and needs much more memory consumption. Therefore, we chose to report our results of SWaPT-AST instead of SWa2-AST in the paper.