Paper: Continual Contrastive Spoken Language Understanding (ICLR-2024)

Reviewer comment (Evaluation):
COCONUT does not outperform other baselines in a more challenging dataset (SLURP), limiting its contribution.

Author response:
For the SLURP dataset, while COCONUT performs worse than S-KD for the less challenging setting with 3 tasks, for the 6-task setup COCONUT slightly outperforms S-KD. As discussed in the paper, this indicates that COCONUT is more appropriate for settings involving more tasks, where the effect of catastrophic forgetting is exacerbated. Nevertheless, we think it is interesting that COCONUT can synergize and be combined with the S-KD method. As COCONUT and S-KD operate on two different parts of the model, that is the encoders and ASR decoder respectively, it seems natural to exploit both of them such that forgetting is mitigated at both levels. We speculate that for SLURP, since the batch size is limited and the number of classes is higher than FSC, having more tasks and thus fewer classes for each task can be more beneficial for the contrastive objectives of COCONUT.

Reviewer comment (Reproducibility):
The architectural design of COCONUT is rather complicated and onerous.

Author response:
We are not sure what the reviewer refers to with the term “complicated”. COCONUT combines supervised contrastive learning and knowledge distillation concepts tailored for the CL setting at hand. Indeed, it does not require more compute power than a standard contrastive loss, which is considered an efficient method to learn multimodal representations (i.e., CLIP [1]), but it requires an attentive selection of how to select the anchors, positives, and negatives. Moreover, we do not agree with the term “onerous” since the additional compute time added by integrating COCONUT is limited and reasonable, as explained below in more detail, and the contrastive learning approach is not considered an onerous objective by default. We finally emphasize that COCONUT does not introduce any modifications to the model itself (audio and text encoders, feature projectors, ASR decoder are fixed throughout the continual learning process), and neither does it introduce new parameters like other architectural CL strategies for each new emerging task.

Reviewer comment (Evaluation):
What is the running time of COCONUT compared to other baselines? Does it cost significant more time to run COCONUT?

Author response:
The main difference between the ER baselines (ER, A-KD, S-KD) and COCONUT is that the former focus on the rehearsal data only, while COCONUT is applied to both the rehearsal data (NSPT loss) and the new data (MM loss), and so COCONUT requires an additional compute time due to the MM loss. Nevertheless, this additional time does not hinder its applicability as it is limited. Indeed, for the FSC-6 setting, the KD baselines require an additional 5/6 % of compute time with respect to the fine-tuning baseline, while COCONUT requires around 15%. For SLURP-3, the KD baselines require around 10/13% of additional compute time, whereas COCONUT requires around 35%. Undoubtedly COCONUT requires slightly more running time than the other KD baselines that are only applied to the rehearsal samples, but this overhead is minimal and consequently we believe this is not an issue for a practical scenario, considering also that COCONUT leads to much improved performance. Additionally, from a memory overhead point of view, COCONUT requires the storage of the rehearsal samples and a copy of the model from the previous task. These storage requirements are the same as the audio and text KDs. Instead, the S-KD approach, in addition to the aforementioned storage requirements, also necessitates the storage of the rehearsal text transcriptions generated with beam search from the previous task, thus increasing the requested memory overhead with respect to COCONUT.

Reviewer comment (Experiments):
For the design of NSPD loss, since the author highlights that the NSPD loss is conducted only on rehearsal samples, what if the loss is conducted on all samples?

Author response:
**Question 1**. **1)** First of all, we want to say that in CL, it is not univocal whether to use only the rehearsal samples or all the samples (rehearsal + new ones) in the distillation process. Some papers use only the rehearsal samples, others exploit both rehearsal and new ones. It seems that this choice depends on the task at hand. In our case, as we stated in the paper, we decided to use only the rehearsal samples because we believe it is not reasonable to use the teacher model for the new data since the teacher was not trained on these new data. To verify this choice, we ran two experiments on SLURP-6 and FSC-6 using all the data for the NSPT loss, and we share them below (R means rehearsal and it is our proposed version, R + N also includes the new (N) data). We can observe that the additional use of the new data leads to poorer performance, thus proving that using only the rehearsal samples is the best alternative. We plan to include this additional experiment in the supplementary material as we think it is compelling for the paper.  

|Method|Avg Acc|Last Acc|  
|-|-|-|  
|**FSC-6**|||  
|NSPT (R)|**77.09**|**74.01**|   
|NSPT (R+N)|76.30|72.34|  
|**SLURP-6**||  
|NSPT (R)|**70.17**|**63.66**|   
|NSPT (R+N)|69.74|62.54|

Reviewer comment (Theory):
For the design of NSPD loss, in equation 5, why does i belong to I instead of I_c? According to figure 2, it seems that the repulsion is applied on new class (i.e., I_c) samples.

Author response:
**Question 1**. **2)** I is correct in equation 5 as we treat as negatives both the samples coming from the new task (new classes) and the rehearsal samples in the current minibatch whose class differs from the current anchor. Thus, given a rehearsal sample from the current minibatch (the anchor), we push away the samples having a different class from these samples, either it comes from the rehearsal buffer or the new task. In Figure 2 we decided to avoid including rehearsal samples with different class labels to not complicate the figure. Following the reviewer’s advice, we will add a sentence in the figure’s caption explaining this.

Reviewer comment (Reproducibility):
For the design of multimodal loss, when the loss is applied, is it only in the first task (to initialize the projection) or in every task?

Author response:
**Question 2**. We thank the reviewer for highlighting parts of the paper that might be unclear. In the following, we answer point by point. **1)** The MM loss is applied to all the tasks of the CL learning process.

Reviewer comment (Theory):
The main motivation of this paper is to overcome catastrophic forgetting. However, the multimodal loss seems more like a trick for better results instead of being targeted at catastrophic forgetting.

Author response:
**Question 2**. We thank the reviewer for highlighting parts of the paper that might be unclear. In the following, we answer point by point. **2)** The MM loss is buttressed by two motivations: first, we need to train the projection layers from the very first task, otherwise the NSPT loss would use randomly initialized projections. Second, by aligning audio and text embeddings of the data belonging to the new task, we get more consistent embeddings in the shared audio-text space such that the NSPT loss can rely on better feature representations.

Reviewer comment (Novelty):
Is this paper the first work to apply multimodal loss on SLU tasks?

Author response:
**Question 2**. We thank the reviewer for highlighting parts of the paper that might be unclear. In the following, we answer point by point. **3)** We are not the first paper to apply multi-modal losses for SLU, but all the other papers using multi-modal approaches are studied under an offline i.i.d. setting. We cited some recent works that use MM contrastive objectives in an offline scenario, including a paper related to SLU (Yi Zhu et al., Cross-modal transfer learning via multi-grained alignment for end-to-end spoken language understanding, Interspeech 2022).

Reviewer comment (Writing):
The text encoder in the paper is quite confusing. Please clarify if this understanding is mistaken.

Author response:
**Question 3**. We call it “text” encoder as it produces a feature representation for the text data, in a similar way as the audio encoder does for the audio counterpart. In ASR it is quite common to exploit a simple embedding layer as a text encoder. In our preliminary results, we tried to use more complex text encoders such as BERT, but we did not observe valuable improvements in response to the large overhead required by employing a big pre-trained encoder like that. So the term “encoder” has to be interpreted as the module that produces the representations that will be fed to the ASR decoder and to the projection layers. We will clarify this point in the paper as we think it is important.

Reviewer comment (Experiments):
For the experiments, more results are needed at different ER ratios (2-4%, 5-10%) to show that the proposed method can produce consistent improvement in different settings.

Author response:
**Question 4**. **1)** Regarding the additional experiments with higher ER ratios such as 4%-10%, we note that when we increase the ER ratio we are implicitly moving towards the offline setting (the upper bound), and this is something we do not desire because catastrophic forgetting is highly reduced. As we can see from the bar plot in Figure 4, when we use an ER ratio of 2% (16 samples) the results we get start approaching the offline upper bound. With this in mind, we believe that ER is a viable CL method only if a limited ratio is used, preserving the conditions a CL setting abides by. Nevertheless, we ran some additional experiments (see table below) where we used 4% of rehearsal data (30 samples) for FSC-6 setting. As we can observe, COCONUT and COCONUT + S-KD still outperform the ER baseline when 4% of memory is used, confirming the trend for smaller ER ratios. At the same time, the Avg and Last accuracies are saturating towards the upper bound. This is why we chose to use 1% as the ER ratio since we think it is a reasonable value to test our proposed approach.  

| **Samples per class** | **2** | **4** | **8** | **16** | **30** |  
| ------ | ------ | ------ | ------ | ------ | ------ |   
|ER | 46.27 | 57.21 | 69.76 | 82.37 | 89.60 |  
| COCONUT |  51.76 | 66.48 | 76.45 | 85.72 | 93.41 |  
| COCONUT + S-KD | 52.37 | 70.32 | 77.57  | 87.91 | 93.93 |

Reviewer comment (Experiments):
In the experiments, why are the results on FSC-6 in Table 1 (76.46) and Table 2 (77.09) different?

Author response:
**Question 4**. **2)** We thank the reviewer for pointing out this typo. The correct value is 77.09. We forgot to update the Avg Acc value for COCONUT for FSC-6 in Table 1.

Reviewer comment (Experiments):
more details on continual learning setting used would have been welcome (ref to (Capellazzo & al 2023) is not very self-explanatory)

Author response:
Regarding some missing details for the CL setting, we included the fundamental information for its definition in the first paragraph of section 5.1. We tried to include the most important details, but some details about the SLURP setting have been omitted for lack of space and because its definition is quite cumbersome. However, we agree with the reviewer that it is important and useful to include these additional details about the definition of the CL setting for the SLURP dataset in the supplementary material so that the reader can dive into more details. In the following, we elaborate a bit more on the CL setting defined for SLURP. As the SLURP dataset provides multiple levels of annotations (scenario, action, entity[es]), in principle one could decide to divide the dataset into tasks following one of these criteria. However, in the original paper where a CIL setting for SLURP is defined [1], the authors propose to use the scenarios as splitting criterion because they represent more general concepts than the actions and entities, and then the accuracy is computed on the intent, defined as the pair (scenario,action). As you may understand from this brief explanation, the definition of such a setting is pretty involved, and we thought it was not necessary to weigh down the flow of the experimental section. In addition to this, we mention that, following [1], we define the order of the classes in the various tasks depending on their cardinality, meaning that the classes with more samples are seen first by the model. This is done because the cardinality of SLURP scenarios varies consistently from class to class. We finally underline that we tried to be as consistent with the original implementation of [1] as possible.

Reviewer comment (Experiments):
experience replay (ER) baseline with buffer capacity of 2% is still better than COCONUT and it is unclear how using twice memory (2% instead of 1%) is a real bottleneck in real applications (authors could have commented this more)

Author response:
ER with 2% performs better than COCONUT with 1%, but in principle, it is not fair to compare two approaches that use a different amount of rehearsal data. Indeed, from the right figure of Table 2 we can see that if COCONUT also uses 2% of rehearsal data, it outperforms the ER baseline by a large margin. By including the ER baseline with 2% in Table 1 we wanted to stress the fact that COCONUT + S-KD for some settings (i.e., FSC-3, SLURP-3) manages to approach the ER baseline that uses twice as many rehearsal samples.

Reviewer comment (Experiments):
how COCONUT could be adapted to more speech tasks ?

Author response:
We are not sure whether the reviewer with the term “speech tasks” refers to other speech “problems” or more CL tasks. Unfortunately, in CL the term task is a bit ambiguous in that it can refer to both the two mentioned concepts. Regarding the first possibility, COCONUT has been designed to work in a multimodal setting, in our case audio-text. Nevertheless, the NSPT loss can be used in an unimodal setting by including only the audio or text component. Since the NSPT loss, and also the MM loss, only requires as input the output of an audio/text encoder, it can be used in any audio or speech classification task, such as keyword spotting or speaker identification. In our paper, we found that the audio component has a larger impact on the CL process than the text counterpart, while for other tasks this might not be the case. Additionally, COCONUT requires only two parameters to be tuned: the hidden dimension of the projected layer and the temperature parameter. However, these parameters are standard tunable parameters for any contrastive learning loss and we found that standard values worked well also in our CL setting. We also emphasize that we believe COCONUT is a promising approach for other multimodal settings such as audio-vision and language-vision tasks, as well as other audio-language tasks like audio-text retrieval. Finally, if the mentioned tasks are classification ones, our joint ASR-SLU problem is more challenging as the ASR decoder generates the final transcription in a seq2seq way, thus making our setting more specific. For the second interpretation, we have seen from the SLURP results that COCONUT performs better when the CL setting includes more tasks, so we believe COCONUT can scale to more tasks efficiently.

Reviewer comment (Experiments):
experience replay (ER) baseline with buffer capacity of 2% is still better than COCONUT and it is unclear how using twice memory (2% instead of 1%) is a real bottleneck in real applications (authors could have commented this more)

Author response:
ER with 2% performs better than COCONUT with 1%, but in principle, it is not fair to compare two approaches that use a different amount of rehearsal data. Indeed, from the right figure of Table 2 we can see that if COCONUT also uses 2% of rehearsal data, it outperforms the ER baseline by a large margin. By including the ER baseline with 2% in Table 1 we wanted to stress the fact that COCONUT + S-KD for some settings (i.e., FSC-3, SLURP-3) manages to approach the ER baseline that uses twice as many rehearsal samples.