Paper: UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling (ICLR-2024)

Reviewer comment (Experiments):
Although the authors propose using residual learning to preserve the integrity of language queries during the cross-attention process in multimodal encoders, it is not clear why only textual queries need to be preserved, not visual queries. Specifically, in Fig. 2(b), why if text info may be missing which needs a residual learning, why not visual info after up-projection linear layer not be added to the cross-modal output? Similarly for Fig. 2(a), why there is no extra UniAdapter for visual modal be needed in multimodal encoder? This needs to be explained carefully, with evidence/experiments.

Author response:
Thank you for your thoughtful question regarding our decision to focus on preserving textual information over visual information in our UniAdapter model.

* Our decision to focus on textual information preservation **instead of visual** is informed by the operational structure of prevalent multimodal cross-attention mechanisms, such as those in ALBEF, BLIP, and Perceiver architectures. In these models, the visual information (the final token sequence output $f_v$ from the visual encoder) is inherently integrated into the each layer of the cross-attention module. This is exemplified in the following equation from our paper:

   $$ h_i = q_i + MCA(Q = q_i, K = f_v, V = f_v) $$

    In this design, $f_v$ consistently serves as both the key and value in each layer of the cross-attention module, ensuring that visual information is intrinsically embedded at every layer.

* Due to this structural design, visual information is naturally preserved and maintained throughout the cross-attention layers. This effective integration prevents the loss or diminishment of visual information as it progresses through these layers. Consequently, this inherent preservation of visual information within the cross-attention mechanism eliminates the need for additional measures, such as extra up-projection linear layers or UniAdapters, specifically for the visual modality in the multimodal encoder.
* Our experimental trials also support this design choice. We tested the addition of extra mechanisms for visual information preservation and found that it did not lead to performance improvements. Instead, it introduced unnecessary parameter overhead. 

Based on these findings, we concluded that there was no necessity for additional mechanisms like extra UniAdapters for the visual modality or additional up-projection linear layers for the visual information in the cross-modal output of our UniAdapter model.

Reviewer comment (Theory):
Regarding equation (7), how to justify only using text token feature is good in this case? How about other features f^{t}_{CLS,i}?

Author response:
Thank you for your insightful question regarding our decision to use the text token feature $f^t_{CLS}$. Our choice is primarily based on the effectiveness of the contrastive learning process, in which the overall semantic representation of a sentence is efficiently encapsulated into the $f^t_{CLS}$ token. This makes $f^t_{CLS}$ an optimal representation for the entire sentence in our model.

Theoretically, incorporating a more granular calculation that includes each token and text token could potentially yield better results. However, such  approach would require the introduction of an additional module to handle the alignment of varying token lengths. To balance model efficiency and simplicity, we chose $f^t_{CLS}$ as our representative token.



|  | $f^t_{CLS}$ | mean($f^t_{CLS,i}$) | max($f^t_{CLS,i}$) | $f^t_{CLS}$ + mean($f^t_{CLS,i}$) |
| -------- | -------- | -------- |-------- |-------- |
| Didemo R@1     |  52.1    | 51.9     | 51.2 | 52.0 |


Additionally, we experimented with using the average or maximum of all $f^t_{CLS,i}$ tokens as an alternative. These variations, however, did not show any significant performance differences in our experiments, further validating our decision to use $f^t_{CLS}$ for maintaining both model efficiency and performance.

Reviewer comment (Experiments):
In Table 2, need ablation studies on top of UniAdapter, not Adapter. On Adapter is good and helpful, but it is needed to be put on top of the proposed solution to see its full benefit, i.e., +Query-residual, +PFA, +Weight-sharing all in Full UniAdapter. Would like to see the enhancement on the full version.

Author response:
Thank you for your valuable feedback on the necessity of providing more explicit ablation studies on the full UniAdapter model in Table 2. We realized that this progression was not as clear as it could be in our original presentation. To address this, we have revised Table 2 to make it more clearly.

Reviewer comment (Novelty):
Lack of novelty: the overall design and each of the three components of UniAdapter are not interesting. In particular, using shared weights in the lower layers followed by layers with specialized weights is common in multi-task learning literature. Weight-sharing has also been employed by previous parameter-efficient fine-tuning work like Compacter (Karimi Mahabadi et al., 2021). Using residual connections is again a commonly seen trick. More importantly, the performance improvement that resulted from combining these three techniques is not impressive at all compared with vanilla adapters, as shown in the middle rows of Table 2.

Author response:
**Q1: The overall design and each of the three components of UniAdapter are not interesting.**

> **(1) The overall design and each of the three components of UniAdapter are not interesting. In particular, using shared weights in the lower layers followed by layers with specialized weights is common in multi-task learning literature. Weight-sharing has also been employed by previous parameter-efficient fine-tuning work like Compacter (Karimi Mahabadi et al., 2021).**

**A:** Thank you for your comments regarding the design and components of UniAdapter, particularly about the use of parameter-sharing, a common strategy in multi-task learning and parameter-efficient fine-tuning as seen in works like Compacter.

We acknowledge that parameter-sharing is not a novel concept per se, but its effective application in different contexts is key. In contrast to previous works where weight-sharing mainly supports multi-task learning, our approach utilizes it to enhance multimodal transfer learning, specifically aiming at the fusion of different modalities within the adapter. 

* Although previous works (e.g., Compacter) utilize the concept of weight-sharing, the motivations and goals of the two models are different. Most multi-task work aimed to make the model perform well on multi-tasks, thus opting for weight sharing at low layers, and they have achieved impressive performances on a wide range of benchmarks.  In UniAdapter, we leverage parameter-sharing in the down-projection layer of the UniAdapter the nuanced similarities and differences between modalities, which bring better performance with fewer data. Meanwhile, the design of weight-sharing in UniAdapter is at adapter level, will previous multi-task methods directly shared weights in the lower layers.
* To provide deeper insight, we have conducted additional visualizations (**see Figure4 and Figure5 in the Appendix part**). We select input representations from our UniAdapter at the 12th layer originating from different modality modules. After that, we conduct joint t-SNE and PCA visualizations on these representations. Our experimental data is derived from 128 randomly selected images from Flickr30K. The findings are as follows:
    - Inputs representations to the UniAdapter from various modalities exhibit similar spatial distributions, suggesting that they reside within a shared space (or overlapped spaces across distributions). This supports the effectiveness of our down-projection sharing strategies.
    - Further, we compared the input representations of the down-projection layer with those of the up-projection layer. Here, we noted that different modalities' data distributions occupy distinct spaces when it comes to the up-projection layer. This finding validates our experiments that sharing the up-projection layer negatively impact the model's performance, as it appears to interfere with the distinct representation spaces required for different modalities.
 
   - We hope the underlying insight behind this approach will inspire future works in this domain.
* We also have compared different layers of the modality sharing in **Table 12** in the appendix. The results indicate that **sharing weight for lower layers may lead to worse performance** for multimodal tasks. Our findings are different from multi-task learning methods (e.g., using shared weights in the lower layers).
* Our parameter-sharing mechanism in UniAdapter is meticulously designed, encompassing not only down-projection but also query residual adapters, query-redisual up-projection layers, and encoder-decoder models for tasks like Visual Question Answering (VQA). This comprehensive approach is specifically engineered for cross-modal parameter-efficient transfer learning, addressing unique challenges in this domain.

> **(2) Using residual connections is again a commonly seen trick.**

**A:** Thank you for noting the widespread use of residual connections in the field. We agree that employing residual connections is a known technique. However, we would like to emphasize that the contribution of our work is not rooted in the mere utilization of residual connections but in the strategic insights we have developed regarding their implementation: **Language Query information progressively diminishes in the cross-attention arthicture**. In response to this finding, we have strategically incorporated residual connections to mitigate this loss of information. This methodological choice was a targeted response to the specific challenge we identified through our study. This approach has notably improved our model's performance, illustrating the effectiveness.

Reviewer comment (Experiments):
Related work: The absence of related work published in 2023 from the first three sections of the paper is surprising. Only a few recent methods are used as baselines in the last experiment section.

Author response:
**Q2: Related work: The absence of related work published in 2023 from the first three sections of the paper is surprising. Only a few recent methods are used as baselines in the last experiment section.**


**A:** Thank you for pointing out the need to incorporate more recent related work from 2023 in the initial sections of our paper. We value your constructive feedback and recognize the importance of reflecting the latest advancements and trends in our field.

In response to your feedback, we have carefully reviewed and included additional related work published in 2023 in the first three sections of our manuscript. 
Furthermore, we have expanded the set of baselines in the last experiment section to encompass more recent methods. 


Thank you again for your constructive suggestion regarding the inclusion of more recent related work in the initial sections of our paper. Should there be any specific recent publications that we may have overlooked or that you believe should be included in our manuscript, please feel free to let us know.

Reviewer comment (Experiments):
Comparison fairness: In Table 2, the highlighted best-performing result is UniAdapter with r=512, which uses 19.0M parameters, significantly more than the middle rows. The comparison is kind of unfair, and it would be better to include "Adapter with r=512" for a fair evaluation.

Author response:
**Q3: Comparison fairness: In Table 2, the highlighted best-performing result is UniAdapter with r=512, which uses 19.0M parameters, significantly more than the middle rows. The comparison is kind of unfair, and it would be better to include "Adapter with r=512" for a fair evaluation.**

**A:** We apologize for any confusion caused by the presentation of our results in Table 2. Thank you for bringing this to our attention.

* We would like to clarify that the inclusion of UniAdapter with r=512 in Table 2 was primarily for **the purpose of comparing different configurations of UniAdapter itself** (specifically, r=128 vs. r=512) as well as **full finetuning baseline**. This was not intended as a direct comparison with the standard Adapter model. We realize now that this might not have been clearly conveyed in our original presentation.

* Acknowledging your valid point about ensuring a fair comparison, we have revised Table 2 to include results for the Adapter model with r=512. This adjustment ensures a more balanced and comprehensive evaluation, allowing for a direct comparison of UniAdapter's performance with that of an equivalently parameterized Adapter model.

Reviewer comment (Novelty):
The adapter mainly implements some reuse design for multimodal tasks, with its structure not deviating significantly from classical approaches. It is unclear if this is optimal for cross-modal applications. Has the author explored distinct design strategies for different modalities?

Author response:
**A:** Thank you for your thoughtful question about our design strategy for different modalities in multimodal tasks.

* We have indeed explored unique adapter designs for varying modalities. Specifically, for video tasks, we experimented a video-specific adapter (ST-Adapter [1]), for the visual backbone. Simultaneously, we used Parallel Adapters for the textual and multimodal backbones. Despite this modality-specific approach, our results, detailed in Table 4, reveal that the ST-Adapter was marginally less effective than employing Parallel Adapters across all modalities. We observed a slight performance difference (69.7 vs. 69.6 on R@Mean) while achieving a significant cost in parameter count (from 42.5M to 28.4M).

* Meanwhile, our findings in Table 1 highlight that for multimodal tasks, the integration of adapters in the cross-modal modality is more impactful than in the visual or textual modalities. This insight is pivotal to our UniAdapter design, where we focus on optimizing the multimodal aspect. Consequently, we have adopted a strategy of minimizing parameter count in the visual and textual modalities through parameter sharing. This approach avoids over-specialization in either modality, aligning with our goal of a more streamlined and efficient design.

* Our investigations suggest that a more uniform module design across different modalities tends to yield superior results compared to the introduction of modality-specific, specialized designs. This finding supports the notion that for cross-modal applications, a unified approach to adapter design can be more effective, as opposed to distinct, modality-specific implementations.

Reviewer comment (Experiments):
The experimental design appears to be somewhat disorganized; it is challenging to discern controlled variables in the comparative analysis presented in each table. This lack of clarity complicates the evaluation of the actual impact of different components of the method.

Author response:
**A:** Thank you for your feedback on the organization of our experimental design, we acknowledge that the presentation in Table 2 might have led to some confusion.

In response to your valuable input, we have revised Table 2 to include results for r=512. This update ensures that all ablation studies are consistently conducted at r=512. By doing so, we aim to provide a more coherent and straightforward comparison across different experimental setups. We believe that this adjustment will significantly improve the clarity of our experimental design and thank you for your suggestion in making our research more accessible and comprehensible.

Reviewer comment (Novelty):
Some of the ideas presented in this paper have been explored in previous works. For example, (1) the sharing of layers across multiple modalities has been addressed in [1]; (2) the aggregation of video (frame) features in a parameter-free manner, such as through attention or averaging, has also been discussed in [2].

Author response:
**Q1: Some of the ideas presented in this paper have been explored in previous works.**

> **(1) the sharing of layers across multiple modalities has been addressed in BEIT-3**

**A:** Parameter-sharing is a commonly used strategy, but how to correctly applied it is much more important. Although full parameter-sharing methods have achieve great success, recent arts (e.g., BLIP-2) show that proper partial parameter-sharing is more important. 
1. In this paper, we have carefully designed our parameter-sharing approach, considering not only down-projection parameter-sharing but also parameter-sharing for query residual adapters, up-projection adapters, and encoder-decoder models on VQA tasks. Our approach is tailored specifically for cross-modal parameter-efficient transfer learning. 
2. Regarding the comparison with BEIT-3, it's important to note that BEIT-3 employs direct Multi-Head Self-Attention sharing for each modality, a standard practice. In contrast, our UniAdapter sharing strategy is custom-designed for efficient cross-modal Adapter modules.
3. Meanwhile, designing an effective parameter-sharing module for parameter-efficient cross-modal modeling is non-trivial and sometimes even "counter-intuition". Our proposed strategy is carefully designed to meet the demands of cross-modal parameter-efficient transfer learning, and it is not a direct increment of previous methods. 

We believe that our work will inspire and assist researchers in designing better models for this task.


> **(2) the aggregation of video (frame) features in a parameter-free manner, such as through attention or averaging, has also been discussed in CLIP4Clip.**

**A:** Parameter-free aggregation of video (frame) features has indeed been discussed before. Nevertheless, how to better perform aggregation is even more important. In CLIP4Clip, the authors explored different schemes (including average, seqLSTM, seqTransf, tightTransf). However, they found that a simple average surpassed other careful designs, suggesting that parameter-free aggregation is a challenging task. 

Our UniAdapter employs language-guided free aggregation. Without adding extra parameters and with minimal computational overhead, it yields improved results. Additionally, we offer comparison results with various parameter-free aggregation strategies in the table below.

| Didemo       | R@1  | R@5  | R@10 | R@Mean |
|-|------|-|-|-|
| Average      | 47.4 | 70.1 | 79.7 | 65.7   |
| Token-Concat (our Baseline) | 51.0 | 75.8 | 84.2 | 70.3   |
| PFA (ours)   | **52.1** | **77.3** | **85.2** | **71.5**   |

Meanwhile, it is important to note that parameter-free aggregation (PFA) is **not** the core contribution of this work. Our UniAdapter mainly aims to construct a unified parameter-efficient training framework designed for multimodal tasks.

Reviewer comment (Experiments):
Regarding feature visualization, I suggest conducting a comparison between the results of the non-shared architecture, the up-shared one, and the all-shared one.

Author response:
**Q2: Regarding feature visualization, I suggest conducting a comparison between the results of the non-shared architecture, the up-shared one, and the all-shared one.**

**A:** Thank you for your valuable suggestion to conduct a comparative analysis of non-shared, up-shared, and all-shared architectures within our UniAdapter model. Following your recommendation, we have conducted an in-depth comparison, the results of which are now included in **Figure 4 and 5 in the Appendix**.

1. We observed that input representations to the UniAdapter from various sharing-strategy exhibit similar spatial distributions. This finding suggests that these representations reside within a shared space or have overlapped distributions across different modalities. Interestingly, the type of adapter employed (e.lg, non-shared vs. shared) does not seem to significantly impact this distribution, which we attribute to the majority of the model's parameters being frozen. This observation supports the effectiveness of our down-projection sharing strategies.
2. Further, we compared the input representations of the down-projection layer with those of the up-projection layer. Here, we noted that different modalities' data distributions occupy distinct spaces when it comes to the up-projection layer. This finding validates our experiments that sharing the up-projection layer negatively impact the model's performance, as it appears to interfere with the distinct representation spaces required for different modalities.

These additional visualizations and analyses provide valuable insights into the distribution of multimodal data within our UniAdapter model and underscore the impact of our architectural choices on the model's performance. We believe these findings significantly contribute to understanding the nuances of our UniAdapter's design and functionality.