Paper: Prompt Learning with Quaternion Networks (ICLR-2024)

Reviewer comment (Theory):
Quaternion hidden space seems to be more sophisticated than linear space which might be better than linear projection. However, it's not obvious why Quaternion Networks is better than the previous prompting technique; or why tunning multimodal pre-trained networks needs quaternion hidden space.

Author response:
**Cons2: The superiority of quaternion spaces.** 
We acknowledge that quaternion hidden spaces are more sophisticated than linear spaces, resulting in better performance. However, current multimodal fusion methods often rely on explicit interaction structures that struggle to fully capture the varied and complex patterns inherent in multimodal data. For example, MaPLE with linear mappings oversimplifies the associations between different modalities. Similarly, the cross-attention method assigns maximum weight to the modality that is beneficial to the current task, but fails to leverage the rich complementary information among different modalities, inadvertently discarding essential data. We have presented detailed ablation experiments in Appendix Figure B. In our approach, QNet, we use quaternion hidden space to construct more complex and non-linear relationships among diverse modalities in a more compact manner. First, we merge output image features from a pre-trained model with text context features. Then, we process these integrated features through a quaternion encoder. Within the quaternion hidden spaces, our QNet employs orthogonal imaginary axes (i.e., i, j, and k) to further tackle the fused features in a higher dimension. Our approach provides a holistic understanding and captures intricate details that might be overlooked from a single-dimensional perspective. Additionally, our QNet integrates features across hierarchical levels to enrich the representation from various perspectives. Extensive experiments on multiple datasets demonstrate the superiority of our approach over the state-of-the-art methods. Furthermore, our QNet can be used as a plug-and-play module to support various multimodal approaches.