Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment (2024)

Abstract: Transformer-based methods have gone mainstream in multimodal sequential learning. The intra and inter modality interactions are captured by the query-key associations of multi-head attention. In this way, the calculated multimodal contexts (attentional results) are expected to be relevant to the query modality. However, in existing literature, the alignment degree between different calculated attentional results of the same query are under-explored. Based on this concern, we propose a new constrained scheme called Multimodal Contextual Contrast (MCC), which could align the multiple attentional results from both local and global perspectives, making the information capture more efficient. Concretely, the calculated attentional results of different modalities are mapped into a common feature space, those attentional vectors with the same query are considered as a positive group and the remaining sets are negative. From local perspective, we sample the negative groups for a positive group by randomly changing the sequential step of one specific context and keeping the other stay the same. From coarse global perspective, we divide all the contextual groups into two sets (i.e., aligned and unaligned), making the total score of aligned group relatively large. We extend the vectorial inner product operation for more input and calculate the aligned score for each multimodal group. Considering that the computational complexity scales exponentially to the number of modalities, we adopt stochastic expectation approximation (SEA) for the real process. The extensive experimental results on several tasks reveal the effectiveness of our contributions.

## introduction multimodal sequential learning, which aims to process and understand the semantic information from multiple modalities (e.g., vision, language, audio) * † denotes corresponding author. and i would want the sequences to be even longer rather than shorter r ɑː ð ə not aligned query key key figure 1 : an example of calculated attentional results, where the colours denote attention weights and we present the text, visual, audio content of a speaker. the text query is "rather", the visual result ("shorter") is not aligned with audio result ("rather"). with machine learning skills, has drawn increasing attention recently. many endeavors @xcite @xcite are devoted to the design of multimodal interactive mode and effective individual representation learning. transformer-based multimodal interaction methods @xcite occupy the mainstream position in multimodal interaction area. compared with vanilla methods, transformer-based methods could achieve relatively superior performances with deep stacked attention blocks @xcite and a suitable number of training samples. concretely, by treating one modality as query (e.g., text) and the other modalities as keys (e.g., visual, audio), the multimodal contextual sequences (attentional results) can be obtained by query-key associated mechanism. in this way, the calculated attentional results are expected to be relevant to the query modality. however, the alignment degree between different calculated attentional results of the same query are under-explored. specifically, if we have three modalities (i.e., text, visual, audio), we employ transformer to correlate text-visual and textaudio interaction respectively. as shown in fig. 1 , when the text query is "rather", the attentional results ("shorter") of text-visual pair has obvious problems, while the audio localization ("rather") is correct. during training, the mistakes of query-key association are inevitable. at this time, constraining the alignment of audio and visual attentional results can polish the capture mistakes indirectly. motivated by the observations, in this paper, we propose a new constrained strategy called multimodal contextual contrast (mcc), which could align multimodal attentional results from both local and global perspectives, making the attentional capture more efficient. specifically, the multimodal attentional results of different modalities are calculated with multi-head attention first and then mapped into a common feature space. the sequence lengths of the multimodal attentional results are same as that of query modality, we denote the multiple attentional sequences as c @xmath0, where d is the feature dimension and n is the number of modalities. those attentional vectors with the same query (at the same sequential step @xmath1. from the local perspective, we sample the negative groups for a positive group by randomly changing the sequential step of one specific context vector and keeping the other stay the same. totally, the number of negative groups is n(t a -1) for a positive group. from the global perspective, we divide all the groups into two sets (i.e., aligned and unaligned), making the total score of aligned groups relatively large. for the implementation of contrastive constraints, we extend the vectorial inner product operation for more input and compute the aligned score for each multimodal group. considering that the computational complexity of relevance scores scales exponentially to the number of modalities, we adopt stochastic expectation approximation for the real process. we conduct extensive experiments on three tasks, the experimental results show that mcc could achieve competitive results compared with the state-of-the-art methods. to sum up, the contributions of our work are four-folded: the experimental results show that mcc could achieve competitive results compared with state-of-the-art methods. 2 related work multimodal interaction. existing multimodal interaction methods could be categorized into transformer-based and non-transformer-based methods. as for the former, zadeh et al. (2016b) proposes to train the model on simply concatenated multimodal features for prediction. poria et al. (2017) correlates multiple modalities with a context-dependent fusion method. zadeh et al. (2018b) explicitly accounts for both interactions in a neural architecture and continuously models them through time. as for the latter, tsai et al. (2019) proposes multimodal transformer to boost the interactions between multiple modalities. lmf-mult (sahay et al., 2020) combines the lmf and mult to process multimodal sequential information, first lmf-mult aligns different modalities at sequential-step level, fusing different modalities. the results are input to the transformer module. @xcite proposes the modal-order-aware network to integrate the three modalities in a certain order to distinguish the importance of different modalities. @xcite conducts contrastive feature decomposition from the sample level. ## contrastive contextual alignment in this section, we would introduce mcc with the following scheme. first, we give simple illustration of the definition and background of some symbols. second, we would introduce the local contrastive alignments and global contrastive alignments in detail (as shown in fig. 2 ). third, considering the complexity, we adopt the theoretical approximation for computation reduction. to be more intuitive, we further provide the complexity analysis of vanilla mcc and improved mcc. ## conclusion in this paper, we propose a generalizable constrained scheme for multimodal sequential learning, which could align attentional sequences from both local and global perspectives. concretely, the multimodal contexts at the same sequential step are considered as a positive set and the remaining sets are negative. we adopt additional random feature mechanism to approximate the real process, reducing the complexity. we conduct extensive experiments on several traditional tasks, the experimental results reveal the effectiveness of our contributions. in the future, we would focus on how to sample negative groups more effectively and simulate the constrained process without approximation error.