Unimodal and Crossmodal Refinement Network for Multimodal Sequence Fusion (2021)

Abstract: Effective unimodal representation and complementary crossmodal representation fusion are both important in multimodal representation learning. Prior works often modulate one modal feature to another straightforwardly and thus, underutilizing both unimodal and crossmodal representation refinements, which incurs a bottleneck of performance improvement. In this paper, Unimodal and Crossmodal Refinement Network (UCRN) is proposed to enhance both unimodal and crossmodal representations. Specifically, to improve unimodal representations, a unimodal refinement module is designed to refine modality-specific learning via iteratively updating the distribution with transformer-based attention layers. Self-quality improvement layers are followed to generate the desired weighted representations progressively. Subsequently, those unimodal representations are projected into a common latent space, regularized by a multimodal Jensen-Shannon divergence loss for better crossmodal refinement. Lastly, a crossmodal refinement module is employed to integrate all information. By hierarchical explorations on unimodal, bimodal, and trimodal interactions, UCRN is highly robust against missing modality and noisy data. Experimental results on MOSI and MOSEI datasets illustrated that the proposed UCRN outperforms recent state-of-the-art techniques and its robustness is highly preferred in real multimodal sequence fusion scenarios. Codes will be shared publicly.

## introduction motivated by recent research achievements on modality representations in language @xcite @xcite , audio @xcite @xcite , and vision (he et al., 2016; woo 1 https://github.com/hw-aarc-club/ emnlp2021_ucrn @xcite , multimodal learning @xcite that aims to efficiently extract joint representations from multiple sensory data, has drawn much attention recently. by mining complementary relations across modalities, multimodal fusion could present a more reliable and comprehensive interpretation of the world. with the development of various network architectures, considerable progress has been achieved in multimodal fusion @xcite @xcite , showing that multimodal representation outperforms unimodal ones on emotion and sentiment prediction tasks. additionally, recent works seek to improve the efficacy of the multimodal fusion methods by assuming that one modality can be translated to another @xcite , or modulated by pivot modality @xcite , so as to align pairwise representations in a common space. however, converting one modality to another modality appears to be inadequate in projecting all modalities into one feature space. a specific pattern may not coexist in all modalities (e.g., it expresses happiness in language/audio while showing neutrality on facial expression). also, the conversion is usually followed by a downstream fusion to produce the final fusion result. therefore, the downstream fusion may have already encompassed it as a byproduct making conversion redundant in terms of multimodal fusion tasks. it has been frequently reported that lexical representation is a stronger predictor than audio and vision representations. hence, @xcite leverages language feature to modulate others. as a result, it becomes the main contributor towards multimodal fused representation. on the contrary, through this process, those weak predictors are prone to undermine the modality-common representations. from this view, the strategy of direct modality modulation or pairwise translation may jeopardize learning rich-fused representation and lead to suboptimal results. moreover, most of those network architectures require all modalities as input. as a result, the learned representations may perform poorly in the real world where complete modalities might not always be simultaneously available (e.g., some specific modality is missing or noisy). this may be because of over fusing or losing sight of addressing the importance of unimodal refinement. although the presence of multiple modalities provides additional information, there are two key challenges to be addressed when learning from multimodal data: 1) models must learn the complex intramodal and crossmodal interactions for predictions, and 2) models must be robust to unexpected missing or noisy modalities during testing. to address the aforementioned problems, the unimodal and crossmodal refinement network (ucrn) is proposed, which takes both robust unimodal representation and efficient crossmodal representation into consideration. our hypotheses are 1) a robust unimodal representation is essential for efficient multimodal fusion, 2) it is beneficial to reduce modality gaps before modality fusion, and 3) stacks of attention-based mechanism can efficiently select the most salient features within a representation, favoring robust representation learning. guided by the above hypotheses, the proposed ucrn consists of the following sub-modules with inputs of basic modality sequence representations: 1) a unimodal refinement module is proposed to yield robust modality-specific representations; 2) the robust unimodal representations are projected to different latent spaces with the aims of reducing representation gaps and 3) the latent spaces are concatenated to correlate modality-common features and to produce robust multimodal representation. the final predicted sequence label is the one with the highest probability in the network output. to recap, the contributions of this paper can be summarized as follows: ## unimodal and crossmodal refinement network (ucrn) in this paper, unimodal and crossmodal refinement network (ucrn) is proposed. as shown in figure 1 , ucrn is comprised of three main parts, where the first part conducts unimodal representation refinement, the second part refines all the previous information for fusion and learns a modality-common representation, and the last part performs prediction. the unimodal refinement module (urm) takes unimodal features (i.e., language, audio, vision features) as input to learn the refined unimodal representations. then the refined unimodal representations are mapped to a common latent space by imposing a multimodal jenson-shannon (mjs) divergence regularizer. following this, the self-quality improvement layers (sqil) are used to further extract desired weighted unimodal representations for fusion. lastly, the crossmodal refinement module (crm) integrates all information and extracts multimodal interactions. ## conclusion in this work, the unimodal and crossmodal refinement network (ucrn) is proposed for robust and efficient multimodal representation learning. we hypothesis that unimodal representation is better to be refined before crossmodal fusion, and it is beneficial to reduce modality gaps before crossmodal refinement. following the line, the proposed network is designed with a unimodal refinement module, a multimodal js divergence regularizer, self-quality improvement layers, and a crossmodal refinement module. the experimental results validated all our assumptions. in particular, the robustness experiments evinced high efficiency and validated that ucrn can handle the modality missing and noise issues. experimental results showed ucrn achieves state-of-the-art results on multiple multimodal datasets.