CTFN : Hierarchical Learning for Multimodal Sentiment Analysis Using Coupled-Translation Fusion Network (2021)

Abstract: Multimodal sentiment analysis is the challenging research area that attends to the fusion of multiple heterogeneous modalities. The main challenge is the occurrence of some missing modalities during the multimodal fusion procedure. However, the existing techniques require all modalities as input, thus are sensitive to missing modalities at predicting time. In this work, the coupled-translation fusion network (CTFN) is firstly proposed to model bi-direction interplay via couple learning, ensuring the robustness in respect to missing modalities. Specifically, the cyclic consistency constraint is presented to improve the translation performance, allowing us directly to discard decoder and only embraces encoder of Transformer. This could contribute to a much lighter model. Due to the couple learning, CTFN is able to conduct bi-direction cross-modality intercorrelation parallelly. Based on CTFN, a hierarchical architecture is further established to exploit multiple bi-direction translations, leading to double multimodal fusing embeddings compared with traditional translation methods. Moreover, the convolution block is utilized to further highlight explicit interactions among those translations. For evaluation, CTFN was verified on two multimodal benchmarks with extensive ablation studies. The experiments demonstrate that the proposed framework achieves state-of-the-art or often competitive performance. Additionally, CTFN still maintains robustness when considering missing modality.

## introduction sentiment analysis has witnessed many significant advances in the artificial intelligence community, in which text @xcite , visual @xcite , and acoustic @xcite modalities are primarily employed to the related research respectively, allowing to exploit the human emotional characteristic and intention effectively @xcite . intuitively, due to the consistency and complementarity among different sources, the joint representation attend to reason about multimodal messages, which are capable of boosting the performance of the specific task @xcite @xcite . multimodal fusion procedure is to incorporate multiple knowledge for predicting a precise and proper outcome @xcite . historically, the existing fusion has been done generally by leveraging the model-agnostic process, considering the early fusion, late fusion, and hybrid fusion technique @xcite . among those, early fusion focussed on the concatenation of the unimodal presentation (d'mello and kory, 2015). on the contrast, late fusion performs the integration at the decision level, by voting among all the model results @xcite . as to the hybrid fusion, the output comes from the combination of the early fusion and unimodal prediction @xcite . nevertheless, multimodal sentiment sequences often consists of unaligned properties, and the traditional fusion manners are failed to take the heterogeneity and misalignment into account carefully, which raises a question on investigating the more sophisticated models and estimating emotional information. @xcite . recently, transformer-based multimodal fusion framework has been developed to address the above issues with the help of multi-head attention mechanism @xcite @xcite . by introducing the standard transformer network @xcite as the basis, tsai et al. @xcite captured the integrations directly from unaligned multimodal streams in an end-to-end fashion, latently adapted streams from one modality to another with the cross-modal ## related work the off-the-shelf multimodal sentiment fusion architecture comprises two leading groups: translation-based and non-translation based model. non-translation based: recently, rnn-based models, considering gru and lstm, have received significant advances in exploiting the context-aware information across the data @xcite . bc -lst m @xcite and gm e -lst m @xcite presented a lstm-based model to retrieve contextual information, where the unimodal features are concatenated into a unit one as the input information. similarly, meldbase (poria et al., 2019) leveraged the concatenation of audio and textual features on the input layer, and employed gru to model sentimental context. in contrast, chf usion @xcite employed the rnn-based hierarchical structure to draw fine-grained local correlations among the modalities, and the empirical evidence illustrates superior advances compared to the simple concatenation of unimodal presentation. on the basis of rnn, mmmu -ba @xcite further employed multimodal attention block to absorb the contribution of all the neighboring utterances, which demonstrates that the attention mechanism can utilize the neighborhood contribution for integrating the contextual information. however, all these methods are suitable for the low-level presentation within the single modality with a nontranslation manner, which may be easily sensitive to the noisy terms and missing information in the sources. translation-based model: inspired by the recent success of sequence to sequence (seq2seq) models @xcite in machine translation, @xcite and @xcite presented multimodal fusion model via the essential insight that translates from a source modality to a target modality, which is able to capture much more robust associations across multiple modalities. mct n model incorporated a cyclic translation module to retrieve the robust joint representation between modalities in a sequential manner, e.g., the language information firstly associated with the visual modality, and latently translated into the acoustic modality. compared with the mct n, seq2seq2sent introduced a hierarchical fusion model using the seq2seq methods. for the first layer, the joint representation of a modality pair is treated as an input sequence for the next seq2seq layer in an attempt to decode the third modality. inspired by the success of the transformer-based model, tsai et al. introduced a directional crossmodality attention module to extend the standard transformer network. follow the basic idea of @xcite provided a novel multimodal fusion cell which is comprised of two standard transformers, embracing the association with a modality pair during the forward and backward translation implicitly. however, all existing models adopt sequential multimodal fusion architecture, which requires all modalities as input, therefore they can be sensitive to the case of multiple missing modalities. moreover, the explicit interactions among cross-modality translations were not considered. ## methodology in this section, we firstly present ctfn (figure 2 ), which is capable of exploring bi-direction crossmodality translation via couple learning. on the basis of ctfn, a hierarchical architecture is established to exploit multiple bi-direction translations, leading to double multimodal fusing embeddings (figure 4 ). then, the convolutional fusion block (figure 3 ) is applied to further highlight explicit correlation among cross-modality translations. ## conclusion in this paper, we present a novel hierarchical multimodal fusion architecture using coupled-translation fusion network (ctfn). initially, ctfn is utilized for exploiting bi-directional interplay via couple learning, ensuring the robustness in respect to missing modalities. specifically, the cyclic mechanism directly discards the decoder and only embraces the encoder of transformer, which could contribute to a much lighter model. due to the couple learning, ctfn is able to conduct bi-direction cross-modality intercorrelation parallelly. based on ctfn, a hierarchical architecture is further established to exploit multiple bi-direction translations, leading to double multimodal fusing embeddings compared with traditional translation methods. additionally, a multimodal convolutional fusion block is employed to further explore the complementarity and consistency between crossmodality translations. essentially, the parallel fusion strategy allows the model maintains robustness and flexibility when considering only one input modality. ctfn was verified on two public multimodal sentiment benchmarks, the experiments demonstrate the effectiveness and flexibility of ctfn, and ctfn achieves state-of-theart or comparable performance on cmu-mosi and meld (sentiment). for future work, we like to evaluate ctfn on more multimodal fusion tasks. the source code can be obtained from https://github.com/deepsuperviser/ctfn .