Integrating Multimodal Information in Large Pretrained Transformers (2020)

Abstract: Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straightforward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). More specifically, this is due to the fact that pre-trained models don’t have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.

## introduction human face-to-face communication flows as a seamless integration of language, acoustic, and vision modalities. in ordinary everyday interactions, we utilize all these modalities jointly to convey our * -equal contribution intentions and emotions. understanding this faceto-face communication falls within an increasingly growing nlp research area called multimodal language analysis @xcite . the biggest challenge in this area is to efficiently model the three pillars of communication together. this gives artificial intelligence systems the capability to comprehend the multi-sensory information without disregarding nonverbal factors. in many applications such as dialogue systems and virtual reality, this capability is crucial to maintain the high quality of user interaction. the recent success of contextual word representations in nlp is largely credited to new transformer-based @xcite models such as bert @xcite and xlnet @xcite . these transformer-based models have shown performance improvement across downstream tasks @xcite . however, their true downstream potential comes from finetuning their pre-trained models for particular tasks @xcite . this is often done easily for lexical datasets which exhibit language modality only. however, this fine-tuning for multimodal language is neither trivial nor yet studied; simply because both bert and xlnet only expect linguistic input. therefore, in applying bert and xlnet to multimodal language, one must either (a) forfeit the nonverbal information and fine-tune for language, or (b) simply extract word representations and proceed to use a state-of-the-art model for multimodal studies. in this paper, we present a successful framework for fine-tuning bert and xlnet for multimodal input. our framework allows the bert and xl-net core structures to remain intact, and only attaches a carefully designed multimodal adaptation gate (mag) to the models. using an attention conditioned on the nonverbal behaviors, mag essentially maps the informative visual and acoustic factors to a vector with a trajectory and magnitude. during fine-tuning, this adaptation vector modifies the internal state of the bert and xlnet, allowing the models to seamlessly adapt to the multimodal input. in our experiments we use the cmu-mosi @xcite and cmu-mosei @xcite datasets of multimodal language, with a specific focus on the core nlp task of multimodal sentiment analysis. we compare the performance of mag-bert and mag-xlnet to the above (a) and (b) scenarios in both classification and regression sentiment analysis. our findings demonstrate that fine-tuning these advanced pre-trained transformers using mag yields consistent improvement, even though bert and xlnet were never trained on multimodal data. the contributions of this paper are therefore summarized as: ## related works the studies in this paper are related to the following research areas: ## bert and xlnet to better understand the proposed multimodal framework in this paper, we first present an overview of both the bert and xlnet models. we start by quickly formalizing the operations within transformer and transformer-xl models, followed by an overview of bert and xlnet. ## multimodal adaptation gate (mag) in multimodal language, a lexical input is accompanied by visual and acoustic information -simply gestures and prosody co-occurring with language. consider a semantic space that captures latent concepts (positions in the latent space) for individual words. in absence of multimodal accompaniments, the semantic space is directly conditioned on the language manifold. simply put, each word falls within some part of this semantic space, depending only on the meaning of the word in a linguistic structure (i.e. sentence). nonverbal behaviors can have an impact on the meaning of words, and therefore on the position of words in this semantic space. together, language and nonverbal accompaniments decide on the new position of the word in the semantic space. in this paper, we regard to this new position as addition of the language-only position with a displacement vector; a vector with trajectory and magnitude that shifts the language-only position of the word to the new position in light of nonverbal behaviors. this is the core philosophy behind the multimodal adaptation gate (mag). a particularly appealing implementation of such displacement is studied in raven @xcite , where displacements are calculated using cross-modal self-attention to highlight relevant nonverbal information. figure 1 shows the studied mag in this paper. essentially, a mag unit receives three inputs, one is purely lexical, one is visual, and the last one is acoustic. let the triplet (z i , a i , v i ) denote these inputs for ith word in a sequence. we break this displacement into bimodal factors [z i ; a i ] and [z i ; v i ] by concatenating lexical vector with acoustic and visual information respectively and use them to produce two gating vectors g v i and g a i : we then create a non-verbal displacement vector h i by fusing together a i and v i multiplied by their respective gating vectors: ## experiments in this section we outline the experiments in this paper. we first start by describing the datasets, followed by description of extracted features, baselines, and experimental setup. ## results and discussion ## conclusion in this paper, we introduced a method for efficiently finetuning large pre-trained transformer models for multimodal language. using a proposed multimodal adaptation gate (mag), bert and xlnet were successfully fine-tuned in presence of vision and acoustic modalities. mag essentially poses the nonverbal behavior as a vector with a trajectory and magnitude, which is subsequently used to shift lexical representations within the pre-trained transformer model. a unique characteristic of mag is that it makes no change to the original structure of bert or xlnet, but rather comes as an attachment to both models. our experiments demonstrated the superior performance of mag-bert and mag-xlnet. the code for both mag-bert and mag-xlnet are publicly available here foot_1