A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles” (2021)

Abstract: Imposing the style of one image onto another is called style transfer. For example, the style of a Van Gogh painting might be imposed on a photograph to yield an interesting hybrid. This paper applies the adaptive normalization used for image style transfer to language semantics, i.e., the style is the way the words are said (tone of voice and facial expressions) and these are style-transferred onto the text. The goal is to learn richer representations for multi-modal utterances using style-transferred multi-modal features. The proposed Style-Transfer Transformer (STT) grafts a stepped styled adaptive layer-normalization onto a transformer network, the output from which is used in sentiment analysis and emotion recognition problems. In addition to achieving performance on par with the state-of-the art (but using less than a third of the model parameters), we examine the relative contributions of each mode when used in the downstream applications.

## introduction multi-modal language analysis expands textual analysis by utilizing co-occurring acoustic and visual information, and has recently become a popular topic in machine learning @xcite . in both sentiment analysis @xcite and emotion recognition @xcite , the three modalities are combined to better represent the sentiment or emotional meaning of a passage. the idea of combing textual, acoustic, and visual features is obvious: individual modalities are not always able to convey as accurate an impression as multi-modal features, which typically provide more complete information. for instance, fig. 1 * work done while at uw-madison shows a caricature where the sentiment may be easily understood from the audio and video, but not from textual analysis alone. combining textual, acoustic, and visual features can be accomplished in a variety of ways ( @xcite @xcite @xcite ). among these, @xcite ) consider text as the backbone and study methods to inject acoustic and visual information into the textual features that are typically extracted via pre-trained word/language models, e.g., glove @xcite , elmo @xcite , and bert @xcite . this paper also uses textual features as a backbone and studies a novel way of injecting non-text features into a primarily text-only model. a style vector is learned from the acoustic and visual features. this style vector is then transferred to a text input transformer encoder via stepped adaptive layer normalization (sadlan). while adaptive style transfer in image processing literature is well studied @xcite @xcite , the novelty in our work is to consider the corresponding audio and video to be the style of the text and transfer the nonverbal features' information to text-based models via the effective sadlan. concretely, our model replaces the original layer normalization in the text transformer encoder with the proposed sadlan, which learns style scale and style bias from nontext features. the proposed model is named as @xcite and it is tested on three benchmark datasets. this paper makes three contributions. first, our model's performance is on par with the state-of-theart but using only less than one third of the model parameters. since our model does not require training a multimodal transformer from scratch to achieve the same results, the style transfer method benefits from both reduced model size and training time. second, we introduce the stepped adaptive layer normalization (sadlan), which performs adaptive normalization as a function of the layer of the dnn encoder. third, we study the contributions of each modality towards use in our downstream applications. while we know that multimodal embeddings would contribute more than each individual modality, such an examination highlights the relative strength of each mode, particularly on sentiment classification tasks. the rest of this paper is organized as follows. section 2 reviews the related work. section 3 describes our proposed method style transfer transformer (stt). section 4 and 5 presents experimental results, and section 6 concludes this work. ## related work multi-modal language analysis: previous work on multi-modal language analysis typically learns a novel network structure to capture interactions between text, audio, and video features via supervised learning. @xcite develop an efficient low-rank tensor fusion mechanism to learn the outer-product representation of multi-modal features, while @xcite ) learn a multi-modal memory gate that is applied to an lstm to capture the flow of information in the different modalities. in their work @xcite learn a specialized multi-modal fusion model by applying a novel multistage fusion in the recurrent network. recently, the multi-modal transformer introduced by @xcite achieves the state-of-the-art performance by using bi-directional cross modal relationships between the different modalities. note that the transformer architecture in stt is the same as in @xcite . however, we depart from their modeling procedure by encoding audio and video jointly using a single bimodal transformer block, thereby eliminating additional cross modal attention blocks. adaptive normalization: adaptive normalization is widely used in image processing and computer vision. @xcite proposes adaptive instance normalization, which learns the affine parameters from the style vector to perform a style transfer in the image encoder's feature space. karras et al. (2018) applies the adin in the generative adversarial network (gan) to generate fake multi-styled human faces. park et al. (2019) proposes a spatially-adaptive normalization for image semantic synthesis. wang et al. (2018a) applies a spatial feature transformation for image superresolution. this paper introduces the sadlan framework in which a bimodal style vector is transferred as a function of the dnn's depth. adapter method in pre-trained language models: due to the enormous size of pre-trained language models (e.g. bert, xlnet), the procedure of fine-tuning the models on downstream data may be inefficient. to address this "adapter" based methods have been developed. in their work @xcite apply the task-specific layer at each encoder layer of bert. each taskspecific layer contains two feed-forward projectors: one down-projects (maps the input vector to a low-dimensional space) and the other up-projects (maps the prior layer's output back to its original dimension). during training, only the parameters in these task-specific layers are updated. stickland and @xcite proposes another method: the task-specific layer is applied between the two layernormalization in the encoder. @xcite applies parallel adapters which each learn different information from the knowledge base in order to enable the model to deal with the multi-task learning task. our proposed sadlan is motivated by the the adapter designed in @xcite . in their work, the authors find that lower layers of the transformer have less impact on the fine tuning objective of a given task, while higher layers are more vital (this is an intuitive observation, because, lower layers are more likely to learn semantic meanings while the higher layers are more related to the specifics of the task). when using sadlan, lower layers of the stt are changed as little as possible to enable the model to learn basic semantic information from the input text, while the top layers allow a larger effect influenced by the style factors. ## style transfer transformer (stt) we define the text, audio, and video features for a given utterance as f @xmath0, f @xmath1, f @xmath2, where l is the length of the modality sequence and d t , d a , d v are embedding dimensions of each modality (each modalities' sequence length is forced to be the same by applying alignment). figure 2 (left) shows an overview of our model stt. it consists of the following four steps: step 1 learn a style vector: in keeping with the hypothesis that the text modality is the major contributor to the learned multimodal embedding, a style encoder is first applied to acoustic (f a ) and visual (f v ) features to learn the non-verbal style vector f @xmath3. the stt first concatenates audio and video sequences at each time step, i.e., f @xmath4. the concatenated f av is then input into the transformer model transformer av , which is a basic self-attention multi-head transformer encoder (with @xmath5; see @xcite ). the final state of transformer av is the style vector f s . step 2 adaptive style transfer onto text: the original layer norm @xcite is defined as: the style transfer techniques for image processing applications use factors s, b learned from a style vector to replace γ, β @xcite . ## experimental setup this section provides a brief overview of the experimental setup, datasets, and baseline methods used for comparison. ## experimental results tables 2 and 3 report results from the baselines and stt on the three benchmark data sets. we can see that stt performs on par with the baselines and sota considered in our experiments. since there exists a disparity between the results reported in @xcite and the results obtained from reproducing the source code released by the authors, we report numbers from reproduced mult as mult rep and report numbers from the paper as mult paper . note that stt beats the performance of mult rep while matching the performance of mult paper . to keep comparisons fair, we compare stt with the averages of scores attained by mult paper and mult rep and note that on all three benchmark datasets on average we do better than mult. while one may argue that reproducing the baseline is essential, we can offer as an explanation that the lack of clarity in reproducibility should not penalize our modeling efforts. to document our experimental evaluations and hypothesis in a credible manner, we report numbers from the paper as well as the reproduction. furthermore, we note that stt is more efficient than mult: it achieves better performance using less than a third number of parameters. see to verify if the performance of stt increases with the number of parameters, we investigate increasing: 1) the dimension of the transformer encoder's hidden layer, 2) the number of encoder layers and number of heads, and 3) the number of layers in the gru. however, we do not find improvements in performance by increasing the number of model parameters. we posit that this performance limitation is due to the relatively crude features extracted for audio and video analysis and suggest exploration in this direction for improved performance. ## conclusion and future work inspired by the success of style transfer algorithms in image processing, this paper proposed the novel style transfer transformer (stt) in which layernormalization in the transformer model is replaced with a style stepped adaptive layer normalization (sadlan). the model is used to learn comprehensive multimodal representations for sentiment analysis and emotion recognition. experiments on benchmark data sets established the effectiveness of the proposed method. furthermore, ablation studies provided supports for our hypothesis of injecting audio and video to highly efficient text embeddings enhances the performance of the text embedding in multimodal tasks without the need for larger models or training data. while our work is a first step towards learning multimodal embeddings via style transfer of non textual features onto text, as part of future work we will consider learning to inject non-verbal information into the text model in a recursive manner in order to achieve a higher model performance. the acoustic and visual features can be processed separately; besides of analyzing examples that benefit from the stt, it's also worthwhile to study examples that are negatively impacted by the method. while transformer models promise improved results, reproducibility in these models is a cause for concern since transformer models are particularly sensitive to initial conditions. as part of future work we will perform experiments to establish significance in observed results and report averaged hyper-parameters.