Amanda: Adaptively Modality-Balanced Domain Adaptation for Multimodal Emotion Recognition (2024)

Abstract: This paper investigates unsupervised multimodal domain adaptation for multimodal emotion recognition, which is a solution for data scarcity yet remains under studied. Due to the varying distribution discrepancies of different modalities between source and target domains, the primary challenge lies in how to balance the domain alignment across modalities to guarantee they are all well aligned. To achieve this, we first develop our model based on the information bottleneck theory to learn optimal representation for each modality independently. Then, we align the domains via matching the label distributions and the representations. In order to balance the representation alignment, we propose to minimize a surrogate of the alignment losses, which is equivalent to adaptively adjusting the weights of the modalities throughout training, thus achieving balanced domain alignment across modalities. Overall, the proposed approach features A daptively m odality-bal an ced d omain a daptation, dubbed Amanda , for multimodal emotion recognition. Extensive empirical results on commonly used benchmark datasets demonstrate that Amanda significantly outperforms competing approaches. The code is available at https://github.com/sunjunaimer/Amanda .

## introduction emotion recognition has gained increasing attention in recent years in a wide spectrum of applications, including emotional support @xcite , conversation system @xcite and healthcare @xcite . multimodal emotion recognition which takes advantage of heterogeneous and complementary signals, such as acoustic, visual, lexical information, has demonstrated superior performance to its unimodal counterpart @xcite . nevertheless, one of the notable drawbacks of multimodal * corresponding author: jun @xcite . in this regard, unsupervised domain adaptation techniques are popular for promoting the generalization capability of a model from a labeled source domain to an unlabeled target domain. domain adaptation typically fills the model's performance gap between the target and source domains via matching the data distributions of the two domains with sample-based, feature-based and inferencebased approaches @xcite . accordingly, numerous schemes have been developed for various tasks in the fields of computer vision @xcite and natural language processing @xcite . in contrast, domain adaptation in multmodal learning settings remains relatively less researched, not to mention in multimodal emotion recognition. the previous literature on multimodal domain adaptation broadly falls into two categories, multiple visual modalities and general multiple modalities. the former tackles multimodal computer vision tasks, where different modalities correspond to rgb and optical flows @xcite , @xcite , or 2d image and 3d point cloud @xcite . in these scenarios, the modalities are similar and share the same environment, suggesting a close distribution gap between source and target domains of different modalities. therefore, no specific effort is required to address modality differences when aligning the domains. the latter category focuses on more general multimodal domain adaptation approaches, applicable to text/image and video/audio applications. however, in these studies, the source and target domains of different modalities are aligned uniformly without recognizing modality disparity @xcite , or they are not considered jointly, leading to the situation where some modalities are well aligned while others not @xcite . in multimodal emotion recognition tasks, the commonly utilized modalities-linguistic, visual and acoustic, exhibit high heterogeneity. moreover, these modalities live in decoupled spaces, as opposed to the visual modalities mentioned above. consequently, from the source to target domains, different modalities experience varying degree of distribution shift. for example, consider a shift in the working scene of a multimodal emotion recognition system from the day (source domain) to the night (target domain). in this scenario, the distribution of visual features noticeably shifts due to the variation of illumination conditions, while that of acoustic features remains relatively unchanged. henceforth, directly applying existing domain adaptation approaches to multimodal emotion recognition might result in an imbalanced alignment of different modalities. the model may then rely heavily on the well aligned modalities in the target domain and under-utilize others; in other words, well aligned modalities dominate others, causing the latter to be under-trained. with the above analysis, in this paper, we advocate modality independence @xcite and align the source and target domains of different modalities, taking their varying distribution gaps into consideration. to be specific, we design our model based on the information bottleneck (ib) theory @xcite , which enforces each modality to perform label prediction, thereby encouraging each to obtain its optimal representation independently. as for the domain alignment, we first introduce label distribution alignment under the practical assumption that the label distributions remain consistent across the source and target domains. we then employ correlation alignment @xcite to match the optimal representations in the two domains for each modality. to balance the representation alignment, we minimize a surrogate of the alignment losses rather than minimizing a weighted sum of the losses with fixed weights. via judiciously devising the surrogate function, minimizing it is tantamount to minimizing the weighted sum of the losses with the weights being adaptively tuned throughout the training progress. concretely, the modalities with larger (resp. smaller) losses receive proportionally larger (resp. smaller) weights, which achieves dynamically balanced domain alignment across modalities. in summary, our work features adaptively modality-balanced domain adaptation (abbreviated as amanda) for multimodal emotion recognition. the contributions are primarily threefold. ## method: amanda prior to delving into our method, amanda, we introduce the notations and assumptions below. notations: suppose the multimodal training dataset contains n samples, each with m modalities. for ease of expression, let us define an auxiliary modality as a union of all modalities, and thus the total number of modalities is m + 1. let [p ] for any positive integer p denote the set {1, 2, • • • , p }. the training samples are denoted by ({x n,m } @xmath0, {y n,m } @xmath1, where @xmath2, x n,@xmath3, ∀@xmath4, and y n,m represents the label corresponding to modality m, ∀@xmath5, y n,1 = y n,2 = • • • = y n,m +1 holds). suppose the number of emotion categories is c; then the label y n,m can be a one-hot vector or a scalar in [c], and we adopt either of these two forms when necessary in the rest of the paper. for the consistency of expression, we use x n,m +1 := [x n,1 ; x n,2 ; • • • ; x n,m ] to collect all features of sample n. let x m and y m represent general feature and label random variables for all @xmath6, with x n,m and y n,m as their realizations. let vector z @xmath7, a map of x m , denote the representation of modality m, and z n,m is a realization of z m (for brevity, we assume the representations of all modalities are d-dimensional vectors). we use superscript s and t to distinguish variables of source and target domains. for instance, x s m and x t m denote the features of modality m from source and target domains, respectively. assumptions: in this paper we consider unsupervised domain adaptation problem for multimodal learning, for which the following assumptions are satisfied: 1) the label target domain data is inaccessible; 2) the feature distributions shift with the domains, yet the label distributions remain unchanged, meaning that p(x t m ) ̸ = p(x s m ) and p(y t m ) = p(y s m ) hold for any @xmath8, where p(•) represents the distribution of a random variable. the second assumption holds true when the domain changes the feature but is not a causal factor of the considered event. for example, although the illumination (the feature) of the vision system varies between the day and the night (different domains), one's emotion (the label) distribution remains relatively stable with the day and the night. ## numerical results benchmark datasets: we assess our method on four widely used benchmark multimodal emotion recognition datasets, iemocap @xcite , meld @xcite , cmu-mosei @xcite , and msp-improv @xcite , which all contain acoustic, visual and lexical modalities. iemocap and msp-improv are composed of dyadic conversations collected in the laboratory setting, and the latter is of higher recording quality. cmu-mosei gathers monologue videos from more than 1000 speakers on youtube over various topics. meld consists of fragments from the tv series "friends", which contains multi-party conversations with over two participants. these datasets are collected from different scenarios and exhibit different characteristics, and hence represent different domains. following work @xcite , we select samples in the four classes-neutral, happy, sad and angry, to construct datasets for our experiments. feature extraction: for the visual modality, we first sample each video uniformly to obtain 64 frames. then, the frames are processed with s3fd @xcite to attain the speaker's faces which are then fed into vision model apvit @xcite , resulting in 64 × 768 sequential feature. bert-base @xcite and wav2vec2 @xcite are employed to extract lexical and acoustic features, respectively. to retain the feature of different levels, the outputs from the 1st, 7th, and 12th transformer blocks are concatenated as the final feature. the generated feature sequences are of dimension 2304, and their lengths are determined by the lengths of the text and audio, respectively. baseline models: we compare our model, amanda, with dann @xcite , cdan @xcite , mada @xcite , daln @xcite , and cdan-e, of which the first four are introduced in the related works sections, and cdan+e is an extension of cdan with the incorporation of entropy-aware reweighting for the domain discrimination loss. implementation details: the multimodal emotion recognition model involves three modalities, in which we employ one-layer lstm for visual modality, and textcnn for acoustic and lexical modalities as study @xcite . the dimension of the representations is chosen as 128. we adopt optimizer adam with learning rate 5 × 10 -4 , momentum coefficient (0.9, 0.999) and batch size 128 for model training. the parameter settings are γ = 5 × 10 -4 , α 1 = 0.08; and α 2 = 0.1 is selected for the comparison studies, and we will investigate how α 2 impacts the model performance in the ablation studies. more details of the implementation can be found from the code in the supplementary material. throughout this section, we use weighted f1 score as model performance metric. the reported f1 scores are obtained by averaging results from 3 repeated experiments, conducted on 2 nvidia a100 gpus with 40gb memory. ## limitations in light of the future work, the limitations of the present work are mainly twofold. 1) although our method is applicable to more general multimodal supervised learning problems, we only validate it on emotion recognition tasks. 2) we have not established the theoretical upper bound of the target domain risk for the proposed approach.