QAP : A Quantum-Inspired Adaptive-Priority-Learning Model for Multimodal Emotion Recognition (2023)

Abstract: Multimodal emotion recognition for video has gained considerable attention in recent years, in which three modalities ( i.e., textual, visual and acoustic) are involved. Due to the diverse levels of informational content related to emotion, three modalities typically possess varying degrees of contribution to emotion recognition. More seriously, there might be inconsistencies between the emotion of individual modality and the video. The challenges mentioned above are caused by the inherent uncertainty of emotion. Inspired by the recent advances of quantum theory in modeling uncertainty, we make an initial attempt to design a quantum-inspired adaptive-priority-learning model (QAP) to address the challenges. Specifically, the quantum state is introduced to model modal features, which allows each modality to retain all emotional tendencies until the final classification. Additionally, we design Q-attention to orderly integrate three modalities, and then QAP learns modal priority adaptively so that modalities can provide different amounts of information based on priority. Experimental results on the IEMOCAP and MOSEI datasets show that QAP establishes new state-of-the-art results.

## introduction multimodal emotion recognition (mer) has attracted more and more interest due to the rapid growth of multimedia information. mer aims to recognize the emotions of the speaker in the video. multiple modalities enrich human emotional expression and they are all closely related to emotion. generally, textual modality provides the most basic semantic information, visual modality provides emotional expressions, and acoustic modality provides the changing tone. three modalities also bring greater challenges to emotion recognition. due to the different amounts of information related to emotions, the priority of each modality varies from sample to sample. if different modalities are not discriminated in fusion, information related to emotion cannot be fully extracted. in the example on the left of figure 1 (a), the dejected expression, wrinkled eyebrows and drooping corners of the eyes all show anger and disgust, so visual modality contributes more to emotion. in the example on the right, a rising tone shows the emotion of happiness, so acoustic modality has higher priority than visual modality. most previous works @xcite @xcite treat modalities equally and do not pay attention to the important role of modal priority. some other works @xcite integrate modalities in a certain order, but the order is not adaptively adjusted for different samples. in practical scenarios, a fixed order cannot fit all samples. more seriously, there might be inconsistencies between the emotion of individual modality and the video. in the example in figure 1 (b), happiness is expressed in the text modality, but the emotion of the video is anger. some previous methods @xcite @xcite do not consider this issue and still integrate three modalities together, resulting in a negative impact on final emotions. some other methods @xcite remove the modality with inconsistent emotions and replace it with other features, which lose the semantic information contained in the modality. as part of human cognition, emotion is always in an uncertain state and constantly evolving until the final decision is made @xcite . specifically, the emotion of a video is considered to be uncertain until it is measured and collapses to an eigenstate, and so does one of the modalities. conceptually, in non-quantum models, the emotions in the video are pre-defined values and a measurement (classification) merely records them. in other words, three modalities are always aligned to a certain emotional label throughout the entire process before recognition. however, the generation of emotions is often spontaneous and intuitive so the cognitive system is fundamentally uncertain and in an indefinite state. in quantumlike frameworks, the emotion of each modality is treated as an indefinite state @xcite . the final quantum measurement creates a definite state and changes the state of the system. despite the above advantages compared to previous models, it is also challenging to complete the mer task in a quantum-like framework due to the complex processes such as feature extraction and modal fusion. technically, we must ensure that the model conforms to the evolution process of the quantum system and that the characteristics of the density matrix remain unchanged. inspired by the excellent performance of quantum-like networks in other tasks @xcite @xcite , we propose an adaptive-priority-learning model (qap) for mer. qap uses quantum states instead of traditional feature vectors to represent modal features from the initial feature extraction step to the final emotion classification step. modal features in each step no longer correspond solely to the final emotional label, but are in a state where emotion is uncertain. in this way, the opposite emotion of a single modality will not affect the final emotion because all modalities are in an uncertain state with multiple emotions. in mer, it is an inherent problem to effectively extract the features of the raw modalities. previous works either use pre-extracted features with handcrafted algorithms or extract end-to-end features with pre-trained models. but these two features are not effectively combined together. in qap, the complex-valued density matrix is used as the unit of modal representation due to the stronger representation ability @xcite . by this means, end-to-end features and pre-extracted features are effectively combined by a non-linear method. for the fusion in the quantum-like framework, q-attention based on the density matrix is designed to orderly integrate the three modalities. after that, since three modalities can form several fusion orders, we use a quantum measurement operator to select the most appropriate fusion order. in this way, qap can learn modal priority adaptively. finally, we use another quantum measurement operator to collapse all states in the density matrix to the pure state representing emotion to recognize the emotion. the main contributions of our paper are as follows: besides, integrating three modalities in a quantumlike framework is a challenging task, and we make an initial attempt in this field to make the modalities with opposite emotions be integrated effectively. ## preliminaries on quantum theory the construction of a quantum-inspired model is based on quantum theory (qt) @xcite . in this section, we will briefly introduce the basic concepts of qt. the state vector in qt is defined on a hilbert space h, which is an infinite inner product space over the complex field. with dirac notations, we denote a complex unit vector µ as a ket |u , and its conjugate transpose µ h is denoted as a bra u|. the inner product and outer product of two state vectors |u and |v are denoted as u|v and |u v|. ## model in this section, we will describe the details of qap. the overall architecture of qap is shown in figure 2. qap consists of three modules: unimodal complex-valued representation, adaptive priority learning and emotion recognition. firstly, the complex density matrix of the single modality is constructed for modal representation(section 4.1). in the representation, end-to-end features and preextracted features are respectively aligned to the amplitude and the phase of the complex value. secondly, q-attention is designed to integrate three modalities orderly and we use a quantum measurement operator to select the appropriate order, then qap can learn modal priority adaptively (section 4.2). finally, another measurement operator is employed to recognize the final emotion (4.3). ## conclusion we propose qap, a quantum-inspired adaptivepriority-learning model for multimodal emotion recognition. first, the quantum state is introduced to model the uncertainty of human emotion, which allows modalities with inconsistent emotions can be effectively integrated. secondly, a novel mechanism q-attention is designed to orderly integrate three modalities in a quantum-like framework. while selecting the appropriate fusion order, qap learns modal priority adaptively. in this way, modalities make varying degrees of contributions based on priority. experiments on two widely used datasets show that qap establishes the new sota.