Q e MMA : Quantum-Enhanced Multi-Modal Sentiment Analysis (2023)

Abstract: Multi-modal data analysis presents formidable challenges, as developing effective methods to capture correlations among different modalities remains an ongoing pursuit. In this study, we address multi-modal sentiment analysis through a novel quantum perspective. We propose that quantum principles, such as superposition, entanglement, and interference, offer a more comprehensive framework for capturing not only the cross-modal interactions between text, acoustics, and visuals but also the intricate relations within each modality. To empirically evaluate our approach, we employ the CMUMOSEI dataset as our testbed and utilize Qiskit by IBM to run our experiments on a quantum computer. Our proposed Quantum-Enhanced Multi-Modal Analysis Framework (QeMMA) showcases its significant potential by surpassing the baseline by 3.52% and 10.14% in terms of accuracy and F1 score, respectively, highlighting the promise of quantum-inspired methodologies.

## introduction sentiments play a vital in our daily lives, serving as crucial elements in communication, decisionmaking, situational awareness, and learning. the commercial value of user-generated service and product reviews is substantial, as they aid fellow users in their decision-making processes, such as when purchasing a new product. moreover, these reviews prove invaluable to businesses, offering insights into product monitoring, enhancing customer relationships, crafting more effective marketing strategies, and refining services. traditional systems typically rely on a single modality, text @xcite @xcite , or other source @xcite to gauge a person's sentiment or emotional state. conversely, multi-modal sentiment analysis seeks to empower machines with a deeper understanding of sentiment by leveraging information from multiple modalities, including text, visuals, and acoustics @xcite . video content, in particular, serves as a rich source for extracting multi-modal information. in addition to visual frames, videos provide acoustic and textual imitation of verbal language. furthermore, a single video may contain multiple utterances from a single speaker, each potentially expressing different sentiments. thus, the sentiment polarity of one statement often hinges on contextual utterances, creating intricate interdependencies. prior studies in multi-modal sentiment analysis have explored various approaches, such as applying attention mechanisms to contextual utterances for classification @xcite , computing correlations among modalities in target and contextual utterances @xcite , and dynamically adjusting word representations based on nonverbal cues via lstms @xcite . however, these methods are computationally intensive, imposing significant overhead for only marginal gains in accuracy. in this paper, we address this challenge by introducing quantum machine learning. our novel approach combines a variational quantum circuit (vqc) @xcite with a recurrent neural network-based multi-modal multi-utterance attention architecture for sentiment analysis. we propose the following: quantum superposition (qs): expressing intra-modal affinities: qs, where a quantum state simultaneously occupies numerous mutually incompatible ground states with a probability distribution until measured, offers an apt analogy for multi-modal tasks. we hypothesize that an utterance can be viewed as a qs of various text-to-text, audio-to-audio, or visual-to-visual instances, yielding complex probability amplitudes suitable for quantum probability theory. ## related work quantum natural language processing (qnlp): in qnlp, two primary perspectives have emerged. first, there is a theoretical application of quantum probability to address classification @xcite and generation tasks @xcite in classical computing systems. second, there is a practical implementation angle, wherein natural language tasks are executed on quantum simulators @xcite and actual quantum computers @xcite . of note, the categorical distributional compositional (discocat) model for natural language, leveraging quantum instantiable algorithms, has gained prominence due to the growing availability of quantum resources. in a recent study conducted by @xcite , the discocat model was scrutinized for sentence classification. they instantiated sentences and word meanings as parameterized quantum circuits, with quantum entanglement being used to manifest the grammatical structure. additionally, @xcite delved into the mapping of sentence representations into quantum circuits, exploring syntax-independent models like the "bagof-words" paradigm in the quantum realm. researchers have also endeavored to theoretically establish quantum probability as an effective means of fusing different modalities @xcite . sentiment analysis: multi-modal sentiment prediction has gained popularity in recent years, with researchers experimenting with various methodologies for merging distinct modalities. hybrid fusion methods, which concurrently scrutinize featurelevel and decision-level attributes, have garnered attention @xcite . tensor fusion approaches, exemplified by t2fn @xcite , and mtfn-cmm @xcite involve the construction of a 3-fold cartesian product through the utilization of modality embeddings. these methods explicitly model unimodal, bimodal, and trimodal interactions using a tensor fusion layer. for word-level fusion, lstm-based gated modality mixing networks, such as rmfn @xcite , and raven @xcite rely on bidirectional recurrent neural networks. meanwhile, multi-head attention-based networks like bimha @xcite and mmha @xcite extract contextual features using a bidirectional recurrent neural network-based model. ## proposed methodology to obtain quantum representations of text, visual, and acoustic inputs, their dimension needs to be reduced due to constraints in qubit availability on to ensure a transparent and equitable comparison, we adopt the mmmu-ba model architecture proposed by @xcite . their framework is designed to process multi-modal information, encompassing text, visual, and acoustic data across a sequence of utterances. this information is channeled into three distinct bi-gru layers. subsequently, a dense operation is applied uniformly across the time steps, with dedicated layers for each modality. following this, they apply multi-modal attention to the outputs of these dense layers. their approach involves implementing a bi-modal attention framework, where an attention mechanism is employed to weigh the contributions of pairwise modalities, namely text-acoustic, visual-text, and acoustic-visual. the results from these pairwise attention mechanisms, combined with modality representations, are concatenated and added to the vqc output. this combined representation is then passed through a softmax layer for classification. we illustrate the model architecture in figure 2 . the execution of our qemma model is carried out using the qasm_simulator from the aer backend and the ibmq_lagos quantum computer via the qiskit library. ## conclusion we introduced a novel vqc to the mmmu-ba baseline, leveraging quantum properties (superposition, entanglement, and interference). this enhanced performance by just 309 parameters, highlighting the efficiency of our quantum approach. furthermore, it is worth noting that the choice of the specific quantum circuit itself is a hyperparameter, suggesting potential optimization for specific tasks. in the future, we aim to refine our vqc and architecture to enhance multi-modal sentiment analysis with quantum capabilities.