What Does Your Smile Mean? Jointly Detecting Multi-Modal Sarcasm and Sentiment Using Quantum Probability (2021)

Abstract: Sarcasm and sentiment embody intrinsic uncertainty of human cognition, making joint detection of multi-modal sarcasm and sentiment a challenging task. In view of the advantages of quantum probability (QP) in modeling such uncertainty, this paper explores the potential of QP as a mathematical framework and proposes a QP driven multi-task (QPM) learning framework. The QPM framework involves a complex-valued multi-modal representation encoder, a quantum-like fusion subnetwork and a quantum measurement mechanism. Each multi-modal (e.g., textual, visual) utterance is first encoded as a quantum superposition of a set of basis terms using a complex-valued representation. Then, the quantum-like fusion subnetwork leverages quantum state composition and quantum interference to model the contextual interaction between adjacent utterances and the correlations across modalities respectively. Finally, quantum incompatible measurements are performed on the multi-modal representation of each utterance to yield the probabilistic outcomes of sarcasm and sentiment recognition. The experimental results show that our model achieves a state-of-the-art performance.

## introduction multi-modal sarcasm and sentiment analysis, as a challenging problem, has attracted an increasing attention in the recent literature @xcite . sarcasm is a subtle form of human language that intends to express criticism, humor or mock sentiments by means of hyperbole, figuration, etc @xcite . the literal meaning of an ironic expression differs from its real implication, which can completely flip the polarity of sentiment. hence, sentiment comes into view and tightly couples with sarcasm in that one helps the * yazhou zhang and yaochen liu contribute equally and share the co-first authorship. ## quantum probability preliminaries quantum superposition and density matrix. the mathematical base of quantum probability is established on a complex hilbert space, denoted as h. a quantum state vector u is expressed as a ket |u , its transpose is expressed as a bra u|. the inner product and outer product of two state vectors |u and |v are denoted as u|v and |u v|. quantum superposition states that a pure quantum state can be in multiple mutually exclusive basis states simultaneously, with a probability distribution until it is measured. a quantum mixture of states gives rise to a mixed state represented by a density matrix, ρ = i p i |u u|, where p i denotes the probability distribution of each pure state. quantum interference. in the double-slit experiment, two paths interfering with each other affects the probability distribution of the particle reaching the final position of the detection screen. we use the wave function ϕ(x) to interpret this behavior. the wave function represents the probability amplitude of a particle be at a position x, and the square of the wave function represents the possibility. the state of the photon is in a quantum superposition of the state of path 1 and path2, which is formulated as: ϕ p (x) = αϕ 1 (x) + βϕ 2 (x), where ϕ 1 (x) and ϕ 2 (x) are the wave function of path1 and path2. α and β are complex numbers. its probability is: quantum measurement. quantum measurement is described by a set of measurement operators acting on the state space of the system being measured {m m }, where m represents the possible measurement outcomes. suppose the quantum system is in a state of |u , then the probability to obtain the outcome m after the measurement is p (m) = u|m † m m m |u . the gleason's theorem @xcite has proven the existence of a mapping function m (|u u|) = tr (ρ|u u|) for any event |u u|. ## theoretical justification of the proposed qpm framework based on the general qp and a few previous studies @xcite , this section proposes theoretical justification of our qpm framework in the form of four claims. claim 1 quantum probability is more general to capture the uncertainty in human language. assume z (x) represents a complex probability amplitude of an event x, where z (x) = re iθ . qp defines the modulus square of this complex probability amplitude to represent a classical probability p (x) = |z (x)| 2 = r 2 . it defines a many-to-one relationship between complex probability amplitude and probability. for example, the probability of a word w is 0.5, i.e., p (@xmath0, then the corresponding probability amplitude may be z ( ## conclusions we have proposed a quantum probability driven multi-task learning framework. the main idea is to treat each utterance as a complex-valued vector. the contextual interaction between utterances and the correlations across modalities are modeled via quantum composition and quantum interference. quantum incompatible measurement is performed to yield the probabilistic outcomes. the experimental results verify the effectiveness of the qpm.