SEMG raph: Incorporating Sentiment Knowledge and Eye Movement into Graph Model for Sentiment Analysis (2022)

Abstract: This paper investigates the sentiment analysis task from a novel perspective by incorporating sentiment knowledge and eye movement into a graph architecture, aiming to draw the eye movement-based sentiment relationships for learning the sentiment expression of the context. To be specific, we first explore a linguistic probing eye movement paradigm to extract eye movement features based on the close relationship between linguistic features and the early and late processes of human reading behavior. Furthermore, to derive eye movement features with sentiment concepts, we devise a novel weighting strategy to integrate sentiment scores extracted from affective commonsense knowledge into eye movement features, called sentiment-eye movement weights. Then, the sentiment-eye movement weights are exploited to build the sentiment-eye movement guided graph (SEMGraph) model, so as to model the intricate sentiment relationships in the context. Experimental results on two sentiment analysis datasets with eye movement signals and three sentiment analysis datasets without eye movement signals show that the proposed SEMGraph achieves state-of-the-art performance, and can also be directly generalized to those sentiment analysis datasets without eye movement signals.

## introduction as one of the hot research directions in natural language processing (nlp), sentiment analysis (sa), aiming to extract knowledge and analyze the sentiment of texts, is crucial in a wide range of applications across multiple domains such as customer opinion identification @xcite , product recommendation @xcite , and mental health monitoring based on patients' social media posts @xcite , etc. some pioneering research efforts in cognitive science affirm the association between cognitive * corresponding authors processing and emotional manifestations @xcite . intuitively, humans are quite apt in extracting subjective content and emphases from texts when reading a context. based on the intuition that the cognitive data which covers eye movement signals or electroencephalography (eeg) signals might bring substantial improvements to sentiment analysis, recent research on sentiment analysis relying on eye movement signals has been attached importance @xcite @xcite @xcite . despite promising progress made by existing eye movement-based sentiment analysis studies, they largely rely on the data containing eye movement signals. in the real-world scenario, however, the acquisition of eye movement signals is expensive, and many commonly used sentiment analysis datasets lack eye movement signals. in addition, existing eye movement-based sentiment analysis work ignores the role of sentiment concepts in the learning of eye movement signals. therefore, in this paper, we propose a novel framework to incorporate sentiment knowledge and eye movement, so as to derive the sentiment-eye movement weights. then, the sentiment-eye movement weights are exploited to build the sentimenteye movement guided graph (semgraph) model. the most significant thing to note here is that the semgraph model can not only tackle the sentiment analysis of the data with eye movement signals, but also be directly generalized to deal with sentiment analysis of the data without eye movement signals. to be specific, we first propose a linguistic probing eye movement paradigm to extract eye movement features from linguistic features. motivated by @xcite and @xcite about the early and late processes of human reading behavior, we establish relationships from two perspectives based on a regression model by using eye movement features as dependent variables and the linguistic features as independent variables. one is the relationship of two linguistic features (character and word features) and first fixation duration (ffd), the other is three linguistic features (character, word, and complexity feature illustrated in section 3.2) and total reading time (trt). furthermore, leveraging sentiment concepts is a key to improving the learning of sentiment analysis @xcite . therefore, we extract the sentiment concepts from an affective commonsense knowledge @xcite , and then devise a novel weighting strategy to integrate the sentiment concepts into eye movement features, so as to derive sentiment-eye movement weights including ffd weight and trt weight. based on it, to model the intricate sentiment and eye movement relationships between contextual words, we incorporate the sentiment-eye movement weights into a gated graph neural network (ggnn) architecture to build a sentiment-eye movement guided graph (semgraph) model. more concretely, considering the close relationships between ffd and word features, we adopt ggnn to study the embeddings of words, and add ffd weight for better information interaction. meanwhile, according to the connection of trt and more complex linguistic features, after obtaining the word embedding by ggnn, soft attention weight and trt weight are combined to enhance graph-level information aggregation for learning the sentiment cues. to sum up, the contributions of this work can be summarized as follows: ## method in this section, we describe our proposed sentimenteye movement guided graph neural network (sem-graph) in detail. as illustrated in figure 1 , our semgraph comprises five components: 1) embedding module, which utilizes bert to create embeddings of the vertices. 2) eye movement feature extraction, which derives eye movement features from the text by the linguistic probing eye movement paradigm. 3) sentiment-eye movement feature incorporation, which derives sentimenteye movement weights by incorporating sentiment knowledge and eye movement features. 4) graph-based information interaction, which adopts ggnn and injects ffd weight to implement word interaction. 5) attention-based readout function, which designs an attention layer consisting of soft attention and trt weight to obtain a graph-level representation for sentiment prediction. ## conclusion in the current work, we investigate the sa task from a novel perspective by incorporating sentiment knowledge and eye movement into a graph architecture. moreover, a linguistic probing eye