Neuro-Symbolic Sentiment Analysis with Dynamic Word Sense Disambiguation (2023)

Abstract: Sentiment analysis is a task that highly depends on the understanding of word senses. Traditional neural network models are black boxes that represent word senses as vectors that are uninterpretable for humans. On the other hand, the application of Word Sense Disambiguation (WSD) systems in downstream tasks poses challenges regarding i) which words need to be disambiguated, and ii) how to model explicit word senses into easily understandable terms for a downstream model. This work proposes a neurosymbolic framework that incorporates WSD by identifying and paraphrasing ambiguous words to improve the accuracy of sentiment predictions. The framework allows us to understand which words are paraphrased into which semantically unequivocal words, thus enabling a downstream task model to gain both accuracy and interpretability. To better fine-tune a lexical substitution model for WSD on a downstream task without ground-truth word sense labels, we leverage dynamic rewarding to jointly train sentiment analysis and lexical substitution models. Our framework proves to effectively improve the performance of sentiment analysis on corpora from different domains.

## introduction to improve the accuracy of sentiment analysis, it is crucial to disambiguate polysemous words according to the context @xcite . a word may have senses that carry opposite sentiments, e.g., "fine" has the positive meaning of being delicate and refined, and the negative meaning of monetary penalty. additionally, words may appear in positive, negative, or neutral senses in different domains. for instance, the word "bearish" is often used in a negative sense to describe the stock market in the financial domain, while more likely to be used neutrally to describe hulking figures in other domains. in traditional black-box sentiment analysis, the representations of word meanings are generated by neural networks @xcite . as such, the disambiguation process is implicit and opaque. it is challenging to investigate what implicit senses are assigned to ambiguous words by a neural network model for it to make accurate predictions. coupling symbolism and sub-symbolism, a neurosymbolic approach to sentiment analysis is transparent and interpretable, regarding the extraction and distinction of word meaning as a sub-problem @xcite . however, research that explicitly deals with ambiguous words in sentiment analysis is limited @xcite @xcite . sense-annotated resources such as sentiwordnet @xcite have been utilized, but only to detect and score sentimental words in the input @xcite @xcite . the disambiguation of polysemous words in sentiment analysis inputs remains under-explored. thus, we are motivated to develop a neurosymbolic framework for sentiment analysis with explicit word sense disambiguation (wsd). traditional wsd aims to classify which sense of the target word is appropriate according to its context @xcite @xcite @xcite . however, the limitations (ℓ) of their task setups are that (ℓ1) the output of those wsd systems, whether sense label @xcite or gloss definition @xcite , is difficult to integrate into downstream input; (ℓ2) the target words must be pre-defined for wsd systems to disambiguate, making it difficult to apply them directly to downstream tasks; and (ℓ3) the reliability of sense distinction of the benchmark has been questioned @xcite , as word senses are not discrete. attempts have been made to incorporate the disambiguation of word senses into downstream applications. nonetheless, they are either impractical in real-world settings @xcite , or only applicable to a specific downstream task, i.e., machine translation @xcite . to address these issues, we leverage dynamic wsd in our neurosymbolic framework. we regard wsd as a lexical substitution task (for ℓ1), where ambiguous words in sentiment analysis input are paraphrased into ones that cause less sentiment confusion using a lexical substitution model with dynamic wsd. our dynamic wsd method comprises two distinct aspects: the dynamic selection of the target word for disambiguation, based on downstream contextual information (for ℓ2); and the dynamic learning of the most suitable lexical substitutions, based on the prediction accuracy of the downstream model (for ℓ3). to identify target words that need to be substituted, we first adopt an attention-based explainable encoder to rank input words by the level of influence they have on the output, motivated by the assumption that not all polysemous words affect sentiment predictions @xcite . then, the top-ranking words with the most diverse senses are selected to be targets, based on our hypothesis that an ambiguous word may coincide with significant disparities in the semantic space between the word itself and its potential substitution candidates. as such, we can trace back which words are selected as targets, and which substitute words are used to disambiguate them, making our method interpretable and transparent. furthermore, we finetune the lexical substitution model whilst training the sentiment analysis model by adopting a dynamic rewarding mechanism that does not require gold standard substitution labels. as such, the substitution model can be adjusted to provide more appropriate output for the downstream task as well as the target domain. our model achieves more accurate sentiment analysis with explicit lexicalsubstitution-based wsd, despite not specifically trained on wsd and sentiment parallel datasets. since the disambiguated word senses are represented as the lexical substitutions of the original words and an explainable encoder is used for retrieving the most contributing words, the neurosymbolic sentiment inference process is more interpretable for humans compared to black-box models. our model is examined with three publicly available datasets from movie review, finance, and twitter domains, outperforming strong baselines by 1.23% on average. the contributions of our work are twofold: (1) we propose a neurosymbolic sentiment analysis framework that leverages wsd as an auxiliary lexical substitution task, achieving stateof-the-art performance on our examined sentiment analysis datasets from different domains; (2) we design a transparent mechanism to select wsd target words, based on a) the importance of words to the learning and inference of a downstream task, and b) word sense diversity. ## related work word sense disambiguation is a task that assigns the appropriate meaning to an ambiguous word according to its context. recent wsd systems attained f1 scores higher than 80% on the benchmark datasets @xcite @xcite , which is regarded as the estimated human performance since it is the highest inter-annotator agreement @xcite . however, research on the application of wsd systems on downstream tasks is limited. @xcite utilized clustering algorithms to obtain sense embeddings by inferring from the similarity of the context vectors of a target word. the heavy requirement for disambiguating every homographs prior to downstream applications makes their method less feasible in many real-world situations. @xcite achieved the disambiguation of word sense in neural machine translation by leveraging a multilingual wsd system to establish a sensetagged parallel corpus. nonetheless, their approach relied on cross-lingual word alignments for reliable annotations, meaning that it cannot be adopted by other types of downstream tasks. sentiment analysis can be generally defined as the task that determines if a piece of text expresses positive, negative, or neutral opinion @xcite . word sense information has been applied in sentiment analysis in the form of sentimentannotated lexicons @xcite @xcite , such as sentiwordnet. however, these methods incorporate the sentiment information of a word without disambiguating its word sense according to the context, but resorting to the emotional score of the most frequent sense, or the average score of all senses. additionally, the main objective of this approach is to identify sentimental words in a sentence, instead of disambiguating polysemous words that might affect model predictions. hung and chen (2016); @xcite ; @xcite attempted to disambiguate according to target domain and context, but either costed extensive human labour, or fail to incorporate word senses explicitly and transparently. to address these limitations, our neurosymbolic framework aims to identify ambiguous target words in the context and replace them with clarifying ones automatically, making them interpretable. we also leverage a dynamic rewarding mechanism so that the substitutions can be fine-tuned to better serve the downstream task and the text domain. ## task definition in this work, we regard word sense ambiguity as the potential cause of errors in a downstream task, e.g., sentiment analysis. inspired by the work of @xcite , we reframe wsd as a lexical substitution task, aiming to replace an ambiguous word with a synonym that not only represents the true sense of the original word in the context @xcite , but also reduces misclassifications in the downstream task. given an input sentence @xmath0, w 2 , . . . , w l ), we first aim to identify the target word t that is likely troublesome for the downstream prediction from w. to resolve the potential ambiguity, we extract m of its synonyms from wordnet @xcite as candidates @xmath1, . . . , c m ), forming the input (s) as @xmath2, w1, w2, . . . , t, . . . , wl, 
, c1, 
, c2, 
, . . . , cm , 
. ## methodology our proposed neurosymbolic framework can be viewed in figure 1 . there are two models in our framework, namely sentiment analysis and lexical substitution models. the sentiment analysis model consists of an encoder, an interpretable attention module, and multiple feedforward neural networks. first, through figure 1 (1) and ( 2 ), the original input is fed into the encoder and the interpretable attention module to obtain the top i tokens with the highest attention weights, which contribute the most to sentiment prediction. in (3), the sense diversity of each token is computed as the average distance of the token's hidden state to those of its substitution candidates. then, through (4), top j tokens with the most diverse senses are selected as disambiguation targets, because these target tokens may have very different senses in different contexts. subsequently, the pretrained lexical substitution model is applied to provide the best substitution to replace each target token, as shown in ( 5 ) and ( 6 )a. the new input sentence is passed onto the sentiment analysis model for final prediction ((7)a and (8)a), which is used for the sentiment model backpropagation ((9)a). in order to fine-tune the lexical substitution model to provide better substitutions that improve the accuracy of sentiment analysis, we adapt a dynamic rewarding mechanism @xcite that was proposed for multi-task learning. as shown in figure 1 (5) and ( 6 )b, for each target word, we choose the top n candidates. each of them is seen as a substitute candidate to calculate the probability of correct sentiment prediction after the replacement ((7)b and (8)b). then, each probability is used to dynamically compute the loss weight when the corresponding candidate is learned by the lexical substitution model as a ground truth ((9)b and (10)b). if the replacement by a candidate leads to a higher chance of predicting the right sentiment, the candidate is more likely given a higher rank by the lexical substitution model. otherwise, the model would be less likely to select the candidate as the best replacement. the important notations used in this paper are summarized in ## results we compare our model with the baselines in our framework consistently demonstrates improvements across all datasets, substantiating its ability to achieve higher gains in accuracy while minimizing any potential loss. furthermore, we can interpret the wsd outcomes and the sentiment analysis decision-making process before and after lexical substitutions from the proposed neurosymbolic framework. we will demonstrate the interpretability of the framework in section 6.2. ## conclusion in this work, we propose a neurosymbolic framework for sentiment analysis that incorporates the disambiguation of word senses by identifying and paraphrasing ambiguous words in the input, which is realized by leveraging a pre-trained lexical substitution model. additionally, to fine-tune the lexical substitution model to provide better disambiguating substitutes for sentiment analysis with no ground-truth word sense labels, we adopt the dynamic rewarding mechanism to jointly train the sentiment analysis and lexical substitution models. experiments show that our framework is indeed effective in improving the performance of sentiment analysis, obtaining state-of-the-art performance on three corpora from different domains.