S entic V ec: Toward Robust and Human-Centric Neurosymbolic Sentiment Analysis (2024)

Abstract: The success of state-of-the-art Natural Language Processing (NLP) systems heavily depends on deep neural networks, which excel in various tasks through strong data fitting and latent feature modeling abilities. However, certain challenges linked to deep neural networks and supervised deep learning deserve considerations, e.g., extensive computing resources, knowledge forgetting, etc. Previous research attempted to tackle these challenges individually through irrelative techniques. However, they do not instigate fundamental shifts in the learning paradigm. In this work, we propose a novel neurosymbolic method for sentiment analysis to tackle these issues. We also propose a novel sentiment-pragmatic knowledge base that places emphasis on human subjectivity within varying domain annotations. We conducted extensive experiments to show that our neurosymbolic framework for sentiment analysis stands out for its lightweight nature, robustness across domains and languages, efficient few-shot training, and rapid convergence.

## introduction deep neural networks have demonstrated remarkable capabilities in the natural language processing (nlp) domain because of their strong capacity to capture latent features and fit data. currently, the advancement of large language models (llms) shows that when the neural networks become very large and deep, sufficient pre-training grants them impressive capability, making them a staple as foundation models for a variety of tasks @xcite . despite the fascination the academic community holds for llms @xcite , a pressing question remains: is the relentless pursuit of model enlargement truly the paramount focus for nlp? as models become larger and deeper, many machine learning problems arise. for instance, larger models normally require more computing resources to train, resulting in more carbon emission @xcite . traditional supervised learning, one of the most significant deep learning paradigms, suffers from issues including knowledge forgetting @xcite , extensive data annotation efforts @xcite , weak cross-domain @xcite and cross-lingual @xcite adaption, and data-centric learning pipelines @xcite . for nlp, the need for deep neural networks is usually due to the complexity of modeling language, i.e., syntax @xcite and semantics @xcite exhibiting different patterns in different contexts. as such, the learning of complex word sequence patterns is achieved by making the model deeper and adding different learning mechanisms @xcite . furthermore, models trained with standard annotation sets where labels were aggregated based on the majority opinion of annotators could potentially disregard nuanced individual variations @xcite . in subjective tasks such as affective computing, a uniform decision-making mechanism may struggle to adequately cater to individual subjective experiences. the judgment of sentiment and emotional states from different people can vary in different contexts or domains, necessitating humancentric systems. in this research, we investigate whether the coupling of syntactic and pragmatic features can alleviate the need for semantic features, thereby significantly reducing the number of model parameters needed to retain semantic knowledge @xcite . hence, such an approach potentially tackles the challenges prevalent in deep neural networks, including heavy reliance on extensive downstream training data and computing resources @xcite , slow training rates, limited adaptability across domains and languages, knowledge forgetting, and the ignorance of humans' subjectivity. previous sentiment analysis works target to address one of these issues with different techniques, e.g., few-shot learning for eliminating the hungry of label data @xcite ; pruning neural networks or using lightweight encoders to reduce the number of learning parameters and computing resources @xcite ; using pre-training @xcite to improve the generalization of a model in different domains and languages; leveraging continues learning techniques to mitigate the knowledge forgetting @xcite . however, these approaches do not instigate fundamental shifts in the deep learning paradigm, nor do they comprehensively resolve all the previously mentioned issues. to alleviate the reliance on large neural networks and downstream training data while improving model performance in cross-domain and cross-language tasks, we propose a neurosymbolic method and a knowledge base for sentiment analysis foot_0 . our hypothesis is that in affective computing, models can reduce the need for semantic learning by learning pragmatic knowledge and syntactic patterns, thereby reducing the number of learning parameters. and mitigating issues associated with deep neural networks. the advantage is threefold: a) the exclusion of semantic learning substantially reduces the parameters of neural networks as it obviates the necessity for the model to grasp the meaning of an extensive vocabulary across diverse contexts. b) incorporating multi-dimensional symbolic representations (sentiment-pragmatic knowledge) from our knowledge base enables the construction of a humancentric system that circumvents the knowledgeforgetting issue, where different dimensions signify subjective judgments from different contributors, offering the flexibility to augment the symbolic representations with new dimensions acquired from new data without changing the existing elements. c) by leveraging the combination of syntactic and pragmatic features as input, it is possible to alleviate issues related to learning and inference across diverse domains and languages, primarily stemming from semantic variations in different linguistic environments. this aligns with our plans to enhance current sub-symbolic ai approaches through task decomposition, panalogy, symbol grounding, and more @xcite . our neurosymbolic framework for sentiment analysis consists of two components, a knowledge base and a neurosymbolic model. the former, termed senticvec, consists of sentiment-pragmatic representations for 388,158 lexical units, i.e., tokens. each representation is in vector form, where each element is a sentiment score ranging from -1 (negative) to 1 (positive), pre-trained from a public sentiment dataset. the sentiment score is automatically generated via an explainable encoder @xcite that signifies the contribution of each lexical unit to the sentiment prediction. lexical units that contribute more will receive more extreme scores. the sentiment vector of a lexical unit consists of multiple sentiment scores, learned and computed using multiple labeled sentiment datasets. the intuition is that different annotators, annotation task instructions, and text domains can yield different annotation outcomes. for a human-centric system with cross-domain robustness, it is essential to acknowledge these sentimental variations; therefore, we opt for multidimensional representations, integrating subjectivity instead of relying on conventional unified sentiment scores to depict sentiment-pragmatic features. our neurosymbolic model consists of a 3-layer multi-layer perceptron (mlp) to learn the pragmatic representations and pos patterns. we use an existing pos tagger to generate pos tag sequences from the inputs. the motivation is that 1) the size of pos tags is smaller than the vocabulary size of words; 2) different languages can share the same pos tag set; 3) senticvec provides useful sentiment-pragmatic representations and can be easily mapped to other languages using bilingual dictionaries. thus, we can use a shallow neural network by leveraging symbolic representations (e.g., pos tags and senticvec) to process multilingual sentiment analysis tasks. this is entirely different from previous learning paradigms that were proposed for addressing the aforementioned deep learning issues. we performed thorough experiments to demonstrate the superiority of our approach in conventional supervised learning, cross-domain inference, and multilingual learning and inference for sentiment analysis. our method achieves comparable performance to robertabase @xcite while requiring 1/7723 of its training parameters, and enables faster training in the conventional supervised learning evaluation. in cross-domain inference evaluation, the sen-ticvec neurosymbolic model demonstrates overall better performance than roberta. in the fewshot multilingual learning evaluation, senticvec surpasses multilingual pre-trained language models (plms) on three languages. in cross-lingual inference evaluation, senticvec obtains comparable performance to plms. furthermore, incorporating senticvec knowledge into multilingual plm consistently brings accuracy gains in all settings. the contributions of this work include: (1) we develop a novel knowledge base, incorporating sentiment-pragmatic representations that prioritize human subjectivity in annotating diverse corpora from varying domains while ensuring retention of original knowledge with the influx of new labeled data. (2) our novel neurosymbolic framework for sentiment analysis stands out for its lightweight nature, robustness across domains and languages, efficient few-shot training, and rapid convergence. ## results we evaluate our neurosymobic method in conventional supervised learning and cross-domain inference (section 5.1); cross-lingual inference (section 5.2) and multilingual learning (section 5.3). our ablation analysis (appendix a) verifies that as the dimensionality of senticvec increases, our model can achieve better accuracy. we further compare the convergence curves of our method and the baselines (appendix b), demonstrating the training speed advantage inherent in our method. ## conclusion we propose a novel neurosymbolic model and a sentiment-pragmatic knowledge base, senticvec. senticvec consists of multi-dimensional sentimentpragmatic representations, making it humancentric and cross-domain efficient. we propose to use pos tags together with senticvec for sentiment analysis. the streamlined pos vocabulary as input markedly reduces the need for deep neural networks and overcomes language barriers. consequently, employing a shallow neural network for sentiment analysis in multilingual learning and cross-lingual inference scenarios becomes viable, substantially decreasing computational demands. the proposed framework can be adapted to other text classification tasks with multiple labeled datasets, e.g., emotion detection @xcite , depression detection @xcite , and topic classification @xcite , which will be examined in our future works.