The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis (2023)

Abstract: We conduct an inquiry into the sociotechnical aspects of sentiment analysis (SA) by critically examining 189 peer-reviewed papers on their applications, models, and datasets. Our investigation stems from the recognition that SA has become an integral component of diverse sociotechnical systems, exerting influence on both social and technical users. By delving into sociological and technological literature on sentiment, we unveil distinct conceptualizations of this term in domains such as finance, government, and medicine. Our study exposes a lack of explicit definitions and frameworks for characterizing sentiment, resulting in potential challenges and biases. To tackle this issue, we propose an ethics sheet encompassing critical inquiries to guide practitioners in ensuring equitable utilization of SA. Our findings underscore the significance of adopting an interdisciplinary approach to defining sentiment in SA and offer a pragmatic solution for its implementation.

## introduction sentiment analysis (sa) has emerged as a significant research focus in natural language processing (nlp) over the last decade. it has now become an indispensable tool in discerning opinions and emotions in written text @xcite , evaluating social entities' reputation @xcite , analyzing and predicting financial needs @xcite , and aiding in effective political decision-making @xcite . this is illustrated in figure 1 which shows the rising numbers of peer-reviewed articles on sentiment analysis published in scopus every year. existing research reveals a notable absence of interdisciplinary endeavors to comprehend the social dimensions of sa, encompassing aspects like emotion and fairness @xcite ; blodgett * authors contributed equally figure 1 : number of articles published each year (from 2010 to 2022) in scopus that contain the term 'sentiment analysis' in the title, abstract, or keywords. @xcite . this lack of collaborative thinking has resulted in flawed analyses and biased outcomes. given the extensive range of applications of sa spanning diverse domains such as healthcare, finance, and policymaking, it is crucial to avoid replicating such tendencies. furthermore, sa, despite addressing social constructs like emotion, subjectivity, and opinion, has been limited in its incorporation of psychological and sociological definitions of sentiment @xcite . while numerous studies have examined the utilization of sa, encompassing its inherent challenges and future directions @xcite , the interdisciplinary and sociotechnical dimensions of sa have received limited exploration. to this end, we explore this gap in the literature by examining sentiment through a technical perspective concentrating on the evolution of sa into a social system. we then evaluate sentiment, examining the various definitions of sentiment through a sociotechnical lens. we also investigate the application of sa, presenting insights into its utilization. these investigations will shed light on the interdisciplinary divide of the term sentiment. next, we evaluate the motivation behind establishing necessary frameworks for measuring sentiment by examining various different sa models and datasets. through our critical survey of 189 unique works in sa (as shown in ## a survey of surveys we now chronologically analyzing various surveys published in the field of nlp. @xcite surveyed 54 articles and categorized them based on utility. they showed that sa was synonymous with opinion mining and subjective analysis, and was primarily utilized to analyze product reviews. similarly, @xcite presented a summary of sa, stating it to have evolved into a sociotechnical system @xcite often used in the fields of politics, public actions, and finance. further, ribeiro et al. ( 2016 ) reviewed sa models and benchmarked a comparison of 24 sa models. they found that most models were developed to measure sentiment in social posts, product reviews, and texts in news articles. however, the metrics of measurement varied considerably across datasets and models, highlighting the need for uniformity in the field of sa. with the advent of deep learning, more sa models were developed using deep learning architectures, as summarized by @xcite . the work demonstrated how similar architectures could now be used in applications such as emo-tion analysis, sarcasm analysis, and toxicity analysis. sánchez-rada and iglesias (2019) surveyed the social context of sentiment analysis, reviewing its applications, limitations, and utilities as a sociotechnical system. @xcite surveyed works on sa from 2014 to 2019 to understand its social utility. they found that most of the work in sa was used in interdisciplinary contexts related to world events, healthcare, politics, and business. recent surveys by @xcite ; guo et al. (2021); wankhade et al. (2022); zad et al. (2021) provide up-to-date perspectives on sa reflecting a shift towards fine-grained approaches, including deep learning and aspect-based sentiment analysis, enabling a more contextual understanding of sentiment. similarly, recent works by @xcite ; soni and rambola (2022) have specifically focused on aspect-based sentiment analysis and implicit aspect detection methods. overall, these surveys reflect a scoping of sentiment analysis to include people's sentiments, opinions, attitudes, evaluations, appraisals, and emotions towards services, products, individuals, organizations, issues, topics, events, and their attributes. however, none of these works discuss the interdisciplinary framework of emotion or sentiment. ## examination of sentiment we start by analyzing the various sentiment frameworks in sa and comparing them to existing social frameworks. by doing so, we aim to uncover the distinctions between the different notions of this term, shedding light on the gap between the technical and social aspects of sentiment. in this context, we define a sociotechnical system as a composite of social and technical components that collectively contribute to goal-oriented behavior, impacting both social and technical actors engaged with the system @xcite . throughout this work, we use the term 'framework' to denote a conceptual structure or set of principles that offer guidance for measuring or defining a specific concept within a study. ## critical analysis of sentiment analysis as shown in the previous sections, the sentiment framework employed in sa differs substantially from the established social frameworks of sentiment. this disparity can pose challenges when applying sa in sociotechnical systems @xcite . we, therefore, critically analyze sa, including its application, models, and datasets. our goal is to assess the suitability of sa in a sociotechnical system, which aims to address societal problems by integrating people and technology @xcite . the detailed roadmap of our survey is depicted in the appendix (figure 3 ). ## the weaknesses in sentiment analysis based on our survey analysis, we outline the key weaknesses encountered in sa within nlp. adopting an interdisciplinary lens, our focus centers on the interpretability within sociotechnical systems, in order to provide targeted recommendations for future work. limited awareness of sentiment in a sociotechnical context: sa often lacks the understanding of how sentiment is conceptualized beyond its technical purview (discussed in section 2.2). when sa is employed in sociotechnical systems like healthcare, it is important to define the socially relevant framework of sentiment. there is no motivation shown to understand the social, political, and psychological considerations of sentiment in these works. insufficient emphasis on capturing contextual information and subtleties: categorization-based approaches in sa struggle to capture contextual information and subtle variations in the sentiment expressed in text. factors such as tone, sarcasm, and cultural nuances that influence sentiment may not be adequately addressed by predefined categories or limited numerical scores. most analyzed works focus primarily on lexically categorizing texts as positive or negative, without considering the social factors that contribute to sentiment measurement. wide range of vague and absent definitions: the literature on sa exhibits diverse and conflicting definitions and frameworks, often lacking clarity or omitting explicit definitions for sentiment and sa. ambiguity arises from the use of terms like 'attitude,' 'tone,' 'subjectivity,' and 'tone' interchangeably, without clear definitions in the context of sentiment analysis. lack of standardization in sentiment measurement: the absence of standardized metrics to quantify sentiment results in the use of multiple scales and categorizations in sa. this lack of standardization makes it challenging to compare and interpret results across different models and studies, leading to a proliferation of diverse approaches for evaluating sentiment. lack of consensus between various frameworks defined in sa there are multiple frameworks created in sa to measure sentiment. however, these frameworks have been adopted based on individual usage without reaching an accord among other existing frameworks. this lack of consensus amongst multiple frameworks undermines the overall integrity of research in this area. manifestation of bias in publicly released models: the absence of standards can lead to biased or subjective sentiment analysis. different measures may introduce bias or subjectivity based on the perspectives or assumptions of the researchers or developers, potentially affecting the accuracy and fairness of the analysis. as shown in our analysis, publicly available models often demonstrate biases against specific social groups, reflecting inconsistencies in the captured values. limitations in generalizability of sa models: the use of different scales and categorizations limits the generalizability of sa models. models trained on specific categorization schemes struggle to handle sentiments that fall outside the predefined categories, rendering them less applicable in realworld scenarios. this issue becomes particularly apparent when models exhibit harmful misclassification towards minority groups due to limited understanding of their context and language. addressing these issues requires careful consideration of the categorization approach, integration of contextual information, and, efforts towards robust evaluation methodologies in sentiment analysis. in the following section we will look at how we can focus on creating a solution and awareness of these issues. ## recommendations and ethics sheet in prior works like blodgett et al. (2020), gebru et al. (2021) & bender and friedman (2018) have created data statements and ethics sheets as a means to audit and provide noteworthy indications to resolve issues in ai, through a list of meaningful ques-tions. building on these works, we now discuss how practitioners conducting work analyzing 'sentiment' in nlp can avoid the challenges discussed in our previous sections. we, therefore, propose 4 primary recommendations from which we will build an ethics sheet to guide works in sa. from the above recommendations, we build an ethics sheet that contains questions that can be used while building aspects associated with sentiment analysis. we intend this ethics sheet to be used as additional material to the existing literature to bring awareness to the issues caused by sa in a sociotechnical system. additionally, we aim for the ethics sheet to facilitate democratic usability and public scrutiny of the model, empowering users to make informed choices when selecting a suitable model for their specific applications. ## conclusion in our survey of 189 papers foot_3 on sa, we observe that, firstly, sa has shifted from analyzing product reviews to being widely used in sociotechnical systems like health and medicine. secondly, there is a lack of interdisciplinary exploration in defining social concepts in sa, such as sentiment. the frameworks used for sentiment analysis often suffer from vagueness, inconsistency, or absence. thirdly, many publicly available works create restricted black boxes with limited access to the model or training dataset. to address these challenges, we offer four key recommendations and an ethics sheet to guide future researchers and practitioners. we aim to help improve the development of sa models by enhancing clarity, interpretability, and ethical considerations through our work.