Sentiment Analysis in the Era of Large Language Models: A Reality Check (2024)

Abstract: Sentiment analysis (SA) has been a long-standing research area in natural language processing. With the recent advent of large language models (LLMs), there is great potential for their employment on SA problems. However, the extent to which current LLMs can be leveraged for different sentiment analysis tasks remains unclear. This paper aims to provide a comprehensive investigation into the capabilities of LLMs in performing various sentiment analysis tasks, from conventional sentiment classification to aspect-based sentiment analysis and multifaceted analysis of subjective texts. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets. Our study reveals that while LLMs demonstrate satisfactory performance in simpler tasks, they lag behind in more complex tasks requiring a deeper understanding of specific sentiment phenomena or structured sentiment information. However, LLMs significantly outperform SLMs in few-shot learning settings, suggesting their potential when annotation resources are limited. We also highlight the limitations of current evaluation practices in assessing LLMs’ SA abilities and propose a novel benchmark, SentiEval, for a more comprehensive and realistic evaluation. Data and code are available at https://github.com/DAMO-NLP-SG/LLM-Sentiment .

## introduction sentiment analysis 1 (sa) has been a longestablished area of research in natural language processing (nlp), which aims to study people's opinions, sentiments, emotions, etc, through computational methods @xcite . since its inception @xcite , this field has attracted significant interest from both academia and industry given its wide range of applications, such as analyzing product reviews and gaining insights from social media posts @xcite . furthermore, achieving a deep understanding of human subjective feeling through sentiment analysis is undoubtedly an important step toward developing artificial general intelligence @xcite . in recent years, large language models (llms) have demonstrated impressive performance on various nlp tasks @xcite @xcite . they can directly perform tasks in zero-shot or few-shot incontext learning manner and achieve strong performance without the need for any in-domain supervised training @xcite @xcite . although there have been some initial attempts to apply llms to sentiment analysis @xcite @xcite , these studies are often limited to some specific tasks and adopt different models, datasets, and settings in experiments. as such, the extent to which existing large language models can be leveraged for sentiment analysis problems remains unclear. in this work, we aim to conduct a reality check on the current state of sentiment analysis in the era of large language models. specifically, we seek to answer the following research questions: 1) what is the current maturity of various sentiment analysis problems? 2) compared to small specialized models trained on domain-specific data, how do large models fare in both zero-shot and few-shot settings? 3) are current sa evaluation practices still suitable to assess models in the era of llms? to this end, we first conduct a systematic review of various sentiment analysis-related tasks, from conventional sentiment classification (sc, classifying the sentiment orientation of a given text @xcite ) to aspect-based sentiment analysis (absa, analyzing sentiment and opinion information at the more fine-grained aspect level @xcite ) and the multifaceted analysis of subjective texts (mast, focusing on specific sentiment or opinion phenomena such as hate speech detection and comparative opinion mining @xcite ). in total, we consider 13 sentiment analysis tasks across 26 datasets. these tasks were often studied in isolation in the past due to their unique characteristics. this fragmentation, while reasonable before, offered a somewhat incomplete understanding of how well models could comprehend human subjective information. for llms, we consider both open-source models including flan-t5 @xcite and flan-ul2 @xcite , along with gpt-3.5 model series, namely chatgpt (gpt-3.5-turbo) and instructgpt (text-davinci-003) @xcite . we also establish comparison baselines using smaller language models foot_0 (slms) such as t5 @xcite , which allows us to measure the performance of llms against these specialized models trained with indomain labeled data. our investigation yields several insights: firstly, llms already show strong sentiment analysis ability in zero-shot settings. on some simple sa tasks such as sentiment classification, they can perform on par with slms trained with full training data. secondly, when it comes to more complex tasks such as absa tasks that require structured sentiment information, or mast tasks requiring a deep understanding of specific sentiment phenomena, llms still lag behind slms trained with in-domain data. moreover, llms appear to be sensitive to prompt design when encountering tasks with complex input and output formats. thirdly, with a limited quantity of annotated data under the few-shot setting, llms with in-context learning consistently outperform slms trained with the same amount of data for all types of tasks. this suggests that the application of llms is advantageous when annotation resources are scarce. during the investigation, we also identify several limitations of current practice in evaluating a model's sa capability. for example, the evaluations often only involve specific tasks or datasets; and inconsistent prompts are utilized across different studies to evaluate models. while these evaluation practices might have been appropriate in the past, they fall short of accurately assessing llms' sa abilities. to address these issues, we propose a novel benchmark called sentieval. it breaks the boundary of a wide range of sa tasks, enabling a more comprehensive evaluation of models. it also employs varied task instructions, paired with the corresponding text, alleviating the sensitivities associated with prompt design during the evaluation of different llms. furthermore, by framing these tasks as natural language instructions, we create a more realistic evaluation environment akin to a real-world practical use case. ## background sentiment analysis sa has received lots of attention since its early appearance @xcite @xcite and remained an active research area in the field of nlp nowadays @xcite @xcite . such enduring interest stems from both the importance of comprehending human subjective sentiments and opinions toward achieving human-level intelligence @xcite , and its broad practical applications, such as analyzing customer reviews @xcite and digesting social media opinions @xcite . sa comprises a broad spectrum of tasks, from sentiment classification that determines the overall sentiment polarity of a given text @xcite , to aspect-based sentiment analysis (absa) @xcite and multifaceted analysis of subjective texts (mast) @xcite in recent years. all these tasks collectively contribute to a holistic understanding of sentiment in language and demonstrate the wide range of tasks falling under the umbrella of sentiment analysis. large language models (llms) recently, there has been a remarkable advancement in the development of llms, such as @xcite , @xcite , flan-ul2 @xcite , llama @xcite and chatgpt. there are some initial attempts on evaluating llms for sa tasks. @xcite observe that the zero-shot performance of llms is comparable to fine-tuned bert model. @xcite conduct a preliminary study with chat-gpt for some sa tasks, specifically investigating its ability to handle polarity shifts, open-domain scenarios, and sentiment inference problems. in addition, @xcite focus on chatgpt's emotional conversation capability and indicate it exhibits promising results in generating emotional responses. moreover, @xcite explore the fine-tuning of a small student model with an llm to generate weak labels, and the final model performs on par with existing supervised models. despite those existing efforts, their scope is often limited to specific tasks and involves different datasets and experimental designs. the true capacity of llms for sa remains unclear. ## investigated tasks and datasets we conduct an extensive survey of a wide range of sa tasks and categorize different tasks into three types: sentiment classification (sc), aspect-based sentiment analysis (absa), and multifaceted analysis of subjective texts (mast). we briefly describe investigated tasks of each type, along with the datasets and evaluation metrics in this section. the detailed descriptions are in appendix a.1. for each dataset, we sample a maximum of 500 examples from its original test set, to ensure balance across various tasks and datasets. ## conclusions in this study, we conduct a systematic evaluation of various sentiment analysis tasks using llms, which helps better understand their capabilities in sentiment analysis problems. experimental results reveal that while llms perform quite well on simpler tasks in a zero-shot setting, they struggle with more complex tasks. in a few-shot learning context, llms consistently outperform slms, suggesting their potential in scenarios where annotation resources are scarce. this work also highlights the limitations of current evaluation practices and then introduces the sentieval benchmark as a more comprehensive and realistic evaluation tool.