Do we still need Human Annotators? Prompting Large Language Models for Aspect Sentiment Quad Prediction (2025)

Abstract: Aspect sentiment quad prediction (ASQP) facilitates a detailed understanding of opinions expressed in a text by identifying the opinion term, aspect term, aspect category and sentiment polarity for each opinion. However, annotating a full set of training examples to fine-tune models for ASQP is a resource-intensive process. In this study, we explore the capabilities of large language models (LLMs) for zero- and few-shot learning on the ASQP task across five diverse datasets. We report F1 scores almost up to par with those obtained with state-of-the-art fine-tuned models and exceeding previously reported zero- and few-shot performance. In the 20-shot setting on the Rest16 restaurant domain dataset, LLMs achieved an F1 score of 51.54, compared to 60.39 by the best-performing fine-tuned method MVP. Additionally, we report the performance of LLMs in target aspect sentiment detection (TASD), where the F1 scores were close to fine-tuned models, achieving 68.93 on Rest16 in the 30-shot setting, compared to 72.76 with MVP. While human annotators remain essential for achieving optimal performance, LLMs can reduce the need for extensive manual annotation in ASQP tasks.

## introduction transformer-based large language models (llms) have gained significant attention due to their capability to address a broad spectrum of natural language processing (nlp) tasks, such as text summarization, translation, reading comprehension and text classification @xcite . noteworthy llms include llama-3.1 @xcite , gemma-3 @xcite , which are accessible in various parameter sizes with open model weights and commercial models like gpt-4 @xcite and claude 3 @xcite . previous research explored zero-and few-shot scenarios in which the llm generates outputs with either none or only a few labelled examples provided in the prompt @xcite . this eliminates the need for supervised model training, such as for small language models foot_0 (slms) using annotated datasets @xcite . this approach is particularly appealing because data annotation is often deemed complex and expensive, both in terms of time or financial cost, thereby complicating the development of text classification solutions tailored to specific tasks @xcite @xcite . an extensively studied task in nlp where manual annotations pose significant challenges is aspect-based sentiment analysis (absa) @xcite . this task facilitates the understanding of customer opinions expressed in reviews or feedback @xcite . unlike traditional sentiment classification, which assigns a single sentiment label (commonly positive, negative, or neutral) to an entire text document, absa requires annotators to identify all aspects within the text and determine the sentiment associated with each one @xcite . a prominent subtask of absa is aspect sentiment quad prediction (asqp), which provides exceptionally detailed insights into the author's opinions by identifying four sentiment elements for each opinion: aspect term (a), aspect category (c), sentiment polarity (p) and opinion term (o) @xcite . consequently, the annotation process for training examples is highly demanding, particularly when multiple opinions need to be annotated within a single text. previous research has predominantly concentrated on 0-to 10-shot learning, exclusively utilizing the english-language restaurant domain datasets @xcite . in this study, we extend the analysis to include up to 50 few-shot examples and evaluate the approach on a diverse series of five datasets. the datasets utilized in this work include @xcite and we incorporate the oats dataset by @xcite , which consists of hotel reviews from tripadvisor and online learning reviews collected from coursera. finally, we introduce a novel asqp dataset, comprising annotated sentences from airline reviews, which is published as part of this work. we considered the following research questions: rq1: how does varying the number of few-shot examples (from 0 to 50) impact performance on the asqp task? rq2: how do llms perform on the asqp task compared to slms trained on annotated examples? rq3: does self-consistency (sc) prompting @xcite , where multiple outputs are generated from the same prompt and the most consistent response is selected, improve performance on the asqp task? we employed google's gemma-3-27b @xcite and report the performance for the smaller-sized gemma-3-4b. in addition, we report the llms' performance on the target aspect sentiment detection (tasd), which focuses on the identification of (a, c, p)-triplets. all code and results of this study is publicly available on github foot_1 . the development of methodologies for addressing the asqp task was strongly influenced by the work of @xcite , which introduced two annotated datasets for the asqp task: rest15 and rest16. an example of such annotations is illustrated in figure 1 . both datasets comprise annotated sentences derived from restaurant reviews. the annotations are sourced from the semeval shared task datasets from @xcite @xcite , which originally included only (a, c, p)-triplets and thus did not include annotations for opinion terms. since the release of rest15 and rest16, generative methods within a unified framework have emerged as the state-of-the-art (sota) approach for the asqp task. various strategies have been explored to generate sentiment elements in specific formats that exploit label semantics. these include approaches employing structured extraction schemas @xcite , sequential representations of sentiment elements @xcite and natural language formats @xcite , wherein quadruples are systematically converted into natural language sentences. performance scores for these methods are presented in all the aforementioned approaches rely on small text generation models, such as t5-base @xcite , which utilizes an encoder-decoder ## methodology we utilized llms to tackle the asqp task across 0-, 10-, 20-, 30-, 40-, and 50-shot settings on different datasets. the performance is compared to that achieved using a dedicated training set to fine-tune smaller pre-trained language models. furthermore, we report performance results for the tasd task. ## results the performance scores for the evaluated configurations are shown in performance gains with an increasing number of few-shot examples. in most cases, increasing the number of few-shot examples resulted in incremental improvements in f1 scores across both asqp and tasd tasks. the difference between zero-and few-shot prompting is substantial. for instance, on the rest16 dataset under the sc prompting condition, the f1 score improved from 28.96 (0-shot) to 51.10 (50-shot) for the asqp task. to further highlight this trend, we provide line plots (see figure f ) that depict the influence of the number of few-shot examples on the f1 scores across all tasks, datasets, and models. llm performance slightly lower compared to sota fine-tuned approaches. for both tasd and asqp, the performance achieved through zeroand few-shot prompting did not surpass that ob- in case only 10 to 50 annotated examples were used for prompting or training, few-shot prompting consistently outperformed fine-tuning approaches across all sample sizes, with only a few exceptions. massive performance enhancements achieved through self-consistency. sc enabled consid-erable boosts of the f1 score, regardless of the amount of few-shot examples. however, recall was occasionally higher without sc. precision, on the other hand, was improved with sc in both tasks and across datasets. for instance, in the case of gemma-3-4b, precision was increased in most instances. the llm's parameter size matters. gemma-3-4b demonstrated lower performance in terms of f1 scores for both asqp and tasd. across the five datasets, the f1 scores in the asqp task were approximately 10 percentage points lower when using gemma-3-4b instead of gemma-3-27b. for example, on the rest15 dataset, the best f1 score achieved with gemma-3-4b was 44.20, while the best score for gemma-3-27b was 62.12 on the tasd task. lower performance in identifying opinion terms compared to other sentiment elements. as shown in the tables in appendix e, performance in identifying sentiment (positive, negative, or neutral) is highly performant, with f1 scores exceeding 90. however, performance in identifying aspect and opinion terms is comparatively much lower. ## discussion the results demonstrated performance improvements in f1 scores for both asqp and tasd as the number of few-shot examples increases, highlighting the gap between zero-and few-shot prompting. in this chapter, we put the results of this work into the context of previous research and provide an outlook on the direction of future work. new sota performance of llms. the llm zero-and few-shot learning performance scores reported in previous studies by @xcite and @xcite for the asqp task on the rest15 and rest16 datasets fall below those achieved by gemma-3-27b in both zero-and 10-shot learning settings. the only exception is rest15, where chatabsa @xcite outperformed gemma-3-27b in zero-shot learning except for tasd + rest16. unlike prior studies, which have primarily evaluated up to 10shot settings, we extended the investigation to a 10-to 50-shot setting for the first time. in this expanded range, gemma-3-27b achieved notable f1 scores exceeding 50 for the asqp task (e.g., rest16 with sc: 51.54) and surpassing 60 for the tasd task (e.g., rest16 with sc: 68.93). notably, these substantial gains are also attributed to the use of sc prompting. furthermore, this is in contrast to the work of @xcite , whose sc approach for e2e-absa did not lead to an improvement in performance. model size and prompting strategy affect few-shot performance. although gemma-3-27b achieved competitive results in both asqp and tasd, its performance remained slightly below fine-tuned sota approaches such as those by @xcite , and @xcite when full training sets were employed. however, in scenarios with limited annotated examples, few-shot prompting consistently outperformed fine-tuning. the parameter size of the model also influenced performance, with gemma-3-27b consistently outperforming its smaller counterpart, gemma-3-4b. directions for enhancing low-resource task performance. building on the promising results of this study, future research could focus on improving low-resource task performance through advanced prompt engineering techniques. approaches such as chain-of-thought prompting @xcite or plan-and-solve prompting @xcite , which allowed for performance gains in other nlp tasks, hold significant potential. furthermore, refining annotation guidelines or representing labels as natural language text, as proposed by @xcite , could contribute to improved outcomes. bigger llms, e.g. with 70b parameters, may provide additional performance benefits, given that our 27b model demonstrated superior results compared to the 4b variant. exploring less complex tasks and many-shot learning. in a broader context, future research could extend our approach to less complex tasks, in terms of the amount of considered sentiment elements, such as e2e-absa or aspect category sentiment analysis (acsa) which focuses on aspect category and the sentiment expressed towards them. beyond the low-resource setting considered in this study, one could explore the so-called "many-shot in-context learning" paradigm described by agarwal et al. for absa, where hundreds or even the full training set is provided in the prompt. observing that our approach achieved performance scores on the tasd task close to fine-tuned models, future work could investigate whether further increasing the number of shots lead to surpassing fine-tuned approaches.