Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation (2025)

Abstract: Aspect-based sentiment analysis (ABSA) is a challenging task of extracting sentiments along with their corresponding aspects and opinion terms from the text.The inherent subjectivity of span annotation makes variability in the surface forms of extracted terms, complicating the evaluation process.Traditional evaluation methods often constrain ground truths (GT) to a single term, potentially misrepresenting the accuracy of semantically valid predictions that differ in surface form.To address this limitation, we propose a novel and fully automated pipeline that expands existing evaluation sets by adding alternative valid terms for aspect and opinion. Our approach facilitates an equitable assessment of language models by accommodating multiple-answer candidates, resulting in enhanced human agreement compared to single-answer test sets (achieving up to a 10%p improvement in Kendall’s Tau score).Experimental results demonstrate that our expanded evaluation set helps uncover the capabilities of large language models (LLMs) in ABSA tasks, which is concealed by the single-answer GT sets.Consequently, our work contributes to the development of a flexible evaluation framework for ABSA by embracing diverse surface forms to span extraction tasks in a cost-effective and reproducible manner.Our code and dataset is open at https://github.com/dudrrm/zoom-in-n-out-absa.

## introduction aspect-based sentiment analysis (absa) is a sophisticated natural language processing task that aims to extract fine-grained sentiment information from text. absa is usually utilized to provide insights into opinions about specific attributes of products or services, enabling organizations or persons to precisely identify consumer preferences and criticisms. the granularity of absa facilitates ♣ work is done while affiliated to sait. original gts variants of aspect / opinion (a) (b) n't worth waiting not worth not worth waiting lacks quality lacks taste guacamole at pacifico steak guacamole at pacifico lacks quality and taste the 9 oz steak is n't worth waiting alternative valid expressions for these terms are highlighted: aspects in yellow and opinions in blue. under conventional evaluation approaches using the original gt set, these highlighted candidates are assessed as wrong because they do not match exactly despite being semantically consistent with the original terms. here, the gray-shaded terms represent the ground truth (gt) annotations: "guacamole" as the aspect and "lacks quality and taste" as the opinion. due to the richness and variability of natural language, there are diverse forms in which these elements are expressed and consequently annotated. different annotators may select different spans to represent the same semantic content, producing various surface forms in span annotations. in figure 1 , the terms highlighted in color illustrate alternative candidates that can be considered valid annotations. however, conventional absa benchmarks @xcite @xcite @xcite typically provide a single gt term for each aspect and opinion pair and employ exact match criteria for evaluation. this approach fails to account for the compatibility of multiple surface forms. as illustrated in figure 1 , semantically valid surface forms (highlighted in color), such as "guacamole at pacifico" for the aspect or "lacks quality" and "lacks taste" for the opinion, are evaluated as incorrect, while they preserve the essential meaning of the original gt quadruple. consequently, this rigid evaluation schema may underestimate the performance of advanced language models in understanding and generating semantically valid predictions, raising concerns about the fairness and comprehensiveness of current model assessments in absa. to address the limitations of conventional absa evaluation approaches, which often fail to account for the variability of natural language expressions, we propose zoom in-n-out, a novel approach for evaluation in absa task 1 . our method aims to accommodate diverse surface forms that preserve the semantic content of the original gt quadruple. zoom in-n-out pipeline comprises three key steps: (i) identifying alternative terms within the original gt span, (ii) exploring potential expressions outside the span, and (iii) verifying that new candidates are semantically consistent with the original gt. this process leverages the capabilities of large language models (llms), specifically gpt-4, which has demonstrated quality in annotation tasks @xcite and efficacy in generating synthetic datasets @xcite . notably, our approach offers several advantages over existing methods, such as n-gram overlap metrics or direct llm/human evaluation. n-gram-based methods like f-measure @xcite may fail to capture semantic nuances; for instance, they may incorrectly score "not worth" and "worth" as similar when gt is "n't worth". direct llm/human evaluation @xcite can be costly and difficult to reproduce since the evaluator model should be run when a new prediction occurs. in contrast, our method provides a balanced solution by ensur1 while implicit terms (e.g., "null") are considered in absa benchmarks @xcite , our method aims to expand explicit gt expressions in aspect and opinion terms. we leave this exploration for future work. experimental results demonstrate the quality and effectiveness of our method. our expanded gt set achieves over 90% validity in human assessment and shows improved human alignment, with up to a 10%p increase in kendall tau score compared to the original gt set. furthermore, compared to t5-based models, the f1 scores of llms jump up in our evaluation set when the train set (in-context examples) comes from the original dataset. for instance, four llms increase their f1 score by 9.8%p, but t5 models increase by only 2.3%p on average. this significant score gap suggests that the conventional evaluation framework may have underestimated the capabilities of llms. it highlights how our method uncovers their hidden ability through our flexible evaluation framework. in summary, the contributions of our paper can be summarized as follows: ## method our pipeline aims to expand the original gt aspect and opinion terms to various equivalent expressions, to encompass the diversity of surface forms in the absa task. based on figure 2 , we introduce zoom in-n-out and describe how the exact match measure is calculated on our gt sets. ## experiments in this section, we first assess the validity of our expanded gts, show that our evaluation set aligns more closely with human evaluators than the original, and analyze diverse baseline models with the original and our gt sets.