From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models (2025)

Abstract: This study examines the performance of Large Language Models (LLMs) in Aspect-Based Sentiment Analysis (ABSA), with a focus on implicit aspect extraction in a novel domain. Using a synthetic sports feedback dataset, we evaluate open-weight LLMs’ ability to extract aspect-polarity pairs and propose a metric to facilitate the evaluation of aspect extraction with generative models. Our findings highlight both the potential and limitations of LLMs in the ABSA task.

## introduction absa is a nuanced form of sentiment analysis that focuses on identifying sentiments related to specific aspects within a text @xcite . researchers have decomposed absa into various subtasks, such as aspect extraction, sentiment classification, aspect category detection, and opinion term extraction, each contributing to a comprehensive understanding of the problem. llms with their in-context learning (icl) capabilities @xcite and parameter-efficient fine-tuning methods, such as low-rank adaptation (lora) with quantization @xcite , offer straightforward yet effective approaches for complex absa tasks. these approaches facilitate the extraction of implicit aspects, which are aspects that are not explicitly stated in the text but can be inferred based on context, sentiment, or background knowledge. this study examines the performance of llms in extracting aspect-polarity pairs within the underexplored and unanticipated domain of sports feed-back. this domain poses unique challenges for absa due to its reliance on implicit references and domain-specific terminology. by evaluating llms in this context, we provide critical insights into their capacity to adapt to novel data. moreover, recognizing the linguistic variability involved in expressing implicit aspects, we propose an evaluation metric that calculates precision and recall while accounting for this variability in settings with a high prevalence of implicit aspects. we also demonstrate the broader applicability of this metric, showing its utility in assessing generative llms on classic absa datasets. finally, we explore various strategies for adapting llms to domain-specific datasets, highlighting key challenges and offering insights for future research. ## related work aspect-based sentiment analysis traditional approaches to absa, extensively reviewed in the literature @xcite @xcite , primarily utilize bidirectional encoders @xcite , recurrent networks @xcite , graph networks @xcite @xcite , sequence-to-sequence models @xcite , and ensembles of models @xcite . various techniques have recently been proposed to improve accuracy, precision, and recall in absarelated tasks, for example, context denoising @xcite , abstract meaning representation @xcite , and global semantic features @xcite . these methods have achieved robust results in within-domain explicit aspect extraction and polarity classification @xcite @xcite . recent studies have investigated the ability of llms to perform absa tasks on both traditional @xcite and more complex datasets ## metrics automated evaluation of models for the aspect detection subtask faces several challenges. first, documents may contain implicit aspects that do not directly match with individual words. for example, the sentence from our dataset: i found that some locations had multiple volunteers that didn't appear to be overly busy and could have been useful at other locations where there were shortages. this sentence alludes to the aspect 'allocation of volunteers' without explicitly stating it in the text. moreover, the definition of what constitutes an aspect is often fuzzy: in the cited example, 'placement of volunteers' could also be interpreted as a valid aspect. second, when llms are used for aspect extraction instead of traditional span-based approaches, relying on exact matches to compute metrics such as precision, recall, and f-score without accounting for linguistic variation can be problematic. to address these evaluation challenges, we propose a generalized method for assessing precision (p ) and recall (r) inspired by the work of euzenat (2007) on ontology alignment. specifically, to account for partial matches and linguistic variation between predicted and true aspect sets, we define precision and recall as follows. for a given document, we define s d as the set of detected aspects and s g as the set of true (gold) aspects. the function ι, parameterized by a threshold θ ∈ [0, 1], returns the set of partial matches between s d and s g . figure 2 illustrates the concept of the intersection ι between the two sets of aspects. the threshold θ serves as a filter for the minimal similarity required between pairs of matching aspects. in the special case where θ = 1, the function ι(s d , s g ) reduces to the intersection of the two sets, enforcing exact aspect matches. conversely, when 0 ≤ θ ≪ 1, it permits the matching of semantically unrelated pairs, making values of θ close to zero impractical. for the purpose of experiments in this study, we set θ = 0.95. an empirical analysis of the impact of θ on matching errors in the context of this study is provided in appendix f. with these definitions, the generalized precision, denoted as p θ , is given by: similarly, the generalized recall, denoted as r θ , is formulated as: algorithm 1 provides the implementation of the function ι(s g , s d , θ) used in this study. a similarity measure σ : s 1 × s 2 → [0, 1] quantifies the resemblance between individual elements from the sets, resulting in a similarity matrix with values ranging from 0 to 1. to avoid false positive matches, values below a specified threshold θ are set to zero. the similarity matrix is then converted into a cost matrix, and the linear sum assignment problem is solved to determine the optimal pairing of elements between the sets, minimizing the total cost. this procedure yields a set of optimal element pairs, i. ## models we evaluated two open-weight models, mistal 7b instruct @xcite and llama-3 8b instruct @xcite , against the baseline pyabsa @xcite on the aspect-polarity pair extraction (aspe) task. the selection of the open-weight models was motivated by their stateof-the-art performance within the parameter range 4 , ease of deployment, and computational efficiency. their relatively compact sizes (7-8 billion parameters) allow local deployment without reliance on external computational resources, a material factor for practical applications. pyabsa is an actively maintained, ensemble4 as of july 2024 based framework trained on publicly available datasets. it serves as a reliable baseline representing traditional yet robust absa methodologies. for measuring phrase similarity, we selected sentence-t5 (large) @xcite . despite its smaller size compared to more recent largescale models, sentence-t5 demonstrates strong performance on text embedding benchmarks @xcite , making it well-suited for experiments with limited computational resources. ## evaluation of open-weight models our experiments aim to address the following research questions: for model fine-tuning, we utilized: 1. the training portion of the composite dataset, containing 14,046 samples. open-weight llms' performance varies by dataset when used without fine-tuning. they performed worse than the pyabsa baseline on the composite dataset (which matches pyabsa's training data), but outperformed it on the novel dataset (which differs in domain and implicit aspect frequency). using icl with sampling significantly improved performance across both datasets, showing that providing relevant examples is an effective way to enhance llms' aspect extraction abilities. fine-tuning effectiveness depends on the similarity between training and evaluation data. when fine-tuned on the composite dataset, both llama-3 and mistral showed significant performance gains on composite samples compared to their non-finetuned versions, but their performance on novel samples declined, falling below that of icl with sampling. the reverse held true when fine-tuning on the novel dataset: while significant improvements were observed on novel samples, performance on composite samples degraded below that of icl with sampling. in contrast, fine-tuning on a mixed dataset combining both novel and composite samples yielded consistent performance gains across both dataset classes. appendix i presents detailed experimental results for individual datasets on the aspect extraction task, evaluated using both adjusted metrics θ = 0.95 and exact match criteria. ## discussion and further research our study reveals several key findings and corresponding future research directions. sota llms demonstrated effectiveness in generating initial annotations for the proposed dataset, despite inherent limitations like restricted context windows and occasional inaccuracies. the implemented multi-step annotation process, combining automated llm-generated annotations with human validation, successfully streamlined the traditionally labor-intensive workflow while maintaining annotation quality through human oversight. the employment of icl with sampling proved effective for enhancing llm performance in extracting absa pairs, offering advantages over finetuning approaches that can lead to overfitting and reduced generalizability. to build upon this success, future research should explore more sophisticated icl strategies, such as retrieval-augmented icl @xcite , which could further enhance the extraction of aspect-sentiment pairs. our proposed metric for generalized precision and recall captures model performance on the aspect extraction task while accounting for linguistic variability. future work should focus on developing methods for automatic determination of the optimal threshold θ value, investigating its relationship with various semantic similarity models. additionally, implementing error detection methods could enable dynamic θ adjustment, ensuring accurate performance measurement across both explicit and implicit aspect extraction scenarios. finally, adopting multi-step reasoning approaches like chain-of-thought prompting @xcite or iterative refinement @xcite presents a promising direction for improving both data annotation and pair extraction processes, potentially reducing the need for human intervention while maintaining output quality.