PASTEL : Polarity-Aware Sentiment Triplet Extraction with LLM -as-a-Judge (2025)

Abstract: Aspect Sentiment Triplet Extraction (ASTE) is a subtask of Aspect-Based Sentiment Analysis (ABSA) that aims to extract aspect terms, corresponding opinion terms, and their associated sentiment polarities from text. Current end-to-end approaches, whether employing Large Language Models (LLMs) or complex neural network structures, struggle to effectively model the intricate latent relationships between aspects and opinions. Therefore, in this work, we propose Polarity-Aware Sentiment Triplet Extraction with LLM-as-a-judge (PASTEL), a novel pipeline that decomposes the ASTE task into structured subtasks. We employ finetuned LLMs to separately extract the aspect and opinion terms, incorporating a polarity-aware mechanism to enhance opinion extraction. After generating a candidate set through the Cartesian product of the extracted aspect and opinion-sentiment sets, we leverage an LLM-as-a-Judge to validate and prune these candidates. Experimental evaluations demonstrate that PASTEL outperforms existing baselines. Our findings highlight the necessity of modular decomposition in complex sentiment analysis tasks to fully exploit the capabilities of current LLMs.

## experiments we conducted our experiments using the llama 3.2 (1b) models and evaluated the proposed system component-wise for task-specific extraction i.e., the ate and ote pipelines, along with the overarching task of aste. for ate and ote tasks we have benchmarked the approach against @xcite . for aste, we compared our pipeline against the performance of gpt-4o in zero-shot and few-shot settings as highlighted in @xcite . additionally we also evaluated our pipeline against existing methods, such as span-bart @xcite , aste transformer (naglik and lango, 2024), minicongts @xcite , and in-structabsa @xcite . dataset and metrics: for the aste task, @xcite introduced a dataset that builds upon the semeval 2014 task dataset @xcite . later @xcite released an improved version foot_0 , which includes explicitly defined triplets for cases where a single opinion span is associated with multiple aspect terms. we evaluate all models based on their precision, recall, and f1 score. triplet pruning using llm-as-a-judge: we experimented with various prompt templates for pruning candidate triplets @xcite . the first prompt type scored each triplet based on the supporting evidence. however, this method was unreliable as the ratings were inconsistent, making it difficult to set a meaningful threshold ( §a.2.4). we then tried a yes/no type of prompt, which was more consistent but lacked reasoning for complex sentences ( §a.2.5). to address this, we incorporated chain of thought along with few shot examples. this approach not only gave stabilized results but also enhanced the model's reasoning, resulting in transparent and accurate pruning. ## results we evaluated paote (llama 3.2-1b) against two strong baselines-hast + towe @xcite and spanmlt @xcite . as shown in table @xcite , paote attains the highest f1 scores, underscoring the effectiveness of our approach. table @xcite demonstrates that the full finetuning significantly improves accuracy, particularly recall in llama 3.2 1b for ate task. it achieves the highest f1 across all datasets, outperforming span-mlt @xcite and instructabsa @xcite . the results in table @xcite indicate pastel achieves state-of-the-art performance across most datasets in aste, outperforming existing methodologies. pastel also performs comparably to the aste transformer @xcite on 15res dataset, further validating its effectiveness. the observed performance loss in 15res arises primarily from dataset limitations. as illustrated below in table @xcite , there is a strong positive correlation between the number of training sentences, average sentence length, and the change (∆) in f1 score over previous state-of-the-art method @xcite . 15res has fewer training sentences and shorter average sentence lengths compared to 14lap, 14res and 16res, limiting contextual information essential for effective learning. pastel excels when trained on abundant and contextually rich data, as reflected by its superior f1 scores on larger datasets like 14res and 16res. conversely, the limited data and shorter contexts in 15res restrict such performance gains. ## conclusion we present pastel, a novel approach for aste using llms, which leverages ate, paote and llm-as-a-judge for structured triplet validation. according to tables [1] and [2], our approach consistently performs better than the benchmarks on both ate and ote, showing the effectiveness of full fine-tuning to achieve optimal model performance. by splitting the pipeline to extract the aspects and opinion terms separately, we are able to mitigate the limitation of llms in understanding the syntactic and semantic dependencies between the aspect and opinion terms, ensuring more precise extraction of triplets. pastel outperforms on majority of the datasets and remains on par with aste transformer on 15res. additionally, llmas-a-judge effectively prunes noisy candidates generated with cartesian product of the outputs of the two pipelines. we leverage llms' reasoning abilities (via chain-of-thought prompts) to ensure that extracted triplets are coherent within each sentence. both modular fine-tuning strategy and llm-asa-judge mechanism contribute significantly to better accuracy, with a better alignment of human annotations. overall, all of these results verify the impact of our structured pipeline approach, mirroring the need for modular decomposition and llm-based verification within aste applications.