Are You Sure You’re Positive? Consolidating Chain-of-Thought Agents with Uncertainty Quantification for Aspect-Category Sentiment Analysis (2025)

Abstract: Aspect-category sentiment analysis provides granular insights by identifying specific themes within product reviews that are associated with particular opinions. Supervised learning approaches dominate the field. However, data is scarce and expensive to annotate for new domains. We argue that leveraging large language models in a zero-shot setting is beneficial where the time and resources required for dataset annotation are limited. Furthermore, annotation bias may lead to strong results using supervised methods but transfer poorly to new domains in contexts that lack annotations and demand reproducibility. In our work, we propose novel techniques that combine multiple chain-of-thought agents by leveraging large language models’ token-level uncertainty scores. We experiment with the 3B and 70B+ parameter size variants of Llama and Qwen models, demonstrating how these approaches can fulfil practical needs and opening a discussion on how to gauge accuracy in label-scarce conditions.

## introduction tracking customer satisfaction is critical for organisations aiming to improve their products and services. however, traditional supervised approaches require bespoke datasets, necessitating time and human labour @xcite . the costs of annotation can be a significant challenge for many organisations, prohibiting access to adapted state-of-the-art solutions, and even to evaluation. hence, in this paper, we explore and compare two popular large language models (llms) across various low-training scenarios, specifically zero-shot, for analysing customer satisfaction. sentiment analysis approaches are commonly used to classify feedback as positive, neutral, or negative for customer satisfaction, but this method often lacks granularity. a single piece of feedback can express multiple sentiments targeting different product or service aspects, leading to ambiguous classifications and limiting actionable insights. to address this, aspect-based sentiment analysis (absa) methodologies @xcite have been developed to more accurately capture the nuanced sentiments present in customer feedback. in our study, we investigated the absa task of aspect category sentiment analysis (acsa), a type of absa. for a given piece of text, acsa considers the aspect categories and sentiment polarities, as shown in figure 1 . the categories are a set of classes which are predefined by domain experts or stakeholders. characteristically, in the figure, the example demonstrates two tuples from the same text. firstly, the review found the pepperoni pizza, a type of #food (category), to be delicious, representing a positive sentiment (polarity). secondly, the review found the service to be terrible, representing negative polarity. ## related work absa has emerged as a particularly interesting research topic due to its increasing popularity and widespread applicability across various domains @xcite @xcite @xcite . in this paper, we address the acsa task, a type of absa. the acsa task is especially relevant for industry applications, as it involves associating predefined category terms with corresponding sentiment polarities, which can be readily quantified and analysed (more on section 3.2). although a few studies have considered the acsa task from a supervised learning perspective @xcite @xcite , it would be beneficial to address it in an unsupervised manner because annotation requires considerable time and human labour, and may also obfuscate reproducibility, as discussed later in this paper. zero-shot and few-shot methodologies have been employed for the broader absa task. for instance, @xcite use llms with a few-shot in-context learning (icl) approach, while @xcite evaluate different prompting techniques for sentiment analysis using llms. fei et al. (2023) investigated a multi-hop cot approach, and @xcite introduced chatabsa, an icl few-shot prompt framework for absa that also extends to the acsa task we address. in our paper, we aim to push the boundaries of zero-shot learning by examining the use of multiple llm agents provided with no prior knowledge. more agents can yield better results; indeed, @xcite discuss how employing additional agents and subsequently performing majority voting can improve outcomes. to the best of our knowledge, there is no prior work on multi-agent collaboration for acsa. we therefore experiment with this approach, inspired by previous absa research on @xcite , and we investigate different aggregation techniques that leverage tokenlevel uncertainty in llms @xcite . ## results & discussion element order debunking we tested three element permutations for each llm model (i.e., aspect, category and opinion). our experiments on the four datasets demonstrated that the order of the three elements between the two larger llm models was fairly consistent, but it differed for each dataset. for instance, order 1) opinion 2) category 3) aspect produced higher f1 scores for the lap-top16 dataset, whereas order 1) category 2) aspect 3) opinion was best for the mams dataset. for the other two datasets, although they differed, the position of the opinion and the categories always followed the same pattern, indicating a cot trend per dataset. when comparing the 3b llms, we notice the same pattern in that the cot element order stays fairly consistent across each dataset. moreover, for all the llms, we did not notice any cot agent that consistently underperformed; similarly, it varied across datasets but not across the models. thus, we draw two conclusions from our experiments. firstly, as opposed to previous research @xcite , starting from the aspects in the cot prompts does not always provide optimal results. secondly, the order is rather datadependent and stays consistent across our four models. in the case of our multi-hop cot preliminary experiments, the best-performing agents always had the opinion preceding the category and aspect, see ## future direction the results demonstrated that the joined cot agents achieved quite high recall, outperforming any individual cot agent. this demonstrates that there is potential for improvement in the use of agents. one possible direction for future research could be to iteratively guide the inference process until a satisfactory summed log probability is reached. moreover, one can try combining other cot agents to evaluate possible improvements. or utilising other aggregating techniques to filter the pairs. research has raised concerns regarding the reliability of token-level uncertainty, as token conditional distributions may sometimes be misleading. this issue arises particularly in cases where an initial token is incorrect, yet all subsequent tokens appear highly probable given that initial token @xcite . investigating more robust methods for estimating token-level confidence scores could improve the downstream acsa performance or other tasks as well. ## conclusion in our experiments, we evaluated two larger (70b+) llms, qwen and llama, along with their smaller 3b parameter-size counterparts. we observed that the optimal cot order for the acsa task is rather dataset-dependent. additionally, we found that employing a multi-hop cot approach does not yield better results compared to our enumerated cot version. furthermore, confidence scores derived from token-level uncertainty proved somewhat informative, appearing to be a more effective strategy than relying solely on majority voting among agents. moreover, we discovered that using the variance in confidence between agents can be a good indicator to view challenging examples. intuitively, larger models tend to produce more reliable confidence estimates. given the high recall achieved by our combined cot aggregation method, we suggest there remains considerable room for improvement, potentially through exploring alternative aggregation techniques or developing more accurate tokenlevel uncertainty estimation methods.