Domain-Expanded ASTE : Rethinking Generalization in Aspect Sentiment Triplet Extraction (2024)

Abstract: Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. However, existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains, raising concerns about the generalization of proposed methods. Furthermore, it remains unclear if large language models (LLMs) can effectively handle complex sentiment tasks like ASTE. In this work, we address the issue of generalization in ASTE from both a benchmarking and modeling perspective. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings. Additionally, we propose CASE, a simple and effective decoding strategy that enhances trustworthiness and performance of LLMs in ASTE. Through comprehensive experiments involving multiple tasks, settings, and models, we demonstrate that CASE can serve as a general decoding strategy for complex sentiment tasks. By expanding the scope of evaluation and providing a more reliable decoding strategy, we aim to inspire the research community to reevaluate the generalizability of benchmarks and models for ASTE. Our code, data, and models are available at https://github.com/DAMO-NLP-SG/domain-expanded-aste.

## introduction opinions and sentiments are essential to human communication, beliefs, and behaviors @xcite . although sentiment analysis is often performed at the sentence or document level, it is insufficient to capture the fine-grained sentiment information and nuances of human opinions @xcite . to this end, aspect sentiment triplet extraction (aste) @xcite is a challenging and well-established task of aspect-based sentiment analysis @xcite which aims to extract richer and more interpretable sentiment information from natural language. concretely, aste considers how each opinion term in a text may express sentiments towards specific aspect targets. although aste has become a more established task with many existing methods @xcite , we are concerned that they may not generalize well due to limitations in the existing benchmark datasets. notably, the established benchmarks are limited to two domains, which limits the evaluation scope of model capabilities and does not represent the diversity of real-world data. on the other hand, it is also important to assess how models generalize to unseen domains as domainspecific labeled data is often scarce @xcite , and models may face domain shift during deployment @xcite . hence, this mo-tivates us to propose a domain-expanded aste benchmark which not only considers the in-domain performance, but also evaluates out-of-domain generalization across a more diverse set of domains. we support the new benchmark by annotating more than 4,000 data samples for two new domains based on hotel and cosmetics product reviews. therefore, we can construct a domain-expanded dataset with four domains as shown in figure 1 . to investigate the domain generalization of existing aste methods, we evaluate five existing methods based on pretrained language models (plms) for the in-domain and out-of-domain settings. on the other hand, while large language models (llms) have recently enabled breakthroughs in many nlp tasks, it is unclear if they can surpass specialized pretrained language models (plms) on sentiment tasks such as aste @xcite . despite the impressive language understanding and general-purpose capabilities of llms, it is challenging to adapt them to aste due to several reasons. notably, black-box models like gpt-4 (ope-nai, 2023) are less trustworthy and interpretable as it is not clear how to estimate the confidence of their predictions. for instance, as each text may contain multiple sentiment triplets, it is useful to know which of the predicted triplets have higher confidence or lower confidence. hence, the lack of interpretability hinders the trustworthiness of llms in practical applications, and limits indepth analysis of their performance. on the other hand, it is generally not possible or feasible to train llms for specific tasks, leading to greater focus on prompt-based methods to improve performance. thus, we introduce confidence-aware sentiment extraction (case), a simple and effective decoding strategy to improve the trustworthiness and performance of llms for complex sentiment tasks like aste. inspired by self-consistency @xcite which samples diverse reasoning paths to select the most consistent answer, we sample diverse sets of sentiment triplets to select the most consistent triplets. intuitively, sentiment triplets which are most consistent, i.e., occur most often when sampling diverse sets of triplets, can be assigned a higher confidence. notably, it is simple to integrate case with any language model that supports stochastic sampling, and it does not require any model re-training or access to model logits. compared to conventional decoding methods such as greedy search or beam search, case en-hances interpretability by estimating the confidence of each predicted triplet, and improves performance by explicitly considering a larger pool of sentiment triplets. in summary, our main contributions include: (1) to evaluate aste methods more holistically, we propose a domain-expanded benchmark which covers in-domain and out-of-domain performance across diverse domains. (2) we annotate more than 4000 samples for two new domains based on hotel and cosmetics product reviews to support the new benchmark. (3) we propose case, a simple and effective decoding strategy to enhance the trustworthiness and performance of llms for aste. our experiments demonstrate its effectiveness across different models, tasks, and settings. ## domain-expanded aste benchmark to evaluate the performance of aste methods more holistically and encourage development of more robust methods, we propose a domainexpanded benchmark. the benchmark assesses models not only in-domain, but also in terms of outof-domain generalization across diverse domains. hence, we construct the benchmark by leveraging two domains from existing datasets, while annotating samples for two new domains. in this section, we detail the dataset construction process and dataset statistics for each domain. ## confidence-aware sentiment extraction (case) to enhance the trustworthiness and effectiveness of large language models (llms) on aste, we propose confidence-aware sentiment extraction (case), a simple and effective decoding strategy. compared to conventional decoding methods such as greedy search or beam search, case enhances interpretability by estimating the confidence of each predicted triplet, and improves performance by explicitly considering a larger pool of sentiment triplets. inspired by self-consistency @xcite which samples diverse reasoning paths to select the most consistent answer, we sample diverse sets of sentiment triplets to select the most consistent triplets. as shown in figure 2 , case consists of four main steps: (1) given the input text, we sample diverse output sequences from the language model, where each output sequence represents a set of candidate sentiment triplets. ## results and analysis to provide a holistic study of aste methods, we evaluate on the proposed domain-expanded aste benchmark, reporting the fully supervised indomain results in ## conclusions in conclusion, this work addressed the task of aspect sentiment triplet extraction (aste) in sentiment analysis, focusing on the issues of limited benchmark domains and the challenges of large language models (llms) in handling complex sentiment tasks. we introduced a domain-expanded aste benchmark by annotating samples from diverse domains, enabling the evaluation of models in both in-domain and out-of-domain settings. this expanded benchmark provided a more comprehensive assessment of model performance, addressing concerns regarding the generalizability of proposed methods. secondly, a novel decoding strategy called case (context-aware sampling and enhancement) was proposed to enhance the trustworthiness and performance of llms in aste. the experimental results demonstrated its effectiveness across multiple tasks, settings, and models. its simplicity and efficacy make it a promising general decoding strategy for complex sentiment tasks. by expanding the scope of evaluation and providing a reliable decoding strategy, we hope to encourage the research community to rethink the generalizability of benchmarks and models for aste. the findings highlight the importance of considering diverse domains and utilizing appropriate decoding strategies when tackling fine-grained sentiment analysis tasks. with these contributions, we hope to foster the development of more robust and capable sentiment analysis methods in the future.