Measuring Your ASTE Models in The Wild: A Diversified Multi-domain Dataset For Aspect Sentiment Triplet Extraction (2023)

Abstract: Aspect Sentiment Triplet Extraction (ASTE) is widely used in various applications. However, existing ASTE datasets are limited in their ability to represent real-world scenarios, hindering the advancement of research in this area. In this paper, we introduce a new dataset, named DMASTE, which is manually annotated to better fit real-world scenarios by providing more diverse and realistic reviews for the task. The dataset includes various lengths, diverse expressions, more aspect types, and more domains than existing datasets. We conduct extensive experiments on DMASTE in multiple settings to evaluate previous ASTE approaches. Empirical results demonstrate that DMASTE is a more challenging ASTE dataset. Further analyses of in-domain and cross-domain settings provide some promising directions for future research.

## introduction aspect sentiment triplet extraction (aste; @xcite , a fine-grained task in sentiment analysis @xcite , has attracted considerable interest recently @xcite . the objective of this task is to extract the sentiment triplet, comprising of an aspect term, an opinion term, and a sentiment polarity, from a given review. as depicted in figure 1 , an example of the sentiment triplet is ("curly cord", "hate", neg), representing a negative sentiment toward the aspect term "curly cord" using the opinion term "hate". the aste task requires a deep understanding of linguistic forms and structures (e.g., aspect terms are usually nouns or verbs used as subjects or objects in a sentence), as well as the ability to identify the relationships between the various linguistic components (e.g., how to pair the aspect terms and opinion terms) in a given text. prior aste methods @xcite have achieved promising results on exist-electronics i hate the curly cord . other than that , they re great . very light weight , perfect for recording and @xcite , great, pos), (curly cord, hate, neg), (mixing, perfect, pos), (recording, perfect, pos), (w i hate the curly cord . other than that , they' re great . ing academic datasets @xcite @xcite , greatly promoting the development and application of aste. however, the datasets employed in these studies remain comparatively uncomplicated, leading to disparities between these datasets and real-world settings in terms of various factors, such as length, expression diversity, domain distribution, etc. for instance, most reviews in existing datasets are of short length, with an average of 16 words per review, while reviews in real-world scenarios are longer (an average of more than 50 words). additionally, expressions used in these datasets are typically simple and straightforward, with limited diversity in lexicality and syntactic. furthermore, existing datasets typically contain two domains, i.e., restaurant and laptop, with very limited domain distributions. in a nutshell, these gaps hide the complexity of real-world scenarios, and therefore, impede the exploration to fully understand and address the challenges presented in real-world aste tasks. in order to bridge the gap and better simulate real-world scenarios, we create a new dataset, named diversified multi-domain aste (dmaste), which is manually annotated to provide a more diverse and realistic set of reviews for the task. as overall, the results of dmaste on multiple settings provide a deeper understanding of the challenges and future directions for aste research. we believe this work will bring a valuable research resource and benchmark for the community. ## dataset to construct a dataset that is more representative of real-world scenarios, we manually annotate a new dataset, named diversified multi-domain aste (dmaste). in this section, we first present a detailed description of the data collection and annotation process. then we demonstrate the superiority of dmaste, through a comparison with previous datasets in terms of key statistics and characteristics. ## experiment settings to thoroughly understand dmaste and provide some promising directions for future aste research, we conduct experiments in multiple settings. this section will first provide an overview of the different experimental setups, then introduce the evaluation metric, and finally, describe the models employed in the experiments. ## results this section presents thorough analyses of the challenges of the dmaste dataset and aste task, with the aim of better understanding these challenges and suggesting promising directions for future research. for each setting, we first show experimental results. then we perform comprehensive analyses to investigate the challenges of the dataset and task, highlighting the limitations of current approaches. finally, we provide promising directions for future research in this area. ## conclusion and future directions we propose dmaste, a manually-annotated aste dataset collected from amazon. compared with existing aste datasets, dmaste contains reviews of various lengths, diverse expressions, more aspect types and covers more domains, which indicates dmaste is a suitable testbed to verify the ability of aste methods in real-world scenarios. we explore the dataset in multiple scenarios, i.e., in-domain, single-source cross-domain, and multi-source cross-domain and provide some promising directions for future research. for the in-domain setting, we compare the results between existing datasets and dmaste and find that the long reviews, complex sentences, and implicit aspect terms make dmaste a challenging dataset. for the single-source cross-domain setting, we observe that domain similarity and cross-domain performance are positively correlated. furthermore, analysis of adversarial training shows that simply learning domain-invariant features may lead to feature collapse and result in the loss of task-specific knowledge. therefore, it is important to design appropriate methods to reduce the domain discrepancy while preserving fine-grained task features for aste tasks. in multi-source domain adaptation, we find that most of the negative transfer comes from dissimilar source-target pairs, pointing out that domain similarity can be a domain selection guideline for future research. in conclusion, we hope that our dataset dmaste and analyses will contribute to the promotion of aste research.