SSP -Based Construction of Evaluation-Annotated Data for Fine-Grained Aspect-Based Sentiment Analysis (2022)

Abstract: We report the construction of a Korean evaluation-annotated corpus, hereafter called ‘Evaluation Annotated Dataset (EVAD)’, and its use in Aspect-Based Sentiment Analysis (ABSA) extended in order to cover e-commerce reviews containing sentiment and non-sentiment linguistic patterns. The annotation process uses Semi-Automatic Symbolic Propagation (SSP). We built extensive linguistic resources formalized as a Finite-State Transducer (FST) to annotate corpora with detailed ABSA components in the fashion e-commerce domain. The ABSA approach is extended, in order to analyze user opinions more accurately and extract more detailed features of targets, by including aspect values in addition to topics and aspects, and by classifying aspect-value pairs depending whether values are unary, binary, or multiple. For evaluation, the KoBERT and KcBERT models are trained on the annotated dataset, showing robust performances of F1 0.88 and F1 0.90, respectively, on recognition of aspect-value pairs.

## introduction aspect-based sentiment analysis (absa) is a sentiment analysis technique which extracts users' opinions on the basis of opinion quintuples consisting of an entity or target, an aspect, a sentiment value, an opinion holder, and a time @xcite . since the opinion holder and time can be identified by meta-information sources such as user ids and content posting time, opinion triples including an entity (e), an aspect (a), and a sentiment value (s) are the main components to be extracted from texts associating a semantic orientation with aspects of a target product or service. for example, in (1), the opinion triple plays a vital role in identifying a user's 'positive' (s) sentiment attributed to the 'design' (a) aspect of a 'jacket' (e) target. ## related studies as several deep learning models have been used to perform absa, many researchers have developed datasets to train the models. @xcite points out that in existing benchmark data, most examples are not significantly different from sentence-level sentiment analysis because they contain only one aspect, or all aspects have the same sentiment polarity. therefore, the study introduces mams, a dataset for more sophisticated absa, allowing all examples to contain two or more polarities. the dataset is extracted from the same corpus as the existing benchmark data. @xcite notices that existing sentiment analysis studies use data from a limited range of domains, and presents yaso, a dataset from yelp, amazon, sst @xcite , and opinosis @xcite for absa learning and evaluation. the yaso dataset covers diverse domains and allows for measuring cross-domain performance. however, it does not provide meta-information on the domains of the reviews. other studies define new problems for absa. for example, @xcite points out that it is essential to find which words convey the sentiment related to a specific aspect. therefore, a task called towe is defined and a dedicated dataset is provided. @xcite uses towe and semeval data and argues that complete absa requires pairing aspects with the corresponding 'opinion terms', i.e. the 'why' expressions that give a clue of why the aspect has the polarity expressed in the sentence, e.g. friendly in waiters/friendly. these 'opinion terms' describe a value of the aspect. in some state-of-the-art datasets, expressions denoting aspect values are annotated, but only when they also convey sentiment information by themselves, as in waiters/friendly. our study differs from that practice in that it includes in its subject non-sentiment information about aspect values, when that information is relevant to sentiment analysis. ssp is a method of generating learning datasets by annotating selected expressions in collected data such as reviews, discussions or questions in the relevant domain in social media. this technology aims to enhance learning data with the aid of sophisticated, large-scale language resources. once the first dataset is semiautomatically annotated, the language resources are edited based on this first result and applied to the corpus. this bootstrapping approach @xcite is applied to the first and second versions of the annotated dataset, producing a third version @xcite . ## method with evaluation triples in section 3, we propose a method of sentiment analysis that focuses on evaluation triples (et) and that we have applied to clothing reviews. ets allow for classifying purchasers' objective or subjective evaluations on aspects such as color, material, pattern..., no matter whether the evaluation is expressed with or without an explicit sentiment polarity. an et comprises a topic, an aspect, and a value. in the clothing domain, a topic is an evaluation target such as 티셔츠 (thisyechu) 't-shirts' or 바지 (paci) 'pants'. an aspect of an evaluation target can be 가격 (kakyek) 'price' or 색 (sayk) 'color'. a value indicates a purchaser's specific evaluation of a topic or aspect. the main characteristic of ets compared with traditional absa opinion triples is that values include not only single words and mwes with a sentiment polarity, but also non-sentiment expressions representing purchasers' evaluations about each aspect. examples of non-sentiment expressions include 길어요 (kileyo) 'it is long' and 검정색이에요 (kemcengsaykieyyo) 'it is black'. the following sections describe the individual characteristics of each element of ets in the clothing domain. ## resource construction we built linguistic resources using the ssp method, which annotates various language patterns in data with the aid of lggs. we utilized lexical databases (deco-dom) to specify the language patterns in lggs. the main advantage of the ssp methodology is that it produces training data efficiently in terms of time and cost as compared to crowdsourcing, which has been widely used to create large-scale training data. the following sections provide further details. ## evaluation of the evad dataset to evaluate the performance of the evad dataset, and indirectly of the ssp method used to generate it, we experimented about aspect-value pairs as they are the core of et. there are frequent aspect-value pairs appearing in the review corpus, such as size-large, fabric-thin, or price-good. for these categories, we evaluated the annotation of evad. we used 1,000 sentences extracted from the clothing reviews, manually annotated them with the correct tags, and evaluated the automatically annotated evad sentences by comparing them with the test set. the results in ## experiment we used the kobert foot_1 and kcbert 4 models in an experiment of training them on evad to recognize aspect-value pairs. these models are suitable for this task since they analyze sequences in syllable units. the pre-trained language models kobert and kcbert were used to train the model. then, the performance of the models was evaluated against the manually annotated test set (1,000 sentences) of section 5. the plms show f1-scores of 88% and 90%, respectively. the reason why kcbert performance is slightly better than kobert is that its pre-trained data comprises informal texts such as news comments. ## conclusion this study proposes an application of the evaluation triple (et) concept in a clothing domain, and the construction of a linguistic resource, evad. the et consists of three components: topic, aspect, and value. the first two are components of traditional absa, whereas figure 5: examples of annotated text @xmath0. <color-@xmath1. <price-@xmath2. <@xmath3. recall precision f1-score evad 0.8804 0.8739 0.8772 the value is introduced to extract detailed information about aspects. we classify 79 aspectvalue pairs described from the clothing domain into three types: unary, binary, and multiple. evad is generated with the ssp method and shows a robust f1-score performance of 90%. ssp-based linguistic resources using an et frame can be applied to various domains. we expect that the concept and method implemented in this study will be a key asset for research on constructing sophisticated annotated training data for deep learning language models.