YASO : A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews (2021)

Abstract: Current TSA evaluation in a cross-domain setup is restricted to the small set of review domains available in existing datasets. Such an evaluation is limited, and may not reflect true performance on sites like Amazon or Yelp that host diverse reviews from many domains. To address this gap, we present YASO – a new TSA evaluation dataset of open-domain user reviews. YASO contains 2,215 English sentences from dozens of review domains, annotated with target terms and their sentiment. Our analysis verifies the reliability of these annotations, and explores the characteristics of the collected data. Benchmark results using five contemporary TSA systems show there is ample room for improvement on this challenging new dataset. YASO is available at https://github.com/IBM/yaso-tsa .

## introduction targeted sentiment analysis (tsa) is the task of identifying the sentiment expressed towards single words or phrases in texts. for example, given the sentence "it's a useful dataset with a complex download procedure" the desired output is identifying dataset and download procedure, with a positive and negative sentiments expressed towards them, respectively. our focus in this work is on tsa of user reviews data in english. till recently, typical tsa evaluation was indomain, for example, by training on labeled restaurant reviews and testing on restaurant reviews. new works (e.g. @xcite ) began considering a cross-domain setup, training models on labeled data from one or more domains (e.g., restaurant reviews) and evaluating on others (e.g., laptop reviews). for many domains, such as car or book reviews, tsa data is scarce or non-existent. this suggests that cross-domain experimentation is more realistic, as it aims at training on a small set of labeled domains and producing predictions for reviews from any domain. naturally, the evaluation in this setup should resemble real-world content from sites like amazon or yelp that host reviews from dozens or even hundreds of domains. 1 existing english tsa datasets do not facilitate such a broad evaluation, as they typically include reviews from a small number of domains. for example, the popular semeval (se) datasets created by @xcite @xcite (henceforth se14, se15, and se16, respectively), contain english reviews of restaurants, laptops and hotels (see §2 for a discussion of other existing datasets). to address this gap, we present yaso, foot_1 a new tsa dataset collected over user reviews taken from four sources: the yelp and amazon @xcite datasets of reviews from those two sites; the stanford sentiment treebank (sst) movie reviews corpus @xcite ; and the opinosis dataset of reviews from over 50 topics @xcite . to the best of our knowledge, while these resources have been previously used for sentiment analysis research, they were not annotated and used for targeted sentiment analysis. the new yaso evaluation dataset contains 2215 annotated sentences, on par with the size of existing test sets (e.g., one of the largest is the se14 test set, with 1,600 sentences). the annotation of open-domain reviews data is different from the annotation of reviews from a small fixed list of domains. ideally, the labels would include both targets that are explicitly mentioned in the text, as well as aspect categories that are implied from it. for example, in "the restaurant serves good but expensive food" there is a sentiment towards the explicit target food as well as towards the implied category price. this approach of aspect-based sentiment analysis @xcite is implemented in the se datasets. however, because the categories are domain specific, the annotation of each new domain in this manner first requires defining a list of relevant categories, for example, reliability and safety for cars, or plot and photography for movies. for open-domain reviews, curating these domain-specific categories over many domains, and training annotators to recognize them with per-domain guidelines and examples, is impractical. we therefore restrict our annotation to sentiment-bearing targets that are explicitly present in the review, as in the annotation of open-domain tweets by @xcite . while some information is lost by this choice, which may prohibit the use of the collected data in some cases, it offers an important advantage: the annotation guidelines can be significantly simplified. this, in turn, allows for the use of crowd workers who can swiftly annotate a desired corpora with no special training. furthermore, the produced annotations are consistent across all domains, as the guidelines are domain-independent. tsa annotation in a pre-specified domain may also distinguish between targets that are entities (e.g., a specific restaurant), a part of an entity (e.g., the restaurant's balcony), or an aspect of an entity (e.g., the restaurant's location). for example, @xcite use this distinction to exclude targets that represent entities from their annotation. in an open-domain annotation setup, making such a distinction is difficult, since the reviewed entity is not known beforehand. consequently, we take a comprehensive approach and annotate all sentiment-bearing targets, including mentions of reviewed entities or their aspects, named entities, pronouns, and so forth. notably, pronouns are potentially important for the analysis of multi-sentence reviews. for example, given "i visited the restaurant. it was nice.", identifying the positive sentiment towards it allows linking that sentiment to the restaurant, if the coreference is resolved. technically, we propose a two-phase annotation scheme. first, each sentence is labeled by five annotators that should identify and mark all target candidates -namely, all terms to which sentiment is expressed in the sentence. next, each target candidate, in the context of its containing sentence, is labeled by several annotators who determine the sentiment expressed towards the candidate -either positive, negative, or mixed (if any). 3 the full scheme is exemplified in figure 1 . we note that this scheme is also applicable to general non-review texts (e.g., tweets or news). several analyses are performed on the collected data: (i) its reliability is established through a manual analysis of a sample; (ii) the collected annotations are compared with existing labeled data, when available; (iii) differences from existing datasets are characterized. lastly, benchmark performance on yaso was established in a cross-domain setup. five state-of-the-art (sota) tsa systems were reproduced, using their available codebases, trained on data from se14, and applied to predict targets and their sentiments over our annotated texts. in summary, our main contributions are (i) a new domain-independent annotation scheme for collecting tsa labeled data; (ii) a new evaluation dataset with target and sentiment annotations of 2215 opendomain review sentences, collected using this new scheme; (iii) a detailed analysis of the produced annotations, validating their reliability; and (iv) reporting cross-domain benchmark results on the new dataset for several sota baseline systems. all collected data are available online. 4 2 related work review datasets the darmstadt review corpora @xcite ) contains annotations of user reviews in two domains -online universities and online services. later on, se14 annotated laptop and restaurant reviews (henceforth se14-l and se14-r). in se15 a third domain (hotels) was added, and se16 expanded the english data for the two original domains (restaurants and laptops). @xcite created a challenge dataset with multiple targets per-sentence, again within the restaurants domain. @xcite annotated opinions from discussions on urban neighbourhoods. clearly, the diversity of the reviews in these datasets is limited, even when taken together. ## input data the input data for the annotation was sampled from the following datasets: -yelp: 5 a dataset of 8m user reviews discussing more than 200k businesses. the sample included 129 reviews, each containing 3 to 5 sentences with a length of 8 to 50 tokens. the reviews were sentence split, yielding 501 sentences. overall, the input data includes reviews from many domains not previously annotated for tsa, such as books, cars, pet products, kitchens, movies or drugstores. further examples are detailed in appendix a. the annotation input also included 200 randomly sampled sentences from the test sets of se14-l and se14-r (100 per domain). such sentences have an existing annotation of targets and sentiments, which allows a comparison against the results of our proposed annotation scheme (see §5). ## yaso next, we detail the process of creating yaso. an input sentence was first passed through two phases of annotation, followed by several post-processing steps. figure 2 depicts an overview of that process, as context to the details given below. ## benchmark results recall the main purpose of yaso is cross-domain evaluation. the following results were obtained by training on data from se14 (using its original training sets), and predicting targets over yaso sentences. the results are reported for the full tsa task, and separately for the te and sc subtasks. baselines the following five recently proposed tsa systems were reproduced using their available codebases, and trained on the training set of each of the se14 domains, yielding ten models overall. evaluation metrics as a pre-processing step, any predicted target with a span equal to the span of a target candidate annotated with low-confidence was excluded from the evaluation, since it is unclear what is its true label. the use of clusters within the evaluation requires an adjustment of the computed recall. specifically, multiple predicted targets contained within one cluster should be counted once, considering the cluster as one true positive. explicitly, a predicted target and a cluster are span-matched, if the cluster contains a valid target with a span equal to the span of the prediction (an exact span match). similarly, they are fully-matched if they are span-matched and their sentiments are the same. predictions that were not span-matched to any cluster were considered as errors for the te task (since their span was not annotated as a valid target), and those that were not fully-matched to any cluster were considered as errors for the full task. using span-matches, precision for the te task is the percentage of spanmatched predictions, and recall is the percentage of span-matched clusters. these metrics are similarly defined for the full task using full-matches. for sc, evaluation was restricted to predictions that were span-matched to a cluster. for a sentiment label l, precision is the percentage of fully-12 github.com/yangheng95/lcf-atepc 13 github.com/nlpwm-whu/racl 14 github.com/lixin4ever/bert-e2e-absa 15 github.com/lixin4ever/hast 16 github.com/xuuuluuu/ aspect-sentiment-classification matched predictions with sentiment l (out of all span-matched predictions with that sentiment); recall is the percentage of fully-matched clusters with sentiment l (out of all span-matched clusters with that sentiment). macro-f 1 (mf 1 ) is the average f 1 over the positive and negative sentiment labels (mixed was ignored since it was scarcely in the data, following @xcite ). our data release is accompanied by code for computing all the described evaluation metrics. results tsa task performance was lowest for sst, perhaps due to its domain of movie reviews, which is furthest of all datasets from the product reviews training data. interestingly, it was also the dataset with the lowest level of agreement among humans (see figure 3a ). the choice of the training domain is an important factor for most algorithms. this is notable, for example, in the te performance obtained for yelp: the gap between training on data from the laptops domain or the restaurants domain @xmath0. a likely cause is that the yaso data sampled from yelp has a fair percentage of reviews on food related establishments. future work may further use yaso to explore the impact of the similarity between the training and test domains, as well as develop new methods that are robust to the choice of the training domain. ## conclusion we collected a new open-domain user reviews tsa evaluation dataset named yaso. unlike existing review datasets, yaso is not limited to any particular reviews domain, thus providing a broader perspective for cross-domain tsa evaluation. benchmark results established in such a setup with contemporary tsa systems show there is ample headroom for improvement on yaso. yaso was annotated using a new scheme for creating tsa labeled data, that can be also applied to non-review texts. the reliability of the annotations obtained by this scheme has been verified through a manual analysis of a sample and a comparison to existing labeled data. one limitation of our scheme is that aspect categories with a sentiment implied from the reviews were excluded, since their annotation requires prespecifying the domain along with its associated categories. while this may limit research for some applications, the dataset is useful in many realworld use cases. for example, given a brand name, one may query a user reviews corpus for sentences containing it, and analyze the sentiment towards that brand in each sentence along with the sentiment expressed to other terms in these sentences. future work may improve upon the presented results by training on multiple domains or datasets, adapting pre-trained models to the target domains in an unsupervised manner (e.g., @xcite ), exploring various data augmentation techniques, or utilizing multi-task or weak-supervision algorithms. another interesting direction for further research is annotating opinion terms within the yaso sentences, facilitating their co-extraction with corresponding targets @xcite , or as triplets of target term, sentiment, and opinion term @xcite . all benchmark data collected in this work are available online. 17 we hope that these data will facilitate further advancements in the field of targeted sentiment analysis. biqing zeng, heng yang, ruyang xu, wu zhou, and xuli han. 2019. lcf: a local context focus mechanism for aspect-based sentiment classification. applied sciences, 9(16). lei zhang, shuai wang, and bing liu. 2018. deep learning for sentiment analysis: a survey. wiley interdisciplinary reviews: data mining and knowledge discovery, 8(4):e1253.