Multi-Domain Targeted Sentiment Analysis (2022)

Abstract: Targeted Sentiment Analysis (TSA) is a central task for generating insights from consumer reviews. Such content is extremely diverse, with sites like Amazon or Yelp containing reviews on products and businesses from many different domains. A real-world TSA system should gracefully handle that diversity. This can be achieved by a multi-domain model – one that is robust to the domain of the analyzed texts, and performs well on various domains. To address this scenario, we present a multi-domain TSA system based on augmenting a given training set with diverse weak labels from assorted domains. These are obtained through self-training on the Yelp reviews corpus. Extensive experiments with our approach on three evaluation datasets across different domains demonstrate the effectiveness of our solution. We further analyze how restrictions imposed on the available labeled data affect the performance, and compare the proposed method to the costly alternative of manually gathering diverse TSA labeled data. Our results and analysis show that our approach is a promising step towards a practical domain-robust TSA system.

## introduction customer reviews of products and businesses provide insights for both consumers and companies. they help companies understand customer satisfaction or guide marketing campaigns, and aid consumers in their decision-making. sentiment analysis plays a central role in the analysis of such material, by aiming to understand the sentiment expressed in a review document or in a single review sentence @xcite . beyond these high-level trends, identifying the sentiment towards a specific product feature or an entity is important. such a fine-grained analysis includes the key task of targeted sentiment analysis (tsa), aimed at detecting sentiment-bearing terms in texts and classifying the sentiment towards them. for example, in the sentence "the room was noisy, but the food was tasty," the targets are room and food with negative and positive sentiments, respectively. our focus in this work is on tsa of user reviews in english. a real-world tsa system has to successfully process diverse data. from toothbrushes to phones, airline companies to local retailers, the online content today covers a broad range of reviews in many domains. ideally, a system for such a multi-domain scenario should be able to cope with inputs from any domain, those that were seen during training, and, perhaps more importantly, those that were not. to the best of our knowledge, this work is the first to pursue tsa in a multi-domain setup, intending to support input from multiple unknown domains. many previous works have used the indomain setup of training and testing on data from the same domain (e.g. @xcite ). newer works focus on the cross-domain setup, yet most have explored a pairwise evaluation of training on one source domain and evaluating on a single known target domain (e.g. @xcite ; @xcite ). broadly, multi-domain learning @xcite includes training and evaluation using data from multiple domains (e.g. @xcite ; @xcite ; @xcite ). sometimes, it is assumed that the input texts are accompanied by a domain label (e.g. @xcite ). here, we do not assume a domain label is given -this has the advantage of allowing easier practical use of our model, without having to specify the domain as part of the input. in other cases, evaluation is limited to domains represented in training, or otherwise performed in a zero-shot setup only on unseen domains @xcite . our system handles both cases simultaneously, processing data from domains well-represented in the training data as well as from unseen domains. for practical reasons, implementing a multidomain system with a single model that can handle all domains is desirable. this can save valuable resources such as memory or gpus, which are in high demand by contemporary language models (lms). for example, it is impractical to expect that an online service providing tsa analysis will have a per-domain model, each keeping its many parameters in memory, along with perhaps a set of pre-allocated gpus. our goal is therefore to have a single multi-domain model that performs well on both seen and unseen domains. this is reminiscent of works in multilingual nlp that develop a single model that handles multiple languages (e.g. m-bert released by @xcite al. (2019), liang et al. (2020), toledo-ronen et al. (2020) ). a possible approach to our setting is training on a diverse tsa dataset, potentially encompassing many of the domains that the system is applied to. however, obtaining such a dataset is a challenge. the existing tsa datasets are limited in their diversity, and the collection of a new large scale diverse tsa dataset is complex @xcite . the road we take is therefore based on augmenting a tsa dataset of limited diversity with assortment of weak labels, through self-training -one of the earliest ideas for utilizing unlabeled data in training @xcite . to show that our approach is feasible, we performed an extensive empirical evaluation with several lms that were fine-tuned with labeled data from the semeval dataset of @xcite (henceforth se). this dataset is limited to two domains: restaurants or laptops. each model went through several selftraining iterations and evaluated on three tsa publicly available datasets: se, the mams dataset of restaurant reviews @xcite , and the yaso dataset of open-domain reviews @xcite . as part of our evaluation, we created two new tsa resources. the first is an annotation layer on top of the yaso dataset, specifying the domain of each review. this allows a per-domain evaluation providing insights on the performance of seen and unseen domains. the second resource is a set of manually annotated tsa reviews, which can be an ad-hoc diverse tsa training set, an alternative to the proposed method. we show that even in the presence of such data in training our approach is valuable. both resources are available online. 1 in summary, the main contributions of this work are: (i) the first exploration of tsa in a multi-1 github.com/ibm/yaso-tsa domain setup; (ii) demonstrating the feasibility of multi-domain tsa by an extensive evaluation on three datasets and the use of self-training; (iii) the release of additional tsa resources: a new annotation layer for the yaso dataset, and a set of fully annotated reviews. ## method our self-training approach augments a given tsa training set with weak-labels (wl) generated from a large multi-domain corpus. the process (depicted in figure 1 ) starts by training an initial tsa model on that given training data. then, that model produces tsa predictions on a large unlabeled corpus of diverse reviews. finally, some of the predictions are selected and added as weak labels to the original training set. a new model is then trained with the augmented data, applied to produce new predictions on the unlabeled data, and the whole process (detailed below) can repeat for several iterations. ## manual error analysis the automatic evaluation reported above is based on exact-span matches, and may be too strict in some cases. for example, in "the best thing about this place is the different sauces," the yaso labeled data contains the target the different sauces, thus counting a prediction of sauces as an error. alternative evaluation options may circumvent this problem. for example, the above prediction would be considered as correct using overlapping span matches. however, changing the automatic evaluation can introduce new issues and and may be too lenient. continuing the above example, with an overlapping span match, a prediction of the entire sentence is also considered as correct. due to these issues, we complement the automatic evaluation with a manual one, comparing the output of an initial ld model to its self-trained counterpart. the error analysis was performed on one experimental setup, with the bert-mlm pretrained model and the entire se dataset for training the ld model. we further focus on the yaso dataset: for each model, 30 predictions considered as errors by the automatic evaluation were randomly sampled from each of the 6 domains. one of the authors categorized these predictions into one of four options: invalid target, correct target identified with wrong sentiment or span, borderline target that can be accepted, and a clearly correct target. the latter are presumably due to the strictness of the exact-matches based evaluation. yaso mams se lm train set p r f1 p r f1 p r f1 ## conclusion this work addressed a multi-domain tsa setting in which a system is trained on data from a small number of domains, and is applied to texts from any domain. our proposed method has employed self-learning to augment an existing tsa dataset with weak labels obtained from a large corpus. an empirical evaluation of our approach has demonstrated that the self-supervision technique, often used when having a training set of limited size, is also effective for enhancing the diversity of the training data. specifically, our results show that the self-trained multi-domain model consistently improves performance, for various underlying lms, and with different starting points: data from two domains, removing half of the data, or restricting to only one domain. interestingly, even in the presence of a diverse tsa labeled data, our approach was comparable to the performance obtained with that data. this allows avoiding the burden and costs associated with manual tsa data collection. in addition to finding targets and their sentiments, other related tasks aim to extract the corresponding opinion term @xcite , identify the relevant aspect category @xcite , or both @xcite . as future work, our approach may be applied to these more complex tasks as well. similarly, it may be useful for developing a multilingual tsa system, by utilizing weak labels produced on unlabeled reviews data in non-english languages.