N ews MTSC : A Dataset for (Multi-)Target-dependent Sentiment Classification in Political News Articles (2021)

Abstract: Previous research on target-dependent sentiment classification (TSC) has mostly focused on reviews, social media, and other domains where authors tend to express sentiment explicitly. In this paper, we investigate TSC in news articles, a much less researched TSC domain despite the importance of news as an essential information source in individual and societal decision making. We introduce NewsMTSC, a high-quality dataset for TSC on news articles with key differences compared to established TSC datasets, including, for example, different means to express sentiment, longer texts, and a second test-set to measure the influence of multi-target sentences. We also propose a model that uses a BiGRU to interact with multiple embeddings, e.g., from a language model and external knowledge sources. The proposed model improves the performance of the prior state-of-the-art from F1_m=81.7 to 83.1 (real-world sentiment distribution) and from F1_m=81.2 to 82.5 (multi-target sentences).

## introduction previous research on target-dependent sentiment classification (tsc, also called aspect-term sentiment classification) and related tasks, e.g., aspectbased sentiment classification (absc, or aspectcategory sentiment classification) and stance detection, has focused mostly on domains in which authors tend to express their opinions explicitly, such as reviews and social media @xcite @xcite @xcite @xcite . we investigate tsc in news articles -a much less researched domain despite its critical relevance, especially in times of "fake news," echo chambers, and news ownership centralization @xcite . how persons are portrayed in news on political topics is very relevant, e.g., for individual and societal opinion formation @xcite . we define our problem statement as follows: we seek to detect polar judgments towards target persons @xcite . following the tsc literature, we include only in-text, specifically insentence, means to express sentiment. in news texts such means are, e.g., word choice and generally framing foot_0 @xcite , e.g., "freedom fighters" vs. "terrorists," or describing actions performed by the target, and indirect sentiment through quoting another person @xcite . other means may also alter the perception of persons and topics in the news, but are not in the scope of the task @xcite , e.g., because they are not on sentence-level. for example, story selection, source selection, article's placement and size @xcite , and epistemological bias @xcite . the main contributions of this paper are: (1) we introduce newsmtsc, a large, manually annotated dataset for tsc in political news articles. we analyze the quality and characteristics of the dataset using an on-site, expert annotation. because of its fundamentally different characteristics compared to previous tsc datasets, e.g., as to sentiment expressions and text lengths, newsmtsc represents a challenging novel dataset for the tsc task. ( 2) we propose a neural model that improves tsc performance compared to prior state-of-the-art models. additionally, our model yields competitive performance on established tsc datasets. (3) we perform an extensive evaluation and ablation study of the proposed model. among others, we investigate the recently claimed "degeneration" of tsc to sequence-level classification @xcite finding a performance drop in all models when comparing single-and multi-target sentences. in a previous short-paper, we explored the characteristics of how sentiment is expressed in news articles by creating and analyzing a small-scale tsc dataset @xcite . the paper at hand addresses our former exploratory work's critical findings, including essential improvements to the dataset. key differences and improvements are as follows. we significantly increase the dataset's size and the number of annotators per example and address its class imbalance. further, we devise annotation instructions specifically created to capture a broad spectrum of sentiment expressions specific to news articles. in contrast, the early dataset misses the more implicit sentiment expressions commonly used by news authors @xcite . also, we comprehensively test various consolidation strategies and conduct an expert annotation to validate the dataset. we provide the dataset and code to reproduce our experiments at: https://github.com/fhamborg/newsmtsc ## related work analogously to other nlp tasks, the tsc task has recently seen a significant performance leap due to the rise of language models @xcite . pre-bert approaches yield up to f 1 @xmath0.3 on the semeval 2014 twitter set @xcite . they employ traditional machine learning combining hand-crafted sentiment dictionaries, such as sentiwordnet @xcite , and other linguistic features @xcite . on the same dataset, vanilla bert (also called bert-spc) yields 73.6 @xcite . specialized downstream architectures improve performance further, e.g., lcf-bert yields 75.8 @xcite . the vast majority of recently proposed tsc approaches employ bert and focus on devising specialized down-stream architectures @xcite @xcite . more recently, to improve performance further, additional measures have been proposed. for example, domain adaption of bert, i.e., domain-specific language model finetuning prior to the tsc finetuning @xcite ; use of external knowledge, such as sentiment or emotion dictionaries @xcite , rule-based sentiment systems @xcite , and knowledge graphs @xcite ; use of all mentions of a target and/or related targets in a document @xcite ; and explicit encoding of syntactic information @xcite . to train and evaluate recent tsc approaches, three datasets are commonly used: twitter @xcite @xcite , laptop and restaurant @xcite . these and other tsc datasets @xcite suffer from at least one of the following shortcomings. first, implicitly or indirectly expressed sentiment is rare in them. in their domains, e.g., social media and reviews, typically authors explicitly express their sentiment regarding a target @xcite . second, they largely neglect that a text may contain coreferential mentions of the target or mentions of different concepts (with potentially different polarities), respectively @xcite . texts in news articles differ from reviews and social media in that news authors typically do not express sentiment toward a target explicitly (exceptions include opinion pieces and columns). instead, journalists implicitly or indirectly express sentiment (section 1) because language in news is typically expected to be neutral and journalists to be objective @xcite @xcite . our problem statement (section 1) is largely identical to prior news tsc literature @xcite with key differences: we do not generally discard the "author-" and "reader-level." doing so would neglect large parts of sentiment expressions. thus, it would degrade real-world performance of the resulting dataset and models trained on it. for example, word choice (listed as "author-level" and discarded from their problem statement) is in our view an in-text means that may in fact strongly influence how readers perceive a target, e.g., "freedom fighters" or "terrorists." while we do not exclude their "reader-level," we do seek to exclude polarizing or contentious cases, where no uniform answer can be found in a set of randomly selected readers (sections 3.3 and 3.4). as a consequence, we generally do not distinguish between the three levels of sentiment ("author," "reader," and "text") in this paper. previous news tsc approaches mostly employ sentiment dictionaries, e.g., created manually @xcite or extended semi-automatically @xcite , but yield poor or even "useless" @xcite performances. to our knowledge, there exist two datasets for evaluation of news tsc methods @xcite , which -perhaps due to its small size (@xmath1. recently, @xcite proposed a dataset (@xmath2. the dataset suffers from various shortcomings, particularly its small size, class imbalance, and lacking the more ambiguous and implicit types of sentiment expressions described above. another dataset contains quotes extracted from news articles, since quotes more likely contain explicit sentiment (@xmath3. ## newsmtsc: dataset creation in creating the dataset, we rely on best practices reported in literature on the creation of datasets for nlp @xcite , especially for the tsc task @xcite . compared to previous tsc datasets though, the nature of sentiment in news articles requires key changes, especially in the annotation instructions and consolidation of answers @xcite . ## methodology the goal of tsc is to find a target's polarity @xmath4., neu., neg.} in a sentence. our model consists of four key components (figure 1 ): a pretrained language model (lm), a representation of external knowledge sources (eks), a target mention mask, and a bidirectional gru (bigru) @xcite . we adapt our model from @xcite and change the design as follows: we employ a target mask (which they did not) and use multiple eks simultaneously (instead of one). further, we use a different set of eks (section 5) and do not exclude the lm's parameters from finetuning. ## error analysis to understand the limitations of gru-tsc, we carry out a manual error analysis by investigating a random sample of 50 incorrectly predicted examples for each of the test sets. for test-rw, we find the following potential causes (not mutually exclusive): edge cases with very weak, indirect, or in part subjective sentiment (22%) or where both the predicted and true sentiment can actually be considered correct (10%); sentiment of given target confused with different target (14%). further, sentence's sentiment is unclear due to missing context (10%) and consolidated answer in newsmtsc is wrong (10%). in 16% we find no apparent reason. for test-mt, potential causes occur approximately similarly often except that targets are confused more often (20%). ## future work we identify three main areas for future work. the first area is related to the dataset. instead of consolidating multiple annotators' answers during the dataset creation, we propose to test to integrate the label selection into the model @xcite . integrating the label selection into the machine learning part could improve the classification performance. it could also allow us to include more sentences in the dataset, especially the edge cases that our restrictive consolidation currently discards. to improve the model design, we propose to design the model specifically for sentences with multiple targets, for example, by classifying multiple targets in a sentence simultaneously. while we early tested various such designs, we did not report them in the paper due to their comparably poor performances. further work in this direction should perhaps also focus on devising specialized loss functions that set multiple targets and their polarity into relation. lastly, one can improve various technical details of gru-tsc, e.g., by testing other interaction layers, such as lstms, or using layer-specific learning rates in the overall model, which can increase performance @xcite . ## conclusion we present newsmtsc, a dataset for targetdependent sentiment classification (tsc) on news articles consisting of 11.3k manually annotated examples. compared to prior tsc datasets, it is different in key factors, such as its texts are on average 50% longer, sentiment is expressed explicitly only rarely, and there is a separate test set for multi-target sentences. as a consequence, state-of-the-art tsc models yield non-optimal performances. we propose gru-tsc, which uses a bidirectional gru on top of a language model (lm) and other embeddings, instead of masking or weighting mechanisms as employed by prior stateof-the-art. we find that gru-tsc achieves superior performances on newsmtsc and is competitive on prior tsc datasets. roberta yields best results compared to using bert, because roberta is pre-trained on news and we find it can better resolve in-sentence relations of multiple targets.