Author’s Sentiment Prediction (2020)

Abstract: Even though sentiment analysis has been well-studied on a wide range of domains, there hasn’tbeen much work on inferring author sentiment in news articles. To address this gap, we introducePerSenT, a crowd-sourced dataset that captures the sentiment of an author towards the mainentity in a news article. Our benchmarks of multiple strong baselines show that this is a difficultclassification task. BERT performs the best amongst the baselines. However, it only achievesa modest performance overall suggesting that fine-tuning document-level representations aloneisn’t adequate for this task. Making paragraph-level decisions and aggregating over the entiredocument is also ineffective. We present empirical and qualitative analyses that illustrate thespecific challenges posed by this dataset. We release this dataset with 5.3k documents and 38kparagraphs with 3.2k unique entities as a challenge in entity sentiment analysis.

## introduction how can we automatically infer the sentiment of an author towards an entity based only on the text of their news article? this task can be seen as a part of complete document understanding, with potential uses in detecting journalistic bias, and in collecting articles that express certain viewpoints towards entities. there are no readily useful datasets or effective solutions for author sentiment inference in the news domain. sentiment analysis solutions have covered a wide range of domains including movie reviews @xcite @xcite , product reviews @xcite @xcite , and social media @xcite . in the news domain, the closest is inferring the sentiment expressed by one entity (or a group) towards another entity (or a group) mentioned in a news article @xcite . however, this does not necessarily cover the sentiment expressed by the author towards a specific entity. to address this gap, we introduce persent, a crowdsourced dataset of sentiment annotations on news articles about people. for each article, annotators judge what the author's sentiment is towards the main (target) entity of the article. the annotations also include similar judgments on paragraphs within the article. our experiments with multiple strong classification models show that this is a difficult task that introduces multiple unmet challenges. the task we propose is difficult mainly because most information contained in a news article is likely irrelevant for the purposes of inferring the author's sentiment. this is a key difference compared to previously studied sentiment problems in domains such as product or movie reviews, where most information is likely relevant for sentiment inference. on the one hand, a single global document-level representation requires careful aggregation of information related to the target entity that is relevant for sentiment inference. on the other, it is also not easy to make local decisions (say at a paragraph level) and aggregate them -in many cases paragraph-level decisions can be noisy due to discourse gaps, and furthermore not all paragraph-level decisions should contribute equally to the final decision. indeed neither approach works satisfactorily. for instance, fine-tuning pre-trained bert @xcite base model only yields an f1 of 0.48 on one test set and 0.42 on the more difficult test set for the document-level task. paragraph-level prediction using bert is even more difficult, with 0.43 and 0.40 f1 values for the two test sets respectively (see section 3.4 for more details). motivated by the nature of the task -requiring focus towards the target entity and aggregating information across the discourse -we also benchmark models that focus on entity-oriented representations (recurrent entity network), discourse-based representations. however, we find that their performance is worse than the pre-trained bert classifier. we conduct further analysis to show the nature of the challenges introduced by this news domain dataset. one of the main challenges in this task is as shown in figure 1 , not all paragraphs necessarily convey the author's sentiment towards the target entity. thus, making a single decision from a representation of the entire document can be ineffective. furthermore, our analysis shows that prediction in documents with many unique entities is more difficult. we also find that it is easier to predict documentlevel sentiment when there are many paragraphs that convey the same sentiment. last, we also provide a qualitative error analysis that indicates the main categories of inference errors. in summary, this work introduces a new sentiment inference task in the news domain that introduces new challenges in document-level sentiment inference along with a dataset that will support further research in this area. ## persent dataset given a news article about a person (we refer to this as the main or target entity), the author sentiment prediction task is to infer the authors' view towards the main entity. this task requires models to carefully track information that is specific to the entity and aggregate sentiment related information across multiple mentions of the entity in the document. since there are no datasets that directly support this task of author sentiment prediction, we create a new dataset persent. it consists of news articles that have been previously used for other types of sentiment analysis and related tasks. the criteria for choosing a document is that the news article should mainly talk about a person entity and that it should not be very long or short. we used three resources to select news articles: we used the following pre-processing steps on the collected news articles. first we find all the person entities in each article, using stanford ner (name entity resolution) tagger @xcite and all mentions of them using co-reference resolution @xcite . we removed articles which are not likely to have a main entity of focus. we used a simple heuristic of removing articles in which the most frequent person entity is mentioned only three times or less (even when counting co-referent mentions). for the articles that remain we deemed the most frequent entity to be the main entity of the article. we also filtered out extremely long and extremely short articles to keep the articles which have at least 3 paragraphs and at most 16 paragraphs. documents are randomly separated into train, dev, and two test sets. we ensure that each entity appears in only one of the sets. our goal here is to avoid easy to learn biases over entities. to avoid the most frequent entities from dominating the training or the test sets, we remove articles that covered the most frequent entities and use them as a separate test set (referred to as frequent test set) in addition to the randomly drawn standard test set. ## evaluation we benchmark the performance of strong classification baselines to understand and highlight the difficulty of this task. ## conclusions & discussion in this work, we introduced author sentiment inference in the news domain, a new sentiment task and release a challenge dataset. most information contained in a news article might be irrelevant for the purposes of inferring the authors sentiment towards the target entity. this makes the task quite challenging for both models that operate over global representation and for models that aggregate local decisions. we hope the task and the dataset will be a useful resource for the community to build more effective text classification models.