Fact vs. Opinion: the Role of Argumentation Features in News Classification (2020)

Abstract: A 2018 study led by the Media Insight Project showed that most journalists think that a clearmarking of what is news reporting and what is commentary or opinion (e.g., editorial, op-ed)is essential for gaining public trust. We present an approach to classify news articles into newsstories (i.e., reporting of factual information) and opinion pieces using models that aim to sup-plement the article content representation with argumentation features. Our hypothesis is thatthe nature of argumentative discourse is important in distinguishing between news stories andopinion articles. We show that argumentation features outperform linguistic features used previ-ously and improve on fine-tuned transformer-based models when tested on data from publishersunseen in training. Automatically flagging opinion pieces vs. news stories can aid applicationssuch as fact-checking or event extraction.

## introduction subjectivity in news reporting is rising in the recent years, especially in online-only publications @xcite . it was estimated that only 41% of publishers label their type of articles (e.g., editorial, review, analysis), and among those who label the types, there is a lack of consistency and clarity @xcite . a major finding of a 2018 study led by the media insight project showed that most journalists (nearly 80%) think that their news organizations should clearly mark what is news reporting and what is opinion/commentary in order to combat fake news and gain public trust @xcite . broadly, there are two types of news articles: 1) opinion articles written to present the opinion of the editor or board and aimed to persuade the readers with respect to a particular point of view, and 2) news stories, which aim to report factual news or events. given that the intent of opinion articles is persuasion, we hypothesize that one of the key difference between news stories and opinion articles rests in the discourse structure and, in particular, the argumentative and persuasive aspects of the article. figure 1 shows an example of a news story and an opinion article with two coarse-grained types of argumentative components highlighted (i.e., claims and premises). we can see that claims are more prevalent in the opinion article, while the news story contains more premises to support a small amount of claims. we study the predictive power of such coarse-grained argumentation features (claims and premises) for the task of news articles classification into news stories and opinion pieces. for short, we will refer to this binary task as news vs. opinion classification. to train our sentence-level argument component classification model @xcite , we use the corpus of editorial news labeled with argumentation strategies introduced by al @xcite . we compare our approach that uses argumentation features to models using discrete linguistic features from previous work @xcite and to document-level transformer-based models such as bert @xcite fine-tuned for the document-level news vs. opinion classification task. we focus in particular on the transferability of these classifiers, as this task is particularly sensitive to changes in topic or publishers. therefore, we train and test our models under two regimes. first, we train on articles from one publisher and test on articles from another publisher (including two different domains). for this, we used the dataset introduced by @xcite . second, we train on articles from multiple publishers and test on articles from an unseen publisher. we demonstrate gains of using argumentation features on both collections and on all modeling approaches, with a wider margin of improvement in the smaller data scenario (i.e., data from a single publisher is used in training). our contributions are two-fold: these models can be used to flag content to readers or fact-checkers who seek to check verifiable factual information and not personal opinions. ## related work subjectivity detection has been studied extensively in previous work, especially in the context of sentiment analysis (e.g., @xcite @xcite ). the majority of these approaches study subjectivity at the sentence level, while some previous work considered document-level classification on genres such as newspaper articles @xcite . detection of subjective language in newspaper articles focused on lexical features of subjectivity, while observing that subjective words also appear in objective contexts and vice versa @xcite @xcite . this lead to approaches that do not generalize across publishers or topics and therefore a contextualized view of the problem is necessary. @xcite focus on the task of document-level classification of news articles into the broad categories of opinion and news stories. opinion articles can be further split in multiple types such as: editorials -which express the opinion of journalist or the editorial board, op-eds -which expresses the opinion of a contributor, and letters-to-the-editor -which express the opinion of the readers to an editorial. @xcite showed that linguistic features such as sentiment, quotation, and use of personal pronouns achieve good performance in document-level classification. however, their heavily engineered features are not as robust against changes in topics and are expected to not generalize well to data from different publishers than the ones used in training, which we aim to test in our experiments. another approach to detect the types of newspaper articles is through argument mining. argument mining is a field concerned with finding argument structure in text from argument components (claim, premises) to relations (support, attack) @xcite . there are many argument mining corpora available on text from multiple genres @xcite @xcite . one relevant corpus for our task is the news editorial corpus of 300 articles from three publishers @xcite . the authors annotate argumentative strategies at the token level into one of six possible strategies. they report agreements and differences in argumentative writing between publishers. editorials across different publishers consist of a majority of assumptions (claims), while they employ evidence supporting strategies differently. this corpus has been used to analyze persuasion in editorials @xcite , and to study patterns of argumentative strategies across topics @xcite . we hypothesize that predicting argumentative strategies of newspaper articles is also useful in predicting the overall type of the article to be news or opinion. @xcite used argumentation to predict sentiment of reviews, however, to the best of our knowledge, there is no previous work on using argumentation features for news vs. opinion classification. ## data in our experiments on news vs. opinion classification, we use two data collections that aim to test the generalizability of the modeling approaches. details about sizes, publishers, and data set splits in both collections are shown in ## features we run our experiments on three feature sets testing all possible combinations among them. ## models we describe below the three models we use in our experiments, which include a machine learning model with discrete features (svm) and deep learning models (rnn and bert). svm. we use a support vector machine (svm) classifier with a linear kernel using scikit-learn implementation @xcite . the svm model can take as input the linguistic features, similar to the ones introduced by @xcite , the contextualized document representation generated by the bert model, argumentation features or any combination of these. argumentation features are represented as the distribution across the three classes (claims, premise, none) in a given article, since our hypothesis is that editorials tend to have a majority of claim sentences, while news articles tend to have a majority of premise or other sentence types. bert. the bert model is used to predict the type of article based on the [cls] token that represents each article. bert is implemented using the hugging face transformers library @xcite . we train for 3 epochs, use maximum sequence length of 512 tokens, a learning rate of 2e-5, and a batch size of 16 on the training sets from both data collections. rnn. we use a recurrent neural network (rnn) to bridge the gap between the sentence-level predictions for argumentation types and our document-level task of article-level classification. we hypothesize that the discourse relationships between the sentence-level predictions can be leveraged to improve classification, when compared to only using the distribution over types. for the rnn model, we use the argumentative labels of sentences as a sequence input to an rnn layer of size 128, with 20% dropout and 20% recurrent dropout for regularization. we pass the output of the rnn model to a softmax dense layer for prediction (figure 2a ). the input sequence to the rnn has a maximum length of 100 sentences, which covers more than 95% of the articles. rnn+bert. in addition, to modeling the document-level content, we also use the fine-tuned bert embeddings as input to a dense layer of size 128 with 50% dropout, and concatenate the output with the rnn layer then pass the concatenated layer to a final softmax layer. we introduce a dense layer with dropout after the bert embeddings, such that the bert and rnn output have equal layer sizes before concatenation. we denote this model as rnn+bert. the diagram of the model is presented in figure 2b . ## results the results of our experiments on the wsj-nyt collection are shown in tables 2 and 4 , while our results on the multi-publisher collection are shown in tables 3 and 5 . ## analysis of argumentation features to further understand the relation between argumentative types of sentences and the discourse structure of the articles, we study the frequency of claims and premises at each sentence position. figure 3 shows the number of times a claim (or a premise) is predicted at each sentence position normalized by the number articles that have this sentence position, e.g. sentence 30 shows the number of times it is classified as claim (or premise) divided by the number of articles of length 30 or more. these percentages are calculated on the first 40 sentences from the articles in the multi-publisher training data set, in order to limit variability caused by low counts. figure 3 shows that opinion articles tend to have a majority of claims and news articles tend to have a majority of premises, which explains the ability of features such as sentence types distributions to classify the article type as we show in section 6. in addition, we also see a trend in the opinion articles to contain less premises, and conversely slightly more claims, as the article progresses. this trend is much less pronounced in news stories. these trends indicate that editorial and news stories follow, in aggregate, distinct discourse patterns. these differences in base-rates justify why the svm model using aggregate counts is able to predict with good accuracy the type of article with only a few features. in addition, by modelling more complex discourse dynamics across sentences, the rnn is able to further improve performance when predicting documentlevel outcomes. editorials tend to have a majority of claims (assumptions) as mentioned by al @xcite , which is consistent with our results. however, we see in our results that news articles tend to have a majority of premises, which could be the case for some but not all news articles. we think our model could be overestimating the number of premises in news articles due to being trained strictly on data annotated from editorials. also, the training data has very small number of sentences from the 'non-argumentative' type and as a result this class is under-predicted by the sentence-level model. however, we believe that our predictions of sentence types are good estimates for article-level and possibly paragraph-level tasks, but a more balanced training data is needed to apply this approach for sentence-level tasks. ## conclusion in this paper we studied the role of argumentative discourse in news articles. we show that with a small corpus of annotated argumentative types of sentences in editorials, we can train a sentence classification model and use those argument component predictions to generate argumentation features useful for classifying articles into news or opinion. the argumentation features show the best generalization performance to new topics/domains and publishers when the model is trained on data of low variety. also, the argumentation features are largely able to further improve upon rich contextualized models trained on more data from multiple publishers for this specific document-level task. these results and our analysis on the frequency and distribution of argumentation features shows that there are distinctive discourse patterns related to claims and premises that are able to generalize well across publishers and topics. future work will expand on the use of discourse features to categorize news articles, with the goal of improving generalization of models across publishers and topics. discourse features can include finer-grained argumentative styles and other types of news discourse categories such as explanations, background, context, reactions and evidence @xcite . we will also aim to expand the types of articles studied beyond the two types and two subtypes explored in this paper.