Improving the Identification of the Discourse Function of News Article Paragraphs (2020)

Abstract: Identifying the discourse structure of documents is an important task in understanding written text. Building on prior work, we demonstrate an improved approach to automatically identifying the discourse function of paragraphs in news articles. We start with the hierarchical theory of news discourse developed by van Dijk (1988) which proposes how paragraphs function within news articles. This discourse information is a level intermediate between phrase- or sentence-sized discourse segments and document genre, characterizing how individual paragraphs convey information about the events in the storyline of the article. Specifically, the theory categorizes the relationships between narrated events and (1) the overall storyline (such as Main Events, Background, or Consequences) as well as (2) commentary (such as Verbal Reactions and Evaluations). We trained and tested a linear chain conditional random field (CRF) with new features to model van Dijk’s labels and compared it against several machine learning models presented in previous work. Our model significantly outperformed all baselines and prior approaches, achieving an average of 0.71 F1 score which represents a 31.5% improvement over the previously best-performing support vector machine model.

## introduction news articles usually follow strong principles of journalistic structure. by design, they often begin with a introductory summary of main events, followed by detailed exposition of the main events and consequences, interspersed in a stereotyped fashion with relevant background information, current and past evidence, and reported speech. @xcite demonstrated the feasibility of detecting this type of discourse structure for news articles using an established hierarchical theory of news discourse @xcite . in their study, they showed that it was feasible to identify the discourse function of news paragraphs using a support vector machine (svm) model and a small set of simple linguistic features, with a performance of 0.54 f 1 . similar to @xcite approach, we demonstrate an improved approach to automatically labeling news article paragraphs with the van dijk discourse functions @xcite applied in their study. our work uses a conditional random field (crf) model, along with new features, to obtain an improved performance of 0.71 f 1 . most importantly, our model is able to precisely capture the interdependencies between the various discourse label types, which flows from our hypothesis that each paragraph in an article is dependent not only on the previous one but rather on a longer sequence of previous paragraphs. the remainder of this paper is structured as follows. we first provide a definition of van dijk's theory as was presented in @xcite . second, we describe the dataset we used in training and testing our crf model ( §3). we then detail the discourse label identification methods, including the crf model and how it captures both section ordering and section content, how the model is trained, and the features it leverages ( §4). we next compare the performance of the crf model with various baselines, demonstrating that it performs better than prior models ( §5). we then discuss related work ( §6), and conclude with a summary of contributions ( §7). summary elements express the major subject of the article, with the headline being the actual headline of the article, and the lead being the first sentence, which is often a summary of the main events of the article. situation elements are the actual events that comprise the major subject of the article. episodes concern main events, which are those events that directly relate to the major subject of the article, and the consequences of those events. the background provides important information about the relation of each paragraph with respect to the central events of a news story. background includes the context, of which circumstances are temporally or spatially non-specific states that contribute to understanding the subject, while previous events are specific recent events that enhance understanding of the main events. history paragraphs are another type of background describing events that have not occurred recently, typically referenced in terms of years prior, rather than months, weeks, or days. comments provide further supporting context for the central events of an article. comments may include verbal reactions solicited from an external source, such as a person involved in the events, or an expert. conclusions, by contrast, are comments made by a journalistic entity (the newspaper, reporter, etc.) regarding the subject. conclusions can be separated into expectations about the resolution or consequences of an event, or evaluations of the current situation. ## dataset we used a gold-standard corpus previously developed by @xcite of van dijk's labels applied to a subset of the automated content extraction (ace) phase 2 corpus @xcite . the ace phase 2 corpus is a major standard corpora of news articles that boasts three advantages: it is widelyused, has relevance to other tasks, and was readily available to researchers. this dataset comprises 50 documents containing 28,236 words divided in 644 paragraphs. yarlott et al. (2018) doubly annotated 50 randomly selected news articles, divided into ten sets of five documents each. within these sets, documents were swapped or replaced in order to obtain uniform sets in terms of total document lengths. the majority of texts were already divided into paragraphs in an obvious manner, either with empty lines or with indentation. the remaining texts were divided by the adjudicator based on either contextual or structural clues, such as abrupt change in topic or unnatural line breaks. the authors report an all-around high agreement with the gold standard (f 1 = 0.85, κ = 0.75) which demonstrates that the gold-standard was not dominated by a single annotator. although the dataset discussed was annotated for all labels discussed here, the headline label could be computed automatically from the structure of ace phase 2 corpus, as the files has the headline separate as part of its markup scheme. label count label count headline 50 lead 42 main events 60 consequences 19 circumstances 103 previous events 64 history 27 verbal reactions 252 expectations 21 evaluations 56 ## identifying discourse labels in contrast to the approach reported by @xcite , we the treated label identification for paragraphs as a sequence modeling task. formally, the task is as follows: given a news report with n discourse labels and m paragraphs, where the paragraphs are unlabeled, identify the optimal sequence (order) of discourse labels h * = (l * 1 , . . . , l * n ) from among all possible label sequences, and assign every paragraph a discourse label h * = (h 1 , . . . , h m ) consistent with l * . sequence labeling problems in nlp, medical informatics, and discourse parsing have been studied by both generative and discriminative approaches, including hidden markov models (hmms; generative) and conditional random fields (crfs; discriminative). @xcite used hmm and n-gram models to detect the orders or labels of sections within clinical reports, while modeling the observation probabilities at the section level. @xcite used hmms as well as n-gram models to detect topic shifts in meeting minutes, and, in contrast to @xcite , modeled the observation probabilities on the sentence level. our approach was inspired by the method described in @xcite which identifies section labels in clinical psychiatric reports. their approach combined a hierarchical hidden markov model (hhmm)-which used section statistics as the model's transition probabilities-with n-grams for the observation probabilities of words. in this paper we substitute a crf for the hhmm. generative models such as hmms have more explanatory power when compared with their discriminative counterparts such as crfs. however, hmms, rely on the assumption that observations are statistically independent from one another. for our problem, this means that an hmm assumes that the presence of certain paragraphs corresponding to a certain discourse label or function a is independent from other paragraphs within another section b. in practice, however, this is not the case: for example a paragraph following the main events are often either consequences or circumstances. ## results and discussion in order to test our model, we randomly split each corpus into training and testing sets in a crossvalidation setup, using five folds, resulting in 40 news reports for training and 10 for testing in each fold. our model was trained to learn a total of 9 distinct discourse label types as represented in 2 (all leaf labels minus headline). in this section we describe our baseline comparisons and overall experiments and results. ## related work there has been substantial work describing how the structure of news operates with regards to the chronology of real-world events. much news follows an inverted chronology-called the inverted pyramid @xcite or relevance ordering @xcite )-where the most important and typically the most recent events come first. bell claims that "news stories. . . are seldom if ever told in chronological order" @xcite , which is demonstrated by rafiee et al. for both western (dutch) and non-western (iranian) news (2018). rafiee et al. also show that many stories follow a hybrid structure, which combines characteristics from both inverted and chronological structures. our approach was inspired by @xcite 's hhmm approach to section identification in clinical notes. in turn, their work extend an earlier study on section identification of psychiatric evaluation reports that combined the work of @xcite on identifying section types within clinical reports and that of @xcite on text segmentation of meeting minutes. li et al. modeled hmm emissions at the section level using bigrams, while sherman and liu modeled the emissions at the sentence level and used unigrams and trigrams. other approaches followed similar strategies in segmenting story text and in creating generative models for detecting story boundaries @xcite . more recently, @xcite used a hybrid deep neural network combined with a hidden markov model (dnn-hmm) to segment speech transcripts from broadcast news to a sequence of stories. similar to our approach, @xcite used crfs and svms for the classification of automatic classification of content types, a novel task that was introduced to provide cues to access the structure of a document's types of functional content. discussing van dijk's theory of news discourse, bekalu stated that analysis of "the processes involved in the production of news discourses and their structures will ultimately derive their relevance from our insights into the consequences, ef-fects, or functions for readers in different social contexts, which obviously leads us to a consideration of news comprehension" @xcite . the theory proposed by van dijk has also been proposed for use in annotating the global structure of elementary discourse units in dutch news articles @xcite . @xcite , in a similar analysis, presented a framing-based approach that provides four structural dimensions for the analysis of news discourse: syntactic structure, script structure, thematic structure, and rhetorical structure. of these, the syntactic structure is most closely aligned with van dijk's theory. in this paper, we chose to focus on van dijk's theory as pan and kosicki do not provide a list or description of the structure that could be readily translated into an annotation scheme. while white (1998) treats the structure of news as being centered around the headline and lead. white suggests that the headline and lead, which act as a combination of both synopsis and abstract for the news story, serve as the nucleus for the rest of the text: "the body which follows the headline/lead nucleus-acts to specify the meanings presented in the opening headline/lead nucleus through elaboration, contextualisation, explanation, and appraisal" (1998, p. 275). we focus on van dijk's theory for this paper as we find it to provide a higher degree of specificity: white's specification modes serve roughly the same purpose as higher-level groupings in van dijk's theory. ## contributions we extend earlier work on news paragraph discourse function labeling. we built a linear chain crf model incorporating various lexical, positional, syntactic, and semantic features that improves detection of the order of discourse labels in a news article at the paragraph level as well as models the paragraph content of each label type. we evaluated our model's performance against two baselines and three existing models with various subsets of features. we showed that the crf model represents a significant improvement in this task. most importantly, our work demonstrated the importance of modeling paragraph and discourse label type inter-dependencies.