Detecting Narrative Elements in Informational Text (2022)

Abstract: Automatic extraction of narrative elements from text, combining narrative theories with computational models, has been receiving increasing attention over the last few years. Previous works have utilized the oral narrative theory by Labov and Waletzky to identify various narrative elements in personal stories texts. Instead, we direct our focus to informational texts, specifically news stories. We introduce NEAT (Narrative Elements AnnoTation) – a novel NLP task for detecting narrative elements in raw text. For this purpose, we designed a new multi-label narrative annotation scheme, better suited for informational text (e.g. news media), by adapting elements from the narrative theory of Labov and Waletzky (Complication and Resolution) and adding a new narrative element of our own (Success). We then used this scheme to annotate a new dataset of 2,209 sentences, compiled from 46 news articles from various category domains. We trained a number of supervised models in several different setups over the annotated dataset to identify the different narrative elements, achieving an average F_1 score of up to 0.77. The results demonstrate the holistic nature of our annotation scheme as well as its robustness to domain category.

## introduction automatic extraction of narrative elements from texts is a multidisciplinary field of research, combining narrative theories with computational models, which has been receiving increasing attention over the last few years. examples include modeling narrative structures for story generation @xcite , using unsupervised methods to detect narrative event chains @xcite and detecting content zones @xcite in news articles, using semantic features to detect narreme boundaries in fictitious prose @xcite , identifying turning points in movie plots @xcite and using temporal word embeddings to analyze the evolution of characters in the context of a narrative plot @xcite . a recent and more specific line of work focuses on using the theory laid out by @xcite and later refined by @xcite to characterize narrative elements in personal experience texts. @xcite relied on @xcite to annotate a corpus of 50 personal stories from weblogs posts, and tested several models over hand-crafted features to classify clauses into three narrative clause types: orientation, evaluation and action. @xcite constructed a corpus from 20 oral narratives of personal experience collected by @xcite , and utilized logistic regression over hand-crafted features to detect instances of complicating actions. more recently, @xcite utilized a combination of ideas from @xcite and @xcite to annotate a collection of short stories, and saldias and roy (2020) used convolutional neural networks (cnns) to classify clauses from spoken personal texts into the same three narrative clause types as @xcite . while these works concentrated their effort on narrative analysis of personal experience texts, we direct our focus to detecting narrative patterns in informational texts, such as news stories. the social impact of news stories distributed by the media and their role in creating and shaping of public opinion incentivized our efforts to adapt narrative analysis approaches to this domain. to the best of our knowledge, this is the first attempt to automatically detect narrative elements based on @xcite and later works by @xcite in news articles. in this work, we introduce neat (narrative ele-ments annotation) -a novel nlp task for detecting narrative elements in raw text. for this purpose, we adapted two elements from the narrative theory presented in @xcite ; @xcite , namely complication and resolution, while adding a new narrative element, success, to create a new multi-label narrative annotation scheme. this scheme was designed with two main objectives in mind. first, capturing elements oriented towards discourse structure, rather than semantic content. second, possessing the flexibility required to capture narrative characteristics within a wide variety of text types, specifically informational text (as opposed to personal experience), and not only literary and wellstructured stories. we used this scheme to annotate a newly-constructed dataset of 2,209 sentences, compiled from 46 english news articles; each sentence was tagged with a subset of the three narrative elements (or, in some cases, none of them), thus defining a novel multi-label classification task. we explored two different approaches towards solving our new task: splitting into three unrelated binary classification tasks (complication, resolution and success), and jointly learning the three narrative categories as a multi-label classification task. we experimented with three supervised models, each based on fine-tuning a different pre-trained language model: bert @xcite , roberta @xcite and dis-tilbert @xcite , achieving an average f 1 score of up to 0.77. an analysis of the results indicates that our narrative categories are strongly connected and form a coherent narrative scheme which is more than just the sum of its parts. additional experimentation with cross-domain classification demonstrates the task's robustness to domain category, suggesting that our annotation scheme is more grounded in discourse characteristics rather than semantic context. the remainder of this paper is organized as follows: section 2 gives a theoretical background and describes the adjustments we have made to the scheme in @xcite in order to adapt it to informational text. section 3 provides a complete description of the dataset and of the processes and methodologies which were used to construct and annotate it, along with a short analysis and some examples for annotated sentences. section 4 describes the experiments conducted on the dataset, and section 5 provides an analysis and a discus-sion of the results. finally, section 6 contains a summary of our contributions as well as several potential directions for future work. ## results & discussion results are reported in first, we observe that addressing the task as a multi-labeled one proved to be a better strategy than learning each narrative category separately. this is evident across backbone models as well as across narrative categories; for each backbone model, the multi-label model produced a higher f 1 score for each and every one of the narrative categories. this is a clear indication that these categories are substantially connected as they constitute intertwining elements in an underlying story. therefore, the three categories form a coherent narrative scheme that is more than just the sum of its parts. interestingly, while this effect is relatively small for complication (f 1 increased by 0.00-0.02), it is much more prominent for resolution (f 1 increased by 0.09-0.15) and success (f 1 increased by 0.03-0.12), meaning that incorporating all three categories into one coherent scheme contributes mostly to learning the resolution and success categories. this suggests that perhaps the narrative properties of the complication category make it more independent and self-contained than the other two categories. resolution and success, on the other hand, are more relative in nature, and seem to be anchored, implicitly or explicitly, by a prior situation or condition. comparing different backbone models, the distilbert-based model performed similarly to the bert-based one -an average f 1 score of 0.61 for both in the separate-task setting, and 0.69 compared to 0.68 in the multi-label setting -suggesting that the task is fairly robust to a decrease in the backbone model's size. however, roberta significantly outperformed the other two language models as the backbone in both settings -an average f 1 score of 0.71 compared to 0.61 in the separate-task setting, and an average f 1 score of 0.77 compared to 0.68 and 0.69 in the multi-label setting. we also note that the difference in performance between complication and the other two categories is less extreme in the roberta-based models compared to the other backbone models. ## conclusion we introduced neat (narrative elements anno-tation) -a novel nlp task for detecting narrative elements in raw text. for this purpose, we designed a new flexible multi-label narrative annotation scheme, specifically suited for informational text, by adapting two elements from the theory introduced in @xcite ; @xcite -complication and resolutionand adding a new element -success. the scheme was used to annotate a new dataset of 2,209 sentences, compiled from 46 articles, which were collected from news websites. we explored two alternate settings for solving this task -one in which each narrative category was treated as a separate classification task, and another in which the entire task was addressed as multi-label classification. in each of these setups, we experimented with fine-tuning three different language models, achieving an average f 1 score of up to 0.77 on the test set, and showcasing the potential of supervised-learning methods in detecting the narrative information encoded into our scheme. the multi-label setting consistently provided significantly better results across all models and narrative categories, demonstrating that our narrative categories are strongly connected and form a coherent narrative scheme which is more than just the sum of its parts. additional cross-domain classification results demonstrate the task's invariance to domain category, suggesting that our annotation scheme is more grounded in discourse characteristics rather than semantic context. we are currently engaged in an ongoing effort for improving the annotation quality of the dataset and increasing its size. in addition, we have several interesting directions for future work. the first one, which we are currently pursuing, includes enriching the scheme with token-level annotation of the narrative elements, effectively converting the task from multi-label classification to a sequence prediction one. alternatively, we could introduce additional layers of information to encode more global narrative structures in the text, such as inter-sentence -or even interarticle -references between narratively-related elements (e.g., a resolution referencing its inducing complication). another potential direction is incorporating additional narrative elements into our annotation scheme. for example, the evaluation element from @xcite may be beneficial in encoding additional information in the context of news media, such as the sever-ity of a complication or the 'finality' of a resolution. we could also add completely new narrative elements, tailored to capture specific informational aspects, such as actor-based elements identifying entities which are related to one or more of the currently defined narrative categories.