Evaluation Metrics for Depth and Flow of Knowledge in Non-fiction Narrative Texts (2023)

Abstract: In this paper, we describe the problem of automatically evaluating quality of knowledge expressed in a non-fiction narrative text. We focus on a specific type of documents where each document describes a certain technical problem and its solution. The goal is not only to evaluate the quality of knowledge in such a document, but also to automatically suggest possible improvements to the writer so that a better knowledge-rich document is produced. We propose new evaluation metrics to evaluate quality of knowledge contents as well as flow of different types of sentences. The suggestions for improvement are generated based on these metrics. The proposed metrics are completely unsupervised in nature and they are derived from a set of simple corpus statistics. We demonstrate the effectiveness of the proposed metrics as compared to other existing baseline metrics in our experiments.

## introduction documents containing non-fiction narrative text occur in many practical applications; e.g., essays, news, emails, safety or security incident reports, insurance claims, medico-legal reports, troubleshooting guides, user manuals etc. it is important to ensure that each such document is of high quality, for which purpose we need metrics that measure their quality. while metrics for readability (or comprehensibility) are obviously usable, we need specialized metrics that attempt to measure quality of nonfiction narrative text in terms of the specific characteristics. fictional narratives are characterized in terms of structural elements such as conflicts, plot points, dialogues, characters, character arcs, focus, etc.; there is extensive literature about their linguistic analysis. however, non-fiction narrative texts are comparatively less studied in linguistics; e.g., @xcite ; mckenzie * work done while working at @xcite . in this paper, we identify following characteristics of non-fiction narrative texts: (i) depth and variety of factual and conceptual knowledge elements present; (ii) distribution of different classes of sentences that represent essential aspects of information content; and (iii) flow and coherence of different types of sentences. we also propose novel quantitative metrics for measuring the quality of non-fiction narrative texts in terms of these characteristics. in this paper, we focus on a specific type of non-fiction narrative text documents -contextual master (cm) stories. contextual master t m is a registered trademark of tcs foot_0 , which refers to an associate who has over the time gained a significant contextual knowledge or understanding of a business domain or a particular client's business. an cm story is a short narrative text that a cm writes to describe a particular instance where he/she has used the expert-level knowledge to solve a specific problem or to address a specific challenge. each such cm story generally consists of 25-30 sentences (details in section 7.1). a typical process of writing these stories is that a cm first writes some initial version which is reviewed by reviewers for knowledge contents, readability, narration flow and other aspects like grammar. over a few iterations of incorporating reviewers' suggestions, a story is accepted to be published internally and for marketing purposes. in this paper, our goal is to develop a system for -(i) automatic evaluation of a cm story for its knowledge contents and narration flow quality, and (ii) automatic generation of suggestions for improvement so that the time needed to produce a publishable final version of a story from its initial version is reduced. the main motivations for building this system are as follows: automatic essay scoring or grading (ke and ng, 2019) is a related problem but it differs from our problem in some key aspects. essay grading is a task of automatically scoring essays based on multiple dimensions like grammar, word usage, style, relevance to the essay topic (prompt), cohesion, coherence, persuasiveness etc. on the other hand, evaluation of non-fiction narrative texts like cm stories emphasizes more on the depth of the knowledge contents which are often not explicitly evaluated by the most essay grading techniques. to some extent, cohesion and coherence are common desirable aspects for essays as well as non-fiction narrative texts like cm stories. however, cohesion and coherence of ideas or topics is expected in essays whereas in cm stories, cohesion and coherence of certain types of sentences is expected. therefore, in this paper, we propose new metrics to specifically evaluate the knowledge depth and the narration quality in terms of flow of sentence types. here, it is important to note that we refer to knowledge as a more conceptual and abstract notion as compared to factual and data-oriented information. for example, we consider task as one of the knowledge markers (section 3) which is defined as a volitional activity which needs expert knowledge to carry out @xcite . a task such as "analysed the configuration of the security protocol" clearly represents an aspect of knowledge of a cm rather than mere factual information. similarly, we consider specialized sentence categories (such as solution, benefit) introduced in section 4 as another aspects of knowledge and hence considered as part of knowledge quality metrics. all the proposed metrics are unsupervised in nature, i.e., they do not need any set of stories which are explicitly annotated for knowledge quality by human reviewers. the specific contributions of this paper are: ## problem definition our goal is to determine the quality of knowledge and narration flow of a cm story with respect to a set of knowledge quality and narration flow quality metrics. each metric is designed to capture and evaluate a certain aspect of the story, as described in detail in later sections. the problem can be specifically defined in terms of input, output and training requirements as follows: ## knowledge markers we hypothesize that the knowledge needed for solving a particular domain or technical problem is expressed in terms of certain knowledge markers. these knowledge markers are mentions of some key entity types as follows: ## sentence categories in addition to the knowledge markers, an ideal cm story should describe all the aspects of a certain problem being solved such as a brief background of the problem, the problem itself, the solution that was provided, and finally what were the benefits that were achieved. therefore, it is important to identify presence of these aspects in a given story. we propose to identify these aspects in the form of the following sentence categories: we modelled the problem of identifying appropriate sentence categories as a multi-label, multiclass sentence classification problem. we used a multi-label setting because in some cases, a sentence may have more than one valid category. for example, the following sentence belongs to solution as well as benefit -he used his understanding of the client's applications and restructured the database accordingly to reduce recurring issues, which resulted in reduction in incidents by 70%. we use a sentence classification model which is based on distilbert @xcite , a lighter version of bert @xcite . distilbert model is 40% smaller than bert while retaining its 97% language understanding capabilities. dis-tilbert 2 is capable of producing semantically rich representations for any input text and the individual words in it. these representations are 768 dimensional dense vectors of real numbers (r 768 ). we use these representations for building our classifier to predict appropriate sentence categories for a sentence in a cm story. we now explain the model architecture in detail. let the input sentence be s which is first passed through the pre-trained distilbert model to obtain -(i) @xcite token encoding which provides the representation of the entire input text s, and (ii) the representations for each word in s. here, x @xmath0. let x @xmath1. we use attention mechanism so that the contribution of each word in s is determined based on its importance for prediction of each of the sentence categories. we use 6 attention layers corresponding to the 6 sentence categories. each attention layer is similar to the one described in @xcite . here, w c @xmath2, respectively. a c @xmath3. these scores are normalized across all the words in s to obtain final attention weights (α c i 's) which are used to obtain a weighted average of word representations. finally, the overall representation (x c @xmath4. this final representation is then passed through a linear transformation layer to obtain a hidden representation. here, w c @xmath5. cross entropy loss is computed using the predicted and the goldstandard label distributions which is summed over all categories to get the overall loss. the model is then trained to minimize this loss over the labelled training data. we used a training set of 1618 sentences which were labelled manually using a few active learning iterations. we evaluated the trained sentence classification model on a held out evaluation dataset of 636 sentences. ## knowledge quality metrics in this section, we describe our proposed knowledge quality metrics based on the knowledge markers and the sentence categories described in the previous sections. for a cm story s, for each knowledge marker and sentence category, we compute a metric which measures its density within the story as follows: here, the division by the number of sentences in s offsets the effect of the length of the story. we similarly compute such metrics for all knowledge markers as well as sentence categories -skills_density, tasks_density, roles_density, concepts_density (based on knowledge markers), background_density, problem_density, expert_knowledge_density, solution_density, bene-fit_density, and client_appreciation_density (based on sentence categories). one limitation of these knowledge quality metrics is that the metrics are dependent on the density of multiple knowledge markers but do not explicitly check whether multiple such markers are relevant or pertinent to each other. we plan to handle this as a future work and currently assume that there is no malicious intent in writing the document (e.g., by adding multiple irrelevant entities in text to artificially boost the quality score). ## narration flow quality metrics in addition to the knowledge content, it is also important to evaluate the narration quality of any narrative text such that it measures how wellstructured the flow of narration is. in this section, we describe our proposed metric to evaluate the flow of different sentence categories in a cm story. sentence categories flow metric: a good flow of sentence categories is that sequence of sentence categories which is generally used to describe an ideal story. for example, generally any story begins with some background of the problem followed by the description of the problem itself. then the contextual knowledge of the cm is discussed followed by the proposed or implemented solution. finally, the story concludes by discussing the benefits that were achieved by the solution and whether any appreciations were received for it. though it is not mandatory to strictly follow this flow of narration and some sentences can be out of place, the good stories are generally structured in this way. moreover, a good cohesive story will contain all the sentences describing a certain aspect (say problem) in close proximity of each other and also at a proper relative position within the entire story. hence, we propose a new metric -scf (sentence categories flow) which tries to capture these aspects of an ideal flow of sentence categories in a cm story. first, a relative position of each sentence within the cm story is determined as follows. for any i th sentence in a cm story consisting of n sentences, the relative position is i n . for a particular sentence category (say solution), we create a sample of relative positions of all sentences belonging to that category from all the stories in our training corpus. we compute mean (µ rp ) and standard deviation (σ rp ) of this sample (e.g., for solution, µ @xmath6.6 and σ @xmath7.22; this means that normally the solution sentences occur in a story after 60% of the overall sentences are written). now, given any new story s, the metric scf solution (s) is computed as the number of sentences of category solution in s whose relative position is more than one standard deviation away from the mean, i.e., relative position outside the range [µ rpσ rp , µ rp + σ rp ]. similar metrics are computed for other sentence categories in the same way (note that µ rp and σ rp are specific to each sentence category). lower the value of this scf metric, better is the narration flow quality, because it simply counts the number of sentences of a particular sentence category which are at un-usual relative positions within a story. based on this metric, suggestions for improvement are generated for those sentences in a cm story for which the relative position is outside the expected range. e.g., please consider re-positioning the solution sentence [x] which is appearing too early (or late) in your story. we also compute a single aggregate metric to combine the scf metrics for individual sentence categories: scf @xmath8. ## experimental analysis in this section, we describe our experiments in terms of datasets, baselines, and the evaluation strategy. ## conclusions and future work we proposed a set of novel evaluation metrics for depth and flow of knowledge in non-fiction narrative texts that are unsupervised as well as interpretable. we focused on a specific type of documents identified as cm stories. two different types of evaluation metrics were proposed: (i) for measuring the quality of the knowledge contents in a cm story, and (ii) for evaluating flow of different categories of sentences in a cm story. we demonstrated the effectiveness of the proposed metrics as compared to the existing metrics like perplexity, readability, and essay grading. in future, we plan to explore how the proposed metrics can be adapted to other types of non-fiction narrative texts such as security incident reports. one interesting research direction is whether we can discover the key sentence categories automatically for a new type of documents. we also plan to develop some new narration flow quality metrics such as a metric based on sequence entropy.