Entailment Progressions: A Robust Approach to Evaluating Reasoning Within Larger Discourse (2025)

Abstract: Textual entailment, or the ability to deduce whether a proposed hypothesis is logically supported by a given premise, has historically been applied to the evaluation of language modelling efficiency in tasks like question answering and text summarization. However, we hypothesize that these zero-shot entailment evaluations can be extended to the task of evaluating discourse within larger textual narratives. In this paper, we propose a simple but effective method that sequentially evaluates changes in textual entailment between sentences within a larger text, in an approach we denote as “Entailment Progressions”. These entailment progressions aim to capture the inference relations between sentences as an underlying component capable of distinguishing texts generated from various models and procedures. Our results suggest that entailment progressions can be used to effectively distinguish between machine-generated and human-authored texts across multiple established benchmark corpora and our own EP4MGT dataset. Additionally, our method displays robustness in performance when evaluated on paraphrased texts a technique that has historically affected the performance of well-established metrics when distinguishing between machine generated and human authored texts.

## introduction as large language models (llms) expand and evolve to accommodate more complex language generation tasks (e.g., significant advances in machine translation @xcite , logical reasoning @xcite , summarization @xcite , complex question answering @xcite )), we are witnessing a growing number of machine-generated text (mgt) in both online and offline environments. 1 this, in turn, has raised concerns regarding authenticity and regulations, 2, 3 drawing attention to mgt detection as both a safeguard and indicator for authentic human authorship, which has become quite a hot topic in natural language processing (nlp). 4 intuitively, machine-generated texts can display lexical, syntactic, and semantic properties that are distinguishable from human authored texts, potentially guiding mgt detection implicitly, as a latent property, or explicitly as a directly encoded feature @xcite . for example, mgt detection methods like entropy and log-likelihood, which assess the probability of a text being machine generated based upon individual token probabilities encoded by a given llm, take into account how llms functionally operate as next word predictors @xcite . thus, evaluating where llms situationally differ from human authorship in relation to both their observed behaviour and functionality can expand the scope of feature selection within mgt detection to capture these differences more effectively and in a more interpretable manner. textual entailment, or the relationship between a given premise and its potentially inferred hy-pothesis, has been previously used to evaluate how llm text generation differs from human authorship in regard to an llm's ability to generate text in accordance with prior informational constraints @xcite . in areas like question answering and dialogue systems, calculating the textual entailment between a prior conversation and a machine-generated response can examine whether a model produces relevant and accurate text, a behaviour assumed to be exhibited in human authorship and communication @xcite . based on observations of differences in textual entailment between mgts and human-authored texts in relation to prior conversations, an interesting question arises: can textual entailment be directly encoded and utilized as a feature for mgt detection? in this paper, we: (1) introduce entailment progressions, a framework in which a given piece of text can be represented as a series of values, with each value representing the level of textual entailment between sentences in a text. these entailment progressions aim to measure the extent to which a model generates each individual utterance in logical reference to its previously generated utterances (i.e., identifying how new information is introduced in relation to the preceding content: in support, in contradiction, or with no relation (neutral)). we believe that entailment progressions provide a unique perspective and should be considered in qualifying llm behaviour to achieve a more in-depth analysis. the definition of recognizing textual entailment (rte) as outlined by @xcite and later expanded upon by @xcite is as follows: "a text t textually entails a hypothesis h relative to a group of end users g just in case, typically, a member of g reading t would be justified in inferring the proposition expressed by h from the proposition expressed by t". this definition incorporates three key aspects of rte. first, it does not require any knowledge beyond the justifiable inference that can be made between a given text and its hypothesis @xcite . second, this justifiable inference is subject to the characteristics exhibited by a group of end users g, in which users outside this group may differ in their inferences due to personal factors that may influence how they interpret logical relationships @xcite . third, the logical component of entailment is textually constrained, rendering it dependent on linguistic factors such as grammar, semantic, and syntactical choices @xcite . current rte modelling approaches require two main steps. first, the features of premise t and hypothesis h are extracted in order to represent the statements in accordance with relevant linguistic mechanisms associated with textual entailment. second, the statements are fed into a supervised multi-class classification model which predicts whether a premise-hypothesis pair possesses positive (the hypothesis can be inferred to be true if the premise is true), negative (the hypothesis can be inferred to be false if the premise is true), or neutral (the hypothesis' truth is not sufficiently conditional upon the premise being true) entailment. for an in-depth overview of rte resources, approaches, and applications, see @xcite . ## results and discussion in figure 2 we showcase two mgts from the ep4mgt dataset. although these two mgts are generated by different models (i.e., gpt4 and mistral-large), pertain to different subject matters, and display low textual similarity (0.0718 as calculated using sentencebert @xcite , a modified bert that derives semantically sentence embeddings that can be compared using cosine similarity), they exhibit high entailment progression similarity (5.9948 using dynamic time warping distance, that measures the similarity between time series @xcite ) between each other. similar to recent work leveraging paraphrasing as a means of evaluating the robustness of different mgt detection approaches @xcite , we also examine the change in performance exhibited by our mlp model when trained on the entailment progressions of the paraphrased texts (where ∆ = best model f 1best model paraphrased f 1 ). for this, we leveraged the same methodology as @xcite and @xcite , in which each sentence is individually paraphrased using the pegasus transformer model @xcite . when trained on the entailment progressions of the paraphrased texts from the ep4mgt dataset, the model exhibits a performance degradation of up to 5% in terms of f 1 score. in addition to these scores, figure 3 illustrates the changes in between the mean positive and negative entailment progressions for the ep4mgt dataset and their paraphrased counterpart. ## conclusion in this paper, we introduce entailment progressions, a novel representation of the underlying logical structures of textual narratives for identifying human and model authorship. we also present ep4mgt, a dataset specifically designed to evaluate the logical approaches of humans and those produced by a suite of state-of-the-art llms, highlighting new avenues for exploring the properties and scope of entailment progressions as a latent descriptor of authorship. given that entailment progressions can be generated from any multi-sentence text, their potential applications could extend to the broader area of text attribution, thus providing insights in their utility as identifiers of authorship (be it human or model-based). this would also position our framework alongside more traditional lexical, syntactic, and semantic descriptors of style. in future work, we plan on examining the effectiveness of entailment progressions in other experimental settings, across different languages, tasks, and genres. although through our framework we have successfully detected mgts in several english corpora with fixed narrative structure (i.e., personal claims, news articles), testing entailment progressions on datasets in languages with different underlying logical conventions or within conversational settings (dialogue) with variable logical constraints could reveal broader applicability.