Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs (2024)

Abstract: With the advent of large language models (LLM), the line between human-crafted and machine-generated texts has become increasingly blurred. This paper delves into the inquiry of identifying discernible and unique linguistic properties in texts that were written by humans, particularly uncovering the underlying discourse structures of texts beyond their surface structures. Introducing a novel methodology, we leverage hierarchical parse trees and recursive hypergraphs to unveil distinctive discourse patterns in texts produced by both LLMs and humans. Empirical findings demonstrate that, although both LLMs and humans generate distinct discourse patterns influenced by specific domains, human-written texts exhibit more structural variability, reflecting the nuanced nature of human writing in different domains. Notably, incorporating hierarchical discourse features enhances binary classifiers’ overall performance in distinguishing between human-written and machine-generated texts, even on out-of-distribution and paraphrased samples. This underscores the significance of incorporating hierarchical discourse features in the analysis of text patterns. The code and dataset will be available at [TBA].

## introduction the emergence of powerful instruction-tuned large language models (llms) @xcite @xcite has led to an explosion of machine-generated texts in both offline and online domains. consequently, discerning the authorship of texts has become a significant challenge, spanning from educational settings to the landscape of online advertising @xcite @xcite . indeed, many efforts have been made to tackle this issue by constructing corpora of machine-generated and human-authored texts @xcite generate a motivational quote on finding a lifelong passion. ## modeling hierarchical structures to explore unique patterns that may extend beyond surface-level structure, it is imperative to employ an expressive framework capable of representing hierarchical structures within texts. human writers strategically utilize textual structures to systematically convey meaning, thereby augmenting the clarity, coherence, and persuasiveness of their written compositions. in the field of linguistics, the study of document structure is within the domain of discourse frameworks @xcite @xcite . this paper adapts rhetorical structure theory (rst) @xcite due to its widespread acceptance and the availability of openly accessible pre-trained parsers. in this paper, we make use of the dmrst parser released by @xcite . ## proposed method: revealing key patterns via discourse motifs this paper focuses on finding distinctive structural patterns, namely, hierarchical discourse relations based on the rst framework ( §2). our main investigative approach involves analyzing the distribution of discourse motifs, which are recurring and statistically significant subgraph patterns within a larger network. these motifs are often considered as the building blocks of larger networks and can provide insights into the organization and dynamics of the system @xcite . as illustrated in figure 2 , our proposed approach involves gathering documents written by humans and llms, parsing their rst trees, transforming them into hypergraphs, and running motif analyses to understand how individuals (both humans and machines) craft texts. the following subsections illustrate how the discourse motifs are systematically formed ( §3.1) and scored using our proposed metrics ( §3.2). ## experimental setup following the generation of three sets of discourse motifs for single-, double-, and triple-triads ( §3.1), we evaluate them using the mf-idf metric and retain those with scores surpassing at least one standard deviation. this yields a total of 207 motifs, with a detailed breakdown presented in section 4.1 outlines the utilization of our proposed features ( §3) within both the baseline models and analyses; and section 4.2 describes the datasets designated and proposed for experimentation. ## 96 80 ## results we present findings from three sets of experiments: ## conclusion as texts generated by llms become increasingly challenging to differentiate from those authored by humans, our approach involves identifying discernible, unique properties inherent in humancrafted texts. we posit that these distinctive signatures manifest at levels beyond mere surface structure and thus opt for looking into their hierarchical discourse structures. viewing the structures as recursive hypergraphs and conducting motif analyses on them, we find that these motifs are useful in the authorship detection task and can highlight the subtle structural differences in the two author groups depending on the domains of texts and their corresponding nature. future plans include extending this approach to long documents by merging multiple document graphs and incorporating topological information beyond discourse.