TM - TREK at S em E val-2024 Task 8: Towards LLM -Based Automatic Boundary Detection for Human-Machine Mixed Text (2024)

Abstract: With the increasing prevalence of text gener- ated by large language models (LLMs), there is a growing concern about distinguishing be- tween LLM-generated and human-written texts in order to prevent the misuse of LLMs, such as the dissemination of misleading information and academic dishonesty. Previous research has primarily focused on classifying text as ei- ther entirely human-written or LLM-generated, neglecting the detection of mixed texts that con- tain both types of content. This paper explores LLMs’ ability to identify boundaries in human- written and machine-generated mixed texts. We approach this task by transforming it into a to- ken classification problem and regard the label turning point as the boundary. Notably, our ensemble model of LLMs achieved first place in the ‘Human-Machine Mixed Text Detection’ sub-task of the SemEval’24 Competition Task 8. Additionally, we investigate factors that in- fluence the capability of LLMs in detecting boundaries within mixed texts, including the incorporation of extra layers on top of LLMs, combination of segmentation loss, and the im- pact of pretraining. Our findings aim to provide valuable insights for future research in this area.

## introduction large language models (llms), particularly since the debut of chatgpt, have made significant advancement and demonstrated the ability to produce coherent and natural-sounding text across a wide range of applications. however, the proliferation of generated text has raised concerns regarding the potential for misuse of these llms. one major issue is the tendency of llms to produce hallucinated content, resulting in text that is factually inaccurate, misleading, or nonsensical. inappropriate utilization of llms for text generation purposes, such as in news articles @xcite , social media posts @xcite , and app reviews @xcite , can propagate misinformation and influence public perceptions. furthermore, the use of machine-generated text can also facilitate academic dishonesty. therefore, accurately distinguishing between human-authored and machinegenerated texts is crucial in order to address these challenges effectively. the majority of existing studies addressing this challenge have approached it as a machinegenerated text classification problem, aiming to determine whether a given text is generated by llms or not. however, this approach assumes that the text is either completely machine-generated or entirely human-written. with the increasing collaboration between humans and ai systems, mixed texts containing both human-authored and machinegenerated portions have emerged as a new scenario that simple machine-generated text classification methods cannot effectively address @xcite . therefore, a more nuanced approach to machine-generated text classification for mixed texts is necessary. this study addresses the challenge of token-level boundary detection in mixed texts, where the text sequence starts with a human-written segment followed by a machine-generated portion. the objective is to accurately determine the transition point between the human-written and llm-generated sections. to achieve this, we frame the task as a token classification problem, thus the turning point of the label sequence will be the boundary. through experiments utilizing llms that excel in capturing long-range dependencies, we demonstrate the effectiveness of our approach. notably, by leveraging an ensemble of multiple llms to harness the robust of the model, we achieved first place in task 8 of semeval'24 competition. furthermore, we explore factors that impact the effectiveness of llms in boundary detection, including the integration of additional layers on top of llms, the combination of segmentation loss and pretraining techniques. our experiments indicate that optimizing these factors can lead to significant enhancements in boundary detection performance. the main contribution of this papers include: 1) we explore llms' capability to detect boundaries within human-machine mixed texts, compare the performance of various llms, and present a benchmark based on the new released data set. and we rank 1st in the corresponding semeval'24 competition @xcite . ## related work previous research has predominantly focused on machine-generated text classification @xcite , where the text is attributed to either human authors or large language models. the objective is to determine whether a given text is human-written or specifically generated by a particular llm. these studies can be classified into two main categories: metric-based methods and model-based methods. metric-based approaches leverage metrics such as word rank, predicted distribution entropy, and log-likelihood @xcite @xcite . on the other hand, model-based methods involve training models on labeled data @xcite . however, these methods are not directly applicable to boundary detection for mixed human-machine texts. recently, there are a few works investigating the detection of mixed human-machine text. these texts consist of both human-written and machinegenerated content, and the objective is to accurately identify the boundary between these two segments. @xcite delved into the human ability to discern boundaries between human-written and machine-generated text. their study revealed significant variations in annotator proficiency and analyzed the impact of various factors on human detection performance. @xcite were the first to formalize the task as identifying transition points between human-written and ai-generated content within hybrid texts, and they examined automated approaches for boundary detection. one limitation of these studies is that the transitions typically occur between sentences rather than at the word level. this paper aims to address the token-level boundary detection of mixed texts. ## conclusion this paper introduces llm-based methodology for detecting token-wise boundaries in humanmachine mixed texts. through an investigation into the utilization of llms for boundary detection, we have achieved optimal performance by leveraging an ensemble of xlnet models in the semeval'24 competition. furthermore, we explore factors that could affect the boundary detection capabilities of llms. our findings indicate that (1) loss functions considering segmentation intersection can effectively handle tokens surrounding boundaries; (2) supplemental layers like lstm and bilstm contribute to additional performance enhancements; and (3) pretraining on analogous tasks aids in reducing the mae. this paper establishes a state-of-art benchmark for future researches based on the new released dataset. subsequent studies aims to further advance the capabilities of llms in detecting boundaries within mixed texts.