AI pom at S em E val-2024 Task 8: Detecting AI -produced Outputs in M4 (2024)

Abstract: This paper describes AIpom, a system designed to detect a boundary between human-written and machine-generated text (SemEval-2024 Task 8, Subtask C: Human-Machine Mixed Text Detection). We propose a two-stage pipeline combining predictions from an instruction-tuned decoder-only model and encoder-only sequence taggers. AIpom is ranked second on the leaderboard while achieving a Mean Absolute Error of 15.94. Ablation studies confirm the benefits of pipelining encoder and decoder models, particularly in terms of improved performance.

## introduction semeval-2024 task 8 @xcite focuses on multigenerator, multidomain, and multilingual machine-generated text detection based on the m4 corpus @xcite . the shared task offers three subtasks, which correspond to standard task formulations in the rapidly developing field of artificial text detection @xcite : (a) classifying if a given text in a particular language is human-written or machinegenerated, (b) attributing the author of a given text, and (c) detecting a boundary between humanwritten and machine-generated text. developing generalizable solutions to these problems helps mitigate the risks of misusing generative language models (lms) for malicious purposes @xcite and improve human performance in identifying ai-produced content @xcite . this paper proposes aipom 1 , a novel method for human-machine mixed text detection (subtask c). the boundary detection setup aligns with common user scenarios for applying generative lms in practice, e.g., text continuation, creative writing, and 1 aipom is named after a simian pokémon aipom, and stands for detecting ai-produced outputs in m4. our ablation studies confirm that using decoder or encoder models individually leads to lower performance. thus, employing the pipeline of decoder and encoder models proves to be an effective solution. additionally, these studies highlight domain shift issues, as there is a significant score disparity between the development and official evaluation sets. future efforts should focus on enhancing the aipom robustness with respect to the text domain and text generator. the codebase and models are publicly released foot_0 . ## background the m4 corpus consists of human-written and machine-generated texts in six languages (english, chinese, russian, urdu, indonesian, and arabic) across various domains, ranging from wikipedia to academic peer reviews. the generative lms include the openai models (chatgpt and textdavinci-003), llama-1 @xcite , @xcite , cohere, dolly-v2 foot_1 , and bloomz @xcite . the organizers provide 3649, 505, and 11123 dataset instances in subtask c training, development, and official evaluation sets, respectively. task formulation human-machine mixed text detection requires predicting the index corresponding to the first machine-generated word, as shown below: ## aipom first, we overview the aipom pipeline. next, we detail the fine-tuning procedures for encoder and decoder models. overview the aipom pipeline (see figure 1 ) consists of multiple consecutive steps of fine-tuning language models: after receiving the predicted text from the decoder, we post-process the original training text and insert a special @xmath0. this allows us to pass the prediction further to the encoder. ## experiments overview we design a series of experiments using two recent language models, a decoder mistral-7b-openorca foot_2 (jiang et al., 2023) and an encoder debertav3-large foot_3 @xcite , selected based on their performance on standard nlp benchmarks and computational requirements. first, we establish a baseline for the decoder model by zero-shot prompting and then compare it to low-rank adaptation (lora) tuning @xcite , which yields significantly better results. second, we look into improving the performance of the stands for fine-tuning a language model, -predicting with the language model, -for aggregating the predictions by averaging. we only use the development set to evaluate our pipeline and choose our final submission based on the mae on the development set. in §5, we report the ablation studies results on both development and official test sets foot_4 . ## results ## conclusion this paper presents the aipom system submitted to semeval-2024 task 8. our solution achieves 2nd place out of 33 participating teams in subtask c. we introduce a novel method that utilizes a pipeline of decoder and encoder models. the advantage of this approach is that the models are exposed to both the original data and the predictions from previous steps. we believe this approach holds significant potential, as it allows for creating pipelines comprising various models, mimicking the transfer of learned knowledge. we plan to further improve our system by exploring different combinations of models and longer pipelines. additionally, we aim to enhance the system's robustness to handle domain-shift scenarios.