Overview of the 2024 ALTA Shared Task: Detect Automatic AI -Generated Sentences for Human- AI Hybrid Articles (2024)

Abstract: The ALTA shared tasks have been running annually since 2010. In 2024, the purpose of the task is to detect machine-generated text in a hybrid setting where the text may contain portions of human text and portions machine-generated. In this paper, we present the task, the evaluation criteria, and the results of the systems participating in the shared task.

## introduction the advent of large language models (llms) has revolutionized artificial intelligence (ai), leading to a significant surge in ai-generated text and the rise of human-ai collaborative writing. while this collaboration offers exciting opportunities, it also introduces challenges -particularly in distinguishing between human-authored and ai-generated content within a single document. although ai refers to various technologies, our focus in this shared task is specifically on the text generated by llms. detecting such content has become essential not only as a deterrent against misuse but also as a safeguard, particularly in news reporting, journalism, and academic writing. previous efforts, such as the 2023 alta shared task @xcite , focused on corpus-level detection of ai-generated text, assuming that entire documents are either human-written or aigenerated. however, with the rise of human-ai collaborative writing, it is increasingly common for a single document to contain a mix of sentences authored by human and ai. our proposed task addresses this realistic scenario by automatically identifying ai-generated sentences within hybrid articles. detecting ai-generated content at the sentence level is crucial for analyzing hybrid texts, which are becoming more prevalent in fields like news reporting, content marketing, and academic writing @xcite . identifying ai-generated content at a finer granularity introduces a more nuanced challenge than distinguishing entirely aigenerated documents from those solely by human writers. to tackle this challenge, our study leverages a newly available public dataset from @xcite and a private test set we collected for this shared task, both of which contain diverse and realistic hybrid articles. these datasets offer ideal benchmarks for exploring ai-generated text detection, as they include a mixture of human-written and ai-generated sentences across a range of topics within two key domains: academic writing and news reporting. by examining the accuracy of identifying aigenerated sentences within texts that combine human and ai-authored content, we aim to develop more sophisticated and effective detection methods for collaborative writing scenarios. this work complements existing corpus-level detection efforts by offering a more comprehensive approach to understanding and identifying ai-generated content at different scales and contexts. the insights gained from this shared task will be valuable not only for preserving integrity in written communication but also for promoting transparency and responsibility in ai-assisted content creation. the website of the 2024 alta shared task is https://www.alta.asn.au/events/ sharedtask2024/ . ## related work recent advances in llms have created unprecedented challenges for content authenticity. following the comprehensive related work presented by @xcite , we examine how the ability of ai to generate human-like text raises significant concerns across multiple scenarios -from education and journalism to scientific research @xcite and social media. while these technologies offer tremendous benefits, they also present risks of academic dishonesty @xcite and the potential spread of misinformation. current detection approaches predominantly employ binary classification at the document level @xcite @xcite @xcite @xcite . these methods assume the content is either entirely ai-generated or entirely human-written, an assumption that fails to reflect real-world usage patterns. as noted in emerging research @xcite , modern content creation often involves human-ai collaboration, requiring more fine-grained detection approaches. a promising direction in hybrid text analysis has emerged, focusing on the identification of mixed authorship within documents. this approach draws inspiration from classical text segmentation techniques while addressing the unique challenges of ai text detection @xcite . recent work has explored both boundary detection methods @xcite @xcite @xcite @xcite and more sophisticated approaches that integrate boundary identification with content classification @xcite @xcite @xcite . ## data description for this shared task, we constructed a dataset comprising hybrid articles with mixed human-written and gpt-3.5-turbo-generated foot_0 content to facilitate the evaluation of ai-generated sentence detection methods. ## baselines to establish baseline performance metrics for the task, we have implemented three approaches for ai-generated sentence detection: ## participating systems and results as in previous years, there were two categories of participating teams: a total of 4 teams made submissions in the test phase, and the results are shown in the difference between the top team and second best is statistically different foot_4 , so the winning team is "null-error". a brief description of the participating systems who provided their information follows. team dima @xcite used a 4-bit quantized llama 3.1-8b-instruct fine-tuned on domainspecific data. they also tested their system's ability to handle automatic rewrites. team adsn @xcite used an ensemble of lightweight classification methods inspired on traditional authorship attribution approaches. ## conclusions this paper described a shared task for sentencelevel detection of gpt-3.5-turbo-generated content within hybrid texts. by moving beyond traditional corpus-level detection to sentence-level analysis, this task addresses the practical challenges of identifying ai-generated sentences in collaborative writing scenarios. the multi-domain training