WETB ench: A Benchmark for Detecting Task-Specific Machine-Generated Text on W ikipedia (2025)

Abstract: Given Wikipedia’s role as a trusted source of high-quality, reliable content, there are growing concerns about the proliferation of low-quality machine-generated text (MGT) produced by large language models (LLMs) on its platform. Reliable detection of MGT is therefore essential, yet existing work primarily evaluates MGT detectors on generic generation tasks, rather than on tasks more commonly performed by Wikipedia editors. This misalignment can lead to poor generalisability when applied to real-world Wikipedia contexts.We introduce WETBench, a multilingual, multi-generator, and task-specific benchmark for MGT detection. We define three editing tasks empirically grounded in Wikipedia editors’ perceived use cases for LLM-assisted editing: Paragraph Writing, Summarisation, and Text Style Transfer, which we implement using two new datasets across three languages. For each writing task, we evaluate three prompts, produce MGT across multiple generators using the best-performing prompt, and benchmark diverse detectors.We find that, across settings, training-based detectors achieve an average accuracy of 78%, while zero-shot detectors average 58%. These results demonstrate that detectors struggle with MGT in realistic generation scenarios and underscore the importance of evaluating such models on diverse, task-specific data to assess their reliability in editor-driven contexts.

## introduction wikipedia serves as a vital source of high-quality, trustworthy data across artificial intelligence (ai) communities. its scale and richness have played a foundational role in the development of large language models (llms) @xcite . however, the wikipedia community has expressed growing concern about the increasing prevalence of machine-generated text (mgt) produced by llms on its platform. 1 the wikimedia foundation warns that the spread of low-quality, unreliable mgt in its projects could undermine its knowledge integrity. 2 specifically, unverified mgt poses challenges such as factual fabrication @xcite and the perpetuation of biases present in training data @xcite , both of which jeop-ardise wikipedia's core content policies. 3 additionally, given wikipedia's frequent inclusion in llm training corpora, undetected mgt on the platform may contribute to performance degradation in future models @xcite . consequently, distinguishing human-written from machine-generated text has become increasingly important, leading to community efforts to identify and remove mgt, 4 and to a growing body of research on estimating the prevalence of mgt on wikipedia @xcite . prior work on benchmarking mgt detectors (e.g., @xcite @xcite has included the wikipedia domain but typically fails to reflect the complexities of editor-driven mgt instances on the platform. existing experimental setups generally assume that mgt on wikipedia results from (i) open-ended, topic-to-text generation and (ii) simplistic prompting techniques. these setups usually rely on a single prompt to generate an entire article, which diverges significantly from real-world wikipedia editing practices that are task-specific and incremental. in fact, prompting an llm to verbatim "write a wikipedia article about [...]," as done in earlier work, is explicitly discouraged by the community. 5 these limitations in existing setups may obscure the actual performance of state-of-the-art (sota) detectors when applied to real-world wikipedia contexts. figure 2 shows that the textual characteristics of task-specific mgt-unlike open-ended, topic-to-text mgt-more closely resemble their human-written text (hwt) references. detectors trained and evaluated on generic generation tasks may learn high-level textual patterns that are less transferable to task-specific mgt instances. consequently, detectors may not generalise well to detecting diverse, task-specific mgt on wikipedia, leaving an unknown number of instances with potentially harmful characteristics-such as hallucination or bias-largely undetected. to address this issue, we advocate for evaluating detectors on data that reflect practical use cases of editors integrating llms into their editorial workflows. this is es-sential for understanding the capacity of automatic detection methods to safeguard wikipedia's knowledge integrity and to assist editors in identifying and removing low-quality mgt. to this end, we build an mgt detection benchmark for task-specific editing scenarios on wikipedia. to create our benchmark, we construct and release two new wikipedia text corpora covering three languages with varying resource availability, enabling conclusions beyond the predominantly studied english wikipedia. we then propose three editing tasks-paragraph writing, summarisation, and text style transfer-grounded in practical use cases identified by @xcite , who analysed wikipedia editors' perceived opportunities for llm-assisted editing. for each task, we test various prompting techniques, generate mgt using diverse llms, and benchmark sota detectors across languages, generators, and tasks (see figure 1 ). we hope that our multipurpose datasets will benefit the broader wikipedia and ai communities in areas such as multilingual bias detection and single-document summarisation. we further aim to offer insights into the feasibility and reliability of automated detection methods for identifying mgt on wikipedia. our contributions are as follows: ## related work wikipedia editing tasks we concentrate on three common editing tasks with varying degrees of llm involvement: paragraph writing, summarisation, and text style transfer. paragraph writing generating new, encyclopaedic content-such as full paragraphs-is central to expanding knowledge on wikipedia. this includes writing paragraphs from scratch, expanding article stubs, or rewriting existing content. with nearly half of all wikipedia articles classified as stubs, researchers have extensively studied wikipedia content generation. 6 the scope of generated content varies from paragraph-level (e.g., @xcite @xcite to full-article generation (e.g., @xcite @xcite @xcite . the methods employed range from early templatebased approaches @xcite to more recent work using retrieval-augmented generation (rag) with pre-trained language models (plms) @xcite or llms @xcite . summarisation according to wikipedia's manual of style, foot_6 each article should begin with a lead section that serves as an introduction by summarising its most important points. the liter-ature treats lead section generation either as a multi-document (e.g., @xcite @xcite or singledocument (e.g., @xcite @xcite summarisation problem. a model's objective is typically abstractive summarisation, that is, generating a lead section from scratch based on the article body. text style transfer maintaining a neutral point of view foot_7 (npov) is a core wikipedia policy, which states that all content must be written from a perspective that is fair, proportionate, and, as far as possible, free from editorial bias. @xcite introduce the wikipedia neutrality corpus (wnc), a large-scale parallel corpus of biased and neutralised sentence pairs retrieved from npovrelated revisions. they further introduce the task of neutralisation, a text style transfer task that aims to reduce subjectivity in a sentence while preserving its meaning. recent work has used the wnc to improve data quality @xcite , test generalisation to other domains @xcite , or examine the ability of llms to detect and neutralise bias @xcite . ## dataset construction we construct two corpora for three languages with varying resource levels: english (high), portuguese (medium), and vietnamese (low). wikips includes paragraphs and lead-content pairs. mwnc is a multilingual version of the wnc @xcite . appendix a provides detailed descriptions of the dataset construction, and appendix ## editing tasks design we define three editing tasks with varying degrees of llm intervention: paragraph writing, summarisation, and text style transfer. these tasks are empirically motivated by @xcite , who found that wikipedia editors see potential in llms for generating article drafts or stubs, summarising content, and improving language. we implement paragraph writing and summarisation using the wikips corpus, and text style transfer using the mwnc. for each task and language, we evaluate three prompting strategies on a length-stratified 10% sample of the target data using gpt-4o mini, 10 and select the best-performing prompt to generate mgt for our benchmark. appendix b provides implementation details and prompt templates. ## experimental setup in this section, we introduce generators, detectors, benchmark construction, and evaluation metrics. ## results paragraph writing across languages and models, training-based detectors outperform zero-shot methods by 19-30% accuracy on average. blackbox detectors are 3-6% more accurate than whitebox detectors in english and portuguese but perform slightly worse in vietnamese. only white-box detectors show a slight increase in accuracy when moving from high-to low-resource languages. within detector families, we do not find any substantial differences between the two training-based models. among white-box detectors, binoculars achieves up to 11% higher accuracy, with performance on low-resource languages approaching that of training-based methods. for black-box detectors, gecscore exhibits up to 25% higher accuracy compared to other models in its category. considering generators, training-based detectors achieve on average 2-7% higher accuracy on smaller-sized generators. this pattern is reversed for zero-shot detectors, where accuracy is higher for larger models, with the gap widening in lowerresource languages. these results suggest that when generating paragraphs from scratch, smaller models leave more detectable semantic and syntactic traces for training-based detectors. in contrast, the internals and token-level patterns of larger models seem to exhibit stronger signals than those of smaller models for zero-shot methods. we observe substantial anomalies in mistral's output for vietnamese. the model often fails to follow prompts, producing unclear outputs that provide simple cues for training-based detectors but appear to confuse zero-shot methods. we provide an analysis of mistral's generation issues in appendix d.1. summarisation among all tasks, supervised detectors perform best on summarisation, achieving an average accuracy of 89% across languages and generators. an exception is gemini 2.0, for which detection accuracies are on average 6-17% lower, suggesting that its summaries may more closely resemble human-written references. while most llms are trained on summarisation tasks, enabling strong zero-shot performance @xcite , wikipedia lead sections follow a distinctive style and formatting that seem to provide strong cues for training-based detectors. compared to paragraph writing, average detection accuracies for zero-shot models are slightly higher. white-box detectors achieve 4% higher accuracy on english summaries compared to blackbox models, while performance is similar for portuguese and vietnamese. as in paragraph writing, binoculars achieves the highest average accuracy (65%) among black-box detectors across languages and models, while gecscore performs best among white-box methods (68%). in contrast to paragraph writing, zero-shot metrics show little variation across generators for english. however, for portuguese and vietnamese, a similar pattern emerges: summaries generated by larger models are slightly easier to detect. this effect is less pronounced than in paragraph writing, with an average accuracy difference of around 6%. we attribute the similar trends between paragraph writing and summarisation to the nature of both tasks: each involves generating text from scratch, conditioned either on retrieved context or article content. we observe the same issues as before for mistral's vietnamese summaries. tst for sentence-level tst, we observe the lowest accuracy scores across all tasks and detector we attribute part of the reduced performance to the sentence-level setting. comparing the english sentence-level results in compared to paragraph writing and summarisation, tst involves only minimal modifications to human-written text. while detection scores on english paragraphs are slightly lower than for full generation from scratch, they remain substantially higher than for sentence-level tst. this suggests that training-based detectors can identify similarly strong mgt signals in paragraph-level text, regardless of whether the content is generated from scratch or modified at the token level. ## conclusion we present wetbench, a multilingual, multigenerator benchmark for detecting mgt in taskspecific wikipedia editing scenarios. we build the benchmark from two new large-scale, multilingual wikipedia text corpora-wikips and mwnc-which support a range of tasks relevant to the wikipedia and ai communities. based on these data, we define three representative tasks, evaluate multiple prompting strategies, generate mgt from diverse llms using the bestperforming prompts, and benchmark detectors. our benchmark reveals that detectors from di-verse families underperform on our data, with substantial variation across languages, models, and tasks. training-based detectors consistently outperform zero-shot methods but achieve only moderate detection accuracy. these results indicate that existing detectors struggle to generalise beyond generic setups, highlighting uncertainty around their reliability and effectiveness in real-world, editor-driven mgt scenarios on wikipedia. in future work, we plan to extend the benchmark with additional tasks, generators, and languages. we also aim to investigate the generalisability of our findings to open-ended generation tasks and other domains.