Beemo: Benchmark of Expert-edited Machine-generated Outputs (2025)

Abstract: The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most existing MGT benchmarks include single-author texts (human-written and machine-generated). This conventional design fails to capture more practical multi-author scenarios, where the user refines the LLM response for natural flow, coherence, and factual correctness. Our paper introduces the Benchmark of Expert-edited Machine-generated Outputs (Beemo), which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases, ranging from creative writing to summarization. Beemo additionally comprises 13.1k machine-generated and LLM-edited texts, allowing for diverse MGT detection evaluation across various edit types. We document Beemo’s creation protocol and present the results of benchmarking 33 configurations of MGT detectors in different experimental setups. We find that expert-based editing evades MGT detection, while LLM-edited texts are unlikely to be recognized as human-written. Beemo and all materials are publicly available.

## introduction the rapid advancement of large language models (llms) has significantly improved their ability to assist users with a wide range of writing tasks @xcite @xcite . while the benefits of llms are commendable, their widespread adoption has raised concerns regarding authenticity of textual content and potential malicious uses within the domains of news, social media, science, and education @xcite @xcite @xcite . over the last few years, a broad range of machine-generated text (mgt) detection bench-marks have been created to facilitate the development of reliable detectors, aimed at mitigating the risks associated with the misuse of llms across different domains and languages (e.g., @xcite @xcite @xcite . however, most of them focus on a single-author scenario, comprising only machine-generated and humanwritten texts. this well-established design does not account for common applications of llms, where the user refines the llm's response for natural flow, coherence, and factual correctness. this paper introduces the benchmark of expertedited machine-generated outputs (beemo foot_0 ), which consists of 6.5k texts written by humans, generated by ten open-source instruction-finetuned llms and edited by expert annotators, who are well-experienced in refining llm-generated content. furthermore, each mgt is edited by two state-of-the-art llms using several diverse editing prompts, which results in 13.1k machine-generated & llm-edited texts. beemo covers five use cases: open-ended generation, rewriting, summarization, and open & closed question answering (qa). our design enables various diagnostic evaluation scenarios, ranging from out-of-domain mgt detection to analyzing how an mtg detector's behavior changes after the llm response is refined by experts and "humanized" by other llms. our main contributions are: (i) we create beemo, one of the first multi-author benchmarks of llm-generated & expert-edited responses for fine-grained mgt detection, which counts 19.6k texts in total; (ii) we evaluate 33 configurations of zero-shot and pretrained mgt detectors; (iii) we release beemo foot_1 and all annotation materials foot_2 . ## related work ## beemo figure 1 outlines our high-level methodology for creating beemo, which includes the following stages: generating instruction-finetuned llms' responses ( §3.1), editing the responses by expert annotators ( §3.2) and state-of-the-art llms ( §3.3). ## experimental setup we use beemo as an out-of-domain benchmark to evaluate generalization abilities of 33 zero-shot and pretrained mgt detectors' configurations in seven binary classification task formulations, which rely on single-and multi-author text versions. we list the detectors below and detail them in appendix g. pretrained mgt detectors we consider the following pretrained detectors: (i) radar 14 @xcite ; and (iv) openai robertabase foot_16 /large 18 . ## results & analysis this section describes our empirical evaluation results on beemo. we report the overall results in expert-based editing evades detection analyzing the results in the h vs. m/e/l/g scenarios, we find that all detectors often fail to recognize an mgt after it is refined by experts, with the auroc scores decreasing by up to 22% for zeroshot (e.g., binoculars, log probability, and log-rank) and pretrained (e.g., aigc mpu, openai roberta-base/large) detectors. in contrast, many zero-shot detectors identify llm-edited texts as mgts, with binoculars, rank, log-rank, and log probability demonstrating stronger generalization abilities. however, radar, aigc mpu, and ope-nai roberta-base/large exhibit a random guessing performance across various edit types, which suggests a strong distribution shift. llm-edited texts are less likely to be recognized as human-authored a comparison of the results in the e/l/g vs. m scenarios confirms our previous finding that the expert-edited texts are more likely to be classified as humanwritten compared to the llm-edited ones. notably, gpt-4o-edited texts are generally easier to identify as mgts compared to expert-and llama3.1-70b-instruct-edited. analysis by detector type here, we compare the results between the zero-shot and pretrained detectors in more detail. the zero-shot detectors (i) better distinguish between human-written and machine-generated texts (h vs. m); (ii) are more generalizable to expert-and llm-edited texts, which is indicated by lower auroc δ-scores (e.g., up to 10% for log-rank, detectllm llr/npr, detectgpt and up to 20% for aigc mpu, mage, and openai roberta-base/large); and (iii) perform better with a larger backbone model in the h vs. m/e/l/g scenarios, however, this effect is less pronounced in the e/l/g vs. m scenarios.