Benchmarking the Detection of LLM s-Generated M odern C hinese Poetry (2025)

Abstract: The rapid development of advanced large language models (LLMs) has made AI-generated text indistinguishable from human-written text. Previous work on detecting AI-generated text has made effective progress, but has not involved modern Chinese poetry. Due to the distinctive characteristics of modern Chinese poetry, it is difficult to identify whether a poem originated from humans or AI. The proliferation of AI-generated modern Chinese poetry has significantly disrupted the poetry ecosystem. Based on the urgency of identifying AI-generated poetry in the real Chinese world, this paper proposes a novel benchmark for detecting LLMs-generated modern Chinese poetry. We first construct a high-quality dataset, which includes both 800 poems written by six professional poets and 41,600 poems generated by four mainstream LLMs. Subsequently, we conduct systematic performance assessments of six detectors on this dataset. Experimental results demonstrate that current detectors cannot be used as reliable tools to detect modern Chinese poems generated by LLMs. The most difficult poetic features to detect are intrinsic qualities, especially style. The detection results verify the effectiveness and necessity of our proposed benchmark. Our work lays a foundation for future detection of AI-generated poetry.

## introduction large language models (llms) have made nonnegligible progress in various tasks @xcite @xcite @xcite . however, the rapid development of advanced llms has made ai-generated text indistinguishable from human-written text @xcite . this ai-generated content could deceive readers through novel forms of manipulation @xcite @xcite , which makes reliable detection of ai-generated text a challenge @xcite . this phenomenon extends even to the domain of poetry, traditionally considered the exclusive realm of human creative expression @xcite . ai-generated poetry has become increasingly prevalent in the real world. it is foreseeable that the total amount of poetry generated by ai will surpass the cumulative amount of all poetry created in human history @xcite . professional models were shown to have the potential to generate high-quality poems that are engaging to readers @xcite . however, humans have been demonstrated to exhibit limited accuracy in distinguishing between aigenerated poetry and human-written poetry @xcite . empirical studies indicate that humans cannot distinguish poems generated by chatgpt-3.5 and may misclassify aigenerated poems as human-written literary works @xcite . with the development of literature, modern chinese poetry has become one of the most popular literary genres in the current chinese world. modern chinese poetry is a unique genre of poetry, which is obviously different from classical chinese poetry and rhymed english poetry @xcite . specifically, modern chinese poetry is free in form and innovative in language, and is not restricted by format, sentence length, rhythm, or meter @xcite @xcite @xcite . can detection models accurately identify aigenerated modern chinese poetry? the meaningful evaluation of llms must be grounded in cultural context @xcite . previous work on detecting llm-generated text has made effective progress @xcite , but these methods are not applicable to modern chinese poetry. for instance, @xcite proposed that incoherence in text can serve as an indicator of llm generation, but this metric fails to generalize to modern chinese poetry because coherence and fluency are not necessary factors for modern chinese poetry. in recent work, @xcite introduced gecscore, a detection method based on grammatical error correction scores, to distinguish between llm-generated and human-written texts. their core idea relies on the disparity in grammatical errors as a discriminative feature, because they found that human-written texts typically contain more grammatical errors than those produced by llms. this method performs well across diverse real-world text sources. however, gecscore is still not applicable to modern chinese poetry because one of the most unique intrinsic features of modern poetry is its deliberate violation of grammatical conventions @xcite . modern poets achieve maximum rhetorical tension and novel aesthetics by breaking grammatical rules @xcite . therefore, the seemingly incorrect grammar in modern poetry is actually the result of careful design, rendering grammar-based detection methods ineffective. recently, there are authors who submit aigenerated poems to journals or publish them on online platforms without indicating that they come from llms. the proliferation of ai-generated modern chinese poetry has significantly disrupted the poetry ecosystem: (1) it deceives both readers and journal editors, and (2) misleads poetry novices. therefore, developing reliable text source identification techniques is critically urgent and practically significant for distinguishing human-written poetry from ai-generated poetry. however, no detection research has focused on modern chinese poetry. based on the urgency of identifying ai-generated poetry in the real chinese world, in this paper, we propose a novel benchmark for detecting llms-generated modern chinese poetry. our goal is to determine the recognition capabilities of detectors on llms-generated modern chinese poetry. we first constructed a high-quality modern chinese poetry dataset named aigenpoetry, which contains 800 poems written by six professional poets and 41,600 poems generated by four mainstream llms. this dataset focuses on different aspects of poetry, including intrinsic qualities, external structures, and emotions. then, we conducted detection experiments on this dataset using six detectors. the experimental results demonstrate that current detectors cannot be used as reliable tools to detect modern chinese poems generated by llms. among them, the most difficult poetic features to detect are intrinsic qualities, especially style. when conducting in-domain detection experiments on poems with different features generated by different models, roberta-based detector has the best comprehensive detection performance. for data with different characteristics, poems with the same style as human poems are the most difficult to detect, while poems that literally express specific emotions, especially fear, are the easiest to detect. these detection results verify the effectiveness and necessity of our proposed benchmark. main contributions of our work are as follows: ## related work with the improvement in the quality of aigenerated poetry, some studies have made efforts to identify ai-generated poetry, but primarily focused on english poetry. @xcite evaluated humans' ability to discriminate english poems generated by gpt-2 @xcite that meet appearance criteria such as rhyme and repetition. they introduced a new turing test by incentivizing the accuracy of human judgments (i.e., increasing judges' motivation). specifically, they evaluated humans' preference for model-generated poems and human-written poems. the results showed that participants preferred human-written poems to model-generated poems, regardless of whether they were told the source of the poems. furthermore, although humans were motivated to participate in the experiment because of monetary rewards, they were still unable to accurately identify ai-generated poems. porter and machery (2024) explored whether non-professional readers can distinguish between english poems of the same style generated by gpt-3.5 and poems written by famous poets. their experiments proved that poems generated by general ai models are more likely to be misidentified as human-written because these poems are considered simpler and easier to understand. for example, ai-generated poems have obvious themes and emotions. in addition, humans' ability to discern ai-generated poems does not improve with more poetry experience. @xcite used four traditional models to classify ai-generated texts and human-written texts, including random forest (rf) @xcite , support vector machine (svm) (burges, 1998), logistic regression (lr) @xcite and long short-term memory (lstm) networks @xcite . among them, the aigenerated english poems are derived from gpt-3.5 @xcite and bard @xcite . 3 task in our work, the detection task of llm-generated poetry is defined as a binary classification task, formulated as: ## aigenpoetry dataset a high-quality benchmark dataset is a prerequisite for advancing research on detecting poetry generated by llms @xcite . currently, there is no publicly available dataset specifically designed for identifying llms-generated modern chinese poetry. to address this gap, we constructed a novel modern chinese poetry dataset named aigenpoetry. aigenpoetry includes both 800 high-quality poems written by six professional human poets and 41,600 poems generated by four state-of-the-art llms. to the best of our knowledge, this constitutes the first modern chinese poetry dataset comprising data pairs of both poems written by professional poets and those generated by ai. human-written we collected 800 high-quality modern chinese poems written by six professional modern chinese poets, which are particularly valuable due to three distinctive characteristics: llms-generated integrating text generated by multiple llms as training data can significantly enhance the detector's performance in cross-model detection @xcite . therefore, the aigenerated poems in our work are produced by various mainstream llms, including gpt-4.1 (ope-nai) @xcite , deepseek-v3 @xcite , deepseek-r1 @xcite , and glm-4 (zhipu ai) (zhipu, 2024). the performance of llms is highly sensitive to input prompts @xcite . based on the unique characteristics of modern chinese poetry and the way people often use llms to generate poetry in the real chinese world, we carefully designed 13 prompts, recorded as p @xmath0...13} . ## experiment detector we employed a variety of detectors to evaluate aigenpoetry. for statistics-based methods, we utilized fast-detectgpt @xcite , lrr @xcite , log-likelihood @xcite , log-rank @xcite , and binoculars @xcite . for fine-tuningbased approaches, we adopted the roberta @xcite based classifier, which has demonstrated superior performance on detectrl benchmark @xcite . since previous work has mainly been validated on english data, migrating to chinese requires certain configuration modifications. for the log-likelihood, log-rank, and lrr methods, we employed qwen2.5-3b foot_1 as the scoring model to optimize detector efficiency while maintaining a model size comparable to the widely used english scoring model gpt-neo-2.7b foot_2 . for fast-detectgpt, we used qwen2.5-3b as the reference model and qwen2.5-7b foot_3 as the scoring model. for binoculars, we used qwen2.5-7b as the observer, and qwen2.5-7b-instruct foot_4 as the performer. for the roberta-based classifier, we fine-tuned the chinese roberta model foot_5 with 3 epochs, a learning rate of 1e-6, and a batch size of 16. we used the above detectors to conduct both in-domain and generalization (out-of-domain) experiments on aigenpoetry dataset. in-domain ex-periments were trained and tested on datasets with the same characteristics generated by individual llms. both the training set and the test set contain 400 pairs of poems. each pair consists of a humanwritten poem and a llm-generated poem. the data for generalization experiments merged poems generated by multiple llms, but the test set had different features from the training set. we balanced the dataset by up-sampling the human-written poems to match the size of the llm-generated poems across all prompts. temperature experiments to explore the effects of different temperatures, we conducted additional experiments with temperatures of 0.7 and 0.0. the experiment is divided into two steps: ## analysis tables 2 and 3 present the performance of various detectors on the d 1 data pairs and the d 2-13 data pairs (in-domain) with different characteristics, respectively. the analysis of emotions is shown in appendix a.4. in summary, when conducting in-domain detection experiments on poems with different features generated by different models, robertabased detector has the best comprehensive detection performance. for data with different characteristics, poems with the same style as human poems are the most difficult to detect, while poems that literally express specific emotions, especially fear, are the easiest to detect. ## conclusion in this paper, we propose the first benchmark for detecting llms-generated modern chinese poetry. experimental results demonstrate that current detectors cannot be used as reliable tools to detect llms-generated modern chinese poems. the most difficult poetic features to detect are intrinsic qualities, especially style, while poems that literally express specific emotions, especially fear, are the easiest to detect. gpt-4.1-generated poems with the same style as human poems are the most difficult to detect among all the data. the detection results verify the effectiveness of our proposed benchmark. our work reveals the vulnerabilities of existing detectors in this task, and lays a foundation for future detection of ai-generated poetry. meanwhile, we call on the research community to pay attention to the oversight and protection of modern chinese poetry and other forms of artistic creation.