Human- LLM Coevolution: Evidence from Academic Writing (2025)

Abstract: With a statistical analysis of arXiv paper abstracts, we report a marked drop in the frequency of several words previously identified as overused by ChatGPT, such as “delve”, starting soon after they were pointed out in early 2024. The frequency of certain other words favored by ChatGPT, such as “significant”, has instead kept increasing. These phenomena suggest that some authors of academic papers have adapted their use of large language models (LLMs), for example, by selecting outputs or applying modifications to the LLM-generated content. Such coevolution and cooperation of humans and LLMs thus introduce additional challenges to the detection of machine-generated text in real-world scenarios. Estimating the impact of LLMs on academic writing by examining word frequency remains feasible, and more attention should be paid to words that were already frequently employed, including those that have decreased in frequency due to LLMs’ disfavor. The coevolution between humans and LLMs also merits further study.

## introduction after the launch of chatgpt, large language models (llms) began to be widely used and are now transforming many aspects of our work and life, with academic writing being one of them. the coevolution of ai and humans has also been recognized by researchers @xcite . for example, empirical studies from april 2024 observed that the frequency of certain words used in academic papers published in 2023 had changed and confirmed a strong correlation between these changes and the use of llms @xcite . survey results also show that many researchers are utilizing llms in their work @xcite . the detection of machine-generated text (mgt) has also attracted a lot of attention @xcite @xcite , figure 1 : the frequency evolution of some words in arxiv abstracts (they were singled out around april 2024 as either favored or disfavored by chatgpt). but the performance of detectors has also been questioned early on @xcite @xcite . recent studies continue to show that some methods are not sufficiently robust @xcite @xcite . the effectiveness of mgt detectors is also related to the model of llms and the type of text @xcite , and their accuracy may also be exaggerated @xcite . the situations likely to arise in reality are more complicated and are not limited to a binary classification framework @xcite . thus, examining and analyzing the ongoing evolution of word usage remains a useful and meaningful task. figure 1 illustrates the evolution in frequency usage of some of the words that were singled out as favored or disfavored by chatgpt around april 2024. the frequency of "significant" and "additionally" continues to grow, while that of "is" and "are" continues its declining trend, as noted by @xcite . meanwhile, the frequency of some other words (e.g., "intricate" and "delve") associated with llms begins to decrease after march and april 2024, which corresponds to the time when researchers identified these words in ai conference peer reviews @xcite and academic papers @xcite . changes in the words used in academic writing, as discussed above, serve as an excellent example of ai and human coevolution. researchers are constantly proposing new detection techniques, but the language and expressions of llm users are also likely to evolve due to their use of llms @xcite . given the lack of a precise definition, llmgenerated text might be undetectable in certain individual instances. therefore, statistically measuring the impact of llms on a large corpus of texts is a more practical option. this paper focuses on the following key points: withdrawn arxiv papers data withdrarxiv dataset @xcite , containing more than 14,000 arxiv withdrawn papers up to september 2024. ## word frequency analysis the analysis presented in figure 2 is based on the abstracts of all arxiv papers submitted between 2018 and 2024. the frequency of words is calculated on a monthly basis and normalized per 10,000 abstracts. figure 2a shows the frequency of the 4 words highlighted by @xcite and figure 2b presents the frequency of the 6 words emphasized by @xcite . the former paper analyzes academic papers, while the latter focuses on ai conference peer reviews, and the average frequency of these words is shown in figure 2c . the trend is clear: starting from april 2024, the frequency of these well-known llm-style words began to decrease. some other words show patterns of consistent growth or a rise followed by a decline, as illustrated in figure 8a of the appendix. a study published in december 2024 also observed a decline in the use of certain words, such as "delve", in some selected arxiv papers @xcite . while they suggested that this was likely due to the release of gpt-4o in may 2024, we suggest that the main reason is that llms may have given these words a bad reputation. many re-searchers noticed such kind of words in march and april and quickly changed their arxiv abstracts. if new llms were the cause, the drop in word frequency would have been delayed. in addition, the frequency of words like "significant", specifically pointed out by @xcite , continues to grow, as shown in figure 2d . this may be because these terms are relatively common and frequently used, their presence alone would not easily lead one to suspect the text as the product of llms. besides, as presented in we compared the results with the abstracts of the withdrawn papers, as illustrated in figures 3a and 3b . given the small number of withdrawn papers, the 12-month rolling averages of their word frequency are used in the graphs. the frequency of some words, such as "intricate", is higher in the withdrawn papers, but the difference is not very large, as is also the case in figures 8b , 8c and 8d of the appendix. we also categorized the abstracts into two groups based on the first category of the papers: computer science (cs) and others. figure 3c shows that the frequency of two words preferred by llms is higher in the cs category, but their frequency in other categories is also increasing. the pattern holds for different cs subcategories, as seen in figure 3d . to better compare the changes in word frequency, we define r ij (t 1 , t 2 ) (the ratio of word i in the abstracts of category j between periods t 1 and t 2 ) as follows: r ij (t 1 , t 2 ) = f ij (t 1 ) f ij (t 2 ) , where f ij (t ) is the frequency of word i in the abstracts of category j during the time period t . figure 4a represents the ratio r between 2023 and 2022, where some words, like "diffusion", are related to the research topics, but some other words have also become much more common in different fields. the ratio r in figure 4b is calculated using the word frequency in the first quarter of 2024 divided by the word frequency from january 2023 to december 2024. some words like "delve" and "showcasing" actually reached their peak usage from january to march 2024, and such words are very few. figure 7 provides more detailed examples. words that appear more often in cs paper abstracts have also clearly increased in other disciplines. more researchers have now noticed issues with word usage and diversity in llm-generated content @xcite @xcite . based on the above results, people are likely still using llms, but they may avoid some words that are typical of llm output. therefore, detecting llm-generated content in real-world scenarios may become more difficult. ## challenges in machine-generated text detection the first 1000 arxiv papers submitted each year from 2018 to 2025 were utilized for this part of the analysis. we also used the following two simple prompts to examine the differences between original arxiv abstracts and those revised by gpt-4o-mini (@xmath0, top-@xmath1.9): the results in figure 5 reinforce the point that the frequency of certain words increases after llms revision. using prompt p2, aimed at suppressing them, reduces the frequency of such words, although it does not completely eliminate them. figure 6 presents the detection results based on binoculars @xcite , one of the stateof-art mgt detectors, where a lower score indicates a greater probability that the text is machinegenerated. unlike the results obtained with our frequency analysis, the outcomes of binoculars detector on average do not return any difference in score for the real abstracts of papers as a function of time. moreover, the change in the detection score between the original abstracts and the texts processed by llms (true positives) is not significant. furthermore, the prompts used for processing can influence the results of mgt detectors. these results raise doubts about the accuracy of the detectors, given that they are analyzing texts that have been fully processed by llms. ## conclusion and discussion humans and llms are coevolving and we can already conclude that, for this reason, the impact of llms on academic writing will fully assert itself over the long term. according to recent studies, people who frequently use chatgpt for writing tasks can accurately distinguish ai-generated text @xcite , which implies that they are also able to foil mgt detectors. grammarly can sometimes achieve effects simi-lar to those of chatgpt @xcite , and the mix of human-written text and machine-generated text should be very common in academic writing. detecting llm-generated content with accuracy is becoming more difficult, perhaps impossible on a text-by-text basis. our findings suggest that some researchers may intentionally avoid using llm-characteristic terms, and they are not as sensitive to some relatively common words. the gradual decrease in the occurrence of "is" and "are" in arxiv abstracts is an excellent example of such a trend, which we ascribe to a more subtle -and continually increasing-llm influence. therefore, using the frequency of more common words to measure the impact of llms on a vast number of publications will be more reliable, although this approach is less suitable for the precise detection of short texts.