Analyzing the Evolution of Scientific Misconduct Based on the Language of Retracted Papers (2025)

Abstract: Amid rising numbers of organizations producing counterfeit scholarly articles, it is important to quantify the prevalence of scientific misconduct.We assess the feasibility of automated text-based methods to determine the rate of scientific misconduct by analyzing linguistic differences between retracted and non-retracted papers.We find that retracted works show distinct phrase patterns and higher word repetition.Motivated by this, we evaluatetwo misconduct detection methods, a mixture distribution approach and a Transformer-based one.The best models achieve high accuracy (>0.9 F1) on detection of paper mill articles and automatically generated content, making them viable tools for flagging papers for closer review.We apply the classifiers to more than 300,000 paper abstracts, to quantify misconduct over time and find that our estimation methods accurately reproduce trends observed in the real data.

## introduction the integrity of scientific research is increasingly threatened by the rise of so-called paper mills, forprofit organizations that produce and sell fraudulent academic manuscripts to researchers, academics, or students who are under pressure to publish in peerreviewed journals @xcite . often disguised as editing or translation services, paper mills sell manuscripts, author slots on peer-reviewed papers, and citations for existing papers @xcite . papers produced by paper mills can have negative consequences for society as they circulate false claims, erode trust in science, or lead to unjustified academic promotions @xcite . since 2010, at least 5402 retracted papers have been connected to paper mills according to the retraction watch database (the center for scientific figure 1 : we create a dataset of retracted papers by merging resources from retractionwatch.org and openalex.org. we add to that a reference dataset of non-retracted articles to train various classifiers for identifying papers with scientific misconduct. finally, we estimate the development of misconduct over time by running a classifier on a large inference dataset of papers from diverse domains. @xcite . this would be around 2 in every 10,000 papers indexed by scopus in the same time frame. however, this number reflects only cases where the paper mill activity was uncovered. the real number of fabricated papers is likely considerably higher due to their convincing nature @xcite . in this paper, we evaluate the feasibility of estimating the true rate of scientific misconduct by analyzing linguistic differences between retracted and non-retracted papers and specifically papers retracted for reasons of scientific misconduct. based on preliminary data review, the central hypothesis is that paper mill articles share a distinctive writing style characterized by words and phrases stemming from the methods used to produce these articles. we conjecture that the automated tools and/or human ghostwriters share a common style or produce unusual expressions @xcite . to investigate this hypothesis, we construct a text corpus of retracted and non-retracted articles (figure 1 and section 2) and assess linguistic characteristics of retracted papers (section 3). next, we evaluate the performance of two text-based misconduct detection methods -a mixture distribution model and a transformer-based text classifierand apply them to an inference corpus stretching a 42-year time-frame @xcite . we find that (1) retracted articles have distinctive language patterns and lower lexical diversity than non-retracted articles, (2) paper mill content is detected with an f1 score of 0.93, (3) there is high correlation (𝜌 = 0.79) between the models' estimates and the observed rate of misconduct retractions, and (4) predicted rates of misconduct are hard to interpret but accurately capture trends. more details on the results can be found in section 6 and a relation to previous work in section 7. the main contributions of our work are (1) a balanced dataset of retracted and non-retracted articles, containing abstracts and full-text sections, (2) a quantification model based on a mixture distribution to directly estimate the rate of papers containing misconduct from a collection of articles and (3) three transformer-based classifiers each classifying one of the labels paper mill, randomly generated content and falsification. we make our code, data, and trained models available on github foot_0 . ## a dataset of retractions figure 1 gives an overview of the provenance and size of the datasets used in this study. we utilize a dataset of retracted papers originating from the blog retractionwatch.com merged with information from the scientific publication repository openalex.org (see subsection 2.1). then, we crawl open-access pdf articles from the web to add full-text data to this dataset (see subsection 2.2). for comparative analysis, we create a reference corpus of non-retracted articles with the same temporal and topical distribution (see subsection 2.3). ## language characteristics of retracted papers to analyze linguistic differences between retracted and non-retracted papers, we compare log-odds of word and n-gram occurrences (subsection 3.1) and investigate the significance of differences in word repetitions (subsection 3.2). ## identifying scientific misconduct the language analysis results in section 2 suggest that retracted papers can be identified through statistical methods based on the differing linguistic structure. building on this, we focus on specifically identifying retractions involving scientific misconduct next. we present two methods to achieve this, a quantification framework based on a mixture distribution (subsection 4.1) and a classifier fine-tuned on a pre-trained transformer model (subsection 4.2). ## experiments we evaluate the dqf and transformer approaches at the level of the document collection and the individual documents. first, we infer the ratio of papers retracted for misconduct from a collection of articles using the dqf (section 5.1). second, we compare both approaches by classifying misconduct on the document level (section 5.2). finally, we use both the best-performing transformer from section 5.2 and the dqf to quantify misconduct through inference on a collection of randomly sampled research articles to explore temporal trends (section 5.3). ## discussion in this section, we revisit the most important results and discuss implications of the findings. the dqf estimates seem as accurate as the classifier-based ones except in health sciences. we observe that the dqf approach seems to function well in the domains of life sciences and physical sciences and a little less in social sciences. for health sciences, it returns seemingly overestimated and highly varying results. this might be explained by the widespread use of formulaic language in the health sciences domain. the 𝛼 is not a precise estimate for the true ratio of misconduct. our methods estimate that 10-15% of papers are involved in misconduct. the evaluation in figure 2 shows that the method tends to overestimate the 𝛼 for a low ground-truth ratio. this indicates that the actual ratio is likely smaller. this method may not be precise enough to reliably detect small effects. rather than providing an exact fraction of misconduct cases, the 𝛼 value should be interpreted as the proportion of papers that exhibit writing style similarities with paper-milled papers or have a slightly higher-than-average probability of resulting from misconduct. in many cases, the estimated ratio correlates strongly with the reported number of misconduct-related retractions, as the inference study shows. trend analysis is a promising application of these models. the practical use case of this becomes clear if we examine the median delay between the date of publication and the date of retraction in our data, which is 475 days, with the 80th percentile at 3.6 years. the dqf method delivers a computationally efficient way to estimate how the incidence of certain types of misconduct changes in real time in a large collection of scientific articles. this could be used, for example, to measure the effectiveness of anti-fraud policies at large publishers. ## related work many studies examine the phenomenon of retractions and misconduct @xcite @xcite @xcite but they primarily focus on metadata such as citation information rather than textual content. @xcite use the retraction watch dataset to analyze author collaboration networks. @xcite and @xcite study linguistic characteristics of retraction notices but not the article content. we do not find any datasets that combine information about retractions with the content of the associated articles. especially looking at the full text and not only abstracts. detection of scientific misconduct. the extensive use of llm chatbots has led to numerous works focusing on identifying ai-generated content in scientific articles and peer reviews. @xcite and @xcite investigate detecting modified sentences by openai's gpt-3 and gpt-4o models. earlier work from @xcite trains a model to distinguish humanwritten sentences from gpt-2 generated ones. (cabanac and labbé, 2021) present a detection method that identifies "tortured phrases" which they attribute to using scientific text generators such as scigen. however, their work is limited to articles from a single journal. aside from detecting ai-generated text, some authors explore more general methods to automatically detect fraud and misconduct in science. (usman and balke, 2024, 2023) use citation information from retraction cascades to identify potentially retractable articles. @xcite use benford's law to identify falsified data specifically. similar to our work, @xcite use a transformer-based model for paper mill content detection. our work presents a new dataset for this task, which stems from a wider range of science domains and is slightly larger. further, we extend their analysis by more pre-trained models and an analysis of large-scale inference on a longitudinal dataset. ## conclusion in this work, we find distinct linguistic patterns in articles retracted for misconduct, such as overuse of certain expressions and frequent repetition of adjectives, nouns, and adverbs. based on these findings, we train a distributional quantification framework and a transformer-based classifier to track growth trends of scientific misconduct. our classifier achieves an f1 score of 0.93 in detecting paper mill articles and automatically generated content, making it a viable option for flagging fraud-ulent papers for human review. further, we show that a computationally simpler approach based on a mixture distribution model can estimate trends of misconduct in life and physical sciences. however, in health sciences, the transformer-based classifier performs better. for future work, we will investigate how metadata such as citation networks and affiliation can be incorporated into detecting misconduct and lead to increased performance in cases where the text-based approach does not yield sufficient results.