Almost AI , Almost Human: The Challenge of Detecting AI -Polished Writing (2025)

Abstract: The growing use of large language models (LLMs) for text generation has led to widespread concerns about AI-generated content detection. However, an overlooked challenge is AI-polished text, where human-written content undergoes subtle refinements using AI tools. This raises a critical question: should minimally polished text be classified as AI-generated? Such classification can lead to false plagiarism accusations and misleading claims about AI prevalence in online content. In this study, we systematically evaluate *twelve* state-of-the-art AI-text detectors using our **AI-Polished-Text Evaluation (APT-Eval)** dataset, which contains 15K samples refined at varying AI-involvement levels. Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models. These limitations highlight the urgent need for more nuanced detection methodologies.

## introduction the rapid advancement of llms has enabled ai to generate highly fluent, human-like text, raising concerns about detectability and prompting the development of various ai-text detectors @xcite @xcite . however, the distinction between ai-generated and human-written text remains a gray area, particularly when human-authored content is refined using ai tools. if a human-written text is slightly polished by ai, should it still be classified as human-written, or does it become ai-generated? classifying such text as ai-generated can lead to false plagiarism accusations and unfair penalties, especially when detectors flag minimally polished content as aigenerated. 1 1 usatoday news on false ai allegations additionally, reports suggesting that a significant share of online content is ai-generated -such as analyses indicating that over 40% of medium posts are likely ai-written -often overlook the nuance of ai-polished text, which detection tools may misclassify. 2,3 these sweeping claims risk misrepresenting the actual extent of ai involvement, leading to misleading statistics and misplaced skepticism about human authorship. motivated by these issues, our study systematically examines how ai-text detectors respond to ai-polished text and whether their classifications are both accurate and fair. to investigate this issue, we introduce the ai-polished-text evaluation (apt-eval) dataset of size 15k, which systematically examines how aitext detectors respond to varying degrees of ai involvement in human writing. our dataset is built from pre-existing human-written samples that are refined using different llms, such as gpt-4o (openai, 2023), llama3-70b @xcite , deepseek-v3 @xcite , etc., applying degree and percentage based modifications. this allows us to assess how detectors respond to minor and major ai polishing. we analyze the classification accuracy, false positive rates, and domain-specific sensitivities of 12 state-of-the-art detectors, spanning model-based, metric-based, and commercial systems. our findings reveal critical weaknesses in existing ai-text detection systems. ai-text detectors exhibit alarmingly high false positive rates, often flagging very minimally polished text as aigenerated. for instance, minimal polishing with gpt-4o can lead to detection rates ranging from 10% to 75%, depending on the detector. furthermore, detectors struggle to differentiate between minor and major ai refinements, suggesting that they may not be as reliable as previously assumed. we also uncover biases against smaller or older llms, where polishing done by less advanced models is more likely to be flagged than text refined by state-of-the-art llms. on average, 46% of samples polished by llama2-7b are classified as ai-generated, whereas this drops to 23% for deepseek-v3-polished samples. furthermore, our study uncovers domain-specific inconsistencies in detection accuracy. detectors show the greatest vulnerability for the 'speech' domain, while exhibiting comparatively higher robustness to 'paper-abstract' texts. these findings raise concerns about the fairness and generalizability of current detection methods. by shedding light on these issues, our research provides valuable insights into the evolving challenges of ai-assisted writing and the limitations of current ai-text detection methodologies. our code and dataset are publicly available. ## ai-text detectors in this work, we evaluate a total of twelve detectors from three different categories: ## key findings we evaluate the detectors on our apt-eval dataset from multiple perspectives to analyze their response to ai-polished text. our key findings are as follows - ## discussion the findings of this study reveal several critical limitations in current ai-text detectors, particularly in distinguishing between human-authored content that has been subtly refined by ai and fully machine-generated text. a key concern is the high false positive rate associated with minimally polished text. many detectors classify such lightly edited content as ai-generated, which poses serious risks of unjust accusations of plagiarism or academic dishonesty. to address this, we recommend moving beyond binary classification frameworks and adopting tiered or probabilistic labeling schemes that reflect varying degrees of ai involvement. a promising direction for future work is to train detectors not only on purely human-written and fully ai-generated texts but also on ai-polished samples. we hope our released apt-eval dataset will serve as a valuable resource for developing and evaluating such models. furthermore, rather than producing a definitive label, detectors should output prediction probabilities, enabling users to better interpret and trust the system's verdict. we also observe detector biases against older or smaller llms, and recommend training on a more diverse range of llm outputs-including from newer, more human-like models-to improve fairness. domain-specific biases can also be mitigated by fine-tuning detectors for particular genres or text types. also, in the early stage of the pipeline, there can be another model to find the text-domain that will trigger a specific detector accordingly. lastly, we emphasize the importance of interpretability and human oversight in detection tools. developing interpretable detectors that can highlight suspicious segments or stylistic anomalies will empower users to make informed decisions. in high-stakes scenarios, integrating human-in-theloop review mechanisms can further enhance the reliability and fairness of the process. ultimately, addressing these challenges requires a multi-faceted approach that balances technical sophistication with transparency, fairness, and adaptability. ## conclusion our study exposes key flaws in ai-text detectors when handling ai-polished text, showing high false positive rates and difficulty distinguishing minor from major ai refinements. detectors also exhibit biases against older or smaller models, raising fairness concerns. we highlight the need for more nuanced detection methods and release our apt-eval dataset to support further research.