Quantifying the Risks of LLM - and Tool-assisted Rephrasing to Linguistic Diversity (2025)

Abstract: Writing assistants and large language models see widespread use in the creation of text content. While their effectiveness for individual users has been evaluated in the literature, little is known about their proclivity to change language or reduce its richness when adopted by a large user base. In this paper, we take a first step towards quantifying this risk by measuring the semantic and vocabulary change enacted by the use of rephrasing tools on a multi-domain corpus of human-generated text.

## introduction writing assistants such as grammarly or quillbot are widely used in writing tasks by native and nonnative speakers alike. to aid the user in the composition of text, writing assistants (wats) provide sophisticated and comprehensive functions, such as grammar correction, spell-checking, and specialized rephrasing and embellishment. more recently, large language models (llms) are increasingly being integrated into writing assistants @xcite , including grammarly and google's smart compose @xcite . these advanced wats offer substantial assistance in the writing process @xcite , but also introduce challenges such as hallucinations or inconsistent content and style @xcite @xcite . after the introduction of chatgpt, language models can also be -and are -used as writing assistants directly due to their chat functionality and ease of use. despite their widespread use, however, there is a notable lack of systematic quantitative investigations into how the use of writing assistants alters the produced text. existing studies tend to focus on the effectiveness and accuracy of wats @xcite , but discount the downstream impact that a pervasive adoption of such systems might have on the style and diversity of the language we use in written day-to-day communication. contributions. in this paper, we address this research gap with a quantitative analysis of the effect that the use of (semi)automated rephrasing tools has on the produced text. we experiment with four traditional wats and six llms to determine the effect of tool-assisted rephrasing from the perspective of corpus-level text diversity. using token-level and vector-level metrics, we provide a comprehensive overview of the ways in which these tools modify text -and where worries about linguistic diversity may or may not be warranted. ## related work academic work on writing assistants is relatively scarce, and focuses predominantly on measuring the effectiveness of wats as a tool for improving writing efficiency, the accuracy of grammar, or for spell checking @xcite , and we are unaware of any studies of language diversity as a result of writing assistant usage. furthermore, analyses of wats with respect to content are predominantly qualitative or based on manual evaluation @xcite , which introduces the potential for subjective biases. prior research tends to examine individual tools, such as grammarly @xcite or quillbot (amyatun and kholis, 2023) rather than providing a comparative analysis, thereby limiting generalizability. with regard to language models, prior work more commonly focuses on the collaborative writing effectiveness between llms and humans, rather than considering textual change @xcite , or on evaluating grammar and style, rather than corpus-level diversity @xcite . while @xcite investigate the risk of diversity reduction as a result of language model usage, they only provide a case study on the example of chatgpt. otherwise, prior investigations have left the dimension of linguistic diversity largely unexplored to focus on quality @xcite , while their application scenarios (often limited to academic writing tasks) are narrower than the crossdomain texts we consider here. in contrast to the above works, we conduct a quantitative evaluation that systematically compares both traditional writing assistants and multiple llm-based tools across texts from diverse domains. our analysis considers not only surfacelevel measures but also semantic-level metrics, with a particular focus on linguistic diversity. ## data we use texts from a variety of domains that we rephrase with the help of writing assistants and language models, as described in the following. ## experimental setup to measure changes to the texts as a result of rephrasing, we consider word-based measures and embedding-based measures, both computed at the paragraph level. for definitions, see appendix e. ## results in the following, we discuss text changes due to rephrasing by writing assistants (wats) and language models (llms) from the perspective of word-based measures and vector-based measures. ## discussion and conclusion our findings indicate three major take-aways.