CLEAR : A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Models (2025)

Abstract: While LLMs have been extensively studied on general text generation tasks, there is less research on text rewriting, a task related to general text generation, and particularly on the behavior of models on this task. In this paper we analyze what changes LLMs make in a text rewriting setting. We focus specifically on argumentative texts and their improvement, a task named Argument Improvement (ArgImp). We present CLEAR: an evaluation pipeline consisting of 57 metrics mapped to four linguistic levels: lexical, syntactic, semantic and pragmatic. This pipeline is used to examine the qualities of LLM-rewritten arguments on a broad set of argumentation corpora and compare the behavior of different LLMs on this task and analyze the behavior of different LLMs on this task in terms of linguistic levels. By taking all four linguistic levels into consideration, we find that the models perform ArgImp by shortening the texts while simultaneously increasing average word length and merging sentences. Overall we note an increase in the persuasion and coherence dimensions.

## introduction text rewriting is an important task in natural language processing, with applications in style transfer @xcite @xcite , paraphrase generation @xcite , and text simplification @xcite @xcite , among others. it can be seen as a form of controllable text generation @xcite , where a given text is modified based on specific requirements, such as improving readability, accuracy, or suitability for a particular context @xcite . recent advancements in large language models (llms) have shown promising performance on a wide range of text generation tasks, allowing them to refine text based on natural language instructions to produce high-quality rewrites @xcite . a relevant but underexplored application of text rewriting is the task of argimp, i.e. rephrasing an argumentative text, with the objective of enhancing its overall quality. arguments can be refined through various linguistic modifications, including lexical, syntactic, semantic, and pragmatic changes. llms have been increasingly studied in the domain of computational argumentation, with recent works showcasing their capabilities in the tasks of argument mining @xcite , argument generation @xcite @xcite , and argument quality assessment @xcite . this work aims to bridge this gap by investigating the linguistic transformations performed by llms when prompted to improve an argumentative text. specifically, we analyze how these models alter texts at four distinct linguistic levels: word choice (lexical), sentence structure (syntactic), meaning shifts (semantic), and rhetorical effectiveness (pragmatic). by systematically categorizing and evaluating these modifications, we aim to better understand the role of llms in argimp and their potential for enhancing argumentative writing (see figure 1 ). llms are known to exhibit biases in text generation settings @xcite . due to a lack of research investigating llms in an argimp scenario, it is not clear what, if any, biases they exhibit in this setting. a so called verbosity bias can be observed in llms @xcite , as well as a positivity bias @xcite @xcite . these biases are of particular interest in an argimp setting. the models may consider longer texts as better, and thus make less changes, or shift the tone and inadvertently change the meaning of original improved lexical syntactic semantic pragmatic how does the vocabulary change? how is the sentence structure modified? how does the meaning shift? how is the argument's effectiveness affected? ## related work the capabilities of llms in the field of computational argumentation have been previously explored, particularly in the areas of argument mining @xcite and argument quality assessment @xcite . 2024 ) make use of llms to rewrite existing arguments to make them more appealing to readers of a certain political ideology. argumentative writing support there is a growing line of research that focuses on developing argumentative writing support tools that provide users with feedback on the quality of their argumentative texts with the objective of guiding them in generating high-quality persuasive texts @xcite . for instance, al, an adaptive learning support system for argumentation skills, offers formative feedback through in-text highlighting of argumentative components, qualitative scores and graph-based visualizations of argument discourse structures @xcite . more recently, @xcite introduced artist, a framework that integrates llm-based improvement suggestions. they conducted a user study with students to evaluate the effectiveness of such feedback. the results indicate that the students generally find the feedback provided by the llms to be helpful and of high quality. these frameworks primarily adopt a reader-oriented perspective, as their goal is to provide feedback that helps students develop the skills to refine their texts manually. our work, in contrast, takes a text-centric approach, focusing on the linguistic quality of llm-generated improvements in argumentative texts. ## argument improvement with llms we aim to evaluate the quality of argumentative texts improved by llms. argumentation occurs in various contexts; our work centers on the following setting: (i) we focus on global argumentation rather than local arguments. (ii) our analysis is limited to monological texts, excluding dialogical debates. (iii) we primarily assess intrinsic, i.e. text-focused, quality rather than extrinsic reader-focused text effectiveness @xcite . we consider an argument a ′ to be improved, if q(a ′ ) > q(a), where q is an abstract function that measures argumentative quality, a is the orig-inal argument and a ′ is an updated version of a that had changes made. we explicitly do not define a universal q here, as what constitutes quality in argumentation is dependent on both intrinsic as well as extrinsic factors. an argument made by a 5-year old with grammar on the level of a high school student could generally be considered good, whereas the exact same argument, if made by a professor at a research conference, would likely be considered bad. furthermore, an argument with wrong premises can still convince people, such as in the case of fake news, and while its intrinsic quality may be low, it can be considered successful if people believe it. with our analysis we aim to answer the following research questions: (i) what changes on linguistic levels do llms make in an argimp setting? (ii) what biases do llms exhibit in an argimp setting? (iii) do models of different sizes behave differently in an argimp setting? ## evaluation setup we split our evaluation into three analyses: the linguistic analysis on four linguistic levels, the bias analysis and the analysis of the argumentative discourse structure. the last one is specific to our use case of argument improvement, whereas the other analyses can be applied to all other text generation domains. ## discussion the analysis based on the scores of our text generation evaluation pipeline shows that on the lexical level overall text length decreases. we further observe an increase in 4 to 6 syllable words and a strong decrease in shorter words. on the syntactic level we note many merge and fuse operations, which means that the original text is shortened or remixed into existing sentences. then, on the semantic level, we note a decrease in the depth of the rst parse trees. finally, on the pragmatic level, we observe an increase in terms of coherence and persuasion, which indicates that the argument quality, in general, improved. manual analysis supports these empirical findings. we note that the models keep the overall structure, where it exists, and do not delete or add significant chunks of text. instead, the models refine and enhance what is already there. notably, manual analysis revealed weak parts of certain arguments, which the model did not address or remove. these results together suggest that the models perform the improvement by focusing the texts: ## conclusion by making use of the clear pipeline, consisting of commonly used text generation metrics mapped to linguistic levels and performing an analysis on the individuals levels, we have found that llms make the texts more focused in an argimp setting, in the sense that (i) the texts become shorter, (ii) the average length of words increases, (iii) semantically the texts do not change. our results suggest that the models perform well for text improvement. we note two positive factors: (i) the length of the texts decreases, but notably not in the case of the microtexts corpus, where the input texts are already quite short, and (ii) the quality increases. we note small differences in model behavior in this task. larger models performed better in both quality of the output texts and appear to make the texts more focused than the small models. a positivity bias could not be identified, instead the models appear to aim to make the texts more neutral, instead of shifting the tone consistently to positive or negative levels. lastly, we could not identify a length bias. the models do not appear to prefer texts of certain lengths. we note the tendency of llama 3.1 in particular to use longer words, which could be a form of bias. our results suggest that this is done to increase information density without negatively impacting readability as evident by the scores on the lexical level of our analysis.