Free-text Rationale Generation under Readability Level Control (2025)

Abstract: Free-text rationales justify model decisions in natural language and thus become likable and accessible among approaches to explanation across many tasks. However, their effectiveness can be hindered by misinterpretation and hallucination. As a perturbation test, we investigate how large language models (LLMs) perform rationale generation under the effects of readability level control, i.e., being prompted for an explanation targeting a specific expertise level, such as sixth grade or college. We find that explanations are adaptable to such instruction, though the observed distinction between readability levels does not fully match the defined complexity scores according to traditional readability metrics. Furthermore, the generated rationales tend to feature medium level complexity, which correlates with the measured quality using automatic metrics. Finally, our human annotators confirm a generally satisfactory impression on rationales at all readability levels, with high-school-level readability being most commonly perceived and favored.

## introduction over the past few years, the rapid development of machine learning methods has drawn considerable attention to the research field of explainable artificial intelligence (xai). while conventional approaches focused more on local or global analyses of rules and features @xcite , the recent development of llms introduced more dynamic methodologies along with their enhanced capability of natural language generation (nlg). the self-explanation potentials of llms have been explored in a variety of approaches, such as examining free-text rationales @xcite or combining llm output with saliency maps @xcite . although natural language explanation (nle) established itself to be among the most common approaches to justify llm predictions @xcite , free-text rationales were found to potentially misalign with the predictions and thereby mislead human readers, for whom such misalignment seems hardly perceivable @xcite . furthermore, it remains unexplored whether freetext rationales represent a model's decision making, or if they are generated just like any other nlg output regarding faithfulness. in light of this, we aim to examine whether free-text rationales can also be controlled through perturbation as demonstrated on nlg tasks @xcite . if more dispersed text complexity could be observed in the rationales, it would indicate a higher resemblance between rationales and common nlg output, as we assume the llms to undergo a consistent decision making process on the same instance even under different instructions. targeting free-text rationales, we control text complexity with descriptive readability levels and evaluate the generated rationales under various frameworks to investigate what effects additional instructions or constraints may bring forward to the nle task (figure 1 ). although the impact of readability (stajner, 2021) has rarely been addressed for nles, establishing such a connection could benefit model explainability, which ultimately aims at perception @xcite and utility @xcite of diverse human recipients. our study makes the following contributions: first, we explore llm output in both prediction and free-text rationalization under the influence of readability level control. second, we apply objective metrics to evaluate the rationales and measure their quality across text complexity. finally, we test how human perceive the complexity and quality of the rationales across different readability levels. 2 2 background text complexity the notion of text complexity was brought forward in early studies to measure how readers of various education levels comprehend a given text @xcite . prior to recent developments of nlp, text complexity was approximated through metrics including flesch reading ease @xcite , gunning fox index @xcite , and coleman-liau index @xcite . these approaches quantify readability through formulas considering factors like sentence length, word counts, and syllable counts. as the most common readability metric, fre was often mapped to descriptions that bridge between numeric scores and educational levels @xcite . @xcite applied readability level control to text summarization through instruction-prompting. in their study, descriptive categories were prompted for assigning desired text complexity to llm output. ## method readability level control as demonstrated in figure 1 , in step 1, we incorporate instructionprompting into the prompt building. the prompts consist of three sections: task description, few-shot in-context samples, and instruction for the test instance. after task description and samples, we add a statement aiming for the rationale: elaborate the explanation in {length} foot_2 to a {readabil-ity_level} student. then we iterate through the data instances and readability levels in separate sessions. we adapt the framework of @xcite to four readability levels based on fre score ranges ( evaluating free-text rationales in light of the problematic adaption to readability-related tasks and major issues in reproducibility of the aforementioned nle evaluation metrics, we exploit the overlap between nle and nlg, we adopt tiger-score @xcite , an nlg metric that is widely applicable to most tasks, for evaluating the generated free-text rationales ( §4.2). applying finetuned llama-2 @xcite , the metric was proposed to require little reference but instead rely on error analysis over prompted contexts to identify and grade mistakes in unstructured text. nevertheless, the approach could sometimes suffer from hallucination (or confabulation), similar to the common llm-based methodologies. ## results we collect predictions and rationales from four models over four datasets ( §4.1). figure 2 presents a data instance to exemplify the output of llm inference as well as each aspect of evaluation. more rationale examples are provided in appendix a.2. the four models achieve divergent accuracy scores on the selected tasks ( free-text rationales generated under instructionprompting show a correlative trend in text complexity. figure 3 reveals that the requested readability levels introduce notable distinction to text complexity, though the measured output readability may not fully conform with the defined score ranges ( evaluation with tigerscore is based on error analyses through score reduction: each identified error obtains a penalty score (<0), and the entire text is rated the summation of all the reductions. such design gives 0 to the texts in which no mistake is recognized; in contrast, the more problematic a rationale appears, the lower it scores. in our results (figure 4 ), we derive non-zero score through further dividing the full-batch score by the amount of non-zero data points, since around half of the rationales are considered fine by the scorer. we also apply the same processing method to selfevaluation with the original model. in most cases, full-batch tigerscore proportionally decreases along with text complexity, whereas non-zero and self-evaluation do not follow such trend. in comparison to tigerscore, bert similarity provides rather little insight into rationale quality (appendix c). although complex rationales resemble the references more, the correlation between readability and similarity remains weak. plus, the scores differ more across datasets than across models, making the outcomes less significant. we conduct a human study ( §4.2) with five annotators, who took around five hours for the 200 samples. while calculating agreement, we simplify the results on readability, coherence, and informativeness into two classes owing to the binary nature of 4-point likert scale; the originally annotated scores are used elsewhere. we register an agreement of krippendorff's α = 3.67% and fleiss' κ = 13.92%. ## discussions our study aims to respond to three research questions: first, how do llms generate different output and free-text rationales under prompted readability level control? second, how do objective evaluation metrics capture rationale quality of different readability levels? third, how do human assess the rationales and perceive the nle outcomes across readability levels? ## related work rationale evaluation free-text rationale generation was boosted by recent llms owing to their capability of explaining their own predictions @xcite . despite lacking a unified paradigm for evaluating rationales, various approaches focused on automatic metrics to minimize human involvement. ν-information @xcite provided a theoretical basis for metrics such as receval (prasad et al., 2023), rev @xcite , and rora @xcite . however, these metrics require training for the scorers to learn new and relevant information with respect to certain tasks. alternatively, several studies applied llms to perform reference-free evaluation @xcite . similar to tigerscore @xcite , instructscore (xu et al., 2023) took advantage of generative models, delivering an reference-free and explainable metric for text generation. however, these approaches could suffer from llms' known problems such as hallucination. as the common methodologies hardly considering both deployment simplicity and assessment accuracy, luo and specia (2024) pointed out the difficulties in designing a paradigm that faithfully reflects the decision-making process of llms. readability of llm output rationales generated under readability level control share features similar to those reported by previous studies on nlg-oriented tasks, such as generation of educational texts @xcite ), text simplification @xcite , and summarization @xcite , given that instruction-based methods was proven to alter llm output in terms of text complexity. @xcite found the readability of llm output to vary even when controlled through designated prompts. gobara et al. (2024) pointed out the limited influence of model parameters on delivering text output of different complexity. while tuning readability remains a significant concern in text simplification and summarization, llms were found to tentatively inherit the complexity of input texts and could only rigidly adapt to a broader range of readability @xcite . ## conclusions in this study, we prompted llms with distinct readability levels to perturb free-text rationales. we confirmed llms' capability of adapting rationales based on instructions, discovering notable shifts in readability with yet a gap between prompted and measured text complexity. while higher text complexity could sometimes imply better quality, both metric scores and human annotations showed that rationales of approximately high-school complexity were often the most preferred. moreover, the evaluation outcomes disclosed llms' sensitivity to perturbation in rationale generation, potentially supporting a closer connection between nle and nlg. our findings may inspire future works to explore llms' explanatory capabilities under perturbation and the application of other nlg-related methodologies to rationale generation.