Flesch-Kincaid is Not a Text Simplification Evaluation Metric (2021)

Abstract: Sentence-level text simplification is currently evaluated using both automated metrics and human evaluation. For automatic evaluation, a combination of metrics is usually employed to evaluate different aspects of the simplification. Flesch-Kincaid Grade Level (FKGL) is one metric that has been regularly used to measure the readability of system output. In this paper, we argue that FKGL should not be used to evaluate text simplification systems. We provide experimental analyses on recent system output showing that the FKGL score can easily be manipulated to improve the score dramatically with only minor impact on other automated metrics (BLEU and SARI). Instead of using FKGL, we suggest that the component statistics, along with others, be used for posthoc analysis to understand system behavior.

## introduction critical to any application area is evaluation. evaluation is often accomplished using one or more quantifiable evaluation metrics. evaluation metrics are the main tool for comparing and analyzing approaches @xcite and are often used to define whether progress is being made in a field. a good evaluation metric should be a proper measure of the quality of a particular algorithm and, importantly, should not be "gameable". specifically, an approach should not be able to obtain a better score on the evaluation metric by manipulating the algorithm or output in ways that do not improve the actual quality of the output. in this paper, we examine evaluation for text simplification, specifically, sentence-level text simplification. text simplification aims to transform text into a variant that is easier to understand by a broader range of people while retaining as much of the original content as possible. a range of approaches for text simplification have been pro-posed ranging from lexical simplification @xcite , where only words and phrases are changed, to fully generative approaches that leverage models from machine translation @xcite and recent sequential neural networks @xcite @xcite ). text simplification evaluation has been done with two general approaches: human evaluation and automated metrics. human evaluation relies on annotators to judge the quality of the simplifications on three dimensions: fluency/grammaticality, how well the sentence represents fluent, grammatical text; adequacy, how well the content is preserved; and, simplicity, how simple the text is @xcite . the first two metrics were adapted from other text generation tasks @xcite with the addition of simplicity for text simplification. when human evaluation is used, these three metrics have been consistently employed. human evaluations provide concrete analysis of texts simplification systems along important dimensions, however, human evaluation is costly and is not practical for development, tuning, and other real-time uses. as such, text simplification has also relied on automated metrics for evaluation. automatic evaluation of text simplification has varied more across papers, though three metrics are most commonly employed: bleu, sari, and flesch-kincaid. bleu @xcite compares the n-gram overlap via precision of a system simplification with a human reference simplification and was borrowed from machine translation. bleu was the first metric suggested for text simplification that utilized reference simplifications @xcite , however, it focuses less on simplicity and more on fluency and content preservation. to counter this, sari was proposed as an alternate metric @xcite . sari also compares against human references, but also utilizes the input sentence allowing it to better capture addition and deletion of information. finally, a third automated metric that has been used to measure readability and fluency is @xcite . fkgl was initially proposed in the 1940s @xcite and since then has been used extensively in the medical domain, though it has never been shown to affect actual comprehension @xcite . fkgl combines two text statistics to calculate the score: the average number of syllables per word and the average number of words per sentence: in recent text simplification papers, both bleu and sari are common evaluation metrics @xcite @xcite @xcite . fkgl is not as popular as it was before sari was introduced, but it continues to be used as an evaluation metric in recent papers @xcite @xcite @xcite @xcite . in this paper, we argue that fkgl is not a proper evaluation metric for text simplification and should not be used to evaluate text simplification systems, i.e., alongside other metrics like bleu and sari. fkgl was one of the first metrics suggested for text simplification @xcite and has been used by many as an evaluation metric to compare systems. however, fkgl was not originally designed to evaluate system output (it was designed to measure human output) and, because of its simplistic nature, is very easy to game, either explicitly (as we do in this paper) or implicitly by certain model biases (e.g., text simplification algorithms that split sentences will tend to have better fkgl scores). recent work has shown that systems with good fkgl scores are not necessarily correlated with high-quality simplifications @xcite , however, this is the first in-depth analysis of the fkgl metric for evaluation and where specific system transformations are analyzed. to explore how fkgl can be manipulated, we introduce six simple methods for modifying system output and examine the impact these modifications have on automated evaluation metrics. the modifications could be made explicitly by a system in an attempt to improve their score, or, more worrisome, implicitly. in addition to the fkgl scores, we also present and and discuss how bleu and sari respond to the modifications. we show that with some very minor modifications, fkgl can be improved dramatically with minimal effect on the other two evaluation metrics. we conclude with some recommendations on how to incorporate fkgl-like metrics into text simplification analysis. ## history of flesch-kincaid the earliest version of the flesch-kincaid readability formula appears in flesch's doctoral dissertation @xcite and calculated based on the the average number of words per sentence, the number of affixes, and the number of references to people. the formula was derived based on the mccall-crabbs standard test lessons in reading @xcite , a standardized test given to children in grades 3-7. the mccall-crabbs tests contains 376 passages with 8 reading comprehensive questions per passage. each lesson is labeled with its difficulty as a grade level. based on these texts, flesch developed the formula to predict the grade of children in grades 3-7 who answered at least 75% of the questions correctly about a given passage. the original goal of the formula was to help students track their progress. five years later, he published a new formula: the reading ease score @xcite . he adjusted the original formula by recomputing the coefficients and replacing previous text measurements with the ones used today, the average number of syllables and the average sentences length. like the original study, this new formula was validated with children and was based on the same criterion, mccall-crabbs standard test lessons in reading. flesch-kincaid grade level is a variation of the reading ease formula with readjusted weights and is the formula that has been commonly used in text simplification evaluation. the formula was derived three decades later @xcite specifically to evaluate the readability of technical materials for military personnel. 531 navy personnel in four technical training schools at navy bases were tested for their reading comprehension level according to the comprehension section of the gates-mcginitie reading test as well as their comprehension of 18 passages from rate training manuals. despite the fact that this formula was derived from navy personnel, with military-based material, and specifically for navy use, it has been broadly used in a range of settings to evaluate the readability of text, for example, it is commonly used to guide text generation by medical writers in the medical domain and even microsoft word includes both the flesch reading ease and fkgl scores @xcite . we provide this background to raise some concerns based on its origins for its application for text simplification evaluation. the inputs of the formula -sentence count, word count, and syllable countwere decided based on a study in the 1940s where modern text analysis tools were not available. both the flesch reading ease and fkgl scores were developed based on very specific corpora and very targeted populations, children grades 3-7 in the former case and navy personnel in the latter case. most importantly, the text passages used to collect data were always written by people and assumed to be mostly free of errors in terms of writing. these assumption cannot be made for text generated by automated systems. ## data to understand the problems with fkgl, we analyzed the output from the five text simplification systems examined by @xcite , a number of which are state-of-the-art: pbmt-r @xcite , a phrase-based approach based on statistical mt; hybrid @xcite , a model that combines sentence splitting and deletion with pbmt-r; encdeca, a basic neural encoder-decoder model with attention; and two deep reinforcement learning models, dress and dress-ls @xcite . there are two main corpora that are used to train and evaluate text simplification systems: wikipedia @xcite , which consists of automatically aligned sentences between english wikipedia and simple english wikipedia, and newsela @xcite , which consists of news articles manually simplified at varying levels of simplicity. we present the results for the newsela corpus since it involves explicit human simplification and has been shown to be less noisy than the wikipedia corpus @xcite . we also conducted the experimental analysis on the wikipedia corpus and saw similar results. ## experimental analysis we applied each of the modification techniques to a varied percentage of output sentences, from 10% to 100% in increments of 10%, for the five text simplification systems. the sentences to be modified were randomly selected from the system output. we calculated fkgl 1 as well as bleu @xcite and sari 2 @xcite to observe how the modifications affect other common text simplification evaluation metrics. to account for per-sentence variation and randomness in some of the modification approaches, we repeated the experiments 100 times and averaged the results. ## a better approach fkgl should not be used as an evaluation metric. instead, it can be used for post-hoc analysis to understand the behavior of the systems. even better, rather than reporting the fkgl score, which can be affected by multiple types of changes in the system, papers can report the individual components of fgkl, i.e., the average sentence length and the average number of syllables. this demystifies the readability score and provides concrete information about the types of changes that are being made by the systems. a comparative analysis of 30 metrics showed that these features are better correlated with human judgement than fkgl @xcite , and some recent papers have reported the average sentence length statistic already @xcite @xcite . these two metrics can be supplemented with other corpus statistics that also help understand what changes the systems are making, e.g., the proportion of sentences that are split.