Controlling Pre-trained Language Models for Grade-Specific Text Simplification (2023)

Abstract: Text simplification systems rewrite text to make it more readable while preserving its content. However, what makes a text easy to read depends on the intended readers. Recent work has shown that pre-trained language models can simplify text using a wealth of techniques to control output simplicity, ranging from specifying only the desired reading grade level, to directly specifying low-level edit operations. Yet it remains unclear how to set these control parameters in practice. Existing approaches set them at the corpus level, disregarding the complexity of individual inputs and considering only one level of output complexity. In this work, we conduct an empirical study to understand how different control mechanisms impact the adequacy and simplicity of text simplification systems. Based on these insights, we introduce a simple method that predicts the edit operations required for simplifying a text for a specific grade level on an instance-per-instance basis. This approach improves the quality of the simplified outputs over corpus-level search-based heuristics.

## introduction in the nlp task of text simplification, systems are asked to rewrite, restructure or modify an original text such that it improves the readability of the original text for a target audience while preserving its meaning. however, text can be simplified in many different ways and what makes a text simple to read depends on the reader. replacing complex or specialized terms with simpler synonyms might help non-native speakers @xcite , restructuring text into short sentences with simple words might better match the literacy skills of children @xcite . acknowledging that text simplification is highly audience-centric @xcite , recent work has focused on developing techniques to control the original: paracho, the "guitar capital of mexico," makes nearly 1 million classical guitars a year, many exported to the united states. grade 5: paracho is known as the "guitar capital of mexico." the town makes nearly 1 million classical guitars a year, with many exported to the united states. word length ratio (w): 1.19 dependency tree depth ratio (dtd): 1.50 grade 3: paracho is known as the "guitar capital of mexico." the town makes many guitars and sells some in the united states. word length ratio (w): 0.96 dependency tree depth ratio (dtd): 1.00 figure 1 : simplified texts can be obtained by either specifying the target audience (via grade level) or by using low-level control tokens to define the ts operation to be performed relative to the complex text (w, dtd). while specifying a reading grade level might be more intuitive for lay users, it provides weaker control over the nature of simplification to be performed. on the other hand, controlling the outputs' simplicity by setting several low-level properties, such as the number of words or dependency tree depth, provides finer-grained control but can be cumbersome to set by readers, teachers, or other users. as a result, it remains unclear how to operationalize the control of text simplification in practice. prior work sets low-level control values (length, degree of paraphrasing, lexical complexity, and syntactic complexity) at the corpus level by searching for control token values on a development set. this is done via maximizing a utility computed using an automatic evaluation metric, sari, a metric designed to measure lexical simplicity @xcite . while this approach is appealing in its simplicity, it remains unclear whether this approach actually helps control complexity for individual inputs, as the control token values are always set at the corpus level. this work presents a systematic empirical study of the impact of control tokens on the degree and quality of simplifications achieved at the instance level as measured by automatic text simplification metrics. our empirical study shows that most corpus-level control tokens have an opposite impact on adequacy and simplicity when measured by bleu and sari respectively. as a result, selecting their values based on sari alone yields simpler text at the cost of misrepresenting the original source content. to address this problem, we introduce simple models to predict what control tokens are needed for a given input text and a desired grade level, based on surface-form features extracted from the source text and the desired complexity level. we show that the predicted low-level control tokens improve text simplification on a controllable ts task compared to corpus-level searchbased optimization. ## background on controllable text simplification while text simplification has been primarily framed as a task that rewrites complex text in simpler language in the nlp literature @xcite @xcite @xcite , in practical applications, it is not sufficient to know that the output is simpler. instead, it is necessary to target the complexity of the output language to a specific audience @xcite . controllable text simplification can be framed as a conditional language modeling task, where the source text x is rewritten as an output y that presents attributes v as scored by a model p (y |x, v ) @xcite . in sequence-to-sequence models, techniques to control the properties v during generation fall under two categories depending on whether they modify the training process @xcite @xcite as described below, or are supplied as constraints during inference @xcite @xcite . 1 control token mechanisms a straightforward method to capture a target attribute, v , in text generation models is to represent it as a special token appended to the input sequence, [v ; x], which acts as a side constraint @xcite . these constraints can be appended to the source or the target sequence. 2 the encoder learns a hidden representation for this token as for any other vocabulary token, and the decoder can attend to this representation to guide the generation of the output sequence. this simple strategy has been used to control second-person pronoun forms when translating into german @xcite , formality when translating to french @xcite , the target language in multilingual scenarios @xcite and to control style, content, and taskspecific behavior for conditional language models @xcite . we provide an overview of the control tokens introduced in prior work for text simplification in tables 1 and 2 . coarse-grained control over the degree and the nature of the simplification, e.g. via source and target grade levels is easier to interpret by end users ( ## how do control tokens impact ts? study settings we study the impact of setting the low-level control values at the corpus level on the instance-level simplification observed using automatic text simplification metrics. we conduct our analysis on the newsela-grade dataset @xcite , which consists of news articles associated with multiple reference simplifications for diverse reading grade levels, and thus lets us analyze the degree and nature of simplification observed across inputs and target readability levels. this data is collected from newsela, 3 an instructional content platform meant to help teachers prepare a curriculum that matches the language skills required at each grade level and has been used in prior work to benchmark controllable ts models @xcite . it includes up to 4 text rewrites at various complexity levels (defined by u.s. reading grade levels 2-12) for an originally complex text. we use the control tokens defined in sheang et al. (2022) (see instance-level findings following prior work @xcite , we select the control tokens that maximize sari on the newsela-grade development set. we measure the complexity of the outputs generated and compare them with the complexity of the newsela references by computing the automatic readability index @xcite , ari, refer equation 1 ). as can be seen in figure 2 , control tokens set at the corpus level using sari tend to over or undersimplify individual input instances. when the reference distribution exhibits diversity in the degree of the simplification performed, setting corpus-level values is sub-optimal: outputs are frequently overor under-simplified as illustrated by the difference in the reference and predicted grade levels. corpus-level findings figure 3 shows the correlation between 100 sets of control values set at the corpus level and automatic ts metrics computed using the outputs generated: sari, bleu, and fr 3 newsela.com (flesch reading ease): most control tokens have an opposite impact on sari and bleu, except the character length ratio, (c). this suggests that setting their values by optimizing for sari alone at the corpus level can be misleading, as a high sari score can be achieved at the expense of a lower adequacy score. these findings are consistent with the observation of @xcite who note a similar negative correlation between human judgments of simplicity and adequacy and caution that: "improvement in one metric and not the other may be due to this inverse relationship rather than actual system performance". these results lead us to concur with the recommendation of alva-manchego et al. (2021), which advocates for always augmenting sari with an adequacy metric for text simplification evaluation. overall, this analysis highlights important limitations of the simplification abilities provided by setting control tokens based on optimizing sari at the corpus level. we propose instead a simple method to predict these values based on each input instance and the desired output complexity. ## grade-specific text simplification with instance-level control since the simplification for a given instance should depend on the original source text, its complexity, and the desired target complexity, we introduce a control predictor module (cp) that predicts a vector of control token values v for each input x at inference time. figure 4 shows the overall inference pipeline for generating the simplified text using the control token values predicted using cp . predicting control tokens we thus directly train a control predictor (cp(θ): @xmath0. let {x i , y i } ∈ d represent a complex-simple pair, and the access controls associated with this pair be we propose both single and multi-output regression solutions for predicting v as described below: we train a simple feature-based gradient boosting decision trees classifier 4 to predict the control values, v using the catboost library using several surface-form features extracted from the source text as described below: ## results we first discuss the accuracy of the control predictor in estimating the access control values on the newsela-grade dataset and then show the impact of using the predicted control tokens as constraints towards controlling the degree of simplification in the generated outputs. the correlation scores for the control tokens (w, c, dtd, levsim, wr) range from -0.33 (s_w, w) to 0.49 (tg, levsim). the moderate correlation between the source features (prepended with s) and the low-level tokens suggests that the source text influences the nature of simplification that can be performed. additionally, levsim controls the degree of simplification as suggested by its moderate-high correlation with the target grade level and can be considered the most prominent token for balancing the adequacysimplicity tradeoff. ## conclusion we present a systematic analysis of the impact of control tokens set at the corpus level on the degree and quality of simplification achieved by controllable text simplification models at the instance level. our findings show that control tokens exhibit an opposite correlation with adequacy and simplicity. hence, selecting their values at the corpus level based on sari alone leads to over or undersimplifying individual instances. this motivates a new approach to set low-level control tokens during inference by predicting them given a source text and desired target grade level. we show that this approach is effective at improving the quality and controlling the degree of simplification in generated outputs based on automatic evaluation. furthermore, predicted low-level control tokens yield more diverse edit operations than alternative ways of setting control on the newsela-grade dataset. our proposed simple solutions improve the inference capability of the controllable ts model for grade-specific ts and reduce the gap with the oracle over a corpus-level baseline approach. however, more sophisticated techniques can benefit the design and prediction of low-level control values and their usage during inference which we leave to future work.