Prompt-based Learning for Text Readability Assessment (2023)

Abstract: We propose the novel adaptation of a pre-trained seq2seq model for readability assessment. We prove that a seq2seq model - T5 or BART - can be adapted to discern which text is more difficult from two given texts (pairwise). As an exploratory study to prompt-learn a neural network for text readability in a text-to-text manner, we report useful tips for future work in seq2seq training and ranking-based approach to readability assessment. Specifically, we test nine input-output formats/prefixes and show that they can significantly influence the final model performance. Also, we argue that the combination of text-to-text training and pairwise ranking setup 1) enables leveraging multiple parallel text simplification data for teaching readability and 2) trains a neural model for the general concept of readability (therefore, better cross-domain generalization). At last, we report a 99.6% pairwise classification accuracy on Newsela and a 98.7% for OneStopEnglish, through a joint training approach. Our code is available at github.com/brucewlee/prompt-learning-readability.

## introduction readability assessment evaluates the reading difficulty of a given piece of text @xcite . the early traditional readability assessment methods like @xcite utilized a linear regression formula fitted to data from large-scale reading experiments on human subjects. more recently, readability assessment has often been viewed as a classification task @xcite . under this classification-based task formulation, models using various handcrafted features @xcite , computer-generated features @xcite , or both (lee et al., 2021) ‡ woong sung (bruce) lee was on leave from the university of pennsylvania during the research period. among the varying approaches to readability assessment, fine-tuning deep transformer models @xcite , that are pre-trained with language modeling objectives (e.g. masked language modelling, next sentence prediction), has proven highly effective in multiple reports @xcite . so far, encoder-only transformer architectures like bert @xcite and roberta @xcite have been the go-to approach, and few reports discuss the applicability of other architecture types. further, there is no report on how readability assessment can be cast in a text-to-text task formulation ( §2). but recent reports @xcite show that a text-to-text is promising for multiple downstream tasks. the main contribution of this paper is that we fine-tune full encoder-decoder transformer architectures (also referred to as sequence-to-sequence) and check if they can learn about text readability. a sequence-to-sequence model has been previously fine-tuned for downstream tasks like document ranking @xcite , but few reports discuss whether the architecture can learn about readability. we fine-tune bart @xcite and t5 @xcite on the popular onestopenglish @xcite and newsela @xcite data. then, we measure their performance on two other datasets with readability annotations for us and cefr curricula, respectively. we also conduct methodological explorations on how a sequence-to-sequence model can be welltrained to learn text readability. this includes how input and output format should be structured, considering that the fine-tuning for a sequence-tosequence model is naturally cast in a text-to-text format @xcite . we summarize our research questions in two: these research questions and our task approach are intentionally formulated in reference to what previous literature @xcite have proposed as future directions. as we elaborate further in the following sections, our approach simplifies some inherent problems that we had in the readability assessment. but our study has limitations and requires further explorations ( §5). ## conclusion so far, we have reported our exploratory work on casting readability assessment tasks in a text-to-text formulation for bart and t5. we summarized our observations into five categories in §4, which can serve as base guidelines for future work. our experimental setup and data permutation methods allow the joint training of more than one dataset, regardless of whether the dataset construction is parallel or distinct ( §3). using news as extra training data further to fine-tune an osen-trained model greatly improved model performance. however, we did not train the other way around (@xmath0, which should be proved in the future.