BERT Embeddings for Automatic Readability Assessment (2021)

Abstract: Automatic readability assessment (ARA) is the task of evaluating the level of ease or difficulty of text documents for a target audience. For researchers, one of the many open problems in the field is to make such models trained for the task show efficacy even for low-resource languages. In this study, we propose an alternative way of utilizing the information-rich embeddings of BERT models with handcrafted linguistic features through a combined method for readability assessment. Results show that the proposed method outperforms classical approaches in readability assessment using English and Filipino datasets, obtaining as high as 12.4% increase in F1 performance. We also show that the general information encoded in BERT embeddings can be used as a substitute feature set for low-resource languages like Filipino with limited semantic and syntactic NLP tools to explicitly extract feature values for the task.

## introduction automatic readability assessment is the task of evaluating the level of ease or difficulty of text documents such as web articles, story and picture books, test materials, and medical prescriptions. often readability levels can be expressed in many forms: discrete values with grade and age levels such as in the common european framework of reference for languages (cefr) foot_0 , or with continuous values from a given range such as in the famous lexile reading framework foot_1 . in machine learning setting, this task is most often viewed as a classification task where an annotated set of corpora is trained with its corresponding gold-standard labels evaluated by an expert as mostly done in previous works @xcite @xcite @xcite @xcite . recent works have tried testing unexplored resources by utilizing large pre-trained language models such as bidirectional encoder representations or bert @xcite which is based on the attention-driven transformer architecture by @xcite directly processing the data to the network @xcite or by (b) using the discrete output of the network via transfer learning @xcite as an additional feature. for these methods, however, evidence of efficacy are only seen in highresources readability datasets in english. thus, we propose an alternative way of incorporating the knowledge of large language models such as bert by combining its information-rich sentence embeddings as a separate feature set for traditional machine learning algorithms with handcrafted linguistic features. we argue that this method is not only low-resource friendly but also preserves the semantic and syntactic information encoded by the attention heads of bert since the embeddings itself will be used. we show that such information can act as a substitute for languages with limited tools for explicitly extracting semantic and syntactic features where results describe non-significance in difference of performances between models using semantic and syntactic features versus models using bert embeddings. ## previous work the first generation of readability formulas and indices date as early as 1920-1940s with the works of @xcite , @xcite , and @xcite primarily using surface-based variables such as raw frequencies and average values of sentences and words per document. the process for using such indices requires manual computa-tion and plugging of values to formulas which can be tedious as the length of a document increases. likewise, experts argue that considering narrow, surface-based features do not entirely capture the linguistic complexity of a given text @xcite @xcite . thus, incorporation of deeper, linguistic variables such as a language's semantics, syntax, morphology, and discourse properties are imperative and worth exploring for the task. to answer this call, the use of handcrafted linguistic features remained the most popular type of input for training readability assessment models through the years. handcrafted linguistic features are often represented as real-valued numbers serving as potential predictors of the difficulty of reading materials. these features span on a wide range of linguistically motivated factors that base on syntax, semantics, morphology, cohesion, and cognition to name a few. these features also serve as the input in the form of vectors for conventional readability assessment setups using traditional classification-based algorithms. to note, not all linguistic features can be applied or extracted for all languages as some have limited nlp tools suitable for use especially for low-resource languages. notable works in various languages such as greek @xcite , german @xcite @xcite , bangla @xcite , and filipino (imperial and ong, 2021a, 2020) have used this approach in combination with traditional machine learning algorithms such as logistic regression and support vector machines. likewise, another reason why studies have resorted to the classical approach of model building is that deep neural models are not practical for the task without a large amount of training data. the advent of large and complex pre-trained language models such as bert and its variations spawned a handful of studies on how these models fare with the readability assessment tasks. the work of @xcite on the supervised experiment setup explored directly using english benchmark corpus such as weebit and on-estopenglish as input for bert via transfer learning while @xcite explored using the final discrete output of bert as a feature for the same datasets. results from both studies show effectiveness of bert for english data as direct input while no significant improvement is seen when the discrete output itself is used as a feature. while these results are remarkable, bert's effectiveness remain a gray area for low-resource languages. ## task definition we define our task at hand as a supervised learning setup. given a text document d where a feature vector @xmath0, x 2 . . . , x n ] is extracted, a model m is trained using said collection of features x along with the gold label y or expert-identified readability level. the label is relative in form (discrete or continuous) based on how readability levels are categorized for each corpus. ## corpus we describe each corpus used in the study below as well as the statistics and breakdown in common core exemplars. the cce dataset contains 168 prose texts from the appendix b of the common core state standards initiative (ccss) 4 for english language studies and first used by flor et al. (2013) for readability assessment. the initiative was a project of the national governors association and the council of chief state school officers in usa 5 . the dataset is divided into three age-range categories: 2-5, 6-7, and 9-12. adarna house. the adarna house corpus is a collection of 265 story books for grades 1-3 from adarna house inc. 6 , the largest children's litera-613 ture publisher in the philippines. this corpus has been used by imperial et al. (2019); imperial and ong (2020, 2021a) for readability assessment in filipino foot_6 bert's efficacy on a wide range of nlp tasks stems from its implicit capability to encode linguistic knowledge such as hierarchical parse trees (hewitt and manning, 2019), parts of speech and syntactic chunks @xcite , semantic roles @xcite as well as entity types and relations @xcite to name a few. in view with this, we find such amount of knowledge an extremely valuable resource which can potentially improve performances of readability assessment models especially for low-resource languages if used correctly. thus, to maximize the potential of bert for low-resource readability assessment, we propose a combined training of its raw embeddings with handcrafted linguistic feature sets through a concatenation process and feeding them to traditional machine learning algorithms. the embeddings of bert generated by the multihead attention layers are information-rich, specifically on semantic and syntactic knowledge @xcite , due to the nature of its training. we describe our proposed architecture in figure 1 with a sample filipino sentence for context. ## experiment setup for the ose and cce corpus in english, we extracted over 155 linguistic features covering lexical diversity and density features, syntactic features based on parse trees, morphosyntactic properties of lemmas, and word-level psycholinguistic features. for the adarna house corpus in filipino, we extracted over 54 linguistic features covering traditional surface-based features, lexical features based on pos tags, language model features, morphology based on verb inflection, and orthographic features based on syllable pattern. for the complete and detailed list of features for english and filipino, please refer to the resources at @xcite and imperial and ong (2020) respectively. the size of the bert embeddings for all datasets remain equal with a fixed dimension of @xmath1. the embeddings and extracted linguistic feature sets were concatenated, for a total of 923 dimensions for combined features for both english datasets and 823 for the filipino dataset. recipes for feature extraction were obtained from the studies of @xcite for english and imperial and ong (2020, 2021a,b) for filipino. we used the sentence-transformers library by @xcite with mean pooling option to extract bert embedding representations for the readability corpora foot_7 . for the traditional machine learning algorithms, we used three of the commonly utilized in previous works: logistic regression, support vector machines, and random forest. models for each dataset were trained on a 5-fold cross validation procedure. we used weighted f1 as the overall metric for performance evaluation. ## conclusion in this study, we proposed an alternative way of combining information-rich bert embeddings with handcrafted linguistic features for the readability assessment task. results from our experiments showed that the method outperforms classical, vanilla approaches in readability assessment using english (ose and cce) and filipino (adarna) datasets in various machine learning algorithms such as logistic regression, support vector machines, and random forest. we also demonstrated that the knowledge implicitly encoded in bert embeddings (semantic and syntactic information) can be used as a full substitute feature set for lowresource languages like filipino with limited nlp tools to explicitly extract feature values for the task. we are looking forward to the application of our proposed method to other languages struggling with the extraction deep linguistic features to trained readability assessment models. future directions of the study include deeper exploration of bert such as isolating extracted embeddings for each of the twelve attention layers.