Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features (2021)

Abstract: We report two essential improvements in readability assessment: 1. three novel features in advanced semantics and 2. the timely evidence that traditional ML models (e.g. Random Forest, using handcrafted features) can combine with transformers (e.g. RoBERTa) to augment model performance. First, we explore suitable transformers and traditional ML models. Then, we extract 255 handcrafted linguistic features using self-developed extraction software. Finally, we assemble those to create several hybrid models, achieving state-of-the-art (SOTA) accuracy on popular datasets in readability assessment. The use of handcrafted features help model performance on smaller datasets. Notably, our RoBERTA-RF-T1 hybrid achieves the near-perfect classification accuracy of 99%, a 20.3% increase from the previous SOTA.

## introduction the long quest for advancing readability assessment (ra) mostly centered on handcrafting the linguistic features that affect readability @xcite . ra is a time-honored branch of natural language processing (nlp) that quantifies the difficulty with which a reader understands a text @xcite . being one of the oldest systematic approaches to linguistics @xcite , ra developed various linguistic features. these range from simple measures like the average count of syllables to those as sophisticated as semantic complexity @xcite . perhaps due to the abundance of dependable linguistic features, an overwhelming majority of ra systems are support vector machines (svm) with handcrafted features @xcite . such traditional machine learning (ml) methods were linguistically explainable, expandable, and most importantly, competent against the modern neural models. as a fragmentary example, @xcite reports that a large ensemble of 6 bilstms with bert @xcite , elmo @xcite , word2vec @xcite , and glove @xcite embeddings showed only ∼1% accuracy improvement from a single svm model developed by @xcite . even though deep neural networks have achieved state-of-the-art (sota) performance in almost all semantic tasks where sufficient data were available @xcite , neural models started showing promising results in ra only quite recently @xcite . a known challenge for the researchers in ra is the lack of large public datasets -with the unique exception of weebit @xcite . technically speaking, even weebit is not entirely public since it has to be directly obtained from the authors. @xcite raised the sota classification accuracy on the popular weebit dataset @xcite by about 4% using bert. this was the first solid proof that neural models with auto-generated features can show significant improvement compared to traditional ml with handcrafted features. however, neural models, or transformers (which is the interest of this paper), still show not much better performance than traditional ml on smaller datasets like onestopenglish @xcite , despite the complexity. from our observations, the reported low performances of transformers on small ra datasets can be accounted for two reasons. 1. only bert was applied to ra, and there could be other transformers that perform better, even on small datasets. 2. if a transformer shows weak performance on small datasets, there must be some additional measures done to supply the final model (e.g. ensemble) with more linguistic information, but such a study is rare in ra. hence, we tackle the abovementioned issues in this paper. in particular, we 1. perform a wide search on transformers, traditional ml models, and handcrafted features & 2. develop a hybrid architecture for sota and robustness on small datasets. however, before we move on to hybrid models, we begin by supplementing an underexplored linguistic branch of handcrafted features. according to survey research on ra @xcite , the study on advanced semantics is scarce. we lack a model to capture how deeper semantic structures affect readability. we attempt to solve this issue by viewing a text as a collection of latent topics and calculating the probability distribution. then, we move on to combine traditional ml (w handcrafted features foot_0 ) and transformers. such a hybrid system is only reported by @xcite , concluding, "(hybrid models) did not achieve sota performance." but we obtain contrary results. through a large study on the optimal combination, we obtain sota results on weebit and onestopenglish. also, our bert-gb-t1 hybrid beats the (previous) sota accuracy with only 30% of the full dataset, in section 4.7. our main objectives are creating advanced semantic features and hybrid models. but our contributions to academia are not limited to the abovementioned two. we make the following additions: 1. we numerically represent certain linguistic properties pertaining to advanced semantics. ## cross domain evaluation it would be very favorable to run an extra crossdomain evaluation (which we believe to be a nextlevel topic). but realistically, performing a crossdomain evaluation requires a thorough study on at least two datasets, which is potentially out of scope in this research. the readability classes/levels are labeled by a few human experts, making the standards vary among datasets. to make two datasets suitable for cross-domain evaluation, much effort is needed to connect the two, such as the class map-ping used in @xcite . however, it should be noted for future researchers that the notion of domain overfitting is indeed a common problem faced in ra, which often uses one dataset for train/test/validation. without a new methodology to connect several datasets or a new large public dataset for ra, it will forever be challenging to develop a ra model for general use @xcite . ## conclusion we have reported the four contributions mentioned in section 1. we checked that the new advanced semantic features add orthogonal information to the model. further, we created hybrid models ( in semantic richness. list 4 and list 5 have the same lengths. however, list 5 contains more meaningful topics (↑ p) throughout the list, resulting in higher overall semantic richness. as such, semantic richness rewards long probability lists (↑ n) with more meaningful (↑ p) topics. similarly, list 3 (↓ n,↑ p) has higher richness than list 2 (↑ n,↓ p). semantic clarity. list 3 and list 4 have the same max(p) and two other same elements (1). however, the second element in list 3 is the same as the first element, resulting in increased difficulty in identifying the main topic (max(p)). likewise, semantic clarity rewards the deviation between the max(p) and the other elements & short probability lists (↓ n). hence, list 1 has the highest clarity. semantic noise. list 2 and list 4 have the same lengths of 6 elements. however, list 2 contains more extraneous topics (↓ p), resulting in higher semantic noise. as such, semantic noise rewards longer lists (↑ n) with more extraneous elements (↓ p). as a result, list 5 has the least semantic noise.