Cross-Lingual Leveled Reading Based on Language-Invariant Features (2021)

Abstract: Leveled reading (LR) aims to automatically classify texts by the cognitive levels of readers, which is fundamental in providing appropriate reading materials regarding different reading capabilities. However, most state-of-the-art LR methods rely on the availability of copious annotated resources, which prevents their adaptation to low-resource languages like Chinese. In our work, to tackle LR in Chinese, we explore how different language transfer methods perform on English-Chinese LR. Specifically, we focus on adversarial training and cross-lingual pre-training method to transfer the LR knowledge learned from annotated data in the resource-rich English language to Chinese. For evaluation, we first introduce the age-based standard to align datasets with different leveling standards. Then we conduct experiments in both zero-shot and few-shot settings. Comparing these two methods, quantitative and qualitative evaluations show that the cross-lingual pre-training method effectively captures the language-invariant features between English and Chinese. We conduct analysis to propose further improvement in cross-lingual LR.

## introduction imagine searching the appropriate reading materials for a 10-year-old child in the bookstore: the tale of peter rabbit is a bit outdated; animal farm, though sounds suitable, is too allegorical; the harry potter series may be just right for the age. leveled reading (lr) provides such selection guides by automatically classifying texts with regard to the reading level appropriate for readers, which has proven to be of importance in multiple fields, including education @xcite , health @xcite and advertisement @xcite . different from the traditional readability assessment @xcite which is formulated as a binary classification problem, lr * equal contribution † corresponding author can be regarded as a multi-class classification task that provides specific reading levels with regard to the cognitive reading level instead of text quality. this fine-grained leveling forms a fundamental component in downstream applications, since it is essential to label different levels even within the harry potter series when the stories get darker. most previous research focus on english by extracting language-specific features, ranging from traditional readability formulas to using machine learning methods. as the most widely studied language in lr, english holds mature lr standards with abundant reading materials, such as lexile @xcite and accelerated reader @xcite , and has recently developed a set of lr datasets for training automatic methods, such as the weebit @xcite , newsela @xcite and on-estopenglish @xcite ) corpus. by contrast, low-resource languages like chinese lack both established lr standards and training data, which results in only a few lr research conducted in chinese @xcite . can we use the existing resources of english to guide the crosslingual lr of low-resource languages such as chinese. there has been a recent trend towards learning language-invariant features to ease the crosslingual generalization from high-resource languages to low-resource languages @xcite . we hypothesize that these language-invariant features also exist in lr, especially in the equivalent level of reading among different languages, which may be automatically extracted through deep learning methods. for example, the reading materials in different languages in the equivalent level may talk about the similar story and express same thoughtsand they may have similar changes in text structure and vocabulary as the level changes. thus, to verify our hypothesis and transfer en-glish lr knowledge into chinese, we explore both adversarial training and cross-lingual pre-training method to extract language-invariant features for english and chinese lr corpora to guide lr in chinese. overall, our contributions are summarized as follows: ## lr datasets in this section, we elaborate on the lr datasets collected which are classified by the standards of gradeletterand number. and we re-align these datasets using an age-based standard because these standards have been designated approximate age range. english datasets: we compile the english dataset based on both the weebit (vajjala and meurers, 2012) corpus and reading a-z (raz) foot_0 reading materials. the weebit corpus contains 3,125 texts of five classes, based on weeklyreader and bbc-bitesize for readers aged from 6 to 17. i'm following (deutsch et al., 2020), we apply additional preprocessing to remove extraneous materials in each text, such as copyright declaration and links. the raz reading materials originally contain books at 29 levels from level aa to z2. and each level has a corresponding suitable age. level aa to c are suitable for children aged from 4 to 6 which are not in our consideration. among the materials available to the public, we select and compile 360 texts from level d to z for readers aged from 6 to 17 ## methodology adversarial training and pre-training are recently popular deep learning methods, which can better learn text representation, and have been applied to cross-lingual tasks to extract common features. inspired by this, we also try to apply these two methods to our cross-lingual lr task. ## conclusion in our work, we explore two methods to tackle chinese lr using deep neural networks without any extra features, the adversarial training model adan and cross-lingual pre-trained language model xlm. we organize and re-classify the lr datasets, including three lr corpora for english, and a variety of textbooks across 12 age levels and extracurricular books recommended in chinese. to the best of our knowledge, this is the first attempt to integrate different corpora and leverage neural language models for cross-lingual lr. experimental results show that cross-lingual language model is more effective, and we can leverage only english corpus to predict the reading level of chinese text, which solves the insufficient data problem in the low-resource chinese language. after the summary of our experiment, there are still some flaws in both our datasets and methods, we have suggested some directions for future development.