CEFR -Based Sentence Difficulty Annotation and Assessment (2022)

Abstract: Controllable text simplification is a crucial assistive technique for language learning and teaching. One of the primary factors hindering its advancement is the lack of a corpus annotated with sentence difficulty levels based on language ability descriptions. To address this problem, we created the CEFR-based Sentence Profile (CEFR-SP) corpus, containing 17k English sentences annotated with the levels based on the Common European Framework of Reference for Languages assigned by English-education professionals. In addition, we propose a sentence-level assessment model to handle unbalanced level distribution because the most basic and highly proficient sentences are naturally scarce. In the experiments in this study, our method achieved a macro-F1 score of 84.5% in the level assessment, thus outperforming strong baselines employed in readability assessment.

## introduction controllable text simplification, first proposed by @xcite , is the automatic rewriting of sentences to make them comprehensible to a target audience with a specific proficiency level. among its primary applications are providing reading assistance to language learners and helping teachers adjust the difficulty level of their teaching materials @xcite @xcite . the finegrained control of output levels to match the linguistic ability of the readership is crucial for these educational applications. while readability assessments have been actively studied (e.g., in @xcite @xcite ), linking readability to language ability is difficult. readability scores, such as the flesch-kincaid grade level @xcite , are intended for native speakers, not for language learners to whom very different considerations apply. @xcite and @xcite revealed that readability metrics designed for l1 do not apply to l2 learners. furthermore, readability definitions use documents rather than sentences, which are required by text simplification at the sentence-level, as their unit. the lack of a corpus annotated by sentence difficulty level hinders the advancement of controllable text simplification. previous studies @xcite @xcite necessarily used corpora annotated for readability rather than difficulty; furthermore, they assumed that all sentences in a document had the same readability (i.e., the document level in newsela @xcite ). to solve these problems, we created a large-scale english corpus annotated by sentence difficulty levels based on the common european framework of reference for languages (cefr), foot_0 the most widely used international standard describing learners' language ability. our cefr-based sentence profile (cefr-sp) corpus adapts cefr to sentence levels. a sentence is categorised as a certain level if a person with the corresponding cefr-level can readily understand it. cefr-sp provides cefr levels for 17k sentences annotated by professionals with rich experience teaching english in higher education. a major challenge in sentence-level assessment is the unbalanced distribution of levels: sentences at the basic (a1) and highly proficient (c2) levels are naturally scarce. to handle this, we propose a sentence-level assessment model with a macro-f1 score of 84.5%. we designed a metric-based classification method with a simple inductive bias that avoids overfitting to majority classes @xcite . our method generates embeddings representing each cefr-level and estimates a sentence's level based on its cosine similarity to these embeddings. empirical results confirm that our method effectively copes with unbalanced label distribution and outperforms the strong baselines employed in readability assessments. this study makes two main contributions. first, we present the largest corpus to date of sentences annotated according to established language ability indicators. second, we propose a sentencelevel assessment model to handle unbalanced label distribution. cefr-sp and sentence-level assessment codes are available foot_1 for future research at https://github.com/yukiar/cefr-sp . ## related work related studies have assessed text levels on different granularity (document and sentence) and level definitions (readability/complexity and cefr). ## cefr-sp corpus this section describes the design of the annotation procedure and discusses sentence-level profiles. cefr describes language ability on a 6-point scale: a1 indicates the proficiency of beginners; a2, b1, b2, c1, and c2 indicates mastery of a language at the basic (a), independent (b), and proficient (c) levels. because cefr is skill-based, each level is defined by 'can-do' descriptors indicating what learners can do, 4 cefr levels for sentences cannot be defined directly. therefore, we used a bottom-up approach, assigning cefr levels to sentences based on the 'cando' descriptors of reading skills under the definition that a sentence is, for example, at a1 level if it can be readily understood by a1-level learners. we hypothesise that with sufficient teaching experience and cefr knowledge, it is possible to objectively determine at which level a learner can understand each sentence. we therefore carefully selected annotators with sufficient expertise through pilot and trial sessions. ## sentence-level assessment we propose a sentence-level assessment model robust to imbalances in label distribution. ## evaluation in this section, the proposed level assessment model is evaluated using the cefr-sp corpus. ## summary and future work in this study, we introduced cefr-sp, the first english sentence corpus annotated with cefr levels. the carefully designed annotation procedure involved recruiting experts with strong backgrounds in english education to ensure the reliability of the assigned labels. cefr-sp allows the development of an automatic sentence-level assessment model. the proposed method can handle unbalanced level distributions using a metric-based classification. our future work will involve collecting parallel sentences of cefr-sp to make it directly applicable for training text simplification models. we will also develop controllable text simplification models based on reinforcement learning: the proposed level assessment model will be employed to reward the generation of lower-level sentences.