CEFR -based Contextual Lexical Complexity Classifier in E nglish and F rench (2023)

Abstract: This paper describes a CEFR-based classifier of single-word and multi-word lexical complexity in context from a second language learner perspective in English and in French, developed as an analytical tool for the pedagogical team of the language learning application Mauril. We provide an overview of the required corpora and the way we transformed it into rich contextual representations that allow the disambiguation and accurate labelling in context of polysemous occurrences of a given lexical item. We report evaluation results for all models, including two multi-lingual lexical classifiers evaluated on novel French datasets created for this experiment. Finally, we share the perspective of Mauril’s pedagogical team on the limitations of such systems.

## introduction the lexical complexity classification task exists in its simplest form as the binary complex word identification task (cwi) and at its most complex, as a multi-class classification where the class nomenclature and granularity is determined by the labelling of the training data. lexical complexity finds application in text and lexical simplification systems, in automated language proficiency assessment, as well as in the creation of level-appropriate pedagogical content, which also happens to be the use case of mauril foot_0 . mauril is a new, free digital platform leveraging a wide range of stimulating and entertaining content from cbc and radio-canada to help users learn english and french. financed and endorsed by the government of canada, this new tool is designed and deployed by cbc/radio-canada, in collaboration with a committee of pedagogical experts. it's meant to help improve oral comprehension and integrate language knowledge in everyday life. the language learning process in mauril begins with a placement test and is then organized by levels covering beginner, intermediate and advanced proficiency foot_1 . each level contains units and each unit consists of a video clip of varying length (anywhere from 1 to 22 min) punctuated by comprehension questions and accompanied by highlighted vocabulary (words and expressions) with a corresponding difficulty level. the creation of pedagogical content (from the selection of video segments, through questions and vocabulary definition to the assignment of difficulty level) is performed manually by experienced foreign language teachers. this labour intensive process of content creation was the target of a lexical processing pipeline designed to streamline and facilitate the extraction and addition of more levelappropriate vocabulary to all existing units. the system in question had to be able to parse the subtitle file of a video segment, reconstruct and then segment the text into tokens, detect and extract multi-word expressions and then assign a complexity label to all occurrences of words and expressions in context. the central component of this system and the current publication is a cefr-based foot_2 lexical complexity classifier for both french and english. in this paper, we apply a novel approach to lexical complexity prediction (lcp), based on rich contextual representations. we show that our system is capable of: in the context of their language-learning platform offering a digital language proficiency assessment exam, duolingo had developed and released a cefr checker (now discontinued) allowing users to validate the difficulty of words and text in english and spanish. the lexical complexity component of the tool was described in settles et al. (2020) as a cefr-based vocabulary scale model based on cefr vocabulary wordlist (an inventory of 6,823 english words labelled by cefr level, mostly in the b1/b2 range). the authors proposed two regression models fit on lexical item representations composed of surface-based features aimed as a proxy of frequency. the models did not seem to handle multi-word expressions, nor common contractions such as doesn't and you've. their complexity predictions were lemma-based and did not take inflection into account, which was evident and consequential in spanish more than it was in english. finally, the misclassifications reportedly attributed to polysemy (settles et al., 2020, p.6) were in fact cases of homonymy, since the representations did not include pos information. disambiguating polysems is a challenge for all lemma-based complexity corpora @xcite @xcite @xcite which conflate polysemous entries into a single entry and assign it a single level. however, not all senses of a polysemous word are learned at once and the different meanings of polysemous words are not uniformly distributed across texts of varying difficulty. @xcite even found a negative association between frequency and complexity with more frequent words being associated with more complex texts. this may be attributable to the fact that frequent words tend to be more polysemous @xcite and complex texts are likely using more than one of those meanings disguised as occurrences of the same lemma. in fact, learners encounter highly polysemous words most often @xcite , hence the importance of disambiguating and accurately predicting the complexity of word senses. the role of context in lcp is two-fold. firstly, it is crucial in deriving the correct sense of a polysemous word (word sense disambiguation), as words in isolation provide no information as to their intended meaning. secondly, it has an incidence on a word's complexity as a source of complementary information. learners acquire much of their vocabulary knowledge from context rather than from decontextualized forms such as word lists, definitions, etc.) (nagy, 1995) @xcite were some of the first to recognize the importance of context for the task of cwi. as they correctly pointed out, the perceived complexity of the lexeme molars in the phrase elephants have four molars... may be higher than in the phrase ... new molars emerge in the back of the mouth. since the second occurrence is surrounded by familiar words that imply its meaning, while the first cooccurs with the rarer and less semantically similar elephants. in more recent work, @xcite found that one of the most important predictors of complexity in their experiments was topic distribution -a context feature modelling polysemy and defined as a vector indicating all topics under which a word occurred. effects of the inclusion of context on predicting lexical complexity are also discussed in a recent survey of lcp by @xcite . in contrast, lexical complexity work on french has mostly focused on representing and classifying lexical items in isolation @xcite , independently of the context in which they appear. this position is reflected in the lexical complexity corpora available in french @xcite which provides no means of contextualizing or disambiguating word senses. @xcite have presented lexical classification models trained on these corpora where lexical items were represented by 49 orthographic, morphologic and statistical features. their l2 classifier achieved 43% accuracy on the six-way classification task. approaches based on such linguistic features often struggle to represent mwes since the latter are absent from vocabulary lists despite their high frequency in everyday interactions 4 and invite the use of simplifying techniques such as averaging the constituents of the mwe (which in turn wrongly assume compositionality). at the same time, studies in both french and english have shown the importance of mwe-based features for the accurate assessment of lexical complexity @xcite . ## training data to train a contextual classifier of lexical items, we needed a collection of words and expressions associated with complexity levels and accompanied by at least one sentence illustrating their usage in context. ## lexical classification in this section, we describe the creation of a classifier able to assign a complexity level between 1 ≡ a1 and 6 ≡ c2 to the meaning of any word or multi-word expression as determined by its context. ## results and discussion to establish the effectiveness of feature-based representation as lexical complexity predictors on the evp dataset, we trained the model me6 baseline, a support vector classifier (with @xmath0: the length of the word form in characters and in tokens. without contextual information, we could only disambiguate some of the homographs by part-of speech and had to reduce multiple occurrences of a single token+pos pair to the one with the lowest complexity level. this resulted in 11,133 data points of which we used 90% for training and 10% for evaluation. the resulting confusion matrix with normalized scores per class on figure 2 shows poor recall for all inner classes, especially the a2 level. we then trained a model called me6 contextual which fits the same support vector classifier on the 6-class training set of the evp corpus (cf. §3.1) with word embeddings extracted from a monolingual language model (as described in §5.3). for this experiment, we could disambiguate and use all training points, including polysems, resulting in a larger training set @xcite . the evaluation on 10% of the corpus (1,418 data points) produced the results on figure 3 . the improved performance of the contextual model is consistent across all six classes and visible on both the confusion matrix as precision and recall and we further trained a 4-class classifier (me4 contextual) on a rebinned and rebalanced 15 version of the dataset. the reduced number of classes provides a further improvement of f1 scores ( to train the multilingual model mme4 contextual, capable of classifying not only english but also french words and expressions, we used the method described in section 5.5. when evaluated on english data, the model performs almost as well as the one trained on contextual vectors from a monolingual bert ( the last model we evaluated, mmefr4 contextual, was trained on a combination of english and french labelled data from evp and fec1. despite the shortcomings of the evaluation 15 to balance the classes, we reduced the size of the largest class c to the size of the second largest -b2. corpora we produced in french, it could be used for training, especially the fec1 which has 904 examples labelled from a1 to b2 foot_12 . after rebalancing the resulting joined dataset, we trained the same support vector classifier on 99% of the data, leaving 1% for evaluation. the results listed in we will be releasing foot_13 code and english data used for training as well as trained models with the exception of any model trained on a combination of english and french data. analysis of the errors on the french evaluation corpora showed the significance of the context's complexity for the individual lexical item's complexity prediction. naturally, contextual representations of lexical items are influenced by the surrounding words and syntactic structures, but the extend to which this affects the lexical classifier becomes more visible in a competence type of corpora such as fec1-3. furthermore, analysis of the errors on french corpora show that when the model has only seen english data, it has a tendency to overestimate the complexity of inflected french verbs since the training data does not reflect the complex morphology of the french language. still, the rich contextual representations encode enough information to allow the model to correctly distinguish and classify, for example, instances of the verb faire used as a main verb vs. as an auxiliary. the early adoption and tests of the vocabulary processing pipeline by mauril's team of foreign language teachers highlighted some of the models' limitations. for example, none of the models (not even the multilingual one) give a special treatment to cognates (sets of words in one of the two languages that have been inherited in direct descent from the other one) which are normally considered easier to acquire. another concern has been the lack of transparency in the classifier's predictions, a direct consequence of the dense representations we favoured over the more interpretable linguistic features. finally, a contextual classifier may predict different levels for occurrences of the same lexeme in different contexts. those limitations underline the need for human validation of the output of such systems. ## conclusion in this article, we detailed the creation and evaluation of a lexical complexity classifier in french and english, predicting contextually-aware cefrbased labels for words and multi-word expressions alike. we established a baseline for the six-way lexical classification on the evp corpus and showed that replacing the representation by statistical features such as frequency for a dense contextual embedding from a masked language model such as bert achieves a significantly improved accuracy in english and a moderate one in french. the most significant obstacle laying before the creation of an equally performant model in french is the lack of appropriate training data. the ideal corpus would not only contain contextually grounded lexemes, but would reflect productive rather than receptive knowledge of vocabulary. the utility of a graded lexical classifier goes beyond mauril's use case of vocabulary analysis. such a model may be used in modular text simplification systems to help adjust the level of simplification and adapt it to the user's competence level. in pipelines for lexical simplification, a cefr-based classifier might help with the ranking of substitution candidates by providing an estimation of their complexity (in context) @xcite . it is also a fine-grained tool for complex word identification and readability analysis.