A comparative study of word embeddings and other features for lexical complexity detection in F rench (2018)

Abstract: Lexical complexity detection is an important step for automatic text simplification which serves to make informed lexical substitutions. In this study, we experiment with word embeddings for measuring the complexity of French words and combine them with other features that have been shown to be well-suited for complexity prediction. Our results on a synonym ranking task show that embeddings perform better than other features in isolation, but do not outperform frequency-based systems in this language.

## introduction complex word identification (cwi) is an important step in text simplification systems which aim at modifying the complexity level of texts to make them more accessible to readers with reading difficulties. there are many factors that can make a text complex or difficult, at the syntactic level (e.g. texts using passive voice), but also at the pragmatic and lexical levels (e.g. texts with a high number of domain-specific or rare words). lexical complexity is the property of words that may pose comprehension difficulties to some readers, and which would not be part of the vocabulary of young children or low-proficiency foreign language learners. complex words have a high impact on the readability of a text. @xcite showed that lexical complexity affected comprehension of elementary science texts by children, and manual lexical simplification has been shown to improve understanding and reading speed in dyslexics @xcite . cwi can serve to identify the lexical elements in a text that need to be substituted by simpler synonyms or paraphrases for the meaning of the text to become more accessible. we carry out a comparative study of different types of systems for measuring the complexity of french words. we experiment with features that have been identified as good indicators of complexity in the literature and word embeddings which, although successful in numerous semantics-related tasks, have not yet been applied to lexical complexity prediction. we assume that complex words appear in complex contexts, and simple words occur in texts that are easier to understand. we consider lexical complexity to be a property of words that is reflected in their context of use. word embeddings encode rich contextual information, so we expect them to be particularly useful for complexity detection. we train our systems on two french lexical resources that encode the distribution of words across different levels of difficulty : manulex, where categories correspond to school levels @xcite , and the flelex database where words are categorized into second language proficiency levels @xcite . we evaluate our systems on lists of synonyms that have been manually ordered by complexity. ## related work due to the importance of lexical complexity detection for simplification, there have been numerous approaches to automatically predicting complexity. the task is often regarded as a classification problem into two or more levels of difficulty. @xcite performs binary classification of words into simple or complex by means of a support vector machine (svm) classifier based on features such as frequency, number of syllables and senses. @xcite classify french words into three or six complexity levels, which indicate the distribution of words in the three school levels and six second language proficiency levels found in manulex and flelex, respectively. gala et al.'s (2014) systems achieve accuracies close to 63% and 43% when trained and evaluated on the two resources. interestingly, both @xcite and @xcite systems obtain results that are close to those of baseline models based on frequency only, and show that complex words occur more rarely in texts. another approach proposed by @xcite consists in ordering synonyms by complexity, creating thus a ranking instead of a classification. their system is trained on pairs of french synonyms based on a complexity score derived from manulex, and uses 21 features. we use in this study three of their best performing features : number of characters, number of phonemes and frequency. other works have focused on creating readability lexicons. @xcite develop a model that infers the age of aquisition of a word from its presence in graded texts and use it to assess text readability. @xcite build a readability lexicon based, among others, on word co-occurrence information, which is the closest feature to the word embeddings we use in our experiments. in the semeval 2012 shared task on lexical simplification @xcite , where systems had to choose a simpler synonym for words in context, the frequency baseline was very powerful and was only beaten by one system using psycholinguistic features including concreteness, imageability, familiarity and age of acquisition @xcite . and word length is the best approach for choosing a substitute for a complex word. ## data and resources we train and evaluate our systems on two large lexical resources for french, manulex and flelex. manulex @xcite is a lexical database with word frequency information calculated from 54 french textbooks used in three school levels, with a total of 1.9 million word tokens. the levels correspond to ages of 6 (cp), 7 (ce1) and 8 to 11 (ce2-cm2) years old. we use its non-lemmatized version, which contains 48,886 word forms, and keep only unigrams with open parts of speech (nouns, adjectives, verbs and adverbs), which reduces it to 39,839 words. flelex @xcite is a graded lexicon based on textbooks and simplified readers for learners of french as a foreign language which contains a total of 777,000 tokens. this resource is organized in 6 levels corresponding to the proficiency levels of the cefr scale (conseil de l'europe, 2001), ranging from a1 to c2. the resource contains 14,236 lemmas from which we retain 13,250 after filtering for part of speech and multi-words, as for manulex. we perform feature extraction and remove words for which no word embedding is available, which leaves us with 36,845 and 12,942 words in each dataset. since words might occur in different levels, we assign each word in its first level of appearance, indicating its moment of acquisition, and we observe that the distribution of words is biased towards the third level in manulex and the three first levels in flelex. we believe this is because the third level in manulex groups more ages than the first two, and in the case of flelex the bias probably reflects the need of constructing a substantial lexicon during the first stages of learning. in order to learn from a more balanced dataset, we randomly remove words from the majority classes, obtaining a total of 26,424 words for manulex and 9,194 for flelex. the contents of the original and balanced resources are given in for evaluation, we use sets of synonyms manually ranked for complexity and made available by @xcite . one example is @xmath0, where words are ordered from more complex to simpler. forty annotators were asked to order the synonyms according to their reading and comprehension difficulty without any given context. the resource consists of 36 groups of synonyms and 134 lexical units, with a few words being present in more than one group, ranked according to the majority ranking amongst all annotators. the reported krippendorff's α of 0.399 reflects the difficulty of this task even for humans. because of the unavailability of word embeddings for multiword expressions and for some synonyms present in the dataset, 1 we remove 13 words and 3 groups of synonyms (for which there was none or only one synonym left after removing the words), which results in a dataset of 33 sets of synonyms and 121 words. 5 of these sets are used as a development set, which results in a test set of 28 synsets and 104 words. we extract frequency and number of phonemes of lexical items to be used by our models from lexique3 @xcite , a large resource with various kinds of lexical information (phonetic transcription, syllabic structure, number of morphemes, etc.) for 142,728 french words. ## model description we view complexity as a continuous, rather than categorical, notion with highly complex/simple words situated at the two extremes of the spectrum. in order to induce a continuous representation from the three manulex levels and the six flelex levels, we implement a simple feed-forward neural network that has as learning objective a number corresponding to the first level of appearance of a word in the corresponding resource. this serves as an indication of the age level or moment of acquisition of the word. the network takes as input features that represent the target word and outputs a real number that indicates its complexity. this way, we obtain a continuous representation of complexity from categorical data, which allows for a more fine-grained ranking than the one found in the original resources. we experiment with different feature combinations, number of layers and layer sizes on a development set, and choose the best configuration for each feature combination for evaluation (see section 5). we use three different architectures : one with no hidden layer, similar to logistic regression (no hl) ; one with a hidden layer of size 100 (1 hl) ; and one with two hidden layers of sizes 150 and 100 (2 hl). in all settings, rectified linear unit (relu) is used as an activation function and iterations are limited to 100. for our experiments, we use the following lexical features which have been shown to be useful for predicting complexity in past work (gala et al., 2014) : french subtitles of 4 types of films and series differing in their original language. word frequency has been shown to be a very effective predictor of complexity in past work. -300-dimensional word embeddings 3 previously trained on wikipedia with fasttext using the skip-gram model @xcite . word embeddings encode distributional information of words, and we expect them to be able to capture differences in complexity. our hypothesis is that complex words appear in more complex texts. ## evaluation we evaluate our systems on the manually ordered synonym lists described in section 3 measuring the pair-based and rank-based accuracy of the predictions against human judgments. for the pair-based evaluation, we exhaustively create @xmath1. for example, from the group @xmath2, ordered from complex to simple, we derive 6 pairs : (@xmath3, @xmath4, @xmath5, @xmath6, @xmath7, @xmath8, preserving the directionality indicated by the annotators. we expect a good system to assign higher scores to more complex words, producing the correct ordering. we report the proportion of pairs that were correctly ordered by the system. in the rank-based evaluation, the system has to order a full group of synonyms based on the score that it assigns to each word. we report the percentage of cases in which the system correctly predicts the exact position of a synonym. following @xcite , we additionally report the proportion of cases where a synonym was placed in the original place or only one position away. for calculating the latter, we exclude groups of synonyms with only two words, which would otherwise always count as correct (there is only one such pair in the development set and one in the test set). in the case of ties between the scores of two synonyms, they are considered to be incorrectly placed, since this means the system is not able to detect the difference in complexity between the two words. we test the three proposed architectures (with 1 or 2 hl(s), and without hl) on the development sets extracted from manulex and flelex and from the manually-ranked synonym resource. we retain as the best model for each feature combination the one that obtains the strongest correlation (spearman's ρ) with manulex and flelex original classes and the highest accuracy scores in the pair-based and rank-based evaluations on the synonym development set. in the case of two models performing equally or very similarly, we choose the simplest one (i.e. the one with fewer layers). we compare the results of the best model for different feature combinations to the ones produced by françois et al.'s model on the same test set. the size of the dataset used in our evaluation is slightly different from the one used in their original work, since lexical items that had no embeddings were left out. we recompute the accuracy of their system output on the reduced test set. as a baseline, we use the simple frequency feature with no training involved, which considers more frequent words to be simpler and orders them accordingly. ## results and discussion results are presented in tables 2 (manulex) and 3 (flelex). the patterns observed and the scores obtained by the systems trained on manulex and flelex are, in general, similar. one could expect flelex to result in a system producing better rankings because of its finer granularity with respect to manulex, but the much bigger size of the latter probably counteracts this effect. among all feature combinations, frequency combined with number of characters or with phonemes seem to be the best predictors of complexity for both manulex and flelex. the number of phonemes is the worst performing feature when used on its own, and obtains the lowest score in all measures. this is somewhat surprising, since gala et al. @xcite show that phonemes are slightly more strongly correlated to manulex and flelex classes than characters, which do a bit better in this setting but still are the next poorest performing feature. however, these features alone produce a considerable amount of ties which substantially lower their performance. the combination of word embeddings with a strong feature such as frequency does not seem to improve the results of embeddings alone on flelex. the system that relies only on embeddings obtains better results on this dataset than the system using all features. on manulex, on the contrary, embeddings seem to benefit from being combined with other features. in the frequency-based baseline is very powerful and is only beaten with a small margin in the rank-based accuracy evaluation by more complex models that have frequency among their features. françois et al. ## conclusions in this work, we have explored the use of word embeddings alone and combined with other features for lexical complexity prediction in french. our system learns a complexity score from complexity levels based on age of acquisition (manulex) and second language proficiency (flelex). the evaluation on a synonym ordering task seems to indicate that whereas word embeddings obtain better results than other features in isolation, frequency-based systems -even a simple frequency baseline -are better suited for this task. the best performing systems use frequency or combine it with characters or phonemes, two features that have been proven to be useful in past work. in the future, we plan to experiment and compare with other kinds of word embeddings built from corpora with a higher level of stylistic variation.