ITEC at MLSP 2024: Transferring Predictions of Lexical Difficulty from Non-Native Readers (2024)

Abstract: This paper presents the results of our team’s participation in the BEA 2024 shared task on the multilingual lexical simplification pipeline (MLSP; Shardlow et al., 2024). During the task, organizers supplied data that combined two components of the simplification pipeline: lexical complexity prediction and lexical substitution. This dataset encompassed ten languages, including French. Given the absence of dedicated training data, teams were challenged with employing systems trained on pre-existing resources and evaluating their performance on unexplored test data.Our team contributed to the task using previously developed models for predicting lexical difficulty in French (Tack, 2021). These models were built on deep learning architectures, adding to our participation in the CWI 2018 shared task (De Hertog and Tack, 2018). The training dataset comprised 262,054 binary decision annotations, capturing perceived lexical difficulty, collected from a sample of 56 non-native French readers. Two pre-trained neural logistic models were used: (1) a model for predicting difficulty for words within their sentence context, and (2) a model for predicting difficulty for isolated words.The findings revealed that despite being trained for a distinct prediction task (as indicated by a negative R2 fit), transferring the logistic predictions of lexical difficulty to continuous scores of lexical complexity exhibited a positive correlation. Specifically, the results indicated that isolated predictions exhibited a higher correlation (r = .36) compared to contextualized predictions (r = .33). Moreover, isolated predictions demonstrated a remarkably higher Spearman rank correlation (ρ = .50) than contextualized predictions (ρ = .35). These results align with earlier observations by Tack (2021), suggesting that the ground truth primarily captures more lexical access difficulties than word-to-context integration problems.

## introduction the aim of predicting and simplifying lexical difficulty is to enhance text readability by focusing on vocabulary. drawing from a simplified perspective on reading @xcite , we can divide these difficulties into two main categories: decoding and comprehension. decoding issues relate to difficulties in accessing words (also known as "lexical access"), where readers struggle to recognize and recall the form and meaning of words from memory. conversely, comprehension difficulties involve struggles in integrating words into the broader textual context (sometimes termed "word-to-context integration"). therefore, simplifying lexical difficulty entails employing various strategies to boost clarity and comprehension. this may involve substituting complex terms with simpler alternatives or providing contextual clues or definitions. ultimately, the goal is to enhance accessibility while maintaining the integrity of the conveyed message. over the last decade, several tasks have been organized to advance the development of automated models, including the complex word identification shared task @xcite , the second complex word identification shared task @xcite , the shared task on lexical complexity prediction @xcite , and the shared task on multilingual lexical simplification @xcite . lastly, @xcite organized the shared task on multilingual lexical simplification pipeline (mlsp). 1 this system description paper outlines our team's involvement in the mlsp shared task, focusing on our approach. specifically, we leveraged predictions of lexical difficulty for french from previous research @xcite in the initial phase of the lexical simplification pipeline, known as lexical complexity prediction. our approach also entailed comparing predictions for individual words (approximating lexical access difficulties) with predictions for words within context (approximating word-to-context integration difficulties). subsequent sections detail our methodology and findings. ## method the shared task progressed through two distinct phases. in the development phase, which took place from february 15 to march 14, 2024, teams were tasked with developing systems using existing resources. due to the absence of dedicated training data and the small size of only 30 trial items per language, our emphasis was on employing pre-trained models for making zero-shot predictions of lexical difficulty (see section 2.1). during the evaluation phase, from march 15 to march 26, 2024, teams were provided with test data for ten languages within the multils framework @xcite . during this phase, we used our pre-trained models to predict scores of lexical complexity for the french test set (see section 2.2) and made two submissions. in the subsequent sections, we will provide a more detailed description of the pre-trained models and test data. ## results figure 4 illustrates the model predictions for the french test dataset. as shown, both models generally predicted a high difficulty level (> 0.5) for most test items, with the isolated model (run 2) indicating a higher difficulty level compared to the contextualized model ( run 1 1 2 model 0.0 0.2 0.4 0.6 0.8 1.0 predictions figure 4: predictions of lexical complexity for the french test data however, the findings presented in these results align with earlier observations by @xcite , indicating that the ground truth predominantly reflects greater challenges in lexical access (i.e., difficulty recognizing the form and meaning of the word, regardless of its context) rather than issues in word-to-context integration (i.e., difficulty in interpreting the word within its context). specifically, @xcite noted that words identified as challenging by non-native readers exhibited more lexical access difficulties, as indicated by the higher predictive power of features associated with isolated word surprisal compared to contextualized word surprisal. this finding is unsurprising, given that the annotators had elementary to intermediate proficiency levels and, therefore, had a significantly smaller vocabulary size compared to native speakers. consequently, it is reasonable to assume that the non-native annotators of the french test dataset also had a lower vocabulary size and were thus more susceptible to encountering words not yet ingrained in their mental lexicon, resulting in greater challenges in recognizing the form and meaning of words. furthermore, the results depicted in ## conclusion this study delved into predicting lexical complexity in french test data employing two models: an isolated model and a contextualized model. the findings underscore that while the transfer of difficulty scores remains uncertain, the ranking of lexical difficulty from this model can still be applied to new data. this emphasizes the potential usefulness of the models in comprehending lexical complexity in french texts, while also spotlighting the limitations in transferring the raw predicted scores. moving forward, we also intend to explore the implications of transferring zero-shot predictions made with pre-trained french models to other languages.