Archaeology at MLSP 2024: Machine Translation for Lexical Complexity Prediction and Lexical Simplification (2024)

Abstract: We present the submissions of team Archaeology for the Lexical Simplification and Lexical Complexity Prediction Shared Tasks at BEA2024. Our approach for this shared task consists in creating two pipelines for generating lexical substitutions and estimating the complexity: one using machine translation texts into English and one using the original language.For the LCP subtask, our xgb regressor is trained with engineered features (based primarily on English language resources) and shallow word structure features. For the LS subtask we use a locally-executed quantized LLM to generate candidates and sort them by complexity score computed using the pipeline designed for LCP.These pipelines provide distinct perspectives on the lexical simplification process, offering insights into the efficacy and limitations of employing Machine Translation versus direct processing on the original language data.

## introduction in the realm of natural language processing (nlp), the twin challenges of lexical complexity prediction and language simplification play pivotal roles in advancing text comprehension and promoting accessibility. lexical complexity prediction refers to the difficulty of understanding phrases based on their lexical features, while simplification aims to enhance accessibility by offering simplified, easier-to-understand alternatives. the importance of addressing these challenges is underscored by their wide-ranging implications across various domains @xcite @xcite . our approach is guided by the idea to extend such methods beyond the languages that currently have available data sets or corpora; thus, our first set of submissions to the 2024 mlsp shared task @xcite uses machine translation to translate all datasets and languages into english, which has been the central language of text simplification and complexity research in recent years @xcite . both the lexical simplification (ls) and the lexical complexity prediction (lcp) pipelines are using only data in english in this case 1 . the second approach is trained on the original texts as released by @xcite and uses an lcp pipeline trained with languageindependent hand-crafted features such as word length, syllables, vowels, etc. and a regression method trained on the small trial data from the original language. for generating candidates for lexical simplification, we have opted for an llm that can be run locally using a quantized version of openhermes 2.5 based on mistral @xcite that has been fine-tuned on code. according to the authors foot_1 , the model was trained on a good ratio of code instruction (7-14% of the total dataset) that boosted several noncode benchmarks, including truthfulqa, agieval, and gpt4all suite. the quantized llm is not inherently multilingual, however, in our small-scale tests we have observed some ability to generate simplification candidates for non-english language, the llm we used to generate the alternatives does not guarantee the correct form of the generated alternative and this problem is amplified by using machine translation to get the phrases in original languages, which could incorrectly translate words without context. regarding machine the llm works better at generating candidates directly using english translations as the number of empty candidates is lower; however, the actual candidates generated tend to be multi-word expressions instead of simple lexical substitutions. in summary, combining our approach for predicting lexical complexity and simplification in a unified framework may not be the best solution for text comprehension, but it can provide a source of interesting results for different languages. ## lexical complexity prediction for lexical complexity prediction we reuse an approach that has been previously tested at the lcp2021 shared task @xcite that obtained a pearson correlation of .75 using a regression method trained on hand-crafted features. ## lexical simplification for lexical simplification, we employ the locally run quantized openhermes 2.5 based on mistral (jiang et al., 2023) using llama-cpp foot_3 and langchain @xcite libraries. the context contains the entire sentence and the target word, and the model is prompted to generate a json with potential replacement candidates. we run the model on the english-translated texts and the results are then back-translated into the initial language. the model prompt is as follows: this sentence "transla-tion" is a translation of "original" and the word "translated_word" is a translation of "original_word" provide a list of 10 alternative simpler words (as a json object) that a child would understand easily to replace the word ""translated_word"" in the following sentence. it is mandatory for pattern of the answer to be displayed as a json with words as keys and complexity scores as values with all the 10 alternatives. the second set of submissions is generated with the model running on the original language date. nevertheless, it is imperative to acknowledge that the model's capability is constrained when handling multilingual data, often leading to hallucinations. the prompt used for original language data is: provide a list of 10 alternative simpler words (as a json object) that a child would understand easily to replace the word "original_word" in the context of the following sentence. it is mandatory to use suitable meanings for the context of the sentence and for the pattern of the answer to be displayed as a json with words as keys and complexity scores as values with all the 10 alternatives. provide only words in "language". sentence: "original. here are some possible synonyms: "synonyms" the synonyms are given in the context extracted from conceptnet @xcite with a quick request to the api. ## results our first set of submissions (suffixed with "*_1.tsv") contain lcp only run with english-only models and ls predictions run on translated texts. the translation model tends to increase the number of words, as seen in our second set of submissions (suffixed with "*_2.tsv") are lcp predictions run on the original target words. figure 1 shows the density plots of the predictions on the test set. the translationsto-english complexity scores (a) are in the same range for all langauges (except for sinhala) while the predictions on the original texts (b) show more divergent patterns due to different features available for each language. here we only report the results on the lcp task as these are the only ones that proved to be competitive in the shared task. for a complete set of results we point the reader to the official task page foot_4 . we perform several experiments on the trial data to verify which features of the original language have the strongest correlation with the complexity scores provided. this should give us a rough idea of the features that contribute the most to the final prediction. the correlations computed on the trial data are reported as a proxy for potential feature impact; given the small sample size, they may also show accidentally high values. for english-language predictions (originals and translations included) word frequency achieves between -.64 and -.7 ρ, followed by mrc features, such as the kucera-francis @xcite number of categories feature (-.55 ρ). mrc features are generally well correlated among each other. sparse features such as dale-chall, europarl frequency, hyponyms, synsets, and other mrcbased features contribute significantly because they create a boundary between words that are not in the external resource (feature value 0) and words that are (@xmath0. for french: word frequency (-.4 ρ) and the number of immediate children in syntactic dependency parse (.4 ρ) show the best correlation with the complexity annotations. german trial data shows that word frequency is at -.76 ρ followed by character length .46 ρ, this could be a lucky coincidence from the small size or the distribution in the trial data. similarly, filipino (-.61 ρ), spanish (-.6 ρ), portuguese (-.71 ρ), and italian (-.63 ρ) show relatively good correlations between complexity scores and word frequency. catalan shows weak correlations of all individual characteristics (-.2 ρ on frequency), which also confirms our overall scores @xcite , and so is japanese (-.58 ρ). sinhala is a special case of language where we do not use word frequency nor other resource and the only relevant features is the character length (-.3 ρ). ## conclusions translating documents into english and making lexical simplification predictions using translated texts introduces noise and severely limits the ability of the model to produce coherent substitutions, especially since our approaches with translations did not take into consideration the proper morphological form of the substitution or of the original word. our results (reported in the appendix) show that complexity prediction is significantly affected by the translation as much as lexical simplification. our mt approach was surpassed by models trained only on english data, which appear to have a better ability to generate good lcp predictions on other languages (especially latin-script languages or languages with a historical contact). this approach can yield decent results, achieving similar correlations to models trained directly on the source language or models using llms and transformer features @xcite . last but not least, we conclude that frequency-and string-based approaches might be a powerful alternative for lcp on low-resource languages.