RETUYT - INCO at MLSP 2024: Experiments on Language Simplification using Embeddings, Classifiers and Large Language Models (2024)

Abstract: In this paper we present the participation of the RETUYT-INCO team at the BEA-MLSP 2024 shared task. We followed different approaches, from Multilayer Perceptron models with word embeddings to Large Language Models fine-tuned on different datasets: already existing, crowd-annotated, and synthetic.Our best models are based on fine-tuning Mistral-7B, either with a manually annotated dataset or with synthetic data.

## introduction history has shown that technology can mean a step forward for inclusion and social development. for instance, nlp can change how different social groups interact with texts, by automatically adapting texts to the reader's needs and hence improving digital accessibility. one of the many nlp tasks devoted to this objective is lexical simplification, where systems are built to replace complex words by simpler ones. this has an immediate impact on language learners and children, but also on people with different types of learning or reading difficulties @xcite . the bea-mlsp 2024 shared task @xcite proposes an excellent opportunity to explore two problems related to this path: to score how complex a word is in a given context (task 1), and to find simpler substitutes for that word (task 2). the dataset used both as trial and test sets covers 10 different languages: catalan, english, filipino, french, german, italian, japanese, portuguese, sinhala and spanish @xcite . this dataset was annotated using the multils framework @xcite . in this paper we present the participation of the uruguayan retuyt-inco team at this shared task, describing the approaches followed and the datasets used. the main challenge to solve these tasks is the scarcity of data: only 30 examples for each language were given as trial data, and no training data. we decided to use the trial data as a development set to compare our experiments against each other, and rely on other sources of data (already existing datasets, crowd-sourced, or synthetic). ## related work lexical complexity prediction and lexical simplification tasks have been addressed in different challenges in the past. we discuss the most recent ones for each task. in the semeval-2021 task 1: lexical complexity prediction @xcite , participants developed systems that, given a word within a sentence, assign it a complexity value on a continuous scale. an extended version of the complex corpus @xcite ) was used, with 10,800 instances of words and multi-word expressions scored according to their complexity. deep learning based systems performed the best, followed closely by feature-based approaches. the tsar-2022 shared task on lexical simplification @xcite hosted a shared task on multilingual lexical simplification for english, portuguese, and spanish. the participants had to propose simpler substitutes for a complex word in a given context. some trial examples were provided in each language (10 for english, 10 for portuguese, and 12 for spanish). the best results were obtained by approaches based on masked language models. ## approaches in this section we detail the five different approaches followed. we experimented with static word embeddings, contextual embeddings, finetuning mistral 7b on synthetic data, crowd-sourced data and existing data, and also with the groq platform. we describe each of them next. ## results in the appendix b we include tables 3 and 4, which show the results of our methods over the trial data. we used those preliminary results to choose which submissions to send to the competition, trying to keep the most promising systems but also a mix of different approaches. the experiments selected for submission are underlined in the tables. tables 1 and 2 show the results of the submitted systems over the test set. ## conclusions we presented a series of experiments for solving the complex word prediction and lexical simplification tasks, ranging from simpler non-contextual static embeddings baselines, to more advanced finetuning of llms. the most important challenge in these tasks was the data scarcity, and because of this we had to use different resources like synthetic datasets, adapting existing datasets, or crowdannotating new data. our best approaches for both tasks where achieved by fine-tuning mistral 7b, either with synthetic data or with already existing resources.