GMU at MLSP 2024: Multilingual Lexical Simplification with Transformer Models (2024)

Abstract: This paper presents GMU’s submission to the Multilingual Lexical Simplification Pipeline (MLSP) shared task at the BEA workshop 2024. The task includes Lexical Complexity Prediction (LCP) and Lexical Simplification (LS) sub-tasks across 10 languages. Our submissions achieved rankings ranging from 1st to 5th in LCP and from 1st to 3rd in LS. Our best performing approach for LCP is a weighted ensemble based on Pearson correlation of language specific transformer models trained on all languages combined. For LS, GPT4-turbo zero-shot prompting achieved the best performance.

## introduction understanding lcp and ls is crucial for enhancing communication accessibility and readability across diverse linguistic contexts. lcp involves analyzing linguistic features to understand text difficulty, while ls focuses on making complex language more readable without losing its meaning. therefore, lcp and ls provide inclusive communication and broadening access to information. nowadays, nlp research is interested in identifying complex words which may be difficult for certain readers @xcite . these difficult words requires various types of intervention, such as direct replacement in the setting of ls @xcite , or generating further explanation @xcite previously, the task of lcp involved labelling the complex words by binary classification @xcite @xcite . this approach was referred to as complex word identification (cwi) which means a word can either be complex or not. however, in practice, word complexity should be a continuous value representing from the least to the most complex. @xcite and @xcite were the first to introduce the task of lcp where a continuous value is assigned to identify a word's complexity. ls is the task of replacing difficult words with easier synonyms while preserving the information and intelligibility of the original text. this is a sub-task of automatic text simplification (ats) @xcite . recently, similar to lcp, this task has also gained considerable amount of attention @xcite . in this paper, we use a cross-lingual weighted ensemble of transformer models to find lcp of a word in context of a sentence for 10 languages. for ls, we use gpt4-turbo (openai, 2023) zero-shot prompting and also top 10 suggestions of gpt4turbo and transformers models in terms of cosine similarity for 10 languages. ## datasets the mlsp shared-task @xcite covers 10 different languages -catalan, english, filipino, french, german, italian, japanese, portuguese, sinhala, spanish and it has two sub-tasks-lcp and ls. lcp data instances include a sentence of a specific language and a specific word from that language of various text genre like news, religious, educational, wikibooks etc. @xcite . then a complexity value ranging from 0-1 of that specific word in the context of that sentence is given. ls also has similar types of data instances but instead of a complexity value 10 simplified substitutions of the target word are provided for each instance. moreover, multils sp/ca dataset was used for both the lcp and ls task for spanish and catalan language @xcite . for each language, the data annotators are from different age group and professions like students, language learners, university faculty, freelancers. the data was annotated by both native and nonnative speakers of each specific language. the data count for all the languages are shown in ## experiments trial data provided for all the languages of the lcp task is very small. in general, it is common to use data augmentation and back-translation techniques to increase the number of data instances in such conditions @xcite . however, it will not work here as these techniques can change the word or even the context of the word after augmentation and back-translation causing change to the complexity also. as such, we use the idea of cross-lingual weighted ensemble approach by using trial data of all the languages for training and validation. we used 80-20 train and validation split. after that we use the test data of the target language for predicting lexical complexity. for training we have used weighted ensemble of mbert @xcite , xlm-r @xcite and language specific bert models. for catalan, filipino, french, german, italian, japanese, sinhala and spanish we used cal-bert @xcite , roberta-tagalog (cruz and cheng, 2021), flaubert @xcite , ger-manbert @xcite , italianbert @xcite , japanesebert (tohoku-nlp, 2020), sin-halaberto @xcite and span-ishbert @xcite ) respectively. for english we used bert @xcite , roberta @xcite and deberta @xcite as language specific model. for all language combined -ensemble of mbert, xlm-r calbert, deberta, roberta-tagalog, flaubert, germanbert, italianbert, japanese-bert, bertimbau, sinhalaberto and spanish-bert were used. pearson correlation coefficient was used as weight for the ensemble. we use gpt4-turbo @xcite zero shot prompting which provides the best result for ls on both trial and test phase. additionally, we used the same set up of bert based models like lcp for all the languages to find the best 10 simplified substitutions for trial and test data. then for each instances of a language, we took the set of all the words suggested by the bert based models and gpt4-turbo together. after that, we find the embeddings of those words and the target token by labse sentence transformer @xcite . furthermore, we find the cosine similarity of the target token to the set of suggested word embeddings. lastly, we choose the best 10 words by the decreasing order of cosine similarity of the embeddings. ## results trial results of lcp and ls are available in ## error analysis for lcp the highest mean absolute and squared error are 0.2089 and 0.0589 for french and the lowest mean absolute and squared error are 0.1018 and 0.0168 for sinhala. this is an acceptable mar for ls, zero-shot prompting by gpt4-turbo alone provides the best result but when we try to find the best 10 suggestions from the set of suggestions generated by the bert based models and gpt4-turbo together, the result significantly decreases. this was because the target token in the sentence varied be in different grammatical form. therefore, finding proper simplified suggestions that fits the context proves to be a struggle for the bert based model. ## conclusion our team gmu's approaches in mlsp 2024 shared task achieved competitive results across multiple languages for both the lcp and ls sub-tasks. the weighted ensemble technique based on transformer models proved effective for lcp, while gpt-4 zero-shot prompting excelled at ls. the multilingual nature of this shared task also highlights the importance of developing techniques that can generalize across languages. one key limitation of our approach is the reliance on cross-lingual transfer due to limited language-specific training data for most languages. while this allowed sharing resources across languages, having larger datasets to each language could potentially boost performance. additionally, the error analysis revealed some remaining challenges in handling complex word expressions and phrases during ls. further improvements in modeling could address these cases more effectively for mlsp in future.