ISEP _ P residency_ U niversity at MLSP 2024 Shared Task: Using GPT -3.5 to Generate Substitutes for Lexical Simplification (2024)

Abstract: Lexical substitute generation is a task where we generate substitutes for a given word to fit in the required context. It is one of the main steps for automatic lexical simplifcation. In this paper, we introduce an automatic lexical simplification system using the GPT-3 large language model. The system generates simplified candidate substitutions for complex words to aid readability and comprehension for the reader. The paper describes the system that we submitted for the Multilingual Lexical Simplification Pipeline Shared Task at the 2024 BEA Workshop. During the shared task, we experimented with Catalan, English, French, Italian, Portuguese, and German for the Lexical Simplification Shared Task. We achieved the best results in Catalan and Portuguese, and were runners-up in English, French and Italian. To further research in this domain, we also release our code upon acceptance of the paper.

## introduction test simplification is an important educational application. it aims to simplify text to make the generated simpler text easier for reading and comprehension by different readers who may be either young learners, people with language disabilities (eg. aphasia), second-language learners, etc. a lot of the research done in the area of text simplification is split into mainly 2 parts, namely syntactic simplification and lexical simplification. syntactic simplification involves splitting the sentences into smaller sentences @xcite . lexical simplification, on the other hand, involves simplifying the text by replacing more complex words and phrases with simpler, and in context, synonyms @xcite . the lexical simplification pipeline consists of multiple sub-tasks, @xcite as shown in figure 1. these subtasks are complex word identification (where we identify which word we have to consider for simplification), substitution generation (where we generate candidate synonyms for the given complex word), substitution selection (where we select the candidate synonyms which are contextually correct), and substitution ranking (where we rank the selected candidates from easiest to most complex). with the advent of large language models (llms) like gpt-3, the potential for automating this task has increased significantly. these models, trained on vast amounts of text, have shown remarkable proficiency in understanding context and generating human-like text. unlike pre-trained language models like bert @xcite , llms are significantly harder to fine-tune due to the massively larger number of parameters (bert has about 110 million parameters, while gpt-3 has about 175 billion parameters). because of this, we use gpt-3 using prompt-engineering, where we provide a prompt to the system to generate substitutes. ## problem statement the multilingual lexical simplification pipeline (mlsp) shared task dealt with 2 problems. the first was lexical complexity prediction (lcp). in this task, the participants had to develop a system where they were given a context and word in a given language, and they had to assess how easy / complex the word was foot_1 . this is similar to the semeval 2021 shared task on lexical complexity prediction @xcite . the second problem was called lexical simplification (ls), where we are given an input context and complex word and we need to generate a ranked list of upto 10 simplifications in increasing order of complexity (i.e. from the simplest substitute to the most complex substitute). this is similar to the 2022 text simplification, accessibility and readability shared task on lexical simplification @xcite . both these problem statements required participants to build systems in multiple languages, such as english, catalan, french, german, italian, brazilian portuguese, spanish, bengali, sinhala, filipino, and japanese. in our paper, we mainly focus on the second task (lexical simplification) for the first 6 languages listed (english to brazilian portuguese). more details on the shared task are available inn the shared task report @xcite . ## related work there has been a number of shared tasks dealing with different aspects of the lexical simplification pipeline. for complex word identification, one of the earliest shared tasks was held in 2016 @xcite . the winners of that shared task used a system of soft voting with different "voters", where the voters are either lexicon based, thresholdbased, or machine-learning assisted @xcite . in 2018, another shared task on complex word identification was held as part of the bea workshop collocated with naacl (yimam et al., 2018). it had a monolingual track and multilingual tracks where the systems would be tested on german, french and spanish. the winning team @xcite , used a similar approach as paetzold and specia (2016b), but with a much wider range of features. one of the challenges is trying to assess a score for how simple / complex a word is, given the context. this step is critical for complex word identification. in light of this, @xcite conducted a lexical complexity prediction task at semeval 2021. the advent of llms inspired a significant change in task specification. the 2022 tsar shared task on lexical simplification @xcite had 3 languages -english, spanish and brazilian portuguese. the participants in that shared task had to generate a set of substitutes for each language. while some systems such as @xcite used a procedure of candidate generation (using pre-trained language models like bert @xcite ), cosine similar-ity and part-of-speech tagging as filters, the winning team @xcite for the shared task, participants were provided only with trial data. that is, a very few contextcomplex word pairs. each language had a trial dataset of 30 context-complex word pairs foot_2 . our systems were then evaluated on test sets of varying sizes. ## results and analysis we report the results of our experiments in one of the challenges that we faced was in constructing the prompts for different languages. while the authors of the paper are l1 / fluent speakers of english and french, we needed the help of google translate to translate the prompts from english to other languages like german / italian / portuguese. one of the challenges of using llms currently is that they are computationally intensive, requir-ing hundreds of gb of gpu power to fine-tune. another challenge is that the current llms are focused on generating ranked substitutes irrespective of the target user. for example, young learners may have different requirements for simplification, as opposed to second-language learners, or people with reading disabilities. this can be tackled by modifying the prompts (especially the one-shot / few-shot prompts) to generate different simplifications based on the target user. ## conclusion and future work although we have performed reasonably well in the shared task for lexical simplification, we would like to extend our work for other languages which we were not able to participate in. most of the other languages possess orthographic challenges because they do not use the roman script, such as bengali, japanese, etc. in the future, we would also like to focus on instruction tuning to improve the performance for personalizing the llm for simplification. currently, the predictions from the llm are independent of the user. this means that a system built using this approach may generate the same output irresppective of the user the text should be simplified for. one method for resolving this is to utilize a user's cognitive information to try to perform complex word identification, as well as generate and rank candidate simplifications.