TMU - HIT at MLSP 2024: How Well Can GPT -4 Tackle Multilingual Lexical Simplification? (2024)

Abstract: Lexical simplification (LS) is a process of replacing complex words with simpler alternatives to help readers understand sentences seamlessly. This process is divided into two primary subtasks: assessing word complexities and replacing high-complexity words with simpler alternatives. Employing task-specific supervised data to train models is a prevalent strategy for addressing these subtasks. However, such approach cannot be employed for low-resource languages. Therefore, this paper introduces a multilingual LS pipeline system that does not rely on supervised data. Specifically, we have developed systems based on GPT-4 for each subtask. Our systems demonstrated top-class performance on both tasks in many languages. The results indicate that GPT-4 can effectively assess lexical complexity and simplify complex words in a multilingual context with high quality.

## introduction the presence of unfamiliar words within a sentence can significantly impede its comprehension for readers. such complex words may cause misunderstandings of the sentence's content or result in wasted time as readers may find themselves compelled to consult definitions of unfamiliar words. the development of a system capable of automatically simplifying complex words would enable readers to proceed without interruption. to achieve this, it is essential to first identify complex words and then replace them with more comprehensible alternatives. numerous researchers have been undertaken focusing on each challenge, engaging in specialized endeavors known as lexical complexity prediction (lcp) @xcite and lexical simplification (ls) @xcite @xcite . lcp is a task that assesses the complexity of a target word, i.e. its level of difficulty for understanding. various methodologies have been proposed to tackle this task. a classical strategy is the frequency-based approach @xcite , which attributes higher complexity scores to words of lower frequency. given the availability of supervised data, one viable option is to train a regression model to evaluate the word's complexity @xcite . however, such abundant linguistic resources for supervised learning are scarce for many languages @xcite . therefore, there exists a need for an approach capable of determining lexical complexity without reliance on supervised data. ls is a task that replaces a complex word with easier synonyms while maintaining the original meaning or information of the sentence. the ls pipeline comprises three primary components @xcite : substitute generation (sg), substitute selection (ss), and substitute ranking (sr). sg returns several candidate alternative words for the target word in a given sentence. ss then shifts through these alternatives, eliminating those unsuitable for the target word, such as more complex words or words with different parts of speech. sr sorts the candidate alternatives to prioritize words more suitable as alternatives, ensuring they appear higher in the ranking. recent ls studies @xcite have often employed pre-trained language models, and systems using gpt-3.5 @xcite have demonstrated particularly effective performance. this paper outlines our approach to the mlsp-2024 shared task @xcite . our system was based on gpt-4 and exclusively utilized the trial data provided by the organizers @xcite @xcite , without incorporating any supplementary datasets. in the lcp task, our system achieved the highest pearson's correlations in nine out of the ten languages. additionally, in the ls task, our system exhibited the highest performance in seven out of the ten languages. the findings indicate that gpt-4 has capabilities of assessing lexical complexity and replacing complex words with simpler alternatives in a multilingual context with a high level of quality. ## lcp subtask in lexical complexity prediction (lcp), we assessed the complexities using gpt-4 foot_0 based system. specifically, we employed g-eval @xcite , a scoring framework using large language models with a chain-of-thoughts, for our system. ## ls subtask in tsar-2022 shared task @xcite of ls, the system using gpt-3.5 @xcite demonstrated a significant lead over other neural approaches such as those using mask language models. following these findings, we employed a gpt-based method using the latest available gpt-4 foot_3 for ls. ## conclusion in this paper, we introduced gpt-4-based systems designed to assess word complexities and replace complex words with simpler ones. our systems achieved superior performance in multiple languages for both lcp and ls tasks within mlsp-2024 shared task. to score complexities, we created a base prompt following g-eval @xcite and added to the base prompt supplementary prompts to delineate the language of the test example and the role of the llm. our prompt, when applied within a three-shot setting, consistently achieved the highest pearson's correlation across the majority of languages. furthermore, our experiments suggest the potential for performance enhancement through the augmentation of few-shot examples. therefore, we plan to explore the change in performance resulting from an increment in the number of few-shot examples. for the task of replacing complex words with simpler alternatives, we manually crafted prompts. the experimental results indicate that these prompts yield alternatives of commendable quality. additionally, we explored the possibility of enhancing the selection of generated alternatives by employing a re-ranking strategy using either gpt-4 or xglm that were instruction-tuned by trial data. however, the re-raking approaches degraded the scores compared to the ones before re-ranking. for future work, we plan to devise an improved re-ranking methodology.