Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups (2024)

Abstract: Complex Word Identification (CWI) is an essential step in the lexical simplification task and has recently become a task on its own. Some variations of this binary classification task have emerged, such as lexical complexity prediction (LCP) and complexity evaluation of multi-word expressions (MWE). Large language models (LLMs) recently became popular in the Natural Language Processing community because of their versatility and capability to solve unseen tasks in zero/few-shot settings. Our work investigates LLM usage, specifically open-source models such as Llama 2, Llama 3, and Vicuna v1.5, and closed-source, such as ChatGPT-3.5-turbo and GPT-4o, in the CWI, LCP, and MWE settings. We evaluate zero-shot, few-shot, and fine-tuning settings and show that LLMs struggle in certain conditions or achieve comparable results against existing methods. In addition, we provide some views on meta-learning combined with prompt learning. In the end, we conclude that the current state of LLMs cannot or barely outperform existing methods, which are usually much smaller.

## introduction complex word identification (cwi) aims to determine whether words or phrases are difficult for a target group of readers to understand. in the lexical simplification task, which targets replacing complex words and expressions with simplified alternatives @xcite , cwi is the first step, and it was treated as part of this task until 2012 when it became a standalone task @xcite . cwi was initially addressed as a binary classification task @xcite , identifying whether a word is complex in a given sentence. when the task became more popular @xcite , it was extended to the continuous domain as lexical complexity prediction (lcp, also referred to as the probabilistic classification for cwi) @xcite , addressing multilingual and multidomain settings. then, it was extended to multi-word expressions @xcite . recently, new datasets started to emerge in various languages and domains @xcite @xcite . previous approaches to cwi ranged from using support vector machines @xcite to deep neural networks based on bidirectional representation from encoder transformers @xcite , multi-task learning with domain adaptation @xcite , and sequence modeling @xcite . with the recent breakthrough in large language models (llms), particularly the work of openai with generative pre-trained transformer (gpt) models @xcite , natural language processing has seen a significant leap. these models have shown the potential to improve performances on various tasks as we scale up the model size and the amount of training data. the announcement of chatgpt foot_0 and its conversational capabilities has sparked a race in developing and fine-tuning new models for general purpose and domain-specific applications using palm @xcite , llama @xcite , orca @xcite , @xcite , @xcite , gpt-4o foot_1 , and many others. our work aims to provide the current state of llms in addressing cwi and lcp compared against state-of-the-art approaches. we focus on evaluating open-source (pre-trained llama 2 @xcite , llama 3 @xcite , vicuna @xcite ) and openai's closesource chatgpt-3.5-turbo and gpt-4o. we summarize the contributions as follows: ## conclusions in conclusion, we addressed cwi and lcp using llms, specifically llama-based and openai's gpt models. we observed that these models can determine the word difficulty level in multiple domains and languages, although with limited performances. meanwhile, these models struggle to label very difficult phrases correctly. future directions imply investigating multiple models in more languages. also, the prompts and example selection greatly influence the models' performance. thus, other future work should rely on reducing hallucination and determining which adversarial examples affect the model's capabilities the most.