LSL lama: Fine-Tuned LL a MA for Lexical Simplification (2023)

Abstract: Generative Large Language Models (LLMs), such as GPT-3, have become increasingly effective and versatile in natural language processing (NLP) tasks. One such task is Lexical Simplification, where state-of-the-art methods involve complex, multi-step processes which can use both deep learning and non-deep learning processes. LLaMA, an LLM with full research access, holds unique potential for the adaption of the entire LS pipeline. This paper details the process of fine-tuning LLaMA to create LSLlama, which performs comparably to previous LS baseline models LSBert and UniHD.

## introduction lexical simplification (ls) is a sub-task within the field of text simplification @xcite in which complex words are substituted with simpler words while maintaining the meaning of the surrounding sentence @xcite . this is done to improve the comprehension of text for those who do not have sufficient reading proficiency, such as a language learner, young child, or someone with a learning disability @xcite . current ls models usually involve a multi-step process, including 1) the identification of complex words; 2) the generation of substitution words; 3) the selection of the substitutes based on context; 4) ranking substitutes by their simplicity; and 5) further context adaptation @xcite . recently, deep learning has been incorporated into some of these steps, such as a method that adapted bert @xcite , a bidirectional encoder, to generate substitution words @xcite . however, the development of generative llms, such as gpt-3 @xcite , presents an opportunity to drastically simplify this multi-step process by utilizing their ability to process and evaluate natural language. while these llms have significantly more parameters (110m for bert vs. 175b for gpt3), and their training requires a substantially greater amount of computation than bert-based models, this cost could be mitigated by fine-tuning a pre-trained llm on a selected task. this fine-tuning could enable the model size to be drastically reduced while maintaining similar performance. large language model meta ai (llama) is a recently released generative llm @xcite . the size of its smallest variant, at 7b parameters, and its full research access provide a unique potential for adapting it to perform the ls task. this paper details the novel fine-tuning of llama on a lexical simplification task to create lsllama. in order to determine if lsllama is effective at the task, its performance was evaluated and compared to previously existing benchmark ls models that use deep learning, such as lsbert @xcite and unihd @xcite using three testing datasets foot_0 . the structure of the following sections is as follows: section 2 details related work in ls and deep learning. section 3 details the method used to finetune lsllama and evaluate the models. section 4 details the results of this evaluation, and section 5 discusses the implications of these results, error analysis, and limitations of the method. section 6 concludes the paper and points to possible future work. ## related work the first method to incorporate deep learning into ls involved the use of neural networks to rank substitution candidates after being generated by a word embeddings model @xcite . a later implementation @xcite ) used bert, a bidirectional encoder transformer model. bert was used as a masked language model to predict the masked word in a sample sentence for simplification. the proposed substitution candidates were then ranked using a combination of lsbert, semantic similarity, and a frequency feature. recently developed generative llms such as gpt-3 @xcite ) can be trained on billions of tokens and can process and respond in natural language. in the tsar-2022 shared task on lexical simplification @xcite , the highest scoring model on the english language task, called unihd, used gpt-3 inference with six prompt variations to generate substitution candidates and a ranking algorithm that combined these candidate lists @xcite . while this method outperformed all other bert-based models, gpt-3 is not fully publicly available, and inference had to be done using paid api requests. a recent generative llm, llama, @xcite was shown to achieve similar performance to gpt-3 in various nlp tasks at a fraction of the size with architecture improvements and a 1t token training set. alpaca, a model that was created by fine-tuning llama on 52k questionanswer prompts, was also found to often behave similarly to chatgpt in answering broad sets of questions @xcite . ## method in order to evaluate lsllama and compare it to lsbert and unihd, llama was first fine-tuned on an ls task to produce lsllama. the specific version of lsbert used was adapted from its original version to act as a benchmark of an ls task @xcite . all three models were then used to propose substitution candidates on three different datasets and evaluated on performance metrics. ## discussion on the benchls dataset, lsllama achieved the best scores on all evaluation metrics, so lsllama consistently outperformed lsbert and unihd on this dataset. on nnseval, there were more mixed results in the acc@k@top1 metrics, with both unihd and lsllama having the highest score in different trials. however, for all other metrics on nnseval, unihd performed the best. on lexmturk, lsllama performed the best in the acc@k@top1 metrics, while unihd performed as well or better than lsllama on the other metrics. lsbert was always outperformed by either unihd or lsllama. unihd was better able to identify the top-ranked substitution candidate, as evidenced in its highest accuracy@1 value, have a better ranking of candidates, as evidenced by its highest map@k value, and propose more relevant candidates when having multiple attempts, as shown by its highest potential@k values on the nnseval and lexmturk datasets. however, on these datasets, lsllama usually best identifies the test dataset's top-ranked candidate in the first few substitution candidates, as evidenced by its highest acc@k@top1 values on all but one trial. on the benchls dataset, lsllama was best able to identify the top-ranked substitution candidates, have a better ranking of candidates, and propose more relevant candidates when having multiple attempts. when looking at results overall, lsllama always outperformed unihd on one dataset. with the other two datasets, lsllama outperformed unihd on some metrics, and unihd outperformed lsllama on some metrics, with unihd outperforming lsllama more often. this indicates that, overall, unihd and lsllama performed comparably when looking at all datasets as a whole. additionally, for all trials, at least one of these two models, unihd and lsllama, always outperformed lsbert, as lsbert was never the top scoring model on any metric. ## conclusion and future work this paper compared a fine-tuned, generative llm, lsllama, to previously existing ls baseline models lsbert and unihd. at evaluation, lsllama outperformed unihd in all metrics on one dataset, while on the other two datasets, unihd outperformed lsllama on most, but not all, metrics. ls-bert also never scored the highest on any metric. regarding their architectures, generative llms require more computational resources than bertbased models, however lsllama is able to simplify the multi-step process of lsbert. lsllama generates and ranks substitution candidates at inference, whereas a separate ranking algorithm is used after inference from lsbert. a separate ranking algorithm is also used in unihd. even though unihd and lsllama are both generative llms, lsllama takes advantage of fine-tuning to significantly reduce its size, from 175b parameters for gpt-3 to 7b parameters for lsllama, while maintaining comparable performance on evaluation metrics. despite some recorded challenges posed by their architecture, this research demonstrates the potential for llama and other llms that are fine-tuned on a ls task to improve upon existing benchmarks in lexical simplification. for future work, fine-tuning could be done using multiple datasets. this could improve the model's specificity, as it would increase the size of the training set used for fine-tuning. testing then can be done on one dataset. in addition, further manipulation of the fine-tuning hyperparameters, inference parameters, and prompt wording could also be pursued to improve the performance of lsllama.