U o M & MMU at TSAR -2022 Shared Task: Prompt Learning for Lexical Simplification (2022)

Abstract: We present PromptLS, a method for fine-tuning large pre-trained Language Models (LM) to perform the task of Lexical Simplification. We use a predefined template to attain appropriate replacements for a term, and fine-tune a LM using this template on language specific datasets. We filter candidate lists in post-processing to improve accuracy. We demonstrate that our model can work in a) a zero shot setting (where we only require a pre-trained LM), b) a fine-tuned setting (where language-specific data is required), and c) a multilingual setting (where the model is pre-trained across multiple languages and fine-tuned in an specific language). Experimental results show that, although the zero-shot setting is competitive, its performance is still far from the fine-tuned setting. Also, the multilingual is unsurprisingly worse than the fine-tuned model. Among all TSAR-2022 Shared Task participants, our team was ranked second in Spanish and third in English.

## introduction we present our system submission for the tsar-2022 shared task (st) on lexical simplification @xcite . the task required participants to develop a lexical simplification system capable of taking a word in context and returning a list of candidate substitutions. the task provided test data in english (en), spanish (es) and brazilian portuguese (pt). we chose to submit for all three tracks a system based on the concept of prompt learning. whereas the previous state of the art for lexical simplification, lsbert @xcite , masked the token in context, our approach, namely promptls, injects prompts within the context that forces the model to generate appropriate substitutions as in ## related work lexical simplification arose as a form of assistive technology @xcite for people with aphasia. early systems used dictionary based replacement methods @xcite , with disambiguation methods to improve the selection of candidates @xcite . recently, simplification systems have focused on the use of transformer architecture to identify appropriate replacements for a given word @xcite . this can be applied at a single or multi-word level @xcite . prompt learning is a method of leveraging the learnt probabilities in a large pre-trained language model to solve nlp tasks @xcite . this can be done in a zero shot @xcite , or fine-tuned setting @xcite . prompt learning requires the design of a prompt @xcite , which can be engineered @xcite , or generated @xcite . ## methodology in this section, we start by the description of our selected datasets (section 3.1) and the design of our prompts (section 3.2). we then describe our proposed method promptls that consists of three modules: 1) a large language model (lm) that generates candidates based on a given prompt (section 3.3), 2) a fine-tuning module that guides the lm to select more appropriate substitutes (section 3.4) and 3) a candidate filtering module which removes incorrect or inappropriate candidates (section 3.5). ## results for english, we executed 48 runs in a zero-shot setting, 240 for the fine-tuned mlm, and 192 for the multilingual settings. for spanish, we executed 160 runs for the zero-shot and fine-tuned model and 140 for the multilingual setting. similarly, for portuguese, we ran 106 runs for zero-shot, 169 for fine-tuned setting and 144 for our multilingual setting. overall, we ran more than 600 experiments for each language with multiple combinations of prompts, context windows, number of candidates for data augmentation, models and settings for the selection of our submitted system. 9 in in the benchmark for each language. nevertheless, for our st submission, we selected the three best-performing runs in the development set using a ranking of all the models results. we show our final systems in ## discussion in in addition, we observed that context words around a complex word are important in this task. in all the settings reported in concerning the model selection, it is noted that t5 model @xcite is also a suitable baseline for a prompt-based setting. however, unlike the experimental mlms, t5 has a decoder, which requires additional effort to apply it to lexical simplification. we therefore leave this implementation for future work. ## conclusion in this paper, we presented the implementation of a prompt-learning system for ls. our experiments indicate we can obtain reasonable results even in zero-shot settings, especially for full resourced languages such as english. we demonstrate that by fine-tuning our prompt templates, we obtain competitive results in all languages. as future work, we intend to experiment with better datasets, including better filtering and ranking methods for ls.