U ni HD at TSAR -2022 Shared Task: Is Compute All We Need for Lexical Simplification? (2022)

Abstract: Previous state-of-the-art models for lexical simplification consist of complex pipelines with several components, each of which requires deep technical knowledge and fine-tuned interaction to achieve its full potential. As an alternative, we describe a frustratingly simple pipeline based on prompted GPT-3 responses, beating competing approaches by a wide margin in settings with few training instances. Our best-performing submission to the English language track of the TSAR-2022 shared task consists of an “ensemble” of six different prompt templates with varying context levels. As a late-breaking result, we further detail a language transfer technique that allows simplification in languages other than English. Applied to the Spanish and Portuguese subset, we achieve state-of-the-art results with only minor modification to the original prompts. Aside from detailing the implementation and setup, we spend the remainder of this work discussing the particularities of prompting and implications for future work. Code for the experiments is available online at https://github.com/dennlinger/TSAR-2022-Shared-Task .

## introduction with recent advancements in machine learning (ml) research coming largely from increasing compute budgets, richard sutton coined the idea of a "bitter lesson", wherein more computational power will ultimately supersede a hand-crafted solution @xcite . more recently, increasing compute power on a general purpose architecture has also shown to be wildly successful in the natural language processing (nlp) community @xcite . in particular, emergent capabilities in very large language models (vllms) have made it possible to approach a variety of tasks wherein only few (if any) samples are labeled, and no further fine-tuning on task-specific data is required at all. in stark contrast to the complex pipelines in modern lexical simplification systems @xcite @xcite , we present a simplistic approach utilizing few-shot prompts based on a vllm with basic instructions on simplification, which returns frustratingly good results considering the overall complexity of the approach, which utilizes a grand total of four hand-labeled instances. we present our results on the tsar-2022 shared task @xcite , which evaluates lexical simplification systems in three available languages (english, spanish and portuguese), with ten labeled instances and around 350 unlabeled test samples provided per language. for the english subset, official results rank our model as the best-performing submission, indicating that this approach may be another instance of the bitter lesson. while the initial findings are indeed promising, we want to carefully evaluate erroneous instances on the test set to analyze potential pitfalls, and further detail some of our experiences in hand-crafting prompts. we also acknowledge the technical challenges in reproducing (and deploying) systems based on vllms, especially given that suitable models exceed traditional computing budgets. ## prompt-based lexical simplification with the public release of the gpt-3 language model @xcite , openai has started the run on a series of now-available vllms for generalpurpose text generation @xcite @xcite . across these models, a general trend in scaling beyond a particular parameter size can be observed, while keeping the underlying architectural design close to existing smaller models. through exhibiting zeroshot transfer capabilities, such models have also become more attractive for lower-resourced tasks; oftentimes, models are able to answer questions formulated in natural language with somewhat sen-sible results. particular template patterns (so-called prompts) are frequently used to guide models towards predicting a particularly desirable output or answer format, without requiring a dedicated training on labeled examples. utilizing this paradigm shift, we experimented with different prompts issued to openai's largest available model, text-davinici-002, which totals 176b parameters. our first approach uses a singular prompt template in a zero-shot setting to obtain predictions for the shared task; we further improve upon these results by combining predictions from different prompt templates later on. ## conclusion and future work utilizing prompted responses from vllms seems to be a promising direction for lexical simplification; particularly in the constrained setting with pre-identified complex words the model performs exceptionally well, even when presented with a severely restricted budget of labeled training data. while the approach also offers promising directions for multi-and cross-lingual approaches, obtaining state-of-the-art results in other languages, we are presented with a prohibitive amount of computation per sample instance. it would therefore be an interesting addition to deal with resource-constraint systems, putting the prediction power into a slightly different perspective. finally, we are reminded of the unstable nature of neural lms; given similar inputs, quality can vary greatly between samples, including a complete breakdown in performance. for future work, we are considering approaches to generate static resources from vllms @xcite , which may require only a onetime commitment to spending on datasets, which can then used as training data for cheaper systems. exploration of prompt tuning approaches for automated search of suitable prompt templates would also greatly accelerate the development process of domain-specific applications @xcite .