WARP : W ord-level A dversarial R e P rogramming (2021)

Abstract: Transfer learning from pretrained language models recently became the dominant approach for solving many NLP tasks. A common approach to transfer learning for multiple tasks that maximize parameter sharing trains one or more task-specific layers on top of the language model. In this paper, we present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. Adversarial reprogramming attempts to learn task-specific word embeddings that, when concatenated to the input text, instruct the language model to solve the specified task. Using up to 25K trainable parameters per task, this approach outperforms all existing methods with up to 25M trainable parameters on the public leaderboard of the GLUE benchmark. Our method, initialized with task-specific human-readable prompts, also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks with just 32 training samples.

## introduction language model pretraining has had a tremendous impact on solving many natural language processing tasks @xcite @xcite . the most popular two approaches take a pretrained model and use a straightforward supervised learning objective. in the first approach, the parameters of the language model are frozen and a task-specific head is trained on top of them @xcite . the second approach fine-tunes all model parameters @xcite . the latter can sometimes yield better results @xcite , while the first one usually offers better stability for smaller datasets. the approach based on frozen features does not require storing task-specific language models. a recent alternative is based on so called adapters @xcite , a technique that adds new weights at every layer of the pretrained language model while the original parameters are kept frozen. this enables a smaller set of task-specific parameters while achieving results comparable to the fine-tuning approach. another approach of leveraging pretrained language models for downstream tasks, introduced by @xcite , provides "task descriptions" without using any labeled examples. gpt-3 @xcite demonstrates impressive few-shot learning performance with priming: by providing the language model a few inputs and outputs ("analogies") as a context. the language model contextually "learns" from these examples and outputs the answer with a single forward pass without any trainable parameters. these methods, however, require huge language models (1.5b and 175b parameters, respectively). the success of task reformulation-based approaches suggest that language models are capable of solving various natural language processing tasks given a well-crafted prompt. we hypothesize that it is possible to find such prompts. in other words, we can discover extra tokens that, when added to the input, can exploit language model capabilities better than the manually-designed ones. in this paper, we introduce a novel technique to find optimal prompts. we call our method warp: word-level adversarial reprograming foot_0 . the method is inspired by adversarial reprogramming @xcite -a method of adding adversarial perturbations to an input image that reprograms a pretrained neural network to perform classification on a task other than the one it was originally trained for. we show that our method, using up to 25k trainable parameters per task, achieves 81.6 test score on the glue leaderboard, outperforming all the other submissions that use up to three orders of magnitude more trainable parameters. we show that it is possible to inject knowledge into warp models using manually designed initialization of the prompt, which is especially useful on tasks with a small number of examples. moreover, warp shows impressive few-shot performance on two tasks from the superglue benchmark with just 32 examples, outperforming gpt-3 results. finally, we discuss the advantages of our method in real-life applications. another approach, called adapters @xcite , introduces new task-specific parameters that are added at every layer of the transformer network. only these newly initialized weights are trained, which allows separation of general and task-specific knowledge. in contrast, our method does not inject taskspecific knowledge inside the body of the pretrained language model. instead, it focuses on learning task-specific input-level prompts. a_lovely_film_... a_pretentious_mess_... two_hours_of_junk. ## warp we follow a setup similar to @xcite with some nlp-specific modifications depicted in figure 2 . our goal is to find the best prompt that will make a pretrained masked language model predict the desired answer (verbalizer token) for a training example's masked token 2 . we search for such prompts in the (continuous) embedding space. in other words, we want to find parameters θ = {θ p , θ v } for prompt and verbalizer embed-dings, respectively, such that: in figure 2 , the template t θ p (x) prepends θ p 1 and appends θ p 2 , θ p 3 , θ p 4 parameters to the word embeddings and uses θ v + and θ v -to calculate the probabilities on the masked token position for positive and negative classes. [mask] token into the input sequence. these tokens might be placed before or after the sentences, depending on the prompt template. we set the optimization objective to a crossentropy loss between the head output of the masked language model and the verbalizer tokens the only trainable parameters are the word embeddings for [p 1], ..., [p k] and in case we want to train models for multiple tasks, these are the only task-specific parameters we need to store. the entire "body" of the large language model (all attention layers, feedforward layers, and all other word embeddings) remains untouched. note that, unlike most adversarial attacks, we do not update the embeddings of the original tokens of the input. this follows the intuition from @xcite , when the pixels of mnist or cifar images are left untouched, and only padding pixels are updated. we train these parameters by minimizing the loss on the training set of the downstream task. ## experiments on glue following prior work, we evaluate our method on the glue benchmark @xcite , which consists of 9 natural language understanding tasks. generally, we perform single-task warp training, with early stopping and model selection using the original validation sets, if not stated otherwise. ## few-shot experiments the fact that warp can be initialized using manually designed natural prompts suggests that we can similarly benefit from such human attribution similar to ipet (schick and schütze, 2021b), especially in scenarios with limited training data. ## discussion we do not see any prompt tokens that are meaningful in the context of the tasks. as expected, the verbalized tokens are more interpretable. for the results for autoprompt and fine-tuning are taken from @xcite . example, the embedding for the "contradiction" class of mnli is close to the token "unless". the embeddings for "negative" and "positive" classes of sst-2 task are close to "defective" and "important", respectively. other verbalized tokens are non-interpretable (e.g. "470" or word pieces with non-latin characters). ## conclusion in this paper we have proposed an alternative way to transfer knowledge from large pretrained language models to downstream tasks by appending carefully optimized embeddings to the input text. the method outperforms existing methods with significantly more trainable parameters on glue benchmark tasks and shows an impressive performance in a few-shot setting on two superglue tasks. on the sentiment analysis task, the performance is comparable to the fully fine-tuned language models. this method can save a lot of storage in software applications designed to serve large numbers of sentence classification tasks.