Inverse is Better! Fast and Accurate Prompt for Few-shot Slot Tagging (2022)

Abstract: Prompting methods recently achieve impressive success in few-shot learning. These methods modify input samples with prompt sentence pieces, and decode label tokens to map samples to corresponding labels. However, such a paradigm is very inefficient for the task of slot tagging. Since slot tagging samples are multiple consecutive words in a sentence, the prompting methods have to enumerate all n-grams token spans to find all the possible slots, which greatly slows down the prediction. To tackle this, we introduce an inverse paradigm for prompting. Different from the classic prompts mapping tokens to labels, we reversely predict slot values given slot types. Such inverse prompting only requires a one-turn prediction for each slot type and greatly speeds up the prediction. Besides, we propose a novel Iterative Prediction Strategy, from which the model learns to refine predictions by considering the relations between different slot types. We find, somewhat surprisingly, the proposed method not only predicts faster but also significantly improves the effect (improve over 6.1 F1-scores on 10-shot setting) and achieves new state-of-the-art performance.

## introduction few-shot learning (fsl) aims at learning a model from only a few examples and is regarded as one of the key steps toward more human-like artificial intelligence @xcite . recently, promptbased methods achieve impressive results and show promising prospects for few-shot learning of natural language processing (nlp) @xcite . prompt-based methods reformulate a target task into the language modeling problem, which takes advantages of the powerful pretrained language models (lm) @xcite 2019; * equal contributions. † corresponding author. g p t 2 none beijing none tomorrow morning g p t 2 new yor none departure refers to " original input " refers to " original input " arrival time refers to " original input " refers to " original input " price tomorrow morning . " original input " time refers to . departure refers to beijing . refers to price mapped labels original labels to.loc time price from.loc time price departure arrival original input: book a flight from beijing to new york tomorrow morning arrival . " original input " refers to tomorrow morning . departure refers to beijing . refers to time 1. 2. 主图 … query lm × 55 [cls] is a [mask] entity . [sep] original input book [cls] is a [mask] entity . [sep] original input book a [cls] is a [mask] entity . [sep] original input morning original input: book a flight from beijing to new york tomorrow morning time price departure arrival none g p t 2 query lm × 4 refers to " original input " price beijing tomorrow morning none new york refers to " original input " arrival refers to " original input " departure refers to " original input " time to.loc time price from.loc time price departure arrival none o original input: book a flight from beijing to new york tomorrow morning for normal prompts, identifying all slots in the query sentence requires enumeration of all spans, while inverse prompt only needs 1-time prediction for each label. @xcite . for example, when classifying the sentiment of the movie review "no reason to watch", prompting methods insert a piece of text "it was", i.e. prompts, to the input example, getting "no reason to watch. it was __". it is natural to expect a higher probability from the lm to fill the template with "terrible" than "great", and the original task is then converted to a language modeling task. such conversion reduces the gap between pretraining and target tasks, which allows less dependency on target task data and helps to achieve better performance in low data scenarios @xcite . however, while achieving great success in sentence level tasks, prompting-based methods show incompatibility for sequence labeling tasks, such as slot tagging. firstly, the aforementioned prompting paradigm is quite inefficient for slot tagging tasks. different from the sentence-level tasks that classify samples of whole sentences, slot tagging sam-ples are multiple consecutive words in a sentence. therefore, as shown in fig. 1 , to find all the possible slots, prompt-based methods have to enumerate all n-gram word spans, and then query lm for each of them, which greatly slows down the prediction @xcite . further, as a structure prediction problem, slot tagging benefits from taking the dependencies between labels into account @xcite . for example in fig. 1 , where the arrival entity often appears after a departure entity. such label dependency is hard to be captured by current prompting methods since they predict labels one-by-one independently. to tackle the above issues, we introduce an inverse paradigm for prompting. different from the classic prompts mapping tokens to labels, we reversely predict slot values given slot types. for the example in fig. 1 , we use an inverse prompt to modify the input as "book a flight from beijing to new york tomorrow morning. arrival refers to __", and then lm is able to decode multi-word span "new york" at a time. compared to the classic prompts that require predictions for every n-gram word span (55-times in fig. 1 ), we only need to perform decoding for v -times, where v is the number of label types (4-times in fig. 1 ), which therefore greatly speeds up the prediction. surprisingly, experiments show the proposed method not only predicts faster but also significantly improves the performance, indicating that prompting lm reversely is a better fit for the slot tagging task. besides, to further improve the prediction accuracy, we propose a novel iterative prediction strategy, from which the model learns to refine predictions by considering the relations between different slot types. to summarize the contribution of this work: the code and data are available at https://github.com/atmahou/ promptslottagging . ## background in this section, we begin with a formal definition of the few-shot slot tagging task ( §2.1), and then introduce the conventional sequence labeling approaches ( §2.2) and recent prompts-based methods ( §2.3) for this task. ## method to remedy the high cost of prompt prediction mentioned in the previous section, we introduce a novel inverse paradigm for prompting of slot tagging task, which significantly improves the speed of prediction by transforming the past fill-in-the-blank problem into a generative task. specifically, we first introduce the construction of our inverse prompts templates ( §3.1), and then describe how to use inverse prompts during training and inference ( §3.2). ## experiment we evaluate the performance of the proposed method on two classic few-shot scenarios: (1) setting with only in-domain data, where all training data are only a few labeled support data. (2) setting with meta source tasks, where some additional data-rich source domains are available for pretraining. evaluation to use same evaluation criteria as conventional sequence labeling methods, we need to label tokens reversely and get output in same format. after generation, we first separate outputs into slot values. for each slot value, we label tokens in the source sentence with three principles: (1) slot value is complete: only if the whole slot value matches a span in the source sentence, we label it with the corresponding label. (2) choose the first overlap predicted slot span: if any token in the source sentence has been labeled, we do not relabel this token even when it matches another slot value. ## conclusion in this paper, to liberate the prompting methods from the burdensome prediction of slot-tagging tasks, we introduce a novel inverse prediction manner to prompting methods of slot-tagging, which significantly improves both the efficiency and accuracy. to further improve performance, we propose an iterative prediction strategy for learning, which enables the prompting model to consider dependency between labels and refine prediction. extensive experiments verify the effectiveness of the proposed method in various few-shot settings, indicating inverse prediction is a better fit for prompting of slot tagging task. all the scientific artifacts used/created are properly cited/licensed, and the usage is consistent with their intended use. this paper does not collect new datasets, nor does the data used contain sensitive information.