Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020)

Abstract: A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained language model and map the predicted words to labels. Manually defining this mapping between words and labels requires both domain expertise and an understanding of the language model’s abilities. To mitigate this issue, we devise an approach that automatically finds such a mapping given small amounts of training data. For a number of tasks, the mapping found by our approach performs almost as well as hand-crafted label-to-word mappings.

## introduction pretraining language models on large corpora has led to improvements on a wide range of nlp tasks @xcite @xcite , but learning to solve tasks from only a few examples remains a challenging problem. as small datasets are common for many realworld applications of nlp, solving this challenge is crucial to enable broad applicability. a promising direction for many tasks is to reformulate them (e.g., by appending an instruction such as "translate into french") so that they can directly be solved by a pretrained language model @xcite @xcite . the key idea of pet @xcite , one such approach aimed at text classification, is to rephrase each input as a cloze question for which the language model's prediction can somehow be mapped to a label; an example is illustrated in figure 1 . while pet achieves remarkable results with little or no labeled training data, manually defining the required mapping between a language model's predictions and labels is difficult as it requires both taskspecific knowledge and an understanding of the language model's inner workings to identify words that it understands sufficiently well. in this work, we show how this mapping can be obtained automatically, removing the need for expert knowledge: we introduce pet with automatic labels (petal), a simple approach for identifying words that can serve as proxies for labels given small amounts of training data. at its core, our approach breaks the intractable problem of finding the mapping that maximizes the likelihood of the training data into several manageable subproblems. integrating our approach into pet significantly outperforms regular supervised training and almost matches the performance of pet with a manually defined mapping. ## related work reformulating problems as language modeling tasks has been explored in fully unsupervised settings @xcite @xcite , in few-shot scenarios with limited amounts of training data @xcite @xcite , and even in high-resource settings @xcite . the same idea is also commonly used for probing the knowledge contained within pretrained language models @xcite @xcite . ## pattern-exploiting training we review pattern-exploiting training (pet) as proposed by @xcite . let m be a pretrained masked language model (mlm), t its vocabulary and [mask] ∈ t the mask token. we consider the task of mapping textual inputs @xmath0.l.o.g. that @xmath1, . . . , k} for some @xmath2. in addition to training data @xmath3, y 1 ), . . . , (x n , y n )}, pet requires a set of pattern-verbalizer pairs (pvps). as exemplified in figure 1 , each pvp @xmath4, v) consists of • a pattern p that is used to convert inputs to cloze questions. formally, p : @xmath5; based on this intuition, schick and schütze (2020a) define the conditional probability distribution q p of y given x as there are several additional details to pet (e.g., an additional language modeling objective to prevent catastrophic forgetting); we skip these details as they are not relevant to our approach. for a more thorough explanation, we refer to @xcite . ## likelihood ratio verbalizer search manually defining the verbalizer v : @xmath6: it requires knowledge not only of a task's labels and how they can best be expressed in natural language using a single word, but also of the used mlm's capabilities as it is crucial to choose only such words as verbalizations that are understood sufficiently well by the language model and correspond to a single token in its vocabulary. we thus aim to automatically find a good verbalizer v for some pattern p without requiring task-or model-specific knowledge. our method requires sets v y ⊆ t of verbalization candidates for each label @xmath7; for now, we simply assume v @xmath8. let v be the set of all verbalizers consistent with these candidate sets, i.e., @xmath9. a natural criterion for measuring the suitability of a verbalizer v is to compute the likelihood of the training data given v, leading to the maximum likelihood estimate unfortunately, iterating over v to find the best verbalizer is intractable: the number of possible verbalizers |v| = |t | k grows exponentially in the number of labels and for a typical mlm, t contains tens of thousands of tokens. to circumvent this problem, we reframe the k-class classification task as k one-vs-rest classifications: for each @xmath10, we search for a verbalization v y that enables m to distinguish examples with label y from examples with any other label. to this end, we introduce binarized training sets t @xmath11, ỹ1 ), . . . , (x n , ỹn )} where ỹi = 1 if y @xmath12. for @xmath13, we define however, this reframing creates a label imbalance: if t is balanced, each t y contains k -1 times as many negative examples as positive ones. to compensate for this, we raise each q (p,vy) (ỹ | x) to the power of this can easily be derived from eq. 4 after compensating for the label imbalance as described above. unfortunately, there is the following problem with eq. 6: as the vocabulary t is quite large for most pretrained mlms, q (p,vy) (0 | x) will almost always be close to 1 and thus, log q (p,vy) (0 | x) ≈ log 1 = 0. this means that negative examples contribute almost nothing to this cross entropy loss, so optimizing for l ce results in verbalizations vy that are overall highly likely, but do not necessarily reflect the meaning of y. we fix this problem by considering not the absolute values of q (p,v i ) (ỹ | x), but the likelihood ratio (lr): independently, this lr criterion was recently shown to compare favorably to cross entropy in gradientbased neural network training for image classification @xcite . to arrive at l lr , we have made quite a number of modifications to our starting point, the intractable maximum likelihood estimate. however, the two objectives are in fact quite similar. the key difference is that eq. 2 enforces a large distance between m (v y | p (x)) and the maximum score assigned to the verbalizations of other labels, whereas eq. 7 enforces a large distance between m (v y | p (x)) and the average score assigned to the verbalizations of other labels; this is shown in appendix a. ## experiments for our experiments with petal, we use the pet implementation of schick and schütze (2020a) and follow their experimental setup. in particular, we use roberta-large @xcite as underlying mlm, we use the same set of hyperparameters for pet, the same evaluation tasks with the same patterns, and the same strategy for downsampling training sets. we deviate from @xcite in that we convert all inputs to single sequences (i.e., we remove all [sep] tokens) as we found this to slightly improve the verbalizers found by our approach in preliminary experiments. to ensure that our results are comparable with previous work and improvements in pet's performance are not simply due to this modification of patterns, we do so only for finding verbalizers and not for actual pet training and inference. we first analyze the verbalizers found by our method qualitatively. to this end, we consider yahoo questions @xcite , a dataset consisting of questions and answers that have to be categorized into one of ten possible categories such as "health", "sports" and "politics". we use the simple pattern p (x) = [mask] question: x and 50 training examples, meaning that we provide just five examples per label. to evaluate our approach quantitatively, we use the yelp review full star (yelp) and ag's news (ag's) datasets @xcite in addition to yahoo questions and mnli. the task for yelp is to guess the number of stars (ranging from 1 to 5) that a customer gave to a restaurant based on their textual review; for ag's, one of the four categories "world", "business", "sports" and "science/tech" has to be assigned to a news article. following schick and schütze (2020a), we again consider a scenario where we have |t | = 50 labeled training examples and a set of 10 000 • k unlabeled examples for each task; the unlabeled examples are only required for pet and not used for finding a verbalizer. for our approach, we consider both a variant where verbalizers are computed for each pattern separately (sep), and a variant were a single verbalizer is computed for all patterns as in avs (joint); for the latter, the likelihood ratio losses for all patterns are simply added up and minimized jointly. we use a multi-verbalizer v where v(y) are the n @xmath16: ## conclusion we have devised petal, a simple approach that enriches pet with the ability to automatically map labels to words. qualitative and quantitative analysis shows that our approach is able to identify words that are suitable to represent labels with as little as 50 examples and almost matches the performance of hand-crafted mappings for some tasks. for future work, it would be interesting to see whether the patterns required by pet can similarly be obtained in an automated fashion.