Noisy Channel Language Model Prompting for Few-Shot Text Classification (2022)

Abstract: We introduce a noisy channel approach for language model prompting in few-shot text classification. Instead of computing the likelihood of the label given the input (referred as direct models), channel models compute the conditional probability of the input given the label, and are thereby required to explain every word in the input. We use channel models for recently proposed few-shot learning methods with no or very limited updates to the language model parameters, via either in-context demonstration or prompt tuning. Our experiments show that, for both methods, channel models significantly outperform their direct counterparts, which we attribute to their stability, i.e., lower variance and higher worst-case accuracy. We also present extensive ablations that provide recommendations for when to use channel prompt tuning instead of other competitive models (e.g., direct head tuning): channel prompt tuning is preferred when the number of training examples is small, labels in the training data are imbalanced, or generalization to unseen labels is required.

## introduction prompting large language models, by prepending natural language text or continuous vectors (called prompts) to the input, has shown to be promising in few-shot learning @xcite . prior work has proposed methods for finding better prompt @xcite @xcite or better scoring of the output from the model @xcite . these studies directly predict target tokens to determine the prediction for an end task. despite promising results, they can be unstable with high variance across different verbalizers (text expression for labels) and seeds, and the worst-case performance is often close to random @xcite . in this paper, we introduce alternative channel models for prompted few-shot text classification with large language models, inspired by noisy channel models in machine translation @xcite @xcite and their extensions to other tasks @xcite . unlike direct models that compute the conditional probability of the label token given the input, channel models compute the conditional probability of the input given the output (figure 1 ). intuitively, channel models are required to explain every word in the input, potentially amplifying training signals in the low data regime. we study the impact of channel models for language model prompting where the parameters of the language model are frozen. in particular, we compare channel models with their direct counterparts for (1) demonstration methods, either concatenation-based @xcite or our proposed, ensemble-based (section 4.1.3), and (2) prompt tuning @xcite . our experiments on eleven text classification datasets show that channel models outperform their direct counterparts by a large margin. we attribute the strong performance of channel models to their stability: they have lower variance and significantly higher worst-case accuracy then their direct counterparts over different verbalizers and seeds. we additionally find a direct model with head tuning-tuning the lm head while freezing other parameters-is surprisingly effective, often outperforming direct models with other forms of tuning. while different methods are preferred given different conditions, the channel model with prompt tuning (denoted as channel prompt tuning) significantly outperforms all direct baselines when @xcite the training data is imbalanced, or (2) generalization to unseen labels is required. in summary, our contributions are three-fold: ## formulation we focus on text classification tasks. the goal is to learn a task function f : x -→ c, where x is the direct computes distributions of labels c @xmath1: p (c i |x). this is the most widely used method in modern neural networks. direct++ is a stronger direct model that computes p (c i |x) p (c i |null) instead of p (c i |x), following the method from @xcite and the nonparametric method from @xcite . this approach is motivated by the fact that language models can be poorly calibrated and suffer from competition between different strings with the same meaning. this approach is used for the demonstration methods in section 4. ## method we explore direct and channel models using a causal language model (lm) p lm that gives the conditional probability of the text y when followed by x. more precisely, given the text @xmath2...x tx and @xmath3...y ty (x 1 ...x tx , y 1 ...y @xmath4, where v is the vocabulary set), p lm (y|x) indicates π ty @xmath5...x tx y 1 ...y t -1 ). 2 when learning a task function f : x -→ c, we also assume a pre-defined verbalizer v : c -→ x which maps each label into a natural language expression. for example, if the task is sentiment analysis with @xmath6, c -}, an example input text x would be "a three-hour cinema master class" and an example v would have v(c + ) ="it was great" and v(c -) ="it was terrible". in a few-shot setup, we are also given a set of k training examples we are interested in methods where there are no trainable parameters (section 4.1) or the number of trainable parameters is very small, typically less than 0.01% of the total (section 4.2). this follows prior observations that updating and saving a large number of parameters for every task is expensive and often infeasible @xcite @xcite . ## experimental results this section reports results from demonstration methods (section 6.1), tuning methods (section 6.2) and ablations (section 6.3). discussion is provided in section 7. ## discussion & conclusion in this work, we introduced a noisy channel approach for few-shot text classification through lm prompting, where we either provide demonstrations to the lm or tune the prompt embeddings in the continuous space. our experiments on eleven datasets show that channel models significantly outperform their direct counterparts, mainly because of their stability, i.e., lower variance and better worst-case accuracy. we also found that direct head tuning is more competitive than previously thought, and different methods are preferred given different conditions. specifically, channel prompt tuning is preferred in the following scenarios. k is small channel prompt tuning is more competitive when there are fewer training examples. we hypothesize two reasons: (1) channel models are more stable (i.e., achieve low variance and high worst-case accuracy), unlike direct models that are highly unstable with small k @xcite @xcite . (2) channel models provide more signals by requiring the model to explain the input word-by-word (as claimed in @xcite ) which is beneficial in the low data regime.