A Universal Discriminator for Zero-Shot Generalization (2023)

Abstract: Generative modeling has been the dominant approach for large-scale pretraining and zero-shot generalization. In this work, we challenge this convention by showing that discriminative approaches perform substantially better than generative ones on a large number of NLP tasks. Technically, we train a single discriminator to predict whether a text sample comes from the true data distribution, similar to GANs. Since many NLP tasks can be formulated as selecting from a few options, we use this discriminator to predict the concatenation of input and which option has the highest probability of coming from the true data distribution. This simple formulation achieves state-of-the-art zero-shot results on the T0 benchmark, outperforming T0 by 16.0%, 7.8%, and 11.5% respectively on different scales. In the finetuning setting, our approach also achieves new state-of-the-art results on a wide range of NLP tasks, with only 1/4 parameters of previous methods. Meanwhile, our approach requires minimal prompting efforts, which largely improves robustness and is essential for real-world applications. Furthermore, we also jointly train a generalized UD in combination with generative tasks, which maintains its advantage on discriminative tasks and simultaneously works on generative tasks.

## introduction generative modeling has been the dominant approach for large-scale pretraining and zeroshot generalization @xcite @xcite . combined with prompts @xcite , most of the natural language processing (nlp) tasks can be formulated into the fill-in-the-blank format and perform generative language modeling. based on the unified generative formulation, pretrained models such as @xcite , bert @xcite figure 1 : average zero-shot performance over 11 zeroshot tasks for our universal discriminator and t0 @xcite . our universal discriminator significantly outperforms t0 across three different scales. more recent work @xcite proposed to further pretrain a generative t5 @xcite with multitask prompted datasets and has substantially enhanced the performance of zeroshot generalization. in contrast, methods based on discriminative modeling @xcite have not been able to achieve state-of-the-art performance on zero-shot learning. the adoption of discriminative approaches for zero-shot learning has been limited in the literature. in this work, we challenge the convention of zero-shot learning and propose to study and improve discriminative approaches. this is motivated by the fact that many nlp tasks can be framed as selecting from a few options; e.g., telling whether sentence a entails sentence b, or predicting which answer is correct for a given question. we call these tasks discriminative tasks. as we will discuss in later sections, a significant portion of nlp tasks is in fact discriminative tasks. we hypothesize that discriminative approaches perform better for discriminative tasks. to verify the hypothesis, we propose the universal discriminator (ud), which substantially improves zero-shot generalization over the previous generative state-of-the-art (sota) @xcite , as figure 1 shows. the main idea is to train a single discriminator to predict whether a text sample comes from the true data distribution of natural language, similar to gans @xcite . given a set of training tasks with labeled data, we construct a dataset with positive and negative examples, where positive ones are indistribution natural language samples and negative ones are out-of-distribution. there are two major types of discriminative tasks. the first type is tasks with multiple options, such as multi-choice question answering and news classification. we fill the options into the sentences and the ones with correct options are considered positive samples. the second type is tasks with yes/no options, which can be formulated as a binary discrimination problem itself. for example, natural language inference aims to predict whether a premise entails a hypothesis. in this case, we use a prompt to concatenate the premise a and the hypothesis b into a sentence "premise: a. hypothesis: b." if entailment holds, this sample is treated as positive in-distribution samples and otherwise negative out-of-distribution ones. for the performance of zero-shot generalization, our approach achieves new state-of-the-art on the t0 benchmark, outperforming t0 by 16.0%, 7.8%, and 11.5% respectively on different scales. ud also achieves state-of-the-art performance on a wide range of supervised nlp tasks, using only 1/4 parameters of previous methods. compared with the previous generative prompt-based methods, our universal discriminator requires minimal prompting, which is simple, robust, and applicable in real-world scenarios. in addition, we also generalize ud to a larger scope of tasks, such that ud can perform discriminative and generative tasks at the same time. specifically, we extend ud to the encoder-decoder architecture for training on generative tasks, and restrict the model's prediction on "yes"/"no" tokens for jointly training discriminative tasks. results prove that generalized ud maintains ud's advantages on discriminative tasks and achieves comparable results on generative tasks (see § 3.4). ## approach previous works @xcite have shown that prompted multi-task training can greatly improve zero-shot performance on unseen tasks. one intuitive reason behind the validity of this improvement is that all the nlp tasks share a common ability that allows lms to solve unseen tasks based on the data from other training tasks. to test this idea and even enhance zero-shot generalization, a direct way is explicitly defining what this "common ability" is. here, we define this "common ability" by designing a new general task of "discriminating whether a text sample comes from the true data distribution of natural language". we will first formulate the learning problem ( § 3.1), and then define the concept discriminative tasks ( § 3.2), followed by describing how we transform discriminative tasks into our shared formulation. in § 3.3 and § 3.4, we will study our ud, respectively on discriminative tasks and on a generalized scope of both discriminative and generative tasks. ## conclusions universal discriminator is a discriminating model for predicting whether a sample comes from the true data distribution, which is a new formulation for all discriminative nlp tasks. experiments show that ud sets the new state-of-the-art for zero-shot generalization on many benchmarks. ud is highperforming with minimal prompting, and thus is more robust and applicable in practice. a generalized ud can also solve generative tasks at the same time which keeps ud's advantage on discriminative tasks and has comparable performance on generative tasks.