Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective (2022)

Abstract: We propose a new paradigm for zero-shot learners that is format agnostic, i.e., it is compatible with any format and applicable to a list of language tasks, such as text classification, commonsense reasoning, coreference resolution, and sentiment analysis. Zero-shot learning aims to train a model on a given task such that it can address new learning tasks without any additional training. Our approach converts zero-shot learning into multiple-choice tasks, avoiding problems in commonly used large-scale generative models such as FLAN. It not only adds generalization ability to models but also significantly reduces the number of parameters. Our method shares the merits of efficient training and deployment. Our approach shows state-of-the-art performance on several benchmarks and produces satisfactory results on tasks such as natural language inference and text classification. Our model achieves this success with only 235M parameters, which is substantially smaller than state-of-the-art models with billions of parameters. The code and pre-trained models are available at https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen/examples/unimc .

## introduction remarkable advances in large-scale language models have brought substantial improvements in a wide variety of tasks such as text classification, natural language inference and commonsense reasoning @xcite . this progress brings opportunities to zero-shot learning (zsl) @xcite , which aims to predict labels on datasets from novel domains. most solutions can be framed in the prompt tuning framework that activate specific parameters in plm @xcite * equal contribution. despite their success, these frameworks suffer from their inherent problems, and thus limit their potential in zero-shot learners. firstly, promptrelated models have an extremely large number of parameters, e.g., gpt-3 has 175b, flan has 137b and palm @xcite has 540b. one immediate problem is that these models are often hard to be trained, making the deployment and consumption difficult. secondly, manual processing is required when addressing zero-shot problems. for instance, t0 builds 2, 073 prompts to handle more than 170 tasks @xcite . lastly, existing models employ a single direction paradigm, either auto-regressive models or sequence-to-sequence, resulting in inadequate usage of information from both directions. as an example, pmlm tries to implement a zero-shot learner, which is shown in figure 1 @xcite . note that recent work @xcite state that pmlm is more suitable than plm for natural language understanding (nlu) tasks. however, it has to be fine-tuned on the task-specific samples to initialize the classifier instead of randomly initializing the classifier. therefore, the ability of pmlm is limited when dealing with zeroshot scenarios. to address the aforementioned problems, we introduce a light-weight framework, called unified multiple choice model (unimc), proposing a novel mc tuning. the proposed mc tuning has the following advantages: i) parameter updating only happens in the mc training phase, and ii) facilitating the deployment. to reduce the manual processing, we only formulate one candidate option prompt format and one question prompt format. note that we also consider the case without any question prompt format. under this setting, we can treat labels as options rather than building verbalizer maps and providing its text information to the models as before. we therefore can learn the information from labels directly. to this end, we convert the problematic classifiers to options. one immediate question is how to choose an option efficiently and unambiguously. therefore, as shown in section 3.2, we develop an option-mask tokens [o-mask] to predict "yes" or "no" before each option. a two-step process is introduced to output the desired options. first, similar to masked language modeling (mlm) @xcite , we conduct option mlm (o-mlm) to recover the "yes" or "no" for each option. next, we propose an option prediction (op) method to compute proper options. with extensive experiments on multiple challenging benchmarks, we demonstrate that our approach's performance outperforms state-of-the-art baselines, while reducing the model size with two orders, as shown in figure 2 . this success suggests the potential of leveraging unimc in large datasets. our contributions are as follows. our proposed method has up to 48% improvement on dbpedia over sota baselines that have a few hundred times larger than our model. ## approaches in this section, we outline the proposed framework, i.e., unimc, and provide the training and inference approaches in detail. ## conclusions in this paper, we introduce a new zero-shot paradigm called mc tuning. this adds flexibility and generalization ability to zero-shot learners. we propose o-mlm and op in both mc training and zero-shot phase, aiming to capture information from both directions. our unimc achieves better performances over sota models that a few hundred times larger than our model. our experiments demonstrate the effectiveness and generalization ability of unimc on zero-shot tasks. in future work, we will extend unimc to few-shot scenarios.