Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks (2023)

Abstract: Although large language models have exhibited impressive zero-shot ability, the huge model size generally incurs high cost. Recently, semi-parametric language models, which augment a smaller language model with retrieved related background knowledge, alleviate the need for storing everything into the model parameters. Although existing semi-parametric language models have demonstrated promising language modeling capabilities, it remains unclear whether they can exhibit competitive zero-shot abilities as their fully-parametric counterparts. In this work, we introduce Zemi, a semi-parametric language model for zero-shot task generalization. To our best knowledge, this is the first semi-parametric language model that can demonstrate strong zero-shot performance on a wide range of held-out unseen tasks. We train Zemi with semi-parametric multitask training, which shows significant improvement compared with the parametric multitask training as proposed by T0. Specifically, during both training and inference, Zemi is equipped with a retrieval system based on the unlabeled pretraining corpus of our backbone model. To address the unique challenges from large-scale retrieval, we further propose a novel retrieval-augmentation fusion module that can effectively incorporate noisy retrieved documents. Finally, we show detailed analysis and ablation studies on the key ingredients towards building effective zero-shot semi-parametric language models. Notably, our proposed Zemi_Large model outperforms T0-3B by 16% across seven diverse evaluation tasks while being 3.8x smaller in scale.

## introduction achieving strong generalization ability on unseen tasks while maintaining a reasonably small param-eter size is a long-lasting challenge for natural language processing (nlp) models. although large language models @xcite @xcite @xcite @xcite have shown impressive zero-shot ability on various nlp tasks, the huge model size generally incurs high cost. alternatively, instead of stuffing everything in the model parameters, recent work on semi-parametric language models @xcite @xcite @xcite demonstrated competitive language modeling performance compared with much larger fully-parametric language models. the intuition is to use a relatively small language model as a reasoning module and augment it with a retriever to retrieve related background knowledge, which effectively alleviates the need for increasing the model capacity to align with the growing data size. however, what really makes large language models the focus of attention in the past two years is their strong zero-shot in-context learning abilities. unfortunately, it is still unclear whether semiparametric language models can exhibit similar zero-shot ability on unseen tasks as their fullyparametric counterparts such as t0 @xcite and gpt-3 @xcite . moreover, improvements in language modeling metrics such as perplexity may not guarantee better performance on downstream tasks especially in low-shot settings @xcite . thus, in this work, we aim to investigate this unexplored research question, can semi-parametric language models exhibit strong zero-shot generalization abilities on various downstream tasks? to this end, we introduce zemi, a zero-shot semi-parametric language model. to the best of our knowledge, this is the first semi-parametric language model that shows strong zero-shot perfor-figure 1 : overview of the semi-parametric multitask prompted training. each training and evaluation instance is formatted with unified text-to-text prompt templates @xcite . in this work, we further augment the prompted instances with retrieved passages from a large-scale task-agnostic corpus, c4 @xcite , which is the same unlabeled pretraining corpus used in t5 @xcite and t0 @xcite . an example of the prompted input and the retrieved documents can be found in figure 2 . mance on a wide range of downstream tasks. in order to effectively train zemi, we propose to extend the multitask prompted training @xcite into semi-parametric settings (section 2.1). specifically, during both the training and the inference stage, we augment the prompted instances with retrieved plain text documents. to cover a wider range of unseen tasks, instead of retrieving from specific corpora for certain tasks, such as exploiting wikipedia for open-domain question answering @xcite ; izacard and grave, 2020), we retrieve documents from a large-scale task-agnostic corpus, c4 @xcite (section 2.2). notably, c4 is the unlabeled pre-training corpus of our backbone model @xcite , which means that every document is seen by the model and we do not require any annotated or curated resources. this guarantees fair comparison with the parametric counterpart t0 @xcite . in our preliminary experiments, we find that existing methods @xcite for incorporating retrieved text cannot effectively handle the noise inevitably introduced by retrieving from large-scale corpora. to address this challenge, we propose a novel retrievalaugmentation fusion module that can selectively ignore noisy retrieved text. specifically, we introduce a light-weight perceiver resampler and a gated cross-attention layer @xcite to enforce the model to attend to salient information of each augmentation and gate out noisy ones (section 2.3). we train zemi on eight multiple-choice question answering (qa) tasks (4.5x fewer than t0) and evaluate on a diverse set of seven unseen tasks from five categories (section 3.1). in order to investigate the impact of the retrieval-augmentation, we favor knowledge-intensive tasks over extractive tasks. experimental results show that zemi outperforms both parametric and semi-parametric baselines. notably, zemi large outperforms t0-3b by 16% across seven evaluation tasks while being 3.8x smaller in scale (section 3.2). we further conduct extensive analysis on why zemi works. we show that the source of the improvements comes from the interplay of our proposed retrieval-augmentation fusion architecture along with the semi-parametric multitask training paradigm. finally, we perform in-depth ablation studies on all aspects of our model design including the gated mechanism. to sum up, the main contributions of this paper are threefold: ## conclusion in this work, for the first time, we show that semi-parametric language models have the potential to exhibit strong zero-shot task generalization ability by introducing zemi. through extensive analysis and ablation study, we further demonstrate that the interplay of the proposed retrieval-augmentation fusion and the semiparametric multitask training is essential towards zemi's empirical success. notably, our proposed zemi large model outperforms t0-3b by 16% across seven diverse evaluation tasks while being 3.8x smaller in scale.