The S elect G en Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generation (2021)

Abstract: We propose a shared task on training instance selection for few-shot neural text generation. Large-scale pretrained language models have led to dramatic improvements in few-shot text generation. Nonetheless, almost all previous work simply applies random sampling to select the few-shot training instances. Little to no attention has been paid to the selection strategies and how they would affect model performance. Studying the selection strategy can help us (1) make the most use of our annotation budget in downstream tasks and (2) better benchmark few-shot text generative models. We welcome submissions that present their selection strategies and the effects on the generation quality.

## introduction few-shot text generation is an important research topic since obtaining large-scale training data for each individual downstream task is prohibitively expensive. recently, pretraining large neural networks with a language modeling objective has led to significant improvements across different fewshot text generation tasks @xcite and many techniques are proposed based on them @xcite @xcite @xcite @xcite . however, all previous works simulate the few-shot scenario by randomly sampling a subset from the full training data. little to no attention has been paid to the selection strategies. the goal of the proposal is to call for innovative ideas on searching for an optimal strategy to select the few-shot training instances, as well as a comprehensive analysis of how the selection strategy would affect the model performance. the study of selection strategies is motivated by two rationales: * equal contribution. x.shen is now at amazon alexa ai. the annotation budget only allows selecting k data for annotating the reference text. we aim to identify the k most representative instances that, when annotated and trained on them, leads to a best model performance. first, random sampling leads to a large variance of model performance @xcite . yet current works sample their own training data which makes it difficult to compare across different models. one can then not be sure whether an improved performance can be really ascribed to the model or the randomness of sampling. using a stable selection strategy to find the most informative few-shot instances can provide a fair platform and better benchmark different few-shot generative models. second, in practical applications, e.g. document summarization, the training data is usually obtained by manually annotating the summaries for some selected documents. in figure 1 , we illustrate the typical training scenario for text generation where the annotation budget only allows annotating a limited amount of data. studying the optimal selection strategy can help make the most use of our annotation budget. specifically, we focus on the label-free setting where the selection can only condition on the unannotated data. although leveraging the reference text may benefit the selection strategy, it conflicts with the realistic setting where we need to first select the data then get its annotated reference text. the selection task resembles the theme of active learning @xcite @xcite , where the model keeps identifying the most informative instances to get labeled. we can consider the task as a starting step before applying active learning, after which more annotations can be continuously collected to further improve the model. ## task description following the training scenario shown in figure 1 , we denote the unlabeled data as u 1 , u 2 , . . . , u n where n is the data size. depending on the downstream text generation task, "data" can mean different types of inputs like unlabeled structured data, documents and paragraphs respectively in the context of data-to-text, document summarization and question generation. we will select k instances from the whole unlabeled dataset, annotate them with reference text, and then train a neural generative model based on the annotated data. k is defined based on the annotation budget. in this work, since we focus on the few-shot scenario, k is set to be small @xcite . the goal is to find the most representative k instances that can lead to the optimal performance when being annotated and trained on them. ## evaluation the final evaluation will be conducted on the following two settings: we will provide scripts for the automatic evaluation. the human evaluation will be conducted after all submissions are received under the same platform and metrics. ## conclusion in this proposal, we target at the problem of training instance selection for few-shot text generation. current research simply applies random sampling which has a large selection variance and can lead to suboptimal performance. the main goal of the task is to call for more attention on this largely under-explored problem, gather innovative ideas on proposing selection algorithms and provide a fair platform for comparison. we believe our shared task can be an important supplement to the study of few-shot text generation, where most works focus solely on the generative algorithm while neglecting the training instance selection. selection strategies proposed in this task can be used to better benchmark model performances for few-shot text generation. importantly, the task was inspired by realistic industrial settings and requirements and will hopefully bene-fit multiple areas of nlp research including humanin-the-loop learning and other active learning based research, where the resource and time constraints calls for the task to be performed.