On Training Instance Selection for Few-Shot Neural Text Generation (2021)

Abstract: Large-scale pretrained language models have led to dramatic improvements in text generation. Impressive performance can be achieved by finetuning only on a small number of instances (few-shot setting). Nonetheless, almost all previous work simply applies random sampling to select the few-shot training instances. Little to no attention has been paid to the selection strategies and how they would affect model performance. In this work, we present a study on training instance selection in few-shot neural text generation. The selection decision is made based only on the unlabeled data so as to identify the most worthwhile data points that should be annotated under some budget of labeling cost. Based on the intuition that the few-shot training instances should be diverse and representative of the entire data distribution, we propose a simple selection strategy with K-means clustering. We show that even with the naive clustering-based approach, the generation models consistently outperform random sampling on three text generation tasks: data-to-text generation, document summarization and question generation. The code and training data are made available. We hope that this work will call for more attention on this largely unexplored area.

## introduction few-shot text generation is an important research topic since obtaining large-scale training data for each individual downstream task is prohibitively expensive. recently, pretraining large neural networks with a language modeling objective has led to significant improvement across different fewshot text generation tasks @xcite and many techniques are proposed based on them @xcite schütze, 2020a; @xcite @xcite @xcite . however, all the previous works simulate the few-shot scenario by randomly sampling a subset from the full training data. little to no attention has been paid to the selection strategies. in this work, we present a preliminary study at searching for an optimal strategy to select the fewshot training instances. studying the selection strategy is motivated by two rationales. first, random sampling leads to a large variance of model performance @xcite . yet current works sample their own training data which makes it difficult to compare across different models. one can then not be sure whether an improved performance can be really ascribed to the model or the randomness of sampling. using a stable selection strategy to find the most informative few-shot instances can provide a fair platform and better benchmark different few-shot generative models. second, in practical applications, e.g. document summarization, the training data is usually obtained by manually annotating the summaries for some selected documents. in figure 1 , we illustrate the typical training scenario for text generation where the annotation budget only allows annotating a limited amount of data. studying the optimal selection strategy can help make the most use of our annotation budget. specifically, we focus on the label-free setting where the selection can only condition on the unannotated data. although leveraging the reference text may benefit the selection strategy, it conflicts with the realistic setting where we need to first select the data then get its annotated reference text. the selection task resembles the theme of active learning @xcite , where the model keeps identifying the most informative instances to get labeled. existing active learning approaches can be roughly divided to uncertainty-based sampling and representative sampling @xcite . uncertainty-based sampling select samples that maximally reduce the uncertainty of the model @xcite . this, however, requires a well-trained model with decent confidence score estimations in order to perform well. therefore, in this paper, we opt for the representative-sampling where the selected training instances are expected to be dissimilar to each other and representative enough to cover all important patterns in the whole data distribution @xcite . this naturally matches the objectives of k-means clustering which minimizes the within-cluster variances while maximizing the between-cluster variances to encourage the diversity and representativeness of each cluster @xcite . as has been shown in image classification tasks, data points closer to the cluster centroids are usually most important, while other faraway points can even be safely removed without hurting model performance @xcite . inspired by this, we propose a simple selection strategy which first clusters the whole unlabeled dataset with the k-means algorithm, and then from each cluster, selects the data point that is closest to the cluster centroid. we conduct experiments on three popular text generation tasks: data-to-text, document summarization and question generation. the proposed selection strategy consistently outperforms random sampling and exhibits much smaller variance. contribution. we present a preliminary study on training instance selection for few-shot text generation and propose a selection strategy based on k-means clustering. the proposed method shows consistent superior performance over random sampling, which can be used to make most use of the annotation budget in practical applications. meanwhile, the selected training instances can serve as a better benchmark for few-shot text generation since they are not biased towards specific generative methods and do not have the large variance issue as found in random sampling. we further perform a set of ablation studies to analyze what contributes to a good selection. our findings can also benefit research in active learning @xcite since identifying the most informative training instances is a critical step before collecting more annotations through active learning. ## problem formulation following the training scenario shown in figure 1 , we denote the unlabeled data as u 1 , u 2 , . . . , u n where n is the data size. depending on the downstream task, "data" can mean unlabeled structured data, documents and paragraphs respectively in the context of data-to-text, document summarization and question generation. we will select k instances from the whole unlabeled dataset, annotate them with reference text, and then train a neural generative model based on the annotated data. k is defined based on the annotation budget. in this work, since we focus on the few-shot scenario, k is set to be small (≤ 100). the goal is to find the most representative k instances that can lead to the optimal performance when trained on them. ## selection by k-means clustering the general idea of our proposed method is to first split the whole unlabeled data into k clusters, then select one data point from each cluster. specifically, we first map each data point into a vector, then cluster the vectors with the k-means algorithm. the objective is sum of the squared errors (sse), which is also called cluster inertia: ## experiments we perform our experiments on the following three representative datasets which cover three different text generation tasks: for all experiments, we finetune the opensourced bart model @xcite as our generative model. bart is pretrained with a denoising autoencoder objective on large amount of text data and has been the state-of-the-arts for many text generation tasks. to extract vectors used for clustering, we finetune the bart model with its original self-supervised objective on the unlabeled data, then apply mean pooling over the last hidden states of the encoder. in the later sections, we will first compare the model performance based on our proposed selection strategy and random sampling, then analyze the variance of them. finally, we perform an ablation study to see the effects of in-cluster selection and embedding choices. ## conclusion in this work, we target at the unexplored problem of training instance selection for few-shot text generation. we show that random sampling can lead to large variance and suboptimal performance.