Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data (2022)

Abstract: Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA. Our code is released, https://github.com/microsoft/REINA .

## introduction in natural language processing, retrieval-based methods work by fetching textual information related to the input from large corpora. the model then takes both the input and retrieved results as input to generate results. this can often improve the performance as the model is exposed to related knowledge not present in the input. as a result, retrieval-based methods have been successfully applied in many tasks such as open-domain question answering @xcite , language modeling @xcite and machine translation @xcite . however, these methods require building an index of large-scale corpus, and the retrieval leads to a significant computational burden. for example, the knn-mt model for machine translation has a generation speed two orders of magnitude slower than traditional mt models @xcite . on the other hand, in the supervised learning setting, the text most similar in distribution to the figure 1 : reina pipeline of model training/inference with retrieval from training data. filter only happens at training, as the same training sample will be retrieved from the index. for each instance, we concatenate the input with the retrieved content, i.e., data and/or labels, for model training and inference. we note that our method is similar to recent works in prompt learning @xcite , where a set of labeled data is carefully chosen based on the input and then included in the prompt for few-shot learning. our method also bears a resemblance to non-parametric instancebased learning @xcite . however, a critical difference is that we focus on the supervised learning setting, where the model parameters are fine-tuned to learn from given examples to achieve much higher performance than few-shot learning or non-parametric methods. in the experiments, we evaluate our method on four popular types of nlp tasks: summarization, language modeling, machine translation, and question answering. we find that i) after integrating reina, we can achieve significantly better performance on these tasks, 11 datasets in total, than models with different pre-trained models; ii) reina leads to sota performance on the datasets of xsum, commonsenseqa (leaderboard no.1), and bigpatent; iii) reina can scale up more easily by leveraging more labeled data from other datasets via retrieval, outperforming baselines which is trained on the same set of data. the effectiveness of our approach on summarization tasks provides insights into the core of supervised learning. even with hundreds of millions of parameters, a model cannot memorize all the patterns in the training data. thus, recapturing related training data as a side-by-side reminder can explicitly provide needed information to enhance the model's performance at inference. it also points out that instead of building models of ever increasing sizes, we can make a decent-size model output high-quality results by leveraging those training data that resemble the instance at hand. this can significantly reduce the computational cost while achieving a similar or better performance of a megasized model. ## related work retrieval-based methods even a pre-trained model as large as @xcite cannot remember everything, and it is important to leverage information retrieval to collect external knowledge to solve different nlp tasks. there are two types of representations for retriever: bag-ofword (bow) based sparse representation @xcite and dense representation from neural networks @xcite . for the sparse representation, as the method is based on bow and usually rule-based score, such as bm25, is used for ranking, it can be easily adapted to a general large-scale search. this method has also been widely explored to solve open domain question answering @xcite @xcite and machine translation @xcite . dense representation based retrieval (dpr) @xcite is the most widely explored area in recent years. dense representations come from encoders, such as transformer, trained with task-specific data. and these methods can achieve better recall performance than sparse representation on different tasks, such as open domain question answering @xcite @xcite , knowledge-grounded generation @xcite , and machine translation @xcite . one drawback of dpr is that it cannot process longer documents, usually less than 128 tokens @xcite . another drawback is that it needs parallel data for model training on specific tasks. considering the generalization and efficiency of sparse representation, in this paper, we use bm25 score @xcite to retrieve from the training data, and our method is more flexible with no requirement of parallel data for model training. compared to nonparametric systems guided by search engine @xcite , our proposed method is based on supervised learning and is more general. @xcite is related to our work by retrieving related questions from pre-built largescale question-answer pairs. however, our method doesn't need addition data augmentation method, and we have successfully applied reina to a wide range of downstream tasks, including summarization, question answering, machine translation and language modeling. prompt engineering with the success of largescale language models (brown et al., 2020) on fewshot learning, prompt engineering comes to be a popular research direction. the idea is to prepend several labeled instances to the input sequence and then conduct the classification or generation. @xcite proposes to prepend the most related labeled data as prompt to help fewshot inference. li and liang (2021) optimizes the prompt in continuous space. motivated by these works where a good labeled prompt can help fewshot learning, we also prepend/append the most similar labeled training data for all the data in training, validation, and test set. however, different from prompt learning, we focus on supervised learning settings. ## model in this section, we will introduce the details of our proposed method. briefly, given the input, we first retrieve the most matched instances with labels for language modeling, we prepend the retrieved data to the query data, and append the retrieved data to the query for all the other tasks. after concatenation, we will directly feed them into transformers, either seq2seq or encoder-only frameworks, for text generation and answering selection. as we focus on the question answering tasks requiring commonsense reasoning, we have another version of index integrating knowledge graph for more precise retrieval. k: external knowledge from conceptnet and wiktionary, src: source language, tgt: target language. from the training data. we then concatenate them with the input sequence to feed into the model for generating the output. an overview of the whole method is shown in figure 2 . ## experiment in this section, we will introduce more details about experiments and the corresponding analysis. ## conclusion in this paper, we propose a simple and effective method to fully make use training dataset. our