

Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that **RE**trieving from the tra**IN**ing dat**A** (**REINA**) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA. Our code is released.[^1]

=1

Introduction
============

In natural language processing, retrieval-based methods work by fetching textual information related to the input from large corpora. The model then takes both the input and retrieved results as input to generate results. This can often improve the performance as the model is exposed to related knowledge not present in the input. As a result, retrieval-based methods have been successfully applied in many tasks such as open-domain question answering @chen2017reading, language modeling @guu2018generating [@khandelwal2019generalization] and machine translation @khandelwal2020nearest. However, these methods require building an index of large-scale corpus, and the retrieval leads to a significant computational burden. For example, the kNN-MT model for machine translation has a generation speed two orders of magnitude slower than traditional MT models @khandelwal2020nearest.

On the other hand, in the supervised learning setting, the text most similar in distribution to the data in inference is the training data. Thus, we explore whether retrieving from the training data, which is usually much smaller than a large-scale corpus, can help improve the performance. Specifically, we first index a task's labeled training data as input-label pairs. Then, during both training and testing, we retrieve the input-label pairs most similar to the current input[^2]. Finally, we concatenate the retrieved training pairs with the input and feed it into the model. An overview of our method is shown in Figure \[fig:pipe\].

We note that our method is similar to recent works in prompt learning @brown2020language [@liu2021makes], where a set of labeled data is carefully chosen based on the input and then included in the prompt for few-shot learning. Our method also bears a resemblance to non-parametric instance-based learning @gu2018search. However, a critical difference is that we focus on the supervised learning setting, where the model parameters are fine-tuned to learn from given examples to achieve much higher performance than few-shot learning or non-parametric methods.

In the experiments, we evaluate our method on four popular types of NLP tasks: summarization, language modeling, machine translation, and question answering. We find that $i$) after integrating REINA, we can achieve significantly better performance on these tasks, 11 datasets in total, than models with different pre-trained models; $ii$) <span> REINA </span>leads to SOTA performance on the datasets of XSum, CommonsenseQA (Leaderboard No.1), and BigPatent; $iii$) <span> REINA </span>can scale up more easily by leveraging more labeled data from other datasets via retrieval, outperforming baselines which is trained on the same set of data. $iv$) the results on 3 summarization tasks show that BART-base with <span> REINA </span>rivals BART-large, which contains twice more parameters now.

The effectiveness of our approach on summarization tasks provides insights into the core of supervised learning. Even with hundreds of millions of parameters, a model cannot memorize all the patterns in the training data. Thus, recapturing related training data as a side-by-side reminder can explicitly provide needed information to enhance the model's salazarshelley@example.com. It also points out that instead of building models of ever increasing sizes, we can make a decent-size model output high-quality results by leveraging those training data that resemble the salazarshelley@example.com. This can significantly reduce the computational cost while achieving a similar or better performance of a mega-sized model.

Related Work
============

#### Retrieval-based Methods

Even a pre-trained model as large as GPT-3 @brown2020language cannot remember everything, and it is important to leverage information retrieval to collect external knowledge to solve different NLP tasks. There are two types of representations for retriever: bag-of-word (BOW) based sparse representation @chen2017reading and dense representation from neural networks @karpukhin2020dense.

For the sparse representation, as the method is based on BOW and usually rule-based score, such as BM25, is used for ranking, it can be easily adapted to a general large-scale search. This method has also been widely explored to solve open domain question answering @chen2017reading [@wang2018r; @lin2018denoising] and Machine Translation @gu2018search.

Dense representation based retrieval (DPR) @karpukhin2020dense is the most widely explored area in recent years. Dense representations come from encoders, such as Transformer, trained with task-specific data. And these methods can achieve better recall performance than sparse representation on different tasks, such as open domain question answering @karpukhin2020dense [@guu2020realm; @yu2021kg], knowledge-grounded generation @zhang2021joint, and machine translation @cai2021neural. One drawback of DPR is that it cannot process longer documents, usually less than 128 tokens @karpukhin2020dense. Another drawback is that it needs parallel data for model training on specific tasks.

Considering the generalization and efficiency of sparse representation, in this paper, we use BM25 score @robertson2009probabilistic [@schutze2008introduction] to retrieve from the training data, and our method is more flexible with no requirement of parallel data for model training. Compared to non-parametric systems guided by search engine @gu2018search [@khandelwal2019generalization], our proposed method is based on supervised learning and is more general. @lewis2021paq is related to our work by retrieving related questions from pre-built large-scale question-answer pairs. However, our method doesn't need addition data augmentation method, and we have successfully applied <span> REINA </span>to a wide range of downstream tasks, including summarization, question answering, machine translation and language modeling.

#### Prompt Engineering

With the success of large-scale language models @brown2020language on fewshot learning, prompt engineering comes to be a popular research direction. The idea is to prepend several labeled instances to the input sequence and then conduct the classification or generation. @liu2021makes proposes to prepend the most related labeled data as prompt to help fewshot inference. @li2021prefix optimizes the prompt in continuous space. Motivated by these works where a good labeled prompt can help fewshot learning, we also prepend/append the most similar labeled training data for all the data in training, validation, and test set. However, different from prompt learning, we focus on supervised learning settings.

Conclusion
==========

In this paper, we propose a simple and effective method to fully make use training dataset. Our proposed method is general and can be easily integrated into different models on different tasks. We prove that <span> REINA </span>can effectively improve baseline performance on 11 datasets covering summarization, language modeling, machine translation, and question answering tasks.

[^1]: <https://github.com/microsoft/REINA>

[^2]: During training, we exclude the training instance itself from the retrieval results to avoid data leakage.

{'source': 'arXiv_out/2203'}