The Web Can Be Your Oyster for Improving Language Models (2023)

Abstract: Pretrained language models (PLMs) encode a large amount of world knowledge. However, as such knowledge is frozen at the time of model training, the models become static and limited by the training data at that time. In order to further improve the capacity of PLMs for knowledge-intensive tasks, we consider augmenting PLMs with the large-scale web using search engine. Unlike previous augmentation sources (e.g., Wikipedia data dump), the web provides broader, more comprehensive and constantly updated information. In this paper, we present a web-augmented PLM – UniWeb, which is trained over 16 knowledge-intensive tasks in a unified text-to-text format. Instead of simply using the retrieved contents from web, our approach has made two major improvements. Firstly, we propose an adaptive search engine assisted learning method that can self-evaluate the confidence level of PLM’s predictions, and adaptively determine when to refer to the web for more data, which can avoid useless or noisy augmentation from web. Secondly, we design a pretraining task, i.e., continual knowledge learning, based on salient spans prediction, to reduce the discrepancy between the encoded and retrieved knowledge. Experiments on a wide range of knowledge-intensive tasks show that our model significantly outperforms previous retrieval-augmented methods.

## introduction with large-scale neural networks, pretrained language models (plms) @xcite can encode a large amount of world knowledge, showing phenomenal capability in knowledge-intensive tasks such as fact checking and open-domain question answering (qa). however, this capacity is naturally limited by the information contained in pretraining or finetuning recently, by drawing the idea from semiparametric approaches @xcite @xcite , retrieval-augmented approaches have been proposed to equip plms with the ability to directly access an external database. as a major knowledge resource, wikipedia has been widely used in previous work. while being highly accurate and wellstructured, wikipedia only covers limited information, both in scope and in time. besides, even for the topics that wikipedia covers, grounding plms' decisions on a single source of knowledge may create biases @xcite . considering these issues, it is time to look beyond wikipedia (or similar single-source databases) and access more broader, in-depth, and up-to-date knowledge from more sources. inspired by @xcite , we select the web as the retrieval resource for enlarging the knowledge capacity of plms. to motivate our approach, in this paper, we aim to capitalize on the web as a source of up-to-date and comprehensive knowledge to solve a wide range of knowledge-intensive tasks. unlike previous web-augmented studies @xcite that mostly focus on single tasks, we seek to develop a unified framework to integrate the use of the web in plms for multi-task solving. although the idea of leveraging the web for improving plms is appealing, it is non-trivial to develop an effective solution. first, plms do not always need external evidence for task solving, especially considering the fact that the web contains noisy, biased, or harmful information @xcite . simply retrieving knowledge without considering the example difficulty and plms' own capabilities may steer models towards unexpected outputs. second, plms are usually pretrained at an earlier time on a limited corpus, leading to a discrepancy between the encoded knowledge and the retrieved knowledge (i.e., web contents). therefore, we need more principled approaches to properly integrating the new knowledge into plms. to address the above issues, we present a webaugmented plms, uniweb, to improve the capacity in knowledge-intensive tasks. instead of using neural network-based retriever, we employ a commercial search engine (i.e., google search) to obtain high-quality and comprehensive retrieval results from the web. based on this idea, we make two major technical contributions. first, we propose a search engine assisted learning method that can selectively query the web only when plm is unconfident in its predictions. for this purpose, we design a self-evaluation mechanism to estimate the confidence level of plms on the task exam-ples. secondly, to reduce the discrepancy between the encoded and retrieved knowledge, we design a pretraining task, continual knowledge learning, to integrate the retrieved knowledge into plms by predicting the salient masked spans in web documents. to train the uniweb model, we convert different knowledge-intensive tasks into a unified text-to-text format, and conduct supervised multitask training over 16 tasks across seven categories. to the best of our knowledge, our model is the first unified web-augmented plm for a wide range of knowledge-intensive tasks. extensive experiments show that plms can significantly benefit from such an approach and a single unified plm (uniweb) is able to achieve (near) state-of-the-art performance on all 16 tasks. ## approach our proposed web-augmented plm, uniweb, is depicted in figure 1 . we first transform knowledgeintensive tasks into a unified text-to-text paradigm and consider the web as a general form of knowledge source. based on the retrieved knowledge, we further design two training objectives to build our model. in the next sections, we will describe our method in detail. ## experiments in this section, we detail the experimental setup and then highlight the main observations of our results. ## conclusion this paper presented a unified web-augmented framework for a wide range of knowledge-intensive tasks, called uniweb. we convert 16 tasks into a text-to-text generation task for training. we propose a search engine assisted learning method to selectively retrieve documents from the web through google search. furthermore, to reduce the discrepancy between the encoded and retrieved knowledge, we design a pretraining task, i.e., continual knowledge learning, to integrate the retrieved knowledge into llms. experiments on 16 tasks show the effectiveness of our web-augmented model compared to previous retrieval-augmented models. in future work, we will investigate the effect of web content in detail and consider applying our model to more types of downstream tasks.