Plug-and-Play Document Modules for Pre-trained Models (2023)

Abstract: Large-scale pre-trained models (PTMs) have been widely used in document-oriented NLP tasks, such as question answering. However, the encoding-task coupling requirement results in the repeated encoding of the same documents for different tasks and queries, which is highly computationally inefficient. To this end, we target to decouple document encoding from downstream tasks, and propose to represent each document as a plug-and-play document module, i.e., a document plugin, for PTMs (PlugD). By inserting document plugins into the backbone PTM for downstream tasks, we can encode a document one time to handle multiple tasks, which is more efficient than conventional encoding-task coupling methods that simultaneously encode documents and input queries using task-specific encoders. Extensive experiments on 8 datasets of 4 typical NLP tasks show that PlugD enables models to encode documents once and for all across different scenarios. Especially, PlugD can save 69% computational costs while achieving comparable performance to state-of-the-art encoding-task coupling methods. Additionally, we show that PlugD can serve as an effective post-processing way to inject knowledge into task-specific models, improving model performance without any additional model training. Our code and checkpoints can be found in https://github.com/thunlp/Document-Plugin .

## introduction in recent years, large-scale pre-trained models (ptms) @xcite have been widely adopted and achieved breakthrough performance for document-oriented nlp tasks, such as question answering. however, due to the tight coupling of document encoding and concrete tasks, ptms have to dynamically generate document representations according to specific tasks and queries, leading to the repeated encoding of the same documents in different applications. for example, wikipedia documents are commonly used in various knowledge-intensive tasks such as question answering @xcite , fact verification @xcite , and dialogue generation @xcite . in this case, existing methods separately encode one document for each task or even for each input query (e.g., a question for question answering, a claim for fact verification), making them highly computationally inefficient. to this end, it raises a natural question: can we decouple document encoding from concrete tasks, encoding documents only once and with guaranteed transferability across multiple tasks? for this question, we propose a novel framework based on ptms to decouple document encoding from tasks, named plugd. specifically, plugd incorporates plug-and-play modules to store document information and utilizes a ptm backbone to capture information from plugins for task reasoning. as shown in figure 1 , documents are encoded into pluggable plugins once and for all before task adaptation. the semantics and knowledge of documents can be injected into task-specific models by plugging document plugins. during task reasoning, the task-specific models can activate the information encoded in the document plugins to handle the input queries. in this way, plugd can decouple the document encoding from downstream task reasoning and reduce the computation costs. for representing documents as pluggable modules, there are two main challenges: (1) plugin learning: the document plugins must be effective for various downstream tasks, requiring them to contain sufficient semantics and knowledge. as for plugin learning, we adopt a selfsupervised method, which requires document plugins to provide sufficient knowledge for the ptm to make predictions. specifically, for each document, we first randomly select parts of sentences as a query and use the remaining sentences as context to learn plugins. then, after encoding the context into plugins, the model is required to recover the masked recurring spans or generate the next sentences for the query based on the plugin knowledge. as for plugin utilization, we propose two strategies to utilize document plugins for downstream tasks: plugging during tuning and plugging after tuning foot_0 . for plugging during tuning, document plugins are utilized in both tuning and inference stages. task data and document plugins are combined together to train task-specific models. for plugging after tuning, document plugins are only utilized in the inference stage to provide external knowledge. document plugins are adopted as a post-processing way to inject knowledge into task-specific models without additional training. to verify the effectiveness of our plug-and-play framework, we adopt wikipedia as our document collection and conduct experiments on 8 datasets of 4 typical knowledge-intensive nlp tasks. the results show that we can generate document plugins once and successfully adapt plugins to various downstream tasks. compared to competitive baselines that encode documents and task-specific inputs simultaneously, our plugin-based method can save 69% computational costs with comparable performance. besides, utilizing document plugins works as an effective post-processing approach to introducing the knowledge of documents into task-specific models and achieving performance improvements without model training. we argue that with the current trend of increasing the model size of ptms, decoupling document encoding from concrete tasks like plugd can be a promising direction that enables large ptms to effectively and efficiently serve diverse downstream tasks. ## methodology in this section, we will first present the paradigm description and the overall framework of plugd then we introduce the self-supervised plugin learning method to make document plugins contain sufficient semantics and two strategies about how to utilize document modules. ## conclusion in this paper, we explore a new paradigm, which aims to represent documents as pluggable modules for ptms. in this setting, we can get rid of encoding the same document multiple times for different tasks. the extensive experiments prove that our proposed plugd can significantly reduce the computational cost and effectively inject document knowledge into ptms to improve performance. in the future, we will explore more effective plugin learning tasks and further attempt to represent knowledge graphs, and figures as plugins to provide knowledge for ptms.