MP rompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension (2023)

Abstract: The large language models have achieved superior performance on various natural language tasks. One major drawback of such approaches is they are resource-intensive in fine-tuning new datasets. Soft-prompt tuning presents a resource-efficient solution to fine-tune the pre-trained language models (PLMs) while keeping their weight frozen. Existing soft prompt methods mainly focus on designing the input-independent prompts that steer the model to fit the domain of the new dataset. Those methods often ignore the fine-grained information about the task and context of the text. In this paper, we propose a multi-level prompt tuning (MPrompt) method for machine reading comprehension. It utilizes prompts at task-specific, domain-specific, and context-specific levels to enhance the comprehension of input semantics at different granularities. We also propose an independence constraint to steer each domain-specific prompt to focus on information within its domain to avoid redundancy. Moreover, we present a prompt generator that incorporates context-related knowledge in the prompt generation to enhance contextual relevancy. We conducted extensive experiments on 12 benchmarks of various QA formats and achieved an average improvement of 1.94% over the state-of-the-art methods.

## introduction in recent years, pre-trained language models (plms) have been widely applied in questionanswering tasks @xcite , particularly in machine reading comprehension @xcite , and achieved remarkable success through the pretrain-then-finetune paradigm @xcite . despite the excellent performance, due to the explosive growth of parameter sizes in plms, the fine-tuning paradigm has become resource intensive. recently, soft-prompt tuning has been widely explored as a parameter-efficient approach to addressing the aforementioned issues @xcite . for example, li and liang (2021) proposed prefixtuning, which prepends a sequence of optimizable prefixes to each transformer layer while keeping the parameters of plms frozen. prefix-tuning provides a lightweight alternative to fine-tuning and has achieved comparable performance with fewer trainable parameters. @xcite proposed prompt-tuning, which only prepends optimizable prompt vectors to the input sequence, which used fewer parameters compared to prefix-tuning. @xcite discovered negative tokens in prompttuning that have a detrimental effect on downstream tasks and proposed xprompt to mask these negative tokens, resulting in improved performance. however, the aforementioned methods are inputindependent, i.e., assigning a uniform prompt to all inputs of a given task, which under-utilizes the input semantics for the answer generation in machine reading comprehension. there is a growing trend towards designing input-dependent prompts (a.k.a dynamic prompts) for various tasks @xcite @xcite . for example, @xcite proposed dialogprompt for a dialog system, which dynamically generates prompt vectors according to the input dialogue context. @xcite extracts input-related information from bert @xcite as contextualized prompts for natural language generation @xcite , which improves the relevance between the generated text and the input text. however, to the best of our knowledge, there has been little research exploring input-dependent prompt methods for question-answering tasks, especially for machine reading comprehension. it is challenging to apply input-independent methods to ma-chine reading comprehension where the answer is context-sensitive. to address the above issues, we propose mprompt, a novel multi-level prompt tuning approach for machine reading comprehension. our method utilizes the dataset and the context information to create three levels of prompts: task-specific, domain-specific, and context-specific. the taskspecific prompts are input-independent and generate a prompt based on the tasks. the domainspecific prompts utilize the domain knowledge generated from the dataset while context-specific prompts rely on the input context. these multilevel prompts endow plms with multiple finegrained considerations of input semantics. to further enhance the domain-specific prompts and avoid information redundancy, we propose the independence constraint to steer each prompt to focus on knowledge within the domain rather than cross-domain knowledge. furthermore, we extract context-related knowledge from a small-scale plm, such as t5-small (raffel et al., 2020), and integrate it into the prompt generation process to enrich the context sensitivity of prompts. with the help of these three levels of prompts, we achieve an average improvement of 1.94% over the state-of-the-art methods on 12 benchmark datasets. our main contributions are as follows: ## methodology our proposed multi-level prompt tuning (mprompt) framework is illustrated in figure 1 . the framework consists of a prompt generator and a generative question answering model, whereas the former relies on a smaller-sized encoder-decoder architecture. the prompt generator generates domainspecific and context-specific prompts and elicits context-related knowledge from small-scale plms into the generation process. ## conclusion in this paper, we propose a novel multi-level prompt (mprompt) tuning method for machine reading comprehension. our method strengthens plms' utilization of input semantics through three levels of prompts: task-specific prompts, domainspecific prompts, and context-specific prompts. the task-specific prompts are input-independent and generate prompts specific to a task. the domain-specific prompts utilize the domain knowledge generated from the dataset while contextspecific prompts are relying on the input context. our experiments show the combination of three level prompts improves the answer generation performance on different sizes of plms and 12 benchmark datasets. in future work, we will extend our method to more tasks such as summarization, translation, and sentiment analysis.