MVP -Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning (2023)

Abstract: Recent advances in pre-trained language models (PLMs) have facilitated the development ofcommonsense reasoning tasks. However, existing methods rely on multi-hop knowledgeretrieval and thus suffer low accuracy due toembedded noise in the acquired knowledge. In addition, these methods often attain highcomputational costs and nontrivial knowledgeloss because they encode the knowledge independently of the PLM, making it less relevant to the task and thus resulting in a poorlocal optimum. In this work, we propose MultiView Knowledge Retrieval with Prompt Tuning (MVP-Tuning). MVP-Tuning leveragessimilar question-answer pairs in the training setto improve knowledge retrieval and employsa single prompt-tuned PLM to model knowledge and input text jointly. We conduct our experiments on five commonsense reasoning QAbenchmarks to show that MVP-Tuning outperforms all other baselines in 4 out of 5 datasetswith less than 2% trainable parameters. MVPTuning even gets a new state-of-the-art resulton OpenBookQA and is number one on theleaderboard.

## introduction endowing machines with human-like commonsense reasoning capabilities has gained increasing interest in natural language processing in recent years @xcite . large pre-trained language models @xcite @xcite @xcite offer unprecedented potential to mine knowledge because of their unique capability in incontext learning. however, given their black-box nature, these models lack essential interpretability, resulting in the embedded knowledge that is always implicit, difficult to interpret, and fragmented. therefore, people have developed methods to explicitly inject external knowledge, such as knowledge graphs (kg), as contextual knowledge into downstream tasks like commonsense reasoning. the main challenge of the above solution lies in utilizing knowledge to serve individual queries while suffering the scalability issue since there can be millions of nodes in the knowledge graph. intuitively, how to extract a partial knowledge graph, i.e., a subgraph, effectively and efficiently is crucial. recent efforts focus on the multi-hop knowledge retrieval strategy @xcite , which anchors input context entities to kg nodes and obtains relevant subgraphs from these nodes and the corresponding multi-hop neighbors. knowledge triplets retrieved by multi-hop retrieval need to be directly connected in the knowledge graph and form a path. this process is highly sensitive to the quality of the knowledge graph, e.g., it tends to fail when necessary triplets are distant from the query and even in another subgraph. therefore, the knowledge extracted by this strategy is often incomplete and biased as the neighbors of the input context entities bound the search span. to this, we propose multi-view retrieval, which expands the pool of knowledge triplet candidates with additional highly-related question-answer pairs. this method does not suffer from the limitation of multihop retrieval and is able to connect distant or disjoint triplets via some similarity measurements, resulting in broader and more diverse triplets from the kg. figure 1 compares these two retrieval strategies. for example, given the question "what are candles good for eliminating?", two retrieved multiview knowledge triplets "(candle, capableof, emit light)" and "(candle, atlocation, dark)" are sufficient to guide the plm to reason and output the answer "dark" directly. on the other hand, the conventional multi-hop strategy needs to retrieve three triplets "(dark, antonym, light)", "(light, antonym, heavy)", and "(dark, isa, illumination)", which are much noisier and more challenging to reason. having extracted the target knowledge, how can we harness this knowledge to serve the ultimate purpose of reasoning? an intuitive way is to employ graph neural networks (gnns) to output node embeddings for kgs and then fuse them with embeddings of other input texts from plms @xcite @xcite @xcite . despite being straightforward, this solution inherits critical issues from gnns, such as over-smoothness. instead, we explore a new way of encoding knowledge from kgs, simple yet effective. for encoding, we directly combine the retrieved knowledge triplets as texts and concatenate them with other textual information as the input of the plm. our approach can alleviate the computational cost and reduce the information loss compared to previous gnns based approaches. in this paper, our proposed multi-view knowledge retrieval scheme can outperform existing work regarding efficiency and accuracy by a large margin when built with recent successful parameterefficient learning techniques, such as prompttuning @xcite @xcite with a plm. therefore, we name our framework as multi-view knowledge retrieval with prompt tuning (mvp-tuning). the multiview knowledge retrieval strategy brings more accurate knowledge localization with less computational cost. the obtained knowledge is fed into a plm model to augment the information in text. to further improve the capability of our model, we integrate parameter-efficient learning in this context. in summary, our primary contributions are: others attempt to build knowledge encoders with simpler designs that exhibit superior performance. gsc @xcite creates a basic graph neural counter that beats more complicated approaches, indicating that gnn-based encoders are merely doing simple counting. safe (jinhao jiang and wen, 2022) encodes relation routes using a simple twolayer mlp. however, these approaches encode text and knowledge independently. greaselm @xcite , on the other hand, integrates the representations of both kg and plm encoders over multiple layers of modal interaction processes. prompt-based unstructured knowledge utilization a line of research has investigated the use of unstructured knowledge sources, such as wikipedia and dictionaries, for commonsense reasoning @xcite . these methods append related knowledge to the input context as a prompt to improve commonsense reasoning. for example, @xcite combined knowledge from conceptnet, wordnet, and other corpora to create 3.5 million generic statements and show that this knowledge can enhance both accuracy and explanation quality. other studies have focused on comparing different methods for incorporating external knowledge from relevant corpora @xcite . additionally, there have been efforts to generate missing facts from plms to supplement external knowledge sources. for instance, @xcite fine-tuned a plm on atomic for commonsense knowledge graph completion, and @xcite prompted gpt-3 @xcite directly to obtain knowledge for reasoning. prompt learning prompt learning is a simple yet effective approach to adapt a plm for specific downstream tasks by adding prompt tokens in the input. a line of prompt learning works utilizes automated search techniques to identify appropriate discrete prompting words @xcite . in contrast to these discrete prompt ## problem statement in this work, we study the multiple-choice commonsense question answering @xcite . given a natural language question q and a collection of n response candidates @xmath0, • • • , c n }, the purpose is to pick the most appropriate candidate c ⋆ ∈ c to answer the question q based on the requisite commonsense knowledge. in accordance with previous work @xcite , we address this commonsense reasoning task in a knowledge-aware setting that embraces a commonsense knowledge graph (cskg) as the commonsense knowledge source. an external cskg can be described formally as a multi-relational graph @xmath1, r, e), where v is the set of all concept nodes (e.g., leg and fire), r is the set of relation types (e.g., capableof and isa), and e ⊆ v ×r×v is the set of relational edges that connects two concept nodes in the v. specifically, we employ conceptnet @xcite , which consists of 799,273 nodes and 2,487,003 edges. ## approach: mvp-tuning as shown in figure 2 , mvp-tuning is based on the plm and includes the multi-view knowledge retrieval module and the prompt tuning module. we augment the input context with multi-view retrieved knowledge, and the prompt tuning module optimizes the plm in a parameter-efficiency way. ## experiments as shown in ## analysis 6.1 low-resource setting to test the robustness of mvp-tuning, we examine its performance in low-resource settings, with three different proportions of training data, i.e., 5%, 10% and 20%, in commonsenseqa and openbookqa. for the commonsenseqa, we still use the in-house split setting. we follow safe (jinhao jiang and wen, 2022) setting to report the average test per- ## conclusion in this work, we propose mvp-tuning, a simple and effective approach to building a strong com-13424 65% 70% 75% 80% 85% 0 5 10 15 20 25 84% 85% 86% 87% 88% 10 20 50 100 150 200 acc number of prefix tokens commonsenseqa openbookqa figure 4: effect of the number of soft prefix tokens. monsense reasoning qa system. it strengthens the conventional knowledge retrieval results via multi-view knowledge and unifies the modeling of input text and retrieved knowledge in a single prompt-tuned plm. extensive experiments show the superiority of mvp-tuning, as it beats other sophisticated approaches in 4 out of 5 popular commonsense qa benchmarks while having less than 2% trainable parameters. mvp-tuning achieves a new state-of-the-art performance in openbookqa and wins first place in the leaderboard.