Knowledgeable Parameter Efficient Tuning Network for Commonsense Question Answering (2023)

Abstract: Commonsense question answering is important for making decisions about everyday matters. Although existing commonsense question answering works based on fully fine-tuned PLMs have achieved promising results, they suffer from prohibitive computation costs as well as poor interpretability. Some works improve the PLMs by incorporating knowledge to provide certain evidence, via elaborately designed GNN modules which require expertise. In this paper, we propose a simple knowledgeable parameter efficient tuning network to couple PLMs with external knowledge for commonsense question answering. Specifically, we design a trainable parameter-sharing adapter attached to a parameter-freezing PLM to incorporate knowledge at a small cost. The adapter is equipped with both entity- and query-related knowledge via two auxiliary knowledge-related tasks (i.e., span masking and relation discrimination). To make the adapter focus on the relevant knowledge, we design gating and attention mechanisms to respectively filter and fuse the query information from the PLM. Extensive experiments on two benchmark datasets show that KPE is parameter-efficient and can effectively incorporate knowledge for improving commonsense question answering.

## introduction commonsense question answering is the process of combining observations and the basic knowledge that reflects our natural understanding of the world and human behaviors, to make presumptions about ordinary situations in our daily life @xcite . it has emerged as an important task in natural language understanding. pre-trained language models (plms), which revolutionize many areas with superior performance, have been applied for the commonsense question answering task based on full fine-tuning as shown in figure 1 (a). for example, @xcite fully fine-tuned the plm unicorn and achieved competitive performance on 8 commonsense benchmarks. however, they inevitably incur prohibitive computation costs as the scale of parameters increases, and are lacking in transparency and interpretability @xcite . furthermore, some works couple the plms with knowledge to improve the interpretability of the reasoning process. as shown in figure 1 (b), they typically extract the relevant knowledge subgraphs about entities in the query and then elaborately design a graph neural network (gnn) module to perform reasoning @xcite @xcite . despite the fact that they provide certain evidence for the reasoning process, it requires expertise to design effective gnn modules. additionally, they generally consider only the structured triple knowledge about the entities in the query, while ignoring the textual knowledge about the query itself. in this work, we propose a simple knowledgeable parameter efficient model (kpe) for commonsense question answering. in particular, we design a parameter-sharing adapter plugin for incorporating knowledge into the frozen plm as shown in figure 1 , which largely reduces the scale of trainable parameters. our adapter plugin integrates both the entity-and query-related knowledge (uniformly grounding to the unstructured commonsense knowledge base genericskb) through two auxiliary knowledgerelated tasks (i.e., span masking and relation discrimination). additionally, to make the adapter focus on the relevant knowledge for commonsense question answering, we design gating and attention mechanisms to respectively filter and fuse the query information from the plm. overall, our main contributions can be summarized as follows: ## related work in this section, we review the related works on commonsense question answering and parameter efficient tuning. ## kpe model following previous works @xcite , we focus on the commonsense question answering task in the form of multiplechoice question answering. formally, given a natural language query q and a set of candidate answers @xmath0, we will measure the plausibility score ρ(q, a) for each answer and choose the most plausible one a * . to promote the commonsense question answering process, we resort to external knowledge bases to extract both entity-and query-related knowledge pieces @xmath1. as shown in figure 2 , our kpe couples the plm with external knowledge via a parameter-sharing knowledgeable adapter attached to the frozen plm. the plm takes (q, a) as input and outputs the plausibility score ρ(q, a). the knowledgeable adapter aims to integrate the knowledge pieces k. in the following, we first introduce the knowledge extraction process. then we describe the knowledgeable adapter that effectively integrates the extracted knowledge based on two auxiliary tasks (i.e., span masking and relation discrimination tasks), as well as gating and attention mechanisms for information interaction with the plm. ## experiments in this section, we evaluate the effectiveness of our proposed kpe. ## conclusion in this work, we present a knowledgeable parameter efficient tuning network kpe to effectively incorporate both entity-and query-related knowledge for improving commonsense question answering. particularly, we design a parameter-sharing knowledgeable adapter as the plugin attached to the frozen plm to incorporate knowledge. two auxiliary knowledge-related tasks are specifically designed for the adapter to better model and capture the knowledge. moreover, to make the adapter integrate relevant knowledge, we introduce gating and attention mechanisms to respectively filter and fuse the query information from the plm. experiments on two benchmark datasets have demonstrated the effectiveness and parameter-efficiency of kpe for commonsense question answering. in future work, we will explore to integrate other parameter-efficient tuning tricks in kpe.