CORN : Co-Reasoning Network for Commonsense Question Answering (2022)

Abstract: Commonsense question answering (QA) requires machines to utilize the QA content and external commonsense knowledge graph (KG) for reasoning when answering questions. Existing work uses two independent modules to model the QA contextual text representation and relationships between QA entities in KG, which prevents information sharing between modules for co-reasoning. In this paper, we propose a novel model, Co-Reasoning Network (CORN), which adopts a bidirectional multi-level connection structure based on Co-Attention Transformer. The structure builds bridges to connect each layer of the text encoder and graph encoder, which can introduce the QA entity relationship from KG to the text encoder and bring contextual text information to the graph encoder, so that these features can be deeply interactively fused to form comprehensive text and graph node representations. Meanwhile, we propose a QA-aware node based KG subgraph construction method. The QA-aware nodes aggregate the question entity nodes and the answer entity nodes, and further guide the expansion and construction process of the subgraph to enhance the connectivity and reduce the introduction of noise. We evaluate our model on QA benchmarks in the CommonsenseQA and OpenBookQA datasets, and CORN achieves state-of-the-art performance.

## introduction commonsense question answering (qa) research requires the machine to have a human thought pattern, which is capable of comprehending text content and combining commonsense knowledge to reason and arrive at the correct answer. despite the success of large pre-trained language models (plms) @xcite on various nlp tasks, there is still a large gap between plms and humans on commonsense qa figure 1 : the network architecture of current commonsense qa research. text encoder for encoding the qa content, graph encoder for reasoning on the graph. there is no connection between stacked text layers and stacked gnn layers. there has already been a significant amount of work that combines plms and kgs for reasoning @xcite @xcite . as illustrated in figure 1 , these works are mainly composed of two modules: (1) capturing text features on qa with a text encoder (such as plm). (2) extracting subgraph from kg and reasoning on it with a graph encoder, such as the gnn-based model @xcite . most work @xcite @xcite focuses on building more efficient graph encoders to capture relationships between entities in graphs for reasoning. however, it ignores the interconnections between the qa content and graph due to gnn and plm being treated as independent modules. to address the above problems, yasunaga et al. (2021) perform joint reasoning by explicitly adding the qa content to the graph in the form of a node. nevertheless, this method is a oneway connection structure, only enabling the graph encoder to obtain textual context information for reasoning while the text encoder cannot perceive graph information. we propose a novel model, co-reasoning network (corn), to solve the above problems. corn adopts a bidirectional multi-level connection structure, which connects the language model (lm) and gnn. specifically, we build bridges between each layer of these two types of models by the co-attention transformer. through this bridge, the text representation and graph node representation can be fused bidirectionally and respectively fed into the next lm and gnn layer. therefore, the gnn layer can reason on the subgraph with contextual text representation to enrich the graph node representation, and the lm layer can further encode the text with the graph node representation to improve the text representation. we adopt a multilevel connection structure to connect each layer, enabling text and graph representation with different semantic levels to interact, generating a more comprehensive feature representation. meanwhile, we propose a qa-aware node based kg subgraph construction method. we use a question-aware node and an answer-aware node to aggregate the question entity nodes and the answer entity nodes respectively and then guide the expansion and construction process of the subgraph to enhance the connectivity of the subgraph and reduce the introduction of noise. moreover, the qa-aware node can help the model perceive the difference between different types of nodes and help the model to learn the representation of graph nodes better. the main contributions of this work are summarized as follows: • we propose corn, which adopts a bidirectional multi-level connection structure. corn uses co-attention transformer to connect each layer of the lm and gnn, which allows lm to perceive graph information and enables gnn to integrate contextual text information, so that gnn and lm can generate richer text and graph node representations. ## problem statement in this paper, we focus on the task of multiplechoice question answering which required extra knowledge of reasoning. formally, giving a question q, a set of answer choices c and external knowledge graph, our purpose is to identify the correct answer from c. to be specific, we calculate the probability score between q and each answer choice @xmath0. we construct a multi-relational subgraph @xmath1, e) by kg (detailed in §3.1). here v is the subset of entity nodes from kg and is related to the mentioned entities in the qa content, e ⊆ v × r × v is the set of edges that connect nodes in v, where r represents a set of relation types. ## co-reasoning network the overview of our model is shown in figure 2 . we concatenate a question q and an answer choice a to get the qa content s, where @xmath2; a]. we apply plms (e.g., roberta) on s to get the initial text representation s ′ . for each qa content s, we use the qa-aware node based subgraph construction method ( §3.1) to construct a subgraph g and initialize the graph node representation g ′ . then, we use n-layer co-reasoning network (corn) to reason on the qa content s ′ and subgraph g ′ . each corn layer consists of lm layer ( §3.3), gnn layer ( §3.4) and co-attention transformer ( §3.2), where co-attention transformer bidirectionally connects lm and gnn, lm encodes qa text with graph representation, and gnn reasons on the subgraph with the contextual text representation. finally, we use the pooled graph representation and text representation from the last corn layer to make predictions to get the probability that the current choice a is the correct choice. ## conclusion we propose a novel commonsense qa model, corn, which adopts a bidirectional multi-level connection structure. it bidirectionally connects each layer of the lm and gnn through the co-attention transformer, which enables the lm to perceive the relationship of qa entity nodes to improve the text representation, and allows gnn to utilize contextual text information to enhance the graph node representation. meanwhile, we propose a qa-aware node based kg subgraph construction method. the qa-aware nodes aggregate question entity nodes and answer entity nodes and then guide the subgraph expansion and construction to increase the connectivity of the subgraph, and reduce the introduction of noise. through extensive experiments and visual analysis, corn can perform multi-level bidirectional interaction to improve the lm+kg models, and achieves stateof-the-art performance among them.