KE - GCL : Knowledge Enhanced Graph Contrastive Learning for Commonsense Question Answering (2022)

Abstract: Commonsense question answering (CQA) aims to choose the correct answers for commonsense questions. Most existing works focus on extracting and reasoning over external knowledge graphs (KG). However, the noise in KG prevents these models from learning effective representations. In this paper, we propose a Knowledge Enhanced Graph Contrastive Learning model (KE-GCL) by incorporating the contextual descriptions of entities and adopting a graph contrastive learning scheme. Specifically, for QA pairs we represent the knowledge from KG and contextual descriptions. Then, the representations of contextual descriptions as context nodes are inserted into KG, forming the knowledge-enhanced graphs.Moreover, we design a contrastive learning method on graphs. For knowledge-enhanced graphs, we build their augmented views with an adaptive sampling strategy. After that, we reason over graphs to update their representations by scattering edges and aggregating nodes. To further improve GCL, hard graph negatives are chosen based on incorrect answers. Extensive experiments on two benchmark datasets demonstrate the effectiveness of our proposed KE-GCL, which outperforms previous methods consistently.

## introduction commonsense question answering (cqa) is an emerging task in the domain of machine reading comprehension with the long-term goal for evaluating the language understanding of machines. the cqa task aims to choose answers for natural language questions about commonsense. figure 1 shows an example to illustrate the definition of the cqa task. to solve this task, external knowledge graphs (kgs) of commonsense, such as conceptnet @xcite where numerous triplets are provided to represent relations between entities, are used in reasoning for the correct answers. to take advantage of the commonsense knowledge, a few works @xcite @xcite @xcite directly retrieve and model the relevant evidence to infer answers. with the great success of graph neural networks (gnns) @xcite @xcite @xcite , recent studies @xcite @xcite @xcite @xcite focus on devising exquisite graph networks with task-dependent attention mechanism to model kgs for effective reasoning. however, previous methods ignore the noise, i.e., irrelevant or distracting entities in the kg, resulting in unsatisfactory performance. take the example in figure 1 . compared with those between "steak house" and the other four choices, the relational path between "steak house" and the choice "restaurant" is irrelevant to the question. moreover, the choices with identical relations of "atlocation" are difficult to be discerned. in other words, the choices whose topology is similar to that of the correct answer are strong noise. therefore, the noise problem should be considered seriously. to address the aforementioned noise problem, we adopt two schemes. one is enhancing kg with textual descriptions. as we all know, kg represents the topology among entities. by incorporating contextual meanings, we could further capture the semantic nuances of entities. thus, it would be beneficial for reducing the negative effect of distracting entities in kgs. take figure 1 as an example; with the enhancement of entity descriptions, we undoubtedly exclude distractors (a, d, and e) which are semantically conflict with the location specified in the question ("near south of the u.s."). then, we derive the correct answer c ("mexico"). the other scheme uses graph contrastive learning (gcl). intuitively, the kg of the qa pair with the correct answer is often topologically similar to those with distractors. this inherent noise could hinder the graph reasoning. to this end, we construct graph positive pairs with negative counterparts based on the gcl framework. in this paper, we propose an end-to-end model, knowledge enhanced graph contrastive learning (ke-gcl) for cqa task. first, we concatenate the given qa pair (i.e., a question with its current choice) with the wiktionary foot_0 descriptions obtaining the contextual representation. and we extract subgraphs from conceptnet foot_1 obtaining the graph embeddings. we then insert the contextual representation as a node into the graph and perform attention-based fusion, obtaining the knowledgeenhanced graph. second, we incorporate gcl scheme into our ke-gcl model. to produce the augmented view of the graph, we introduce an adaptive graph augmentation strategy. we adaptively drop out irrelevant edges and mask unimportant node features. the sampling probabilities for nodes and edges are determined by topological connectivity and contextual relevance. furthermore, to achieve efficient message propagation in graph reasoning, we devise a graph attention network (gat) @xcite based reasoning module, by scattering the connected edges and aggregating the adjacent nodes. the knowledge-enhanced graph and its augmented view jointly perform reasoning. thus, we obtain the final graph representations. third, to enhance the training signals, we build positive and negative pairs for computing graph contrastive loss. specifically, for the positive pair, we use the graph augmented view of the correct answer. for the negative pairs, we take the graph and its augmented counterparts of other incorrect choices as hard negatives. we set the other graphs in the mini-batch as common negatives. thus, we train ke-gcl model with a combination of two losses. one is the answer prediction loss and the other is the graph contrastive loss. major contributions are summarized as follows: ## our proposed ke-gcl the ke-gcl model framework is shown in figure 2. we elaborate on the details as follows. ## conclusion and future work in this paper, we propose a novel ke-gcl model for cqa task, which leverages contextual descriptions and gcl to reduce the noise in the kg. first, we integrate contextual descriptions into the kg, forming the knowledge-enhanced graph. then, we devise an adaptive sampling strategy to generate the augmented view of the graph. moreover, we reason over graphs via edge scattering and node aggregation. finally, to further enhance the effect of gcl, we take graph pairs of incorrect answers as hard negatives. extensive experiments on benchmark datasets verify that our ke-gcl model outperforms the baselines consistently. in the future, we will consider how to apply the gcl scheme in the few-shot or unsupervised scenarios for various cqa tasks.