Gold: A Global and Local-aware Denoising Framework for Commonsense Knowledge Graph Noise Detection (2023)

Abstract: Commonsense Knowledge Graphs (CSKGs) are crucial for commonsense reasoning, yet constructing them through human annotations can be costly. As a result, various automatic methods have been proposed to construct CSKG with larger semantic coverage. However, these unsupervised approaches introduce spurious noise that can lower the quality of the resulting CSKG, which cannot be tackled easily by existing denoising algorithms due to the unique characteristics of nodes and structures in CSKGs. To address this issue, we propose Gold (Global and Local-aware Denoising), a denoising framework for CSKGs that incorporates entity semantic information, global rules, and local structural information from the CSKG. Experiment results demonstrate that Gold outperforms all baseline methods in noise detection tasks on synthetic noisy CSKG benchmarks. Furthermore, we show that denoising a real-world CSKG is effective and even benefits the downstream zero-shot commonsense question-answering task. Our code and data are publicly available at https://github.com/HKUST-KnowComp/GOLD.

## introduction the emergence of commonsense knowledge graphs (cskgs) has significantly impacted the field of commonsense reasoning @xcite as cskgs provide commonsense knowledge that is often not explicitly stated in the text and difficult for machines to capture systematically @xcite . while existing methods bank on expensive and timeconsuming crowdsourcing to collect commonsense knowledge @xcite , it remains infeasible to obtain cskgs that are large enough to cover numerous entities and situations in the world @xcite . to overcome this limitation, various automatic cskg construction methods have been proposed to acquire commonsense knowledge at scale @xcite , including prompting large language model (llm) @xcite , rule mining from massive corpora @xcite , and knowledge graph population @xcite . although those methods are effective, they still suffer from noises introduced by construction bias and the lack of human supervision. therefore, how to identify noise in large-scale cskg accurately and efficiently becomes a crucial research question. to tackle this issue, noise detection algorithms have been proposed for conventional entitybased kgs by primarily adopting two approaches: learning-based and rule-based. learning-based methods like transe @xcite learn representations of entities and relations that adhere to specific relation compositions like translation assumption or relational rotation. to enhance their performance, researchers also incorporate local information around the head and tail entities, such as different paths from head to tail @xcite @xcite and neighboring triples @xcite . these methods aim to improve their ability to capture the complex relationships between entities in kgs. however, they are not easily adaptable to the unique characteristics of cskgs. in cskgs, nodes are noncanonicalized, free-form text, meaning nodes with different descriptions may have related semantics. as illustrated in figure 1 , "paint door" and "paint house" are two distinct nodes but imply related semantics @xcite . additionally, when detecting noise (paint, usedfor, brush your teeth), "brush your teeth" is an isolated node that cannot be distinguished based on any structural information. only through the power of a language model can it be learned that "paint" and "brush your teeth" are uncorrelated, thus detecting such noise. the aforementioned methods overlook this semantic information and cannot generalize to semantically similar events with diverse structural information. on the other hand, rule-based methods utilize logical rules in kgs for noise detection. for instance, as shown in figure 1 , the correct relation between "painter" and "paint house" should be capableof. this can be easily detected through the learned logical rule: capableof(x, y) ← capableof(x, z)∧hasprerequisite(y, z). @xcite similarly propose an approach based on information theory that extracts sub-graph patterns to identify the noise. however, the sparsity of edges in cskgs @xcite posits a serious challenge to learning structural information well, as the number of learnable rules decreases significantly. this requires a generalizable rulelearning ability at the noise detector side to expand the rule bank accordingly, which is currently lacking. therefore, applying noise detection models for kgs directly to cskgs can result in incomplete learning of both semantic and structural information in the cskgs. in order to detect noises in cskgs effectively, it is important to consider both the semantic information and the global and local structural information jointly. however, these factors have not been given enough importance in existing denoising approaches. to address this gap, we propose gold (global and local-aware denoising), a cskg noise detector that uses a plm-based triple encoder and two noise detectors that take into account both global and local structures (section 4). specifically, the triple encoder extracts the semantic information contained in the free-text formatted nodes in cskgs. to identify correct patterns, the global detector uses high-frequency patterns extracted through rule mining, which intrinsically uses a rule encoder to generalize the learned rules and guide noise detection. the local detector, inspired by @xcite , adopts a graph neural network to efficiently measure the similarity of aggregated semantic information of neighboring triples of the head and tail nodes to help detect noise. extensive experiments on two manually synthesized noisy-cskg benchmarks demonstrate the efficacy and state-of-the-art performance of gold. further experiments and analyses with atomic 10x @xcite , a large-scale cskg distilled from gpt3, demonstrates its proficiency in identifying noise within real-world cskgs, while also yielding advantages in the downstream zero-shot commonsense question-answering task. in summary, in this paper, we make the following contributions: ## problem definition noises in cskg commonsense knowledge represents not only basic facts in traditional knowledge graphs but also the understanding possessed by most people @xcite , we evaluate whether a triple is a noise from two perspectives: if a triple fails to satisfy any of the aspects mentioned above, we define it as noise. given a triple (h, r, t) ∈ e in a cskg, we concatenate the language descriptions of h, r, and t and determine whether this description conforms to commonsense. we note that each triple violates commonsense to a different degree, and we define noise detection as a ranking problem to standardize the evaluation process better. thus, we model noise detection as a ranking process where a scoring function f : @xmath0. ## the gold method our proposed method gold comprises four components: triple encoder, global noise detector, local noise detector, and comprehensive evaluation scorer. an overview is presented in figure 2 . first, we leverage a plm to encode the natural language descriptions of nodes and relations in cskgs to obtain their sentence embeddings, thus further encoding the triples. when detecting noise, we evaluate the likelihood of a triple being noise from both a global and local perspective. from the global perspective, we aim to identify high-frequency patterns in the knowledge graph, as a small amount of noise is less likely to affect correct high-frequency patterns @xcite . to accomplish this, we employ rule mining to extract high-quality rules from the knowledge graph. from the local perspective, we adopt graph networks to aggregate the neighboring triple information around both the head and tail nodes of a given edge, allowing us to estimate if there is any correlation. finally, based on these two aspects of detection, we obtain a comprehensive score indicating the noise level. ## conclusions in this paper, we propose gold, a noise detection framework leveraging the power of language models, global rules, and local structural information. this method is motivated by the fact that nodes in cskgs are in free-text format, and correct patterns are unlikely to be drowned out by noise. experimental results indicate that our method achieves state-of-the-art performances in cskg noise detection tasks. this method shows promising directions for automatically obtaining a large-scale cskg with minimal noise, as well as effectively representing knowledge for downstream tasks.