Complex Logical Query Answering by Calibrating Knowledge Graph Completion Models (2024)

Abstract: Complex logical query answering (CLQA) is a challenging task that involves finding answer entities for complex logical queries over incomplete knowledge graphs (KGs). Previous research has explored the use of pre-trained knowledge graph completion (KGC) models, which can predict the missing facts in KGs, to answer complex logical queries. However, KGC models are typically evaluated using ranking evaluation metrics, which may result in values of predictions of KGC models that are not well-calibrated. In this paper, we propose a method for calibrating KGC models, namely CKGC, which enables KGC models to adapt to answering complex logical queries. Notably, CKGC is lightweight and effective. The adaptation function is simple, allowing the model to quickly converge during the adaptation process. The core concept of CKGC is to map the values of predictions of KGC models to the range [0, 1], ensuring that values associated with true facts are close to 1, while values linked to false facts are close to 0. Through experiments on three benchmark datasets, we demonstrate that our proposed calibration method can significantly boost model performance in the CLQA task. Moreover, our approach can enhance the performance of CLQA while preserving the ranking evaluation metrics of KGC models. The code is available at https://github.com/changyi7231/CKGC.

## introduction knowledge graphs (kgs) are composed of structured representations of facts in the form of triplets and have been widely used in various domains. one of the key tasks associated with kgs is complex logical query answering @xcite . complex logical queries are typically expressed using first-order logic (fol), which encompasses logical operations such as conjunction (∧), disjunction (∨), * corresponding author. negation (¬), and existential quantifier (∃). for example, the query "which universities do the turing award winners not in the field of deep learning work in?" can be formulated as a fol query, as illustrated in figure 1 . many well-known knowledge graphs (kgs) suffer from incompleteness, rendering it challenging to answer complex queries through simple kg traversal. building on the accomplishments of knowledge graph completion (kgc) methods @xcite @xcite in addressing one-hop kg queries, a research avenue has emerged focusing on learning embeddings for queries to handle complex logical queries @xcite @xcite . nonetheless, these methodologies often require extensive training on numerous complex logical queries, leading to substantial training time overhead and limited generalization to out-ofdistribution query structures. in addressing these challenges, cqd @xcite introduces a method for clqa by leveraging one-hop atom results derived from a pretrained kgc model, thereby removing the necessity for training on complex queries. cqd frames clqa as an optimization problem and employs techniques like beam search or continuous approximation to estimate the optimal solution. despite its effectiveness, the approximations made during the process could lead to a decrease in accuracy for cqd. furthermore, cqd is reliant on a kgc model whose values of output might not be specifically calibrated for clqa, potentially resulting in inaccuracies in the outcomes. in this paper, we introduce a method for calibrating kgc models, namely ckgc, which can make kgc models adapt to handling complex logical queries. our method is lightweight and effective. it is lightweight, as the adaptation function is simple and the model can quickly converge in adaptation process. moreover, it is effective in significantly enhancing the performance of clqa. we first represent every complex logical query as a computation graph over fuzzy sets, of which nodes represent fuzzy sets and edges represent the fuzzy set operations over fuzzy sets. to obtain the answers of a query, we traverse the computation graph and execute the fuzzy set operations through a straightforward forward propagation process, eliminating the need for optimization compared to cqd @xcite . we define four fundamental fuzzy set operations for clqa: projection operation, complement operation, intersection operation, and union operation, all of which require the kgc model to be calibrated. however, a kgc model is typically measured by the ranking metrics, which do not need it to be calibrated. thus, we propose a calibration method for kgc models to make it adapted to clqa. we define the adaptation function as a monotonically increasing function that preserves the ranking evaluation metrics of the kgc model. the main idea of our method is to map the values of output of a kgc model to the interval [0, 1], and makes the values corresponding to true triplets be as close as possible to 1, while the values corresponding to false triplets be as close as possible to 0. our experimental evaluation on three standard datasets demonstrates the efficacy of ckgc in clqa. the results reveal that ckgc achieves state-of-the-art performance across all datasets, showcasing an average relative improvement of 6.7% on existential positive first-order queries and 53.9% on negation queries compared to the prior state-of-the-art method. complex logical query answering clqa over kgs extends kgc to predict answers for fol queries, which additionally requires defining relationships between sets of entities. embeddingbased methods represent sets of entities as geometric objects @xcite @xcite or probability distributions @xcite , and then minimize the distance between embeddings of queries and embeddings of their corresponding answers. however, the quality of representation of sets may be compromised when dealing with large sets. to overcome this limitation, some studies have incorporated powerful fuzzy set theory to handle fol queries @xcite . for instance, fuzzqe @xcite embeds entities and queries into a fuzzy space and leverages fuzzy set operations to perform logical operations on the embeddings. similarly, gnn-qe @xcite decomposes the query into relational projections operations and fuzzy set operations over fuzzy sets, and subsequently learns a graph neural network to execute relational projections. nevertheless, the aforementioned methods generally necessitate training on numerous complex logical queries, resulting in significant training time overhead and limited generalization to outof-distribution query structures. another line of methods first pre-trains a kgc model and then integrates it with fuzzy set theory to infer answers. cqd @xcite formulates clqa as an optimization problem and proposes two strategies to approximate the optimal solution: cqd-co, which directly optimizes in the continuous space, and cqd-beam, which utilizes beam search. despite cqd avoiding the need to train on complex logical queries, it suffers from accuracy loss due to the approximated optimization and uncalibrated kgc models. to enhance accuracy, @xcite introduce qto, which efficiently finds the theoretically optimal solution through a forward-backward propagation on a treelike computation graph. additionally, @xcite propose cqd a , a parameter-efficient adaptation method, to calibrate kgc models. however, the kgc model for qto is not well calibrated for complex queries, and the optimal solution for cqd a is still approximated. in contrast, our proposed model not only delivers accurate solutions but also effectively calibrates the kgc models. ## background in this section, we introduce the related background of our method, complex logical query answering on knowledge graphs and fuzzy sets. ## conclusion clqa is one of the crucial tasks associated with kgs. in this paper, we introduce ckgc, a calibration method developed to enhance the adaptability of kgc models for clqa. through experimental analysis, we illustrate that the implementation of ckgc leads to a substantial improvement in model performance for the clqa task. therefore, the calibration of kgc models holds significant importance for the optimization of clqa models. moving forward, we encourage the exploration and proposal of additional calibration methods to further enhance the performance of clqa.