C omple QA : Benchmarking the Impacts of Knowledge Graph Completion Methods on Question Answering (2023)

Abstract: How much success in Knowledge Graph Completion (KGC) would translate into the performance enhancement in downstream tasks is an important question that has not been studied in depth. In this paper, we introduce a novel benchmark, namely CompleQA, to comprehensively assess the influence of representative KGC methods on Knowledge Graph Question Answering (KGQA), one of the most important downstream applications. This benchmark includes a knowledge graph with 3 million triplets across 5 distinct domains, coupled with over 5000 question-answering pairs and a completion dataset that is well-aligned with these questions. Our evaluation of four well-known KGC methods in combination with two state-of-the-art KGQA systems shows that effective KGC can significantly mitigate the impact of knowledge graph incompleteness on question-answering performance. Surprisingly, we also find that the best-performing KGC method(s) does not necessarily lead to the best QA results, underscoring the need to consider downstream applications when doing KGC.

## introduction the inherent incompleteness of knowledge graphs (kgs) is a well-recognized problem. as such, a multitude of methods have been proposed to address this issue, spanning statistical relational learning @xcite , embedding-based models @xcite @xcite , neural-symbolic methods @xcite @xcite , and recent language model-based methods @xcite @xcite . however, prior studies have largely viewed kg completion as an end in itself, neglecting to investigate its potential impact on subsequent applications that utilize the completed kgs. the applications of kgs span different domains and tasks. among them, knowledge graph question answering (kgqa), designed to answer natural language questions using information from kgs, is one of the most important. extensive approaches have been proposed in recent years @xcite @xcite . despite the rapid advancement, previous studies either assume the kgs are complete or handle incompleteness via specific questionanswering methods @xcite @xcite , lacking a comprehensive investigation into different approaches. the impact of kgc on kgqa performance remains largely unexplored. in our study, we seek to rectify this oversight by introducing a novel benchmark compleqa designed to directly and holistically assess the influence of kg completion on kgqa. this benchmark comprises over three million triplets and approximately 400,000 entities across 5 different domains, collectively forming the kg. the corresponding qa dataset includes over 5,000 questions, featuring both single-hop and multi-hop questions. these questions are sorted into three generalization levels and are sourced from the grailqa @xcite dataset. for kg completion, we employ entitycentric incompleteness to align with the qa dataset, where the entities in the missing triplets correspond to those appearing in the questions. importantly, different from previous studies, we actually incorporate predicted triplets into the kg, and use this completed kg for question answering, allowing a seamless study of various kgc and kgqa methods. our investigation incorporates the study of four representative kg completion models, namely transe @xcite , complex @xcite , and ro-tate @xcite . for the kgqa models, we employ two published leading methods decaf @xcite and pangu @xcite . our experiments show that the incompleteness of kgs can adversely affect qa performance, and effective kg completion can help alleviate this and improve performance by up to 6.1% across varying degrees of incompleteness. furthermore, we discern that kg completion performance does not always align with downstream performance, due to the limited overlap in triplet predictions between models and the disparate influence of these triplets on kgc and qa performance. we hope our findings stimulate more research that perceives kg completion not merely as an isolated objective but as a crucial step toward enhancing the performance of downstream tasks. ## benchmark construction in this section, we delineate the procedural steps employed in the development of our benchmark, including the knowledge graph, the questionanswering dataset, and the completion dataset. ## methods and evaluation metrics this section presents the methods implemented on our benchmark and the evaluation metrics employed to assess their effectiveness. ## experiments in this section, we delve into the outcomes of our empirical investigations. firstly, from to validate this, we construct a scenario that incorporates only the correctly-predicted triplets into the kg, while all the incorrect triplets are discarded. figure 2 , along with other results in appendix c, clearly illustrates a significant performance enhancement in this scenario, especially at the 80% incompleteness level, thereby substantiating the detrimental impact of incorrect predicted triplets on the qa model. q2: does the best kgc method(s) always lead to the best qa performance? according to to quantitatively assess the relationship between the performance of kgc and qa, we use spearman's rank correlation coefficient. this measures the pearson correlation between the ranked values of two variables. in many instances, we find that the correlation between the kgc metric and qa metric is not high, with a value below 0.5. another observation is that the f1 score of kgc performance corresponds better with qa performance than the mean reciprocal rank (mrr). this is reasonable because qa performance relies on the completed knowledge graph, which includes predicted triplets. the f1 score, in this context, is a more direct measure of the quality of predicted triplets compared to mrr. to delve further, we measured the overlap of predicted triplets among various models and found that no model's predictions, correct or incorrect, fully encompassed the other's. for example, in a 20% incomplete kg, complex and rotate had about 80% shared correct predictions and 30% shared incorrect ones. in this case, despite complex's superior performance in kgc, it doesn't solely determine qa performance as various predicted triplets impact qa differently, and this impact may not align well with their contribution to kgc performance. this discrepancy points to the need for kgc methods that optimize both kg completion and downstream task performance. ## conclusion in this study, we introduced a novel benchmark to investigate the impact of representative kgc methods on the knowledge graph question answering (kgqa) task. our findings demonstrate that kg incompleteness negatively affects kgqa, and effective kgc can significantly mitigate this issue. however, we also discovered that best-performing kgc method does not necessarily lead to the best kgqa results. our work underlines the necessity to view kgc not merely as a standalone goal, but as a vital step toward improving downstream tasks.