EmbedKGQA: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings
Anonymous Author(s)
Afﬁliation
Address email
Reproducibility SummaryScope of Reproducibility
Our work consists of four parts: (1) Reproducing results from Saxena et al.
[2020] (2) Adding more experiments byreplacing the knowledge graph embedding method (3) and exploring the question embedding method using varioustransformer models (4) Verifying the importance of Relation Matching (RM) module.
Based on the code shared bythe authors, we have reproduced the results for the EmbedKGQA method.
We have not purposely performed relationmatching to validate point-4.
Methodology
We have used the code provided by Saxena et al.
[2020] with some customization for reproducibility.
In addition tomaking the codebase more modular and easy to navigate, we have made changes to incorporate different transformersin the question embedding module.
Question-Answering models were trained from scratch as no pre-trained modelswere available for our particular dataset.
The code for this work is available on Github 1 .
Results
We were able to reproduce the Hits@ 1 to be within ± 2.35% of reported value (in most cases).
Anomalies were observed in 3 cases.
[1] In MetaQA-KG-Full (3-hop) dataset [2] In WebQSP-KG-50 and [3] WebQSP-KG-Full datasets.
From our experiments on the QA model, we have found that a recent transformer architecture, SBERT (Reimers and Gurevych [2019]) produced even better accuracy than the paper.
Replacing RoBERTa (Liu et al.
[2019]) with SBERT(Reimers and Gurevych [2019]) increased the absolute accuracy by ≈ 3.4% in half KG case and ≈ 0.6% in the full KGcase.
(KG: Knowledge Graph, " ≈ ": Approximately)
What was easy
As the code was open-sourced, we didn’t have to implement the paper giving us the liberty to customize the codebaseto focus on the author’s claim validation, perform extended experiments and explore shared as well as new models.
Inaddition to this, pretrained KG embedding models were shared which helped in the reproduction experiment.
What was difﬁcult
lack of a comprehensive documentation alongwith missing comments deﬁning functions/classes/attributes etc made it laborious to review the code and modify it.
In addition to large training times for question answering models the knowledge graph embeddings also required a signiﬁcant amount of computing resources
with
couple of virtual
Introduction
Knowledge is the key to question answering task.
Knowledge Graph (KG) is a multi-relational graph consisting ofentities as nodes and relations among them as typed edges.
KGs can accommodate a wide variety of facts, makingthem one of the potential candidates for intelligent decision-making.
Question Answering over KG (KGQA) task aimsto answer natural language queries posed over the KG.
Multi-hop KGQA is a trending topic and has gained tractionfrom both academia and industry recently.
Multi-hop KGQA task involves reasoning over multiple edges of the KGto arrive at the correct answer.
Earlier works on KGs(e.g.
Suchanek et al.
[2007], Google [2013], Lehmann et al.
[2015], Mitchell et al.
[2018])have some element of sparsity, i.e. they do not capture all the facts available in the real world.
Recent research onmulti-hop KGQA has attempted to reduce this sparsity with the help of relevant external textual resources that are notreadily available.
On the other side, KG embeddings have emerged as an effective tool to overcome the KG sparsityby predicting missing links in the KG.
Although effective, KG embeddings have not been explored for the multi-hop KGQA task. Saxena et al.
[2020] ﬁlls this gap with the proposed EmbedKGQA method.
This work intends to reproduce and perform an ablation (removing relation matching module) as well as extendedstudy on EmbedKGQA(Saxena et al.[2020]).
EmbedKGQA claims to be the ﬁrst of its kind to use KG embeddingsfor multi-hop KGQA and improves over other state-of-the-art baselines.
Scope of reproducibility
According to Saxena et al.
[2020], using ComplEx (Trouillon et al.
[2016]) KG embeddings signiﬁcantly improves
Hits@1 for multi-hop KGQA task and it has been proved with the help of the results on MetaQA (Zhang et al.
[2018])and WebQSP (Yih et al.
[2016]) datasets.
This reproducibility work tries to test this claim and conducts experimentsas mentioned in table:{2,3} of the original paper.
Section 4.1 contains the corresponding results which support theclaim with some anomalies.
Methodology
The authors of the original paper have open-sourced the code along with the data and pre-trained ComplEx KGembedding models.
We have used the same codebase (commit:5d8fdbd4) and customized it for our purposes. Inaddition to this, we have added a comprehensive documentation to make it more interpretable. Moreover, a commandline functionality is also added to easily conﬁgure various transformers models in the training workﬂow.
3.1 Model descriptions
As shown in Figure:1, EmbedKGQA has three modules:
KG
Embedding Module - This module contains a KG embedding model called ComplEx (Trouillon et al.
[2016]) to learn embeddings for all entities in the input KG.
4 pretrained models have been shared by the
author which contains 2 models for MetaQA-KG-{Full, 50} as well as 2 models WebQSP-KG-{Full,50} dataset.
Details about the dataset are mentioned in section:3.2.
Question Embedding Module: Given a question q, head entity h and set of answer entities A, this module
learns the question embeddings based on the score function deﬁned by the KG embedding method used in 1.
Answer Selection Module: This module uses the outputs of module:1 and 2 to select the ﬁnal answer by scoring the (head entity and question) pair against all possible answers.
The strategy is mentioned in section:4.4and 4.4.1 of Saxena et al.
3.2 Datasets
There are two datasets used in the original paper.
MetaQA (Zhang et al.
[2018]) and WebQSP (Yih et al.
Bothdatasets have two portions.
(1) KG data (2) QA data.
KG data for both are further divided into two categories.
Using the full KG (indicated by sufﬁx KG-Full) and (2) Using only 50% of the facts in the respective KGs (indicatedby sufﬁx KG-50).
The details of generating custom KG datasets are discussed here 3 .
Both datasets are taken fromhere 4 .
Statistics for table:1 and 2 have been taken from Saxena et al.
For generating question embeddings the Image source: Saxena et al.
[Table]
[Table]
[Table]
Experiment-Alias is the name used for the respective datasets in experiments.
γ = Contains all facts that are within 2-hops of any entity mentioned in the questions of WebQSP.
ϕ =
Contains only 50% of the triples ( randomly selected without replacement)
ψ = Contains 50% of the edges sampled randomly from fbwq_full.
question is placed between 
 and 
 tags for all transformers except SentenceTransformer as it takes the inputsentence in its pure form.
The preprocessing used in the original code has been used here.
No additional preprocessinghas been performed from our end.
3.3 Hyper-Parameters
Hyper-parameters used to train the models aren’t explicitly shared in the codebase or the paper, hence we decided touse the default values provided in the codebase 3 to compensate the lack of time.
For reproduction, a pretrained modelshared along with the data was used; ComplEx(Trouillon et al.
[2016]) was used as the knowledge graph embeddingmethod for all the KG types, i.e.,
[Table]
table-3, validate_every = The number of train routines before validation for a single epoch Total
[Table] [Table] [Table]
[Table] excluding For the purpose of reproducibility, hyper-parameters for training MetaQA and WebQSP QA models have been takenfrom Section:MetaQA 5 and Section:WebQuestionsSP 6 of the original codebase respectively.
For RoBERTa (Liu et al.[2019]), a pretrained model ’roberta-base’ has been taken from HuggingFace transformers package (Wolf et al.
Other hyper-parameters are populated by default values in the codebase 3 .
3.4 Experimental setup and code
Experiments have been performed on the NVIDIA DGX-1 server with 8xV100 GPUs, out of which 6 were used inthis work.
The metric used for validating the claims is Hits@1.
According to Wang et al.
[2019b], Hits@ k is theproportion of test triples ranking in the top-k results.
The code for this work is open-sourced on Github 1 .
In additionto this, we have shared couple of Docker images 7 for easy kick-starting of experiments without the hassle of settingup the environment.
Following trained models are made available in our Docker image 7 , chosen on the basis of better performance in ourextended study.
• TuckER KG embedding model for Meta-QA-{Full, 50}
• QA models trained using ComplEx as KG embedding model and SBERT mentioned in table:4 as questionembedding model for WebQSP-KG-{Full, 50}
3.5 Computational requirements
This work has been performed on 6 V100-16GB GPUs connected via NVLink. NVLink reduced multi-GPU trainingtime by 1/4.
The time required for various reproductions are mentioned in table:{5,6}.
3.6 Extended Experiments
Apart from reproducing the results mentioned in the original paper, a couple of extended experiments have beenperformed to ﬁnd answers to the following two questions:
Can recent KG embedding methods like TuckER ( Balaževi´c et al.
[2019]) give higher accuracy on higher
levels of hops,
i
e., 3-hop scenario to be speciﬁc compared to Trouillon et al.
[2016] used in the original paper? [Table]
Can other transformer architectures like
, Longformer (Beltagy et al.
[2020]) and SBERT (Reimers and Gurevych [2019]) improve the results on WebQSP(Yih et al.
Details of hyper-parameters used for these experiments are available on our Github repository 1 .
Various transformermodels used for experiment-2 are mentioned in [-table: 
 [Table] [Table] [Table]-]table:4.
Results
We report results for reproducibility as well as our extended experiments.
The results of reproduction have a mixednature while the ones for our extended experiments show positive signs to support claim-1, 2. The results are discussedin Section:5 in detail.
4.1 Results reproducing original paper
We perform two experiments based on the two datasets introduced in section:3.2.
These experiments provide vitalinformation about the results mentioned in table:{2,5} of the original paper.
The results of the two are reported intable-5 and 6 respectively.
From the results of table:5 in Saxena et al. [2020] and table-6 in this report, it is evident thatrelation matching(RM) is an important component to aid the multi-hop KGQA when the given KG is considerablylarge, i.e. {MetaQA, WebQSP} KG-Full; Deﬁnitely, WebQSP KG-50 also shows improvement in presence of RMbut the performance signiﬁcantly improves when applied to KG-Full setting. The original author, Apoorv has alsoexpressed the same opinion during one of the virtual meetings.
4.2 Results beyond the original paper
We have conducted two additional experiments from our end to ﬁnd an answer to claim:{1,2}.
The results in table:{7,8}support claim-1 but with a caveat.
On the other hand, values in table-9 improve upon the results reported by the originalpaper and creating a new SOTA baseline.
Additional experiments ingest custom hyper-parameters mentioned in ourcodebase in absence of the original hyper-parameters.
Discussion
The reproducibility results from table-5,6 corroborates the claims mentioned in Section:2 to some extent.
The reproduced version is within ± 2 .
4 range (positive value indicates better performance and vice-versa) except for MetaQAKG-Full dataset’s 3-hop scenario which has a signiﬁcant drop of 22.5%.
This happened due to the absence relationmatching module (discussed here 8,9,10 ) and hence it can be concluded that relation matching is a vital component inmulti-hop question answering, especially in Full-KG setting as most of the edges will be present making room for therelation matcher to make maximum use of the available edges in the KG. Moreover, 3-hop KG-50 outperforms theoriginal model by a margin of +0.9%. From table:{7, 8}, it is clear that TuckER (Balaževi´c et al. [2019]) performs better than ComplEx (Trouillon et al.[2016] for the 3-hop scenario for both MetaQA-KG datasets, i.e., Full and 50.
Though these results strengthen the
[Table] [Table] [Table] [Table]
For table:{7,
[Table]
Question-Embedding-Method
[Table]
RoBERTa (Liu et al.
[Table]
XLNet (Yang et al.
[Table]
ALBERT (Lan et al.
[Table]
Longformer (Beltagy et al.
[Table]
SBERT (Reimers and Gurevych [2019]
[Table] [Table]
claim-1, a more comprehensive set of tests may lead to a concrete conclusion.
(e.g., experiments employing a broaderset of hyper-parameters).
According to table-9, in all the cases, SBERT (Reimers and Gurevych [2019]) outperforms RoBERTa (Liu et al.[2019]) used in the original paper creating a new SOTA benchmark which supports the claim-2.
There were some experiments which didn’t work out due to the lack of time.
E.g. Using RelationalTucker3 (Wanget al. [2019a]) and SimplE (Kazemi and Poole [2018]) to test claim-1.
Furthermore, hyper-parameter search couldn’tbe done due to the same reason hence we had to pick the default ones mentioned in the codebase.
All these create aroom for further improvement and experiments.
5.1 What was easy
The paper was straightforward to understand.
The open-sourced codebase helped us get kick-started.
5.2 What was difﬁcult
The structure of the codebase made it difﬁcult to navigate it.
Since the code relied upon different techniques for thetwo datasets, the development of one function that trains different kinds of KG embeddings and another functionthat trains different kinds of QA models for both datasets was difﬁcult.
MetaQA uses LSTM/GRU (Hochreiter and Schmidhuber [1997] / Chung et al.
[2014] while WebQSP uses / RoBERTa (Liu et al.
[2019]) to perform the same taskof generating question embeddings.
Also, training KG embeddings for MetaQA yields ﬁles in the form of NumPyﬁles while WebQSP uses LibKGE (Broscheit et al.
[2020]) for the same purpose which produces LibKGE speciﬁc KGE models.
Reproduction and the extensive study was a bit hard in the beginning as KG embeddings and questionembedding methodology varied for both datasets. After having a couple of virtual meetings with the author and codereview, it became easier to conduct the planned experiments.
The unavailability of hyper-parameters used to train eachmodule increased the experiment cycle by multi-fold.
5.3 Communication with original authors
We had a couple of virtual meetings with Apoorv, the primary author.
Though it was daunting to understand thecodebase due to the reasons mentioned in [-Section:5.
 but-]Section:5.2 with the help and support of the author, it became easier tonavigate the codebase.
5.4 Future Scope
We think that there is a wide range of empirical analysis and experimentation that can be performed for multi-hop QAtask, out of which we’re sharing a few here:
KG embedding compression (Sachan [2020])
Using recent transformer models like Performer (Choromanski et al.
[2020]), Reformer (Kitaev et al.
Gurevych [