EmbedKGQA: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings
Anonymous Author(s)
Afﬁliation
Address email
Reproducibility SummaryScope of Reproducibility
Our work consists of four parts: (1) Reproducing results from Saxena et al.
[2020] (2) Adding more experiments byreplacing the knowledge graph embedding method (3) and exploring the question embedding method using varioustransformer models (4) Verifying the importance of Relation Matching (RM) module.
Based on the code shared bythe authors, we have reproduced the results for the EmbedKGQA method.
We have not performed relation matchingdeliberately to validate point-4.
Methodology
We have used the code provided by Saxena et al.
[2020] with some customization for reproducibility.
In addition tomaking the codebase more modular and easy to navigate, we have made changes to incorporate different transformersin the question embedding module.
Question-Answering models were trained from scratch as no pre-trained modelswere available for our particular dataset.
The code for this work is available on GitHub 1 .
Results
We were able to reproduce the Hits@ 1 to be within ± 2.35% of reported value (in most cases).
Anomalies were observed in 3 cases.
[1] In MetaQA-KG-Full (3-hop) dataset [2] In WebQSP-KG-50 and [3] WebQSP-KG-Full datasets.
From our experiments on the QA model, we have found that a recent transformer architecture, SBERT (Reimers and
Gurevych [2019]) produced better accuracy than the original paper.
Replacing RoBERTa
(Liu et al.
[2019]) with
SBERT (Reimers and Gurevych [2019]) increased the absolute accuracy by ≈ 3.4% in half KG case and ≈ 0.6% in thefull KG case.
(KG: Knowledge Graph, " ≈ ": Approximately)
What was
easy
As the code was open-sourced, we didn’t have to implement the paper giving us the liberty to customize the codebaseto focus on the author’s claim validation, perform extended experiments and explore shared as well as new models.
Inaddition to this, pretrained KG embedding models were shared which helped in the reproduction experiment.
What was difﬁcult
The lack of a comprehensive documentation alongwith missing comments deﬁning functions/classes/attributes etc., made it laborious to review the code and modify it.
In addition to large training times for question answering models the knowledge graph embeddings also required a signiﬁcant amount of computing resources
Communication with
We had a couple of virtual meetings with
Saxena 2 , the primary author of EmbedKGQA
Introduction
Knowledge is the key to question answering task.
Knowledge Graph (KG) is a multi-relational graph consisting ofentities as nodes and relations among them as typed edges.
KGs can accommodate a wide variety of facts, makingthem one of the potential candidates for intelligent decision-making.
Question Answering over KG (KGQA) task aimsto answer natural language queries posed over the KG.
Multi-hop KGQA is a trending topic and has gained tractionfrom both academia and industry recently.
Multi-hop KGQA task involves reasoning over multiple edges of the KGto arrive at the correct answer.
Earlier works on KGs(e.g.
Suchanek et al.
[2007], Google [2013], Lehmann et al.
[2015], Mitchell et al.
[2018])have some element of sparsity, i.e. they do not capture all the facts available in the real world.
Recent research onmulti-hop KGQA has attempted to reduce this sparsity with the help of relevant external textual resources that are notreadily available.
On the other side, KG embeddings have emerged as an effective tool to overcome the KG sparsityby predicting missing links in the KG.
Although effective, KG embeddings have not been explored for the multi-hop
KGQA task.
Saxena et al.
[2020] ﬁlls this gap with the proposed EmbedKGQA method.
This work intends to reproduce and perform an ablation (removing relation matching module) as well as extendedstudy on EmbedKGQA(Saxena et al.[2020]).
EmbedKGQA claims to be the ﬁrst of its kind to use KG embeddingsfor multi-hop KGQA and improves over other state-of-the-art (SOTA) baselines.
Scope of reproducibility
According to Saxena et al.
[2020], using ComplEx (Trouillon et al.
[2016]) KG embeddings signiﬁcantly improves
Hits@1 for multi-hop KGQA task and it has been proved with the help of the results on MetaQA (Zhang et al.
[2018])and WebQSP (Yih et al.
[2016]) datasets.
This reproducibility work tries to test this claim and conducts experimentsas mentioned in table:{2,3} of the original paper.
Section 4.1 contains the corresponding results which support theclaim with some anomalies.
Methodology
The authors of the original paper have open-sourced the code along with the data and pre-trained ComplEx KGembedding models.
We have used the same codebase (commit:5d8fdbd4) and customized it for our purposes.
Inaddition to this, we have added a comprehensive documentation to make it more interpretable.
Moreover, a commandline functionality is also added to easily conﬁgure various transformers models in the training workﬂow.
3.1 Model descriptions
As shown in Figure:1, EmbedKGQA has three modules:
KG
Embedding Module - This module contains a KG embedding model called ComplEx (Trouillon et al.
[2016]) to learn embeddings for all entities in the input KG.
4 pretrained models have been shared by the
author which contains 2 models for MetaQA-KG-{Full, 50} as well as 2 models WebQSP-KG-{Full,50}
dataset.
Details about the dataset are mentioned in section:3.2.2.
Question Embedding Module
: Given a question q, head entity h and set of answer entities A, this module
learns the question embeddings based on the score function deﬁned by the KG embedding method used in 1.
Answer Selection Module: This module uses the outputs of module:1 and 2 to select the ﬁnal answer by scoring the (head entity and question) pair against all possible answers.
The strategy is mentioned in section:{4.4,4.4.1} of the original paper respectively.
3.2 Datasets
There are two datasets used in the original paper.
MetaQA (Zhang et al.
[2018]) and WebQSP (Yih et al.
Bothdatasets have two portions.
(1) KG data (2) QA data.
KG data for both are further divided into two categories.
Using the full KG
Image source: Saxena et al.
[Table] [Table]
[Table]
Experiment-Alias is the name used for the respective datasets in experiments.
γ = Contains all facts that are within 2-hops of any entity mentioned in the questions of WebQSP.
ϕ =
Contains only 50% of the triples (randomly selected without replacement).
ψ = Contains 50% of the edges sampled randomly from fbwq_full.
question is placed between 
 and </s
> tags for all transformers except SentenceTransformer as it takes the inputsentence in its pure form.
The preprocessing used in the original code has been used here.
No additional preprocessinghas been performed from our end.
3.3 Hyper-Parameters
Hyper-parameters used to train the models aren’t explicitly shared in the codebase or the paper, hence we decided touse the default values provided in the codebase 3 to compensate the lack of time.
For reproduction, a pretrained modelshared along with the data was used; ComplEx(Trouillon et al.
[2016]) was used as the knowledge graph embeddingmethod for all the KG types, i.e.,
[Table]
table:3, validate_every
=
The number of train routines before validation for a single epoch
Total
[Table]
excluding early
[Table] [Table]
=
[Table] [Table]
[Table]
For the purpose of reproducibility, hyper-parameters for training MetaQA and WebQSP QA models have been takenfrom section:{MetaQA 5 , WebQuestionsSP 6 } of the original codebase respectively.
For RoBERTa (Liu et al.
[2019]),a pretrained model ’roberta-base’ has been taken from HuggingFace transformers package (Wolf et al.
Otherhyper-parameters are populated by default values in the codebase 3 .
3.4 Experimental setup and code
Experiments have been performed on the NVIDIA DGX-1 server with 8xV100 GPUs, out of which 6 were used inthis work.
The metric used for validating the claims is Hits@1.
According to Wang et al.
[2019b], Hits@
k is theproportion of test triples ranking in the top-k results.
The code for this work is open-sourced on GitHub 1 .
In additionto this, we have shared couple of Docker images 7 for easy kick-starting of experiments without the hassle of settingup the environment.
Following trained models are made available in our Docker image 7 , chosen on the basis of better performance in ourextended study.
• TuckER KG embedding model for Meta-QA-{Full, 50}
• QA models trained using ComplEx as KG embedding model and SBERT mentioned in table:4 as questionembedding model for WebQSP-KG-{Full, 50}
3.5 Computational requirements
This work has been performed on 6 V100-16GB GPUs connected via NVLink.
NVLink reduced multi-GPU trainingtime by 1/4.
The time required for various reproductions are mentioned in table:{5,6}.
3.6 Extended Experiments
Apart from reproducing the results mentioned in the original paper, a couple of extended experiments have beenperformed to ﬁnd answers to the following two questions:
Can recent KG embedding methods like TuckER ( Balaževi´c et al.
[2019]) give higher accuracy on higher
levels of hops,
i
e., 3-hop scenario to be speciﬁc compared to Trouillon et al.
[2016] used in the original
paper?
[Table]
Can other transformer architectures like ALBERT (Lan et al.
[2019]), XLNet (Yang et al.
[2019]), Longformer (Beltagy et al.
[2020]) and SBERT (Reimers and Gurevych [2019]) improve the results on WebQSP(Yih et al.
Details of hyper-parameters used for these experiments are available in our GitHub repository 1 .
Various transformermodels used for experiment-2 are mentioned in table:4.
Results
We report results for reproducibility as well as our extended experiments.
The results of reproduction have a mixednature while the ones for our extended experiments show positive signs to support claim-1, 2.
Detailed discussionabout the results can be found in section:5.
For all tables in this report, bold values indicate better performance.
4.1 Results reproducing original paper
We perform two experiments based on the two datasets introduced in section:3.2.
These experiments provide vitalinformation about the results mentioned in table:{2,5} of the original paper.
The results of the two are reported intable:{5, 6} respectively.
From the results of table:5 in Saxena et al.
[2020] and table:6 in this report, it is evidentthat relation matching(RM) is an important component in multi-hop KGQA when the given KG is considerably large,i.e. {MetaQA, WebQSP} KG-Full; Deﬁnitely, WebQSP-KG-50 also shows improvement in presence of RM but theperformance signiﬁcantly improves when applied to KG-Full setting.
The author of Saxena et al.
[2020] had alsoexpressed the same opinion in one of the virtual meetings.
4.2 Results beyond the original paper
We have conducted two additional experiments from our end to ﬁnd an answer to claim:{1,2}.
The results in table:{7,8}support claim-1 but with a caveat.
On the other hand, values in table:9 improve upon the results reported by theoriginal paper creating a new SOTA baseline.
Additional experiments ingest custom hyper-parameters mentioned inour codebase in absence of the original hyper-parameters.
None of these experiments include the RM module.
Discussion
The reproducibility results from table:{5,6} corroborate the claims mentioned in section:2 to some extent.
The reproduced version is within ± 2 .
4 range (positive value indicates better performance and vice-versa) except for MetaQAKG-Full dataset’s 3-hop and WebQSP-KG-Full scenario which has a signiﬁcant drop of 22.5% and 18.5% respectively.
The absence of RM module has been reported and discussed here 8,9,10 .
For a given question, the RM module uses it’scontext to extract useful information from the available edges present in the KG.
This information is further pluggedinto the answer selection module to select more relevant answers.
Thus, relation matching is a vital component inmulti-hop question answering, especially in KG-Full setting where more number of the edges are present w.r.t.
KG-50setting or any smaller KG w.r.t.
the KG-Full setting.
Results from table:{5,6} corroborates the previous statement.
Moreover, MetaQA-KG-50 3-hop outperforms the original model by a margin of +0.9% without using RM which isan interesting observation.
Apart from one reported anomaly, the reproduced results are pretty close to the original
[Table] [Table]
[Table] [Table]
For table:{7, 8}, ∆ = (TuckER Hits@ k ) - (ComplEx Hits@ k ). [Table]
[Table]
RoBERTa (Liu et al.
[Table]
XLNet (Yang et al.
[Table]
ALBERT (Lan et al.
[Table]
Longformer (Beltagy et al.
[Table]
Reimers and Gurevych [
[Table]
[Table] [Table]
results in case of MetaQA dataset.
Default set of hyper-parameters mentioned in the original codebase(Refer section:3.3) were used in the reproducibility study.
The anomaly in WebQSP-KG-Full,i.e. 18.5% drop bolsters the importanceof RM in KG-Full setting.
The reproduced results for WebQSP-KG-50 are within ± 7 % range.
The use of differenthyper-parameters can be one of the possible answers to this variation.
This value is signiﬁcant but not w.r.t.
WebQSPKG-Full’s drop of 18.5% which again strengthens the importance of RM in KG-Full setting.
As mentioned in 4.1, RMis highly useful when the KG is considerably large.
From table:{7, 8}, it is clear that TuckER (Balaževi´c et al.
[2019]) performs better than ComplEx (Trouillon et al.[2016] for the 3-hop scenario for both MetaQA-KG datasets, i.e., Full and 50.
Though these results strengthen claim1, a more comprehensive set of tests may lead to a concrete conclusion.
(e.g., experiments employing a broader set ofhyper-parameters).
According to table:9, in all the cases, SBERT (Reimers and Gurevych [2019]) outperforms RoBERTa (Liu et al.
[2019])used in the original paper creating a new SOTA benchmark which supports claim-2.
There were some experiments which didn’t work out due to the lack of time.
E.g. Using RelationalTucker3 (Wanget al.
[2019a]) and SimplE (Kazemi and Poole [2018]) to test claim-1.
Furthermore, hyper-parameter search couldn’tbe done due to the same reason hence we had to pick the default ones mentioned in the codebase.
All these create aroom for further experiments and improvements.
5.1 What was easy
The paper was straightforward to understand.
The open-sourced codebase helped us get kick-started.
5.2 What was difﬁcult
The structure of the codebase made it difﬁcult to navigate it.
Since the code relied upon different techniques for the twodatasets, the development of one function that trains different kinds of KG embeddings and another function that trainsdifferent kinds of QA models for both datasets was difﬁcult.
MetaQA uses LSTM/GRU (Hochreiter and Schmidhuber[1997] / Chung et al.
[2014] while WebQSP uses RoBERTa (Liu et al.
[2019]) to perform the same task of generatingquestion embeddings.
Also, training KG embeddings for MetaQA yields ﬁles in the form of NumPy (Harris et al.[2020]) ﬁles while WebQSP uses LibKGE (Broscheit et al.
[2020]) for the same purpose which produces LibKGEspeciﬁc KG embedding(KGE) models.
Reproduction and the extensive study was a bit hard in the beginning as KGEand question embedding methodology varied for both datasets.
After having a couple of virtual meetings with theauthor and code review, it became easier to conduct the planned experiments.
The unavailability of hyper-parametersused to train each module increased the experiment cycle by multi-fold.
5.3 Communication with original authors
We had a couple of virtual meetings with the primary author of Saxena et al.
Though it was daunting tounderstand the codebase due to the reasons mentioned in section:5.2 with the help and support of the author, it becameeasier to navigate the codebase.
5.4 Future Scope
We think that there is a wide range of empirical analysis and experimentation that can be performed for multi-hop QAtask, out of which we are sharing a few here:
KG embedding compression (Sachan [2020])
Using recent transformer models like Performer (Choromanski et al.
[2020]), Reformer (Kitaev et al.
[2020])etc.
for generating question embeddings.
Using low-dimensional hyperbolic KG embeddings (Chami et al.
[2020]) in KG embedding module along
with hyperbolic word embeddings (Dhingra et al.
[2018]) for question embedding module.
A new approach for sentence embedding, SBERT-WK
(Wang and Kuo [2020]) instead of SBERT
(Reimers
and Gurevych [2019]) can be tried out