Q UERY E MBEDDING ON H YPER - RELATIONAL K NOWLEDGE G RAPHS
A BSTRACT
Multi-hop logical reasoning is an established problem in the ﬁeld of representation learning on knowledge graphs (KGs).
It subsumes both one-hop link prediction as well as other more complex types of logical queries.
However, existing algorithms operate only on classical, triple-based graphs, whereas modern KGs often employ a hyper-relational modeling paradigm.
In this paradigm, typed edges may have several key-value pairs known as qualiﬁers that provide ﬁne-grained context for facts.
In queries, this context modiﬁes the meaning of relations, and usually reduces the answer set.
Hyper-relational queries are often observed in real-world
KG applications, and existing approaches for approximate query answering (QA) cannot make use of qualiﬁer pairs.
In this work, we bridge this gap and extend the multi-hop reasoning problem to hyper-relational KGs allowing to tackle this new type of complex queries.
Building upon recent advancements in Graph Neural
Networks and query embedding techniques, we study how to embed and answer hyper-relational conjunctive queries.
Besides that, we propose a method to answer such queries and demonstrate in our experiments that qualiﬁers improve QA on a diverse set of query patterns.
I NTRODUCTION
Query embedding (QE) on knowledge graphs (KGs) aims to answer logical queries using neural reasoners instead of traditional databases and query languages.
Traditionally, a KG is initially loaded into a database that understands a particular query language, e.g., SPARQL.
The logic of a query is encoded into conjunctive graph patterns , variables , and common operators such as joins or unions .
On the other hand, QE bypasses the need for a database or query engine and performs reasoning directly in a latent space by computing a similarity score between the query representation and entity representations 1 .
A query representation is obtained by processing its equivalent logical formula where joins become intersections ( ∧ ), and variables are existentially quantiﬁed ( ∃ ).
A ﬂurry of recent QE approaches (Hamilton et al., 2018;
Ren et al., 2020; Ren & Leskovec, 2020; Kotnis et al., 2020; Arakelyan et al., 2021) expand the range of supported logical operators and graph patterns.
However, all existing QE models work only on classical, triple-based KGs.
In contrast, an increasing amount of publicly available (Vrandeˇci´c & Kr¨otzsch, 2014; Suchanek et al., 2007) and industrial
KGs adopt a hyper-relational modeling paradigm where typed edges may have additional attributes, in the form of key-value pairs, known as qualiﬁers .
Several standardization efforts embody this paradigm, i.e., RDF* (Hartig, 2017) and Labeled Property Graphs (LPG) 2 , with their query languages SPARQL* and GQL, respectively.
Such hyper-relational queries involving qualiﬁers are instances of higher-order logical queries, and so far, there has been no attempt to bring neural reasoners to this domain.
In this work, we bridge this gap and propose a neural framework to extend the QE problem to hyper-relational KGs enabling answering more complex queries.
Speciﬁcally, we focus on logical queries that use conjunctions ( ∧ ) and existential quantiﬁers ( ∃ ), where the function symbols are parameterized with the qualiﬁers of the relation.
This parameterization enables us (cf.
Fig. 1) to answer queries like What is the university U where the one who discovered the law of the photoelectric effect L got his/her BSc.
degree? , which can be written as ?
U : ∃ P. discovered by ( L, P ) ∧ educated at { ( degree : BSc ) } ( P, U ) .
Our contributions towards this problem are four-fold.
First, as higher-order queries are intractable at practical scale, we show how to express such queries in terms of a subset of ﬁrst-order logic (FOL) well-explored in the literature.
Second, we build upon recent advancements in Graph Neural
Networks (GNNs) and propose a method to answer conjunctive hyper-relational queries in latent space.
Then, we validate our approach by demonstrating empirically that qualiﬁers signiﬁcantly improve query answering (QA) accuracy on a diverse set of query patterns.
Finally, we show the robustness of our hyper-relational QA model to a reiﬁcation mechanism commonly used in graph databases to store such graphs on a physical level.
R ELATED W ORK
Query Embedding.
The foundations of neural query answering and query embeddings laid in
GQE (Hamilton et al., 2018) considered conjunctive ( ∧ ) queries with existential ( ∃ ) quantiﬁers modeled as geometric operators based on Deep Sets (Zaheer et al., 2017).
Its further extension,
Q UERY 2B OX (Ren et al., 2020), proposed to represent queries as hyper-rectangles instead of points in a latent space.
Geometrical operations on those rectangles allowed to answer queries with disjunction ( ∨ ) re-written in the Disjunctive Normal Form (DNF).
Changing the query representation form to beta distributions enabled B ETA E (Ren & Leskovec, 2020) to tackle queries with negation ( ¬ ).
Another improvement over Q UERY 2B OX as to answering entailment queries was suggested in
E M QL (Sun et al., 2020) by using count-min sketches.
The other family of approaches represents queries as Directed Acyclic Graphs (DAGs).
MPQE (Daza & Cochez, 2020) assumes variables and targets as nodes in a query graph and applies an R-GCN (Schlichtkrull et al., 2018) encoder over it.
It was shown that message passing demonstrates promising generalization capabilities (i.e., when training only on 1-hop queries and evaluating on more complex patterns).
Additional gains are brought when the number of R-GCN layers is equal to the graph diameter.
Treating the query DAG as a fully-connected (clique) graph,
B I QE (Kotnis et al., 2020) applied a Transformer encoder (Vaswani et al., 2017) and allowed to answer queries with multiple targets at different positions in a graph.
Finally, CQD (Arakelyan et al., 2021) showed that it is possible to answer complex queries without an explicit query representation.
Instead, CQD decomposes a query in a sequence of reasoning steps and performs a beam search in a latent space of KG embeddings models pre-trained on a simple 1- hop link prediction task.
A particular novelty of this approach is that no end-to-end training on complex queries is required, and any trained embedding model of the existing abundance (Ali et al., 2020; Ji et al., 2020) can be employed as is.
Still, all of the described approaches are limited to triple-based KGs while we extend the QE problem to the domain of hyper-relational KGs.
As our approach is based on query graphs, we investigate some of MPQE observations as to query diameter and generalization (see also Appendix G).
Hyper-relational KG Embedding.
Due to its novelty, embedding hyper-relational KG is a ﬁeld of ongoing research.
Most of the few existing models are end-to-end decoder-only CNNs (Rosso et al., 2020; Guan et al., 2020) limited to 1-hop link prediction, i.e., embeddings of entities and relations are stacked and passed through a CNN to score a statement.
On the encoder side, we are only aware of S TAR E (Galkin et al., 2020) to work in the hyper-relational setting.
S TAR E extends the message passing framework of CompGCN (Vashishth et al., 2020) by composing qualiﬁers and aggregating their representations with the primary relation of a statement.
Inspired by the analysis of (Rosso et al., 2020), we take a closer look at comparing hyper-relational queries against their reiﬁed counterparts (transformed into a triple-only form).
We also adopt
S TAR E (Galkin et al., 2020) as a basic query graph encoder and further extend it with attentional aggregators akin to GAT (Veli ˇ ckovi ´ c et al., 2018).
H YPER - RELATIONAL K NOWLEDGE G RAPHS AND Q UERIES
Deﬁnition 3.1 (Hyper-relational Knowledge Graph) .
Given a ﬁnite set of entities E , and a ﬁnite set of relations R , let Q = 2 ( R×E ) .
Then, we deﬁne a hyper-relational knowledge graph as G =
( E , R , S ) , where S ⊂ ( E × R × E × Q ) is a set of (qualiﬁed) statements.
For a single statement s = ( h, r, t, qp ) , we call h, t ∈ E the head and tail entity, and r ∈ R the (main) relation .
This also indicates the direction of the relation from head to tail.
The triple ( h, r, t ) is also called the main triple .
qp =
{ q 1 , . . . }
= { ( qr 1 , qe 1 ) , . . . }
⊂ R ×
E is the set of qualiﬁer pairs , where { qr 1 , qr 2 , . . .
} are the qualiﬁer relations and { qe 1 , qe 2 , . . . }
the qualiﬁer values .
Hyper-relational KGs extend traditional KGs by enabling to qualify triples.
In this work, we solely use hyper-relational KGs and hence use ”KG” to denote this variant.
The qualiﬁer pair set provides additional information for the semantic interpretation of the main triple ( h, r, t ) .
For instance, consider the statement ( AlbertEinstein , educated at , ETHZurich , { ( degree , BSc ) } ) .
Here, the qualiﬁer pair ( degree , BSc ) gives additional context on the base triple ( AlbertEinstein , educated at , ETHZurich ) (also illustrated on Fig. 1).
Note that this statement can equivalently be written in ﬁrst order logic (FOL).
Speciﬁcally, we can write it as a statement with a parameterized predicate educated at { ( degree : BSc ) }
( AlbertEinstein , ETHZurich ) .
Then, we note that Q is a ﬁnite set, meaning that also the number of different parameterizations for the predicate is ﬁnite, which means that this becomes a ﬁrst order logic statement.
In this formalism, the KG is the conjunction of all FOL statements.
Possible monotonicity concerns are discussed in Appendix E.
We deﬁne a hyper-relational query on a hyper-relational KG as follows:
Deﬁnition 3.2 (Hyper-relational Query) .
Let V be a set of variable symbols, and TAR ∈ V a special variable denoting the target of the query.
Let E + = E
(cid:93)V .
Then, any subset Q of ( E + ×R×E + × Q ) is a valid query if its induced graph 1) is a directed acyclic graph, 2) has a topological ordering in which all entities (in this context referred to as anchors) occur before all variables, and 3) TAR must be last in the topological orderings.
The answers A G ( Q ) to the query Q are the entities e ∈ E that can be assigned to TAR , for which there exist a variable assignment for all other variables occurring in the query graph, such that the instantiated query graph is a subgraph of the complete graph G .
Academic degree ( P512 ): Bachelor ( Q787674 )
Academic major ( P812 ): Mathematics ( Q853077 )
Academic degree ( P512 ): Doctorate ( Q849697 )
Academic major ( P812 ): Physics ( Q413 )
B. Hyper-relational facts e d u c a t e d a t e d u c a t e d a t
Albert Einstein
Q
ETH Zurich University of Zurich
P 69 P
Q206702 Q11942 ?
var educatedAt
W q q φ r
W in Σ ?
target r q r1 q v1 q r2 q v2 o ϕ q ϕ q
∑ P 69 P 512 Q 849697 P 812 Q 853077 Q 206702 W q γ ϕ r r q r1 q e1 q r2 q e2 o ϕ q ϕ q ∑ P 69 P 512 Q 849697 P 812 Q 853077 Q 206702 W q h q γ ϕ r
∑ s
W λ (r) s r q r q v q r q v
PAD
PAD s r q r q v q r q v
PAD PAD
V
FC
S t a r E
E n c o d e r
T r a n s f o r m e r
P oo l i n g
E
Initial EmbeddingMatrices R V
StarE Encoder
R
V
Updated EmbeddingMatrices
Updated EmbeddingMatrices R V r q r q v q r q v o ϕ q ϕ q ∑
P
P
Q 849697 P
Q 853077 Q
W q γ
[Table]
r ∑ s
Q
W λ (r) s r q r q v q r q v
PAD PAD
FC
S t a
[Table]
E
E n c o d e r
T r a n s f o r m e r
P oo l i n g s r o q r1 q v1 q r2 q v2 <PAD> <PAD>
Q u e r y
L i n e a r i z a t i o n
V
R ... r s PAD
Transformer + Pooling + Fully Connected
QUERY V ⊙ ( ) → sigmoid
Q
Index Select q r1 q v1 q r2 q v2 PAD
Distribution over .. r q r1 q v1 q r2 q v2 o ϕ q ϕ q ∑
P 69 P 512 Q 849697 P 812 Q 853077 Q
W q γ ϕ r
∑ s W λ (r)
Q r q r1 q v1 q r2 q v2 o ϕ q ϕ q ∑
P 69 P 512 Q 849697 P 812 Q 853077 Q
W q γ ϕ r
∑ s W λ (r)
Q r q r1 q v1 q r2 q v2 o ϕ q ϕ q ∑ P 69 P 512 Q 849697 P 812 Q 853077 Q 206702 W q γ ϕ r
∑ s W λ (r) Q937 r o d i r e c t e d b y D a r k K n i g h t
ϕ
∑ s W λ (r) Nolan
Photoelectric effect degree
BSc h q q r q e h r,q r = φ r ( r , h q ) 𝛾 r m h→t,r,q
Problem Deﬁnition.
Given the incomplete KG G (part of the not observable complete KG ˆ G ) and a query Q. Rank all entities in G such that answers to the query, if it were asked in the context of the complete KG ˆ G , are at the top of the ranking.
Since the given KG is not complete, we cannot solve this problem directly as a graph matching problem as usually done in databases.
Instead, we compute a latent representation of the query, such that it is close to the embeddings of entities which are the correct answers to the query.
M ODEL D ESCRIPTION
Our model is not trained directly on the KG, but rather with a set of queries sampled from it (see Section 5.1).
Hence, to describe our model, we consider a query Q ⊂ ( E + ×R×E + × Q ) , and describe how we learn to represent it.
Our model can be seen as a combination of MPQE (Daza & Cochez, 2020) and StarE (Galkin et al., 2020) with a CompGCN (Vashishth et al., 2020) architecture.
Our model learns representations for entities and the special symbols ( { VAR , TAR } ), ˆ E ∈
R ( |E| +2)
× d , and relation representations R ∈ R 2 |R|× d , where d is a hyper-parameter indicating the embedding dimension.
There are two representations for each relation.
For a statement s = ( h, r, t, qp ) ∈ Q , one is used for computing “forward” messages from h → t , and the other one to compute “backward” messages from h ← t , i.e., inverse relation r − 1 .
We encode the query graph using a sequence of S TAR E layers (Galkin et al., 2020).
First, we select from ˆ E the necessary embeddings.
These correspond to the ones for all entities in the query, VAR if the query has internal variables, and TAR for targets.
The embeddings are then put in E , which also contains one copy of VAR for each unique variable encountered in Q .
A layer enriches the entity and relation representations E , R to E (cid:48) , R (cid:48) as follows:
For each query statement s = ( h, r, t, qp ) ∈ Q , we compute messages m h → t , r , qp and m h ← t , r − 1 , qp .
For brevity, we only describe m h → t , r , qp , the other direction works analogously.
Qualiﬁer Representation.
We begin by composing the representations of the components of each qualiﬁer pair q i
= ( qr i , qe i ) ∈ qp into a single representation: h q i = γ q ( E
[ qe i ] , R [ qr i ]) , where γ q denotes a composition function (Vashishth et al., 2020), e.g., the Hadamard product, and we use X [ y ] to indicate the representation of entity y projected in vector space
X e.g qe i in E .
Next, we aggregate all qualiﬁer pair representations for qp , as well as the representation of the main relation r into a qualiﬁed relation representation using an aggregation function φ r : h r , qp = φ r ( R [ r ] , { h q i } q i ∈ qp ) .
For φ r we experiment with a simple sum aggregation, and an attention mechanism.
The Message-Passing Step.
To obtain a message m h → t , r , qp , the qualiﬁed relation representation is further composed with the source entity representation E [ h ] using another composition function γ r .
The result gets linearly transformed by W → , a layer speciﬁc trainable parameter:
m h → t , r , qp = W
→ γ r ( E [ h ] , h r , qp ) .
For inverse relations, we use a separate weight W ← .
Next, we aggregate all incoming messages a t , → = φ m ( { m h → t (cid:48) , r , qp | ( h, r, t (cid:48) , qp ) ∈ Q, t = t (cid:48) } )
us-
ing a message aggregation function φ m , e.g., a (weighted) sum.
Besides computing these message aggregates in each direction, we also compute a self-loop update a e , (cid:9) =
W (cid:9) γ r ( e , r (cid:9) ) , where W (cid:9) , r (cid:9) are trainable parameters.
The updated entity representation is then obtained as the average over both directions and the self loop, with an additional activation σ applied to it:
E (cid:48)
[ e ] = σ (cid:0) 13 ( a e , (cid:9) + a e , → + a e , ← ) (cid:1)
Finally, the relation representations are updated by a linear transformation R (cid:48)
[ r ] = W r R
[ r ] , where W r is a layer speciﬁc trainable weight.
Query Representation.
After applying multiple S TAR E layers, we obtain enriched node representations E ∗ for all nodes in the query graph.
As ﬁnal query representation x Q , we aggregate all node representations of the query graph, x Q = φ q ( { E ∗ [ e ] | e ∈ E + ∧ (( e, r, t, qp ) ∈
Q ∨ ( h, r, e, qp ) ∈ Q ) } ) , with φ q denoting an aggregation function, e.g., the sum.
Alternatively, we only select the ﬁnal representation of the unique target node, x Q = E ∗ [ TAR ] .
To score answer entity candidates, we use the similarity of the query representation and the entity representation, i.e., score ( Q, e ) = sim ( x Q , E ∗ [ e ]) , such as the dot product, or cosine similarity.
We designate the described model as S TAR QE ( Query Embedding for RDF Star Graphs ) since
RDF* is one of the most widely adopted standards for hyper-relational KGs.
E XPERIMENTS
In this section, we empirically evaluate the performance of QA over hyper-relational KGs.
We design experiments to tackle the following research questions: RQ1) Does QA performance beneﬁt from the use of qualiﬁers?
RQ2)
What are the generalization capabilities of our hyper-relational QA approach?
RQ3) Does QA performance depend on the physical representation of a hyper-relational
KG, i.e., reiﬁcation ?
5. D ATASET
Existing QE datasets based on Freebase (Toutanova & Chen, 2015) and NELL (Carlson et al., 2010) are not applicable in our case since their underlying KGs are strictly triple-based.
Thus, we design a new hyper-relational QE dataset based on WD50K (Galkin et al., 2020) 5 comprised of Wikidata statements, with varying numbers of qualiﬁers.
WD50K-QE.
We introduce hyper-relational variants of 7 query patterns commonly used in the related work (Hamilton et al., 2018;
Ren et al., 2020; Sun et al., 2020; Arakelyan et al., 2021) where each edge can have [0 , n ] qualiﬁer pairs.
The patterns contain projection queries (designated with -p ), intersection queries (designated with -i ), and their combinations (cf.
Fig. 3).
Note that the simplest 1p pattern corresponds to a well-studied link prediction task.
Qualiﬁers allow more ﬂexibility in query construction, i.e., we further modify formulas by conditioning the existence of qualiﬁers and their amount over a particular edge.
We include dataset statistics and further details as to dataset construction in Appendices B and D.
These patterns are then translated to the SPARQL* format (Hartig, 2017) and used to retrieve materialized query graphs from speciﬁc graph splits.
Following existing work, we make sure that validation and test queries contain at least one edge unseen in the training queries, such that evaluated models have to predict new links in addition to QA.
As we are in the transductive setup where all entities and relation types have to be seen in training, we also ensure this for all entity and relation types appearing in qualiﬁers.
Due to the (current) lack of standardized data storage format for hyper-relational (RDF*) graphs in graph databases, particular implementations of RDF* employ reiﬁcation , i.e, transformation of hyper-relational statements as deﬁned in Section 3 to plain triples.
Reiﬁcation approaches (Frey et al., 2019) introduce auxiliary virtual nodes, new relation types, turn relations into nodes, and might dramatically change the original graph topology.
An example of a standard RDF reiﬁcation (Brickley et al., 2014) in Fig.
4 transforms an original hr-2p query with three nodes, two edges, and two qualiﬁer pairs into a new graph with nine nodes and eight edges with rigidly deﬁned edge types.
However, the logical interpretation of a query remains the same.
Therefore, we want hyper-relational QE models to be robust to the underlying graph topology.
For this reason, we ship dataset queries in both formats, i.e., hyper-relational RDF* and reiﬁed with triples, and study the performance difference in a dedicated experiment.
5. E VALUATION P ROTOCOL
In all experiments, we evaluate the model in the rank-based evaluation setting.
Each query is encoded into a query embedding vector x q ∈ R .
We compute a similarity score sim ( x q , E [ e ]) between the query embedding and the entity representation for each entity e ∈ E .
The rank r ∈ N + of an entity is its position in the list of entities sorted decreasingly by score.
We compute ﬁltered ranks (Bordes et al., 2013), i.e., while computing the rank of a correct entity, we ignore the scores of other correct entities.
We resolve exactly equal scores using the realistic rank (Berrendorf et al., 2020), i.e., all entities with equal score obtain the rank of the average of the ﬁrst and last position.
Given a set of individual ranks { r i } ni =1 , we aggregate them into a single-ﬁgure measure using several different aggregation measures: The Hits@k (H@k) metric measures the frequency of ranks at most k ∈ N + , i.e., H @ k = 1 n (cid:80) I
[ r i ≤ k ] , with I denoting the indicator function.
H@k lies between zero and one, with larger values showing better performance.
The Mean Reciprocal Rank (MRR) is the (arithmetic) mean over the reciprocal ranks, i.e., MRR = 1 n (cid:80) r − 1 i , with a value range of (0 , 1] , and larger values indicating better results.
It can be equivalently interpreted as the inverse harmonic mean over all ranks, and thus is stronger inﬂuenced by smaller ranks.
Since the size of the answer set of queries varies greatly, 6 queries with large answer sets would strongly inﬂuence the overall score compared to very speciﬁc queries with a single entity as answer.
In contrast to rank-based evaluation in existing QE work, we propose to weight each rank in the aforementioned averages by the inverse cardinality of the answer set to compensate for their imbalance.
Thereby, each query contributes an equal proportion to the ﬁnal score.
5. H YPER - RELATIONAL QA
As we are the ﬁrst to introduce the problem of hyper-relational QA, there is no established baseline available at the time of writing.
Hence, we compare our method to several alternative approaches.
[Table]
Implementation.
We implement S TAR QE 7 and other baselines in PyTorch (Paszke et al., 2019)
(MIT License).
We run a hyperparameter optimization (HPO) pipeline on a validation set for each model and report the best setup in the Appendix J. All experiments are executed on machines with single GTX 1080 Ti or RTX 2080 Ti GPU and 12 or 32 CPUs.
Triple-Only For the ﬁrst experiment, we remove all qualiﬁers from the hyper-relational statements in the query graph leaving the base triples ( h, r, t ) .
Thus, we isolate the effect of qualiﬁers in answering the queries correctly.
Note that in effect, this is similar to the MPQE approach, but with a better GNN and aggregation.
To do this, we have to retain the same queries as in other experiments, but remove the qualiﬁers upon loading.
The set of targets for these queries remains unchanged, e.g., in the hyper-relational query from Fig.
1 we would drop the (degree:BSc) qualiﬁer but still have only ETHZurich as a correct answer.
Reiﬁcation.
For the second setting, we convert the hyper-relational query graph to plain triples via reiﬁcation (see Section 5.1).
The effects of such a transformation include the addition of two extra nodes per triple in the query, to represent blank nodes and predicates (more details in Appendix A).
Here, we investigate whether S TAR QE is able to produce the same semantic interpretation of a topologically different query.
Note that, while conceptually the default relation enrichment mechanism of S TAR QE resembles singleton property reiﬁcation (Frey et al., 2019), its semantic interpretation is equivalent to standard RDF reiﬁcation.
Zero Layers.
To measure the effect of message passing, we consider a model akin to bag-of-words , which trains entity and relation embeddings without message passing before graph aggregation.
Oracle.
If we take away the qualiﬁer information, we could compare with several QE approaches (e.g. GQE, MPQE, Query2box, BetaE, EmQL, and CQD).
The Oracle setup, is not just an upper bound to these, but to all possible, non-hyper-relational QE models.
It simulates the best possible
QE model that has perfect link prediction and ranking capabilities, but without the ability to use qualiﬁer information.
Using this setting we investigate the impact of qualiﬁer information on our queries.
More details are in Appendix I.
Discussion.
The results shown in Table 1 indicate that S TAR QE is able to tackle hyper-relational queries of varying complexity.
That is, the performance on complex intersection and projection queries is often higher than that of a simple link prediction ( hr-1p ).
Particularly, queries with intersections ( − i ) demonstrate outstanding accuracy, but it should be noted here that also the Oracle setting performs very well in this case.
Importantly, MRR values are relatively close to Hits@ which means that more precise measures like Hits@3 and Hits@1 retain good performance (we provide a detailed breakdown in Appendix J).
[Table]
To investigate if this performance could be attributed to the impact of qualiﬁers
[Table]
we run a TripleOnly and Oracle setup.
These experiments show that for some query patterns, qualiﬁers play an important role.
Without them, we would not be able to get good results for the 2p , 3p , 2i-1p and 1p-2i queries.
The reason that the Oracle cannot answer these well is that despite its access to the test set, the mistakes it makes accumulate when there are more hops akin to beam search (recall that the Oracle does not have access to qualiﬁers and considers all edges with a given relation) .
For queries that only involve one reasoning step, we notice that the Oracle can, and hence a normal link predictor might be able, to perform very well.
When more paths intersect ( 2i and 3i ), the chance of making a mistake goes down.
This observation is similar what can be seen in e.g., CQD (Arakelyan et al., 2021), and can be ascribed to each of the different paths constraining the possible answer set, while cancelling out mistakes.
More experiments on qualiﬁers impact are reported in Appendix H.
We observe a comparative performance running the Reiﬁcation baseline.
It suggests that our QE framework is robust to the underlying graph topology retaining the same logical interpretation of a complex query.
We believe it is a promising sign of enabling hyper-relational QA on a wide range of physical graph implementations.
Finally, we ﬁnd that message passing layers are essential for maintaining high accuracy as the Zero
Layers baseline lags far behind GNN-enabled models.
One explanation for this observation can be that without message passing, variable nodes do not receive any updates and are thus not ”resolved” properly.
To some extent counter-intuitively, we also observed that it does not make a large difference whether relation embeddings are included in the aggregation or not.
Relatively high performance on 1p, 2i, 3i queries can be explained by their very speciﬁc star-shaped query pattern which is essentially 1-hop with multiple branches joining at the center node.
5. G ENERALIZATION
Following the related work, we experiment with how well our approach can generalize to complex query patterns if trained on simple ones.
Note that below we understand all query patterns as hyperrelational, i.e., having the hr- preﬁx.
There exist several styles for measuring generalization in the literature that we include in the experiment:
Q2B-like.
The style is used by Q UERY 2B OX (Ren et al., 2020) and assumes training only on 1p,2p,3p,2i,3i queries while evaluating on two additional patterns 2i-1p, 1p-2i .
EmQL-like.
The other approach proposed by E M QL (Sun et al., 2020) employs only 1p, 2i patterns for training, using ﬁve more complex ones for evaluation.
MPQE-like.
The hardest generalization setup used in MPQE (Daza & Cochez, 2020) allows training only on 1p queries, i.e., vanilla link prediction, while all evaluation queries include unseen intersections and projections.
MPQE-like + Reif.
To measure the impact of reiﬁcation on generalization, we also run the experiment on reiﬁed versions of all query patterns.
Similarly to MPQE-like , this setup allows training only on 1p reiﬁed pattern and evaluates the performance on other, more complex reiﬁed patterns.
Discussion.
Table 2 summarizes the generalization results.
As reference, we include the S TAR QE results in the no-generalization setup when training and evaluating on all query patterns.
Generally, we observe that all setups generalise well on intersection queries ( -i ) even when training in the most restricted ( 1p ) mode.
The Q2B-like regime demonstrates appealing generalization capabilities on ( 2i-1p, 2p-1i ), indicating that it is not necessary to train on all query types.
However, moving to a ﬁne-grained study of most impactful patterns, we ﬁnd that projection ( -p ) patterns are rather important for generalization, as both EmQL and MPQE styles dramatically fall behind in accuracy, especially in the MRR metric indicating that higher precision results are impaired the most.
The MPQE style is clearly the hardest when having only one training pattern.
The higher results on intersection ( -i ) patterns can be explained by a small cardinality of the answer set, i.e., qualiﬁers make a query very selective with very few possible answers.
Finally, it appears that reiﬁcation ( MPQE+ Reif ) impedes generalization capabilities and overall accuracy according to the MRR results.
This can be explained by the graph topologies produced when reifying complex queries, and training only on 1p is not sufﬁcient.
L IMITATIONS & F UTURE W ORK
In this section, we discuss limitations of the current work, and future research directions.
The ﬁrst limitation of our work is that we do not allow literal values like numbers, text, and time in our graph.
This means that we cannot, for example, handle queries asking for people born in the year 1980.
These and more complex values can be incorporated as node features (Wilcke et al., 2020).
Secondly, more logical operators, such as negation (which could be included as a qualiﬁer), disjunctions, cardinality constraints, etc. can be considered.
In this work, we only allow variables in the head and tail positions.
Nonetheless, one can also formulate queries with variables in more diverse positions, e.g., qualiﬁer value or main relation.
Moreover, the current work allows for different query shapes compared to prior work because queries are not limited to DAGs (see Appendix F for details).
However, our work does not allow all shapes.
Speciﬁcally, it is currently unclear how queries with cycles would behave.
Another future direction can be found in the many operators which query languages like SPARQL have.
For example, queries including paths, aggregations, sub-queries, ﬁlters on literals, etc.
Further research work is required towards explainability.
Currently, our system does not provide explanations to the answers.
An initial direction is to analyse the intermediate values in variable positions which can be used as explanations, but likely need to be explicitly trained to behave that way.
Finally, an interesting research direction is the use of our approach for the creation of query plans.
C ONCLUSION
In this work, we have studied and addressed the extension of the multi-hop logical reasoning problem to hyper-relational KGs.
We addressed the theoretical considerations of having qualiﬁers in the context of QA and discussed the effects it can have on it, such as cardinality of the answer set.
We proposed the ﬁrst hyper-relational QE model, S TAR QE, based on a GNN encoder to work in this new setup.
We introduced a new dataset, WD50K-QE, with hyper-relational variants of 7 well studied query patterns and analysed the performance of our model on each of them.
Our results suggest that qualiﬁers help in obtaining more accurate answers compared to triple-only graphs.
We also demonstrate the robustness of our approach to structural changes involved in the process of reiﬁcation.
Finally, we evaluate the generalisation capabilities of our model in all settings and ﬁnd that it is able to accurately answer unseen patterns.
Reproducibility Statement.
The experimental setup and implementation details are described in
Section 5.
We elaborate on the dataset construction process in Appendices B and C. All hyperparameters are listed in Table 6.
The complete source code is available for reviewers in the supplementary material, and will be made available open source.
More experimental evidence is provided in Appendix H and more detailed metrics per query pattern are listed in Appendix J.
Ethics Statement.
The assumption of our approach as well as related techniques is that the data used for training solely contains true facts.
If this data is (intentionally) biased or erroneous to start with, further biased or wrong information will be derived, which could lead to harmful conclusions and decisions.
Besides, even if the input data to the system is correct, the system could due to its imperfections still derive incorrect answers to queries, which might, again lead to wrong conclusions.
Hence, users of this technology must be made aware that answers to their queries are always approximate, and this technology ought not to be used in a process where correctness is critical. Acknowledgements.
This work has been funded by the Elsevier Discovery Lab and the German
Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A.
The authors of this work take full responsibilities for its content.
For performing the experiments the authors made use of the LISA system provided by SurfSara and the DAS cluster (Bal et al., 2016).
A PPENDIX A T YPES OF G RAPHS
In this section we discuss in further detail the different approaches to break free from the restriction of pairwise relations.
The intuition how those approaches are different is presented in Fig.
Hyper-relational Graphs KGs used in our work are a subset of hyper-relational graphs.
In general, hyper-relational graphs can also contain literal values, like integers and string values as qualiﬁer values.
Labeled property graphs are another example of a subset of hyper-relation graphs, but they typically only allow literals as qualiﬁer values.
The Wikidata model (Erxleben et al., 2014) and
RDF* (Hartig, 2017) allows for both literals and entities as values.
Many implementations of RDF* do make the monotonicity assumption, which we also made in the paper.
Some will also merge statements in case they have the same main triple.
The resulting statement will have the same main triple, but as qualiﬁer information, the union of the sets of qualiﬁer pairs of the original statements.
Hypergraphs Hypergraphs are another type of graphs, initially proposed by Berge (Berge, 1984), where hyperedges link one or more vertices.
These hyperedges group together sets of nodes in a relation, but lose information of the particular roles that entities play in this relation.
Besides, each different composition of elements in the relation introduces a new hyperedge type, causing a combinatorial explosion of these types.
Reiﬁed Graphs Hyper-relational and reiﬁed graphs have equivalent expressive power compared to labelled directed graphs.
The reason is that we can deﬁne a bijection between these graphs.
However, depending on the use case, the different graph types have beneﬁts.
When we convert a hyper-relational graph to an RDF graph, we end up with what is called the reiﬁed graph.
There are multiple approaches on how to perform this conversion.
In our work, we use Standard RDF Reiﬁcation 8 .
It introduces new, so called blank nodes to represent statements with all parts of a statement attached.
This allows for all information to exist on the same level.
One of the disadvantages to this approach is the addition of auxiliary nodes, which heavily affect the structure of the graph and quickly inﬂate the graph size.
B T HE WD50K-QE D ATASET
In this section, we describe the generation of the hyper-relational queries that we use for training, validation and testing of our models.
We start with WD50K (Galkin et al., 2020), which is a
WikiData-based dataset created for hyper-relational link prediction.
This dataset already provides with train, validation and test splits, each containing a selection of hyper-relational triples.
It is publicly available by the authors, in CSV format.
In order to utilise it for our work, we converted it from CSV to RDF*.
The overall pipeline is presented on Fig.
To generate the queries, we hosted the converted dataset on a graph database with support for hyper-relational data.
For our use-case, we use anzograph 9 .
We utilize 3 named graphs 10 : triple train , triple validation , and triple test , to prevent validation and test set leakage.
In the evaluation of approximate QA, it is common to have queries with at least one unseen edge from the test set, which is also applied upon our query generation process.
We ensure that:
• For the training set, all statements in a query come from the triple train set (only).
• For the validation set, one statement comes from the triple validation and the other edge(s) come from either triple train or triple validation , and [Table] • For the test set, one statement comes from triple test , and the other statement from any of triple train , triple validation , and triple test
In order to generate the query data splits, we constructed SPARQL* queries that correspond to a speciﬁc pattern.
We call these higher order queries formulas .
Using these formulas to generate queries has several advantages compared to other approaches found in related work.
In the related work, we encounter sampling using techniques such as random walks to create queries which have the shape according to the pattern.
With our approach, we have more control over what the samples are, and thus a ﬁner control of the inputs to our experiments.
Moreover, we can also ensure that we do not sample with replacement, we are not biased towards speciﬁc high degree nodes, and at the same time guarantee that we do not sample queries that are isomorphic with each other.
A ﬁnal beneﬁt is that with our approach, we can immediately retrieve all answers for a query, instead of only one.
One ﬁnal issue we encountered had to do with the nodes that have a high in-degree in our dataset.
Their existence causes a skewing of the distribution of correct answers to queries, and has to be resolved.
In Appendix C, we describe this issue and explain our approach to overcome it.
As a result of this procedure, we obtain all queries of the given shapes, and are guaranteed that there are no duplicates nor isomorphism among the queries.
The queries used in this paper are such that every edge in the query has one qualiﬁer, which means that, in the knowledge graph, the same edge has at least one, but possibly more qualiﬁers.
The amount of queries for the different patterns can be found in Table 3.
C I SSUES S AMPLING Q UERIES WITH J OINS IN A G RAPH WITH H IGH D EGREE N ODES
In our query generation step, we exclude high in-degree nodes as a target for speciﬁc queries.
In this section, we explain why that choice was made.
We will focus on the 3i pattern.
The same argument holds for other patterns with joins, like 2i, 2i-1p, 1p-2i .
The queries are randomly sampled from all possible queries that can be formed by matching the pattern with the data graph.
For the 3i pattern, a node with an in-degree of n , results in (cid:0) n 3 (cid:1) different queries, with that node as a target.
The problem is that if we randomly sample our queries, the answer would usually be one of the highest degree nodes.
Concretely, in our graph data, the highest observed in-degree is 4,424, which would result in 14,421,138,424 possible queries with the corresponding node as an answer.
If we generated all possible queries, we would end up with 38,011,148,464 different ones.
So, the node with the highest in-degree is already responsible for 38% of the queries, meaning that system answering 3-i queries that are randomly sampled, would get 38% correct by always giving that answer.
Moreover, if we look at the Hits@10 metric, we would score 93%, by always predicting the same ranking (the top 10 in descending frequency).
Note that existing query embedding models have been evaluated like this in the past, ignoring this data issue.
Hence, we decided to make the task harder by removing these high degree nodes for queries with joins.
That is, by putting the threshold at an in-degree of 50, we remove the 623 nodes with the highest in-degree which would have represented 99.9% of the possible answers for the 3i queries without this modiﬁcation.
After ﬁltering, the baseline of always predicting the same ranking for the randomly sampled queries, will result in a Hits@10 of 3% in the best case.
For other query shapes where a join is involved, the situation is similar, but less pronounced.
D S IZE OF THE A NSWER S ET OF A Q UERY WITH MORE Q UALIFIERS
When we have a query with or without qualiﬁers, and we add more qualiﬁers, the number of answers to this query can only become smaller.
Intuitively, this happens because the query becomes more speciﬁc.
In this section we prove this intuition correct.
Note that proving this requires monotonicity, as we deﬁned in the paper.
Without this assumption - for example allowing non-monotonic qualiﬁer relations - would result in losing the guarantee that the answer set becomes smaller.
Given a query Q , and a query Q that is the same, except for one set of qualiﬁer pairs of Q which can have extra pairs, then the answers to Q are a subset of the answers to Q .
Formally:
Theorem D.1.
Given a KG G =
( E , R , S ) , where S ⊂ ( E × R × E × Q ) , a query
Q = { ( h 1 , r 1 , t 1 , qp 1 ) , . . .
( h n , r n , t n , qp n ) } , and a second query
Q = { ( h 1 , r 1 , t 1 , qp 1 ) , . . .
( h n , r n , t n , qp n ) } , where ∃ !
k : qp k ⊆ qp k and ∀ x (( x ∈ [1 , . . .
n ] ∧ x (cid:54) = k ) →
qp x = qp x ) .
Then, A G ( Q ) ⊆
A G ( Q ) .
Proof.
Assuming symbols as deﬁned in the theorem, we show that ∀ a : a ∈ A G ( Q ) =
⇒ a ∈ A G ( Q ) .
If a is an answer to Q , then its associated variable substitution v is such that ( v ( h k ) , r k , v ( t k ) , qp k ) ∈ G .
From monotonicty (see Appendix E) , and given qp k ⊆ qp k
we then know that ( v ( h k ) , r k , v ( t k ) , qp k ) ∈ G .
And hence, using the same variable substitution, a is an answer for Q .
Corollary D.1.
By induction, given a query Q , and a query Q that is the same, except for any set of qualiﬁer pairs of Q which can have extra pairs, then the answers to Q are a subset of the answers to Q .
Corollary D.2.
As a special case, given a query Q , and a query Q that is the same, except that Q does not have qualiﬁers, while Q can have qualiﬁer pairs on its statements, then the answers to Q are a subset of the answers to Q . [Figure]
E O N M ONOTONICITY
In this work, we restrict the qualiﬁers to respect monotonicity in the KG in the following sense:
[Equation]
This implies that if a qualiﬁed statement in the KG has a set of qualiﬁer pairs, then the KG also contains the qualiﬁed statement with any subset thereof.
As a result, some types of qualifying information cannot be used, e.g., a qualiﬁer indicating that a relation does not hold (i.e., negation).
This breaks monotonicity since the existence of such a statement would further imply the existence of that statement without the qualiﬁer, leading to contradiction.
F R ELAXING THE R EQUIREMENTS ON Q UERIES
In the paper, we limited our queries with the same limitations as done in prior work.
Here we will discuss why not all of these restrictions are needed for our work, and how our evaluation already includes some of these more general cases.
As a reminder the deﬁnition of our queries is as follows:
Deﬁnition F.1 (Hyper-relational Query) .
Let V be a set of variable symbols, and TAR ∈ V a special variable denoting the target of the query.
Let E + = E
(cid:93)V .
Then, any subset Q of ( E + ×R×E + × Q ) is a valid query if its induced graph
1) is a directed acyclic graph, 2) has a topological ordering in which all entities (in this context referred to as anchors) occur before all variables, and 3) TAR must be last in the topological orderings.
The main reason why we deal with more general queries is because our query encoder learns representation for both normal and inverse relations.
We use r − 1 to indicate the inverse of relation r .
When encoding a query with a (normal) relation r , then both representations R [ r ] and R [ r − 1 ] are used simultaneously.
If we encounter a query using the inverse relation r − 1 , we can use that inverse relation R [ r − 1 ] and the inverse of the inverse R [ (cid:0) r − 1 (cid:1) − 1 ]
= R [ r ] , or in other words the normal relation representation of r .
This means that even if we only ever train with the normal relation, we have the ability to invert relations, and hence we can lift some limitations.
For example, let us look at the query
[Equation]
as shown in Fig.
This query breaks two of the requirements.
It has an entity e 3 occurring after variables VAR 1 and TAR in the topological ordering.
And, TAR is not in the last position.
However, using inverse relations, we can convert this query into:
[Equation]
This transformation converts the query into a 1p-2i query, which is a pattern among the evaluated patterns in the paper.
Because of this transformation, the pattern of Q new pattern is indistinguishable from the pattern of Q known pattern from the perspective of the model.
Besides the example illustrated above, many other graphs can be converted into one of the patterns in the paper.
Since we can perform the conversion in both directions, all these possible patterns are also implicitly included in our used datasets.
In principle, the encoder does also not assume that there are no cycles in the query graph.
However, the effect of cycles requires further investigation.
G N UMBER OF M ESSAGE P ASSING S TEPS E QUAL TO THE D IAMETER
In Daza & Cochez (2020), the authors ﬁnd that the best results are achieved when making the number of message passing steps equal to the diameter of the query, deﬁned as the longest shortest path between 2 nodes in it.
After these steps, they use the embedding of the target variable as the embedding of the complete graph.
Accordingly, we performed additional experiments with this dynamic query embedding setting , and with a variant which uses an extra message passing step.
From these experiments we concluded that this approach did not lead to better results, which contradicts with the ﬁndings in Daza & Cochez (2020).
H Q UALIFIER I MPACT ON P ERFORMANCE
For a more ﬁne-grained analysis of qualiﬁers impact on query answering performance, for each query pattern we ran an experiment measuring relative performance change in the presence and absence of a given qualiﬁer relation in a query.
That is, the main results of StarQE in Table 1 assume all qualiﬁers are enabled.
Then, for each qualiﬁer relation, we remove qualiﬁer pairs containing this relation from all queries in a given pattern and run a model forward pass to obtain new predictions.
We run such experiments 5 times for each relation, each metric, and each pattern to get an average value with standard deviations.
Finally, for each metric, we sort relations in the ascending order of their performance increase.
Table 4 reports top-3 worst (i.e., ﬁrst 3 relations in the sorted list) and top-3 best (last 3 relations in the sorted list)
qualiﬁer relations that have the most impact along the metrics.
For example, P1686 ( for work ), a common qualiﬁer of relation award received , increases Hits@10 performance in 1p2i queries by a large margin of 62% and MRR by 57% compared to queries without this qualiﬁer.
The gains are consistent, although on a smaller scale, in other patterns, too.
On the other hand, P ( character role ), a qualiﬁer of cast member , seems to be the most confusing qualiﬁer in 2p queries leading to lower scores across all metrics.
We note, however, that on a bigger picture (Fig.
8), where impact for all qualiﬁer relations is visualized, total gains of having qualiﬁers outweigh negative effects from some of them.
[Figure]
[Table]
I T HE O RACLE SETUP
There are several QE methods we could compare our work with.
However, these methods are only able to answer queries utilising triple and not qualiﬁer information.
Besides, we see that new methods are introduced regularly, each outperforming the previous method.
Hence, we employ an
Oracle approach to compute the upper bound that triple only QE models can achieve on WD50KQE.
This Oracle is simulating perfect link prediction and ranking capabilities but does not have access to qualiﬁer information.
To achieve this, the Oracle system has access to training, validation, as well as test data.
Given the access to the totality of the data, the system will return an optimal ordering, i.e., one that maximizes our reported metrics.
However, since the system cannot process qualiﬁer information, the ordering of the result set can only be based on the entities and relations in the query.
This means that queries that are the same when ignoring the qualiﬁer information, will get the same list of answers.
In effect, the oracle will, despite its perfect information for a setting without qualiﬁers, not answer perfectly.
Using this setting we can hence investigate how much difference qualiﬁer information makes for our queries.
Finally, note that because multiple ordering can be considered optimal from the information
the
Oracle has available, we report the expected value for these metrics.
J D ETAILED R ESULTS
We provide our chosen hyper-parameters after performing hyper-parameter optimisation in Tableand detailed results including standard deviation across ﬁve runs with different random seeds in Tables 7 and 8.
We report Hits@k for k = 1 , 3 , 10 , mean reciprocal rank (MRR), and adjusted mean rank index (AMRI)
(Berrendorf et al., 2020).
For all metrics, larger values indicate better performance.
We highlight the best result per column and metric in bold font .
For the Hits@k metric, we observe StarQE performing best across patterns, except for the most simple ones, 1p (equivalent to plain link prediction), and 3i .
Standard deviations are usually small (around 1% point).
The base triples baseline sometimes exhibits larger variances across multiple random seeds.
For MRR we make similar observations as for Hits@k, which is intuitive since it can be seen as a soft version of H@k.
For AMRI, the picture differs slightly.
AMRI is a linear transformation of the mean rank, which normalizes the scores such that 0 corresponds to the performance of a random scoring model and 100% to the perfect score.
Besides, AMRI preserves the linearity of mean rank, i.e., it is equally inﬂuenced by improvements on any rank, not just at the top of the ranking.
Across the board, we observe values far beyond 50%, usually exceeding 90%.
The general tendency of some patterns being more difﬁcult than others persists and is coherent with the other metrics.
Comparing different models, we can see that StarQE falls behind, e.g., reiﬁcation and more simple baselines such as zerolayers, which do not use the graph structure.
Since this behavior was not observed for the metrics more focused on the top positions in ranking, the decrease in performance in this metric has to be caused by entities that received large ranks in the ﬁrst place.
Zooming in on the Oracle setup, we make several observations.
First, it becomes increasingly harder for the Oracle to produce good results for smaller values of k in the Hits@k metrics.
Since this is an upper bound to the Base triple setup, also that model shows similar behavior.
The observation that the intersection queries can still perform well is also visible in the more detailed results.
The very high AMRI for the Oracle is also expected.
Since this model is designed as the best possible ranking model, it will place all correct answers for the query (while ignoring qualiﬁers) at the top of the ranking.
This set of answers is a super set of the correct answers to the qualiﬁed query (see Appendix D).
Now, that set of answers is nearly always very small compared to the 50K candidate entities in our dataset.
So, relatively speaking the correct answers are still nearly always ranked high, and hence we obtain a high AMRI score.
K E VALUATING F AITHFULNESS
Following the idea of EmQL (Sun et al., 2020) [Table] we evaluate faithfulness of our hyper-relational approach by evaluating its performance on the training set, that is, an ability to correctly answer already seen queries.
Evaluation on the training set is a suitable proxy for faithfulness since even the union of training, validation and test queries (as done in the original work) is highly incomplete considering the whole background graph (be in Freebase or Wikidata).
To this end, we evaluate the model trained in two regimes: StarQE-like trained on all query types and MPQE-like trained in the hardest setting on only 1p link prediction queries.
The results presented in Table 5 show that
the
StarQE-like model exhibits faithfulness saturating performance metrics to almost perfect results.
As expected, training the model only on one query type inhibits faithfulness qualities
[Table] [Table]
T a b l e 6 :
B e s t hyp e rp a r a m e t e r a s c ho s e n a f t e r hyp e rp a r a m e t e r op ti
m i za ti on
E xp e r
i m e n t
S t a r Q
E
R e i ﬁ ca ti on
B a s e
T r i p l e
Z e r o
L a y e r s
M P Q
E - li k e
M P Q
E - li k e +
R e i f
E m Q
L - li k e
Q 2 Bli k e ac ti v a ti on l ea ky r e l u r e l u r e l u r e l u p r e l u r e
l
u
l ea ky r e l u l ea ky r e l u op ti
m i s e r a d a m a d a m a d a m a d a m a d a m a d a m a d a m a d a m l ea r n i
ng -r a t e 0 .
b a t c h - s i ze        g r a phpoo li ng t a r g e t poo li ng s u m s u m s u m t
a r g e t poo li ng t a r g e t poo li ng t a r g e t poo li ng s u m m e ss a g ew e i gh ti ng a tt e n ti on a tt e n ti on a tt e n ti on s y mm e t
r
i c a tt e n ti on s y mm e t r i c s y mm e t
r
i c a tt e n ti on s i m il a r it y do t p r odu c t n e g a ti v e po w e
r
no r m n e g a ti v e po w e
r
no r m n e g a ti v e po w e
r
no r m do t p r odu c t do t p r odu c t do t p r odu c t n e g a ti v e po w e
r
no r m nu m - l a y e r s
u s e - b i a s
T r u e
T r u e T r u e
F a l s e F a l s e
F a l s e F a l s e
T r u e e m b
e
dd
i ng - d i m        d r opou t 0 .0 .0 .0 .0 .
.c
o m po s iti on m u lti p li ca ti on m u lti p li ca ti on m u lti p li ca ti on m u lti p li ca ti on m u lti p li ca ti on m u lti p li ca ti on m u lti p li ca ti on m u lti p li ca ti on
T a b l e 7 :
F u ll r e s u lt s f o r b a s e li n e e xp e r
i m e n t s , i n c l
ud
i ng s t a nd a r d d e v
i a ti on ac r o ss ﬁ v e r un s w it h
d i ff e r e n t r a ndo m s ee
d s .
W e r e po r t H it s @
k f o r k
= 1 , 3 , 10 , m ea n r ec
i p r o ca l r a nk (
M
RR ) , a nd a d j u s t e d m ea n r a nk
i nd e x ( A
M R I)
B e rr e ndo rf e t a l . ( ) .
F o r a ll m e t
r
i c s , l a r g e r v a l u e s i
nd
i ca t e b e tt e r p e rf
o r m a n ce . W e h
i gh li gh t t h e
[Table]
s t r e s u lt p e r c
o l u m n a nd m e t r i c ( e x c l ud i ng
[Table]
h
[Table]
O
[Table]
ac l e ) i
[Table]
o l d f o n t .
[Table]
2p 3p
[Table] [Table]
±
[Table]
B a s e
T r i p l e 11 .± 0 .
2 .± 0 .
[Table]
± 1 .
[Table]
± 2 .
[Table]
± 6 .
[Table]
± 0 .
[Table]
[Table]
[Table]
R e i ﬁ ca ti on 24 .
± 0 .
[Table]
± 0 .
[Table]
± 0 .
[Table]
±
[Table] [Table]
±
[Table]
[Table] [Table] [Table]
[Table]
Z e r o
L a y e r s
[Table]
± 0 . ± 0 .
[Table]
± 0 .
[Table]
±
[Table] [Table]
±
[Table] [Table]
±
[Table] [Table]
[Table]
O r ac l e
[Table]
[Table] [Table] [Table] [Table]
[Table]
H
[Table]
s @
[Table]
S t a r
Q E
[Table]
± 0 .
± 0 .
[Table]
±
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
B a s e
T r i p l e 25 .± 0 .
6 .± 0 .
[Table]
±
[Table] [Table] ±
[Table] [Table]
±
[Table] [Table]
± 0 .
[Table]
±
[Table]
R e i ﬁ ca ti on 40 .
± 1 .
45 .± 0 .
55 .± 0 .
± 0 .
± 1 .
951 .± 0 .
70 .± 0 .
Z e r o
L a y e r s 31 .± 0 .
± 0 .
25 .± 0 .
55 .± 0 .
85 .± 0 .
25 .± 0 .
35 .± 0 .
O r ac l e 61 .13 .
H it s @( % )
S t a r
Q E 51 .± 0 .
± 0 . ± 0 .
77 .± 0 .
92 .± 0 .
± 0 . ± 0 .
B a s e
T r i p l e 45 .± 1 .
± 0 .
24 .± 2 .
69 .± 1 .
91 .± 1 .
16 .± 1 .
40 .± 1 .
R e i ﬁ ca ti on 55 .
± 1 .
50 .± 0 .
63 .± 0 .
± 0 . ± 0 .
761 .± 0 .
80 .± 0 .
Z e r o
L a y e r s 44 .± 0 .
± 0 .
38 .± 0 .
67 .± 0 .
90 .± 0 .
35 .± 0 .
47 .± 0 .
O r ac l e 81 .24 .
M
RR ( % )
S t a r
Q E 30 .± 0 .
± 0 . ± 0 .
± 0 .
83 .± 1 . 55 .
± 0 .
± 1 .
B a s e
T r i p l e 22 .± 0 .
6 .± 0 .
14 .± 1 .
48 .± 2 .
74 .± 5 . 8 .± 0 .
22 .± 1 .
R e i ﬁ ca ti on 34 .
± 1 .
43 .± 0 .
50 .± 0 .
61 .± 1 .
± 2 .
349 .± 0 .
64 .± [Table]
Z e r o
L a y e r s 27 .± 0 .
± 0 .
21 .± 0 .
50 .± 0 .
80 .± 0 .
24 .± 0 .
30 .± 0 .
O r ac l e 52 .
A M
R I ( % )
S t a r
Q E 78 .± 1 .
± 2 .
74 .± 2 .
93 .± 0 .
98 .± 0 .
73 .± 2 .
88 .± 0 .
B a s e
T r i p l e 88 .± 0 .
± 0 .
95 .± 0 .
99 .± 0 . 99 .± 0 .
83 .± 0 .
98 .± 0 .
R e i ﬁ ca ti on 88 .
± 0 .
86 .± 1 .
± 0 .
± 0 . ± 0 .
085 .± 0 .
± 0 .
Z e r o
L a y e r s 88 .± 0 .
± 0 .
95 .± 0 .
96 .± 0 .
98 .± 0 .
± 0 .
98 .± 0 .
O r ac l e 99 .99 .99 .99 .100 .
T a b l e 8 :
F u ll r e s u lt s f o r t
h e g e n e r a li za ti on e xp e r
i m e n t s , i n c l
ud
i ng s t a nd a r d d e v
i a ti on ac r o ss ﬁ v e r un s w it h
d i ff e r e n t r a ndo m s ee
d s .
W e r e po r t
H it s @
k f o r k
= 1 , 3 , 10 , m ea n r ec
i p r o ca l r a nk (
M
RR ) , a nd a d j u s t e d m ea n r a nk
i nd e x ( A
M R I)
B e rr e ndo rf e t a l . ( ) .
F o r a ll m e t
r
i c s , l a r g e r v a l u e s i
nd
i ca t e b e tt e r p e rf
o r m a n ce .
W e h
i gh li gh t t h e b e s t r e s u lt p e r c
o l u m n a nd m e t r
i c i n b o l d f o n t .
[Table]
tt e
[Table]
1p 2p
[Table]
e m
Q L - li k e
[Table]
± 1 .
[Table]
± 3 .
[Table]
±
[Table]
[Table]
± 3 .
[Table]
±
[Table] [Table]
[Table] [Table] [Table]
[Table]
M P Q
E - li k e
[Table]
± 0 .
[Table]
± 0 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
M P Q
E - li k e +
R e i f 41 .
± 0 .
[Table]
±
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table]
H
[Table]
s @
[Table]
S t a r
E - li k e
[Table]
± 0 .
[Table]
± 0 .
[Table] [Table]
[Table]
[Table]
[Table]
Q 2 B - l i k e
[Table]
± 1 .
[Table]
± 1 .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
e m
Q L - li k e
[Table]
± 1 .
[Table]
±
[Table] [Table]
±
[Table] [Table]
±
[Table] [Table]
[Table] [Table]
[Table]
M P Q
E - li k e 48 .± 0 .
12 .± 0 .
34 .± 3 .
83 .± 1 .
96 .±
[Table]
[Table]
±
[Table] [Table]
[Table] [Table]
M P Q
E - li k e +
R e i f 58 .
± 0 .
12 .± 1 .
31 .± 2 .
± 0 . ± 0 .
13 .±
[Table]
[Table] [Table]
M
RR ( % )
S t a r
E - li k e 30 .± 0 .
± 0 . ± 0 .
[Table]
± 0 .
[Table]
±
[Table] [Table]
± 0 .
[Table]
±
[Table]
Q 2 B - l i k e 33 .± 0 .
41 .± 0 .
51 .± 1 .
61 .± 1 .
[Table]
±
[Table] [Table]
±
[Table] [Table]
±
[Table]
e m
Q L - li
k e 32 .± 1 .
10 .± 2 .
27 .± 5 .
61 .± 3 .
±
[Table] [Table]
±
[Table] [Table]
[Table]
M P Q
E - li k e 26 .± 0 .
6 .± 0 .
19 .± 2 .
56 .± 1 .
[Table]
±
[Table] [Table]
±
[Table] [Table]
± [Table]
M P Q
E - li k e +
R e i f 36 .
± 0 .
6 .± 0 .
17 .± 2 .
56 .± 0 .
77 .± 0 .
8 .± 0 .
29 .± 2 .
A M
R I ( % )
S t a r
E - li k e 78 .± 1 .
68 .± 2 .
74 .± 2 .
93 .± 0 .
98 .± 0 .
73 .± 2 .
88 .± 0 .
Q 2 B - l i k e 88 .± 0 .
± 1 .
± 0 .
99 .± 0 . 99 .± 0 .
83 .± 1 .
98 .± 0 .
e m
Q L - li k
e 73 .±
11 .51 .± 13 .74 .±
.90 .± 5 .
97 .± 1 .
37 .± 22 .82 .± 8 .
M P Q
E - li k e 93 .± 0 .
83 .± 2 .
92 .± 3 .
99 .± 0 . 99 .± 0 .
85 .± 2 .
99 .± 0 .
M P Q
E - li k e +
R e i f 95 .
± 0 .
82 .± 1 .
92 .± 1 .
± 0 . ± 0 .
± 1 .
± 0 .