Regex Queries over Incomplete Knowledge Bases
Abstract
We propose the novel task of answering regular expression queries (containing disjunction ( ∨ ) and Kleene plus (+) operators) over incomplete KBs.
The answer set of these queries potentially has a large number of entities, hence previous works for single-hop queries in KBC that model a query as a point in high-dimensional space are not as effective.
In response, we develop RotatE-Box – a novel combination of RotatE and box embeddings.
It can model more relational inference patterns compared to existing embedding based models.
Furthermore, we deﬁne baseline approaches for embedding based
KBC models to handle regex operators.
We demonstrate performance of RotatE-Box on two new regex-query datasets introduced in this paper, including one where the queries are harvested based on actual user query logs.
We ﬁnd that our ﬁnal RotatE-Box model signiﬁcantly outperforms models based on just RotatE and just box embeddings.
1. Introduction
Knowledge Base Completion (KBC) predicts unseen facts by reasoning over known facts in an incomplete KB.
Embedding-based models are a popular approach for the task – they embed entities and relations in a vector space, and use a scoring function to evaluate validity of any potential fact.
KBC models are typically evaluated using single-hop queries, example, “Who founded Microsoft?”, which is equivalent to the query ( Microsoft → founded by → ?).
A natural extension [Guu et al., 2015] poses multi-hop queries, such as ( Microsoft → founded by → · → lives in → ?), which represents “Where do founders of Microsoft live?”.
Recent works have also studied subsets of ﬁrst-order logic queries with conjunction ( ∧ ), disjunction( ∨ ) and existential quantiﬁers ( ∃ )
[Hamilton et al., 2018, Ren et al., 2020].
We analyzed real-world query logs 1 over the publicly available Wikidata KB 2 and extract its distribution of queries.
We found (see Table 1) that while the single hop queries are indeed most frequent (87%), multi-hop queries comprise only 1% of all queries.
A substantial fraction of queries (12%) are queries using regular expression operators (regex), which use disjunction ( ∨ ) or Kleene plus (+) operators.
In response, we pose the novel problem of answering regex queries using embedding-based KBC models.
[Table]
Regex queries have a potentially large set of correct answers, due to which typical embedding models that model both queries and entities as points are not very eﬀective.
We build on one such model RotatE , which models a relation as a rotation of entity vectors – it can model all relation patterns generally observed in a KB except hierarchy and has not been extended to queries beyond single-hop.
There also exists Query2Box [Hamilton et al., 2018], a box embedding method which models a query as a high-dimensional axis-aligned box.
This can better handle larger answer spaces, but it cannot model symmetric relations (e.g. friend of ).
[Table]
To answer the full repertoire of regex queries over incomplete KBs, we propose a new baseline, RotatEBox – a RotatE model augmented with box embeddings.
In our model, a regex query is an axis-aligned box.
Application of a relation or regex expression rotates the center (vector) of the box, and enlarges/shrinks the box.
Application of a regex operator is handled using two approaches: (1) learned operators over boxes, or (2) introducing separate embeddings for Kleene plus of a relation, and expanding a disjunction into multiple non-disjunctive queries.
RotatE-Box can capture all typical relation patterns – symmetry, antisymmetry, inversion, composition, intersection, mutual exclusion and hierarchy, along with eﬀectively handling a large number of correct answers.
We also contribute two new datasets for our task.
Our ﬁrst dataset, called FB15K-Regex, is an extension of the popular FB15K dataset [Bordes et al., 2013], and is constructed using random walks on the FB15K graph.
Our second dataset is built over a dense subset of
Wikidata (which we name W iki 100) and is based on the real queries from existing query logs.
Table 2 lists example queries in the datasets.
We ﬁnd that models based on RotatEBox outperform all other baselines, including strong models based on just RotatE and just Query2Box on these two datasets.
In summary, our work presents a novel regex query answering task, two new datasets and a strong baseline for our task.
We will release all resources for further research 3 .
2. Related Work
A KB G is a set of triples ( e 1 , r, e 2 ), where e 1 , e 2 ∈ E (the set of entities) and r ∈ R (the set of relations).
For a KBC task, all triples are split into train/dev/test, resulting in graphs
G train , G dev , and G test .
The standard KBC task requires answering single-hop queries of the form ( e 1 , r, ?)
with the answer e ∗ 2 if ( e 1 , r, e ∗ 2 ) ∈ G test .
While many kinds of solutions exist for this problem, we build on the vast literature in embedding-based approaches [Bordes et al., 2013, Socher et al., 2013, Yang et al., 2014, Trouillon et al., 2016, Jain et al., 2018, Sun et al., 2019, Jain et al., 2020].
These methods embed each entity and relation in a latent embedding space, and use various tensor factorization scoring functions to estimate the validity of a triple.
Our work builds on RotatE, which deﬁnes relation as a rotation in a complex vector space.
The distance of target entity from the query constitutes the scoring function.
RotatE has shown high KBC performance and can model many relation patterns (see Section 2.1).
The ﬁrst work on answering complex queries over incomplete KBs extends single-hop queries to path queries (or multi–hop queries) of the type ( e 1 , r 1 , r 2 , . . .
, r n , ?).
Guu et al.
[2015] construct datasets for the task using random walks on FB13 and WN11 KBs.
Models for the task include compositional training of existing KBC scoring functions and models based on variants of RNNs [Yin et al., 2018, Das et al., 2017].
Recent work has started exploring subsets of ﬁrst-order logic queries.
Hamilton et al.
[2018] explore conjunctive queries with conjunction ( ∧ ) and existential quantiﬁcation ( ∃ ).
They propose DeepSets [Zaheer et al., 2017] to model conjunction as a learned geometric operation.
MPQE [Daza and Cochez, 2020] and BIQE [Kotnis et al., 2021] extend this work by constructing a query representation using a graph neural network and transformer encoder respectively.
Probably closest to our work is that of answering Existential Positive First-order (EPFO) logical queries [Ren et al., 2020] – queries that include ∧ , ∨ , and ∃ .
They propose Query2Box, which represents a query as a box (center, oﬀset) and each entity as point in a highdimensional space.
The points closer to the center are scored higher for the query, especially if they are inside the box.
The strength of Query2Box is that it can naturally handle large answer spaces, and also model many relation patterns (see Section 2.1).
[Ren and Leskovec, 2020] extend EPFO to complete set of ﬁrst-order logical operations, by handling negation ( ¬ ) with their proposed model, BetaE . They propose high-dimensional Beta distributions as embeddings of queries and entities.
The KL divergence between query and entity serves as a scoring function. There are also other works that work with probabilistic databases for answering EPFO queries [Friedman and Van den Broeck, 2020], but, to the best of our knowledge, there has been no work on answering regex queries over incomplete knowledge bases.
In particular, regular expressions that contain the Kleene plus operator, are not included in path queries or ﬁrst-order logic queries.
2.1 Relation Patterns in a KB
Embedding models can be critiqued based on their representation power [Toutanova and
Chen, 2015, Guu et al., 2015, Trouillon et al., 2016, Sun et al., 2019] in modeling commonly occuring relation patterns (relationships amongst relations).
We summarize the common patterns and model capabilities in Table 3.
Broadly, translation-based methods (that add and subtract embedding vectors) like TransE and Query2Box can model all patterns but cannot model symmetry (( e 1 , r 1 , e 2 ) ⇒ ( e 2 , r 1 , e 1 )), such as for the relation friend of .
Rotation-based models such as RotatE can model all inference patterns except for hierarchy: ( e 1 , r 1 , e 2 ) ⇒ (
e 1 , r 2 , e 2 ).
Other models have other weaknesses, like similaritybased methods (that use dot products between embeddings) such as DistMult and ComplEx [Table] are incapable of handling composition and intersection.
Our proposed model combines the power of Query2Box and RotatE and can express all seven common relation patterns.
3. Task
We now formally deﬁne the task of answering regular expressions (regex) queries on a KB.
Regular expressions are primarily used to express navigational paths on a KB in a more succinct way and to extend matching to arbitrary length paths.
Any regex c in a KB is an expression over R ∪ { + , /, ∨} , generated from the grammar:
c ::= r | c 1 ∨ c 2 | c 1 /c 2 | c + , where r ∈ R represents a relation in the KB and c 1 and c 2 represent regular expressions.
/ denotes followed by , ∨ denotes disjunction , and + is a Kleene Plus operator denoting one or more occurrences.
We use l ( c ) to denote the set of paths (sequence of relations) compatible with regex c .
As an example, for c = r 1 /r +2 , l ( c ) will have paths like r 1 /r 2 , r 1 /r 2 /r 2 , r 1 /r 2 /r 2 /r 2 and so on.
Their path lengths will be 2, 3 and 4 respectively.
A regex query q over a KB is deﬁned as a pair of head entity e 1 and a regular expression c .
For query q
=
( e 1 , c, ?), we deﬁne the answer set (cid:74) q (cid:75) as the union of entities reachable from e 1 in G , by following the paths in l ( c ): (cid:74) q (cid:75) = [Equation] c
[Equation]
(cid:74)
[Equation]
p, ?) (cid:75) . The goal of the task is to, given a query q , rank entities in (cid:74) q (cid:75) higher than other entities.
4. Model
We now describe RotatE-Box – a model that combines the strengths of RotatE and Query2Box.
Following Query2Box, the relations and queries are modeled as boxes, and entities as points in a high-dimensional space.
However, unlike Query2Box, all points (and boxes) are in a complex vector space C k , instead of a real space.
We ﬁrst describe the model for single-hop queries.
RotatE-Box embeds each entity e as a point e ∈ C k and each relation r as a box r , represented as (Cen( r ) , Oﬀ( r ))
∈ C 2 k .
Here
Cen( r ) ∈
C k is the center of the box such that its modulus in each of the k dimensions is one, i.e., ∀ j ∈ [1 , k ], | Cen( r j ) |
= 1, Cen( r j ) ∈ C .
By doing so, Cen( r j ) can be re-written as e iθ r,j using Euler’s formula (here, e is Euler’s number).
This corresponds to rotation by angle θ r,j in the j th dimension of complex vector space.
In vectorized form, Cen( r ) =
e iθ θ θ r .
Oﬀ( r ) is a positive oﬀset,
i.e., Re(Oﬀ( r )) ∈ R k ≥ 0 and Im(Oﬀ( r ))
∈ R k ≥ 0 .
Any box that satisﬁes these constraints on its center and oﬀset is termed as a rotation box .
A point v ∈ C k is considered inside a box if it satisﬁes:
[Equation]
and
[Equation]
Here, r max =
Cen( r ) + Oﬀ( r ) ∈ C k , and r min =
Cen( r ) − Oﬀ( r ) ∈ C k .
The inequalities represent element-wise inequality of vectors.
A single-hop query q = ( e 1 , r, ?) has a rotation box representation q given by ( e 1 (cid:12) Cen
( r ) , Oﬀ( r )).
Here (cid:12) is the Hadamard product (element-wise multiplication).
Since Cen( r ) = e iθθθ r , e 1 is rotated by an angle θθθ r ∈ R k .
Speciﬁcally e j is rotated by angle θ r,j in the j th dimension.
Given a query q associated with a box q , RotatE-Box ranks entities e in the order of the distance of e from q , deﬁned as:
[Equation]
where α ∈ (0 , 1) is a constant to downweigh the distance inside the query box, and dist out
[Equation]
e
[Equation]
Here q max and q min are deﬁned analogously to r max and r min above.
This completes the description of RotatE-Box for single-hop queries.
As RotatE is a special case of RotatE-Box (by modeling Oﬀ( r ) = 0 , ∀ r ∈ R ), RotatE-Box can model all the relation patterns modeled by RotatE: symmetry, anti-symmetry, inversion, composition, intersection and mutual exclusion.
Additionally, boxes enable modeling hierarchy.
Theorem 1.
RotatE-Box can model hierarchical relation pattern.
(Proof in Appendix A.1)
4.1 Relation Paths
We now deﬁne RotatE-Box operations for a path query, e.g., q = ( e 1 , r 1 /r 2 / . . .
/r n , ?).
It ﬁrst initializes the rotation box embedding for relation path as r 1 (denoted by p 1 ).
Given p i , to continue the path with r i +1 , it rotates the center of p i , and adds the oﬀsets:
[Equation]
The ﬁnal rotation box embedding p of r 1 /r 2 / . . .
/r n is p n as computed above.
It computes query embedding q as described earlier – rotating e 1 by Cen( p ) and keeping the oﬀset as
Oﬀ( p ).
Ranking of answers for the query follows similar approach as speciﬁed in Eq 2.
4.2 Kleene Plus
To extend path embeddings to general regex expression c , we follow the same representation: for a regex query ( e 1 , c, ?), c is embedded as a rotation box, c , in complex space C 2 k .
We ﬁrst describe our formulation for the Kleene Plus (+) operator.
We use two methods for this.
Projection:
Kleene Plus (+) operator may apply compositionally and recursively to any regular expression.
An approach to materialize this is to deﬁne it as a geometric operator [Hamilton et al., 2018], which takes input a rotation box representation c (corresponding to rotation by θθθ c ), applies a function kp to output a new rotation box embedding c (cid:48) that represents c + .
RotatE-Box uses the function kp ( c )
=
c (cid:48) =
( e iθθθ c (cid:48) , K oﬀ Oﬀ( c )), where θθθ c (cid:48) =
K cen θθθ c .
Here, K cen ∈ R k × k , K oﬀ ∈ C k × k are trainable parameter matrices.
Free parameter : A strength of the geometric operator is that it can be applied compositionally, whereas a weakness is that its expressive power may be limited, since the two
K matrices apply on all occurrences of Kleene plus.
As an alternative, for every relation r , we deﬁne free parameter rotation box embeddings of r + denoted by r + .
The query q = ( e 1 , r + , ?) is then represented as ( e 1 (cid:12) Cen
( r + ) , Oﬀ( r + )).
Such a formulation cannot handle compositional queries like ( e 1 , ( r 1 ∨ r 2 ) + , ?) or ( e 1 , ( r 1 /r 2 ) + , ?).
However, the motivation behind this formulation is that most of the Kleene plus queries in user query logs of
Wikidata do not require Kleene plus operator to be compositional (Table 9).
4.3 Disjunction
We follow two approaches for modeling disjunction ( ∨ ).
In one case, we split a disjunction into independent non-disjunctive regexes.
In the second case, we train a disjunction operator.
Aggregation: This approach follows Ren et al.
[2020], who proposed transforming EPFO queries to Disjunctive Normal Form (DNF), moving disjunction operator ( ∨ ) to the last step of computation graph.
While all regex expressions cannot be transformed to a DNF form, a signiﬁcant fraction (75%) of regex queries in our dataset formed from Wikidata user query logs (Section 5) can be expressed as a disjunction of a tractable number of nondisjunctive constituent queries q = q 1 ∨ q 2 ∨ . . .
∨ q N .
For example, q = ( e 1 , r 1 / ( r +2 ∨ r +3 ) , ?) can be expressed as q 1 ∨ q 2 where q 1 = ( e 1 , r 1 /r +2 , ?) and q 2 = ( e 1 , r 1 /r +3 , ?).
For such queries, following Ren et al.
[2020], we compute scores independently for each query, and aggregate them based on the minimum distance to the closest query box, i.e. dist( e ; q ) =
Min( { dist( e ; q 1 ) , dist( e ; q 2 ) , . .
, dist( e ; q N ) } ).
Notice that this approach is not applicable if such a query decomposition is not feasible, such as for ( r 1 ∨ r 2 )
+ .
DeepSets: Zaheer et al.
[2017] introduce a general architecture called DeepSets for functions where the inputs are permutation invariant sets.
As disjunction ( ∨ ) is permutation invariant, we use DeepSets to model it.
We deﬁne an operator D which takes the rotation box embeddings c 1 , c 2 , . . .
, c N of regular expressions c 1 , c 2 , . . .
, c N (corresponding to rotation angles θθθ c 1 ,θθθ c 2 , . . .
,θθθ c N ), and returns the embedding of c
= c 1 ∨ c 2 ∨ . . .
∨ c N , denoted by c .
RotatE-Box deﬁnes D ( c 1 , c 2 , . . . , c N ) =
( e iθθθ c , Oﬀ( c )) as follows:
[Equation]
Oﬀ [Equation] where MLP cen , MLP oﬀ are Multi-Layer Perceptron networks, Ψ is a permutation-invariant vector function (element-wise min, max or mean), and W cen , W oﬀ are trainable matrices.
4.4 Training objective
Following the previous work [Sun et al., 2019, Ren et al., 2020], we optimize a negative sampling loss to train RotatE-Box.
Let γ be a constant scalar margin.
For a query q with answer entity e , and a negative entity not in answer set e (cid:48)
i , the loss is deﬁned as:
[Equation]
5. Dataset
We contribute two new regex query datasets for our task – the ﬁrst is based on Wikidata 4 [Vrandeˇci´c and Kr¨otzsch, 2014a], a large popular open-access knowledge base, and the second is based on FB15K [Bordes et al., 2013], a subset of the Freebase KB commonly used in KBC tasks.
For both datasets, we ﬁrst split facts in original KB G into three subsets to construct G train , G dev , and G test (using existing splits for FB15K).
These are used to split regex queries into train, dev and test sets.
Wiki100-Regex:
The Wikidata KB contains over 69 million entities, which makes it diﬃcult to experiment with directly.
We ﬁrst construct a dense subset based on English
Wikidata, with 100 relations and 41,291 entities, and name it Wiki100 .
This knowledge base contains 443,904 triples (Appendix B.1 has more details on Wiki100 construction).
Existing work on KBC with complex queries uses datasets built using random walks over KB [Guu et al., 2015, Kotnis et al., 2021] or uniform sampling over query DAG structure
[Hamilton et al., 2018, Ren et al., 2020].
The availability of a large-volume of query (SPARQL) logs over Wikidata, oﬀers us a unique opportunity to use real-world user queries for Wiki100-Regex.
On studying the query logs, we ﬁnd that 99% of regex queries (excluding single-hop queries) are of ﬁve types – ( e 1 , r +1 , ?), ( e 1 , r +1 /r +2 , ?), ( e 1 , r 1 /r +2 , ?), ( e 1 , r 1 ∨ r 2 , ?), and ( e 1 , ( r 1 ∨ r 2 ) + , ?).
We retain all regex queries from these types as the ﬁnal set of queries.
To split the user queries into train/dev/test, we traverse each user query ( e 1 , c, ?)
up to depth 5, where c is the regex expression.
For every answer entity e 2 reachable from efollowing c in G train , we place ( e 1 , c, e 2 ) in train split.
If e 2 is not reachable in G train , but reachable in G train ∪ G dev , we place ( e 1 , c, e 2 ) in dev split, else in test split.
We further augment the training split with random walks on G train for these 5 query types.
The ﬁnal regex query type distribution is given in Table 9 in Appendix B.2.
[Table]
FB15K-Regex:
This dataset is constructed with the goal of testing our models on a higher diversity of regex queries than Wiki100-Regex.
We identify 21 diﬀerent types of regex expressions, using up to three diﬀerent FB15K relations along with Kleene plus (+) and/or disjunction ( ∨ ) operators.
For each regex expression type c , we enumerate a set of compatible relations paths l ( c ), up to length 5.
For a random source entity e 1 , the answer set of regex query q = ( e 1 , c, ?) is formed by aggregating the answer sets of compatible relations path queries, i.e.
(cid:74) q (cid:75) = (cid:83) p ∈ l ( c ) ,len ( p ) < =5 (cid:74) ( e 1 , p, ?) (cid:75) .
We discard generic queries that have answer set size of over 50 entities.
This procedure results in a highly skewed dataset, with 3 query types accounting for more than 80% of the regex queries.
We reduce the skew in the query distribution by undersampling frequent query types.
The ﬁnal dataset statistics for each regex query type are summarized in Table 8 in Appendix B.2.
The split into train/dev/test sets follows a procedure similar to one used when constructing Wiki100-Regex data.
The dataset statistics of both the datasets are summarized in Table 4.
6. Experiments
In this section, we run experiments on the two datasets introduced in Section 5.
We compare our RotatE-Box model with two competitive methods for complex query answering – Query2Box and BetaE, along with a close baseline, RotatE. We also wish to ﬁnd out the performance diﬀerence between compositional and non-compositional variants to handle disjunction and Kleene plus.
6.1 Experimental Setup
Baselines and model variants: We note that regex variants of distribution-based models like BetaE and distance-based models like Query2Box and RotatE can also be created by applying similar ideas as in Sections 4.1-4.3 (Appendix A.2).
For each of the three models, we experiment with all possible combination of operators:
• KBC + Aggregation (BASELINE):
In this variant, we only train on single hop queries (link prediction), ( e 1 , r, ?), and evaluate the model on regex queries by treating r + as r , and using Aggregation operator for handling disjunction.
• Free parameter + Aggregation: In this variant, Kleene plus is modeled as a free parameter for every relation and disjunction is handled via Aggregation.
• Free parameter + DeepSets: In this variant, Kleene plus is modeled as a free parameter and disjunction is handled via DeepSets.
• Projection + Aggregation: In this variant, we use Projection for Kleene plus and aggregation for disjunction.
• Projection + DeepSets (COMP): In this variant, we use Projection for Kleene plus and DeepSets for disjunction
Unlike other variants, both the operations are compositional and hence, can be applied to any arbitrary regex query.
Apart from Projection + DeepSets (COMP) , none of the other variants listed above can answer every query type, due to the presence of non-compositional operators for regex.
For example, the query type ( e 1 , ( r 1 /r 2 ) + , ?) cannot be answered by these variants.
In such cases, the query is not evaluated and the rank of correct entity is simply returned as ∞ .
Evaluation metrics:
Given a regex query q
=
( e 1 , c, ?)
, we evaluate validity of e 2 as an answer of the query by ranking all entities, while ﬁltering out other entities from the answer set (cid:74) q (cid:75)
[Bordes et al., 2013].
The candidate set for query q , C ( q ), can then be deﬁned as
C ( q ) = { e 2 } ∪ {E − (cid:74) q (cid:75) } .
We calculate the rank of e 2 amongst C ( q ) and compute ranking metrics such as Mean Reciprocal Rank (MRR) and Hits at K (HITS@K).
Implementation details:
An embedding in complex vector space has two trainable parameters per dimension (real and imaginary part) whereas a real valued embedding has [Table] only one.
Therefore, to keep the number of trainable parameters comparable, we use k =for complex embedding based models (RotatE and RotatE-Box) and k = 800 for real embedding based models (Query2Box).
Relation and entity embeddings of all the models are ﬁrst trained on single-hop queries ( e 1 , r, ?), and then optimized for regex queries along with other trainable parameters of regex operators.
For Kleene plus operator (free parameter variant), r + is initialized as r .
We use element-wise minimum function for Ψ. Other training speciﬁc details and hyperparameter selection are in Appendix C.1.
6.2 Results and Discussions
Table 5 shows the main results of diﬀerent embedding based models with diﬀerent variants for Kleene plus and disjunction on both datasets.
Overall, we observe that the compositional variants of all models outperform non-compositional ones.
RotatE-Box obtains signiﬁcant improvements over other distance-based methods – RotatE and Query2Box (correspondingly for all variants).
RotatE-Box variants also perform comparably to BetaE models for
FB15K-Regex, and signiﬁcantly outperform them for Wiki100-Regex.
Comparing best and second best results, RotatE-Box(COMP) achieves 0.65 points MRR over BetaE(COMP) for FB15K-Regex, and over 3.7 points MRR over RotatE(COMP) for Wiki100-Regex.
The massive improvement of both COMP and other variants over BASELINE highlights the necessity of training KBC models beyond single-hop queries.
Recall that for each model, all variants except COMP contain only non-compositional regex operators, hence, a portion of queries are not evaluated, and the score for those queries is returned as 0.
To get better insights into operators, we look at the subset of regex queries that are answerable by all variants.
In our experiments, this subset includes all regex query types except ( e 1 , ( r 1 ∨ r 2 ) + , ?).
The results of this ablation study are presented in Table 6.
While RotatE-Box maintains its superior performance over other models on [Table] [Table] this subset of regex queries, both the formulations of Kleene plus operator – Projection and Free parameter
[Table] have almost similar performance [Table] with Free parameter variant showing marginally higher MRR and HITS@1 in most cases.
In particular, (Free parameter + DeepSets) for RotatE-Box on Wiki100-Regex is better than (COMP) by more than 1. points MRR.
This suggests that the simple Projection operator does not always achieve quite the same expressiveness as free parameters.
We leave the study of deeper trainable projection operators for future work.
6.2.1 Impact of regex training on single-hop queries
The regex query training is initialized with entity and relation embeddings trained on single– hop queries.
However, when the model is optimized on regex queries, the performance on single–hop queries drops dramatically for both COMP and (Free parameter + Aggregation) variants.
Similar trend is observed in other variants as well.
The results are summarized in
Table 7.
We believe this could the the result of entity embeddings being optimized for two conﬂicting objectives.
Consider the following example.
Let q 1 represent the query ( e 1 , r, ?) and let e 2 be an answer to q 1 .
Similarly, let q 2 = ( e 2 , r, ?) and e 3 be an answer for q 2 .
In theory, from KBC training we get q 1 − e 2 = q 2 − e 3 = 0 , i.e. e 3 − e 2 = q 2 − q 1 (cid:54) =
for any distance-based model.
However, for regex query q (cid:48)
= ( e 1 , r + , ?), for which e 2 and eare both answers, e 3 − e 2 = q (cid:48) − q (cid:48) = 0 .
Hence, distance-based embedding models cannot simultaneously model both KBC and regex queries.
This hypothesis can also be extended to distribution-based models like BetaE, which models distance between entity and query as KL divergence between their respective distributions.
Similar phenomenon has been observed in previous works on complex queries [Ren et al., 2020].
We leave it to future work to optimize for complex queries without losing performance on single–hop queries.
We also observe that on Wiki100, RotatE-Box has a higher performance than RotatE, suggesting that RotatE-Box may have some value for pure KBC tasks also.
7. Conclusion
In this work we present the novel task of answering regex queries over incomplete knowledge bases.
We provide two datasets and query workloads for the task and provide baselines for training regex operators.
Of the two datasets we present, Wiki100-Regex is a novel challenging benchmark based entirely on a real-world KB and query logs.
We also provide a new model RotatE-Box , which models more relational inference patterns than other approaches.
Our code and data is available at https://github.com/dair-iitd/kbi-regex .
While the baselines of this work are the ﬁrst step in modeling regex operators, there is a lot of scope for future research.
Particularly, Kleene Plus poses novel modeling challenges – it is an idempotent unary operator (( r + ) + = r + ) and an inﬁnite union of path queries ( r + = r ∨ ( r/r ) ∨ ( r/r/r ) . . . ).
In the future, we plan to work on constructing parameterized operator functions, which honor these properties of the operator, and can be applied compositionally and recursively, while being suﬃciently expressive.
Acknowledgments
This work was supported by an IBM AI Horizons Network (AIHN) grant, grants by Google,
Bloomberg and 1MG, a Visvesvaraya faculty award by Govt.
of India, and a Jai Gupta
Chair fellowship to Mausam. Srikanta Bedathur was partially supported by a DS Chair of AI fellowship. We thank the IITD HPC facility for compute resources.
Appendix A. Model A.1 Proof of Theorem
Proof.
Consider a relation r 1 represented by (Cen( r 1 ) , Oﬀ( r 1 ))
∈ C k .
The query representation of q 1 = ( e 1 , r 1 , ?) will be
[Equation]
If an entity e 2 is in the answer set of q 1 , then it must be inside the query box of q 1 , i.e:
[Equation]
Now consider a relation r 2 such that Cen( r 2 )
=
Cen( r 1 ), Re(Oﬀ( r 2 )) ≥ Re(Oﬀ( r 1 )) and
Im(Oﬀ( r 2 )) ≥ Im(Oﬀ( r 1 )).
Hence, for query q 2 = ( e 1 , r 2 , ?) represented by:
[Equation]
the following is true:
[Equation]
Hence, the entity e 2 is inside the query box of q 2 .
Therefore, r 1 ( e 1 , e 2 ) implies r 2 ( e 1 , e 2 ).
Thus, RotatE-Box can model hierarchy.
A.2 Regex Variants of Baseline Models
In this section we describe how the formulations of regex operators (Section 4.1- 4.3) apply to Query2box and RotatE .
Similar ideas can be easily extended to BetaE as well.
Query2Box :
Each entity e is modeled as a point e ∈ R k and each relation r is modeled as a box r , represented as (Cen( r ) , Oﬀ( r ))
∈ R 2 k .
Unlike RotatE-Box, the modulus of Cen( r ) is not constrained.
The representation of a single-hop query q = ( e 1 , r, ?) is constructed by translating the center and adding the oﬀset, i.e. q = ( e
+ Cen( r ) , Oﬀ( r )).
Generalizing this to path queries, the path embedding p n for relation path r 1 /r 2 / . . .
/r n is computed as follows:
[Equation]
Here, p 1 = r 1 .
Query q = ( e 1 , r 1 /r 2 / . . .
/r n , ?) is represented as q =
( e 1 +Cen( p n ) , Oﬀ( p n )).
Modeling Kleene plus as Projection follows from Section 4.2, however, for Query2Box, the projection matrices are applied to actual embeddings, rather rather rotation angles.
Given representation c for a regex expression c , c
+ is modeled as kp ( c ) =
c (cid:48)
=
Kc , K ∈ R k × k
While modeling Kleene plus as a Free parameter, the query q = ( e 1 , r + , ?) is represented as ( e 1 + Cen( r + ) , Oﬀ( r + )).
We follow the original implementation of Query2Box [Ren et al., 2020] for modeling
Disjunction as aggregation.
When modeling disjunction with DeepSets, c = c 1 ∨ c 2 ∨ . . .
∨ c N is represented as:
[Equation]
RotatE : For RotatE, entities and relations are modeled as points in the complex space, i.e. e ∈ C k and r ∈ C k .
The modulus of r is constrained to be 1 in every dimension ( r = e iθθθ r ).
The single-hop query q = ( e 1 , r, ?) is constructed by rotating the entity embedding ( q = e 1 (cid:12) r ).
The target entity is scored by its distance from the query embedding dist( e ; q ) =
| q − e | .
Computation of the relation path follows Section 4.1, but ignoring the oﬀset term.
Therefore, q = ( e 1 , r 1 /r 2 / . . .
/r n , ?) is represented as q = e 1 (cid:12) p n , where p n = r 1 (cid:12) r 2 (cid:12) . . .
(cid:12) r n .
Given representation c (corresponding to rotation by angle θθθ c ) for a regex expression c , c
+ is modeled as kp ( c ) =
c (cid:48) =
e iθθθ c (cid:48) , where θθθ c (cid:48)
=
K θθθ c , K ∈ R k × k , when using Projection for Kleene plus.
The free parameter embeddings of r + is denoted by r + .
For RotatE, The query q = ( e 1 , r + , ?) is then represented as q = e 1 (cid:12) r + .
Modeling disjunction as aggregation is similar to that described in Section 4.3, using dist( e ; q ) as deﬁned above.
For DeepSets, given the embeddings c 1 , c 2 , . . .
, c N (corresponding to rotation angles θθθ c 1 ,θθθ c 2 , . . .
,θθθ c N ) of regular expressions c 1 , c 2 , . . .
, c N , c
= c 1 ∨ c 2 ∨ . . .
∨ c N is represented as:
[Equation]
where MLP is a Multi-Layer Perceptron network, Ψ is a permutation-invariant vector function, and W is a trainable matrix.
Appendix B. Dataset B.1 Wiki100 Knowledge Base
Wikidata is a collaboratively edited multilingual knowledge base that manages factual data in Wikipedia [Vrandeˇci´c and Kr¨otzsch, 2014b].
It contains more than 69 million entities which makes it diﬃcult to experiment with directly.
We construct a dense subset based on
English Wikidata as follows:
we ﬁrst selected entities and relations which have at leastinstances in Wikidata and selected triples mentioning only the top-100 frequent relations.
The resulting dataset had highly skewed distribution of relations with 6 relations accounting for more than 95% of all triples.
In order to smooth this distribution and to include a signiﬁcant number of all relations, we undersample facts containing these 6 relations, and oversample others.
Finally, we recursively ﬁltered sparse entities mentioned less than n times (decreasing n from 10 to 1) to obtain a knowledge base which we call as Wiki100 , with 443 , 904 triples over 41 , 291 entities and 100 unique relations, which we randomly split into train/dev/test.
B.2 Regex Query Distribution
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
∨
[Table]
[Table]
( e
[Table]
, ( r
[Table]
∨ r 2 ) / [Table] [Table] ( e
[Table]
r 1 / ( r
[Table]
∨ r 3 ) , ?)
[Table]
( e [Table] r [Table]
1 ∨ r
[Table]
[Table]
[Table]
[Table]
( e 1 , ( r 1 ∨ r 2 ) /r +3 , ?)
27, 5, 9, ( e 1 , ( r +1 ∨ r +2 )
/r 3 , ?)
24, 4, 8, ( e 1 , r +1 /
( r 2 ∨ r 3 ) , ?) 27, 5, 9, ( e 1 , r 1 / ( r +2 ∨ r +3 ) , ?)
28, 5, 9, ( e 1 , ( r 1 ∨ r 2 )
+ , ?)
Total 580, 112, 200,
Query type
Train
Valid
Test ( e [Table]
, r
[Table]
490, 24, 23, ( e 1 , r +1 /r +2 , ?) 6,   ( e 1 , r 1 /r +2 , ?) 85, 10, 8, ( e 1 , r 1 ∨ r 2 , ?)
274, 14, 14, ( e 1 , ( r 1 ∨ r 2 )
+ , ?)
Total 1,205, 66, 62,
Appendix C. Experiments
All the models are implemented in Pytorch-Lightning framework.
We used NVIDIA V (16GB) GPUs from training the models.
For all models, we use 2 GPUs when training on FB15K-Regex, and 4 GPUs when training on Wiki100-Regex.
C.1 Hyperparameters
[Table]
Model-speciﬁc hyperparameters such as margin γ and down-weighting constant α are ﬁrst tuned on single-hop query pre-training.
Apart from learning rate lr , the same hyperparameters are used in regex query training.
The ranges of hyperparameter grid search used for all our models are: γ ∈ { 4 . 0 , 8 . 0 , 12 . 0 , . . .
, 28 . 0 , 32 . 0 } , α ∈ { 0 . 2 , 0 .
8 , 1 . 0 } , learning rate lr = { 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } .
For a particular dataset, we found the same set of hyperparameters to perform best for all three models.
The ﬁnal values are reported in Table 10.
All the models are optimized using Adam optimizer [Kingma and Ba, 2015].
For FB15K, learning rate is set to 10 − 4 for single hop as well as regex queries.
For Wiki100, it is 10 − 3 when training on single hop queries and 10 − 4 when training on regex queries.
We use a batch size of 1024 and sample 256 negative entities for each training query.
The negative entities e (cid:48)
i are sampled uniformly for regex query training and adversarially while training RotatE and RotatE-Box on single–hop queries (as per [Sun et al., 2019]).
For both datasets across all models we train for
1000 epochs for single- hop queries and for regex queries, using early stopping on dev split MRR