Paper:
Probabilistic Regular Graph Languages
Distributions over strings and trees can be represented by probabilistic regular languages, which characterize many models in natural language processing. Recently, several datasets have become available which represent natural language phenomena as graphs, so it is natural to ask whether there is an equivalent of probabilistic regular languages for graphs. To answer this question, we review three families of graph languages: Hyperedge Replacement Languages (HRL), which can be made probabilistic; Monadic Second Order Languages (MSOL), which support the crucial property of closure under intersection; and Regular Graph Languages (RGL; Courcelle 1991), a subfamily of both HRL and MSOL which inherits these properties, and has not been widely studied or applied to NLP. We prove that RGLs are closed under intersection and provide an efficient parsing algorithm, with runtime linear in the size of the input graph.
1 Introduction
NLP systems for machine translation, summarization, paraphrasing, and other problems often fail to preserve the compositional semantics of sentences and documents because they model language as bags of words, or at best syntactic trees. To preserve semantics, they must model semantics. In pursuit of this goal, several datasets have been produced which pair natural language with compositional semantic representations in the form of directed acyclic graphs (DAGs), including the Abstract Meaning Represenation Bank (AMR; Banarescu et al. 2013), the Prague Czech-English Dependency Treebank (Hajič et al., 2012), Deep-
bank (Flickinger et al., 2012), and the Universal Conceptual Cognitive Annotation (Abend and Rappoport, 2013). To make use of this data, we require probabilistic models of graphs.
Consider how we might use compositional semantic representations in machine translation (Figure 1). We first parse a source sentence to its semantic representation, and then generate a target sentence from this representation. Stated more formally, we first predict a graph G from a source string s, and then predict a target string t from G, giving us a conditional model P(t, G|s), which we can decompose as P(t, G|s) = P(t|G)P(G|s). Jones et al. (2012) observe that this decomposition can be modeled with a pair of probabilistic synchronous grammars over domains of strings and graphs. Given a domain of source strings Ls, a domain of source graphs LG, a domain of target graphs LG′ and a domain of target strings Lt, these grammars define relationsR ⊆ Ls×LG and R′ ⊆ LG′ ×Lt. Translation is then the solution to the following inference problem, given input s:
arg max t∈Lt ∑ {G|(s,G)∈R∧(G,t)∈R′} P(t|G)P(G|s)
In practical settings the sum over G is typically replaced with an arg max. Either way, we must
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
define probability distributions over the graph domains and efficiently compute their intersection.
For NLP problems in which data is in the form of strings and trees, such distributions can be represented by finite automata (Mohri et al., 2008; Allauzen et al., 2014), which are closed under intersection and can be made probabilistic. It is therefore natural to ask whether there is a family of graph languages with similar properties to finite automata. Recent work in NLP has focused primarily on two families of graph languages: hyperedge replacement languages (HRL; Drewes et al. 1997), a context-free graph rewriting formalism that has been studied in an NLP context by several researchers (Chiang et al., 2013; Peng et al., 2015; Bauer and Rambow, 2016); and DAG automata languages, (Kamimura and Slutzki, 1981), studied by Quernheim and Knight (2012). Thomas (1991) showed that the latter are a subfamily of the monadic second order languages (MSOL), which are of special interest to us, since, when restricted to strings or trees, they exactly characterize the regular languages of each (Büchi, 1960; Büchi and Elgot, 1958; Trakhtenbrot, 1961).
The HRL and MSOL families are incomparable: that is, the context-free graph languages do not contain the regular graph languages, as is the case in languages of strings and trees (Courcelle, 1990). So, while each formalism has appealing characteristics, none appear adequate for the problem outlined above: HRLs can be made probabilistic, but they are not closed under intersection; and while DAGAL and MSOL are closed under intersection, it is unclear how to make them probabilistic (Quernheim and Knight, 2012).1
This paper investigates the regular graph languages (RGL; Courcelle 1991), defined as a restricted form of HRL (§2). The restrictions ensure that RGLs are a subfamily of MSOL. Courcelle (1991) defined regular graph grammars as an auxiliary result of a theoretical research question quite different from ours.2 As a consequence, they have
1Semiring-weighted MSOLs have been defined, where weights may be in the tropical semiring (Droste and Gastin, 2005). However, for the weights to define a probability distribution, they must meet the stronger condition that the sum of multiplied weights over all definable objects is one. This does not appear to have been demonstrated for DAGAL, which violate the sufficient conditions that Booth and Thompson (1973) give for probabilistic languages. We suspect that there are DAGAL (hence MSOL) for which it is not possible.
2The primary research question of Courcelle (1991) is a conjecture which has only quite recently been proven (Bojanczyk and Pilipczuk, 2016).
HRL∗ MSOL† Graphs
CFTL∗
CFL∗
RTL†∗ Trees
RL†∗ Strings
RGL†∗ DAGAL†
Figure 2: Containment relationships for families of regular and context-free string and tree languages, hyperedge replacement languages (HRL), monadic second order definable graph languages (MSOL), directed acyclic graph automata languages (DAGAL), and regular graph languages (RGL). ∗ indicates that the family of languages is probabilistic and † indicates that the family of languages is intersectible.
not been widely studied, and they have never been applied to NLP. We present two new results.
1. We prove that RGLs are closed under intersection (§3).
2. We give a parsing algorithm for RGL that is linear in the size of the input graph (§4).
Figure 2 summarizes the relationship of RGL to other formalisms and their properties. We conclude with a discussion of how RGL relates to other recently-discovered formalisms and its capacity to represent semantics as graphs (§5).
2 Regular Graph Languages
We use the following notation. If n is an integer, [n] denotes the set {1, . . . , n}. IfA is a set, s ∈ A∗ denotes that s is a sequence of arbitrary length, each element of which is in A. We denote by |s| the length of s. A ranked alphabet is an alphabet A paired with an arity function rank: A→ N. Definition 1. A hypergraph over a ranked alphabet Γ is a tuple G = (VG, EG, attG, labG, extG) where VG is a finite set of nodes; EG is a finite set of edges (distinct from VG); attG : EG → V ∗G maps each edge to a sequence of nodes; labG : EG → Γ maps each edge to a label such that |attG(e)| = rank(labG(e)); and extG is an ordered subset of VG called the external nodes of G.
We assume that both the elements of extG and the elements of attG(e) for each edge e are pairwise distinct. An edge e is attached to its nodes by tentacles, each labeled by an integer indicating the node’s position in attG(e) = (v1, . . . , vk). The
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
tentacle from e to vi will have label i, so the tentacle labels lie in the set [k] where k = rank(e). To express that a node v is attached to the ith tentacle of an edge e then we say vert(e, i) = v. Likewise, the nodes in extG are labeled by their position in extG. In figures, the ith external node will be labeled (i). The rank of an edge e is k if att(e) = (v1, . . . , vk) (or equivalently, rank(lab(e)) = k). The rank of a hypergraph G is the size of extG. Example 1. Hypergraph G in Figure 3 has four nodes (shown as black dots) and three hyperedges labeled a, b, and X (shown boxed). The bracketed numbers (1) and (2) denote its external nodes and the numbers between edges and the nodes are tentacle labels. Call the top node v1 and, proceeding clockwise, call the other nodes v2, v3, and v4. Call its edges e1, e2 and e3. Its definition would state:
attG(e1) = (v1, v2) labG(e1) = a
attG(e2) = (v2, v3) labG(e2) = b
attG(e3) = (v1, v4, v3) labG(e3) = X
extG = (v4, v2).
Definition 2. Let G be a hypergraph with an edge e of rank k and letH be a hypergraph also of rank k disjoint from G. The replacement of e by H is the graph G′ = G[e/H]. Its node set VG′ is VG ∪ VH where the ith node of e in G is fused with the ith external node of VH . Its hyperedge set EG′ is (EG−{e})∪EH . For each e′ ∈ EG′ , attG′(e′) = attG(e′) if e′ ∈ EG and attG′(e′) = attH(e′) if e′ ∈ EH . For each e′ ∈ EG′ , labG′(e′) = labG(e′) if e′ ∈ EG and labG′(e′) = labH(e′) if e′ ∈ EH . Its external node list is ext′G = extG. Example 2. Replacement is shown in Figure 3. We denote the replacement as G[X/H] since the edge is unambiguous given its label.
2.1 Hyperedge Replacement Grammars
Definition 3. A hyperedge replacement grammar G = (NG , TG , PG , SG) consists of a finite set of ranked nonterminal symbols NG , a finite set of ranked terminal symbols TG (disjoint from NG), a finite set of productions PG , and a start symbol SG ∈ NG . Every production in PG is of the form X → G where X ∈ NG and G is a hypergraph over NG ∪ TG . For each production, the rank of e must equal the rank of G.
For each production p : X → G, we will use L(p) to refer to X (the left-hand side of p) and R(p) to refer to G (the right-hand side of p). An
edge is a terminal edge if its label is terminal and a nonterminal edge if its label is nonterminal. A graph is a terminal graph if all of its edges are labeled with terminal symbols. The terminal subgraph of a graph is the subgraph consisting of all terminal edges and their endpoints.
Given a HRG G, we say that graph G derives graph G′, denoted G → G′, iff there is an edge e ∈ EG and a nonterminal X ∈ NG such that labG(e) = X and G′ = G[e/H], where X → H is in PG . We extend the idea of derivation to its transitive closure G →∗ G′. For every X ∈ NG we also use X to denote the connected graph consisting of a single edge e with lab(e)=X and nodes (v1, . . . , vrank(X)) such that attG(e) = (v1, . . . , vrank(X)), and we define the language LX(G) as {G | X →∗ G ∧ G is terminal}. The language of G is then L(G) = LSG (G). We call the family of languages that can produced by any HRG the hyperedge replacement languages (HRL).
We assume that terminal edges are always of rank 1 or 2, and depict them as directed edges where the direction is determined by the tentacle labels: the tentacle labeled 1 attaches to the source of the edge and the tentacle labeled 2 attaches to the target of the edge, if it exists.
Example 3. Table 1 shows a HRG deriving AMR graphs for sentences of the form ‘I need to want to need to want to ... to want to go’. Figure 4 is a graph derived by the grammar. The grammar is somewhat unnatural, a point we will return to (§5).
2.2 Properties of HRGs
A HRG can be made probabilistic just as a CFG can: for each nonterminal symbol, we define a distribution over the right-hand sides of all productions with that symbol as its left-hand side (Booth and Thompson, 1973). HRLs are not closed under intersection, just as the CFLs (which they general-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Sp : X
(1)
1 go 1
2 I
arg0 Y Zs : (1)
(2)
1
2
1
arg0
arg1
Xq : W
Y
(2)
(1) 1
2
1 1
2
arg1
arg0 Wt :
(1) 1
want
Yr : Z
X
(2)
(1) 1
2
1 1
2
arg1
arg0 Zu :
(1) 1
need
Table 1: Productions of a HRG. The labels p, q, r, s, t, and u label the productions so that we can refer to them in the text. Note that Y can rewrite in two ways, either via production r or s.
(1)
arg1
arg1
arg1
arg1
need
want
need
want
go
I
arg0
arg0
arg0
arg0 arg0
Figure 4: Graph derived by grammar in Table 1.
ize) are not: emptiness of HRLs is decidable, but given two HRLs simulating CFLs, the emptiness of their intersection is undecidable.
2.3 Regular Graph Grammars
A regular graph grammar (RGG; Courcelle 1991) is a restricted form of HRG. To explain the restrictions, we first require some definitions.
Definition 4. Given a graph G, a path in G from a node v to a node v′ is a sequence
(v0, i1, e1, j1, v1)(v1, i2, e2, j2, v2)
. . . (vk−1, ik, ek, jk, vk) (1)
such that v0 = v, vk = v′, vert(er, ir) = vr−1, and vert(er, jr) = vr, for each r ∈ {1, . . . , k}. The length of this path is k.
A path is terminal if every edge in the path has a terminal label. A path is internal if each vi is internal for 1 ≤ i ≤ k−1. Note that the endpoints v0 and vk of an internal path can be external.
Definition 5. A HRG G is a Regular Graph Grammar if each nonterminal in NG has rank at least one and for each p ∈ PG the following hold:
(C1) R(p) has at least one edge. Either it is a single terminal edge, all nodes of which are external, or each of its edges has at least one internal node.
(C2) Every pair of nodes in R(p) is connected by a terminal and internal path.
Example 4. The grammar in Table 1 is an RGG. Regular string and tree grammars are also RGGs (Figures 5 and 6).
X
(1)
Y Z
(1)1 a
1 2
1 1
b
Figure 6: RGGs generalize regular tree grammars.
We call the family of languages generated by RGGs the regular graph languages (RGLs).
3 RGLs are closed under intersection
Monadic second-order languages (MSOLs; Courcelle and Engelfriet 2011) are graph languages defined by statements in monadic second-order (MSO) logic. An MSO formula ϕ defines the graph language L(ϕ) = {G | G |= ϕ}. A full explanation is beyond the scope of this paper but we provide a brief discussion of MSO in the supplementary materials. Courcelle (1991) proves that restrictions C1 and C2 ensure that any RGL is also a MSOL.
MSOL is trivially closed under intersection, since the conjunction of two MSO statements is
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
also an MSO statement: if ϕ1 and ϕ2 are both MSO formulae, thenL(ϕ1)∩L(ϕ2) = L(ϕ1∧ϕ2). However, this does not guarantee that an arbitrary subfamily of MSOL is closed under intersection— it only guarantees that the intersection of two languages in subfamilyG are in MSOL, not necessarily inG itself. Here, we give a sufficient condition for a subfamily G to be closed under intersection.
Proposition 1. Let G be (1) a subclass of HRG, defined as a restriction on the right-hand sides of its productions that does not depend on nonterminal labels; and (2) MSO-definable. Then for any pair of languages L1, L2 ∈ G, the language L1 ∩ L2 is also in G.
Proof. Since both L1 and L2 are both in HRL and MSOL, we can look at them from both perspectives. Let G1 be a HRG deriving L1 and let φ2 be an MSO statement defining L2. Propositions 1.10 and 4.8 in Courcelle (1990) prove that the intersection of a HR language and an MSO language is in HRL, by constructing a HRG which derives all and only those graphs in the intersection of the two languages.3 This HRG has finitely many nonterminals defined by the cross product of the nonterminals of the original HRG and a finite set of ‘states’ of the MSO.4 The productions of the intersection grammar are copies of the original HRG, with different nonterminal labels. Hence we can construct HRG G∩ such that L(G∩) = L1 ∩ L2 and the productions in G∩ satisfy any restriction that G1 satisfied since the restriction is in terms of the right-hand sides of the productions but not the nonterminal labels. Therefore, G∩ is in G.
RGG satisfies the conditions of Proposition 1, so RGLs are closed under intersection. Importantly, both proofs—that RGL is in MSOL and that RGL is closed under intersection—are constructive, implying that it is possible to construct the intersection grammar.
4 RGL Parsing
To parse RGG, we will exploit the property that every nonterminal including the start symbol has rank at least one (Definition 5), and we assume
3This is a generalization to graphs of the proof that the intersection of a context-free and regular string language is a context-free string language (Bar-Hillel et al., 1961).
4For each MSO-definable language L in some set of all possible graphs L′, there exists a set A, a homomorphism h : L′ → A, and a finite subset C of A such that L = h−1(C). The finite set of ‘states’ here is the set C.
that the corresponding external node is identified in the input graph. This mild assumption may be a reasonable for applications like AMR parsing, where grammars could be designed so that the external node is always the unique root. Later we will relax this assumption.
The availability of an identifiable external node suggests a top-down algorithm, and we take inspiration from a top-down parsing algorithm the predictive top-down parsable grammars, another subclass of HRG (Drewes et al., 2015). These grammars, the graph equivalent of LL(1) string grammars, are incomparable to RGG, but the algorithms are related in their use of top-down prediction and in that they both fix an order of the edges in the right-hand side of each production.
4.1 Top-down Parsing for RGLs
Just as the algorithm of Chiang et al. (2013) generalizes CKY to HRG, our algorithm generalizes Earley’s algorithm (Earley, 1970). Both algorithms operate by recognizing incrementally larger subgraphs of the input graph, using a succinct representation for subgraphs that depends on an arbitrarily chosen marker node m of the input graph.
Definition 6. (Chiang et al. 2013; Definition 6) Let I be a subgraph of G. A boundary node of I is a node which is either an endpoint of an edge in G\I or an external node of G. A boundary edge of I is an edge in I which has a boundary node as an endpoint. The boundary representation of I is the tuple b(I) = 〈bn(I), be(I),m ∈ I〉 where
1. bn(I) is the set of boundary nodes of I 2. be(I) is the set of boundary edges of I 3. (m ∈ I) is a flag indicating whether the
marker node is in I .
Chiang et al. (2013) prove each subgraph has a unique boundary representation, and give algorithms that use only boundary representations to compute the union of two subgraphs, requiring time linear in the number of boundary nodes; and to check disjointness of subgraphs, requiring time linear in the number of boundary edges.
For each production p of the grammar, we impose a fixed order on the edges of R(p), as in Drewes et al. (2015). We discuss this order in detail in §4.2. As in Earley’s algorithm, we use dotted rules to represent partial recognition of productions: X → ē1 . . . ēi−1 · ēi . . . ēn means that we have identified the edges ē1 to ēi−1 and that we must next recognize edge ēi. We write ē and
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
v̄ for edges and nodes in productions and e and v for edges and nodes in a derived graph. When the identity of the sequence is immaterial we abbreviate it as α, for example writing X → ·α.
We present our parser as a deductive proof system (Shieber et al., 1995). The items of the parser are of the form [b(I), p : X → ē1 . . .· ēi . . . ēn, φp] where I is a subgraph that has been recognised as matching ē1, . . . , ēi−1; p : X → ē1, . . . , ēn is a production (called p) in the grammar with the edges in order; and φp : ER(p) → V ∗G is maps the endpoints of edges in R(p) to nodes in G.
For each production, p, we number the nodes in some arbitrary order. Using this, we construct the function φ0p : ER(p) → V ∗R(p) such that for ē ∈ ER(p) if att(ē) = (v̄1, v̄2) then φ0p(ē) = (v̄1, v̄2). As we match edges in the graph with edges in p, we assign the nodes v̄ to nodes in the graph. For example, if we have an edge ē in a production p such that att(ē) = (v̄1, v̄2) and we find an edge e which matches ē, then we update φp to record this fact, written φp[att(ē) = att(e)]. We also use φp to record assignments of external nodes. If we assign the ith external node to v, we write φp[extp(i) = v]. We write φ0p to represent a mapping with no grounded nodes.
Since our algorithm makes top-down predictions based on known external nodes, our boundary representation must cover the case where a subgraph is empty except for these nodes. If at some point we know that our subgraph has external nodes φ(ē), then we use the shorthand φ(ē) rather than the full boundary representation 〈φ(ē), ∅,m ∈ φ(ē)〉.
To keep notation uniform, we use dummy nonterminal S∗ 6∈ NG that derives SG via production p0. For graph G, our system includes the axiom:
[extG, p0 : S∗ →·SG , φ0p0 [extR(p0) = extG]]. Our goal is to prove:
[b(G), S∗ → SG·, φ] where φ is the union of the φps for each production p used to derive G.
As in Earley’s algorithm, we have three inference rules: PREDICT, SCAN and COMPLETE (Table 2). PREDICT is applied when the edge after the dot is nonterminal, assigning any external nodes
that have been identified. SCAN is applied when the edge after the dot is terminal. Using φp, we may already know where some of the endpoints of the edge should be, so it requires the endpoints of the scanned edge to match. COMPLETE requires that a recognized subgraph J match on the external nodes of its edge in the parent graph, and that the combined subgraphs are edge-disjoint.5
Example 5. Using the RGG in Table 1, we show how to parse the graph in Figure 7, which can be derived by applying production s followed by production u, where the external nodes of Y are (v3, v2). Assume the ordering of the edges in production s is arg1, arg0, Z; the top node is v̄1; the bottom node is v̄2; and the node on the right is v̄3; and that the marker node is not in this subgraph— we elide reference to it for simplicity. The external nodes of Y are determined top-down, so the parse of this subgraph is triggered by this item: [(v3, v2), Y →· arg1arg0Z, φs] where φs(arg1) = (v̄1, v3), φs(arg0) = (v̄1, v2), and φs(Z) = (v̄1).
Table 3 shows how we can prove the item
[〈{v3, v2}, {e3, e2}〉, Y → arg1arg0Z·, φ] The boundary representation 〈{v3, v2}, {e3, e2}〉 in this item represents the whole subgraph shown in Figure 7.
v1
v2
v3 . . .
. . .
need (e1)
arg0 (e2)
arg1 (e3)
Figure 7: Top left subgraph of Figure 4. To refer to nodes and edges in the text, they are labeled v1, v2, v3, e1, e2, and e3.
4.2 Normal Ordering
Our algorithm requires a fixed ordering of the edges in the right-hand sides of each production. We will constrain this ordering to exploit the structure of RGG productions, allowing us to bound
5We provide a soundness and completeness proof of the parser in the supplementary materials.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Name Rule Conditions
PREDICT [b(I), p : X → ē1 . . .· ēi . . . ēn, φp][q : Y → α]
[φp(ēi), Y → ·α, φ0q [extR(q) = φp(ēi)]] lab(ēi) = Y
SCAN [b(I), X → ē1 . . .· ēi . . . ēn, φp][e = edglab(ēi)(v1, . . . , vm)]
[b(I ∪ {e}), X → ē1 . . .· ēi+1 . . . ēn, φp[att(ēi) = (v1, . . . , vm)]] φp(ēi)(j) ∈ VG ⇒ φp(ēi)(j) = vert(e, j)
COMPLETE [b(I), p : X → ē1 . . .· ēi . . . ēn, φp][b(J), q : Y → α·, φq] [b(I ∪ J), X → ē1 . . .· ēi+1 . . . ēn, φp ∪ φq] φp(ēi) = φq(extR(q))
lab(ēi) = Y EI ∩ EJ = ∅
Table 2: The inference rules for the top-down parser.
Current Item Reason 1. [(v3, v2), Y → · arg1arg0Z, φs] Axiom 2. [〈{v3, v2, v1}, {e3}〉, Y → arg1· arg0Z, φs[att(arg1) = (v1, v3)]] SCAN: 1. and e3 = edgarg1(v1, v3) 3. [〈{v3, v2, v1}, {e3, e2}〉, Y → arg1arg0·Z, φs[att(arg0) = (v1, v2)]] SCAN: 2. and e2 = edgarg0(v1, v2)] 4. [(v1), Z → · need, φ0u[extR(u) = φs(Z)]] PREDICT: 3. and Z → need 5. [〈{v1}, {e1}〉, Z → need·, φu[att(need) = (v1)]] SCAN: 4. and e1 = edgneed(v1) 6. [〈{v3, v2}, {e3, e2}〉, Y → arg1arg0Z ·, φs ∪ φu] COMPLETE: 3. and 5.
Table 3: The steps of recognising that the subgraph shown in Figure 7 is derived from productions r2 and u in the grammar in Table 1.
(1) X (2)a 1 2 b
Figure 8: This graph cannot be normally ordered.
parsing complexity. If s = ē1 . . . ēn is an order, define si:j = ēi . . . ēj .
Definition 7. Let s be the order of a right-hand side of a production. s is normal if it has the following properties:
1. ē1 is connected to an external node, 2. s1:j is a connected graph for all j ∈ {1, . . . , n},
3. if ēi is nonterminal, each endpoint of ēi must be shared with ēj where ēj is terminal and j < i.
Example 6. The ordering of the edges of production s in Example 5 is normal.
Arbitrary HRGs do not necessarily admit a normal ordering. For example, the graph in Figure 8 cannot satisfy properties 2 and 3 simultaneously. However, RGGs do admit a normal ordering.
Proposition 2. If G is an RGG, for every p ∈ PG , there is a normal ordering of the edges in R(p).
Proof. If R(p) contains a single node then it must be an external node and it must have a terminal edge attached to it since R(p) must contain at least one terminal edge. If R(p) contains multiple nodes then by C2 there must be terminal internal paths between all of them, so there must be a terminal edge attached to the external node, which we use to satisfy property 1. We select terminal edges once one of their endpoints is connected to an ordered edge, and nonterminal edges once all endpoints are connected to ordered edges, possible by C2. Therefore, properties 2 and 3 are satisfied.
Normal ordering tightly constrains recognition of edges. Property 3 ensures that when we apply PREDICT, the external nodes of the predicted edge are all bound to specific nodes in the graph. Properties 1 and 2 ensures that when we apply SCAN, at least one endpoint of the edge is bound.
4.3 Parsing Complexity
Assume a normally-ordered RGG. Let the maximum number of edges in the right-hand side of any
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
production be m; the maximum number of nodes in any right-hand side of a production k; the maximum degree of any node in the input graph d; and the number of nodes in the input graph n.
Remark 1. The maximum number of nodes in any right-hand side of a production (k) is also the maximum number of boundary nodes for any subgraph in the parser.
COMPLETE combines subgraphs I and J only when the entire subgraph derived from Y has been recognized. Boundary nodes of J are also boundary nodes of I since they are nodes in the terminal subgraph of R(p) where Y connects. The boundary nodes of I ∪J are also bounded by k since are a subset of the boundary nodes of I .
Remark 2. Given a boundary node, there are at most (dm)k−1 ways of identifying the remaining boundary nodes of a subgraph that is isomorphic to the terminal subgraph of the right-hand side of a production.
The terminal subgraph of each production is connected by C2, with a maximum path length of m. For each edge in the path, there are at most d subsequent edges. Hence for the k − 1 remaining boundary nodes there are (dm)k−1 ways of choosing them.
We count instantiations of COMPLETE for an upper bound on complexity (McAllester, 2002), using similar logic to (Chiang et al., 2013). The number of boundary nodes of I, J and I ∪ J is at most k. Therefore, if we choose an arbitrary node to be some boundary node of I ∪J , there are at most (dm)k−1 ways of choosing its remaining boundary nodes. For each of these nodes, there are at most (3d)k states of their attached boundary edges: in I , in J , or in neither. The total number of instantiations is thenO(n(dm)k−1(3d)k), linear in the number of input nodes and exponential in the degree of the input graph. In the case of the AMR dataset, the maximum node degree is 17 and the average degree is 2.12.
We observe that RGGs could be relaxed to produce graphs with no external nodes by adding a dummy nonterminal S′ with rank 0 and a single production S′ → S. To adapt the parsing algorithm, we would first need to guess where the graph starts. This would add a factor of n to the complexity as the graph could start at any node, requiring n runs of the algorithm.
5 Discussion and Conclusions
RGG supports probabilistic interpretation and is closed under intersection, making it similar to regular families of string and tree languages that are widely used in NLP. The constraints of RGG also enable more efficient parsing than general HRG, and this tradeoff is reasonable since HRG is very expressive—when generating strings, it can express non-context-free languages (Engelfriet and Heyker, 1991; Bauer and Rambow, 2016), far more power than needed to express semantic graphs. On the other hand, RGG is so constrained that it may not be expressive enough: it would be more natural to derive the graph in Figure 4 from outermost to innermost predicate; but constraint C2 makes it difficult to express this, and the grammar in Table 1 does not. Perhaps we need less expressivity than HRG but more than RGG.
Since RGLs are a subfamily of both HRL and MSOL, they inherit probability and intersection closure, respectively. Courcelle (1991) discusses exactly those languages that are both HRL and MSOL, which he calls strongly contextfree languages (SCFL).6 SCFLs are defined nonconstructively, but it is natural to ask whether other, less restrictive subfamilies of SCFLs can be constructed using similar mathematical tools. Courcelle (1991) identifies the family of seriesparallel graphs as one such family, but it seems of little relevance to NLP. However, there are two recent independently-developed formalisms that may be useful. Tree-like Grammars (TLG; Matheja et al. 2015) and Restricted DAG Grammars (RDG; Björklund et al. 2016), both explicitly defined as restrictions on HRG. TLGs are in MSOL (and are closed under intersection) but we do not yet know if RDG is a subfamily of MSOL, or whether they are closed under intersection. They are both incomparable to RGG, but they share important characteristics, including the fact that the terminal subgraph of every production is connected. This means that our top-down parsing algorithm is applicable to both. In addition, if RDGs are in fact MSOL, Proposition 1 applies to them and means they are closed under intersection. We conjecture that larger, less restrictive subfamilies of SCFLs may be found based on a weaker restriction of connected terminal subgraphs, and we plan to explore these questions in future work.
6Courcelle’s definition of strongly context-free is unrelated to use of this term in NLP.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Review:
The paper is concerned in finding such a family of graph languages that is closed under intersection and can be made probabilistic.  - Strengths:  The introduction shows relevance, the overall aim, high level context and is nice to read. The motivation is clear and interesting.  The paper  is extremely clear but requires close reading and much formal background. It nicely takes into account certain differences in terminology.  It was interesting to see how the hyper-edge grammars generalize familiar grammars  and Earley's algorithm.  For example, Predict applies to nonterminal edges, and Scan applies to terminal edges.    If the parsing vs. validation in NLP context is clarified, the paper is useful because it is formally correct, nice contribution, instructive and can give new ideas to other researchers.    The described algorithm can be used in semantic parsing to rerank hypergraphs that are produced by another parser.   In this restricted way, the method can be part of the machinery what we in NLP use in natural language parsing and thus relevant to the ACL.  - Weaknesses:  Reranking use is not mentioned in the introduction.  It would be a great news in NLP context if an Earley parser would run in linear time for NLP grammars (unlike special kinds of formal language grammars).  Unfortunately, this result involves deep assumptions about the grammar and the kind of input.   Linear complexity of parsing of an input graph seem right for a top-down deterministic grammars but the paper does not recognise the fact that an input string in NLP usually gives rise to an exponential number of graphs.  In other words, the parsing complexity result must be interpreted in the context of graph validation or where one wants to find out a derivation of the graph, for example, for the purposes of graph transduction via synchronous derivations.  To me, the paper should be more clear in this as a random reader may miss the difference between semantic parsing (from strings) and parsing of semantic parses  (the current work).  There does not seem to be any control of the linear order of 0-arity edges.  It might be useful to mention that if the parser is extended to string inputs with the aim to find the (best?) hypergraph for a given external nodes, then the item representations of the subgraphs must also keep track of the covered 0-arity edges.                          This makes the string-parser variant exponential.    - Easily correctable typos or textual problems:  1)  Lines 102-106 is misleading.   While intersection and probs are true,  such distribution  cannot refer to the discussion in the above.  2) line 173:  I think you should rather talk about validation or recognition algorithms than parsing algorithms as  parsing  in NLP means usually completely different thing that is much more challenging due to the lexical and structural ambiguity.  3) lines 195-196 are unclear:  what are the elements of att_G; in what sense they are pairwise distinct.  Compare Example 1 where ext_G and att_G(e_1) are not disjoint sets.  4) l.206.  Move *rank* definition earlier and remove redundancy.  5) l. 267:  rather  immediately derives , perhaps.  6) 279: add  be   7) l. 352:  give an example of a nontrivial internal path.  8) l. 472:   define a subgraph of a hypergraph  9) l. 417, l.418:  since there are two propositions, you may want to tell how they contribute to what is quoted.  10) l. 458:  add  for   Table:                          Axiom:              this is only place where this is introduced as an axiom.                    Link to the text that says it is a trigger.  - General Discussion:  It might be useful to tell about MSOL graph languages and their yields, which are context-free string languages.                            What happens if the grammar is ambiguous and not top-down deterministic?  What if there are exponential number of parses even for the input graph due to lexical ambiguity or some other reasons.  How would the parser behave then?  Wouldn't the given Earley recogniser actually be strictly polynomial to m or k ?  Even a synchronous derivation of semantic graphs can miss some linguistic phenomena where a semantic distinction is expressed by different linguistic means.                    E.g. one language may add an affix to a verb when another language may express the same distinction by changing the object.  I am suggesting that although AMR increases language independence in parses it may have such cross-lingual challenges.  I did not fully understand the role of the marker in subgraphs.  It was elided later and not really used.  l. 509-510:                 I already started to miss the remark of lines 644-647 at this point.  It seems that the normal order is not unique.  Can you confirm this?  It is nice that def 7, cond 1 introduces lexical anchors to predictions.  Compare the anchors in lexicalized grammars.  l. 760.  Are you sure that non-crossing links do not occur when parsing linearized sentences to semantic graphs?  - Significant questions to the Authors:  Linear complexity of parsing of an input graph seem right for a top-down deterministic grammars but the paper does not recognise the fact that an input string in NLP usually gives rise to an exponential number of graphs.  In other words, the parsing complexity result must be interpreted in the context of graph validation or where one wants to find out a derivation of the graph, for example, for the purposes of graph transduction via synchronous derivations.  What would you say about parsing complexity in the case the RGG is a non-deterministic, possibly ambiguous regular tree grammar, but one is interested to use it to assign trees to frontier strings like a context-free grammar?  Can one adapt the given Earley algorithm to this purpose (by guessing internal nodes and their edges)? Although this question might seem like a confusion, it is relevant in the NLP context.  What prevents the RGGs to generate hypergraphs whose 0-arity edges (~words) are then linearised?   What principle determines how they are linearised?                  Is the linear order determined by the Earley paths (and normal order used in productions) or can one consider an actual word order in strings of a natural language?   There is no clear connection to (non)context-free string languages or sets of (non)projective dependency graphs used in semantic parsing.  What is written on lines 757-758 is just misleading:  Lines 757-758 mention that HRGs can be used to generate non-context-free languages.  Are these graph languages or string languages?    How an NLP expert should interpret the (implicit) fact that RGGs generate only context-free languages?  Does this mean that the graphs are noncrossing graphs in the sense of Kuhlmann & Jonsson (2015)?