Paper:
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Harnessing the statistical power of neural networks to perform language understanding and symbolic reasoning is difficult, when it requires executing efficient discrete operations against a large knowledge-base. In this work, we introduce a Neural Symbolic Machine, which contains (a) a neural “programmer”, i.e., a sequence-to-sequence model that maps language utterances to programs and utilizes a key-variable memory to handle compositionality (b) a symbolic “computer”, i.e., a Lisp interpreter that performs program execution, and helps find good programs by pruning the search space. We apply REINFORCE to directly optimize the task reward of this structured prediction problem. To train with weak supervision and improve the stability of REINFORCE we augment it with an iterative maximum-likelihood process. NSM outperforms state-of-the-art on the WEBQUESTIONSSP dataset when trained from question-answer pairs only, without requiring any feature engineering or domain-specific knowledge.
1 Introduction
Deep neural networks have achieved impressive performance in supervised classification and structured prediction tasks such as speech recognition (Hinton et al., 2012), machine translation (Bahdanau et al., 2014; Wu et al., 2016) and more. However, training neural networks for semantic parsing (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Liang et al., 2011) or program induction, where language is mapped to a symbolic representation that is executed by a com-
puter, through weak supervision remains challenging. This is because the model must interact with a computer through non-differentiable operations at training time.
In semantic parsing, recent work handled this (Dong and Lapata, 2016; Jia and Liang, 2016) by training from manually annotated programs and avoiding program execution at training time. However, annotating programs is known to be expensive and scales poorly. In program induction, attempts to address this problem (Graves et al., 2014; Reed and de Freitas, 2016; Kaiser and Sutskever, 2015; Graves et al., 2016b; Andreas et al., 2016) either utilized low-level memory (Zaremba and Sutskever, 2015), or required memory to be differentiable (Neelakantan et al., 2015) so that the model can be trained with backpropagation. This makes it difficult to use the efficient discrete operations and memory of a traditional computer, and limited the application to synthetic or small knowledge bases.
In this paper, we propose to utilize the mem-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ory and discrete operations of a traditional computer in a novel Manager-Programmer-Computer (MPC) framework for neural program induction, which integrates three components:
1. A “manager” that provides weak supervision (e.g., ‘NYC’ in Figure 1) through a reward indicating how well a task is accomplished. Unlike full supervision, weak supervision is easy to obtain at scale (Section 3.1). 2. A “programmer” that takes natural language as input and generates a program that is a sequence of tokens (Figure 2). The programmer learns from the reward and must overcome the hard search problem of finding correct programs (Section 2.2). 3. A “computer” that executes programs in a high level programming language. Its nondifferentiable memory enables abstract, scalable and precise operations, but makes training more challenging. It provides a friendly neural computer interface by detecting and eliminating invalid choices (Section 2.1).
Within this framework, we introduce the Neural Symbolic Machine (NSM) and apply it to semantic parsing. NSM contains a sequence-to-sequence (seq2seq) “programmer” (Sutskever et al., 2014) and a symbolic non-differentiable Lisp interpreter (“computer”) that executes programs against a large knowledge-base (KB).
Our technical contribution in this work is threefold. First, to support language compositionality, we augment the standard seq2seq model with a key-variable memory to save and reuse intermediate execution results (Figure 1). This is a novel application of pointer networks (Vinyals et al., 2015) to compositional semantics.
Second, to alleviate the search problem of finding correct programs when training from questionanswer pairs (Figure 1), we use the computer to execute partial programs and prune the programmer’s search space by checking the syntax and semantics of generated programs. This generalized past work on training from weak supervision (Liang et al., 2011; Berant et al., 2013) by using the semantic denotations for structural search.
Third, to train from weak supervision and directly maximize the expected reward we turn to the REINFORCE (Williams, 1992) algorithm. Since learning from scratch is difficult for REINFORCE, we combine it with an iterative maximum likelihood (ML) process, where beam search is
used to find pseudo-gold programs, which are then used to augment the objective of REINFORCE.
On the WEBQUESTIONSSP dataset (Yih et al., 2016), NSM achieves new state-of-the-art results with weak supervision, significantly closing the gap between weak and full supervision for this task. Unlike prior work, it is trained end-toend, and does not require feature engineering or domain-specific knowledge.
2 Neural Symbolic Machines
We now introduce NSM by first describe the “computer”, a non-differentiable Lisp interpreter that executes programs against a large KB and provides code assistance (Section 2.1). We then propose a seq2seq model (“programmer”) that supports compositionality using a key-variable memory to save and reuse intermediate results (Section 2.2). Finally, we describe a training procedure that is based on REINFORCE, but is augmented with pseudo-gold programs found by an iterative ML training procedure (Section 2.3).
Before diving into details, we define the semantic parsing task: given a knowledge base K, and a question x = (w1, w2, ..., wm), produce a program or logical form z that when executed against K generates the right answer y. Let E denote a set of entities (e.g., ABELINCOLN),1 and let P denote a set of properties (e.g., PLACEOFBIRTH). A knowledge base K is a set of assertions or triples (e1, p, e2) ∈ E × P × E , such as (ABELINCOLN, PLACEOFBIRTH, HODGENVILLE).
2.1 Computer: Lisp Interpreter with Code Assistance
Semantic parsing typically requires using a set of operations to query the knowledge base and process the results. Operations learned with neural networks such as addition and sorting do not perfectly generalize to inputs that are larger than the ones observed in the training data (Graves et al., 2014; Reed and de Freitas, 2016). In contrast, operations implemented in high level programming languages are abstract, scalable, and precise, thus generalizes perfectly to inputs of arbitrary size. Based on this observation, we implement operations necessary for semantic parsing with an ordinary programming language instead of trying to learn them with a neural network.
1We also consider numbers (e.g., “1.33”) and date-times (e.g., “1999-1-1”) as entities.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
( Hop v p )⇒ {e2|e1 ∈ v, (e1, p, e2) ∈ K} ( ArgMax v p )⇒ {e1|e1 ∈ v,∃e2 ∈ E : (e1, p, e2) ∈ K,∀e : (e1, p, e) ∈ K, e2 ≥ e} ( ArgMin v p )⇒ {e1|e1 ∈ v,∃e2 ∈ E : (e1, p, e2) ∈ K,∀e : (e1, p, e) ∈ K, e2 ≤ e}
( Filter v1 v2 p )⇒ {e1|e1 ∈ v1,∃e2 ∈ v2 : (e1, p, e2) ∈ K}
Table 1: Interpreter functions. v represents a variable, p a property in Freebase. ≥ and ≤ are defined on numbers and dates.
We adopt a Lisp interpreter as the “computer”. A program C is a list of expressions (c1...cN ), where each expression is either a special token ”RETURN” indicating the end of the program, or a list of tokens enclosed by parentheses ”( F A0 ... AK )”. F is one of the functions in Table 1, which takes as input K arguments of specific types. Table 1 defines the semantics of each function and the types of arguments (either a property p or a variable v). When functions are executed, they return an entity list that is the expression’s denotation in K, and save it to a new variable.
By introducing variables that save the intermediate results of execution, the program naturally models language compositionality and describes from left to right a bottom-up derivation of the full meaning of the natural language input, which is convenient in a seq2seq model (Figure 1). This is reminiscent of the floating parser (Wang et al., 2015; Pasupat and Liang, 2015), where a derivation tree that is not grounded in the input is incrementally constructed.
The set of programs defined by our functions is equivalent to the subset of λ-calculus presented in (Yih et al., 2015). We did not use full Lisp programming language here, because constructs like control flow and loops are unnecessary for most current semantic parsing tasks, and it is simple to add more functions to the model when necessary.
To create a friendly neural computer interface, the interpreter provides code assistance to the programmer by producing a list of valid tokens at each step. First, a valid token should not cause a syntax error: e.g., if the previous token is “(”, the next token must be a function name, and if the previous token is “Hop”, the next token must be a variable. More importantly, a valid token should not cause a semantic (run-time) error: this is detected using the denotation saved in the variables. For example, if the previously generated tokens were “( Hop v”, the next available token is restricted to properties {p | ∃e, e′ : e ∈ v, (e, p, e′) ∈ K} that are reachable from entities in v in the KB graph. These checks are enabled by the variables and can
be derived from the definition of the functions in Table 1. With this code assistance, the interpreter prunes the “programmer”’s search space by orders of magnitude, and enables learning from weak supervision on a large KB.
2.2 Programmer: Seq2seq Model with Key-Variable Memory
Given the “computer”, the “programmer” needs to map natural language into a program, which is a sequence of tokens that references operations and values in the “computer”. We base our programmer on a standard seq2seq model with attention, but extend it with a key-variable memory that allows the model to learn to represent and refer to program variables (Figure 2).
Sequence-to-sequence models consist of two RNNs, an encoder and a decoder. We used a 1-layer GRU (Cho et al., 2014) for both the encoder and decoder. Given a sequence of words w1, w2...wm, each word wt is mapped to an embedding qt (embedding details are Section 3). Then, the encoder reads these embeddings and updates its hidden state step by step using ht+1 = GRU(ht, qt, θEncoder), where θEncoder are the GRU parameters. The decoder updates its hidden states ut by ut+1 = GRU(ut, ct−1, θDecoder), where ct−1 is the embedding of last step’s output token at−1, and θDecoder are the GRU parameters. The last hidden state of the encoder hT is used as the decoder’s initial state. We also adopt a dot-product attention similar to Dong and Lapata (2016). The tokens of the program a1, a2...an are generated one by one using a softmax over the vocabulary of valid tokens at each step, as provided by the “computer” (Section 2.1).
To achieve compositionality, the decoder must learn to represent and refer to intermediate variables whose value was saved in the “computer” after execution. Therefore, we augment the model with a key-variable memory, where each entry has two components: a continuous embedding key vi, and a corresponding variable token Ri referencing the value in the “computer” (see Figure 2).
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Key Variable
v1 R1(m.USA) Execute ( Argmax R2 Population )
Execute Return
m.NYCKey Variable
... ...
v3 R3(m.NYC)
Key Variable
v1 R1(m.USA)
v2 R2(list of US cities)
Execute ( Hop R1 !CityIn )
Hop R1 !CityIn( )
Largest city ( Hop R1in US GO !CityIn Argmax R2( )Population)
R2 Population ReturnArgmax )(
Entity Resolver
Figure 2: Semantic Parsing with NSM. The key embeddings of the key-variable memory are the output of the sequence model at certain encoding or decoding steps. For illustration purposes, we also show the values of the variables in parentheses, but the sequence model never sees these values, and only references them with the name of the variable (“R1”). A special token “GO” indicates the start of decoding, and “RETURN” indicates the end of decoding.
During encoding, we use an entity linker to link text spans (e.g., “US”) to KB entities. For each linked entity we add a memory entry where the key is the average of GRU hidden states over the entity span, and the variable token (R1) is the name of a variable in the computer holding the linked entity (m.USA) as its value. During decoding, when a full expression is generated (i.e., the decoder reads ”)”), it gets executed, and the result is stored as the value of a new variable in the “computer”. This variable is keyed by the GRU hidden state at that step. Every time a new variable is added to memory, the variable token is added to the output vocabulary of the decoder. The final answer returned by the “programmer” is the value of the last computed variable.
Note that, similar to pointer networks (Vinyals et al., 2015), the key embeddings for variables are dynamically generated for each example. During training, the model learns to represent variables by backpropagating gradients from a time step where a variable is selected by the decoder, through the key-variable memory, to an earlier time step when the key embedding was computed. Thus, the encoder/decoder learns to generate representations for variables such that they can be used at the right time to construct the correct program.
While the key embeddings are differentiable, the values referenced by the variables (lists of entities), stored in the “computer”, are symbolic and non-differentiable. This distinguishes the keyvariable memory from other memory-augmented neural networks that use continuous differentiable embeddings as the value of each memory entry (Weston et al., 2014; Graves et al., 2016a).
2.3 Training NSM with Weak Supervision
NSM executes non-differentiable operations against a KB, and thus end-to-end backpropa-
gation is not possible. Therefore, we base our training procedure on REINFORCE (Williams, 1992; Norouzi et al., 2016).
When the reward signal is sparse and the search space is large, it is common to utilize some full supervision to pre-train REINFORCE (Silver et al., 2016). Since we assume only weak supervision, we suggest an iterative ML procedure for finding pseudo-gold programs that will bootstrap REINFORCE.
REINFORCE We can formulate training as a reinforcement learning problem: given a question x, the state, action and reward at each time step t ∈ {0, 1, ..., T} are (st, at, rt). Since the environment is deterministic, the state is defined by the question x and the action sequence: st = (x, a0:t−1), where a0:t−1 = (a0, ..., at−1) is the history of actions at time t. A valid action at time t is at ∈ A(st), where A(st) is the set of valid tokens given by the “computer”. Since each action corresponds to a token, the full history a0:T corresponds to a program. The reward rt = I[t = T ] · F1(x, a0:T ) is non-zero at the last step of decoding, and is the F1 score computed comparing the gold answer and the answer generated by executing the program a0:T . Thus, the reward of a program or action sequence is
R(x, a0:T ) = ∑ t rt = F1(x, a0:T ).
The agent’s decision making is characterized by a policy, πθ(s, a) = Pθ(at = a|x, a0:t−1), where θ are the model parameters. Since the environment is deterministic, the probability of an action sequence a0:T is
Pθ(a0:T |x) = ∏ t Pθ(at | x, a0:t−1).
We can define our objective to be the expected reward and use policy gradient methods such as
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
REINFORCE for training. The objective and gradient are:
JRL(θ) = ∑ x EPθ(a0:T |x)[R(x, a0:T )],
∇θJRL(θ) = ∑ x ∑ a0:T Pθ(a0:T | x) · [R(x, a0:T )−
B(x)] · ∇θ logPθ(a0:T | x), where B(x) = ∑
a0:T Pθ(a0:T | x)R(x, a0:T ) is a
baseline that reduces the variance of the gradient estimation without introducing bias.
While REINFORCE assumes a stochastic policy, we rely on beam search for gradient estimation. Thus, in contrast with common practice of approximating the gradient by sampling from the model, we use the top-k action sequences (programs) in the beam with normalized probabilities. This allows training to focus on sequences with high probability, which are on the decision boundaries, and reduces the variance of the gradient.
Empirically (and in line with prior work), REINFORCE converged slowly and often got stuck in local optima (see Section 3). The difficulty of training resulted from the sparse reward signal in the large search space, which caused model probabilities for programs with non-zero reward to be very small at the beginning. If the beam size k is small, good programs fall off the beam, leading to zero gradients for all programs in the beam. If the beam size k is large, training is very slow, and the normalized probabilities of good programs when the model is untrained are still very small, leading to (1) near zero baselines, thus near zero gradients on ”bad” programs (2) near zero gradients on good programs due to the low probability Pθ(a0:T | x). To combat this, we present an alternative training strategy based on maximum-likelihood.
Iterative ML If we had gold programs, we could directly optimize their likelihood. Since we do not have gold programs, we can perform an iterative procedure (similar to hard EM), where we search for good programs given fixed parameters, and then optimize the probability of the best program found so far. We do decoding on an example with a large beam size and declare abest0:T (x) to be the pseudo-gold program that achieved highest reward with shortest length among the programs decoded on x in all previous iterations. Then, we can optimize the ML objective:
JML(θ) = ∑ x logPθ(a best 0:T (x) | x) (1)
A question x is not included if we did not find any program with positive reward.
Training with iterative ML is fast because there is at most one program per example and the gradient is not weighted by model probability. while decoding with a large beam size is slow, we could train for multiple epochs after each decoding. This iterative process has a bootstrapping effect that a better model leads to a better program abest0:T (x) through decoding, and a better program abest0:T (x) leads to a better model through training.
Even with a large beam size, some programs are hard to find because of the large search space. A common solution to this problem is to use curriculum learning (Zaremba and Sutskever, 2015; Reed and de Freitas, 2016). The size of the search space is controlled by both the set of functions used in the program and the program length. We apply curriculum learning by gradually increasing both these quantities (see details in Section 3) when performing iterative ML.
Nevertheless, iterative ML uses only pseudogold programs and does not optimize the objective we truly care about. This has two adverse effects: (1) The best program abest0:T (x) could be a spurious program that accidentally produces the correct answer (e.g., using the property PLACEOFBIRTH instead of PLACEOFDEATH when the two places are the same), and thus does not generalize to other questions. (2) Because training does not observe full negative programs, the model often fails to distinguish between tokens that are related to one another. For example, differentiating PARENTSOF vs. SIBLINGSOF vs. CHILDRENOF can be challenging. We now present learning where we combine iterative ML with REINFORCE.
Augmented REINFORCE To bootstrap REINFORCE, we can use iterative ML to find pseudogold programs, and then add these programs to the beam with a reasonably large probability. This is similar to methods from imitation learning (Ross et al., 2011; Jiang et al., 2012) that define a proposal distribution by linearly interpolating the model distribution and an oracle.
Algorithm 1 describes our overall training procedure. We first run iterative ML for NML iterations and record the best program found for every example xi. Then, we run REINFORCE, where we normalize the probabilities of the programs in beam to sum to (1 − α) and add α to the probability of the best found program C∗(xi).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Algorithm 1 IML-REINFORCE Input: question-answer pairs D = {(xi, yi)}, mix ratio α, reward function R(·), training iterations NML, NRL, and beam sizes BML, BRL. Procedure: Initialize C∗x = φ the best program so far for x Initialize model θ randomly . Iterative ML for n = 1 to NML do
for (x, y) in D do C← Decode BML programs given x for j in 1...|C| do if Rx,y(Cj) > Rx,y(C∗x) then C∗x ← Cj θ ←ML training with DML = {(x,C∗x)}
Initialize model θ randomly . REINFORCE for n = 1 to NRL do
DRL ← ∅ is the RL training set for (x, y) in D do
C← Decode BRL programs from x for j in 1...|C| do
if Rx,y(Cj) > Rx,y(C∗x) then C∗x ← Cj C← C ∪ {C∗x} for j in 1...|C| do
p̂j ← (1−α)· pj∑ j′ pj′
where pj = Pθ(Cj | x) if Cj = C∗x then p̂j ← p̂j + α DRL ← DRL ∪ {(x,Cj , p̂j)}
θ ← REINFORCE training with DRL
Consequently, the model always puts a reasonable amount of probability on a program with high reward during training.
On top of imitation learning, our approach is related to the common practice in reinforcement learning (Schaul et al., 2016) to replay rare successful experiences to reduce the training variance and improve training efficiency. This is also similar to recent developments (Wu et al., 2016) in machine translation, where ML and RL objectives are linearly combined, because anchoring the model to some high-reward outputs stabilizes training.
3 Experiments and Analysis
We now empirically show that NSM can learn a semantic parser from weak supervision over a large KB. We evaluate on WEBQUESTIONSSP, a challenging semantic parsing dataset with strong baselines. Experiments show that NSM achieves new state-of-the-art performance on WEBQUESTIONSSP with weak supervision, and significantly closing the gap between weak and full supervisions for this task. Hyper-parameter settings are in the supplemental material due to page limits.
3.1 The WEBQUESTIONSSP dataset
Collecting semantic parse labels is a difficult, time consuming task, but can increase the accuracy of state-of-the-art question-answering systems by
full supervision. The WEBQUESTIONSSP dataset (Yih et al., 2016) contains semantic parses for the questions from WEBQUESTIONS (Berant et al., 2013) that are answerable using Freebase. It consists of 3,098 question-answer pairs for training and 1,639 for testing. These questions were collected using Google Suggest API and the answers were originally obtained using Amazon Mechanical Turk and updated by annotators who were familiar with the design of Freebase. We further separated out 620 questions from the training set as a validation set. For query pre-processing we used an in-house named entity linking system to find the entities in a question. The quality of the entity linker is similar to that of (Yih et al., 2015) at 94% of the gold root entities being included. Similar to Dong and Lapata (2016), we replaced named entity tokens with a special token ”ENT”. For example, the question ”who plays meg in family guy” is changed to ”who plays ENT in ENT ENT”.
Following (Yih et al., 2015) we used the last publicly available snapshot of Freebase (Bollacker et al., 2008). Since NSM training requires random access to Freebase during decoding, we preprocessed Freebase by removing predicates that are not related to world knowledge (starting with ”/common/”, ”/type/”, ”/freebase/”),2 and removing all text valued predicates, which are rarely the answer. Out of all 23K relations, 434 relations are removed during preprocessing. This results in a graph that fits in memory with 23K relations, 82M nodes, and 417M edges.
3.2 Model And Training Details
For pre-trained word embeddings, we used the 300 dimension GloVe word embeddings trained on 840B tokens (Pennington et al., 2014). On the encoder side, we added a projection matrix to transform the embeddings into 50 dimensions. On the decoder side, we used the same GloVe embeddings to construct an embedding for each property using its Freebase id, and also added a projection matrix to transform this embedding to 50 dimension. A Freebase id contains three parts: domain, type, and property. For example, the Freebase id for PARENTSOF is “/people/person/parents”. “people” is the domain, “person” is the type and ”parents” is the property. The embedding is constructed by concatenating the average of word embeddings in the domain and type name
2We kept “/common/topic/notable types”.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
to the average of word embeddings in the property name. For example, if the embedding dimension is 300, the embedding dimension for “/people/person/parents” will be 600. The first 300 dimensions will be the average of the embeddings for “people” and “person”, and the second 300 dimensions will be the embedding for “parents”.
Inspired by the staged generation process in Yih et al. (2015), curriculum learning includes two steps. We first run iterative ML for 10 iterations with programs constrained to only use the “Hop” function and the maximum number of expressions is 2. Then, we run iterative ML again, but use both “Hop” and “Filter”, and the maximum number of expressions is 3. However, the relations used by “Hop” are restricted to those that appeared in abest0:T (q) in the first step.
3.3 Results and discussion
We evaluate performance using the offical evaluation script for WEBQUESTIONSSP. Because the answer to a question may contain multiple entities or values, precision, recall and F1 are computed based on the output of each individual question, and average F1 is reported as the main evaluation metric. Accuracy measures the proportion of questions that are answered exactly.
A comparison to STAGG, the prior state-of-theart model (Yih et al., 2016, 2015) is shown in Table 2. Our model beats STAGG with weak supervision by a significant margin on all metrics, while relying on no feature engineering or handcrafted rules. When STAGG is trained with strong supervision it obtains an F1 of 71.7, and thus NSM closes half the gap between training with weak and strong supervision.
Model Prec. Rec. F1 Acc. STAGG 67.3 73.1 66.8 58.8 NSM 70.8 76.0 69.0 59.5
Table 2: Results on the test set. Average F1 is the main evaluation metric and NSM outperforms STAGG with no domainspecific knowledge or feature engineering.
Four key ingredients lead to the final performance of NSM. The first one is the neural computer interface that provides code assistance by checking for syntax and semantic errors. We find that semantic checks are very effective for opendomain KBs with a large number of properties. For our task, the average number of choices is reduced from 23K per step (all properties) to less
than 100 (the average number of relations connected to an entity).
The second ingredient is augmented REINFORCE training. Table 3 compares augmented REINFORCE, REINFORCE, and iterative ML on the validation set. REINFORCE gets stuck in local maxima and performs poorly, while iterative ML training is not directly optimizing the F1 measure, and achieves sub-optimal results. In contrast, augmented REINFORCE is able to bootstrap using pseudo-gold programs found by iterative ML and achieves the best performance on both the training and validation set.
Settings Train F1 Valid F1 iterative ML 68.6 60.1 REINFORCE 55.1 47.8 Augmented REINFORCE 83.0 67.2
Table 3: Average F1 on the validation set for augmented REINFORCE, REINFORCE, and iterative ML.
The third ingredient is curriculum learning during iterative ML. We compare the performance of the best programs found with and without curriculum learning in Table 4. We find that the best programs found with curriculum learning are substantially better than those found without curriculum learning by a large margin on every metric.
Settings Prec. Rec. F1 Acc. No curriculum 79.1 91.1 78.5 67.2 Curriculum 88.6 96.1 89.5 79.8
Table 4: Evaluation of the program with the highest F1 score in the beam (abest0:t ) with and without curriculum learning.
The last important ingredient is reducing overfitting. Given the small size of the dataset, overfitting is a major problem for training neural network models. We show the contributions of different techniques for controlling overfitting in Table 5. Note that after all the techniques have been applied, the model is still overfitting with training F1@1=83.0% and validation F1@1=67.2%.
Error analysis Two main sources of errors arise in manual error analysis
1. Search failure: the correct program is not found during search for pseudo-gold programs, either because the beam size is not large enough, or because the set of functions implemented by the interpreter is insufficient.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Settings ∆ F1@1 -Pretrained word embeddings -5.5 -Pretrained property embeddings -2.7 -Dropout on GRU input and output -2.4 -Dropout on softmax -1.1 -Anonymize entity tokens -2.0
Table 5: Contributions of different overfitting techniques on the validation set.
The 89.5% F1 score in Table 4 indicates that at least 10% of the queries are of this kind. 2. Ranking failure: Pseudo-gold programs with high reward are found, but are not ranked at the top. Because training error is low, this is largely due to over-fitting. The 67.2% F1 score in Table 3 indicates that about 20% of examples are of this kind.
4 Related work
Among deep learning models for program induction, Reinforcement Learning Neural Turing Machines (RL-NTMs) (Zaremba and Sutskever, 2015) are the most similar to NSM, as a nondifferentiable machine is controlled by a sequence model. Therefore, both models rely on REINFORCE for training. The main difference between the two is the abstraction level of the programming language. RL-NTM uses lower level operations such as memory address manipulation and byte reading/writing, while NSM uses a high level programming language over a large knowledge base that includes operations such as following properties from entities, or sorting based on a property, which is more suitable for representing semantics. Earlier work such as OOPS (Schmidhuber, 2004) has desirable characteristics, such as the ability to define new functions and modify code storage. These remain to be future improvements for NSM.
We formulate NSM training as an instance of reinforcement learning (Sutton and Barto, 1998) in order to directly optimize the task reward of the structured prediction problem (Norouzi et al., 2016). Compared to imitation learning methods (Daume et al., 2009; Ross et al., 2011) that interpolate a model distribution with an oracle, NSM needs to solve a challenging search problem of training from weak supervisions in a large program space. Our solution employs two techniques (a) a symbolic “computer” helps find good programs by pruning the search space (b) an it-
erative maximum likelihood (ML) process, where beam search is used to find pseudo-gold programs. Wiseman and Rush (Wiseman and Rush, 2016) proposed a max-margin approach to train a sequence-to-sequence scorer. However, their training procedure is more involved, and we did not implement it in this work.
NSM is similar to Neural Programmer (Neelakantan et al., 2015) and Dynamic Neural Module Network (Andreas et al., 2016) in that they all solve the problem of semantic parsing from structured data, and generate programs using similar semantics. The main difference between these approaches is how an intermediate result (the memory) is represented. Neural Programmer and Dynamic-NMN chose to represent results as vectors of weights (row selectors and attention vectors), which enables backpropagation and search through all possible programs in parallel. However, their strategy is not applicable to a large KB such as Freebase, which contains about 100M entities, and more than 20k properties. Instead, NSM chooses a more scalable approach, where the “computer” saves intermediate results, and the neural net only refers to them with variable names (e.g., “R1” for all cities in the US).
5 Conclusion
We propose the Manager-Programmer-Computer framework for neural program induction. It integrates neural networks with a symbolic nondifferentiable computer to support abstract, scalable and precise operations through a friendly neural computer interface. Within this framework, we introduce the Neural Symbolic Machine, which integrates a sequence-to-sequence neural “programmer” with key-variable memory, and a Lisp interpreter with code assistance. Because the interpreter is non-differentiable and to directly optimize the task reward, we apply reinforcement learning and use pseudo-gold programs found by an iterative ML training process to bootstrap training. NSM achieves new state-of-the-art results on a challenging semantic parsing dataset with weak supervision, and significantly closing the gap between weak and full supervision. It is trained endto-end, and does not require any feature engineering or domain-specific knowledge.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplemental Material
A.1 More Model Details The dimension of encoder hidden state, decoder hidden state and key embeddings are all 50. The embeddings for the functions and special tokens (e.g., ”UNK”, ”GO”) are randomly initialized by a truncated normal distribution with mean=0.0 and stddev=0.1. All the weight matrices are initialized with a uniform distribution in [− √ 3 d , √ 3 d ] where d is the input dimension. Dropout rate is set to 0.5, and we see a clear tendency that larger dropout produces better performance, indicating overfitting is a major problem for learning.
A.2 More Training Details In iterative maximum likelihood training, the decoding uses beam size k = 100 to update the approximate gold programs and the model is trained for 20 epochs after each decoding. We use Adam optimizer (Kingma and Ba, 2014) with initial learning rate 0.001 for optimization. In our experiment, this process usually converges after a few (5-8) iterations.
For REINFORCE training, the best hyperparameters are chosen using the validation set. We use beam size k = 5 for decoding, and α is set to 0.1. Because the dataset is small and some relations are only used once in the whole training set, we train the model on the entire training set for 200 iterations with the best hyperparameters. Then we train the model with learning rate decay until convergence. Learning rate is decayed as gt = g0×β max(0,t−ts) m , where g0 = 0.001, β = 0.5 m = 1000, and ts is the number of training steps at the end of iteration 200.
Since decoding needs to query the knowledge graph constantly, the speed bottleneck for training is decoding. We address this problem in our implementation by partitioning the dataset, and using multiple decoders in parallel to handle each partition. We use 100 decoders, which queries 50
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
KG servers, and one trainer. The neural network model is implemented in TensorFlow. Since the model is small, we didn’t see a significant speedup by using GPU, so all the decoders and the trainer are using CPU only.
A.3 Extra Figures
Programmer ComputerManager
program
code assist
question & reward
answer built-in functions
Weak supervision
Neural knowledge base (built-in data)
non-differentiable
Symbolic
Figure 3: Manager-Programmer-Computer framework
Encoder a0 a1 an
SoftMax
q0 q1 qm
dot product
Linear Projection 300*50=15k
Linear Projection 600*50=30k
GRU 2*50*3*50=15k
Attention
Decoder
GRU 2*50*3*50=15k
Linear Projection 100*50=5k
Figure 4: Seq2Seq model architecture with dot-product attention and Dropout at GRU input, output, and softmax layers.
KG server 1
KG server m
…...
Decoder 1
Decoder 2
Decoder n
…...
Trainer
QA pairs 1
QA pairs 2
QA pairs n
Solutions 1
Solutions 2
Solutions n
…...…...
Model checkpoint
Figure 5: System Architecture. 100 decoders, 50 KG servers and 1 trainer.

Review:
This paper introduces Neural Symbolic Machines (NSMs) --- a deep neural model equipped with discrete memory to facilitate symbolic execution. An NSM includes three components: (1) a manager that provides weak supervision for learning, (2) a differentiable programmer based on neural sequence to sequence model, which encodes input instructions and predicts simplified Lisp programs using partial execution results stored in external discrete memories. (3) a symbolic computer that executes programs and provide code assistance to the programmer to prune search space. The authors conduct experiments on a semantic parsing task (WebQuestionsSP), and show that (1) NSM is able to model language compositionality by saving and reusing intermediate execution results, (2) Augmented REINFORCE is superior than vanilla REINFROCE for sequence prediction problems, and (3) NSM trained end-to-end with weak supervision is able to outperform existing sate-of-the-art method (STAGG).  - Strengths  * The idea of using discrete, symbolic memories for neural execution models is novel.                    Although in implementation it may simply reduce to copying previously executed variable tokens from an extra buffer, this approach is still impressive since it works well for a large-scale semantic parsing task.  * The proposed revised REINFORCE training schema using imperfect hypotheses derived from maximum likelihood training is interesting and effective, and could inspire future exploration in mixing ML RL training for neural sequence-to-sequence models.  * The scale of experiments is larger than any previous works in modeling neural execution and program induction. The results are impressive.  * The paper is generally clear and well-written, although there are some points which might require further clarification (e.g., how do the keys ($v_i$'s in Fig. 2) of variable tokens involved in computing action probabilities? Conflicting notations: $v$ is used to refer to variables in Tab. 1 and memory keys in Fig 1.).  Overall, I like this paper and would like to see it in the conference.  * Weaknesses  * [Choice of Dataset] The authors use WebQuestionsSP as the testbed. Why not using the most popular WebQuestions (Berant et al., 2013) benchmark set? Since NSM only requires weak supervision, using WebQuestions would be more intuitive and straightforward, plus it could facilitate direct comparison with main-stream QA research.  * [Analysis of Compositionality] One of the contribution of this work is the usage of symbolic intermediate execution results to facilitate modeling language compositionality. One interesting question is how well questions with various compositional depth are handled. Simple one-hop questions are the easiest to solve, while complex multi-hop ones that require filtering and superlative operations (argmax min) would be highly non-trivial. The authors should present detailed analysis regarding the performance on question sets with different compositional depth.  * [Missing References] I find some relevant papers in this field missing. For example, the authors should cite previous RL-based methods for knowledge-based semantic parsing (e.g., Berant and Liang., 2015), the sequence level REINFORCE training method of (Ranzato et al., 2016) which is closely related to augmented REINFORCE, and the neural enquirer work (Yin et al., 2016) which uses continuous differentiable memories for modeling neural execution.  * Misc.  * Why is the REINFORCE algorithm randomly initialized (Algo. 1) instead of using parameters pre-trained with iterative ML?  * What is KG server in Figure 5?