N EURAL P ROGRAM P LANNER FOR S TRUCTURED P REDICTIONS
Jacob Biloki Inc. jacob.biloki@mosaix.ai
Chen Liang Google Brain crazydonkey@google.com Ni Lao SayMosaic Inc. ni.lao@mosaix.ai
A BSTRACT
We consider the problem of improving reinforcement learning (RL) applied to weakly supervised structured prediction (SP) – for example, given a database table and a question, perform a sequence of computation actions on the table, generating a response and a binary success-failure reward signal.
This line of research has been successful by leveraging RL to directly optimize the desired metrics of a task, such as the accuracy in question answering or BLEU score in machine translation.
However, different from the common RL settings, the environment dynamics are deterministic and known in SP, which haven’t been fully utilized by the modelfree RL methods that are usually applied.
We propose to apply model-based RL methods, which rely on planning as a primary model component.
We demonstrate the effectiveness of planning-based SP by applying our Neural Program Planner (NPP) model to an existing SP model, Memory Augmented Policy Optimization (MAPO)
Liang et al.
The proposed NPP is optimized to score a given a set of candidate programs from a pretrained search policy, deciding which program is the most promising by utilizing the space searched by the underlying model. NPP is applied after beam creation allowing the space to be observed after execution providing more rich program representation only available to NPP.
We evaluate NPP on weakly supervised program synthesis natural language (semantic parsing) by evaluating it in both a stacked learning and non stacked learning environment.
On the W IKI T ABLE Q UESTIONS benchmark, NPP achieves a new state-of-the-art of 47 .
2% accuracy.
I NTRODUCTION
Numerous results from natural language processing tasks have shown that Structured Prediction (SP) can be cast into a reinforcement learning (RL) framework, and known RL techniques can give formal performance bounds on SP tasks (Daume et al., 2009; Ross et al., 2011; Bengio et al., 2015).
RL also directly optimizes task metrics, such as, the accuracy in question answering or BLEU score in machine translation, and avoids the exposure bias problem when compaired to maximum likelihood training that is commonly used in SP (Ross et al., 2011; Ranzato et al., 2016).
However, previous works on applying RL to SP problems often use model-free RL algorithms (e.g., REINFORCE or actor-critic) and fail to leverage the characteristics of SP, which are different than typical RL tasks, e.g., playing video games (Mnih et al., 2015) or the game of Go (Silver et al., 2016b; 2017).
In most SP problems conditioned on the input x , the environment dynamics, except for the reward signal, is known, deterministic, reversible, and therefore can be searched.
This means that there is a perfect model 1 of the environment, which can be used to apply model-based RL methods that utilize planning 2 as a primary model component.
Take semantic parsing (Berant et al., 2013; Pasupat & Liang, 2015) as an example, semantic parsers trained by RL such as Neural Semantic Machine (NSM) (Liang et al., 2017; 2018) typically rely on beam search for inference – the program with the highest probability in beam is used for execution and generating answer.
However, the policy, which is used for beam search, may not be able to assign the highest probability to the correct program.
This limitation is due to the policy predicting locally normalized probabilities for each possible action based on the partially generated program, and the probability of a program is a product of these local probabilities.
For example, when applied to the W EB Q UESTIONS SP task, NSM made mistakes with two common patterns: (1) the program would ignore important information in the context; (2) the generated program does not execute to a reasonable output, but still receives high probability (spurious programs).
Resolving this issue requires using the information of the full program and its execution output to further evaluate its quality based on the context, which can be seen as planning.
This can be observed in Figure 4 where the model is asked a question “Which programming is played the most?” .
The full context of the input table (shown in Table 1) contains programming for a television station.
The top program generated by a search policy produces the wrong answer, ﬁltering by a column not relevant to the question.
If provided the correct contextual features, and if allowed to evaluate the full program forward and backward through time, we observe that a planning model would be able to better evaluate which program would produce the correct answer.
To handle errors related to context, we propose to train a value function to compute the utility of each token in a program.
This utility is evaluated by considering the program and token probability as well as the attention mask generated by the sequence-to-sequence (seq2seq) model for the underlying policy.
We also introduce beam and question context with a binary feature representing overlap from question/program and program/program, such as how many programs share a token at a given timestep.
In the experiments, we found that applying a planner that uses a learned value function to re-rank the candidates in the beam can signiﬁcantly and consistently improve the accuracy.
On the W IKI T ABLE Q UESTIONS benchmark, we improve the state-of-the-art by 0 .
9% , achieving an accuracy of 47 .
B ACKGROUND 2. W IKI T ABLE Q UESTIONS W
IKI T ABLE Q UESTIONS (Pasupat & Liang, 2015) contains tables extracted from Wikipedia and question-answer pairs about the tables.
See Table 1 as an example.
There are 2,108 tables and 18,496 question-answer pairs split into train/dev/test set.
Each table can be converted into a directed graph that can be queried, where rows and cells are converted to graph nodes while column names become labeled directed edges.
For the questions, we use string match to identify phrases that appear in the table.
We also identify numbers and dates using the CoreNLP annotation released with the dataset.
The task is challenging in several aspects.
First, the tables are taken from Wikipedia and cover a wide range of topics.
Second, at test time, new tables that contain unseen column names appear.
Third, the table contents are not normalized as in knowledge-bases like Freebase, so there are noises and ambiguities in the table annotation.
Last, the semantics are more complex comparing to previous datasets like W EB Q UESTIONS SP (Yih et al., 2016).
It requires multiple-step reasoning using a large set of functions, including comparisons, superlatives, aggregations, and arithmetic operations (Pasupat & Liang, 2015).
See (Liang et al., 2018) for more details about the functions.
2. N EURAL S EMANTIC M ACHINE
We adopt the NSM framework open sourced in the Memory Augmented Policy Optimization paper (MAPO) (Liang et al., 2017; 2018), which combines (1) a neural “programmer”, which is a seq2seq model augmented by a key-variable memory that can translate a natural language utterance to a program as a sequence of tokens, and (2) a symbolic “computer”, which is an Lisp interpreter that implements a domain speciﬁc language with built-in functions and provides code assistance by eliminating syntactically or semantically invalid choices.
For the Lisp interpreter, it added functions according to (Zhang et al., 2017; Neelakantan et al., 2016) for W IKI T ABLE Q UESTIONS , refer to (Liang et al., 2018) for further detail of the open-sourced implementation.
Same as (Liang et al., 2018)
we consider the problem of contextual program synthesis with weak supervision – i.e., no correct action sequence a is given as part of the training data, and training needs to solve the hard problem of exploring a large program space.
However, we will focus on improving decision making [Table] (planning) giving a pretrained search policy, while previous work mainly focus on learning the search policies.
P ROBLEM F ORMULATION
The problem of weakly supervised contextual program synthesis can be formulated as: generating a program a by using a parametric mapping function, ˆ a “ f θ p x q , where θ denotes the model parameters.
The quality of a generated program ˆ a is measured in terms of a scoring or reward function R p ˆ a ; x , y q with y being the expected correct answer.
The reward function evaluates a program by executing it on a real environment and comparing the emitted output against the correct answer.
For example, the reward may be binary that is 1 when the output equals the answer and 0 otherwise.
We assume that the context x includes both a natural language input and an environment, for example an interpreter or a database, on which the program will be executed.
Given a dataset of context-answer pairs, tp x i , y i qu Ni “ 1 , the goal is to ﬁnd an optimal parameter θ ˚ that parameterizes a mapping of x Ñ a with maximum empirical return on a heldout test set .
In this study we will further decompose the policy f θ into the stacking of a search model s φ p x q , which produces a set of candidate programs B given an environment x , and a value model v ω p a ; x , B q , which assigns a score s to program a given the environment and all the candidate programs.
Therefore, θ “ [Equation] ; ω s and
[Equation]
The search model s φ can be learnt by optimizing a conditional distribution π φ p a | x q that assigns a probability to each program given the context.
That is, π φ is a distribution over the countable set of all possible programs, denoted A .
Thus @ a P A : π φ p a | x q ě 0 and ř a P
A π φ p a | x q “ 1 .
Then, to synthesize candidate programs for a novel context, one may ﬁnd the most likely programs under the distribution π φ via exact or approximate inference such as beam search.
B “ s φ p x q « argmax B a P A π φ p a | x q .
π φ is typically an autoregressive model such as a recurrent neural network: ( e.g. Hochreiter & Schmidhuber (1997a))
π φ p a | x q ” ś | a | i “ t π φ p a t | a ă t , x q , where a ă t ” p a 1 , . . .
, a t ´ 1 q denotes a preﬁx of the program a .
In the absence of ground truth programs, policy gradient techniques (such as REINFORCE Williams (1992)) present a way to optimize the parameters of a stochastic policy π φ via optimization of expected return .
Given a training dataset of context-answer pairs, tp x l , y l qu Nl “ 1 , the training objective is O ER p θ q “ E a „ π φ p a | x q R p a ; x , y q .
Decision-time planning typically relies on value network Silver et al.
(2016a) trained to predict the true reward.
In the next section, however, we introduce a max-margin training objective for the value model v ω , which optimizes to rank rewarded programs higher than non-rewarded programs.
N EURAL P ROGRAM P LANNER (NPP)
We now introduce NPP by ﬁrst describing the architecture of v ω – a seq2seq model which goes over candidate program answer pairs and the ﬁnal score of a candidate program is simply the sum of its token scores (Section 4.1).
Secondly we describe the program token representation, which considers the program, question and beam context which are used to denote the utility of all tokens within a program (Section 4.2).
Finally, we describe a training procedure that is based on max-margin/ranking objective on candidate programs given a question (Section 4.3).
4. A RCHITECTURE
Here we introduce the NPP architecture (Figure 1) in the context of semantic parsing, but the framework should be broadly applicable to applying RL in structured predictions.
Given a pre-trained search policy π φ and environment x , NPP ﬁrst unrolls the policy with beam search to generate candidate programs (plans) B “ s φ p x q .
Then it scores each program a considering token a t at every step and global statistics among all programs B .
a t is represented as a context feature vector C t (details in Section 4.2).
To capture the sequential inputs the scoring component is a seq2seq model which goes over candidate program answer pairs and assigns preference scores to each program/answer token.
We implement a bi-directional recurrent network with LSTM Hochreiter & Schmidhuber (1997b); Greff et al.
(2015) cells as the ﬁrst layer of our planner C LSTM “ LSTM p C q .
The LSTM hidden state at each step is fed to a one dimensional convolutional layer with kernel size 3, in order to capture inner function scoring as most functions are of size 3-5, as a feature extractor O CNN t “ CNN p C LSTM t q .
Finally we calculate the score per token by feeding into a single node hyperbolic tangent activation layer to compute the score per token v t “ Tanh p O CNN t q of token a t . The ﬁnal score of a candidate [Table] probability
[Table]
to the search policy
[Table] Number of candidate programs having token a t at position t program v ω [Table] ř t “ 1 ..T v t is simply the sum of its token scores.
The choice of simply summing token level scores makes the score very understandable (details in Section 4.2).
Figure 2 gives implementation details of the value model.
4. P ROGRAM T OKEN R EPRESENTATION
To better score tokens based on the overall context of the environment we represent each token with a set of context features C t “ r q tok ; q attn ; p prob ; t prob ; t agree s as in Table 2.
q attn is the softmax attention across question tokens, which helps to discern which part of the question is of most importance to the model when generating the current given token.
Together q tok and q attn represents the alignment between program and query.
t prob and p prob are the probability of token a t and program a assigned by the search policy π φ , which represent the decisions from the search model.
t agree is the number of candidate programs having token a t at position t .
Access to information such as t agree is only available to NPP as it can only be used after all the candidate programs have been generated.
4. T RAINING NPP
We formulate NPP training as a learning to rank problem — optimizing pairwise ranking among candidate programs B .
Given a training dataset of context-answer pairs, tp x l , y l qu Nl “ 1 , the training objective is
[Equation]
where σ p v q “ 1 {p 1 ` e ´ v q is the sigmoid function, and v l,i “ v ω p a l,i ; x l , s φ p x l qq is the estimated value of a l,i , the i -th program generated for context x l .
We compare NPP training in two setups: a single MAPO setup, and a stack learning setup.
For the single MAPO setup the queries used to produce a pretrained MAPO model are also used to train the NPP model.
The dev and test queries are used for ablation study and ﬁnal evaluation.
Since the
NPP training queries are already used to train the MAPO model, the candidate programs are biased towards better reward (compared to those candidate generated for unseen queries).
This setup causes
NPP to learn from a different distribution as the intended dev/test data.
Surprisingly NPP is still able to improve the prediction of MAPO as will be discussed in Section 5.3.
To overcome the issue with single MAPO setup we also generate NPP training data with a stacked learning setup.
First the train and dev queries are merged and splitted into K equal portions, and with Leave-One-Out (LOO) scheme they form K train/dev sets.
Then K MAPO models are trained on K train sets.
Finally we use each of the K MAPO s to produce a stack learning dataset by running these models on their respective dev set.
The stack learning dataset is further splited for train/dev purposes for NPP.
In this way, each NPP training query is decoded by a MAPO model, which has never seen this query before, and therefore avoid the bias in training data generation.
E XPERIMENTS
Our empirical study is based on the semantic parsing benchmark, W IKI T ABLE Q UESTIONS (Pasupat & Liang, 2015).
To focus on studying the planning part of the problem we assume that the search policy is pretrained using MAPO (Liang et al., 2018), and NPP is trained to rescore given candidate programs produced by MAPO.
Additionally we show that stacked learning (Cohen & Carvalho, 2005) is helpful in correctly training a planner given pre-trained policy models.
5. E XPERIMENTAL S ETUP
Datasets.
W IKI T ABLE Q UESTIONS (Pasupat & Liang, 2015) contains tables extracted from
Wikipedia and question-answer pairs about the tables.
See Table 1 as an example.
There are 2,108 tables and 18,496 question-answer pairs.
We follow the construction in Pasupat & Liang (2015) for converting a table into a directed graph that can be queried, where rows and cells are converted to graph nodes while column names become labeled directed edges.
For the questions, we use string match to identify phrases that appear in the table.
We also identify numbers and dates using the
CoreNLP annotation released with the dataset.
Baselines.
We compare NPP to Liang et al.
(2018), the current state of the art on the W IKI T ABLEQ UESTIONS dataset.
MAPO relies on beam search to ﬁnd candidate programs, and uses the program probability according to the policy to determine the program to execute.
MAPO manages to achieve 46 .
3% accuracy on this task when using an ensemble of size 10.
We aim to show that NPP can improve on single MAPO as well as the ensemble of MAPO models.
Training Details.
We set the stacked learning parameter K “ 5 for all our experiments.
We set the batch size to be equal to 16.
We use Adam optimizer for training with a learning rate 10 ´ 3 .
We choose a size of 64 nodes for the LSTM (which becomes 128 as it is bidirectional).
The CNN consists of 32 ﬁlters with kernel size 3.
All the hyper parameters are tuned on the development set and trained for 10 epochs.
Ensemble Details.
We formulate the ensemble of K MAPO models with NPP as the sum of normalized NPP scores under different MAPO models: Let Φ “ t φ k u
Kk “ 1 be K MAPO models to be ensembled.
We deﬁne the ensembled score of a program a under context x as
[Equation]
where ¯ v ω p x q is the average score for programs in beam s kφ p x q
[Equation]
and v 1 ω backs-off v ω to ¯ v ω p x q whenever a is not in beam
[Equation]
[Table]
5. A BLATION S TUDY
In order to evaluate the effectiveness of our proposed programs token representations, we present a feature ablation test in Figure 3.
We can see that the program probability p prob produced by the search policy is the most important feature, providing the foundation to NPP scoring.
The program agreement feature t agree is also very useful.
We often observe cases for which beam B produces program with similar tokens that are not highly valued by the underlying model.
By utilizing this feature, we more strongly consider programs which were repeatedly searched by s φ .
t agree also helps to identify the programs that are very similar throughout most of the sequence to learn their divergence and grade their utility. [Table] [Table] [Table]
[Table] Question referencing features such as q tok provide signiﬁcant importance in providing the program with query level context [Table] ensuring we are ﬁltering or selecting values based on the query context.
While the help from attention feature q attn is not signiﬁcant.
5. M AIN R ESULTS
We evaluate NPP’s impact on MAPO under three different settings, in each of which NPP consistently improves the precision of MAPO.
First we consider a single MAPO trained on a single train-dev data split.
Similar to other RL models in the literature, MAPO training is a procedure with big variances.
Even trained on the same data split multiple times with different random seed gives big variances in accuracy of 0.3% (dev) and 0.5% (test).
We use the MAPO model to decode on its own train, dev and test data, in order to generate parallel splits for NPP training.
Training and applying NPP this way is able to improve precision, despite of the exposure bias in the NPP training data.
However, it does not improve on the variances.
We next consider MAPO models that were trained and evaluated on separate train/dev splits created with a Leave-One-Out (LOO) scheme.
As described in Section 4.3 we also use these splits to generate a stacked learning dataset for NPP to avoid the data bias problem.
We can see that with different data splits MAPO has signiﬁcantly higher variances on the dev set (1.1%), which is a drawback of RL models in general.
Stacked learning helps NPP to improve precision more signiﬁcantly (1.3% for dev and 1.1% for test).
It also helps to reduce the variances to 0 .
2% on both dev and test.
Finally, we consider ensembled MAPO settings, which produces the previous state of the art result.
We use the same NPP model trained from the stacked learning setting, and apply it to an ensemble of either 5, or 10 MAPO models from (Liang et al., 2018).
We can see that when applied to 5 MAPO ensemble, NPP can still improve the precision by 1.1%.
When applied to 10 MAPO ensemble, NPP can improves the precision by 0.9%.
5. E XPLAINABILITY A NALYSIS
Since the score of a program is the sum of its token scores, it is easy to visualize how NPP plan and select the correct program to execute.
We observed that there are two common situations in which MAPO chooses the wrong program from the beam – selecting a spurious program over the semantically correct program and executing the incorrect table ﬁltering or column selection.
NPP aims to overcome these non optimal decisions by taking advantage of the larger time horizon and other programs discovered so far.
For example NPP may reward earlier program tokens based on program tokens chosen much later on due to the bi-directional recurrent network.
We ﬁrst investigate how NPP demotes spurious programs.
MAPO may produce programs which return the correct answer but are not semantically correct.
NPP helps solve this by scoring semantically correct programs higher in the beam.
An example is shown in Figure 3 for the question “What venue was the latest match played at?”
when referring to a soccer (football) player given a table of his matches.
The top program in beam proposed by MAPO was to ﬁrst ﬁlter all rows for the competition string, which is incorrect considering the context of the table is only competitions.
NPP is able to reevaluate the program given the full context.
Although the ﬁrst function ( ﬁlter_in ) is typically used to ﬁlter the table for the correct row/column.
NPP learns that in this situation it is better to ﬁnd the last of all rows using the function last .
NPP, re-evaluates the ﬁrst function of the new best program as being high in utility, and scores all tokens within this function higher than the tokens in the incorrect program.
We then investigate programs from MAPO which produce wrong answers.
An example is shown in Figure 4 which is based on Table 1.
MAPO assigns a higher probability to a program in beam that ﬁlters on an incorrect column.
Because NPP knows program token overlap with the query as well as the attention matrix, it is able to better understand the question and grade the program which is more closely related to the question.
In addition to this we notice that the convolutional layer grades full functions within their context, given a kernel of size 3 the parenthesis at the beginning of program already receives a higher NPP score which we interpret as the overall score of executing the function.
[Figure] [Figure]
C ONCLUSION
Reinforcement learning applied to structured prediction suffers from limited use of the world model as well as not being able to consider future and past program context when generating a sequence.
To overcome these limitations we proposed Neural Program Planner (NPP) which is a planning step after candidate program generation. We show that an additional planning model can better evaluate overall structure value.
When applied to a difﬁcult SP task NPP improves state of the art by 0.9% and allows intuitive analysis of its scoring model per program token.