A Family of Cognitively Realistic Parsing Environments for Deep Reinforcement Learning
Abstract
The hierarchical syntactic structure of natural language is a key feature of human cognition that enables us to recursively construct arbitrarily long sentences, supporting communication of complex, relational information.
In this work, we describe a framework in which learning cognitively realistic left-corner parsers can be formalized as a Reinforcement Learning problem, and introduce a family of cognitively realistic chart-parsing environments to evaluate potential psycholinguistic implications of RL algorithms.
We report how several baseline Q-learning and Actor Critic algorithms, both tabular and neural, perform on subsets of the
Penn Treebank corpus.
We observe a sharp increase in difﬁculty as parse trees get slightly more complex, indicating that hierarchical reinforcement learning might be required to solve this family of environments.
Introduction
We introduce a framework in which we can start exploring how reinforcement learning (RL; Sutton and Barto 2018) algorithms scale up against human cognitive performance, as captured by the syntactic parsing problem.
Parsers grounded in contemporary generative linguistic theory involve rich, hierarchically structured representations and complex rule systems that pose signiﬁcant challenges for RL algorithms.
We begin with a simple example to illustrate the type of psycholinguistic task we modeled our environments on, which is non-cumulative moving-window self-paced reading [Just and Carpenter, 1980, Just et al., 1982].
In such tasks, a sentence is displayed on the screen with the words hidden (covered by dashes), and only one word is uncovered at a time with a key press (usually, the spacebar).
The human reader decides when to press the spacebar to uncover the next word which automatically hides the current word – hence the name of self-paced reading.
Self-paced reading tasks mimic an essential aspect of naturally-occurring language comprehension with auditory stimuli: the signal is strictly linearly and strictly incrementally presented one word at a time.
Just as in naturally-occurring verbal interactions, and unlike in normal reading situations, the linguistic signal cannot be ‘rewound’ to previous words – we cannot just look back and reread previous parts of the text – or ‘fast-forwarded’ to subsequent words – we cannot jump ahead to parts of the text that do not immediately follow the word currently being read.
We use a chart parser (Earley 1970, Tomita 1986, Scott 2008) with a cognitively realistic eager left-corner parsing strategy (Resnik 1992, Hale 2014 a.o.) to provide the reward structure, thereby guiding the reinforcement learning process.
Running this eager left-corner parser on a simple input sentence will shed light on its inner workings.
Assume we have a simple grammar with three phrase structure rules (PSRs) ( i ) S → NP VP, ( ii ) NP → Det N, and ( iii ) VP → V. Also, assume that we are reading the sentence A boy sleeps in a self-paced reading task.
We start with a screen in which all words are covered with dashes: - - - - - - - - - - .
After the ﬁrst spacebar press, the ﬁrst word is revealed: A - - - - - - - - - . The parser recognizes its syntactic category Det (determiner) and takes a series of parsing steps that construct the leftmost tree in Figure 1.
We see here the eager left-corner nature of our parser: PSR ( ii ), which has Det as its left branch/corner, is triggered as soon as we recognize that the ﬁrst word A is a determiner.
This partial tree is only implicitly constructed in the chart parser: the chart does not store trees, but instead contains edges, which are left-corner based hypotheses about the possible syntactic structures we can associate with the linguistic input received so far.
After another spacebar press, the noun is revealed ( - boy - - - - - - ), its syntactic category N is recognized, and the richer partial tree shown in the middle of Figure 1 is constructed after a series of parsing steps.
Finally, the verb is revealed after one more spacebar press ( - - - - sleeps ), its syntactic category V is recognized, and the ﬁnal tree structure in Figure 1 is constructed.
[Equation]
== ⇒
S
VP
V sleeps
NP
N boy
Det
A
Both humans and RL agents may get ‘lost’ in these parsing mazes, for example, when encountering a so-called ‘garden path’ example like The horse raced past the barn fell (Bever 1970 among many others).
This sentence might seem ungrammatical, but it is in fact grammatical under the reduced relative clause interpretation that can be paraphrased as ‘the horse that was raced past the barn fell.’
To access that interpretation, one can compare the previous example with the sentence The children taught by the Berlitz method passed the test (Crain and Steedman 1985).
This second example does not garden-path the reader because the most likely interpretation of taught in this sentence is the non-ﬁnite past-participle interpretation.
This is in contrast to raced in the ﬁrst example, which garden-paths the reader because its most likely interpretation is the incorrect, ﬁnite simple past one.
This interpretation leads the reader down an incorrect garden path during incremental parsing.
In brief, parsing tasks can be viewed as executing a certain protocol, and RL is a family of methods to learn protocols.
The paper makes two contributions.
First, we introduce a new family of cognitively realistic parsing environments for RL that are ordered by the complexity of the parsing problems they pose.
Second, we study the performance of 15 agents in the two easiest parsing environments (height-4 and height-5 trees).
The 15 agents fall into 3 classes: tabular, Deep Q Network (DQN), and Actor-Critic (AC) agents; for the latter
[Table]
Family of Parsing Environments 2.1 Parsing Actions
It is perhaps easiest to introduce our formalization of the parsing environment by examining an example episode perfectly played by a trained parsing agent, shown in Table 1.
The sentence to be parsed is He sighed .
The agent can take one of three types of actions.
The ﬁrst type of action is scan , which ‘reads’ the next word of the sentence.
The words are ‘read’ one at a time [Table] and the agent is not able to access the next word without a scan action.
Furthermore, the ﬁnal word at the end of the sentence is not explicitly marked as such: the agent learns that the end of the sentence has been reached only when it takes a scan action that fails, which receives a steep negative reward of − 2 .
That is, the agent is actively encouraged to predict the end of the sentence
[Table]
The ﬁrst two actions in Table 1 are scan actions, and we see that the trained agent correctly predicts the end of the sentence after the second word has been scanned and does not attempt to scan again.
The results of these actions are two leaf edges which are added to the chart, and these edges are returned to the agent by the environment as (part of) the next state.
Leaf edges consist of a span, which indicates what positions/parts of the input sentence are ‘covered’ by the leaf edge.
The ﬁrst scan adds a leaf edge spanning positions 0 to 1, that is, the ﬁrst terminal, which is the word He .
The second scan adds a leaf edge spanning positions 1 to 2, i.e., the second terminal, which is the word sighed .
Actions that contribute to the ﬁnal parse receive a small negative reward of − 0 .
25 , encouraging the agent to ﬁnish parsing as soon as possible.
The next ﬁve actions in Table 1 are predict actions.
These actions target a complete edge in the chart (leaf edges are by deﬁnition complete; we discuss completion for other edges below), and identify a production in the grammar whose right-hand side starts with the terminal or non-terminal of that completed edge.
That is, we identify productions whose left-corner is the targeted edge.
For example, the ﬁrst predict action (step 3 in Table 1), targets the leaf edge storing the word He and identiﬁes ‘PRP → He’ as a production whose left-corner is that word (PRP stands for personal pronoun).
As a result of this action, we add a new edge to the environment chart, the one listed in step 3 under state .
The span of this edge is 0:1, which is the same as the span of the edge targeted by the predict action (the edge covers only the ﬁrst word of the sentence).
The edge added in step 3 is not a leaf edge, as it builds syntactic structure on top of a previous edge.
That syntactic structure is the unary branching node PRP, whose only daughter is the word He .
This edge is a complete edge, indicated by the ﬁnal dot • after He : ‘complete’ means that the entire right-hand side of the production used to construct the edge has been recognized (has been ‘matched against’ complete edges that are already available).
The next predict action (in step 4 ) targets the edge ‘[0:1] PRP → He • ’, which has just been added to the environment chart / state.
This edge is a complete edge, and is also the left-corner of the production ‘NP-SBJ → PRP;’ NP-SBJ stands for subject noun phrase (NP).
At step 4 , we build more syntactic structure on top of the PRP non-terminal: this syntactic structure is encoded by the new edge ‘[0:1] NP-SBJ → PRP • ’, which is a complete edge spanning position 0:1.
That is, by end of the predict actions in steps 3 and 4 , we have implicitly built the leftmost partial tree in Figure 2.
The predict action in step 5 targets the complete NP-SBJ edge we just added, which is identiﬁed as the left corner of the ‘S → NP-SBJ VP’ production.
The resulting edge ‘[0:1] S → NP-SBJ • VP ’, which is part of the environment state after the action in step 5 , is the ﬁrst incomplete edge in this episode: the dot • precedes the VP, indicating that only the NP-SBJ has been recognized.
Note also that the span of this incomplete edge is still 0:1, as the span always indicates the part of the input {+[Figure] PRP He step 
+}
==
==
⇒ S • VP NP-SBJ PRP He steps 2 , {+6 , 
+}
==
==
==
⇒
VP
VBD sighed step 8 =
=== ⇒
S
VP
VBD sighed
NP-SBJ PRP He The predict actions in steps 6 and 7 build syntactic structure on top of the word sighed (VBD stands for verb in past form, and VP for verb phrase).
The resulting tree is the third one in Figure 2.
The ﬁnal action in the episode is a complete action.
This is the third and ﬁnal type of action that an agent can take.
Complete actions target two edges in the current chart, i.e., in the current environment state.
The ﬁrst edge is incomplete; speciﬁcally, this is the edge ‘[0:1] S → NP-SBJ • VP’ we built in step 5 .
The second edge has to be complete, and its left-hand side non-terminal has to be the very same as the leftmost incomplete non-terminal on the right-hand side of the incomplete edge.
This ﬁnal action completes the parse, building the complete ﬁnal tree in Figure 2.
The reward in this step is a substantial positive reward of 5 .
In sum, agents can take one of three types of actions: scan, predict and complete.
Predict actions target an edge in the current chart that can be the left corner of a grammar production.
Complete actions target two edges in the current chart, an incomplete edge and a complete edge; the complete edge can be used to bring the incomplete edge one step closer to completion.
States returned by the environment consist of the current chart (list of edges), which encode the partially built syntactic structures in a very compact manner.
Charts efﬁciently encode multiple complete parses if the sentence to be parsed is syntactically ambiguous.
Consider, for example, the typical prepositional phrase (PP) attachment example I saw the astronomer with a telescope , where the PP with a telescope can be attached to the noun astronomer (the astronomer has a telescope) or to the verb saw (seeing by means of a telescope).
The parts of the syntactic structure that these two readings share are encoded only once in the chart.
In addition, charts can very gracefully support backtracking of the kind we need when resolving local syntactic ambiguities like garden-path sentences.
2.2 Environment Setup
For every episode, the agent is tasked with parsing a single sentence.
In all our experiments, the sentences come from the parsed Brown Corpus part of the Penn Treebank-3 Corpus [Marcus et al., 1999].
The speciﬁc set of parse trees we load into our parsing environment determine the difﬁculty of the parsing tasks an RL agent will face.
This enables us to create a wide variety of parsing environments that can be ﬁnely tuned in terms of difﬁculty.
The level of difﬁculty is determined by the kind of tree structures we allow in the input set of trees.
One way to decrease the level of difﬁculty is to restrict the kind of syntactic structures we allow; for example, removing trees that contain adjuncts decreases the level of difﬁculty.
Another way is to limit the height of the trees, where tree height is deﬁned as the length of the longest path in the tree starting at the root.
For example, our experiments were run ﬁrst with height-4 trees only, which is the smallest height with a reasonable number of trees (more than 150), after which we investigated height-5 trees.
We have created a variety of tree sets along these lines.
Height-6 trees, for example, even without adjunct structures, already raised the level of difﬁculty to a point that exceeded our computational resources (3 separate
GPUs, the best of which was a Titan RTX, not accessible around the clock).
Once a subset of trees is identiﬁed (by calibrating it for tree height, types of syntactic structures, and sometimes maximum sentence length) and loaded into the parsing environment, it is split into train, validation and test sets according to percentages provided by the user.
The train-validation-test split is determined by a random seed that can be set by the user for reproducibility.
Given our limited computational resources and the pilot benchmarking nature of this work, we couldn’t do a systematic hyperparameter search, so we report results based only on train-test 90% - 10% splits, without a separate validation set for hyperparameter tuning.
We set the hyperparameters to values that seemed reasonable (often defaults), provided in the next section.
After the set of trees is loaded, and before the train-test split is determined, the environment creates a context-free grammar (CFG) based on the productions implicit in all the loaded trees.
The environment uses this CFG to generate the reward structure for any given episode.
Every training episode consists of the agent learning to parse one sentence from the train set.
3 What we essentially do is create a ‘maze’ based on each individual sentence, and train agents on these ‘mazes.’
For every train sentence, the environment uses the CFG to left-corner chart-parse the sentence and identify all its possible parses, as well as all the complete and incomplete parse edges that contribute to these parses.
These parse edges enable us to generate the reward structure for every episode: if the agent takes an action resulting in an edge that contributes to one of the possible parses, the agent receives a small negative penalty, which was − 0 .
25 in our experiments (but this, and all other rewards, can be set by the user).
If the agent selects a parsing action that is licensed by the current state (consisting of the current chart and whether there still are unscanned words), but does not contribute to one of the possible parses, the agent receives a larger negative penalty of − 0 .
Actions are licensed if they create a valid edge that can be added to the current chart (‘valid’ based on the current chart and the background CFG).
If the agent selects a parsing action that is not licensed by the current state, for example, it tries to add an edge that was already added to the chart, or tries to scan a word when there are no more words to be scanned, the agent receives a steep negative penalty of − 2 .
Finally, when the agent selects an action that adds the ﬁnal edge needed to complete a full parse of the sentence, it receives a positive reward of 5 and the episode terminates.
An optimal policy takes the minimum number of actions necessary to construct a complete parse of the input sentence.
Thus, when agents build syntactic structures that are not validated by the PTB-based CFG (basically, when they go down ‘dead-ends’ in the ‘maze’), they are negatively penalized by the environment.
Since charts are able to compactly represent multiple parse trees and to gracefully support backtracking, it’s always possible for an agent to eventually arrive at the correct parse by exhaustively executing all parse actions offered by the environment (basically, by exhaustively exploring the ‘maze’).
However, this results in a signiﬁcantly lower total reward than taking the shortest, most cognitively realistic path to the simplest correct parse (as seen in Sec.
4 Results).
Agents are trained to take the minimum steps possible because this heuristic is part of what the human parser does (Hale 2011 a.o. and references therein).
It is precisely this minimum-cost feature of the human parser that leads it down garden paths (recall the example The horse raced past the barn fell ).
Hale [2011] provides suggestive evidence that a distance metric (i.e. an estimate of steps necessary to complete a parse) inferred based on PTB counts can be used to guide parsing in a way that captures a variety of garden-path phenomena.
The present work is a ﬁrst step towards using RL methods to learn this metric from experience (building on Hale 2014), and in the process, hopefully provide evidence for the cognitive realism of this minimal-effort / minimal-cost hypothesis about the human parser.
In our experiments, we decreased the difﬁculty of learning and provided the agent with the list of all possible predict and complete actions licensed by the environment in any given state.
This simpliﬁes the learning problem by reducing action generation to action selection from a provided set of choices.
The simpliﬁcation can be easily removed, which would force the agent to generate actions.
Experiments 3.1 Agent Architectures
We study the performance of 15 RL algorithms / agents in the height-4 and height-5 parsing environments: ( i ) a tabular Q learning agent (Watkins 1989, Watkins and Dayan 1992), ( ii ) 7 DQN agents (Mnih and al 2015, Sutton and Barto 2018, Ch. 11 and references therein), and ( iii ) 7 Actor-Critic (AC) agents (Sutton and Barto 2018, Ch. 13 and references therein).
The 7 DQN agents differ with respect to their architecture, and so do the AC agents.
Six of them are recurrent: Elman/standard RNN (Elman 1990), GRU (Cho et al. 2014), LSTM (Hochreiter and Schmidhuber 1997), and bidirectional [Figure] versions of these three (Schuster and Paliwal 1997, Graves and Schmidhuber 2005).
We also study a self-attention (Vaswani et al. 2017) agent.
In all our experiments, the learning rate α for all the agents was set to 10 − 3 , and the discount factor γ to 0 . 9 .
The tabular Q agent represents the Q function as a look-up table that stores the estimated values of all possible state-action pairs.
The state s t at time step t consist of all the edges in the current chart, plus a Boolean indicating whether all the words have been scanned, i.e., whether we reached the end of the sentence.
The action a t selected by the agent in state s t can be a scan, predict or complete action, as discussed in the previous section.
Before learning begins, all the entries in the Q table are set to an arbitrary value ( 0 ), and they are updated in an entry-wise fashion at each time step t : the value of the pair ( s t , a t ) is updated based on the reward signal r t +1 and the new state s t +1 that the agent receives from the environment after taking action a t .
The new state s t +1 contains the chart from the previous state s t plus the new edge (if any) added by action a t .
DQN agents approximate the Q function with an artiﬁcial neural network (ANN).
Their basic architecture is provided in Fig.
The edges in the current chart are numericalized (we return to this in a moment) and the resulting tensors are the input to a recurrent or self-attention ANN, the output of which is a chart tensor that ‘summarizes’ the current chart.
We have a variety of choices for how to compute the chart tensor, but we only explore the simplest choices here: for RNNs, we take the chart tensor to be the ﬁnal hidden state (or the two ﬁnal hidden states for bidirectional RNNs); for self-attention, we mean-pool the attention outputs.
This chart tensor is then concatenated with the numericalized action we’re evaluating, and the resulting tensor is the input to a multilayer perceptron (MLP) with a single hidden layer and a ReLU nonlinearity.
The output of the MLP is the predicted
Q value for the current state (chart) and action.
We use an (cid:15) -greedy policy, with (cid:15) annealed from a starting value of 1 to a minimum value of 0 . 01 .
All DQN agents were trained using one-step semi-gradient TD (a.k.a. semi-gradient TD(0); Sutton and Barto 2018, Chapters 9-11), with a squared
TD-error loss function and the Adam optimizer (Kingma and Ba 2015).
The AC agents use the same architecture as the DQN agents for their policy-approximation component: the single-value output is now the estimated logit for the action we’re evaluating.
These logits are soft-maxed together to yield a probability distribution over these actions.
To this main policyapproximating branch, we add a separate state-value estimation MLP head with a single hidden layer that takes the chart tensor as input and outputs the estimated value for the current state (chart).
The state-value head is trained using a squared TD-error loss (we backpropagate these gradients only through the state-value MLP head).
The main policy-network branch is updated using a one-step version of REINFORCE (Williams 1992; see also Sutton and Barto 2018, Ch. 13 and references therein).
All the recurrent and self-attention networks had a state/query/key/value size of 256 , which was also the hidden-layer size of all MLPs.
There are many possible choices for edge-to-tensor numericalization, including trainable embeddings.
In our experiments, we used a simple, deterministic algorithm.
As discussed in the previous section, an edge is basically a CFG production with three associated integers: two integers are used to indicate the span of the edge (the part of the sentential input that is ‘covered’ by the edge), and the third one is used to indicate the dot position, i.e., how much of the right-hand side of the production has been completed.
To numericalize an edge, we need to decide how to numericalize the non-terminals and terminals in the production part, and these three integers.
For the integers, we used a one-hot encoding with a dimensionality equal to the maximum length of the right-hand side of a production in our CFG.
For the non-terminals, we also used a one-hot encoding with the dimensionality provided by the number of non-terminals in our CFG.
For terminals, i.e. words, we used GloVe embeddings (Pennington et al. 2014) and reduced their dimensionality to the dimensionality of the non-terminal one-hot encodings via a principal component analysis (PCA) model.
For out-of-vocabulary (UNK) words without GloVe embeddings, we used the mean GloVE embedding.
Actions were numericalized using the same algorithm.
For complete actions, we concatenated the numericalizations of the two edges in the complete pair.
For predict actions, we numericalized the edge and then numericalized the production as if it was an edge, with null tensors for the three spurious integers.
For scan actions, we concatenated two null edge tensors of the appropriate dimensionality.
3.2 Environment Speciﬁcs
We select the two simplest parsing environments we can create based on the Brown part of the Penn
Treebank corpus: an environment based on a subset of the trees of height 4, and an environment based on a subset of the trees of height 5.
To make the difﬁculty as low as possible for the RL agents, the trees are selected so that they always begin with a subject NP.
The height-4 environment hadtrees, with a random 90% − 10% train-test split that varied across 4 experimental runs; all the results reported in this section average across these runs, hence, across multiple random train-test splits.
All splits had 160 train sentences and 18 test sentences.
We did not have a separate validation set: given our limited computational resources, we didn’t do any hyperparameter tuning (except for a handful of very limited comparisons).
The height-5 environment had 776 trees, and the 90% − 10% split had 698 train sentences and 78 test sentences.
Although these two environments are the simplest in terms of difﬁculty, the jump in difﬁculty from height 4 to height 5 is signiﬁcant (and it only gets more substantial when moving to height-6 trees etc.).
The maximum sentence length was 7 for height-4 trees, and 12 for height-5 trees.
The CFG induced by the height-4 trees had only 269 productions with a maximum right-hand side length of 4 , while the CFG induced by the height-5 trees had 1761 productions with a maximum right-hand side length of 7 .
Because of this, the average number of parses for a sentence from the height-4 corpus (according to the induced CFG) was 1 . 09 , while the average number of parses for a sentence from the height-5 corpus was 18 .
The number of parses increases very quickly as tree height goes up.
For example, height-6 based CFGs associate more than 35 million parses with relatively short sentences like The pale blob of the woman disappeared .
Yet another way to see the jump in difﬁculty from height 4 to height 5 is to compare the average number of valid predict and complete actions per step: for height 4, there are on average 4 .
5 predict actions and 1 .
11 complete actions per step, while for height 5, there are on average 97 .
39 predict actions and 1 .
82 complete actions per step.
The performance of the agents in the height-4 and height-5 parsing environments are provided in
Tables 2 and 3.
The results in these tables (both means and standard errors) are averaged overindependent runs.
The height-5 results for some of the AC agents were computed on less thanruns because of their substantially higher computational-resource demands.
The number of steps was also limited for some of the AC agents on height 5.
Based on a very small set of comparisons, step-limiting did not hurt, and sometimes even slightly helped, the AC agents’ training.
On height 4, the agents were trained for 15 , 000 episodes, while on height 5, they were trained for 5 , 000 episodes.
There are fewer episodes for height-5 because the episodes are much longer than the height-4 ones; 5 , 000 height-5 episodes are roughly equivalent to 15 , 000 height-4 episodes in that the (cid:15) -annealing schedule has a similar proﬁle relative to a full training run.
3.3 Evaluation
To better evaluate the agents, we estimated a ﬂoor and a ceiling for their performance in these environments.
The ﬂoor is provided by agents randomly choosing an action in any state from the set of actions that are valid in that state.
For height 4, the random agent achieves an average total reward per episode of − 4 .
87 , obtained in 18 .
14 steps per episode (on average).
For height 5, the random agent achieves an average total reward of − 96 in 156 .
21 steps per episode.
We estimate the performance ceiling for any given sentence by looking at all the possible parses of the sentence based on the environment CFG, and all the complete and incomplete edges that contribute to any of these parses.
With the edges and parses in hand, we can compute the average reward per parse by multiplying the number of edges by − 0 .
25 (which is the cost of any parsing action that contributes to a successful parse), adding 5 for all ﬁnal edges (this is the ﬁnal reward for completing the parse), and dividing by the total number of possible parses.
We can compute the average minimum number of steps in the same way (we divide the number of edges by the number of parses).
These estimates are fairly accurate for low-ambiguity sentences like the ones in the height-4 environment, but they tend to be overly optimistic for higher-ambiguity sentences like the ones in the height-5 environment.
To see this, take the typical PP-attachment ambiguity example I saw the astronomer with a telescope .
The average number of minimum steps to a successful parse when there are 2 possible parses is likely higher than taking the number of edges in a chart encoding both parses and dividing by 2 : most of the edges are shared by the two parses, so they always have to be added even if we go for only one parse.
This being said, the estimated average maximum reward for the height-4 environment is 2 . 99 , and the average minimum number of steps is 10 . 05 , which are very likely close to the true values because of the low ambiguity of height-4 sentences.
For height 5, the estimated average maximum reward is 4 . 13 , and the estimated minimum number of steps is 5 . 47 .
These height-5 estimates are likely pretty far from the true values, for which the average number of minimum steps seems closer to 25 or 30 , which would put the average max reward for height 5 closer to − 2 .
Results
With these performance ranges in mind, we can turn to a discussion of the results in Tables 2 and 3.
We see that, overall, DQN agents outperform AC agents, with the performance of tabular agents being the poorest.
Tabular agents perform at random baseline level in both the height-4 and the heightenvironments: they effectively memorize the training data, so they only very slightly generalize from train to test in height 4 (many trees have similar subparts, so some generalization is possible), but completely fail to generalize in the more difﬁcult height-5 environment.
[Table]
On height 4, the DQN agent with an Elman (simple) RNN is the best on the test sentences; see the two boldfaced numbers in the left half of Table 2.
This is likely because the other agents end up overﬁtting the training data.
The performance of the DQN RNN agent is very close to ceiling performance, i.e., to the estimated max reward and min steps per episode for height 4, indicating that, for all intents and purposes, we have solved the height-4 environment.
The DQN GRU agent is also a solid performer in height 4, trailing behind DQN RNN only slightly.
However, the extra capacity in the more complex agents is helpful in the height-5 environment, where the bi-directional GRU performs the best; see the two boldfaced numbers in the right half of Table 2.
Once again, the DQN GRU is a solid performer, trailing only slightly behind its bidirectional cousin on height 5.
However, even the performance of the best agent (DQN Bi-GRU) is not at ceiling on height 5.
As mentioned before, we have not done a hyperparameter search because of our limited computational resources, but the sub-ceiling performance of the best agent does not seem to be due to model capacity (only): we compared a DQN RNN agent with double the hidden state in the recurrent component ( 512 ) and double the hidden-layer size in the MLP component ( 512 again), and its performance on height 5 was not distinguishable from the DQN RNN agent in Table 2.
We see that these cognitively realistic parsing environments provide a substantial challenge for current
RL algorithms—recall that the height-4 and height-5 environments provide the simplest levels of difﬁculty, which very abruptly escalates for height 6 and above.
A minimal increase in difﬁculty would be to use all the height-4 and height-5 trees, not only the trees that start with an NP subject.
The AC agents performed more poorly than the DQN agents, which might be due to the fact that DQN is probably more sample efﬁcient than AC.
This is particularly interesting given that AC architectures have been argued to be cognitively realistic (Botvinick et al. 2009).
The self-attention AC agent in particular performed surprisingly poorly.
Self-attention DQN is a pretty solid, middle-of-the-pack performer on both height 4 and height 5.
The training of the self-attention AC agent, however, was unstable, which resulted in poor performance on height 4 and a complete lack of convergence in the more difﬁcult height-5 environment.
We are still diagnosing this issue.
[Table]
[Figure]
N/A
N/A N/A N/A The plots in Fig.
4 show the number of steps per episode (lower is better) for four agents, the tabular one and the three simplest recurrent DQNs.
We see that the tabular agents seem to learn, but very slowly, while the DQN agents learn much faster.
However, the DQN agents top out at a sub-ceiling level of performance on height 5, indicating that this environment remains unsolved.
We leave an extensive hyperparameter search for a future occasion, as well as the explorations of alternative architectures, both for ANNs and for RL agents.
Hierarchical reinforcement learning agents in particular might bring signiﬁcant improvements, given that parsing is an ideal task for learning and deploying ‘macro’ actions (options; Sutton et al.
1998 among many others), and given that hierarchical (AC) agents have been argued to be cognitively realistic in Botvinick et al.
Conclusion and Future Work
This research program has two overarching goals.
On the cognitive side, the goal is to investigate whether experience is sufﬁcient for control in parsing, as hypothesized in Hale [2014, Ch. 6].
For example, we want to investigate whether RL agents trained on natural language corpora like the Penn
Treebank exhibit the same proﬁle in parsing local syntactic ambiguities (e.g. garden path sentences like The horse raced past the barn fell ) as human participants in psycholinguistic experiments.
A wealth of other phenomena investigated in the psycholinguistic literature could also be investigated through this lens.
On the (Deep) RL side, parsing is an ideal task to study hierarchical option induction (Bacon et al. 2017) among others.
For example, seeing a proper name or a pronoun or a determiner at the start of a sentence could trigger an option that builds in one step an entire subtree with an S root and an NP-SBJ and VP daughters (e.g. the second tree in Fig. 2).
Such options might be crucial to solve more complex environments (height 6 and above).
Investigating curriculum learning (from height 4 to height 5, for example) is another promising direction to explore.
In summary, we have seen how deep RL agents can learn from experience to left-corner parse in an environment that maintains a chart and context-free grammar.
While parsing difﬁculty increases very quickly with respect to syntactic tree height, we note the clear performance beneﬁts of certain neural architectures, which opens up the possibility of testing the cognitive realism of various learning algorithms in this family of environments.
The application of deep RL to a cognitive faculty as fundamental as syntactic parsing could enable future work that draws inspiration from empirical psycholinguistic phenomena to craft learning algorithms that better match human performance on a variety of tasks, linguistic and potentially non-linguistic.