Transition-based Parsing with Stack-Transformers Anonymous
Abstract
Modeling the parser state in transition-based parsing is key to good performance.
Recurrent
Neural Networks considerably improved the performance of transition-based systems by global state modeling, e.g. stack-LSTM parser, or local state modeling of contextualized features, e.g. Bi-LSTM parser.
Given the success of Transformer architectures in recent parsing systems, this work explores modiﬁcations of the sequence-to-sequence Transformer architecture to model either global or local parser states.
We show that modiﬁcations of the cross attention mechanism of the Transformer considerably strengthen performance both on dependency and Abstract Meaning Representation (AMR) parsing tasks.
Introduction
Transition-based Parsing transforms the task of predicting a graph from a sentence into predicting an action sequence of a state machine that produces the graph (Nivre, 2003, 2004; Kubler et al., 2009;
Henderson et al., 2013).
These parsers are attractive for their linear inference time and interpretability, however, the performance hinges on effective modeling of parser state at every decision step.
Parser states typically comprise two memories, a buffer and a stack, from which tokens can be pushed or popped (Kubler et al., 2009).
Traditionally, parser states were modeled using hand selected local features pertaining only to the words on the top of the stack or buffer (Nivre et al., 2007; Zhang and Nivre, 2011, inter-alia).
With the widespread use of neural networks, global models of the parser state such as the stack-LSTM (Dyer et al., 2015) allowed encoding the entire buffer and stack.
It was later shown that local features of the stack and buffer extracted from contextual word representations, such as Bi-LSTMs, could outperform global modeling (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016).
With the rise of the Transformer model (Vaswani et al., 2017), various approaches have been proposed that leverage this architecture for parsing (Kondratyuk, 2019; Kulmizev et al., 2019; Mrini et al., 2019; Ahmad et al., 2019; Cai and Lam, 2020).
In this work we revisit the local versus global paradigms of state modeling in the context of sequence-to-sequence Transformers applied to action prediction for transition-based parsing.
We propose a modiﬁcation of the cross-attention mechanism of the Transformer to provide global state modeling of the stack and buffer akin to stackLSTMs.
We ablate the different components and introduce simple state modeling alternatives.
Results show that local and global state modeling of the parser state yields more than 2 percentage points absolute improvement over a strong Transformer baseline, both for dependency and Abstract
Meaning Representation (AMR) parsing.
We also introduce improvements to the AMR transitionbased oracle (Ballesteros and Al-Onaizan, 2017a) yielding best results for a transition-based system and second overall.
Modeling the parser state
Given a sentence s and its graph g , transition-based parsing utilizes an oracle, generally rule-based, to derive a set of actions a = O ( s, g ) .
These actions applied to a state machine generate the graph from the sentence g = M ( a, s ) .
The parser thus needs to learn p ( a t | a <t , s 1: | s | ) at each step.
The state of the parser at step t is deﬁned implicitly by ( a <t , s 1: | s | ) .
This translates to an explicit state at step t where the stack contains some token vectors of the sentence about to be processed, sometimes along with new composed vector representations, and the buffer contains the remainder of tokens in the sentence, see e.g. middle of Fig.
The transition-based formalism relies heavily on the explicit representation of the state i.e. buffer and stack conﬁgurations.
Prior to the widespread use of neural Networks, local features considering only the top of the stack and buffer already achieved good performances (Nivre et al., 2007;
Zhang and Nivre, 2011, inter-alia).
The Use of Recurrent Neural Networks made possible the modeling of the global state of the parser by encoding stack, buffer and action history with LSTMs.
StackLSTMs (Dyer et al., 2015) enriched LSTMs with a pointer that can be moved backwards to allow encoding of the stack under a pop (REDUCE) operation, see top part of Fig.
In addition to this, stack-LSTMs used the transition-based formalism to build vector representations of partial tree elements by non-linear composition making full use of the information in ( a <t , s 1: | s | ) .
Another well known LSTM model is the BiLSTM feature parser (Kiperwasser and Goldberg, 2016).
In this case, a contextual representation of the sentence is ﬁrst built with a Bi-LSTM h 1: | s | = BiLSTM( s 1: | s | ) .
At each Time step t the stack conﬁguration determined by a <t is used to select the top of the stack and buffer elements from h 1: | s | .
Although the features utilize local information of the buffer and stack, the use of a strong contextual representation proved to be sufﬁcient and this remains one of the most widely used forms of parsing today.
State modeling in the Transformer
Unlike RNNs, Transformers are not recurrent and they model time-structure and precedence by concatenating positional embeddings with token embeddings.
Two or more Transformer layers are applied on top of each other to obtain a non-linear contextual representation of every token.
Furthermore, each layer has multiple attention heads that capture different structural patterns.
We take advantage of these characteristics by dedicating two heads of the Transformer to encode stack and buffer information.
At time-step t words in s 1: | s | are either in the stack, buffer or removed.
For the two dedicated attention heads, we add a {−∞ , 0 } mask m ti to the decoder cross-attention mechanism, allowing only to attend the words in the encoder that are present in either the stack or the buffer.
We also add an additional position embedding p ti to the output of the encoder h
i to indicate token position in the stack and the buffer, see Fig.
This leads to the following modiﬁed matching function between the output of previous decoder layer b t (query) and hidden representation of word
h
i (key)
[Equation]
from which the attention would be computed as α ti = softmax( e t ) i .
Both mask and positions change for each word and time-step as the parser state changes, but they imply little computation overhead and can be precomputed for training.
Fig.
1 draws a comparison between a stackLSTM and a sequence-to-sequence Transformer equipped with this modiﬁed attention, henceforth stack-Transformer.
Both provide global state encoding of stack and buffer, modeling the context and their respective sequence order.
The stackLSTM composes LSTM cells while the stackTransformer just sums content and position vectors as in the usual Transformer encoding.
One interesting ablation of this model involves only using m ti on one head and constraining it to point to the top two words in the stack, making it similar to the local features over contextualized encodings of (Kiperwasser and Goldberg, 2016).
The attention mechanism has been used in the past to encode stack and buffer information on
RNN models.
Liu and Zhang (2017) propose the use of a simple boundary to separate stack and buffer attentions, precluding the SWAP operations needed for AMR parsing and non-projective parsing in general.
Zhang et al.
(2017) mask out reduced words and add a bias to the attention weights for words in the stack.
The approach proposed here specializes different heads for different views.
The model is free of other modiﬁcations such as the constrained decoding in (Zhang et al., 2017).
It is common practice for transition-based systems to use a SHIFT(tag) action, to add a tag prediction task e.g. POS (Bohnet and Nivre, 2012).
We experimented with labeling SHIFT actions with the word being shifted.
This provides a coarse view of the parser state when looking at the action history and exploits the Transformers ability to build representations involving far apart words.
This simple enhancement, notably improved the Transformers performance but only when limiting this to a small set of most frequent words.
Experiments and Results
The proposed models were tested on dependency parsing, the English Penn-Treebank (Marcus et al., 1993), and semantic parsing, LDC2017T10 (AMR 2.0).
The dependency parsing setup followed (Dyer et al., 2015), in the setting with no POS tags.
This has only SHIFT, LEFT-ARC(label), and RIGHTARC(label) base action with a total of 82 different actions for a train set of around 40 K sentences.
As it is customary results were measured in terms of (Un)labeled Attachment Scores (UAS/LAS).
AMR is a complex semantic parsing task, encompassing named entity recognition, word sense disambiguation and co-reference among other subtasks.
Nodes in an AMR are not explicitly linked to the surface symbols and are often lemmatizedforms of the words in the sentence with a sense tag for predicates.
The setup followed (Ballesteros and Al-Onaizan, 2017a), which introduced new actions to segment text, derive nodes or sub-graphs in addition to the dependency parsing actions.
This amounted to a total of around 9 K different actions for close to 37 K sentences.
To facilitate training, node labels are limited to those seen during training for the words on the top of the stack.
Results were measured in Smatch (Cai and Knight, 2013).
Ballesteros and Al-Onaizan (2017b), and the follow-up work (Naseem et al., 2019), predict ﬁrst that a node has to be created and then predict the label, or call a lemmatizer if no label is found.
For simplicity, we directly predicted the label and added COPY actions to construct node names from lemmas.
A change that made a large difference was force-aligning unaligned nodes to neighbouring words.
This increased oracle Smatch from 93 .to 97 .
9 and notably improved model performance.
We therefore provide results for the Naseem et al.
(2019)’ oracle for comparison.
All models were implemented on the fairseq toolkit.
Only minor modiﬁcations over the MT model hyper-parameters (Ott et al., 2018) 2 were needed.
Label smoothing was reduced to 0 .
All models used 6 layers of encoding and decoding with size 256 and 4 attention heads, except the normal Transformers in AMR, which performed better on a 3 / 8 layer conﬁguration instead of 6 / 6 .
We used RoBERTa-base (Liu et al., 2019) embeddings without ﬁne-tuning as input, averaging wordpieces.
Weight averaging of the best 3 models (JunczysDowmunt et al., 2016) and beam 10 were used in all models 3 .
Models were trained for a ﬁxed number of epochs ranging from 80 to 120 , selecting the best model on validation by either LAS or
Smatch 4 .
No other hyper-parameters were changed across models or tasks.
Training took at most 6 h on a Nvidia Tesla v100 GPU.
The labeled SHIFT strategy used the 100 most frequent words.
4. Ablation on Global versus Local States
Table 1 compares the standard Transformer and different ways of modeling the parser state including ablations of the global stack-Transformer on the development sets.
The Transformer architecture beneﬁts greatly from the simple labeled SHIFT strategy (a, b).
All ablations of the stack-Transformer perform above this strategy with fairly close results (b-f).
The only exception is the use of positions
[Table]
[Table]
Transformer + (
[Table]
[Table]
Stack
[Table]
Transformer
[Table]
which seems detrimental in both tasks (g).
The largest difference across stack-Transformer variants is between the approach only considering the top of the stack (c) and the full modeling of the global state (f), giving some indication that global state modeling can be of use, even in the presence of strong feature encoders.
Local modeling and the labeled SHIFT strategy seem however enough to attain an equivalent performance (h)
[Table] Tables 2 and 3 compare with prior works on the respective corpora test sets.
Due to their strong impact, pre-trained embeddings are indicated as
XL-net-large X
(Yang et al., 2019), BERT base β and large B (Devlin et al., 2019).
We also indicate
Graph Recategorization (G.R.) that preprocesses named entity recognition among other properties (Lyu and Titov, 2018; Zhang et al., 2019).
Finally a ∗ indicates the (Naseem et al., 2019) oracle.
Overall, the proposed stack-Transformer provides competitive scores compared to existing works, particularly if we take into account that the modiﬁcations over the normal Transformer are relatively straightforward.
This allows using existing tools for fast training and decoding and facilitates incorporation of new developments for these
[Table]
[Table]
models.
The improvements against a normal Transformer are also clear but far larger for AMR, which is likely due to the higher complexity of the task.
The gap against stack-LSTMs is large even when using the same oracle and comparing with its Reinforcement Learning ﬁne-tuning.
It is also worth noting that results stand out furthermore when compared to transition-based systems only.
Unlike recent AMR parsers using Graph Re-categorization, the approach presented does segmentation of entities on the ﬂy, which can be expected to provide better generalization to new domains 5 .
Conclusions
We have explored modiﬁcations of sequence-tosequence Transformers to encode the parser state for transition-based parsing akin to Stack-LSTMs.
These modiﬁcation consistently provide improvements against a normal Transformer for syntactic and particularly semantic parsing (AMR).
Results obtained are also competitive recent parsing approaches, particularly transition-based ones.