AMR Parsing with Action-Pointer Transformer (2021)

Abstract: Abstract Meaning Representation parsing is a sentence-to-graph prediction task where target nodes are not explicitly aligned to sentence tokens. However, since graph nodes are semantically based on one or more sentence tokens, implicit alignments can be derived. Transition-based parsers operate over the sentence from left to right, capturing this inductive bias via alignments at the cost of limited expressiveness. In this work, we propose a transition-based system that combines hard-attention over sentences with a target-side action pointer mechanism to decouple source tokens from node representations and address alignments. We model the transitions as well as the pointer mechanism through straightforward modifications within a single Transformer architecture. Parser state and graph structure information are efficiently encoded using attention heads. We show that our action-pointer approach leads to increased expressiveness and attains large gains (+1.6 points) against the best transition-based AMR parser in very similar conditions. While using no graph re-categorization, our single model yields the second best Smatch score on AMR 2.0 (81.8), which is further improved to 83.4 with silver data and ensemble decoding.

## introduction abstract meaning representation (amr) @xcite is a sentence level semantic formalism encoding who does what to whom in the form of a rooted directed acyclic graph. nodes represent concepts such as entities or predicates which are not explicitly aligned to words, and edges represent relations such as subject/object (see figure 1 ). amr parsing, the task of generating the graph from a sentence, is nowadays tackled with sequence to sequence models parametrized with neural networks. there are two broad categories of methods that are highly effective in recent years. transition-based approaches predict a sequence of actions given the sentence. these actions generate the graph while processing tokens left-to-right through the sentence and store intermediate representations in memories such as stack and buffer @xcite @xcite @xcite @xcite . general graph-based approaches, on the other hand, directly predict nodes and edges in sequential order from graph traversals such as breath first search or depth first search @xcite @xcite . while not modeling the local semantic correspondence between graph nodes and source tokens, the approaches achieve strong results without restrictions of transition-based approaches, but often require graph re-categorization, a form of graph normalization, for optimal performance. the strong left-to-right constraint of transitionbased parsers provides a form of inductive bias that fits amr characteristics. amr nodes are very often normalized versions of sentence tokens and locality between words and nodes is frequently preserved. the fact that transition-based systems for amr have alignments as the core of their explanatory model also guarantees that they produce reliable alignments at decoding time, which are useful for applications utilizing amr parses. despite these advantages, transition-based systems still suffer in situations when multiple nodes are best explained as aligned to one sentence token or none. furthermore, long distance edges in amr, e.g. re-entrancies, require excessive use of swap or i offer a solution to the problem . tokens: actions: graph: copy copy la(1:arg0) shift reduce pred(thing) ra(2:arg1) pred(solve-01) ra(6:arg2-of) i offer thing solve-01 arg0 arg1 arg2-of 1 2 3 4 5 6 7 8 9 figure 2: source tokens, target actions and amr graph for the sentence i offer a solution to the problem (partially parsed). the black arrow marks the current token cursor position. the circles contain the action indices (used as ids), black circles indicate node creating actions. only these actions are available for edge attachments. notice that the edge actions (at steps 3, 7 and 9) explicitly refer to past nodes using the id of the action that created the node. the other participant of the edge action is implicitly assumed to be the most recently created graph node. in this work, we propose the action-pointer transition (apt) system which combines the advantages of both the transition-based approaches and more general graph-generation approaches. we focus on predicting an action sequence that can build the graph from a source sentence. the core idea is to put the target action sequence to a dual use -as a mechanism for graph generation as well as the representation of the graph itself. inspired by recent progress in pointer-based parsers @xcite , we replace the stack and buffer by a cursor that moves from left to right and introduce a pointer network @xcite as mechanism for edge creation. unlike previous works, we use the pointer mechanism on the target side, pointing to past node generation actions to create edges. this eliminates the node generation and attachment restrictions of previous transitionbased parsers. it is also more natural for graph generation, essentially resembling the generation process in the graph-based approaches, but keeping the graph and source aligned. we model both the action generation and the pointer prediction with a single transformer model @xcite . we relate target node and source token representations through masking of cross-attention mechanism, similar to astudillo et al. (2020) but simply with monotonic actionsource alignment driven by cursor positions, rather than stack and buffer contents. finally we also embed the amr graph structural information in the target decoder by re-purposing edge-creating steps, and propose a novel step-wise incremental graph message passing method @xcite en-abled by the decoder self-attention mechanism. experiments on amr 1.0, amr 2.0, and amr 3.0 benchmark datasets show the effectiveness of our apt system. we outperform the best transitionbased systems while using sensibly shorter action sequences, and achieve better performance than all previous approaches with similar size of training parameters. ## amr generation with action-pointer figure 2 shows a partially parsed example of a source sentence, a transition action sequence and the amr graph for the proposed transitions. given a source sentence @xmath0, x 2 , . . . , x s , our transition system works by scanning the sentence from left to right using a cursor c @xmath1, 2, . . . , s}. cursor movement is controlled by three actions: shift moves cursor one position to the right, such that c t+1 = c t + 1. reduce is a special shift indicating that no action was performed at current cursor position. merge merges tokens x ct and x ct+1 and shifts. merged tokens act as a single token under the position of the last token merged. ## training and inference our model is trained by maximizing the log likelihood of equation (1). the valid action space, action-source alignment c t , and the graph embedding mask at each step t are pre-calculated at training time. for inference, we modify the beam search algorithm to jointly search for actions and edge pointers and combine them to find the action sequence that maximizes equation (1). we also consider hard constraints in the searching process such as valid output actions and valid target pointing values at different steps to ensure an amr graph is recoverable. for the structural information that is extracted from the parsing state such as c t and graph embedding masks, we compute them on the fly at each new step of decoding based on the current results, which are then used by the model for the next step decoding. we detail our search algorithm in appendix c. ## experimental setup data and evaluation we test our approach on two widely used amr parsing benchmark datasets: amr 2.0 (ldc2017t10) and amr 1.0 (ldc2014t12). the amr graphs are all human annotated. the two datasets have 36521 and 10312 training amrs, respectively, and share 1368 development amrs and 1371 testing amrs foot_2 . we also report results on the latest amr 3.0 (ldc2020t02) dataset, which is larger in size but has not been fully explored, with 55635 training amrs and 1722 and 1898 amrs for development and testing set. wiki links are removed in the preprocessing of data, and we run a wikification approach in post-processing to recover wikipedia entries in the amr graphs as in @xcite . for evaluation, we use the smatch (f1) scores foot_3 @xcite and further the fine-grained evaluation metrics @xcite to assess the model's amr parsing performance. ## related work with the exception of astudillo et al. ( 2020 ), other works introducing stack and buffer information into sequence-to-sequence attention parsers @xcite @xcite , are based on rnns and do not attain high performances. liu and zhang (2017 ); zhang et al. (2017) tackle dependency parsing and propose modified attention mechanisms while buys and blunsom (2017) predicts semantic graphs jointly with their alignments and compares stackbased with latent and fixed alignments. compared to the stack-transformer (astudillo et al., 2020), regarding the use of pointer mechanisms for arc attachment, @xcite proposed the stack-pointer network to build partial graph representations, and fernández-gonzález and gómez-rodríguez (2020) adopted pointers along with the left-to-right scan of the sentence, greatly improv-ing the efficiency. compared with these works, we tackle a more general text-to-graph problem, where nodes are only loosely related to words, by utilizing the action-pointer mechanism. our method is also able to build up to depth m graph representations with m decoding layers. while not explicitly stated, graph-based approaches @xcite generate edges with a pointing mechanism, either with a deep biaffine classifier @xcite or with attention @xcite . they also model inductive biases indirectly through graph re-categorization, detailed in section 6.1, which requires a name entity recognition system at test time. re-categorization was proposed in @xcite , which reformulated alignments as a differentiable permutation problem, interpretable as another form of inductive bias. finally, augmenting seq-to-seq models with graph structures has been explored in various nlp areas, including machine translation @xcite ), text classification @xcite , amr to text generation @xcite , etc. most of these works model graph structure in the encoder since the complete source sentence and graph are known. we embed a dynamic graph in the transformer decoder during parsing. this is similar to broad graph generation approaches @xcite relying on graph neural networks @xcite , but our approach is much more efficient as we do not require heavy re-computation of node representations. ## conclusion we present an action-pointer mechanism that can naturally handle the generation of arbitrary graph constructs, including re-entrancies and multiple nodes per token. our structural modeling with incremental encoding of parser and graph states based on a single transformer architecture proves to be highly effective, obtaining the best results on all amr corpora among models with similar learnable parameter sizes. an interesting future exploration is on combining our system with large pre-trained models such as bart, as directly finetuning on the latter shows great potential in boosting the performance @xcite . although we focus on amr graphs in this work, our system can essentially be adopted to any task generating graphs from texts where copy mechanisms or hard-attention plays a central role.