Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing (2021)

Abstract: Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.

## introduction the task of abstract meaning representation (amr) parsing translates a natural sentence into a rooted directed acyclic graph capturing the semantics of the sentence, with nodes representing concepts and edges representing their relations @xcite . recent works utilizing pretrained encoder-decoder language models show great improvements in amr parsing results @xcite . these approaches avoid explicit modeling of the graph structure. instead, they directly predict the linearized amr graph treated as free text. while the use of pre-trained transformer encoders is widely extended in amr parsing, the use of pre-trained transformer decoders is recent and has shown to be very effective, maintaining current state-of-the-art results @xcite . these approaches however lack certain desirable properties. there are no structural guarantees of graph well-formedness, i.e. the model may predict strings that can not be decoded into valid graphs, and post-processing is required. furthermore, predicting amr linearizations ignores the implicit alignments between graph nodes and words, which provide a strong inductive bias and are useful for downstream amr applications @xcite @xcite @xcite . on the other hand, transition-based amr parsers @xcite @xcite operate over the tokens of the input sentence, generating the graph incrementally. they implicitly model graph structural constraints through transitions and yield alignments by construction, thus guaranteeing graph well-formedness. 1 however, it remains unclear whether explicit modeling of structure is still beneficial for amr parsing in the presence of powerful pre-trained language models and their strong free text generation abilities. in this work, we integrate pre-trained sequenceto-sequence (seq-to-seq) language models with the transition-based approach for amr parsing, and explore to what degree they are complementary. to fully utilize the generation power of the pre-trained language models, we propose a transition system with a small set of basic actions -a generalization of the action-pointer transition system of @xcite . we use bart @xcite as our pre-trained language model, since it has shown significant improvements in linearized amr generation @xcite . unlike previous approaches that directly fine-tune the model with linearized graphs, we modify the model structure to work with our transition system, and encode parser states in bart's attention mechanism @xcite . we also explore different vocabulary strategies for action generation. these changes convert the pre-trained bart to a transition-based parser where graph constraints and alignments are internalized. we provide a detailed comparison with topperforming amr parsers and perform ablation experiments showing that our proposed transition system and bart modifications are both necessary to achieve strong performance. although bart has great language generation capacity, it still benefits from parser state encoding with hard attention, and can efficiently learn structural output. our model establishes a new state of the art for amr 2.0 while maintaining graph well-formedness guarantees and producing built-in alignments. ## intricacies of amr parsers a frequent complaint about amr parsers is that they involve combining many different techniques and hand-crafted rules, resulting in complex pipelines that are hard to analyze and generalize poorly. this situation has notably improved in the past few years but there are still two main sources of complexity present in almost all recent parsers: graph re-categorization and subgraph actions. graph re-categorization @xcite @xcite normalizes the graph prior to learning. this includes joining certain subgraphs such as entities, dates and other constructs into single nodes, removing special types of nodes like polarity and normalizing propbank names. an example of common normalizations is displayed in figure 1 . training and decoding of models using this technique happens in this re-categorized space. re-categorized graphs are expanded to normal valid amr graphs in a post-processing stage. the type and number of subgraphs normalized vary across implementations, but most high performing approaches @xcite utilize the re-categorization described in appendix a.1 of @xcite . this version requires of an external named entity recognition (ner) system to anonymize named entities, both at train time and test time. it also makes use of look-up tables for nominalizations (e.g. english to england) and other hand-crafted rules. graph re-categorization has been criticised for its lack of generalization to new domains, such as biomedical domain or even the amr 3.0 corpus @xcite . recent top performing systems e.g. cai and lam (2020); @xcite also provide results without re-categorization, but this is shown to hurt performance notably on the amr 2.0 corpus. subgraph actions @xcite ) are used in transition-based systems and play a role similar to re-categorization. instead of normalizing and reverting, transition-based parsers apply a subgraph action that generates an entire subgraph at once. this subgraph action coincides with many of the subgraphs collapsed in re-categorization. subgraph actions bring however fewer external dependencies, since the parser learns to segment and identify subgraphs during training. they still suffer however from data sparsity since some subgraphs appear very few times. as in re-categorization, subgraph actions also make use of lookup tables for nominalization and similar constructs that hinder generalization. furthermore, they create the problem of unattachable nodes. this was addressed in @xcite by ignoring subgraphs for a set of heuristically determined cases. subgraph actions have been used in all transitionbased amr systems @xcite @xcite . aside from ner, past amr parsers have other external dependencies such as pos taggers @xcite and lemmatizers @xcite @xcite . (dotted lines) to its amr graph. arccreating actions are displayed vertically due to space constraints. words are repeated in grey to indicate the word under cursor for each action. the node boston in dotted box is created by copying the token under cursor via copy action at position 13. la(1,arg0) creates a left arc with label arg0 from the top concept like-01 to the concept person at position 1. for the concept trip-03, la(1,arg0) is a co-reference (re-entrancy) to the concept person. ## a simplified transition system in this section we propose a transition system for amr parsing designed with two objectives: maximize the use of strong pre-trained decoders such as bart, and minimize the complexity and dependencies of the transition system compared to previous approaches. similarly to @xcite , we scan the sentence from left to right and use a token cursor to point to a source token at each step. parser actions can either shift the cursor one token forward or generate any number of nodes and edges while the cursor points to the same token. the proposed set of actions is as follows: shift moves token cursor one word to the right. copy creates node where the node name is the token under the current cursor position. la(j,lbl) creates an arc with label lbl from the last generated node to the node generated at the j th transition step. ra(j,lbl) same as la but with arc direction reversed. root declares the last predicted node as the root. unlike previous transition-based approaches, we do not use a reserved action, such as pred @xcite or confirm @xcite , to predict nodes; foot_1 instead we directly use the node @xmath0. this opens the possibility of utilizing bart's target side pre-trained vocabulary. we avoid using any copy actions that involve copying from lemmatizer outputs or lookup tables. our copy action is limited to copying the lower cased word. we also eliminate the use of subgraph @xcite or entity (ballesteros and al-onaizan, 2017b) actions producing multiple connected nodes simultaneously, as well as merge action creating spans of words. in previous approaches these actions were derived from alignments or hand-crafted. they thus did not cover all possible cases limiting the scalability of the approach. finally, we discard the reduce action previously used to delete a source token. the effect can be achieved by simply using shift without performing any other action. figure 2 shows an example sentence with an action sequence and the corresponding graph. this can be compared with the handling of verbalization and named entities in figure 1 . to train a parser with a transition system, we need an action sequence for each training sentence that will produce the gold graph when executed. this action sequence then serves as the target for seq-to-seq models. a simple rule-based oracle algorithm creates these ground-truth sequences given a sentence, its amr graph and node-to-word alignments. at each step, the oracle tries the following possibilities in the order of listing and performs the first one that applies: if possible, nodes are generated by copy and otherwise @xmath1. arcs are generated with la and ra, connecting the nodes closer to the current node before the ones that are farther away. note that the arcs are created by pointing to positions in the action history, where a graph node is represented by the action that creates it, following @xcite . multiple nodes can be generated at a single source word before the cursor is moved by shift. when multiple nodes are aligned to the same token, these nodes are generated in a predetermined topological order of graph traversal, interleaved by edge creation actions. root is applied as soon as the root node is generated. the above oracle circumvents the problem of unattachable nodes by avoiding the use of subgraph actions. this implies that the oracle will always produce a unique action sequence that fully recovers the gold graph as long as every node in the graph is aligned to some token. to guarantee that all nodes are aligned, we improve upon the alignment system from @xcite ; astudillo et al. (2020), which aligns a large majority, but not all amr nodes. 3 in order to do this, we apply a heuristic based on graph proximity to maintain local correspondences between graph nodes and sentence words. if a node is not aligned, we copy the alignment from its first child node, if existing, and otherwise the alignment from its first parent node. for example, in figure 2 if the node person was not provided with an alignment, our oracle would have assigned it to the aligned token of its child node employ-01. this is a recursive procedure -as long as there are some alignments to start with and the ground-truth graph is connected, all the nodes will get aligned. 3 roughly 1% of graphs contain about 1-2 unaligned nodes. our proposed transition system makes better use of bart pre-trained decoder compared to previous transition-based approaches (see section 6) while greatly simplifying the transition set. it also naturally produces node-to-word alignments via source token cursor in the meantime. ## parsing model we build our model on top of the pre-trained seqto-seq transformer, bart @xcite . we modify its architecture to incorporate a pointer network and internalize parser states induced by our transition system, and fine-tune for sentenceto-action generation. ## analysis transition system to assess whether our proposed transition system helps integration with pre-trained bart, we train both the apt model from @xcite and our sep-voc model on the transition system of ## related work using seq-to-seq to predict linearized graph sequences for parsing was proposed in @xcite and is currently a very extended approach @xcite @xcite . however, it is only recently with the rise of pre-trained transformer decoders, that these techniques have become dominant in semantic parsing. @xcite proposed custom multi-task pre-training and fine-tuning approach for conventional transformer models @xcite . the massively pre-trained transformer bart @xcite was used for executable semantic parsing in @xcite and amr parsing in @xcite . the importance of strongly pre-trained decoders seems also justified as bart gains popularity in various semantic generation tasks @xcite . our work aims at capitalizing on the outstanding performance shown by bart, while providing a more structured approach that guarantees well-formed graphs and yields other desirable sub-products such as alignments. we show that this is not only possible but also attains state-of-the art parsing results without graph re-categorization. our analysis also shows that contrary to @xcite , vocabulary sharing is not necessary for strong performance for our structural fine-tuning. encoding of the parser state into neural parsers has been undertaken in various works, including seq-to-seq rnn models @xcite @xcite , encoder-only transformers @xcite , seq-to-seq transformers @xcite and pre-trained language models @xcite . here we explore the application of these approaches to pre-trained seq-to-seq transformers. borrowing ideas from @xcite , we encode alignment states into the pre-trained bart attention mechanism, and re-purpose its selfattention as a pointer network. we also rely on a minimal set of actions targeted to utilize bart's generation with desirable guarantees, such as no unattachable nodes and full recovery of all graphs. we are the first to explore transition-based parsing applied on fine-tuning strongly pre-trained seq-toseq models, and we demonstrate that parser state encoding is still important for performance, even when implemented inside of a powerful pre-trained decoder such as bart. ## conclusion we explore the integration of pre-trained sequenceto-sequence language models and transition-based approaches for amr parsing, with the purpose of retaining the high performance of the former and structural advantages of the latter. we show that both approaches are complementary, establishing the new state of the art for amr 2.0. our results indicate that instead of simply converting the structured data into unstructured sequences to fit the need of the pre-trained model, it is possible to effectively re-purpose a generic pre-trained model to a structure-aware one achieving strong performance. similar principles can be applied to adapt other powerful pre-trained models such as t5 @xcite and gpt-2 @xcite for structured data predictions. it is worth exploring thoroughly the pros and cons of introducing structure to the model compared to removing structure from the data (linearization) in various scenarios.