Inducing and Using Alignments for Transition-based AMR Parsing (2022)

Abstract: Transition-based parsers for Abstract Meaning Representation (AMR) rely on node-to-word alignments. These alignments are learned separately from parser training and require a complex pipeline of rule-based components, pre-processing, and post-processing to satisfy domain-specific constraints. Parsers also train on a point-estimate of the alignment pipeline, neglecting the uncertainty due to the inherent ambiguity of alignment. In this work we explore two avenues for overcoming these limitations. First, we propose a neural aligner for AMR that learns node-to-word alignments without relying on complex pipelines. We subsequently explore a tighter integration of aligner and parser training by considering a distribution over oracle action sequences arising from aligner uncertainty. Empirical results show this approach leads to more accurate alignments and generalization better from the AMR2.0 to AMR3.0 corpora. We attain a new state-of-the art for gold-only trained models, matching silver-trained performance without the need for beam search on AMR3.0.

## introduction abstract meaning representation (amr) was introduced as an effort to unify various semantic tasks (entity-typing, co-reference, relation extraction, and so on; @xcite . of existing approaches for amr parsing, transition-based parsing is particularly notable because it is high performing but still relies on node-to-word alignments as a core pre-processing step @xcite @xcite @xcite . these alignments are not in the training data and must be learned separately via a complex pipeline of rule-based systems, pre-processing (e.g., lemmatization), and post-processing to satisfy domain-specific constraints. such pipelines can fail to generalize well, propagating errors into training that reduce amr performance in new domains (e.g., amr3.0). this work studies how we can probabilistically induce and use alignments for transition-based amr parsing in a domain-agnostic manner, ultimately replacing the existing heuristics-based pipeline. to induce alignments, we propose a neural aligner which uses hard attention within a sequenceto-sequence model to learn latent alignments @xcite @xcite . while straightforward, this neural parameterization makes it possible to easily incorporate pretrained features such as character-aware word embeddings from elmo @xcite and also relax some of the strong independence assumptions in classic count-based aligners such as ibm model 1 @xcite . we find that the neural aligner meaningfully improves upon various baselines, including the existing domain-specific approach. to use the neural aligner's posterior distribution over alignments, we explore several methods. our first approach simply uses the map alignment from the neural aligner to obtain a single oracle action sequence, which is used to train the amr parser. however, this one-best alignment fails to take into account the inherent uncertainty associated with posterior alignments. our second approach addresses this via posterior regularization to push the amr parser's (intractable) posterior alignment distribution to be close to the neural aligner's (tractable) posterior distribution. we show that optimizing this posterior regularized objective results in a simple training scheme wherein the amr parser is trained on oracle actions derived samples from the neural aligner's posterior distribution. our final approach uses the neural aligner not as a regularizer but as an importance sampling distribution, which can be used to better approximate samples from the amr parser's posterior alignment distribution, and thus better approximate the otherwise intractable log marginal likelihood. in summary, we make the following empirical and methodological contributions: ## inducing alignments here, we describe our neural alignment model, which is essentially a variant of sequence-tosequence models with hard attention @xcite @xcite . in contrast to sb-align, our approach requires minimal pre-processing and does not have dependencies on many components or domain-specific rules. the alignment model is trained separately from the amr parser and optimizes the conditional likelihood of nodes in the linearized graph given the sentence. 3 the amr graph is linearized by first converting the graph to a tree, foot_3 and then linearizing the tree via a depth-first search, as in figure 1 . letting 𝑣 = 𝑣 1 , . . . , 𝑣 𝑆 be the nodes in the linearized amr graph, the log-likelihood is given by for parameterization, the sentence 𝑤 is encoded by a bi-directional lstm. each word is represented using a word embedding derived from a pretrained character-encoder from elmo @xcite , which is frozen during training. on the decoder side, the linearized amr tree history is represented by a uni-directional lstm. the decoder shares word embeddings with the text encoder. the prior alignment probability 𝑞(𝑙 𝑠 = 𝑖 | 𝑣 <𝑠 , 𝑤 ; 𝜙) is given by bilinear attention @xcite , ## using alignments the neural aligner described above induces a posterior distribution over alignments, 𝑞(𝑙 | 𝑤, 𝑔 ; 𝜙). we explore several approaches using this alignment distribution. ## experimental setup all code to run our experiments is available online foot_7 with apache license, 2.0. ## results and analysis the full results are shown in pipeline generalization sb-align was developed prior to the amr3.0 release, and because it incorporates a complex pipeline with domainspecific rules, one could argue it is specialized for prior datasets like amr2.0. in ## related work oracles for parsing dynamic oracles in syntactic parsing @xcite enable oracles to recover from imperfect sequences. oracles with random exploration in transition-based amr parsers have been previously explored using imitation-learning @xcite , reinforcement learning @xcite and oracle mining @xcite . in addition to this, @xcite produces multiple alignments via their rule-based system and selects the best based on parser performance. compared to our proposed posterior regularization training, dynamic oracle works exploit specific linguistic properties of syntactic trees not directly transferable to amr graphs. prior work on random exploration and amr selects oracles based on the smatch. our work uses the action space and oracle from structbart @xcite , which requires every node to be aligned to a word so that the amr graph is fully recoverable. we expose the parser to uncertainty by sampling alignment, which does not require computing the smatch metric. rather, the aligner is trained separately from the parser using pairs of sentences and their respective amr graphs. ## conclusion in this work we propose a general-purpose neural amr aligner, which does not use a complex alignment pipeline and generalizes well across domains. the neural parameterization allows the aligner to fully condition on the source and target context and easily incorporates pretrained embeddings, leading to improved performance. simply using our aligner to produce training data for a state-of-theart transition-based parser leads to improved results. we additionally propose a learning procedure using posterior regularization and importance sampling that involves sampling different action sequences during training. these incorporate alignment uncertainty and further improve parser performance. our results on gold-only amr training (i.e., without silver data) show that parsers learned this way match the performance of the prior stateof-the-art parsers without requiring beam search at test time.