AMR Parsing with Causal Hierarchical Attention and Pointers (2023)

Abstract: Translation-based AMR parsers have recently gained popularity due to their simplicity and effectiveness. They predict linearized graphs as free texts, avoiding explicit structure modeling. However, this simplicity neglects structural locality in AMR graphs and introduces unnecessary tokens to represent coreferences. In this paper, we introduce new target forms of AMR parsing and a novel model, CHAP, which is equipped with causal hierarchical attention and the pointer mechanism, enabling the integration of structures into the Transformer decoder. We empirically explore various alternative modeling options. Experiments show that our model outperforms baseline models on four out of five benchmarks in the setting of no additional data.

## introduction abstract meaning representation @xcite is a semantic representation of natural language sentences typically depicted as directed acyclic graphs, as illustrated in fig. 1a . this representation is both readable and broad-coverage, attracting considerable research attention across various domains, including information extraction @xcite , summarization @xcite , and vision-language understanding @xcite . however, the inherent flexibility of graph structures makes amr parsing , i.e., translating natural language sentences into amr graphs, a challenging task. the development of amr parsers has been boosted by recent research on pretrained sequenceto-sequence (seq2seq) models. several studies, categorized as translation-based models, show that fine-tuning pretrained seq2seq models to predict linearized graphs as if they are free texts (e.g., examples in tab.1.ab) can achieve competitive or even superior performance @xcite @xcite . this finding has spurred a wave of subsequent efforts to design more effective training strategies that maximize the potential of pretrained decoders @xcite @xcite , thereby sidelining the exploration of more suitable decoders for graph generation. contrary to preceding translation-based models, we contend that explicit structure modeling within pretrained decoders remains beneficial in amr parsing. to our knowledge, the ancestor parser @xcite is the only translation-based model contributing to explicit structure modeling, which introduces shortcuts to access ancestors in the graph. however, amr graphs contain more information than just ancestors, such as siblings and coreferences, resulting in suboptimal modeling. in this paper, we propose chap, a novel translation-based amr parser distinguished by three innovations. firstly, we introduce new target forms of amr parsing. as demonstrated in tab. 1.c-e, we use multiple layers to capture different semantics, such that each layer is simple and concise. particularly, the base layer, which encapsulates all meanings except for coreferences (or reentrancies), is a tree-structured representation, enabling more convenient structure modeling than the graph structure of amr. meanwhile, coreferences are presented through pointers, circumventing several shortcomings associated with the variablebased coreference representation @xcite for more details) used in all previous translation-based models. secondly, we propose causal hierarchical attention (cha), the core mechanism of our incremental structure modeling, inspired by transformer grammars @xcite . cha describes a procedure of continuously composing child nodes to their parent nodes and encoding new nodes with all uncomposed nodes, as illustrated in fig. 2 . unid name representation a pm ( a / alpha :arg0 ( b / beta ) :arg1 ( g / gamma :arg2 b ) ) 3c ). (d) is for ⇓double(c) (fig. 3b ). (e) is for ⇑ (fig. 3d ). red pointers , which represent coreferences, constitute the coref layer. blue pointers , which point to the left boundaries of subtrees, constitute the auxiliary structure layer. texts, which encapsulate all other meanings, constitute the base layer. ## structure modeling we primarily highlight two advantages of incorporating structured modeling into the decoder. firstly, the sequential order and adjacence of previous linearized form mismatch the locality of real graph structures, making the transformer decoder hard to understand graph data. specifically, adjacent nodes in an amr graph exhibit strong semantic relationships, but they could be distant in the linearized form (e.g., person and tour-01 in fig. 1a ). conversely, tokens closely positioned in the linearized form may be far apart in the amr graph (e.g., employ-01 and tour-01 in fig. 1a ). secondly, previous models embed variables into the linearized form (e.g., b in tab. 1.a @xmath0. 1.b) and represent coreferences (or reentrancies) by reusing the same variables. however, the literal value of variables is inconsequential. for example, in the penman form, (a / alpha :arg0 (b / beta)) conveys the same meaning as (n1 / alpha :arg0 (n2 / beta)). furthermore, the usage of variables brings up problems regarding generalization @xcite . for instance, in the spring dfs form, <@xmath1, @xmath2. ## parsing model our parser is based on bart @xcite , a pretrained seq2seq model. we make three modifications to bart: (1) we add a new module in the decoder to encode generated pointers, (2) we enhance decoder layers with cha, and (3) we use the pointer net to predict pointers. ## conclusion we have presented an amr parser with new target forms of amr graphs, explicit structure modeling with causal hierarchical attention, and the integration of pointer nets. we discussed and empirically compared multiple modeling options of cha and the integration of cha with the transformer decoder. eventually, chap outperforms all previous models on in-distribution benchmarks in the setting of no additional data, indicating the benefit of structure modeling.