Levi Graph AMR Parser using Heterogeneous Attention (2021)

Abstract: Coupled with biaffine decoders, transformers have been effectively adapted to text-to-graph transduction and achieved state-of-the-art performance on AMR parsing. Many prior works, however, rely on the biaffine decoder for either or both arc and label predictions although most features used by the decoder may be learned by the transformer already. This paper presents a novel approach to AMR parsing by combining heterogeneous data (tokens, concepts, labels) as one input to a transformer to learn attention, and use only attention matrices from the transformer to predict all elements in AMR graphs (concepts, arcs, labels). Although our models use significantly fewer parameters than the previous state-of-the-art graph parser, they show similar or better accuracy on AMR 2.0 and 3.0.

## introduction abstract meaning representation (amr) has recently gained lots of interests due to its capability in capturing abstract concepts @xcite . in the form of directed acyclic graphs (dags), an amr graph consists of nodes as concepts and edges as labeled relations. to build such a graph from plain text, a parser needs to predict concepts and relations in concord. while significant research efforts have been conducted to improve concept and arc predictions, label prediction has been relatively stagnated. most previous models have adapted the biaffine decoder for label prediction @xcite @xcite @xcite . these models assign labels from the biaffine decoder to arcs predicted by another decoder, which can be misled by incorrect arc predictions during decoding. the enhancement of message passing between decoders for arc and label predictions has shown to be effective. among these works, @xcite emerge with an iterative method to exchange embeddings between concept and arc predictions and feed the enhanced embeddings to the biaffine decoder for label prediction. while this approach greatly improves accuracy, it complicates the network architecture without structurally avoiding the error propagation from the arc prediction. this paper presents an efficient transformerbased @xcite approach that takes a mixture of tokens, concepts, and labels as inputs, and performs concept generation, arc prediction, and label prediction jointly using only attentions from the transformer without using a biaffine decoder. its compact structure ( §3.3) enables crossattention between heterogeneous inputs, providing a complete view of the partially built graph and a better representation of the current parsing state. a novel levi graph decoder ( §3.4) is also proposed that reduces the number of decoder parameters by 45% (from 5.5 million to 3.0 million) yet gives similar or better performance. to the best of our knowledge, this is the first text-to-amr graph parser that operates on the heterogeneous data and adapts no biaffine decoder. ## related work recent amr parsing approaches can be categorized into four classes: (i) transition-based parsing which casts the parsing process into a sequence of transitions defined on an abstract machine (e.g., a transition system using a buffer and a stack) @xcite @xcite @xcite @xcite @xcite , (ii) seq2seq-based parsing 2 which transduces raw sentences into linearized amr graphs in text form @xcite @xcite @xcite , (iii) seq2graph-based parsing which incrementally and directly builds a semantic graph via expanding graph nodes without resorting to any transition system @xcite @xcite . (iv) graph algebra parsing which translates an intermediate grammar structure into amr @xcite @xcite . our work is most closely related to seq2graph paradigm while we extend the definition of node to accommodate relation labels in a levi graph. we generate a levi graph which is a linearized form originally used in seq2seq models for amrto-text @xcite @xcite . our levi graph approach differs from seq2seq approaches in its attention based arc prediction, where arc is directly predicted by attention heads instead of brackets in the target sequence. ## conclusion we presented two effective approaches which achieve comparable (or better) performance comparing with the state-of-the-art parsers with significantly fewer parameters. our text-to-graph transducer enables self-and cross-attention in one transformer, improving both concept and arc prediction. with a novel levi graph formalism, our parser demostrates its advantage on relation labeling. an interesting future work is to preserve benefits from both approaches in one model. it is also noteworthy that our levi graph parser can be applied to a broad range of labeled graph parsing tasks including dependency trees and many others.