Incorporating Graph Information in Transformer-based AMR Parsing (2023)

Abstract: Abstract Meaning Representation (AMR) is a Semantic Parsing formalism that aims at providing a semantic graph abstraction representing a given text. Current approaches are based on autoregressive language models such as BART or T5, fine-tuned through Teacher Forcing to obtain a linearized version of the AMR graph from a sentence. In this paper, we present LeakDistill, a model and method that explores a modification to the Transformer architecture, using structural adapters to explicitly incorporate graph information into the learned representations and improve AMR parsing performance. Our experiments show how, by employing word-to-node alignment to embed graph structural information into the encoder at training time, we can obtain state-of-the-art AMR parsing through self-knowledge distillation, even without the use of additional data. We release the code at [ http://www.github.com/sapienzanlp/LeakDistill ]( http://www.github.com/sapienzanlp/LeakDistill ).

## introduction creating a machine-interpretable representation of meaning lies at the core of natural language understanding and has been framed as the semantic parsing task. multiple formalisms have been proposed over the years, e.g., prague czech-english dependency treebank @xcite , universal conceptual cognitive annotation @xcite , babelnet meaning representation @xcite ; however, abstract meaning representation @xcite has received more attention thanks to the large corpus available and a well-defined structure. amr captures text semantics in the form of a directed acyclic graph (dag), with nodes representing concepts and edges representing semantic relationships between them (see figure 1 ). currently, amr is widely employed in a plethora of nlp domains, such as information extraction @xcite , text summarization @xcite , question answering @xcite @xcite , human-robot interaction @xcite , and machine translation @xcite , among others. until a short while ago, autoregressive models proved to be the best approach for semantic parsing because of their outstanding performance without relying on sophisticated ad-hoc architectures @xcite . then, more recently, several approaches have emerged to increase performance by including structural information in the model @xcite , adding extra semantic role labeling tasks @xcite or by ensembling strategies @xcite . in this paper, following the effort of strengthening the model's learning phase by incorporating meaningful structural information, we investigate the use of structural adapters @xcite that are basically graph neural networks (gnns) embedded in the encoder of a transformer encoder-decoder architecture. the structural information is derived from intrinsic concept-node alignments from which we build a word-based graph with a structure similar to the original amr. leveraging such a graph implies partial data leakage: the graph structure is revealed to a model during training. to overcome the lack of the leaked information at inference time, we explore knowledge distillation (kd), a technique that transfers knowledge from a teacher model to a student model @xcite . the word-based graph is employed with the structural adapters to obtain soft targets (the teacher path), which are then used for selfdistillation, transferring the knowledge to the student, which only has access to the text. our main contributions are: i) exploring how to add structural information to the amr parsing model using structural adapters and self-knowledge distillation, ii) state-of-the-art results in amr parsing for amr 2.0 and amr 3.0 datasets, and iii) competitive base models for amr parsing. ## related work over the years, multiple trends have appeared to parse amr graphs: using statistical methods @xcite @xcite , neural-transition based parsers @xcite @xcite or bidirectional transformers @xcite @xcite based on bert @xcite . recently, autoregressive models based on bart @xcite have emerged as a dominant approach for amr parsing, since they obtained state-of-the-art performance without complex pipelines. one notable example is spring @xcite , which frames amr parsing as a neural machine translation task, where text is translated into a linearized version of the graph. subsequently, several works extended spring using a variety of different strategies. @xcite leverages multitask learning to improve cross-lingual amr parsing results. atp @xcite expands the dataset with extra auxiliary tasks such as semantic role labeling and dependency parsing, with pseudo-amr graphs constructed based on a particular task. am-rbart @xcite uses a pre-training strat-egy based on masked language modeling where both text and graph need to be denoised, using 200k graphs generated by spring. however, despite their efforts to enhance spring's performance, all these systems rely on additional external data. although ancestor @xcite , which modifies ancestor information during decoding, and bibl @xcite , that adds a secondary graph masking task while training, do not rely on extra data, their performance improvements remain relatively limited. our proposed model effectively bridges the gap in performance between "with" and "without" extra data by integrating explicit structural information during the training phase. ## word-aligned graph our goal is to incorporate graph-structured information into the encoder of a transformer-based parser. however, the model only has access to the input sentence at that stage, with no hidden representation of amr-specific nodes and relations. thus, we simplify the amr structure to a wordbased graph by exploiting a pre-existing alignment between spans in text and semantic units in the corresponding amr graph (see figure 1 ). first, starting with the source amr graph, we replace the labels of the amr nodes and relations with the words of the corresponding sentence as provided by the alignment (figure 2 , left). next, we convert each edge into a node and connect it to its original endpoints (see figure 2 , center). @xcite did for amr graphs, we split each multi-token node (e.g., freedom in figure 2 ) into a parent node represented by the first token and children nodes connected to it which contain the remaining tokens. we name the resulting graph representation the word-aligned graph (wag). we will leverage wags to enrich the encoder's hidden representations of words with the amr graph's structural information. unfortunately, a problem arises with non-aligned nodes (e.g., the :location relation in figure 2 ), since they will not have associated hidden states. therefore, we have two alternatives: i) remove nodes for which we do not have hidden states (contracted wag), or ii) create new hidden states for them (full wag). contracted wag as a first option, we remove non-aligned nodes from the graph. however, deleting the nodes from the original graph would produce a disconnected graph. to obtain a connected structure similar to the original graph, we contract nodes rather than removing them. a contracted wag (cwag) is a graph in which non-aligned nodes are merged with their closest parent node along with all their relations. figure 2 (right) depicts a cwag. full wag alternatively, we preserve the nodes without alignment (e.g., the node "location" in figure 2 (center) ). this type of graph is referred to as a full wag (fwag), figure 2 (center) shows an example of fwag. ## structural adapters for amr parsing in this section, we describe the main components of our structure-enhanced approach to amr parsing. ## experimental setup to demonstrate the benefits of incorporating structural information in amr parsing, we devise a set of experiments to assess its performance in comparison to state-of-the-art models. before delving into details, we provide information regarding the datasets (section 6.1), the metrics (section 6.2) and the model (section 6.3) used in our experiments. ## results in this section, we provide our experimental findings. all tables show single-run results. ## performance analysis seq2seq parsers show decreased performance for longer sentences since a single error at decoding time in an early step can lead to compound errors and suffer from exposure bias. we explore how this affects our model compared to spring, atp and amrbart. figure 5 shows the performance on amr 3.0 test set for buckets of 200 sentences split by the number of words. while performance is similar on shorter sentences, with amrbart showing slightly better performance, in longer sentences of over 14 words leakdistill fares better, especially compared to the baseline, which drops to 80 smatch points. this experiment also shows how performance is relatively stable for mediumlength sentences (10-30 words, oscillating around 85 points), while it starts deteriorating for longer ones. the high performance on short sentences is likely due to easy-to-parse structures, such as single date sentences. ## conclusion we presented a new approach to training the transformer architecture where partial information of the target sequence can be learned via self-knowledge distillation: the information can be leaked in the encoder implicitly through transformer adapters which improve training but are switched off during inference. by employing this approach in amr parsing, we achieved state-of-the-art results among non-ensemble methods. moreover, we produced a lightweight amr parser that outperforms spring while having four times fewer parameters. we also showed that, for all methods, performance degrades as the number of words increases. interestingly, our approach can potentially be used in other tasks, such as relation extraction, where alignments between input and target sequence elements exist, or structural information is unavailable at inference time.