AMR s Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing (2023)

Abstract: In this paper, we examine the current state-of-the-art in AMR parsing, which relies on ensemble strategies by merging multiple graph predictions. Our analysis reveals that the present models often violate AMR structural constraints. To address this issue, we develop a validation method, and show how ensemble models can exploit SMATCH metric weaknesses to obtain higher scores, but sometimes result in corrupted graphs. Additionally, we highlight the demanding need to compute the SMATCH score among all possible predictions. To overcome these challenges, we propose two novel ensemble strategies based on Transformer models, improving robustness to structural constraints, while also reducing the computational time. Our methods provide new insights for enhancing AMR parsers and metrics. Our code is available at [ https://www.github.com/babelscape/AMRs-Assemble ]( https://www.github.com/babelscape/AMRs-Assemble ).

## introduction semantic parsing is the subfield of natural language understanding @xcite that aims to encode the meaning of a sentence in a machineinterpretable structure. one of the formalisms that has gained more attention is the abstract meaning representation @xcite , which embeds the semantics of a sentence in a directed acyclic graph. in amr, concepts are represented by nodes, and semantic relations between concepts by edges (see figure 1 ). amr parsing has been applied to various areas of nlp, including question answering @xcite @xcite , text summarization @xcite , information extraction @xcite , and machine translation @xcite , and has been extended to non-english languages @xcite igit, 2022; @xcite . current amr parsing approaches are based on transformer sequence-to-sequence (seq2seq) models @xcite , which translate text into a linearized representation of the amr graph. recently, there have been some improvements through techniques such as pre-training on structural graph information @xcite , incorporating shallow semantic information @xcite , modifying ancestor information during decoding @xcite , and adding a structural graph prediction task during training @xcite . nevertheless, in an attempt to push smatch @xcite performance, there has been a recent trend towards ensemble models, which merge amr graph predictions from multiple systems. some examples include graphene @xcite , a graph mining algorithm that searches for the largest common structure among the graph predictions, or the maximum bayes smatch ensemble @xcite , which introduces a bayesian ensemble approach in order to create high-quality silver data. however, notwithstanding their higher performance, ensem-ble models are potentially more vulnerable to producing corrupted amr graphs. for instance, @xcite highlighted that better smatch scores do not always correlate with better parsing. in this study, we conduct an investigation into the reasons why ensemble models improve their performance and in which cases they do so despite producing corrupted output. our analysis reveals three significant drawbacks in these approaches: i) ensemble systems do not consider structural constraints in amr, treating amr graphs as regular sets of triplets, ii) they rely on smatch, which does not impose amr constraints, exacerbating the problem of corrupted amr graphs produced by ensemble methods that prioritize a higher score over adherence to structural constraints, as is the case with graphene, and iii) they are computationally expensive. our findings highlight the need for more robust evaluation metrics that hold to the structural constraints of amr. in this paper, we present two novel ensemble strategies that address the above limitations of current approaches. in the first strategy, we follow previous merging methods, showing how to train a seq2seq model to combine different predictions by taking into account both the original sentence and predictions from multiple models. in our second approach, we propose using selection as the ensembling strategy, where we select the best graph instead of merging. specifically, we base our method on the perplexity score of the model. additionally, we propose a graph algorithm that checks the structural constraints in amr graphs. through these contributions, we aim to provide more robust and efficient solutions for ensembling amrs. ## amrs assemble! the task of amr parsing can be framed as a seq2seq task, where the input @xmath0, x 2 , ..., x m ] is a sequence of m tokens and the output @xmath1, g 2 , ..., g n ] is a linearized graph with n tokens. to illustrate, the linearized representation of the amr graph in figure 1 is as follows: the goal of seq2seq amr parsing task is to learn a function that models the conditional probability: in this work, we use longt5 @xcite ) as the seq2seq model, which is specialized for long sequences, making it feasible to provide sentences and linearized graphs as input. ## conclusion in this paper, we leveraged self-supervised pretraining and a denoising autoencoder architecture to achieve strong results in merging amr graph predictions. we also introduced two novel approaches for ensembling that select the best prediction from a set of candidates using simple and efficient perplexity score functions. these results suggest that the selection strategy is a promising alternative for ensembling, since it achieves competitive performance while being less expensive. furthermore, we developed an algorithm that checks the structural amr constraints in parsing outputs. this allowed us to perform an analysis that revealed how previous ensemble models produce higher score graphs but exploit smatch weaknesses that lead to increased structural issues. overall, our findings highlight the need for more robust evaluation metrics and ensemble models that are designed to adhere to the structural constraints.