Cross-lingual AMR Aligner: Paying Attention to Cross-Attention (2023)

Abstract: This paper introduces a novel aligner for Abstract Meaning Representation (AMR) graphs that can scale cross-lingually, and is thus capable of aligning units and spans in sentences of different languages. Our approach leverages modern Transformer-based parsers, which inherently encode alignment information in their cross-attention weights, allowing us to extract this information during parsing. This eliminates the need for English-specific rules or the Expectation Maximization (EM) algorithm that have been used in previous approaches. In addition, we propose a guided supervised method using alignment to further enhance the performance of our aligner. We achieve state-of-the-art results in the benchmarks for AMR alignment and demonstrate our aligner’s ability to obtain them across multiple languages. Our code will be available at [ https://www.github.com/babelscape/AMR-alignment ]( https://www.github.com/babelscape/AMR-alignment ).

## introduction at the core of natural language understanding lies the task of semantic parsing, aimed at translating natural language text into machine-interpretable representations. one of the most popular semantic formalisms is the abstract meaning representation @xcite , which embeds the semantics of a sentence in a directed acyclic graph, where concepts are represented by nodes, such as time, semantic relations between concepts by edges, such as :beneficiary, and the co-references by reentrant nodes, such as r representing rose. in crosslingual amr, the english amr graph represents the sentence in different languages (see figure 1 ). to date, amr has been widely used in machine translation @xcite , question answering @xcite , human-robot interaction @xcite ), text summarization @xcite and information extraction @xcite , among other areas. the alignment between spans in text and semantic units in amr graphs is an essential requirement for a variety of purposes, including training amr parsers @xcite , cross-lingual amr parsing @xcite , downstream task application @xcite , or the creation of new semantic parsing formalisms @xcite . despite the emergence of various alignment generation approaches, such as rule-based methods @xcite and statistical strategies utilizing expectation maximization (em) @xcite , these methods rely heavily on english-specific rules, making them incompatible with cross-lingual alignment. furthermore, even though several attempts extend the alignment to non-english sentences and graphs @xcite , these efforts are inherently monolingual and therefore lack the connection to the richer amr graph bank available in english, which can be exploited as a source of interlingual representations. on the other hand, current state-of-the-art amr parsers are auto-regressive neural models @xcite that do not generate alignment when parsing the sentence to produce the graph. therefore, to obtain both, one needs to i) predict the graph and then ii) generate the alignment using an aligner system that is based on language-specific rules. this paper presents the first amr aligner that can scale cross-lingually by leveraging the implicit information acquired in transformer-based parsers @xcite . we propose an approach for extracting alignment information from crossattention, and a guided supervised method to enhance the performance of our aligner. we eliminate the need for language-specific rules and enable simultaneous generation of the amr graph and alignment. our approach is efficient and robust, and is suitable for cross-lingual alignment of amr graphs. our main contributions are: (i) we explore how transformer-based amr parsers preserve implicit alignment knowledge and how we can extract it; cross-attention most state-of-the-art systems for amr parsing are based on encoder-decoder transformers, specifically on bart @xcite . these models consist of two stacks of transformer layers, which utilize self-and crossattention as their backbone. the popularity of transformer models has led to increased interest in understanding how attention encodes information in text and relates to human intuition @xcite and definitions of explainability @xcite . research has been conducted on how attention operates, relates to preconceived ideas, aggregates information, and explains model behavior for tasks such as natural language inference @xcite , translation @xcite @xcite , summarization @xcite or sentiment analysis @xcite . furthermore, there have been attempts to guide attention to improve interpretability or performance in downstream tasks @xcite . however, to the best of our knowledge, there has been no prior study on attention for amr parsing. this paper fills this gap by investigating the role of attention in amr parsing. ## method originally described by @xcite as "multi-head attention over the output of the encoder", and referred to as cross-attention in @xcite , it enables the decoder to attend to the output of the encoder stack, conditioning the hidden states of the autoregressive component on the input text. self-attention and cross-attention modules are defined as: ## ablation study gold spans leamr relies on a span segmentation phase, with a set of multiword expressions and stanza-based named entity recognition. we use the same system in order to have matching sentence spans. however, these sometimes differ from the gold spans, leading to errors. rules all modern alignment systems depend on rules to some degree. for instance, we use the subgraph structure for named entities, certain relations are matched to their parent or child nodes, etc. (see appendix a for more details). but what is the impact of such rules? as expected, both leamr and our unguided method see a considerable performance drop when we remove them. for relation, leamr drops by almost 60 points, since it relies heavily on the predictions of parent and child nodes to provide candidates to the em model. our unguided approach also suffers from such dependency, losing 25 points. however, our guided model is resilient to rule removal, dropping by barely one point on subgraph and 5 points on relation. layers figure 2 shows how alignment acts differently across heads and layers. we explore this information flow in the decoder by extracting the alignments from the sum of layers at different depths. the right of ## error analysis we identify two main classes of error that undermine the extraction of alignments. ## cross-lingual analysis: a case study to investigate the potential causes of misalignment between english and non-english languages, we conduct a case study that qualitatively examines the differences in alignment generated by different systems and languages. figure 4 illustrates the sentence, "why is it so hard to understand?" with its human translations in german, spanish, and italian, and its amr. in the italian translation, the subject of the verb is omitted, while in the spanish translation the focus of the question is modified from asking the reason why something is difficult to understand to asking directly what is difficult to understand, making "qué" the subject. as a consequence, in both cases making it impossible to align "it" with any word in either the italian or spanish sentence by machine translation alignment. ## conclusion in this paper, we have presented the first amr aligner that can scale cross-lingually and demonstrated how cross-attention is closely tied to alignment in amr parsing. our approach outperforms previous aligners in english, being the first to align cross-lingual amr graphs. we leverage the cross-attention from current amr parsers, without overhead computation or affecting parsing quality. moreover, our approach is more resilient to the lack of handcrafted rules, highlighting its capability as a standard-and language-agnostic aligner, paving the way for further nlp tasks. as a future direction, we aim to conduct an analysis of the attention heads that are not correlated with the alignment information in order to identify the type of information they capture, such as predicate identification, semantic relations, and other factors. additionally, we plan to investigate how the alignment information is captured across different nlp tasks and languages in the cross-attention mechanism of sequence-tosequence models. such analysis can provide insights into the inner workings of the models and improve our understanding of how to enhance their performance in cross-lingual settings.