Error Exploration for Automatic A bstract M eaning R epresentation Parsing (2023)

Abstract: Following the data-driven methods of evaluation and error analysis in meaning representation parsing presented in (Buljan et al., 2022), we performed an error exploration of an Abstract Meaning Representation (AMR) parser. Our aim is to perform a diagnosis of the types of errors found in the output of the tool in order to implement adaptation and correction strategies to accommodate these errors. This article presents the exploration, its results, the strategies we implemented and the effect of these strategies on the performances of the tool. Though we did not observe a significative rise on average in the performances of the tool, we got much better results in some cases using our adaptation techniques.

## introduction semantic parsing of natural language is the task of extracting a formal meaning structure from a natural language sentence. semantics of natural languages can be formalised in various ways, see for instance @xcite and more recently @xcite for overviews; semantic parsing can be performed from any natural language into any of the semantic formalisms. one of these formalisms, abstract meaning representations @xcite ) has been widely used in the context of deep semantic parsing of english and up to at least ten other languages, including french, german, spanish, italian, and polish. there are two main types of approaches to multilingual amrs: either the amr graphs concepts are consistent with the target language (e.g. french concepts for french sentences), or the parsing results in an amr graph with english concepts (propbank-based). in this paper, we work in the scope of the latter approach. machine semantic parsing of english has reached high-quality results, scoring over 83% smatch score @xcite foot_0 for the state of the art approaches @xcite . in this context, we want to focus on the remaining 17%, and investigate both why the parser performs badly on these inputs and why the evaluation techniques would consider these parses as bad ones. we have limited our error explorations to languages we were familiar with; in particular, expert annotators familiar with chinese should be involved in a follow-up study covering chinese, for which a large amount of amr annotation has been done. machine amr parsing works well, making the cases where it performs badly particularly interesting both linguistically and for deep learning studies. to make amr parsing truly usable and reliable for real-life applications such as automatic summarization @xcite , question/answer generation @xcite , and neural machine translation @xcite we need to be able to trust it. we believe that this trust will come from a deep understanding of both our models and our data. the work presented in this article takes roots in explainability of artificial intelligence and computational linguistics. we conduct an error analysis and annotation exploration of the 50 worst examples from development corpora. we work in a multilingual context, on english (en), french (fr), german (de), spanish (es), italian (it), and polish (pl). our aim in this article is to share the error categories that we observed along with our attempts to remediate these errors, and the results of these attempts, in particular in terms of (non-significant) effects on the smatch score. while our work constitutes a negative proof of concept, we still think it is an important contribution to share in the field to help to constitute a baseline for such adaptation techniques and encourage research and dialogues around them. ## amr parsing amr parsing was explicitly developed for english only. its goal was to represent sentences through relations between predicates and their semantic arguments. these representations are now machinegenerated and have been extended to at least ten other languages. ## error exploration we performed an error analysis of the parser's outputs. we identified and implemented two strategies based on this analysis. our results show improvements in the parsing results that are qualitatively interesting but quantitatively not significant enough. our analysis was done on the development corpus of amr 3.0 (ldc2020t02), as we wanted to avoid introducing any bias in our study by using the test corpus. the sentences were machine translated into the given language, parsed using a model trained on the machine-translated training corpus, and then evaluated using the smatch python package against the gold development corpus. then, the sentences were ranked by worst smatch score, and the 50 first ones were annotated. ## related work there are to our knowledge not many publications presenting systematic explorations of machine amr parsing mistakes for the purpose of improving the explored tool. this observation might come from a publication bias, as a scientific community, we tend to publish positive results over negative ones. in @xcite , the authors present a discussion of methodological choices for diagnostic evaluation and error analysis in the context of four semantic parsers, two of which output amr graphs. this article also explores one of the alternatives to smatch, developed for several semantic representations of language (not only amr), as part of the meaning representation parsing task. @xcite presents another way of measuring the quality of automatic parses by using smatch to compute more fine-grained metrics. stemming from this work, @xcite focuses on reentrancy phenomena in amr graphs, categorizes their types, and shows results of experiments performed via an oracle correcting these errors, augmenting the overall parsing performance by 5%. smatch is also being questioned in a multilingual context. in @xcite , the authors argue for the necessity of a multilingual amr evaluation metric and present a multilingual adaptation of s2match called xs2match. the work presented in our article is inspired by the previous work of the same authors @xcite ; in this work, they annotate translation divergences between a corpus of english and a corpus of spanish data, grounding their annotation schema in amr and labelling type and cause of divergences. ## discussion and conclusion as shown in the next steps for our research are twofold. on one hand, we will continue the diagnostic of our approach, in particular for languages other than english, by evaluating our parser using scores such as xs2match and exploring the errors that get the lower scores; conjointly, as translation issues were majoritary in our analyses, we will investigate how manual correction of translations can improve the parsing's quality. on the other hand, we will investigate other approaches for our parser. several categories of errors we diagnosed come from the seq2seq method and from the machine translation tools we use to produce the non-english corpora.