Effect of Source Language on AMR Structure (2022)

Abstract: The Abstract Meaning Representation (AMR) annotation schema was originally designed for English. But the formalism has since been adapted for annotation in a variety of languages. Meanwhile, cross-lingual parsers have been developed to derive English AMR representations for sentences from other languages—implicitly assuming that English AMR can approximate an interlingua. In this work, we investigate the similarity of AMR annotations in parallel data and how much the language matters in terms of the graph structure. We set out to quantify the effect of sentence language on the structure of the parsed AMR. As a case study, we take parallel AMR annotations from Mandarin Chinese and English AMRs, and replace all Chinese concepts with equivalent English tokens. We then compare the two graphs via the Smatch metric as a measure of structural similarity. We find that source language has a dramatic impact on AMR structure, with Smatch scores below 50% between English and Chinese graphs in our sample—an important reference point for interpreting Smatch scores in cross-lingual AMR parsing.

## introduction though the abstract meaning representation (amr; @xcite framework was originally designed for annotating english sentences, and not intended as an interlingua, it has since been adapted to a number of other languages ( §2.1), raising the question of how well it abstracts away from the particularities of individual languages. to investigate amr's ability to serve as an interlingua, previous work has explored methods of characterizing the types of differences between parallel amr graphs (amrs annotating parallel sentences in different languages; §2.2). however, there has not yet been an effort to systematically quantify the effect on amr structure of the language of the sentence being parsed (hereafter, the source language). we hypothesize that regardless of any language-specific information in the amr (i.e. if the labels are made to be in the same language), the structure of amrs across language pairs will likely differ because of the linguistic properties of the source sentence. to better understand the impact of language on amr structure in the pursuit of effective evaluation of cross-lingual amr pair similarity, we aim to quantify the amount of impact in parallel amrs. here we explore the effect of source language on amr structure in the large annotated parallel corpus of mandarin chinese and english amrs @xcite . to quantify the impact of source language on the amr, we eliminate the measurable impact of lexical divergence and focus solely on structural divergences. to do this, we take a pair of parallel english and chinese amrs and manually translate every word in the chinese graph into its english equivalent. structural elements of the amr are largely unchanged ( §3.2). we then evaluate via smatch @xcite , which is an algorithm to compare amr graphs and calculate similarity. ultimately, we have a smatch score quantifying the effect of source language on amr structure. from these smatch scores, we are able to demonstrate that the source language has a dramatic effect on the structure of an amr, even if the amr is a gold annotation with no noise introduced by automatic parsing. this result has important implications for (1) identifying cross-linguistic inconsistencies in the amr schema, and (2) interpreting scores in cross-lingual amr parsing evaluations @xcite . 1 our primary contributions include: • a novel approach to quantifying effect of source language on amr structure; • a small dataset of 120 chinese amrs with english concept labels, following our approach; foot_1 and • an analysis of the smatch score differences between our chinese amrs with english concept labels and the corresponding gold english amrs. ## results & analysis we collect 60 annotations from each annotator, with 20 sentences overlapping so that we can calculate interannotator agreement (120 annotations total, on 100 unique sentences). we calculate the smatch scores between the annotations (chinese amr with english concepts) and the corresponding gold english amr. ## conclusion our case study between chinese and english serves as an analysis of the impact of linguistic divergence between those two languages on amr structure. through our annotation process of translating chinese concepts to english, we find that there is a dramatic impact on amr structures, with smatch scores between our annotations and the gold english amrs falling below 50%. for comparison, inter-annotator smatch scores within a single language (chinese) in the same domain have been reported at 83% @xcite . this substantive impact on amr structure motivates further consideration for source language when working with amr cross-lingually-either in evaluating cross-lingual amr parsers or when developing and comparing amr schema in new languages. as a meaning representation, it is critical that an amr graph effectively reflect the meaning of the sentence being parsed. current cross-lingual amr parsers evaluate accuracy of a parsed non-english sentence by comparing to the corresponding gold english amr. our newfound evidence that source language has a sizable effect on amr structure should be taken into account when interpreting cross-lingual smatch evaluations. ideally, gold amrs should be created in the source language for evaluating cross-lingual parsers (even if sufficient training data is only available in english). future work might investigate steps to mitigate source language impact when evaluating cross-lingual amr parsing, or further investigate the effect in other language pairs.