Accounting for Language Effect in the Evaluation of Cross-lingual AMR Parsers (2022)

Abstract: Cross-lingual Abstract Meaning Representation (AMR) parsers are currently evaluated in comparison to gold English AMRs, despite parsing a language other than English, due to the lack of multilingual AMR evaluation metrics. This evaluation practice is problematic because of the established effect of source language on AMR structure. In this work, we present three multilingual adaptations of monolingual AMR evaluation metrics and compare the performance of these metrics to sentence-level human judgments. We then use our most highly correlated metric to evaluate the output of state-of-the-art cross-lingual AMR parsers, finding that Smatch may still be a useful metric in comparison to gold English AMRs, while our multilingual adaptation of S2match (XS2match) is best for comparison with gold in-language AMRs.

## introduction the abstract meaning representation (amr; @xcite formalism captures the meaning of a sentence or phrase as a rooted, directed acyclic graph. nodes correspond to concepts and the labeled edges reflect the relations between concepts. for example, the annotation in figure 1 features the amr annotation of the sentence "we will try not to make a mistake" in both penman (text-based) and graph form. the edge labels can be arguments (core roles, denoted as :argn), or one of a number of non-core roles such as :location or :manner. cross-lingual amr parsers convert non-english text to (english-focused) amr graphs. as there are no existing multilingual amr evaluation metrics and due to the limited availability of non-english gold amr annotations, these cross-lingual amr parsers have only ever been evaluated in comparison to gold english amrs. this established approach of comparison to english amrs using the monolingual smatch metric needs to be considered figure 1 : an amr annotation for the sentence "we will try not to make a mistake," in penman text-based notation and as a rooted graph. the diagram was made on the amreager website @xcite . we argue that cross-lingual amr parsing should represent the semantics, beyond the lexicon, faithfully to the source language. enabled by work developing sizable spanish @xcite and chinese @xcite gold amr corpora, we propose that cross-lingual amr parsers should be evaluated on gold amrs that match the language being parsed. in this work, we adapt three monolingual amr evaluation metrics to a multilingual setting, and evaluate the performance of these metrics in comparison to human judgments of cross-lingual sentence similarity. we show that our adaptation of s2match @xcite which leverages labse @xcite embeddings is highly correlated with, and most correlated to, human judgment of similarity. additionally, using our new cross-lingual evaluation metric, we evaluate the performance of crosslingual amr parsers and compare that with evaluations using the previously-used smatch metric. this provides a new, informative ranking of existing cross-lingual amr parsers and offers insight into the applicability of the monolingual smatch metric for cross-lingual evaluation. ours is the first work to address the evaluation of cross-lingual amr parsers with respect to the language being parsed. our contributions include: ## background abstract meaning representation parsers produce amr annotations from natural text. "cross-lingual amr parsing" refers to parsing a non-english sentence into a standard english amr @xcite . @xcite introduced the task of cross-lingual amr parsing and developed non-english parsers by projecting from english annotations to their non-english counterparts through the use of parallel corpora. current approaches to cross-lingual amr parsing evaluate amrs produced from text in four languages: mandarin chinese, spanish, italian, and german ( §5.1). language-specific amr parsers have also been developed to parse from mandarin chinese @xcite , portuguese (anchiêta and pardo, 2018), and indonesian @xcite . amr parsers have traditionally been evaluated using the smatch metric @xcite . in order to compare a pair of semantic graphs (system and gold), smatch aligns their nodes, searching for a maximal alignment via hill climbing. with these alignments, triples representing edges of the two graphs are compared to compute an f-score. cross-lingual amr parsers have been evaluated on gold english amr graphs created from english sentences, paired with sentences that were manually translated from those english sentences into other languages. the english amrs come from the amr 2.0 dataset @xcite , and the non-english sentence translations from the amr 2.0 -four translations dataset @xcite with translations into mandarin chinese, italian, spanish, and german. thus, cross-lingual amr parsers have been evaluated by comparing the system-produced amr parsed from the non-english sentence to the gold english amr corresponding to the translated sentence-which does not take into account any effect the source language might have on amr structure. we discuss this concern in §3. in addition to the resources noted above, recent work has produced gold spanish amrs for amr 2.0 -four translations spanish sentences @xcite . as a result, we can compare the system output to in-language (spanish) gold amrs, and develop metrics to enable this comparison. ## developing cross-lingual amr evaluation metrics when comparing english and non-english amrs, the concepts themselves are in different languages and the structure of the amr will also differ, as it is affected by the syntax and semantics of the language being parsed from @xcite @xcite . therefore, to be able to evaluate the similarity of amrs in two different languages, there are likely changes that need to be made to the monolingual metric. a naive assumption is that amr should be structurally the same for parallel sentences regardless of language, because amr encodes meaning and translation preserves meaning. however, previous work has demonstrated that this is not the case, and that even when lexical items are made to be monolingual, the source language has a marked effect on the amr structure itself (for at least english and chinese): @xcite reported smatch scores consistently below 50% between english and chinese parallel gold amr graphs, even when all chinese tokens are replaced by their original gold english amr: (s / surge-01 :arg1 (a / and :op1 (s2 / speed-01) :op2 (a2 / accident)) :mod (a3 / as-well)) original gold spanish amr: (c0 / aumentar-01 :manner (c1 / también) :arg1 (c2 / y :op1 (c3 / exceder-01 :arg1 (c4 / velocidad)) :op2 (c5 / accidente))) our translated version of the gold spanish amr: (c0 / increase :manner (c1 / also) :arg1 (c2 / and :op1 (c3 / exceed :arg1 (c4 / speed)) :op2 (c5 / accident))) corresponding english amrs. therefore, we alter existing metrics to be able to compare (english) amrs parsed from non-english sentences to gold in-language amrs. we consider the applicability of existing amr metrics in cross-lingual parser evaluation and adapt them to function multilingually. cross-lingual amr parsers are currently evaluated via smatch. here we consider three metrics: smatch, sembleu, and s2match. as mentioned in §2, smatch aligns the semantic graphs via hill climbing. s2match @xcite incorporates word embeddings into smatch to account for similarity of concept nodes without the same token being used. sembleu @xcite is based on the machine translation metric bleu @xcite . sembleu does not involve variable alignment and instead converts the graph to a bag of k-grams. broadly, our approach to adapting these monolingual metrics is that we alter smatch ( §3.1) and sembleu ( §3.2) by translating the lexical material in the amr graphs into english, and s2match ( §3.3) by using cross-lingual embeddings. ## analysis of metrics to compare cross-lingual amr metrics, we need a source of ground truth about how the sentences in a translation pair relate to one another. for this we utilize human judgments of cross-lingual similarity. because amr is a meaning representation, the similarity scores of cross-lingual amr pairs ideally should correlate with the similarity scores of their associated sentence pair. in line with previous work @xcite , we determine the accuracy of our amr metrics by calculating pearson's correlation between system output and human judgments of cross-lingual sentence similarity. specifically, we use gold amrs as input to the metrics and calculate how correlated the metric-based similarity scores are to the human similarity ratings for the corresponding sentence pair. we normalize the amrs by removing all wikification (links to the associated wikipedia pages for entities in the amr). ## evaluating cross-lingual amr parsers now that we have assessed the metrics discussed in §4 on gold amrs in comparison to human judgments, we are interested in seeing how existing cross-lingual amr parsers perform on our recommended cross-lingual metric versus on monolingual smatch. ## background on other evaluation metrics other metrics which we did not adapt in this paper have been proposed for amr evaluation. sgl bi xlpt-amr tp xs2match xs2match figure 4: average english smatch vs average spanish xs2match a document-level version of smatch @xcite (as opposed to sentence-level) aligns the roots of its sentence-level amr subgraphs. similarly, @xcite produces a version of smatch designed to specifically consider core semantics, called smatch-weighted. sema @xcite extends smatch by taking a breadth-first search approach to computing the maximum score; smatch relies on one-to-one variable matching. the evaluation is limited and the metric is only shown to be stricter than smatch. another existing monolingual amr metric we did not consider in this work is mf β . mf β @xcite measures how easily an amr can be reconstructed by amr parsers and measures the grammaticality of the produced text. mf β evaluation is more suited to amr-to-text generation evaluation than to text-to-amr parsing. goodman (2019) presents four amr normalization techniques to ensure that isomorphic amrs are evaluated as equivalent. the bamboo suite @xcite houses various amr similarity metrics to be able to assess the strengths and weaknesses of each metric. ## conclusion and future work our analysis of evaluation of cross-lingual amr parsers indicates the usefulness of xs2match as a multilingual evaluation method. we recommend this approach as a way to compare amrs parsed from non-english sentences to their gold non-english equivalents, while exploring additional alternatives in our work. we also find that using smatch in comparison to gold english amrs may be a useful tool for ranking cross-lingual amr parser performance in the absence of in-language gold amrs. with the future production of non-english gold amrs, the evaluation of cross-lingual amr parsers using our proposed metric will be more robust, accounting for the effect of source language on amr.