S panish A bstract M eaning R epresentation: Annotation of a General Corpus (2022)

Abstract: Abstract Meaning Representation (AMR), originally designed for English, has been adapted to a number of languages to facilitate cross-lingual semantic representation and analysis. We build on previous work and present the first sizable, general annotation project for Spanish AMR. We release a detailed set of annotation guidelines and a corpus of 486 gold-annotated sentences spanning multiple genres from an existing, cross-lingual AMR corpus. Our work constitutes the second largest non-English gold AMR corpus to date. Fine-tuning an AMR to-Spanish generation model with our annotations results in a BERTScore improvement of 8.8%, demonstrating initial utility of our work.

## introduction abstract meaning representation (amr) represents the core meaning of a sentence as a directed, rooted graph focused on predicate-argument structure @xcite (figure 1 ). nodes correspond to concepts and labels denote relations between concepts. labels can be core roles functioning as predicates or arguments, or other attributes such as :location or :manner. while there are large amr-annotated corpora available for english, cross-lingual adaptations of amr are necessary if amr is to be useful as an interlingua or intermediate representation for crosslingual tasks @xcite . recent work has adapted amr to a variety of languages ( §2.1), evaluating cross-lingual efficacy of rolesets, word senses, and how effectively amr relations capture "who is doing what to whom" in languages other than english. as amr aims to abstract away from morphosyntax, its graph structure is closer to logic than a syntactic parse. for english, amr removes information such as number, definiteness, tense, word class, and word order. yet, in many languages, morphosyntactic information in languages other than english carries rich, important semantic information beyond the "sugar" amr intends to avoid. therefore, it is important when developing non-english amr annotation schema to both consider consistency with work in other languages (primarily english) as well as effectively reflecting the semantics of the language being annotated as much as possible. spanish is one of the most widely spoken languages in the world. there has been one previous proposal for adapting @xcite presented a corpus of 50 representative annotations for a spanish translation of (the little prince) (lpp) ( §2.2). while @xcite noted that english amr failed to adequately capture semantic phenomena in spanish, they indicated that accurate representation could be accomplished by adding specific roles and constructions. for example, the english and spanish amrs in figure 1 , which annotate parallel sentences, have two syntactic divergences due to inherent differences between the languages @xcite . we extend this prior work on spanish amr and present the first substantial spanish amr corpus of 486 gold-annotated spanish amrs ( §4). specifically, we annotate the spanish sentences from the "abstract meaning representation 2.0 -four translations" dataset @xcite , a corpus from the news domain that has become a popular resource for evaluation of cross-lingual amr parsers @xcite @xcite and that spans more genres than lpp. to support the annotation, we develop annotation guidelines that update and complete those previously established for spanish ( §3). as with prior work, we find that amr's principle of abstracting away from morphosyntax creates challenges for representing meaning in agreement-rich languages such as spanish; we present solutions that may be extendable to other languages that exhibit similar linguistic phenomena ( §3.14). our work adds to the development of non-english amr schema and discusses how to balance consistency and compatibility with standard english amr while capturing pertinent semantic information not explicitly encoded in english. three annotators were involved ( §4); their work is verified with detailed analysis of inter-annotator agreement and disagreement ( §5). our annotations are publicly available on github. 1 finally, to underscore the utility of our gold annotations, we conduct an initial evaluation for a cross-lingual generation task ( §6). we show that by fine-tuning an amr-to-spanish generation model we are able to achieve an 8.8% increase in bertscore @xcite performance. ## aims and guidelines our primary aims with the development of this corpus included the release of a (1) sizable, (2) generalpurpose spanish amr corpus, which can be useful in the evaluation of cross-lingual amr parsers, (3) which effectively represents spanish semantics. we set out to meet these goals by (1) manually annotating 586 amrs, (2) annotating the four translations dataset, often used for evaluation of crosslingual amr parsers, and (3) developing guidelines which consider a range of linguistic phenomena. in this section, we discuss the key considerations and linguistic phenomena we prioritize in our approach to spanish amr annotation. ## fine-tuning a spanish generation model amr generation produces text from an amr. to evaluate the utility of our dataset in practical nlp tasks, we fine-tune the t5wtense generation model of the amr library amrlib to produce spanish sentences. 5 the t5wtense generation model uses the pretrained huggingface t5 transformer to convert amr graphs to text. we split our 486 annotations into 110 sentences (test) and 376 (training). 6 we compare the fine-tuned system output and the un-tuned system output to the corresponding spanish reference sentences from amr 2.0 -four translations @xcite . we use bertscore, an automatic evaluation metric for text generation @xcite , to perform this comparison, as previous work has demonstrated that it is the automatic metric most correlated with human judgments for (english) amr-to-text generation systems @xcite . for evaluating spanish text, the default bertscore model is bert-base-multilingual-cased, which is the model we use here. after fine-tuning t5wtense, we see a marked improvement in performance, increasing in bertscore by approximately 8.8% absolute (11.86% relative improvement). current state-of-the-art cross-lingual generation @xcite achieves a bertscore of 0.8534 on the same test set, 7 which indicates that by fine-tuning on only 376 spanish amr annotations, we are able to achieve results close to the current best performing model. 8 the marked improvement resulting from our fine-tuning demonstrates the utility of our corpus and suggests incorporating our data into more sophisticated generation or parsing models can lead to greater improvements. 9 ## conclusion we have presented an updated approach to spanish amr annotation which considers a broader range of meaningful linguistic phenomena than previous work. using updated guidelines, we constructed a corpus of 486 gold-annotated spanish amrs for the "amr 2.0 -four translations" dataset, achieving high amr inter-annotator agreement (0.83-0.89 iaa via smatch). gold spanish amrs will contribute to ongoing evaluation and training of crosslingual amr models; this is substantiated by our results in §6, which improved an off-the-shelf amrto-spanish generation system by fine-tuning on our data. little prior work on amr has set out to develop large-scale gold corpora in languages other than english; our work suggests that this is a fruitful effort, both to foster a better understanding of the cross-lingual properties of amr and to improve system performance on non-english nlp tasks.