Multilingual AMR -to-Text Generation (2020)

Abstract: Generating text from structured data is challenging because it requires bridging the gap between (i) structure and natural language (NL) and (ii) semantically underspecified input and fully specified NL output. Multilingual generation brings in an additional challenge: that of generating into languages with varied word order and morphological properties. In this work, we focus on Abstract Meaning Representations (AMRs) as structured input, where previous research has overwhelmingly focused on generating only into English. We leverage advances in cross-lingual embeddings, pretraining, and multilingual models to create multilingual AMR-to-text models that generate in twenty one different languages. Our multilingual models surpass baselines that generate into one language in eighteen languages, based on automatic metrics. We analyze the ability of our multilingual models to accurately capture morphology and word order using human evaluation, and find that native speakers judge our generations to be fluent.

## introduction generating text from structured data has a variety of applications in natural language processing. tasks such as decoding from tables @xcite , question answering from knowledge bases @xcite , and generation from rdf triples @xcite , knowledge graphs @xcite and linguistic meaning representations @xcite face similar challenges: interpreting structured input and writing fluent output. we focus on generating from graph structures in the form of abstract meaning representations (amr) @xcite . previous work has largely focused on generating from amr into english, but we propose a multilingual approach that can decode into twenty one different languages. compared to multilingual translation, decoding from structured input has distinct challenges. translation models take natural language input and must faithfully decode into natural language output. however, as shown in @xcite , bridging the gap between structured input and linear output is a difficult task. in addition, in structured input such as graphs, the input is usually semantically under-specified. for example, in amrs, function words are missing and tense and number are not given. thus, generation from structured input must bridge the gap between (i) structure and string and (ii) underspecified input and fully specified output. multilinguality brings a third challenge -that of generating in languages that have varied morphological and word order properties. annotating natural language with amr is a complex task and training datasets only exist for english foot_0 , so previous work on amr-to-text generation has overwhelmingly focused on english. we create training data for multilingual amr-to-text models, by taking the europarl multilingual corpus and automatically annotating the english data with amrs using the jamr semantic parser. we then use the english amrs as the input for all generation tasks. to improve quality, we leverage recent advances in natural language processing such as cross-lingual embeddings, pretraining and multilingual learning. cross-lingual embeddings have shown striking improvements on a range of cross-lingual natural language understanding tasks @xcite @xcite . other work has shown that the pre-training and fine-tuning approaches also help improve generation performance @xcite @xcite . finally, multilingual models, where a single model 2 related work amr-to-text generation. initial work on amr-to-text generation adapted methods from statistical machine translation (mt) @xcite , grammar-based generation @xcite , tree-to-string transducers @xcite , and inverted semantic parsing @xcite . neural approaches explored sequence-to-sequence models where the amr is linearized @xcite or modeled with a graph encoder @xcite @xcite @xcite . as professionally-annotated amr datasets are in english, all this work focuses on english. one exception is the work of @xcite which uses automatic translation to translate the english text of the ldc amr data into brazilian portuguese and align english with the portuguese translation to create portuguese-centric amrs. however, this work focuses only on one language. in contrast, we consider generation into twenty one languages. we use very different methods and generate from english-centric amrs, not target-language amrs. ## method to generate from amrs, we use neural sequence to sequence models that model the input amr with a transformer encoder and generate natural language with a transformer decoder. for all languages, the input is an english-centric amr that was derived automatically using the jamr semantic parser from english text. we pre-train both the amr encoder and the multilingual decoder and we leverage crosslingual embeddings. ## experimental setting we describe the various sources of data used to create multilingual amr-to-text generation models and describe the implementation and evaluation. ## results performance on gold english amr we evaluate our models trained on silver amr on the crosslingual amr dataset from @xcite where the input is a gold englishcentric amr and the output is available in three european languages: spanish, french, and italian. the results are shown in on english ldc data, we compare to existing work. even though it is trained on silver amrs and out of domain, non-ldc data, the multilingual model compares well with previous work (see impact of modeling improvements. for the multilingual model, we display the effect of incrementally adding additional modeling improvements (cf. ## conclusion abstract meaning representations were designed to describe the meaning of english sentences. as such they are heavily biased towards english. amr concepts are either english words, propbank framesets ("want-01") or special, english-based keywords (e.g., "date-entity"). the structure of amrs is also influenced by english syntax. for instance, the main relation of "i like to eat" is the concept associated with its main verb ("like") whereas given the corresponding german sentence "ich esse gern" (lit. "i eat willingly"), the main predicate might have been chosen to be "eat" ("essen"). in other words, amrs should not necessarily be viewed as an interlingua @xcite . nonetheless, our work suggests that it can be used as one: given an english-centric amr it is possible to generate the corresponding sentence in multiple languages. this is in line with previous work by @xcite which shows that despite translation divergences, amr parsers can be learned for italian, chinese, german and spanish which all map into an english-centric amr.