Generalising Multilingual Concept-to-Text NLG with Language Agnostic Delexicalisation (2021)

Abstract: Concept-to-text Natural Language Generation is the task of expressing an input meaning representation in natural language. Previous approaches in this task have been able to generalise to rare or unseen instances by relying on a delexicalisation of the input. However, this often requires that the input appears verbatim in the output text. This poses challenges in multilingual settings, where the task expands to generate the output text in multiple languages given the same input. In this paper, we explore the application of multilingual models in concept-to-text and propose Language Agnostic Delexicalisation, a novel delexicalisation method that uses multilingual pretrained embeddings, and employs a character-level post-editing model to inflect words in their correct form during relexicalisation. Our experiments across five datasets and five languages show that multilingual models outperform monolingual models in concept-to-text and that our framework outperforms previous approaches, especially in low resource conditions.

## introduction recently, neural approaches to language generation have become predominant in various tasks such as concept-to-text natural language generation (nlg), summarisation, and machine translation thanks to their ability to achieve state-of-the-art performance through end-to-end training @xcite @xcite . specifically in machine translation, deep learning models have proven easy to adapt to multilingual output @xcite and have been demonstated to successfully transfer knowledge between languages, benefiting both the low and high resource languages @xcite . in the concept-to-text nlg task, the language generation model has to produce a text that is an input mr: x1 x2 x3 x4 broadcastedby firstaired lastaired where: @xmath0, @xmath1, @xmath2, @xmath3 gold target references: таблица 1: delexicalisation and relexicalisation strategies on english and russian examples from the webnlg 2020 challenge. ## rare and unseen inputs in nlg due to the existence of open-set and numerical attributes in the aforementioned datasets, it is common during testing for mrs to contain rare or unseen values. certain datasets are even more challenging in this regard (e.g. webnlg challenge 2020) as they also contain unseen relations in the development and test subsets. several techniques have been proposed to mitigate this problem. delexicalisation, also known as anonymisation or masking, is a pre/post-processing procedure that attempts to mitigate problems with data sparsity. in preprocessing, all values in the mr that appear verbatim in the target sentence are replaced in both input and output with specific placeholders, e.g. "x-" followed by the corresponding attribute (e.g. "x-type") so that the placeholder still captures rele-vant semantic information. in figure 1 we use numbered placeholders instead, for clarity and space. the model is trained to generate the target text containing these placeholders, which are subsequently replaced with the corresponding true values (i.e. relexicalised) in post-processing. see figures 1 and figure 2 for examples; we mark this strategy as exact due to the exact matching of the values with the text. to improve delexicalisation accuracy, n-gram matching @xcite has been proposed as an alternative. thanks to its simplicity and efficacy, delexicalisation is widely used by many systems, including the winning systems of major concept-to-text nlg shared tasks @xcite @xcite . mapping the values as such can be sufficient for simple datasets, but otherwise, incorrect or incomplete delexicalisation will lead to inconsistent input and deteriorate performance. lastly, problems may also occur during relexicalisation as it does not take into account the context in which the placeholders are situated and may result in disfluent sentence. for a simplified example, observe how placing the unedited dates in the placeholders leads to disfluent output in figure 2 . segmentation strategies are commonly used in neural machine translation to improve the generalisation ability of models. the objective is to break down words into smaller units, reducing the vocabulary and the number of unseen tokens @xcite . unfortunately, applying segmentation in concept-to-text nlg, e.g. using byte-pair-encoding (bpe) subword units @xcite or using characters as basic units @xcite @xcite , underperforms against delexicalisation. challenges include capturing long dependencies between segmented words, and generating non-existing words. copy mechanism is another method to address unseen input, by allowing the decoder of an encoder-decoder model to draw a token directly from the input sequence instead of generating it from the decoder vocabulary @xcite . while applications of the copy mechanism in concept-to-text nlg have achieved overall good results @xcite @xcite , when dealing with rare and unseen inputs delexicalisation is still preferable @xcite . to improve the generalisation ability of copy mechanism models, roberti et al. (2019) propose applying the copy mechanism to character-level nlg systems. this is combined with an additional optimisation phase during training where the encoder and decoder are switched. ## language agnostic delexicalisation in order to address the shortcomings of previous approaches to generalise over rare or unseen inputs, especially in cases of multilingual output, we propose language agnostic delexicalisation (lad). figure 3 shows an overview of lad; the input and output are first delexicalised using pretrained language-independent embeddings, and (optionally) ordered. the multilingual generation model is trained on the delexicalised training data, and the output is relexicalised using automatic value post-editing to ensure that the values fit the context. each component is described in more detail bellow. to enable multilingual generation, we adapt the universal encoder-decoder framework via "target forcing" @xcite since it can be directly applied to any nlg model without the need to modify the latter's architecture. to do so, we extend the input mr in the encoder with a language token that signals which language the model should generate output in. in addition, we follow @xcite and initialise the decoder with the language token. the rest of the components (i.e. delexicalisation, ordering, and value post-editing) are orthogonal to the model's architecture. ## experiments for our experiments we use five datasets and calculate bleu-4 @xcite , meteor @xcite , chrf++ @xcite , and ter @xcite . the webnlg challenge 2017 @xcite data consists of sets of rdf triple pairs and corresponding english texts in 15 dbpedia categories. for our purposes, we will be using a later work @xcite that introduced a machine translated russian version of webnlg17, a part of which was post edited by humans. due to the limited amount of human corrected russian sentences, and to facilitate the most accurate evaluation, we use these solely for testing. to ensure that half of the domains in the new test set remain unseen during training, we create our own train/dev/test split by retaining the following dbpedia categories from training and development sets: astronaut, monument and university. the latest incarnation of the webnlg challenge @xcite is fully human annotated for both english and russian. we use this as the main dataset in our experiments, as it is designed to promote multilinguality. however, due to the fact that the provided test set does not contain unseen russian instances, we perform our experiment on a custom split (webnlg20*) ensuring that part of the domains in the test data remain unseen during training. the split was performed similarly to the previously described webnlg17. multiwoz 2.1 @xcite and cross-woz @xcite are datasets of dialogue acts and corresponding utterances in english and chinese respectively. the two datasets share the same structure, with multiwoz covering 7 domains and 25 attributes, and crosswoz covering 5 domains and 72 attributes; 4 of the domains are common in both datasets though crosswoz has more attached attributes. multilingual woz 2.0 @xcite is also a dialogue dataset with utterances available in three languages: english, italian and german. its scope is more limited than multiwoz and crosswoz as it only covers a single domain. for all models in our experiments, the input consists of a simple linearisation of the mrs. particularly, for the delexicalisation based models, the values are extended with their respective placehold-ers as shown in the following example: "entity 1 meyer werft location entity 2 germany". ## conclusion we proposed language agnostic delexicalisation, a novel delexicalisation framework that matches and delexicalises mr values in the text independently of the language. for relexicalisation, an automatic value post editing model adapts the values to their context. results show that multilingual models outperform monolingual models, and that lad outperforms previous work in improving the performance of multilingual models, especially in low resource conditions. lad also improves on the performance of pre-trained language models achieving state-of-the-art results. the automatic value post editing component is especially beneficial in morphologically rich languages.