W eb NLG Challenge 2020: Language Agnostic Delexicalisation for Multilingual RDF -to-text generation (2020)

Abstract: This paper presents our submission to the WebNLG Challenge 2020 for the English and Russian RDF-to-text generation tasks. Our first of three submissions is based on Language Agnostic Delexicalisation, a novel delexicalisation method that match values in the input to their occurrences in the corresponding text through comparison of pretrained multilingual embeddings, and employs a character-level post-editing model to inflect words in their correct form during relexicalisation. Our second submission forfeits delexicalisation and uses SentencePiece subwords as basic units. Our third submission combines the previous two by alternating between the output of the delexicalisation-based system when the input contains unseen entities and/or properties and the output of the SentencePiece-based system when the input is seen during training.

## introduction recently, neural approaches to language generation have become predominant in various tasks such as concept-to-text natural language generation (nlg), summarisation, and machine translation thanks to their ability to achieve state-of-the-art performance through end-to-end training @xcite @xcite . specifically in machine translation, deep learning models have proven easy to adapt to multilingual output @xcite and have been demonstated to successfully transfer knowledge between languages, benefiting both the low and high resource languages @xcite . in the concept-to-text nlg task, the language generation model has to produce a text that is an accurate realisation of the abstract semantic information given in the input @xcite . it is common practice to perform a delexicalisation @xcite of the mr, in order to facilitate the nlg model's generalization to rare and unseen input; lack of generalisation is a main drawback of neural models @xcite but is particularly prominent in concept-to-text. delexicalisation is a two-step process that starts with a preprocessing step where all occurrences of mr values in the text are replaced with placeholders. this way the model focuses on learning to generate text that is abstracted away from the actual values. in a post-processing step, known as relexicalisation, the placeholders are re-filled with the mr values. delexicalization does not need to be contained to solely the mr values; in the surface realization task, full delexicalization of the input structures has also been explored to great effect @xcite . the main shortcoming of delexicalisation is that its efficacy is bound by the number of values that are correctly identified. in fact, a naive implementation of delexicalisation requires the values provided by the mr to appear verbatim in the text, which is often not the case. this shortcoming is more prominent when expanding context-to-text to the multilingual setting, as mr values in the target language are often only partially provided. additionally, mr values are usually in their base form, which makes it harder to find them verbatim in text of morphologically rich languages. finally, relexicalisation also remains a naive process that does not consider how the context should effect the morphology of the mr value when it is added to the text and vise versa @xcite . for our submissions to the webnlg challange 2020 rdf-to-text generation tasks, we trained two multilingual neural nlg models that use differently-preprocessed data. our first submitted system is based on language agnostic delexicalisation (lad), a novel delexicalisation method that aims to identify and delexicalise values in the text independently of the language. lad expands over previous delexicalisation methods and maps input values to the most similar n-grams in the text, by focusing on semantic similarity, instead of lexical similarity, over a language independent embedding space. this is achieved by relying on pretrained multilingual embeddings, e.g. laser @xcite . in addition, when relexicalising the placeholders, the values are processed with a character-level post editing model that modifies them to fit their context. our second submission does not employ delexicalisation but makes use of an additional segmentation step that breaks down words into subword units. we anticipate lad to perform better for inputs that were not seen during training (due to it abstracting away from specific values) and the subword-based model to perform better for seen inputs. to combine the two models, we submitted a third system that simply selects the output of the lad system when the input contains unseen entities and/or properties or the output of the subword-based system when the input is seen. ## language agnostic delexicalisation in this section, we will describe the language agnostic delexicalisation (lad) framework em-ployed by our first submitted system. figure 1 shows an overview of lad; the input and output are first delexicalised using pretrained languageindependent embeddings, and ordered. the multilingual generation model is trained on the delexicalised data, and the output is relexicalised using automatic value post-editing to ensure that the values fit the context. each component is described in more detail bellow. ## data and preprocessing in the webnlg challenge rdf-to-text task, the goal is to generate text that describes particular entities and their properties as they were extracted from knowledge bases, and as such the input mr consists of a set of rdf (resource description format) triples, each in the form @xmath0, property, object>. our submitted systems are trained solely on the data provided by the organiser of the challenge. the mrs are formed by concatenating the triples in the rdfs sequentially. tokenisation and truecasing is performed with the scripts provided in the moses toolkit @xcite . to enable multilingual generation, we adapt the universal encoderdecoder framework via "target forcing" @xcite since it can be directly applied to any nlg model without the need to modify the latter's architecture. to do so, we extend the input mr in the encoder with a language @xmath1. in addition, we follow @xcite and initialise the decoder with the language token as well. for the lad-based system, both mr and output text are delexicalised. during training, the input mr is ordered by imitating the order in which the triple's objects appear in the corresponding text (according to lad-based matching). during testing, we follow @xcite and perform graph-based ordering. the values in the input are delexicalized and indicated with a property-related placeholder followed by a numbered general placeholder. in the output text, the values are replaced solely with the numbered general placeholders. examples of inputs and outputs are shown in ## configurations the multilingual nlg and vape use a transformer as underlying architecture. we use the fairseq toolkit for our systems @xcite the vape overprocessing similarity check threshold was set to 0.95. ## evaluation we observe that overall for english lad performs better than the subword model sp. both lad and sp performed comparably on instances seen at training time. however, the latter underperforms when dealing with unseen entities and even worse with unseen categories, i.e. domains not seen during training that often contain unseen properties in addition to unseen entities. surprisingly, the combination system lad+sp performs better than sp in seen inputs and better than lad in unseen categories. this implies some inconsistency between how lad+sp distinguishes seen from unseen inputs and how they were divided for the shared task. on the russian test set, the subword system outperformed lad considerably across all the evaluation metrics. we note however that the russian test set contained practically no unseen properties nor unseen entities. this is an unfavourable setting for lad as it is specifically designed to address generalisation to unseen input. some properties do appear inconsistently in testing (e.g. spelled differently from how they occurred in training) which explains why the combination system lad+sp performs worse than single sp. by directly observing the output, we note that lad is less fluent than the non-delexicalisation model, as the generated context is not always consistent with the values. however, its main advantage is that it avoids under-and over-generating values as they are being controlled by the placeholders. in fact, while sp often appears as more fluent than lad, it tends to under-generate and miss values, especially for longer inputs. ## conclusion we presented our submission to the webnlg challenge 2020 english and russian rdf-to-text generation tasks. for both tasks we employ multilingual transformer-based seq2seq nlg systems that differ, however, in the type of data processing. our first system adopts lad, a delexicalisation technique that relies on multilingual embeddings for delexicalisation and uses post-editing to adapt the values to the text during relexicalisation. our second system instead employs a sentencepiece model to perform word segmentation. in addition, we submitted a third system that combines the previous two by selecting the lad output when the input contains an unseen property or entity, and the sp output otherwise. the shared task's automatic evaluation shows that overall lad perform better than sp in english as it is more robust in dealing with unseen cases. however, this is not demonstrated in the russian results as the test set contains only instances that have been seen during training.