Enriching the E 2 E dataset (2021)

Abstract: This study introduces an enriched version of the E2E dataset, one of the most popular language resources for data-to-text NLG. We extract intermediate representations for popular pipeline tasks such as discourse ordering, text structuring, lexicalization and referring expression generation, enabling researchers to rapidly develop and evaluate their data-to-text pipeline systems. The intermediate representations are extracted by aligning non-linguistic and text representations through a process called delexicalization, which consists in replacing input referring expressions to entities/attributes with placeholders. The enriched dataset is publicly available.

## introduction data-to-text nlg is the computational task which aims to generate text from non-linguistic data @xcite . applications of this task have become increasingly common, such as rdf-to-text @xcite , @xcite , dialogue response generation @xcite , robot-journalism @xcite , etc. the growth of the field can be partially explained by increasing availability of focused data-to-text resources, such as webnlg @xcite , e2e @xcite , rotowire @xcite , genwiki @xcite and kelm @xcite . as with other automatic text generation fields, such as machine translation, significant advances in deep learning @xcite , along with an increasing number of data-totext resources, have resulted in upsurge in neural end-to-end applications targeted towards data-totext nlgk @xcite . hence, 1 https://github.com/thiagocf05/ enrichede2e although the current data-resources have benefited the development of end-to-end neural models, the same can not be said for pipeline systems, since these resources usually only consist of raw meaning representations and their final verbalizations. aiming to decrease data sparsity and make data-to-text models more generalizable and generate more adequate texts, many approaches aim to extract alignments between the non-linguistic and text representations, and then use these alignments to build explicit intermediate representations for a more controllable generation process @xcite . as examples, all the data-driven participating models of the e2e work by first converting the meaning representation into an intermediate template which is later realized into the final text. this is also the case in the webnlg challenge, which makes use of the eponymous dataset. in order to make it easy for researchers to rapidly develop and evaluate data-to-text pipeline systems, castro ferreira et al. (2018b) enriched the webnlg corpus, one of the most popular data-totext resources. the study extracts intermediate representations for popular pipeline tasks such as discourse ordering, text structuring, lexicalization and referring expression generation. intermediate representations are automatically extracted by aligning the non-linguistic and text representations through a process called delexicalization, which consists of replacing in the texts referring expressions to input entities/attributes with placeholders. the same extraction process with respect intermediate representations above is applied to the recent cacapo dataset, which is both multilingual (dutch and english) and multi-domain, containing up to 10,000 sentences (van der lee et al., 2020). highly inspired by the work of castro ferreira et al. (2018b) and van der lee et al. ( 2020 ), our study aims to delexicalize and provide pipeline intermediate representations for another very popular data-to-text dataset: the e2e dataset. we believe that the enriched version of the e2e will provide another data-resource so researchers can better investigate data-driven pipeline systems, their subtasks as well as its comparison with state-of-the-art end-to-end systems. ## the e2e dataset the e2e dataset is a resource initially constructed for training end-to-end, data-to-text applications in the restaurant domain. it consists of 50,602 english verbalizations to 5,751 dialog-act-based meaning representations @xcite . the dataset is split into training, validation and test sets in a ratio of 76.5%, 8.5% and 15%, respectively. an example of a pair between a meaning representation (top) and its corresponding text (middle) is depicted in figure 1 . each meaning representation consists of 3-8 attribute-value pairs, picked from a list of 8 attributes depicted in although the crowd-workers were asked to verbalize all the information contained in the meaning representation, the creators of the corpus decided to do not penalize those who skipped some information. for this reason, the corpus may also be used to study experiments for the content selection task of pipeline data-to-text systems. the e2e dataset differs from the webnlg corpus, which focused on semantic variation, as it leverages higher lexical and syntactical variations, having an average of 8.1 reference verbalizations per meaning representation. the corpus is also bigger than other similar corpora such as sfrest @xcite , a corpus in the domain of hotels and restaurants with 5,192 verbalizations to 1,950 meaning representations; and bagel @xcite , with 404 texts verbalizing 202 meaning representations. ## delexicalization following the method used by castro @xcite in the webnlg corpus, we aimed to decrease the data sparsity of the corpus and to align a meaning representation with its corresponding text by delexicalizing the texts. the delexicalization process works by replacing the referring expressions to the values of the attributes for placeholders representing the attributes. figure 1 shows an example of a meaning representation, the final verbalization and its delexicalized version (bottom). the process was conducted differently for training and validation/test parts of the corpus as explained in the following sections. ## explicit intermediate representations based on the alignments between the meaning representation and the text provided by the delexi-calization process, we can extract several explicit intermediate representations that can help to study several generation phenomenon as well as to build traditional pipeline (rule based or data driven) datato-text systems, which may generate more adequate texts and to generalize better for new domains @xcite . @xcite , we have enriched the e2e dataset with several intermediate representations about content selection, discourse ordering, text structuring, lexicalization and referring expression generation. these intermediate representations could be used to study each phenomenon as well as to develop a data-driven, pipeline data-to-text system as envisaged by @xcite . content selection is the task of deciding which information should be verbalized. by comparing the attributes contained in a meaning representation and the presence or absence of their placeholders in the delexicalized template, we are able to automatically extract all the input content for a given verbalisation. in the example of figure 1 for instance, we can see that the placeholder of the attribute eattype (e.g. eattype ) is not present in the delexicalized template, indicating that it was not selected to be verbalized in the text. discourse ordering is the task of sorting the selected content in the order it should be verbalized. by looking at the order of the placeholders in the delexicalized text, we can extract this order. in figure 1 , looking at the order of the placeholders in the delexicalized template, we see that the sorted list of attributes is: near, area, name, food, pricerange and familyfriendly, text structuring is the task within pipeline datato-text systems responsible for structuring the outputs of content selection and discourse ordering into paragraphs and sentences. using stanza @xcite , we tokenized the sentences of each delexicalized template and considering their placeholders, extracted the sentence plan for each one the attributes verbalized. in figure 1 for instance, near, area, name were verbalized in the first sentence, whereas food, pricerange and familyfriendly in the second. lexicalization aims to find the proper phrases and words to express the content to be included in each sentence. to obtain lexicalization templates similar to the ones used for the neural pipeline system of castro @xcite , we again used stanza in the delexicalized templates to lemmatize determiners and verbs and extract their correct morphological inflection information. then in these templates, determiners and verbs were replaced by their morphological inflection information and lemmas. for instance, for the delexicalized template depicted in figure 1 , the lexicalization template would be: near near in area vp[@xmath0, @xmath1, @xmath2, @xmath3, verb-@xmath4. name vp[@xmath5, @xmath6, @xmath7, @xmath8, @xmath9, @xmath10, vp[@xmath11, @xmath12, @xmath13, @xmath14, @xmath15, @xmath16, @xmath17, @xmath18, verb-@xmath19. referring expression generation is the task responsible for generating the references to the entities present in the text @xcite . in our case, these entities are the attributes of the meaning representation. following castro ferreira et al. (2018b), we extract the referring expression to the attributes by overlapping an original text and its delexicalized version. in figure 1 , contains examples of extracted references such as the wrestlers and it for the name attribute value the wrestlers in the meaning representation. surface realization is responsible for taking the last decisions to convert a non-linguistic data into text. in this case, the correct morphological realisation of determiners and verbs as well as detokenizing the text. for this step in specific, we did not extract any kind of information, but refer to the extensive literature that exists on morphological inflection @xcite . these tools can be used to correctly realize our extracted lexicalization templates. moreover, detokenization is a task already solved with high accuracy. ## conclusion this work introduces the enriched version of the e2e dataset @xcite . together with the enriched version of webnlg @xcite and ca-capo van der @xcite , this resource will help researchers to carefully investigate particular pipeline processes in data-to-text systems in the levels of macro-(e.g., content selection, discourse ordering and text structuring), microplanning (e.g., lexicalization, aggregation and referring expression generation) and surface realization. in particular, we will be able to better analyse how such subtasks could obtain better performance when developed using a rule-based approach or a specific/multitask data-driven system. moreover, in future work the community will be able to better compare pipeline and end-to-end data-to-text systems in terms of generalization as well as fluency and adequacy of their generated texts.