Mod- D 2 T : A Multi-layer Dataset for Modular Data-to-Text Generation (2023)

Abstract: Rule-based text generators lack the coverage and fluency of their neural counterparts, but have two big advantages over them: (i) they are entirely controllable and do not hallucinate; and (ii) they can fully explain how an output was generated from an input. In this paper we leverage these two advantages to create large and reliable synthetic datasets with multiple human-intelligible intermediate representations. We present the Modular Data-to-Text (Mod-D2T) Dataset which incorporates ten intermediate-level representations between input triple sets and output text; the mappings from one level to the next can broadly be interpreted as the traditional modular tasks of an NLG pipeline. We describe the Mod-D2T dataset, evaluate its quality via manual validation and discuss its applications and limitations. Data, code and documentation are available at https://github.com/mille-s/Mod-D2T.

## introduction multi-level linguistic representations are inherent to several linguistic formalisms, including lexical functional grammar @xcite and @xcite . they have been widely used in natural language understanding (nlu), e.g. in enhanced universal @xcite and prague dt @xcite , as well as in natural language generation (nlg), e.g. in enhanced webnlg @xcite , srst @xcite , and wikifluent @xcite . reference architectures have been proposed that define modules and/or levels of representation, with the first theoretical architecture probably dating back to the work of @xcite , while a widely accepted applied nlg architecture was described by @xcite . while end-to-end generators are very efficient @xcite , there is evidence that splitting the generation process into sub-steps can lead to improvements @xcite @xcite . however, corresponding datasets with multiple intermediate representational levels are scarce. in this paper, we present the modular data-to-text (mod-d2t) dataset, which comprises the inputs from the webnlg 2020 shared task data @xcite , paired with new output texts and ten intermediate-level representations that incrementally specify the output. the mappings from one level to the next can broadly be interpreted as the modular tasks of linguistic structuring, text planning/sentence aggregation, lexicalisation, communicative structure determination, deep sentence structuring, surface sentence structuring, surface aggregation, referring expression generation (reg), linearisation/morphology resolution and surface form generation. unlike existing multilevel datasets, which were created by adding annotated layers on top of existing text, we leverage the forge rule-based pipeline generator @xcite to produce multiple human-intelligible intermediate (semantic, syntactic, morphological) and final (text) representations starting from abstract structures. ## the mod-d2t dataset in this section, we describe the dataset and how it was built. ## qualitative evaluation we evaluated the quality of the intermediate representations by counting and classifying errors in predarg-th (figure 2 ) and ssynt-pro (figure 3 ) for 30 randomly selected inputs of the webnlg'20 dev split. these inputs contain 1 to 7 triples each, and are rendered as texts of up to 5 sentences. 4 predarg-th: out of the 30 semantic graphs, 20 (66.7%) were considered correct by a semantics expert (an author). the most common error was an overactive aggregation pattern that coordinated incompatible elements, typically a territory and one of its constituents, e.g., aarhus university is in aarhus and denmark. this problem was found in 8 structures (26.7%) and it resulted in sentences that were still grammatical and faithful, but lacked fluency. we also identified one case of unnecessary predicate 'be' ('be' was used for abraham a. ribicoff is american; 'american' itself being predicative, we could do without the copula in the predarg structure). the rest of the structures did not exhibit problems, but some representations could raise debate, such as (i) the use of "phantom agents", as in ait is affiliated with vtu, where the arguments are numbered a2 and a3, supposing an a1 that would affiliate one with the other (6 cases in 4 structures), and (ii) the choice of some edge labelling, e.g. for hypermarcas is in the pharmaceuticals sector, which is represented with a location relation between 'hypermarcas' and 'sector'; a more solid representation would treat 'sector' as a predicate with 'hypermarcas' as its argument. ssynt-pro: out of the 30 syntactic structures, 28 (93%) were considered correct given the predarg-th representation by a syntax expert (another author). the two problems found were one case of superfluous determiner the next to a genitive complement (which would produce the baku's memorial in the final text), 5 and one case of an underspecified dependency dep where adv would have been more appropriate. text: a previous human evaluation of the text quality of the forge generator that we use was provided by castro ferreira et al. (2020). in the "seen" scenario, they reported the following raw scores (out of 100): data coverage: 95.3 (humanwritten text: 95.5), @xmath0.6 (94.1) , correctness: 93.6 (93.4), text structure: 87.0 (91.2), fluency: 82.7 (88.1). 2019 also replaced the mentions of subject and object values with placeholders and lemmatised verbs in texts, allowing for learning both lexicalisation as a whole and reg. @xcite split wikipedia paragraphs and rephrase splits into autonomous minimal sentences, replacing pronouns by their referent. they thus end up with two layers used to train a sentence aggregation module and a paragraph compression module, which includes reg. they then apply their approach to the webnlg dataset by crafting minimal sentence (as opposed to predarg in our case) templates that they instantiate with the input triples. @xcite propose one syntactic and one predicateargument levels using universal dependency annotations (de marneffe et al., 2021) as a source. ## related work and limitations our dataset differs from the previous work in that we do not use human-written texts, and that we provide richer linguistic structures, with multiple semantic, syntactic and morphological levels that are (to the extent of our knowledge) not currently available for triple-to-text generation. the main limitation of our approach is that since the texts are synthetic and produced by a deterministic generator, their variety and quality is limited by the knowledge encoded in the generator (in particular, they generally lack the naturalness of humanwritten texts), and they represent only a fraction of what is possible for a language to express. another (current) limitation is that mod-d2t only contains english; generating texts in other languages requires crafting lexical resources and retrieving the subject and object values in the target language, adding rules to cover language-specific phenomena, and adapting predarg templates; see @xcite . we are currently completing irish and french versions of mod-d2t and will report on the multilingual aspects in future publications. ## conclusions and future work with the mod-d2t dataset, we are making available a large amount of rich and reliable linguistic structures at several levels of representation for a sizeable set of d2t input/output pairs. these can be used, e.g., for experimenting with plug-andplay nlu and nlg, facilitating (human) language learning, or teaching linguistics. 6 the dataset construction process is flexible enough to allow the controlled production of a myriad of variants of the dataset in terms of verbalisation, sentence grouping/structuring, output simplicity/complexity, etc., simply by (de)activating optional modules (see in future work, in addition to the multilingual extension mentioned above, we will make the generation pipeline available for researchers to generate their own datasets, and provide mappings from our representations to standard representation schemes (e.g., surface-syntactic ud). simon mille, stamatia dasiopoulou, beatriz fisas, and leo wanner. 2019a. teaching forge to verbalize dbpedia properties in spanish. in proceedings of the 12th international conference on natural language generation, pages 473-483, tokyo, japan. association for computational linguistics. simon mille, stamatia dasiopoulou, and leo wanner. 2019b. a portable grammar-based nlg system for verbalization of structured data. in proceedings of the 34th acm/sigapp symposium on applied computing, pages 1054-1056. amit moryossef, yoav goldberg, and ido dagan. 2019. step-by-step: separating planning from realization in neural data-to-text generation. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 2267-2277, minneapolis, minnesota. association for computational linguistics. ratish puduppully and mirella lapata. 2021. data-totext generation with macro planning. transactions of the association for computational linguistics, 9:510-527. ehud reiter and robert dale. 1997. building applied natural language generation systems. natural language engineering, 3(1):57-87. sebastian schuster and christopher d. manning. 2016. enhanced english universal dependencies: an improved representation for natural language understanding tasks. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 2371-2378, portorož, slovenia. european language resources association (elra). aleksandr k. žolkovskij and igor a. mel'čuk. 1965. o vozmožnom metode i instrumentax semantičeskogo sinteza [method and instruments of semantic synthesis]. naučnotexničeskaja informacija, 5:23-28.