Enhancing factualness and controllability of Data-to-Text Generation via data Views and constraints (2023)

Abstract: Neural data-to-text systems lack the control and factual accuracy required to generate useful and insightful summaries of multidimensional data. We propose a solution in the form of data views, where each view describes an entity and its attributes along specific dimensions. A sequence of views can then be used as a high-level schema for document planning, with the neural model handling the complexities of micro-planning and surface realization. We show that our view-based system retains factual accuracy while offering high-level control of output that can be tailored based on user preference or other norms within the domain.

## introduction the original vision of data-to-text generation was to take complex data and describe it using natural language such that humans could better understand it @xcite . neural data-to-text systems commonly transcribe an isolated, simple data structure to a natural language. however, in many domains, e.g., finance, cinema box-office, or weather, the structured-data to be described are not independent, but rather exist as points along multi-dimensional axes such as time or entities (like people, companies, or locations). figure 1 shows an example of a human-authored basketball game summary that requires data from outwith an individual game record. such summaries often mention the upcoming games for each team (e.g. last two sentences) and aggregate player statistics over prior games. commercial applications deal with this complex scenario using hand-crafted rules @xcite @xcite , although recent research leveraging deep learning techniques has looked to alleviate this bur-den. however, such systems remain largely endto-end, offering little user specified control foot_0 . on small sets of triples or attributes @xcite , neural systems can produce fluent, in domain generations, although they can struggle with factual accuracy mistakes such as hallucination. additional challenges are introduced when using complex data such as tables @xcite , compounded when considering interlinked structured data and longer texts @xcite . while challenging, this complex setting is also an opportunity. in this paper, we propose the novel concept of data views, where all available data is divided into manageable subsets, describing entities and their attributes along one or more dimensions. views are then aligned with corresponding spans of text. they can then be combined to form high level document plans (schema) for neural data-to-text systems, controlling the generation of text without sacrificing factual accuracy. we investigate the use of views in the domain of automated journalism (english language basketball game summaries), using the sportsett dataset @xcite which extends the ro-towire dataset @xcite . ## related work factualness and controllability are critical issues for data-to-text systems. studies have shown that for end users of systems, accuracy is more of a concern than readability @xcite and users prefer texts tailored to their needs (van der @xcite . recently, the research community has focused on neural approaches, aiming to solve data-to-text tasks by leveraging advances of deep learning in language modeling fields (in particular neural machine translation) @xcite @@xmath0, 118-102, at american airlines arena on friday. <within-@xmath1, they won this game in dominating fashion. <within-@xmath2, a 31-24 first quarter really set the tone, with a 41-29 second quarter sealing the victory. <within-@xmath3. <whole-@xmath4, with oklahoma city hitting 16-of-30 and miami connecting on 11-of-36. <whole-@xmath5, winning the fastbreak differential, 23-8. <whole-@xmath6, as he tallied 43 points, seven rebounds, five assists and two steals. <whole-@xmath7, 12 rebounds and 14 assists. <whole-@xmath8, seven rebounds, two assists and three steals. <whole-@xmath9, as he provided 28 points on 11-of-13 from the field. <whole-@xmath10, as he provided 21 points, seven rebounds and two assists off the bench. <whole-@xmath11, four rebounds, three assists and two steals. <whole-@xmath12. <whole-@xmath13, two rebounds and five assists. <between-@xmath14, as they travel to face the boston celtics. <between-@xmath15, they play host to the struggling indiana pacers on saturday for their next outing. id team name pts reb wins losses . player between-gameviews (partial). et al. @xcite . neural systems can blur the distinction between each sub-task of the pipeline approach, and are able to learn end-to-end to generate in-domain text from structured data @xcite @xcite . our work draws on a wide body of prior research on controllability and factualness, namely data engineering, controllable text generation, as well as planning. data engineering it is increasingly clear that careful design of datasets used to train deep neural models matters significantly @xcite . on simpler data-to-text tasks such as the e2e challenge @xcite , a number of data-level techniques have been proposed to improve factual accuracy and controllability, including dataset curation @xcite , data-to-text alignment @xcite , straightforward control via input manipulation @xcite , and fine-grained annotation @xcite . however, these techniques are not suited for the complexity of the problem at hand, and do not enable interactability. on complex datasets, adding extra records to increase the coverage of the source data over the description @xcite has shown encouraging results, but not all cases of hallucination are clear-cut and easy to solve with a few records. @xcite suggested that it might be possible to model tables as three-dimensional, with rows, column, and time as the dimensions. this is not satisfactory because (1) current neural models cannot correctly perform the arithmetics required to generate the types of utterances found in the reference texts @xcite ; (2) adding dimensions leads to intractable complexity. controllable text generation controllable text generation (ctg) techniques traditionally involve conditioning an nlg system on several control factors of style (e.g. tone, tense, length, etc.) @xcite @xcite , or content (e.g. customized summaries based on aspect queries @xcite ). control factors are often framed as a collection of key-value pairs, similarly to a typical data-to-text setting. in the biography domain @xcite , filippova (2020) explicitly introduced ctg to data-to-text via an hallucination score simply attached as an additional attribute which reflects the amount of divergence in the target reference. prompting (i.e. starting generation from textual instructions @xcite ) can also provide some control, with (li and liang, 2021) obtaining encouraging results at managing the length of generated descriptions. contrasting with documentlevel approaches, @xcite propose a finer-grained controllability, via word-level attributes to learn the relevant parts of each training instance. planning and schema macro-planning, i.e. high-level planning of ideas, has long been used in traditional nlg pipelines, and has recently been introduced to neural systems as well. on smallscale datasets with short inputs/outputs (e.g. the webnlg corpus), these approaches rely on detailed annotations of sentence structure and mention placement @xcite , or the strong assumption that descriptions describe the associated data entirely (and nothing else) @xcite . however, these are unreasonable dependencies for large-scale datasets, with prohibitive size and complexity of inputs and outputs. on more complex tasks, the two-step neural approach of @xcite has proven effective at reducing factual mistakes and provides a small degree of controllability. given all possible combinations of entities, a planner first selects which will be part of the narrative. in a second step, a generative module learns to output descriptions based on the selected entities. designed this way, the planning step scales poorly in the input size, since it needs to consider all possible combinations. furthermore, while the plans can be edited, the impact of individual edits on the final output is unclear (since the model is an end-to-end encoderdecoder). additionally, no restrictions are placed during decoding to ensure that (1) the decoding process follows the order of the plan; (2) the decoder's copy mechanism doesn't copy attributes from entities in other part of the plans. lastly, no restrictions are placed on the length of the texts corresponding to each part of the plan, the decoder has to decide the number of sentences to attribute to each item. generated texts are of similar size, independent of the chosen plan. ## data views and their design in multidimensional settings, descriptions of a data structure mostly focus on an initial point based upon the narrative intent, but also often compare subsets of data along different points of an axis. for example, the best player in the game being summarised might be described in an initial sentence: we make explicit this latent partitioning of the data, via views, as a solution to both the handling of dimensional data, and the alignment of data to text. we split the associated data following the same partitioning, and a view is defined as the records (i.e., key-value pairs) for one entity, from within one partition. in cases where several views are aligned with the same span of text, e.g. a sentence comparing two entities, views can be combined to form view sets. figures 1 and 2 illustrate this mirrored multidimensionality, along the time axis. the opening sentence describes the basketball game in focus, with details of the teams and their respective scores. this team data can be seen in the left (team) <whole-@xmath16. in contrast, the final two sentences report the upcoming opponents for both teams. this data, which lies elsewhere on the axis, is shown in the @xmath17. the third sentence mentions team statistics within quarters of the game, this data is shown in the @xmath18. ## experimental setup data we use the sports journalism (basketball) dataset sportsett @xcite , based on the rotowire datset introduced by @xcite . it consists of game statistics paired with human-authored descriptions. the original dataset contained train-test corruption so we partition by season foot_3 to provide a suitable proxy of a real-world task. ## human evaluation of factual accuracy with the gold standard protocol of @xcite , human annotators highlight nonoverlapping spans of text that contain errors, also selecting an error category. participants are provided with links to basketball-reference.com, which is at least as complete as the system input data, and much easier for humans to navigate. four main error categories are used; name (named the oklahoma city thunder (33-18) defeated the miami heat (24-26) 118-102 on friday. the thunder shot 48 percent from the field and 53 percent from three-point range. they also dominated the rebounding battle, 50-47. the heat n also held the heat to 45 percent shooting and outrebounded them 50-38 u . paul george led the way for the thunder with 43 points, seven rebounds, five assists and two steals. dennis schroder added 28 points of his own off the bench. russell westbrook finished with 14 points, 12 u assists and 12 rebounds. steven adams scored 13 points and grabbed seven rebounds in 27 minutes. jerami grant chipped in eight points, while jerami grant w added eight. patrick patterson led the bench w with five points and two rebounds. kelly olynyk was the only other w heat player c in double figures, finishing with 21 points. josh richardson scored 18 points on 7-of-18 shooting. hassan whiteside had 12 points and 16 rebounds in 19 minutes. bam adebayo, it wasn't enough to overcome the heat w . oklahoma city will travel to boston to take on the celtics on sunday. miami plays host to the pacers, also on saturday. entities), number (ordinal, cardinal, etc), word (a word or phrase that is not a name or number), context (such as implicature errors). there is also the last resort category other, for text that is nonsensical. finally, there is a not checkable category, which covers facts that are impractical or impossible to check using the provided reference data. we performed our experiment with one annotator per text 6 . the original protocol limited annotators to only 4 prior games before defaulting to not checkable errors. we felt this was overly restrictive so we asked annotators to check all assertions within the current season. we compared generations for 35 random basketball games, generating a text for each of the 7 systems. a latin square design was used whereby each participant annotated 5 texts for each system, never seeing the same game twice (245 total annotated texts). ## conclusion increasing concern has been raised regarding the quality of both task setup @xcite , as well as evaluation in nlp/nlg, with caveats of systems and experimental results often going unreported @xcite . we contribute to the meaningful progress of both. in exploring an alternative task where generation of an exact human reference text is not the goal (it is just part of the available information), we bring the generation process more inline with a real-world problem where control is a major requirement. we expanded the data-to-text task by considering the mirrored multi-dimensional aspects of both data and text. we have shown that by splitting this extended data into manageable views based on its dimensionality, meaningful control can be introduced over system output without sacrificing factual accuracy. control comes in the form of ordering views using schema, in the way messages might be ordered in rule-based systems, but allowing the neural model to handle the complexities of micro-planning and surface realization. the method of splitting data into views could be applied to complex data from sources such as relational databases or multi-dimensional arrays. our implementation of views is only one possible way to structure the data. we intuit that views that describe a named entity and a set of direct attributes, in any dimension, work well. considering the multidimensionality of data and text brings the problem closer to that encountered and addressed by humans in the real world. enhancing the structure of datasets, as well as designing models that leverage this multidimensionality, will move systems closer to the goal of human-like descriptions of complex data.