Studying the Impact of Filling Information Gaps on the Output Quality of Neural Data-to-Text (2020)

Abstract: It is unfair to expect neural data-to-text to produce high quality output when there are gaps between system input data and information contained in the training text. Thomson et al. (2020) identify and narrow information gaps in Rotowire, a popular data-to-text dataset. In this paper, we describe a study which finds that a state-of-the-art neural data-to-text system produces higher quality output, according to the information extraction (IE) based metrics, when additional input data is carefully selected from this newly available source. It remains to be shown, however, whether IE metrics used in this study correlate well with humans in judging text quality.

## introduction the ecological validity @xcite of data-to-text tasks requires that tasks resemble, as closely as possible, real-world problems. only if this is the case can neural data-to-text solutions be operationally deployed with confidence. in the context of data-to-text, one of the issues with ecological validity is that most real-world tasks involve sizeable input data, with longer and more complex texts than are found in 'toy-sized' datasets. we must be able to see a path to a machine learning task which closely resembles a real-world scenario and allows us to investigate important research questions. we should aim to improve both the dataset and the task continuously. generating summaries of basketball games from tabled data with the rotowire dataset, as introduced by @xcite for the english language, moves us closer to an ecologically valid data-to-text task. the original rotowire dataset has been found to contain gaps between the information in the input data, and the information content of the training text. this makes the task unfair for evaluating neural data-to-text systems that are required to generate high quality output text, with hardly any factual errors. the sportsett:basketball dataset @xcite addresses these data issues by fixing information gaps in the input data, whilst maintaining the original human-authored texts. given that there is now at least an order of magnitude more data per game, we should consider which subset of data to train the system on, and what if any pre-processing should be preformed. we added some of this newly available data to an existing state-of-the-art neural data-to-text system @xcite and found improvement across a range of metrics. we used this system since, at the time of writing, it is one of the most recent, best performing and easiest to configure. we also discuss here which types of data could be added in the future, as well as some difficulties that may be encountered in doing so. ## related work many systems have been designed and evaluated using the rotowire dataset @xcite @xcite @xcite . most of these works focus on adjusting the architecture of the system whilst using similar input data. @xcite is a notable exception, as their architecture change allows box score and other data from previous games in their input. this was, however, still data from the original rotowire dataset. some works have attempted to better align data to text in other datasets, with techniques such as semantic control @xcite . for rotowire, this has been investigated by @xcite , which aims to prevent generation of sentences not grounded in the data. with this approach, some of the most common sentence types from the human narrative, such as the subsequent opponents of each team, are not generated. this difference is crucial when determining ecological validity of the task. the aim is to replicate the human author in the act of writing a summary of the basketball game, including its full narrative structure. ## information gaps in data-to-text shown in figure 1 is an example textual summary from the rotowire dataset. an example partial box score is shown in we performed a machine-assisted corpus analysis, using the spacy syntactic parser @xcite to group sentences which only differ by entity. we do this using an abstraction process where we replace named entities with special tokens comprised of their part-of-speech and entity label. some manual rules are added to the parser to handle domain specific syntax. by this process the sentence (s01 from figure 1 ) 'the atlanta hawks (41-9) beat the washington wizards (31-19) 105-96 on wednesday.', is transformed to 'propn-org (x-y) beat the propn-org (x-y) x-y on noun-date.'. we then count and read these abstract sentence types to find statements common to the narrative, but with attribute types not present in the data. for example, sentences with the same abstract form as s1 occur 26 times in the training corpus, with more than 800 further sentences of a similar form (using defeat instead of beat, or also including the location/stadium). it is the most common type of thing to say in the opening sentence of these summaries (which teams played, when, and where). there are, however, important attributes in these sentences which are not provided by the original rotowire data. when generating game summaries, systems will often hallucinate these attributes as they deem it probable that such language is included in the summary, but the attribute is not available to the copy attention mechanism. in the case of our above example, the day of the week is not available in the data. the stadium in which the game was played, as well as the city and/or state within which the stadium stands, are also not available despite being common in variants of this opening sentence. ## evaluation by information extraction the information extraction (ie) metrics of relation generation (rg), content selection (cs), and content ordering (co), have been used extensively to compare systems operating on the rotowire based datasets. these metrics are also key to the system design philosophy of @xcite which aims to "[exploit] the fact that post-hoc information extraction is significantly easier than generation itself." all of these metrics are based on ie models which learn to link statements in the text to tuples in the data. these models are trained on a corpus which has been automatically annotated using rules. names or numbers in the text are linked to the possible data tuples which could represent them. during evaluation, tuples are predicted for the test text summaries, with a name, value and a type, e.g. 'atlanta-96-team-pts'. these predicted tuples can be compared with the tuples in the data in order to determine whether facts predicted from the text match those in the data. for example, if the fact extracted from the text is 'atlanta-96-team-pts', but in the data we see 'atlanta-105-team-pts', then based on the match of both name and type, we can determine that the number is wrong. the cs metrics use these tuples to measure how many of the tuples from the predicted text exist in the gold standard text. the co metric measures the order of these tuples. for more details see @xcite (we have purposefully cited the arxiv version of this paper as it includes an additional appendix detailing the procedure). ## results ## conclusion and future work our results show that identifying data which should be included, then modelling it within the system architecture, increased all information extraction based metrics. existing metrics have only been evaluated in limited ways for this domain. improved metrics could help us evaluate systems, as well as find and categorise information gaps. the subset of data selected for input, the form it takes (atomic versus aggregated), as well as the inclusion of system components/techniques (copy attention mechanism, hierarchical encoder, separate document plan, fact grounding, etc.), are all variables which could affect system performance. we plan in future work to perform ablation studies to determine which such variables, and in which combination, produce the best results. as part of this, we aim to create a unified code-base which will allow for components to be selected and configured in combination, for as many different data forms as possible. beyond this, we hope to move away from end-toend system designs. this is similar in spirit to the idea proposed in @xcite , where a single model is not attempting to learn everything, the document plan is learned separately. we would extend such ideas to the data itself, if we can use both the known ontology from the structured data, as well as relationships and other information extracted with nlu or other tools, then this additional information could be input to systems which realize the language, meaning they are not left to solve both data analytic and language problems with a single model. if we can define our data operations in terms of standard data models, such as relational models, then we will be closer to a general approach for filling the information gap in data-to-text.