Content Type Profiling of Data-to-Text Generation Datasets (2022)

Abstract: Data-to-Text Generation (D2T) problems can be considered as a stream of time-stamped events with a text summary being produced for each. The problem becomes more challenging when event summaries contain complex insights derived from multiple records either within an event, or across several events from the event stream. It is important to understand the different types of content present in the summary to help us better define the system requirements so that we can build better systems. In this paper, we propose a novel typology of content types, that we use to classify the contents of event summaries. Using the typology, a profile of a dataset is generated as the distribution of the aggregated content types which captures the specific characteristics of the dataset and gives a measure of the complexity present in the problem. Extensive experimentation on different D2T datasets is performed and these demonstrate that neural systems struggle in generating contents of complex types.

## introduction an ecologically valid task requires the automated systems to resemble the real-world scenario as closely as possible in its output @xcite . accordingly, a data-to-text generation (d2t) system needs to convey important insights extracted from the data in the textual summaries @xcite . most d2t problems can be seen as a stream of time-stamped events with a textual summary of each event presenting the insights. an event is the time-period of interest for which the textual summary is written. for example, in sports reporting -a game played between two teams can be an event; whereas in weather forecasting -the time period and location for which the forecast is written can be considered one full event. the summaries can contain different types of facts with information sometimes coming the bucks (10-10) handled the heat (9-9) 109-85 on friday night in milwaukee. it was the second victory over miami for the bucks this season after emerging victorious in miami 91-84 on nov. 16. milwaukee fell behind early but clawed back into the game in the second quarter and held a four-point advantage at half. the bucks were led by an unlikely face in kendall marshall, who scored a season-high 20 points (7-8 fg, 4-5 3pt) in 24 minutes. 109 ) can be directly copied from input data, while the ones in "italics" needs to be derived from multiple records such as the fact -kendall marshall leading the bucks team. finally, the information in "bold & italics" such as 'this was miami's second victory against bucks' are derived using records from multiple events in the stream. from multiple events in the stream. in most cases, the facts in an event summary are verbatim of input records. other times, these facts are derived from multiple records of either the same or multiple events in the stream. as an example, we show a part of baseball summary with multiple types of content in figure 1 . a distribution of different content types in the summaries of a dataset can be used to generate its profile that can capture the specific characteristics of the dataset and provide a measure of complexity present in the problem. the dataset profile can help us better define the d2t system requirements, such as: the type of system to build -is a complex domain-specific system required or can a general system be effective; or any information gap @xcite that needs to be bridged at the data level. generally a problem's complexity is identified with the evaluation of systems built for the task. there are several methods to evaluate the d2t systems, mostly by measuring the factual accuracy of generated texts @xcite @xcite or lexical similarity of generated texts with reference texts @xcite @xcite . these, whilst being promising are reactive measures where a full cycle of system development is needed to evaluate both the dataset and d2t system together. whereas dataset profiling can be a proactive measure to gain important insights about the task before even starting system development. there are other utilities of dataset profiling method as well, most notably, it can be used as a measure of dataset's complexity in datasheets for dataset @xcite . there are other type of datasets in d2t as well such as e2e @xcite or webnlg @xcite that do not follow the event based time-series setting. these datasets mostly focus on improving general ability of generation models such as transcribing a set of records, possibly in a domain-agonist setting. in this work, we do not address such datasets and rather focus on those which follow a time-series structure, where summaries may contain facts derived from across several event records, and also may require content selection on input data. in this paper, we propose a typology of different content types in d2t summaries based on the source of their information in the event stream. our key contributions are as follows 1 : the rest of the paper is organised as follows: in section 2 we formally define the d2t task and discuss our proposed content type typology. we then go on to describe our experimental set up in section 3, and discuss the results in section 4. some related works are discussed in section 5, before concluding the paper in section 6. 1 code, data, and results are at https://github.com/ ashishu007/content-type-profiling ## methodology the idea proposed in this paper is that we can create a profile capturing the specific characteristics of a dataset by looking at the distribution of content types present in its summaries. the first step towards achieving this is to formally define the main concepts (section 2.1). then, in second step, the content typology is defined by identifying the different types of facts that can be included in an event summary (section 2.2). ## experimental setup the experiments are performed in three phases. in first phase, the aim is to understand the characteristics of human authored summaries, for which, dataset profiles are generated based on human authored summaries using the proposed content type typology (section 4.1). the second phase aims to demonstrate the challenge state-of-the-art generation systems face in attempting to generate complex (inter-event and intra-event complex) content (section 4.2). this is evaluated by comparing the errors made by the generation systems for each content types. finally, the proposed methodology is used to understand the concept-drift issue in a problem domain which can help in building better systems capable of handling such domain-specific issues (section 4.3). ## results and discussions in this section, we use the content type classifiers build for different datasets using method described in section 2.2 for generating their profiles. the performance of best classifier for each dataset with its macro-f1 score and the number of samples used for training is shown in appendix a. as discussed in the previous section, the dataset profiles will be used for: first, analysing the humanauthored summaries from different datasets (section 4.1); second, analysing the system generated summaries from several state-of-the-art neural generative systems (section 4.2); and third, characterising the concept-drift issue in sportsett dataset (section 4.3). ## conclusion in this paper, we presented a typology of different content types in d2t summaries. the proposed typology is used to profile multiple datasets, which captures their characteristics and provide a measure of complexity present in the datasets. extensive experimentation is performed to demonstrate the challenge facing generation systems in producing complex types of content. we further use the profiling method to identify the concept drift problem in a dataset. through this work, we argue that a dataset's content type profile can help us define system requirements for building better generation systems. in future, we plan to employ insights gained from this work in building better d2t generation systems capable of producing accurate complex content for event summaries. this will require a system to be able to: understand the domainspecific rules, for deriving implicit information from the input data: and able to operate in huge search space, to select important content from vast possibility of across-event information. the dataset profiling method will help in identifying any information gap between input data and output summary and characterise the domain-specific rules in different categories based on their information source.