A Baseline Document Planning Method for Automated Journalism (2021)

Abstract: In this work, we present a method for content selection and document planning for automated news and report generation from structured statistical data such as that offered by the European Union’s statistical agency, EuroStat. The method is driven by the data and is highly topic-independent within the statistical dataset domain. As our approach is not based on machine learning, it is suitable for introducing news automation to the wide variety of domains where no training data is available. As such, it is suitable as a low-cost (in terms of implementation effort) baseline for document structuring prior to introduction of domain-specific knowledge.

## introduction automated generation of news texts from structured data -often referred to as 'automated journalism' @xcite @xcite or 'news automation' @xcite @xcite -is of great interest to various news producers. it is seen as a way of 'providing efficiency, increasing output and aiding in reallocating resources to pursue quality journalism' @xcite . while data-to-text nlg systems are still far from common especially among the smaller, regional news industry players, at least among the larger newsrooms the use of nlg approaches has clearly been established @xcite . while secrecy in the industry makes it difficult to establish the commercial reality as an outsider, the limited available evidence indicates that commercial automated journalism is mostly done using rule-based methods despite a surge of academic interest in increasingly complex neural methods for nlg (e.g. @xcite : interviews of news automation users indicate that the employed methods are mostly based on templates @xcite , as are the few open source code repositories of real-world news automation systems @xcite . indeed, some nlg industry experts believe that especially end-to-end neural models do not match customer needs at this time @xcite . contributing factors include a lack of control @xcite ; issues with hallucination of nongrounded output @xcite @xcite ; the difficulty in surgically correcting any issues identified in trained neural models beyond additional training; as well as the difficulty of establishing what the 'worst case' performance of a neural model is. in addition, we believe that that while neural nlg methods are theoretically highly transferable, the practical transferability of neural nlg solutions to many news domains is limited by a lack of training data. while newsrooms have extensive archives of news text, these are rarely associated with the matching data that is the 'input' for each piece of news text (e.g., @xcite . at the same time, the non-trainable methods for nlg, too, suffer from difficulties in transferability and reusability @xcite . in this work, we investigate document planning (selecting what content and in what order should appear in the document) for structured, statistical data-to-text nlg in the context of automated journalism targeting human journalists. we are not in search of a perfect method, but rather something that is relatively easy to implement as a subdomainindependent baseline and which can then be enhanced with domain-specific processing later-on. such a method would make it easier to introduce automated journalism solutions to completely new subdomains within the larger statistical data domain. when queried for insight into news structure, journalists and academics often recite the concept of the "(inverted) news pyramid", where the news article is structured so that the order in which information appears in the text reflects the journalist's belief about the importance of the piece of information @xcite . while the precise origin of the structure is not clear @xcite , it has become so prototypical that it is held selfevident in the journalistic trade literature: "every journalist knows how to write a traditional news text: start with the most important thing and continue until you have either said everything relevant or the space reserved for the story runs out" (sulopuisto, 2018, translated from finnish). a more rigorous analysis of the structures employed in 'hard' news is presented by @xcite , who argues that hard news articles have an 'orbital' structure consisting of a nucleus which represents the main point of the article and satellites that give context and additional information about the nucleus. @xcite assigns the role of the nucleus to the combination of the headline and the lead paragraph of the article, and describes the subsequent paragraphs as the satellites. @xcite identifies five possible relations between a satellite and the nucleus: elaboration, cause-and-effect, justification, contextualization and apprisal. @xcite , in turn, identify that the satellites can elaborate, reiterate, describe causes or consequences, contextualize or provide additional assessment. an important observation is that -as indicated by 'orbital' -these satellites are relatively freely reorderable without affecting readability or meaning. together, these two observations indicate that a good document plan for hard news (1) priorizes more newsworthy items and (2) contains some overarching theme (exemplified by the nucleus) so that the text as a whole is coherent, i.e. the satellites are in some way related to the nucleus. the relations identified by @xcite and @xcite are highly similar to those identified in the more general rhetorical structure theory (rst) @xcite , which uses similar nucleus-satellite terminology. however, whereas @xcite and @xcite analyze news text on the level of paragraphs, rst can be applied on a more fine-grained level to much shorter text spans. as rst shows that similar relations can be applied on a sub-paragraph level, we hypothesize that a reasonably approximation of a news article might be constructed by applying @xcite orbital theory also within paragraphs, by considering the first sentence of the paragraph a nucleus, and the others as satellites. importantly, we interpret the orbital theory of news structuring to suggest that -as the satellites are freely orderable -the actual type of relation is not as important for document planning as knowing that some relation exists between the satellite and the nucleus. we hypothesize that while identifying whether a specific (rst) relation exists between two arbitrary pieces of information requires domain knowledge, an approximation of whether two arbitrary pieces of information are related in some way could be obtained by inspecting their similarity in a domain-independent fashion. that is, we expect that a piece of information regarding the us health care funding in 2020 is more likely to be related in some way to a piece of information discussing the us health care funding in 2020 than to another piece of information discussing the health care funding in sweden in 1978. if a heuristic or similarity measure identifying such relations could be identified, it could be used together with some estimate of newsworthiness to construct paragraph and document plans that seek to maximize both the key aspects identified above: newsworthiness and the relatedness of the content. as noted in the introduction, there is a distinction between the theoretical and the practical transferability of neural processing methods. we believe that a good baseline document planning and content selection approach should avoid the need for training data present in the many of recently proposed document planning and content selection approaches. this rules out as unsuitable most recent work that are based on learning from an aligned corpus of data and human-written texts, such as @xcite al. (2010), konstas and lapata (2013), wiseman et al. (2017), zhang et al. (2017), li and wan (2018), dou et al. (2018) and puduppully et al. (2019). outside of these trainable approaches, to our knowledge, most other document planning approaches are based on 'hand-engineered' (konstas and lapata, 2013), domain-specific methods. a highly relevant survey of various document planning methods is presented by @xcite . while these previous works are -to at least some degree -domain-specific, they establish concepts and ideas that are highly relevant for our goal. both @xcite and @xcite describe a core set of information, called 'summary spine' or 'key events', that they hold as more important than the rest of the available information. they, as well as @xcite , also employ a numeric estimate of importance. @xcite identify that content already selected for inclusion in the document plan affects how well suited so-far unselected content is for inclusion. @xcite identify gricean maxims @xcite as providing requirements for document planning and content selection. ## context our work on document planning is done in the context of a series of data-to-text nlg applications producing short highlights of structured statistical data. importantly, the applications are intended to be deployed in contexts where they must be able to produce texts highlighting between 10 and 30 data points from datasets measured in 100.000s of data points. the resulting texts are intended to both alert journalists to potential news and to provide them with a starting place from which to write the final news text. our system, adapted from @xcite , is based on a pipeline of components with dedicated responsibilities similar to those described by @xcite and @xcite . for this work, the relevant part of the architecture is the document planner component. this component receives as input two sets of message data structures, an example of which is shown in the core set contains messages that are known to be highly relevant to the generation task. unlike the "summary spine' of @xcite , the set is unlinked and unordered, and not all members of the set are guaranteed to be included in the document plan. the expanded set, contains messages that can be, but are not guaranteed to be, relevant for the document. expressed using the terminology from section 2, we assume that only messages in the core set can be nuclei, while messages from either set can be satellites. these core and expanded sets are determined automatically from user input. when requesting a new text, the user of the system must define a dataset the text is to be generated from, for example the consumer price data available from eurostat. this dataset is then divided into the core set and the expanded set by the user when they select what country the generated text should focus on. for example, if the user were to select that the text should discuss french consumer prices, the core set would contain all data from the consumer price dataset that pertains directly to france, while the rest of the consumer price dataset (including data pertaining to the uk, finland, croatia, etc.) would be set as the expanded set. we estimate each message's 'newsworthiness' using the interquartile range based method described by @xcite with the values scaled to have mean 0 and standard deviation 1 for the purposes of this computation. the resulting value is conceptually similar to 'importance' of @xcite and 'risk' of @xcite . the iqr based method compares each data point in turn to a larger distribution, giving it higher scores the further it is from the area between the first and the third quartile of the larger distribution. values between the quartiles are given a minimal, uniform, score that is dependent on the shape of the distribution. in other words, higher iqr values indicate that the value is more of an outlier compared to the rest of related data in the dataset. as such, it captures a degree of 'unexpectedness', which is an important aspect of newsworthiness @xcite . we do not use the domain-specific parts of the method described by @xcite . that is, we make no value judgement of whether messages pertaining to french consumer prices are more newsworthy than messages pertaining to croatian consumer prices, nor do we make judgements of whether changes in the price of education are more or less newsworthy than changes in the price of alcohol and tobacco. however, we do weight the scores so that messages with the timestamp field being closer to present receive higher weights, as recency is an important aspect of newsworthiness. while we have described our method for computing the newsworthiness value in some detail, we emphasize that for the rest of this article we only assume that the newsworthiness values are non-negative and that higher values indicate higher newsworthiness. more crucially for the method described be-low, we specify that the value type fields (which describe how the messages' values are to be interpreted) contain members of a hierarchical taxonomy of data types represented as colon-separated hierarchies of labels. for example, the value type field value health:cost:hc2:mio eur would indicate that the number in the value field is the amount of money (cost), measured in millions of euros (mio eur), spent by some nation (as defined by the location and location type fields) on rehabilitative care (hc2) in some time period (as defined by the timestamp and timestamp type fields) and that this is part of the larger health care topic (health). in our case, these labels are automatically established from the headers of the input data tables. the goal of document structuring is to produce a three-level tree-structure with ordered children. the root node corresponds to the document as a whole and the mid-level structures correspond to paragraphs. the leaves are the messages selected for inclusion in the document. while the messages have not yet, at this stage, been associated with any linguistic structures, they can be conceptualized as being phrases or very short sentences. we are thus concurrently determining both the content and the structure the document. we emphasize that our applications are employed in domains where they must be able to select some 10-30 messages from a pool of potential messages numbering in 100,000s. given infinite computational resources, it would be preferential to construct all possible document plans and then score them in some fashion. this, however, is infeasible given the size of the search space. previously, other authors have employed, for example, stochastic searches with significantly smaller search spaces @xcite . indeed, some kind of a beam search approach could be very useful in smartly searching a subset of the search space. however, we have thus far been unable to identify a document-level metric that adequately balances the 'total amount of newsworthiness' in a text with the length of the text, a requirement for beam search. ## research objective based on the above considerations, our main goal is to identify a widely applicable method for content selection and document planning that matches the following requirements: req1: the method needs to be highly performant req2: the method should not be dependent on domain knowledge req3: the document should have a theme req4: the document should have multiple paragraphs but not be excessively long req5: the paragraphs should have distinct themes related to the document theme req6: the paragraph themes should be newsworthy in their own right req7: the paragraphs should not be excessively long or short req8: all messages should relate to the paragraph theme req9: all messages should be newsworthy req10: within each paragraph, the messages should be presented in an order that produces a coherent narrative again, we emphasize that our goal is not to identify a method that is optimal for any specific scenario, but rather to determine a baseline method that is adequate for a broad spectrum of applications and sub-domains. ## a baseline approach to document planning optimally, we would wish to produce some sort of a globally optimal document plan. however, as discussed above, this would entail significant computational costs and require a scoring function applicable to the document as a whole. as such, we propose a method for producing document plans in a greedy, linear, and iterative fashion. at every stage, decisions are made considering only a limited local context, thus avoiding the need for a method of determining the global quality of the document plan, thus fulfilling req1 ('the method needs to be highly performant'). the document's overall theme, in our use case, is selected by the user who initiates the generation task. in initiating the task, the users selects both a dataset and a focus location. the generation process then derives the core messages and expanded messages sets (the inputs to the document planner, see section 3) so that both sets discuss the dataset field description example value where what location the fact relates to finland where type what the type of the location is country timestamp the time (or time range) the fact relates to 2020m05 timestamp type the type of the timestamp month value a (usually) numeric value 0.01 value type interpretation of value cphi:hicp2015:cp-hi02:rt01 newsworthiness an estimate of how newsworthy the message is 1 ## evaluation the method described above was implemented in a larger nlg application producing news alerts for journalists from datasets provided by eurostat. a variation of the same application was also developed with a simplified document planner. in this simplified planner, the planner always selects the maximally newsworthy available message as the message without any early stopping threshold. nuclei are selected from the core messages set, while satellites can be from either set. contrasting our proposed method with this simplified method enables us to evaluate the importance of narrative coherence in the generated texts. the larger application is multilingual, but the evaluation was conducted using english language texts. three experts were recruited from the finnish news agency stt, a national european news agency, to evaluate documents on the consumer price indices in five different european nations. for all nations, the judges were shown variants produced by both our proposed method and the simplified method. one of the selected countries is the country the news agency is based in, with the assumption that the judges would have high amounts of world knowledge they would be able to use in evaluating these texts. another variant pair describes a country that is both relatively small and geographically remote (but still within eu), with the assumption that the journalists are unlikely to ## conclusions in this work, we have identified a need for, and proposed, a widely applicable baseline document planning method for generating journalistic texts from statistical datasets. our method is based on observations on the similarities between the orbital theory of news structure @xcite and @xcite . while our proposed method is likely to fall short of the performance of subdomain-specific planning methods, results indicate that it achieves adequate performance while fulfilling a set of requirements identified based on the larger application domain of news generation.