Do LLM s Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLM s (2024)

Abstract: Journalists engage in multiple steps in the news writing process that depend on human creativity, like exploring different “angles” (i.e. the specific perspectives a reporter takes). These can potentially be aided by large language models (LLMs). By affecting planning decisions, such interventions can have an outsize impact on creative output. We advocate a careful approach to evaluating these interventions to ensure alignment with human values.In a case study of journalistic coverage of press releases, we assemble a large dataset of 250k press releases and 650k articles covering them. We develop methods to identify news articles that _challenge and contextualize_ press releases. Finally, we evaluate suggestions made by LLMs for these articles and compare these with decisions made by human journalists. Our findings are three-fold: (1) Human-written news articles that challenge and contextualize press releases more take more creative angles and use more informational sources. (2) LLMs align better with humans when recommending angles, compared with informational sources. (3) Both the angles and sources LLMs suggest are significantly less creative than humans.

## introduction in-depth news coverage goes beyond summarizing a story: it confirms or refutes narratives, offers viewpoints, and contextualizes events to expand readers' understanding @xcite . this process requires time and resources @xcite . in an era where journalists are inundated with complex topics to cover and resources are dwindling @xcite , approaches to facilitate such coverage are needed @xcite . figure 1 : two steps that precede writing news articles based on press releases are: formulating an angle (i.e., a specific focus), and selecting sources (i.e., a person or document contributing information). we compare planning steps made by human journalists (left), to those made by llms under various prompts designed to stimulate creative aid (right). we find that llm plans are significantly less creative and diverse. we call for deeper alignment with fundamental human decision-making before creative-aid tools are widely deployed. llms have been proposed as tools to facilitate creative planning in journalism. @xcite , for instance, explored how well llms could recommend unique angles to cover press releases. while llms have been found to contribute positively, important questions remain. how often do llm planning decisions align with human values? how can we adjust such decision-making to ensure better alignment? in this work, we lay the groundwork for more broadly developing ai approaches for aiding creative tasks, ensuring they align with human values, and outlining a path to improvement. with a broad, novel dataset, we compare the planning decisions llms would make to the decisions humans have made in the past. as such, our work represents a generalizable foot_2 benchmark in creative planning tasks and can serve as a template for creative planning evaluation going forward. we start by assembling a corpus of press releases and news articles covering them, and identify articles that have effectively coveraged these releases. according to @xcite , effective coverage substantially challenges and contextualizes press releases. to measure this, we quantify how much the article entails and contradicts a press release. for intuition on why we measure both, consider: complete entailment would simply indicate a vanilla summary @xcite while complete contradiction could indicate off-topic. we find, via extensive manual evaluation, that a mixture of both indicates effective coverage (.81 f1). next, we ask what planning decisions characterize effective coverage. on a dataset of 6,000 human-written news articles and press release pairs, we find strong positive correlations between the overall criticality of a news article's coverage, and: (1) the creativity of the news article's angle (@xmath0.29) and (2) the number of sources used in the article (@xmath1.5). with this in hand, we turn to using our dataset to evaluate how llms might facilitate these two planning steps. first, we explore an llms ability to recommend "angles", or story directions, building off @xcite . next, we compare the kinds of sources suggested by an llm with the sources human journalists used to cover these articles. overall, we have two core findings: (1) we find that llms perform well at recommending angles that humans ultimately took (63.6 f1-score), but perform poorly at recommending kinds of sources (27.9 f1-score). (2) however, the level of creativity for both angles and sources is low. in sum, we make the following contributions: taken together, these indicate that substantial work is needed during the planning stages of creative acts in order to align llms with the creativity of human work. however, our results, especially angle formulation, suggest that narrative planning exists in llms, and future work improving our approach might yield significant progress. ## dataset press releases offer an ideal window into the journalistic process. press releases contain potentially valuable information, but are often "spun" by their authors to portray events positively @xcite . "de-spinning" them involves challenging and contextualizing claims (maat and de jong, 2013) and often requires substantial work prior to writing: as illustrated in figure 1 , journalists engage in multiple planning steps, including developing an angle and finding sources. here, we describe how we construct pressrelease, a large corpus of 650,000 news articles hyperlinking to 250,000 press releases. pressrelease contains data collected in two main approaches in order to avoid biases with either one. press releases ← news outlets, hyperlinks: the first way we discover news articles linking to press releases is to collect html of news articles, and find hyperlinks to known press release domains in these articles. we query common crawl for all urls from 9 major financial newspapersin all scrapes since 2021, resulting in 114 million urls. from these urls, we discover 940,000 urls of news articles, specifically, using a supervised model by @xcite to differentiate news article urls from other pages on news websites (e.g. login pages). then, we find hyperlinks to press releases in these news articles by finding all links to known press release websites. 4 this yields 247,372 articles covering 117,531 press releases. ## press release coverage as contrastive summarization we seek to identify when a news article effectively covers a press release, as defined by (maat and de jong, 2013). identifying effective coverage is not trivial: many articles uncritically summarize press releases or use them peripherally in larger narratives. we examine pairs of news articles and press releases, answering the following two questions: (1) is this news article substantially about this press release? (2) does this news article challenge the information in the press release? while many articles discuss press releases, most of them simply repeat information from the release without offering insights. after examining hundreds of examples, we devise novel framework, contrastive summarization, to describe "effective coverage". a piece of text is a contrastive summary if it not only conveys the information in a source document, but contextualizes and challenges it. can we automatically detect when a piece of text is a contrastive summary? to do so, we represent each press release and news article as sequences of sentences, ⃗ @xmath2, ...p n , ⃗ @xmath3, ...n m , respectively. we establish the following two criteria: we define "references" (or "contradicts") as 1 if any sentence in ⃗ n references (or contradicts) p j , 0 otherwise. viewed in an nli framework @xcite , "contradicts" is as defined in nli, and "references" = ["entails" ∨ "contradicts"]. we expect this approach can get us close to our goal of discovering press releases that are substantially covered and challenged by news articles. a press release is substantially covered if enough of its information is factually consistent or contradicted by the news article. it's substantially challenged if enough of its sentences are contradicted by the news article. @xcite found that aggregating sentence-level nli relations to the document-level improved factual consistency estimation. we take a nearly identical approach to the one shown in their work. 10 first, we calculate sentence-level nli relations, p(y|p i , n j ), between all ⃗ p × ⃗ n sentence pairs. then, we average the topk inner relations for each p i , generating a p i -level score. finally, we average the top-k outer p i -level scores. k inner is the number of times each press release sentence should be referenced before it is "covered", and k outer is the number of sentences that need to by "covered" to consider the entire press release to be substantially covered. using nli to identify press release/news article coverage pairs provides a computationally cheap and scalable method. ## analysis of press releases and news articles we frame three insights to explain more about what effective coverage entails. these insights lay the groundwork for our explorations in our llm planning framework discussed in the next section. insight #1: effective news coverage incorporates both contextualization and challenging statements. our first insight is that nli-based classifiers can be useful for the task of identifying effective coverage. this is not entirely obvious: nli classification is noisy @xcite and contradiction relations might exist not only in directly opposing statements, but in ones that are orthogonal or slightly off-topic @xcite . however, our strong results on a large annotated dataset -our annotators were instructed to determine whether a news article effectively covers a press release -indicate that this method is effective. our performance results, between 70-80 f1-score, are within range of laban et al. (2022) (66.4-89.5 f1 across 6 benchmarks), who first used nli to evaluate vanilla summaries. that a similar methodology can work for both tasks emphasizes the relatedness of the two: identifying effective corr with creativity angle source contradiction 0.29 0.10 entailment 0.27 0.03 neutral -0.07 -0.11 insight #2: articles that contradict and entail press releases (1) take more creative angles and (2) use more sources. we first noticed that articles with more creative angles foot_11 contradict and entail press releases more, as shown in in order to further explore these kinds of articles, we analyze the sources they used. @xcite developed methods to identify informational sources mentioned in news articles. we utilize this work to identify sources in our corpus: as shown in insight #3: news articles that contradict press releases more use more resource-intensive sources. of the kinds of sources used in news articles, the majority are either quotes, 40%, (i.e., information derived directly from people the reporter spoke to), or press reports, 23% (i.e., information from other news articles). we obtain these labels by scoring our documents using models trained and described by @xcite . as shown in take-away: taken together, our three insights suggest that any approach to assisting journalists in covering press releases must have an emphasis on (1) suggesting directions for contrastive summaries and (2) incorporating numerous sources. we take these insights forward into the next section, where we assess the abilities of llms to assist journalists. ## llm-based document planning based on the insights in the previous section, we now study how llms might assist journalists. specifically, we ask: how well can an llm (1) provide a starting-point, or an "angle", for a contrastive summary and (2) how well can an llm suggest useful kinds of sources to utilize? petridis et al. (2023) explored how llms can aid press release coverage. the authors used gpt-3.5 to identify potential controversies, identify areas to in with our dataset, pressreleases, we are able to conduct a more comprehensive experiment to benchmark llms planning abilities. in this section, we identify 300 critical news articles and the press releases they cover. we compare plans generated by llms with the plans pursued by human journalists: such an approach, along with recent work @xcite , is part of an emerging template for comparing llm creativity with human creativity and studying how llms might be used in human-in-the-loop creative pipelines. ## discussion we assessed how llms can help journalists plan and write news articles. we constructed a large corpus of news articles covering press releases to 19 we report our 5-point scale in we found that llm suggestions performed quite poorly compared with the reporting steps actually taken by humans, both in terms of alignment as well as creativity. does this suggest that llms are poor planners in practice? our benchmark provides a useful check for this question, but we do not believe our experiments here are conclusive. instead, we view our approach as a first step: we compare basic prompt engineering with human actions that are observed from final-draft writing. clearly, the final drafts written by humans result from multistep, iterative reporting, accumulated experience, and real-world knowledge. while llms are not able to match many of these plans, they may nevertheless be helpful when paired with journalists. using human-decision making as a basis of comparison for llms is standard, even in creative, open-ended tasks: e.g. story-planning @xcite , computational journalism @xcite @xcite and others @xcite . if this problem were unlearnable (e.g. there were simply too many an-gles to take, or so much prior knowledge needed to form any kind of plan), then we would not see any improvement after fine-tuning. crucially, the 10point improvement we observe from fine-tuning is evidence that there are learnable patterns. existing research into journalism pedagogy, which implies that observation of other journalists' standard practice is as important as gaining subject-matter expertise and conducting on-the-ground work (ryfe, 2023), should further support the hypothesis that planning is learnable. however, the low scores after fine-tuning imply the need for more fundamental work. our current approach is naive: we expect llms to produce human-level plans with simple prompting and no references, besides the press release. there are two major directions for advancement in this task: as llms are increasingly used for planningoriented creative tasks @xcite , careful analysis is required. our goal in this work was to outline a novel task requiring planning and affirm a basic to perform this analysis. we believe that our use of llms in article planning represents an emerging and as-yet-underexplored application of llms to tasks upstream of the final writing output. in these cases, the decisions made by the llm might one day have the ability to impact even more fundamental steps: which sources to talk to, which angles to take, and which details to highlight. professional journalists ground their approach to these decisions in institutional values: fairness, reducing sourcing bias, and confirming details. without carefully comparing the steps that llms make with humans, we risk disregarding these values. ## related work our work is inspired by the task outlined in an-glekindling @xcite , which introduced llm-assistants for press release coverage as a useful writing tool and utilized llms to summarize press releases and suggest angles. our work fits into a larger literature utilizing llms as writing assistants @xcite @xcite . we take a datadriven approach toward identifying journalists' needs through corpus and benchmark construction. whether llms can serve as effective planners in creative acts is currently an unresolved debate @xcite . however, the two-step process of planning then creating has been explored extensively @xcite @xcite . our work aims to build in this direction by constructing an evaluation set. we see broad parallels between the notion of a plan, which is an unobserved generative process preceding the generation of observable text, and earlier generations of discrete latent variable modeling @xcite @xcite . work like @xcite seeks to extend concepts and framing in this work into a more modern era by selecting the best plan from multiple plans. we believe that various approaches are converging to a novel approach to llm and human interaction, and we hope that our work serves as a good addition and a useful benchmark. ## conclusion we have built a corpus to study professional human planning decisions by identifying well-reported news articles covering press releases. these are articles use a variety sources, engage in criticism, and challenge the source material (maat and de jong, 2013). we assessed how llms could suggest plans for covering source documents for these articles. our goal is to ground llm planning in the observation of human dynamics, opening the door to aligning future developments to journalistic practice. our approach captures more broadly the objectives of human journalists across many different organizations, across decades of coverage. our benchmark compares the plans an llm makes to approaches taken by journalists who were covering press releases in real-life settings, and establishes a new direction for exploring how llms can support the journalistic process.