Explaining Mixtures of Sources in News Articles (2024)

Abstract: Human writers plan, _then_ write. For large language models (LLMs) to play a role in longer-form article generation, we must understand the planning steps humans make before writing. We explore one kind of planning, source-selection in news, as a case-study for evaluating plans in long-form generation. We ask: why do _specific_ stories call for _specific_ kinds of sources? We imagine a generative process for story writing where a source-selection schema is first selected by a journalist, and then sources are chosen based on categories in that schema. Learning the article’s _plan_ means predicting the schema initially chosen by the journalist. Working with professional journalists, we adapt five existing schemata and introduce three new ones to describe journalistic plans for the inclusion of sources in documents. Then, inspired by Bayesian latent-variable modeling, we develop metrics to select the most likely plan, or schema, underlying a story, which we use to compare schemata. We find that two schemata: _stance_ and _social affiliation_ best explain source plans in most documents. However, other schemata like _textual entailment_ explain source plans in factually rich topics like “Science”. Finally, we find we can predict the most suitable schema given just the article’s headline with reasonable accuracy. We see this as an important case-study for human planning, and provides a framework and approach for evaluating other kinds of plans, like discourse or plot-oriented plans. We release a corpora, _NewsSources_, with annotations for 4M articles, for further study.

## introduction as language models (lms) become more proficient at long-form text generation and incorporate resources @xcite and tools @xcite to support their writing, recent work has shown that planning before writing is essential @xcite @xcite . however, supervised datasets to support headline: nj schools teach climate change at all grade levels michelle liwacz asked her first graders: what can penguins do to adapt to a warming earth? ← potential labels: academic, neutral gabi, 7, said a few could live inside her fridge. ← potential labels: unaffiliated, neutral tammy murphy, wife governor murphy, said climate change education was vital to help students. ← poten. labels: government, agree critics said young kids shouldn't learn disputed science. ← labels: unaffiliated, refute a poll found that 70 percent of state residents supported climate change being taught at schools. figure 1 : we seek to infer unobserved plans, or schemata, in natural data, focusing on one scenario: source-selection made by human journalists during news writing. although the reasons why sources are chosen are unobservable, we show that one explanation (in the diagram, represented by squares: { , , }), is preferred over another (represented by circles: { , , }) if it better predicts the observed text (conditional perplexity) and the explanation is more internally consistent (posterior predictive). our paper is divided into two parts: in the first part (i.e. section 2.3 and section 3.2), we introduce the different schemata we will compare -i.e. the top half of this diagram. in the first part (i.e. section 4 and section 5) we determine the right schema for a datum among competing schemata -i.e. the bottom half of this diagram -and, given minimal information about a document, we show that we can predict what schema should be used. as can be seen, the journalist planned, before writing, the different kinds of sources (e.g. teachers, kids) she wished to use. why did she choose these groups? was it: a. to include varied social groups? b. to capture different sides of an issue? answering this question, we argue, allows us to infer why she chose each source. if the answer is a, we can infer, then, that the writer probably chose her sources because each fell into a different social group. if the answer is b, the sources were more likely chosen because each agreed or disagreed with the main event. now, the core problem in this endeavor emerges: a document's plan is not typically observable. we directly address this and show that we can differentiate between plans in naturally observed text. inspired by latent variable modeling approaches (airoldi and bischof, 2016), we uncover a document's most likely plan on the following basis: a proposed plan better describes a document's actual plan if it gives more information about the completed document. we introduce simple metrics for this goal: conditional perplexity and posterior predictive likelihood, in figure 1 (section 2.2). next, to create a straightforward setting to demonstrate the power of these metrics, we work with professional journalists from multiple major news organizations to identify planning approaches they regularly take. we operationalize these as schemata, or explanatory frameworks under which each source in the news article is assigned to a different discrete label (e.g. in the affiliation schema, for example, the source-categories would be government, media...). we adapt five schemata from parallel tasks and introduce three novel schemata to better describe sourcing criterion. we implement our schemata by annotating over 600 news articles with 4,922 sources and training supervised classifiers. we validate our approach with these journalists: they deem the plans we infer as correct @xmath0.74 f1 score. finally, the choice of schema, we find, can be predicted with moderate accuracy using only the headline of the article (roc=.67), opening the door to new computational journalism tooling. in sum, our contributions are threefold: with this work, we hope to inspire further unsupervised inferences in document generation. studying journalistic decision-making is important for understanding our information ecosystem @xcite @xcite , can lead to important computational journalism tools @xcite and presents a real-world case-study in planning. ## building a silver-standard dataset of different possible plans the schemata described in the previous section give us theoretical frameworks for identifying writers' plans. to compare schemata and select the schema that best describes a document, we must first create a dataset where informational sources are labeled according to each schema. we describe that process in this section. ## comparing schemata we are now ready to explore how well these schemata explain source selection in documents. we start by describing our experiments, then baselines, and finally results. all experiments in this section are based on the 90, 000 news articles filtered from newsedits, labeled as described in the previous section. we split 80, 000/10, 000 train/eval. ## predicting schemata taken together, our observations from (1) section 3.3) indicate that schemata are largely unrelated and (2) section 4.3 indicate that stance and affiliation both have similar explanatory power (although stance is less predictable). we next ask: which kinds of articles are better explained by one schema, and which are better explained by the other? if we can answer this question, we take steps towards being able to plan source-selection via different schemata. such a step could lead us towards better multi-document retrieval techniques, by giving us axes to combine different documents into a retriever. in ## related work this work focuses on informational sources in news articles and is part of a broader field of characterbased analysis in text. ## conclusions in conclusion, we explore ways of thinking about sourcing in human writing. we compare 8 schemata of source categorization, and adapt novel ways of comparing them. we find, overall, that affiliation and stance schemata help explain sourcing the best, and we can predict which is most useful with moderate accuracy. our work lays the ground work for a larger discussion of discovering plans made by humans in naturally generated documents. it also takes us steps towards tools that might be useful to journalists. naturally, our work is a simplification of the real human processes guiding source selection; these categories are nonexclusive and inexhaustive. we hope by framing these problems we can spur further research in this area.