Tracking the Newsworthiness of Public Documents (2024)

Abstract: Journalists regularly make decisions on whether or not to report stories, based on “news values”. In this work, we wish to explicitly model these decisions to explore _when_ and _why_ certain stories get press attention. This is challenging because very few labelled links between source documents and news articles exist and language use between corpora is very different. We address this problem by implementing a novel _probabilistic relational modeling_ framework, which we show is a low-annotation linking methodology that outperforms other, more state-of-the-art retrieval-based baselines. Next, we define a new task: __newsworthiness prediction__, to predict if a policy item will get covered. We focus on news coverage of local public policy in the San Francisco Bay Area by the _San Francisco Chronicle_. We gather 15k policies discussed across 10 years of public policy meetings, and transcribe over 3,200 hours of public discussion. In general, we find limited impact of public discussion on newsworthiness prediction accuracy, suggesting that some of the most important stories barely get discussed in public.Finally, we show that newsworthiness predictions can be a useful assistive tool for journalists seeking to keep abreast of local government. We perform human evaluation with expert journalists and show our systems identify policies they consider newsworthy with 68% F1 and our coverage recommendations are helpful with an 84% win-rate against baseline. We release all code and data to our work here: https://github.com/alex2awesome/newsworthiness-public.

## introduction despite much qualitative analysis of newsworthiness @xcite very little quantitative work has attempted to analyze: ## policy item ↔ article linking our goal in this section is to determine which articles cover which policies. we seek to model the likelihood a link l exists between an article, a, and a specific policy item, p, or p (l|a, p). we apply the probabilistic relational model (prm) framework @xcite to solve this problem. in prm, we learn conditional attributes h 1 , ...h t of either the article, policy, or both and marginalize over them: where, as shown in figure 2 , h 2 might be "covers sfbos", and h 3 might be "covers sfbos votes/policy." foot_2 (note that the model p(h i |a, p) = p(h i |a) if the attribute h i is only dependent on the article, a.) not all politics articles are about sf-bos, and not all sfbos articles cover policy. such variety confounds unsupervised models, but is solvable when broken into easier-to-supervise subproblems. this is not dissimilar to chain-of-thought (cot) @xcite , where language models decompose complex reasoning tasks. ## newsworthiness prediction next, we wish to ask why certain policy proposals are covered. to address this, we establish a new task, newsworthiness prediction: predict, given a policy item p, if an article will write about it. we use our linked dataset {(a, p)}, in section 2, and treat this problem as a prediction problem where: our goal is twofold: (1) learning a good model can show us which features of policy-items lead to coverage. (2) performing this task well at inference time takes us steps closer to building tools that will be useful for surfacing potential stories. previously, newsworthiness has been addressed as a feature-detection problem, as in @xcite , where engineered-features measured specific criteria 14 . journalists examined combinations of features to find newsworthy items but could miss items if their newsworthiness did not fit the measurements. because we formulate our task as a prediction task, backed by a dataset, we can also expose new and possibly unexpected features. however, a prediction-based approach is limited in its own ways. we assume that past coverage bly, changes in what is considered "newsworthy". ## discussion and conclusion in summary, we established links between a large corpus of news articles and local policy proposals we did so using a classical method, probabilistic relational modeling, that outperformed retrievalbased methods and embedding-based methods with only a small amount of annotated data. we used the assessed newsworthiness of prior articles to build models to predict the newsworthiness of articles. we found that the performance of our models did not degrade over time, and we found that expert journalists agreed with our newsworthiness assessments and found our tools helpful. our work faces many limitations and risks, which we discuss in sections 2.3 and 3. notably, we assumed that historical coverage patterns are a reasonable starting point for modeling future newsworthiness predictions. while we found that this yielded useful models, there might be cases where news values evolve and prior decisionmaking is morally and ethically unacceptable, for example with crime @xcite or suicide coverage @xcite . our work would serve enforce such historical patterns. also, it might miss major, atemporal results, like covid-19. both of these represent considerable risks, and indicate that human involvement remains crucial in any kind of newsworthiness prediction system (a point made by researchers studying other realworld predictive systems @xcite ). despite these risks and limitations, we see this work as presenting a crucial starting point for a larger research direction in newsworthiness prediction. by establishing "newsworthiness" as a welldefined predictive task, we hope to have opened the door to future work applying these concepts. we intend in upcoming work to explore ways to introduce control and explainability into the newsworthiness prediction pipeline that we have outlined here.