A Bit of This, a Bit of That: Building a Genre and Topic Annotated Dataset of Historical Newspaper Articles with Soft Labels and Confidence Scores (2025)

Abstract: Digitised historical newspaper collections are becoming increasingly accessible, yet their scale and diverse content still present challenges for researchers interested in specific article types or topics. In a step towards developing models to address these challenges, we have created a dataset of articles from New Zealand’s Papers Past open data annotated with multiple genre and topic labels and annotator confidence scores. Our annotation framework aligns with the perspectivist approach to machine learning, acknowledging the subjective nature of the task and embracing the hybridity and uncertainty of genres. In this paper, we describe our sampling and annotation methods and the resulting dataset of 7,036 articles from 106 New Zealand newspapers spanning the period 1839-1903. This dataset will be used to develop interpretable classification models that enable fine-grained exploration and discovery of articles in Papers Past newspapers based on common aspects of form, function, and topic. The complete dataset, including un-aggregated annotations and supporting documentation, will eventually be openly released to facilitate further research.

## introduction just over 100 years ago, in an article titled "the natural history of the newspaper", robert park wrote, "the newspaper, like the modern city, is not wholly a rational product." @xcite . he was describing the development of the press from newsletter to political and commercial institution, but his sense of an evolving organism, something familiar yet difficult to specifically define, applies as much to the types of articles inside as it does to the newspaper as a whole. these types, or categories, of newspaper articles can be referred to as genres, although an agreed upon definition of this term is as difficult to pin down as the genres themselves @xcite @xcite @xcite . for our purposes in constructing a dataset of digitised historical newspaper articles annotated with genre and topic labels, we consider genre to be the type of document (or newspaper article in this case) and topic to be what the document is about @xcite . we do not attempt to establish a definitive list of the types of articles found in historical newspapers but instead aim to identify articles that share characteristics of form and function along with the topics within those articles, such as "football" or "politics". we call the article categories genres and label them with common terms such as "editorial", "letter", or "review". today, we have desktop access to millions of digitised newspapers and researchers are investing significant effort in developing datasets, designing interfaces, and training state-of-the-art models to enhance these collections and extract new insights (for example @xcite @xcite @xcite . the text of many digitised newspaper collections has been made searchable through the use of optical character recognition (ocr) software which, along with related technologies and post-ocr correction methods, has improved significantly in recent years @xcite @xcite . however, the fact remains that many historical newspaper collections contain a significant number of errors which affect the reliability of keyword search and have implications for researchers in terms of source criticism, reproducibility, and claims of representativeness @xcite @xcite . ocr errors, misspellings, and diachronic language change also present challenges when it comes to the robust application of new methods such as retrieval augmented generation @xcite to historical text collections @xcite @xcite . several studies have explored the use of features such as page layout data, text statistics, tf-idf metrics, and parts-of-speech frequencies, sometimes in addition to bag-of-words or word embedding representations, to classify news articles by genre (for example @xcite @xcite . these approaches offer alternative ways to identify and retrieve different types of texts, and work at a level of abstraction from the individual characters, which can provide resilience to ocr errors and unusual word forms. when used with interpretable machine learning methods and in transparent pipelines they can provide valuable insights into the characteristics of different genres and the ways in which these genres change and evolve @xcite . misclassifications can also be informative by revealing where and how genres overlap or surfacing unusual examples that would otherwise be difficult to discover @xcite @xcite . the subjectivity and hybridity of genre and its fluidity across time make genre classification a challenging and interesting problem @xcite @xcite . the example shown in figure 1 , a humorous poem that contains aspects of a recipe and advice, illustrates these challenges. this text might be labeled in different ways depending on the perspective of the annotator, and classification using traditional supervised machine learning methods trained on a single hard label per article could lose valuable information. soft labels and confidence scores from multiple annotators provide a way to better reflect the human perspective of genre by capturing its subjectivity and hybridity @xcite . this approach follows the perspectivist paradigm described by @xcite , which builds on previous work encouraging consideration of human label variation in machine learning model training and evaluation @xcite @xcite @xcite . 1 in making room for disagreement, perspective, and subjectivity, such an approach has been argued to improve model calibration, representation, and evaluation @xcite . in this paper, we describe the construction and 1 see also: the perspectivist data manifesto. features of a genre and topic annotated dataset of digitised historical newspaper articles sampled from the national library of new zealand's papers past open data (national library of new zealand te puna mātauranga o aotearoa, 2024). the resulting dataset includes soft genre labels, annotator confidence scores, and topic labels and covers 106 new zealand newspaper titles from the period 1839-1903. the final dataset will be made publicly available and, as far as we are aware, will be the first openly released dataset of digitised historical newspaper articles annotated in this way. it is a key part of an ongoing project focused on developing interpretable genre classification models to enhance the discovery and analysis of articles in papers past newspapers. the key contributions of this work are: (1) a large dataset of more than 7,000 historical newspaper articles with un-aggregated soft genre labels, annotator confidence scores, and topic labels, (2) a detailed description of the sampling and annotation process and results including the genre and topic labels and inter-annotator agreement, and (3) a discussion of the challenges and limitations associated with the development of the dataset. ## annotation results of the 7,791 articles annotated, 652 marked as "illegible" and 103 that were not labeled with a primary genre were removed from the dataset. the ## discussion in this study, we adopted a perspectivist approach to annotating a dataset of historical newspaper articles with soft genre and topic labels and annotator confidence scores. the approach allowed us to capture the inherent subjectivity and hybridity of genre and the resulting dataset contains a wealth of information that can be used in the development and testing of genre classification models for historical newspaper texts. while we have reported krippendorff's α using two approaches to identifying a primary genre label, there is much to be learnt from further analysis of agreement across the genre and topic labels and between annotators. the dataset lends itself to the exploration of new methods for evaluating data quality with multiple soft labels, an active area of research @xcite . in ongoing work the annotated dataset will be used to develop interpretable classification models that enable a more fine-grained exploration of articles in papers past newspapers based on common aspects of form, function, and topic. our focus on interpretable methods is motivated by several factors including improved transparency and reproducibility of results and the value to researchers of being able to understand and interrogate the combinations of features that contribute to an article's classification. as emphasised by @xcite , transparent machine learning methods can support rather than replace qualitative historical research. they can be used to test hypotheses at scale and across different dimensions such as time periods, regions, or newspapers, or reveal patterns worthy of closer investigation @xcite . our annotation approach enhances the exploratory potential of subsequent models by enabling more of the complexity of genre and topic to be reflected in the training data. ## conclusion the sampling and annotation process described in this paper, and the resulting dataset of more than 7,000 articles from new zealand newspapers spanning the period 1839-1903, will be of interest to researchers in digital humanities and computational linguistics, as well as those interested in exploring perspectivist approaches to machine learning. when publicly released, the final dataset will include full un-aggregated annotations and will be supported with detailed documentation that follows the recommendations of @xcite , and more recently @xcite , and includes information on the sampling and annotation process, the distribution of genres and topics, ethical considerations, metrics, and potential use cases. in ongoing work, we will use the dataset to develop interpretable classification models to further explore aspects of genre and topic in papers past newspapers.