The Kronieken Corpus: an Annotated Collection of D utch/ F lemish Chronicles from 1500-1850 (2024)

Abstract: In this paper we present the Kronieken Corpus, a new digital collection of 204 chronicles written in Dutch/Flemish between 1500 and 1850, which have been scanned, transcribed and annotated with named entities, dates, pages and a smaller part with sources and attributions. The texts belong to 308 physical volumes and contain between 23 and 24 million words. 107 chronicles, or 178 chronicle volumes, collected from 39 different archives and libraries in The Netherlands and Belgium and transcribed by volunteers had never been transcribed or published before. The result is a unique enriched historical text corpus of original hand-written, non-canonical and non-fiction text by lay people from the early modern period.

## introduction we present a newly transcribed and annotated dataset of local chronicles in dutch from the period 1500-1850. the corpus has been compiled with the goal of developing a method to track and analyse the circulation, reception, evaluation and acceptance of old and new knowledge over time and across geographical locations by a lay public of mainly middle class authors. this work is part of the project chronicling novelty. new knowledge in the netherlands, 1500-1850 foot_0 and the corpus is available for public use. @xcite the historic period of 1500-1850 was chosen because it covers a number of societal changes that impacted knowledge production and circulation, such as the rise of the printing press, church reformations and the scientific revolution. this period also covered the so-called eighteenth century enlightenment and revolutionary era, as well as the political 'restoration' period of the early nineteenth century. local chronicles are chronologically organized accounts of events in the author's community. following pollmann (2016), who argued that histori-ans of early modern europe should more actively exploit the potential of the thousands of local chronicles that europeans wrote between 1500-1850, we approached chronicles as collections of useful knowledge created by authors for future reference. chroniclers collected information on a range of topics including local politics and history, crime, prices, public space and natural or cultural events that they deemed remarkable. most of these texts were not written with a view to publication in print, but as manuscripts circulated among the literate middle classes of early modern towns and villages. chronicles are one of the very few genres of narrative european texts that remained both ubiquitous and relatively stable throughout the early modern period. therefore they can be used for comparative studies across both time and space about a wealth of topics. in the context of the research project chronicling novelty, we analysed what sources of information people considered reliable and how new information changed the way people reasoned over time (dekker, 2022; lassche and morante, 2021; lassche et al., 2022; kuijpers, 2022). we believe that this dataset will be of unique value for research in history, digital humanities and historical linguistics, as well as for students of e.g. local politics, state formation, religious history, social conflict, and history of emotions. after the discussion of some related work in section 2, we describe the composition of the corpus in section 3, the transcription process in section 4, the annotations in section 5 and the usage, distribution and maintenance in section 6. finally, we present a discussion in section 7 and put forward some conclusions in section 8. ## related work digital historical texts in dutch are made available by different institutions, and in different ways. ex-amples can be found at the website of clarin, 3 the huygens institute 4 and the institute for dutch lexicology. @xcite many texts can be searched and accessed via the nederlab portal especially catering for historical linguists, 6 such as the gysseling corpus, the corpus middelnederlands and the corpus oudnederlands. most of these corpora consist of documents produced by institutions, such as the currently being digitized proceedings of the states general of the dutch republic (1576-1796). @xcite other resources contain newspapers, @xcite or the writings of important political or literary figures and scientists. @xcite in comparison to these existing corpora, this corpus is unique in several ways. it brings together a large set of non-institutional writings by a broad range of lay -often unknown -authors that are not archived in one place but scattered all over the netherlands and belgium. similar to the collection of private letters confiscated from dutch ships during the anglo-dutch wars in the seventeenth and eighteenth century, 10 this collection represents the voices of individuals that belong to various social strata of society, who write on their own initiative and on topics that matter to them. other than the seized correspondences, however, the chronicles are written in a larger geographical area comprising also current day belgium and the eastern and southern inland provinces of the netherlands (rutten and wal, 2011, 2014). we are not aware of a similar dataset in other languages. ## composition of the corpus when searching for chronicles that suited our goals we used the following selection criteria: first, we excluded family chronicles and regional chronicles -family chronicles lack the focus on public affairs, while regional chronicles were more often written for publication and by semi-professional historians. in order for our corpus to be searchable, we also decided we could only include texts that were (mainly) written in dutch, even though french was the selection of the chronicles that would make up the corpus was carried out by the project leaders, both senior researchers in history, with the help and advice of student assistants, historians and archivists. the list of chronicles that were selected can be found in our github repository. @xcite the collection process lasted from 2016 till 2018. after that time some more chronicles were identified and added. to find manuscripts we searched the digital inventories of the provincial archives in the netherlands and belgium for (variants of) words such as chronicle, annals, journal, history and diary. we did the same for local archives and a number of important libraries of which we knew or suspected that they could host chronicles. in this way we were able to add 106 unpublished chronicles (177 volumes) to our collection, that were sourced from 39 different archives and libraries and a few private collections. these archives and libraries had to be visited one by one, and every page of a chronicle manuscript had to be scanned. some archives took on the task of scanning the chronicles themselves, but in most cases the scantent was used by the project team. @xcite in combination with the doc-scan app, the scantent enables the user to hold a document with both hands and scan it with their smartphone without pressing any button. docscan automatically takes a picture once a page is turned. the texts are all in dutch/flemish with sometimes quotations in other languages (mainly french or latin). spelling is very heterogeneous. some texts, especially some sixteenth-century chronicles from the north-eastern netherlands have elements of the local dialect. figure 1 shows a map of the low countries with the distribution of manuscripts over time and space. ## annotations because of the complexity of the task, annotation projects of historical corpora still make use of manual work by experts or volunteers although tools for automatic annotation are currently being developed (tonelli and menini, 2021; arnoult et al., 2021; sluijter et al.; koolen et al., 2020; koolen and hoekstra, 2020). we performed the annotations with the transkribus tool. once the chronicles had been transcribed we performed three annotation tasks. (i) we labeled named entities, dates, and page numbers. these labels should improve the searchability of the corpus for future users as well as enable our own analysis. due to limited time, we were not able to annotate the full corpus. instead, a subset of 139 chronicles (196 volumes) was annotated. (ii) in a smaller subset of 66 chronicles (85 volumes) the referencing to sources of information by the authors was annotated. (iii) in a third annotation project, attribution relations were tagged in another subset of the corpus (17 chronicles, 22 chronicle volumes). in the first task, we chose to annotate the corpus manually because our goal was to create the largest possible gold standard annotated data set. this differed from the annotation tasks 2 and 3, where the goal was to explore whether a limited set of manually annotated data would be sufficient to train computer models that would be able to automatically label source mentions and attribution relations. in the subsections below, the three annotation tasks are discussed in more detail. ## usage, distribution and maintenance the corpus, including the transcriptions, meta-data, manual and automatic annotations and documentation has been made publicly available for future use under the creative commons license cc 4.0. all data is stored in a github repository. @xcite the digitized versions of published material are published on the dbnl website. the scans of the manuscripts that were uploaded in the transkribus tool are accessible side by side with their transcriptions and annotations on the read and search webchronicles-datasets. the corpus of chronicles is also of great value for the digital humanities and computational linguistics communities. to begin with, corpora of this size and diversity of historic variants are very scarce, especially for dutch. such a corpus will allow us to make progress in processing historic variants of dutch not only because it can be used to improve linguistic normalization tools, but also because it will allow users to train new tools. the additional layers of semantic annotation that will be provided with the corpus will allow the computational linguistics community to train new tools for the semantic processing of historical variants of dutch. the corpus can be used for research purposes, as well as for teaching purposes. students can be taught how to process this type of corpora with hands-on assignments. finally, the corpus can be used to organize international shared tasks on processing historic variants of languages. finally, users should take into account that although we believe that chronicles are a genre of texts that have much in common, the diversity in size, topics, writing styles, motives and the pro-ficiency of the authors make for a very heterogeneous corpus that sometimes hinders comparisons over time and across space. ## discussion our initial plan was not only to annotate a part of the corpus, but also to use the annotations to train machine learning systems to complete the annotations, for example to annotate source mentions and attribution relationships. however, this has proven to be more challenging than anticipated. our exploratory experiments on automatically labeling source mentions demonstrated that the mentioning of sources showed so much variation and complexity that the training set was still too small, and the model used (crf) was not the most powerful (lassche and morante, 2021; lassche, 2024). in ongoing experiments aiming to annotate attribution relationships automatically, similar challenges are arising. because the ways to automatically annotate data are rapidly expanding due to the swift developments in the field new avenues are opening for experimentation. we plan to train bert classifiers (devlin et al., 2019) and to annotate more data using generative models and appropriate prompting. the corpus has several limitations. first, due to limited budget, time and staff, it was not possible to annotate the full corpus. we manage to annotate 197 out of 308 volumes, which amounts to 63% of the corpus. for the same reason, only 179 volumes have normalised date labels. @xcite second, some errors and misreadings remain. the transcription and annotation tasks were carried out by volunteers with varying proficiency in paleography and comprehension of historical language. the team of experts could not correct all transcriptions themselves but was assisted in this task by a selected group of volunteers. third, apart from the earlier mentioned bias due to selection procedures by the team as well as the ravages of time, the following types of chronicles may be underrepresented in the corpus: chronicles written by women, chronicles written in rural areas, and chronicles written by lower class authors. moreover, chronicles that are part of private collections, smaller archives, smaller towns and especially in archives that have not yet @xcite an overview with all chronicles and their annotation status can be found on https://github.com/ chroniclingnovelty/chronicles-datasets/blob/ main/handleidingen/overview_chronicles.xlsx . ## conclusions we presented a corpus of 204 dutch language chronicles from the period 1500-1850 counting almost 24 million words. the corpus has been transcribed manually by volunteers combined with automatic hand written text recognition as offered in the transkribus tool. the transcriptions have also been annotated by volunteers in three annotation tasks: a first general annotation of named entities, mentions of dates as well as elements in the lay out of the pages such as images, printed matter, tables and copied text. a second task focused on the annotation of sources of information mentioned by the author as well as the receiver of this information and the medium of communication, and a third task focused on attribution relations. the result will be of value to both historians, students of historical literature as well as historical linguists. the additional layers of semantic annotation that are provided with the corpus will allow the computational linguistics community to train new tools for the semantic processing of historical variants of dutch. although a big effort was made to provide a quality resource, it was not possible to surmount some limitations posed by the magnitude of the project and the nature of textual data. in future work we will explore ways to surmount these limitations.