The Early M odern D utch Mediascape. Detecting Media Mentions in Chronicles Using Word Embeddings and CRF (2021)

Abstract: While the production of information in the European early modern period is a well-researched topic, the question how people were engaging with the information explosion that occurred in early modern Europe, is still underexposed. This paper presents the annotations and experiments aimed at exploring whether we can automatically extract media related information (source, perception, and receiver) from a corpus of early modern Dutch chronicles in order to get insight in the mediascape of early modern middle class people from a historic perspective. In a number of classification experiments with Conditional Random Fields, three categories of features are tested: (i) raw and binary word embedding features, (ii) lexicon features, and (iii) character features. Overall, the classifier that uses raw embeddings performs slightly better. However, given that the best F-scores are around 0.60, we conclude that the machine learning approach needs to be combined with a close reading approach for the results to be useful to answer history research questions.

## introduction it is well known from previous studies that early modern europe witnessed an eruption of news, which was due to developments such as the burgeoning business of print, the professionalization of postal networks, and the invention of the newspaper @xcite . while the production of information by various media is a well-researched topic, the question how people were engaging with this information explosion is still underexposed aside from a few studies @xcite . historians find it difficult to answer questions such as 'who owned book shelves?', 'who read which books?', and 'who bought newspapers?', since they have not yet found a suitable source and method to research these issues concerning the spread of information from the perspective of the consumer. a source of information that might contain the key to answer these questions are early modern chronicles. a chronicle is a text from someone who keeps a record of events happening in their surroundings, who believes that these events are worth recording, and that the best way to structure this information is to do so chronologically @xcite . to fill their chronicle with useful information, early modern chroniclers used a wide variety of information sources, such as pamphlets, newspapers, official announcements, gossips, conversations, songs and other sounds, letters from friends or relatives, or the chronicler themselves functioned as eyewitness. all the sources that a modern chronicler had available constitute the early modern 'mediascape', a term coined by @xcite . furthermore, the genre of the chronicle has remained rather stable over the centuries, and since many texts have been preserved in both public and private archives, a rather homogeneous collection of manuscripts still exists. this is why chronicles are a suitable source for historians to get more insight in the reception of news and information. two aspects are interesting to analyse from a history perspective: (i) the media that are mentioned by the author of the chronicle (such as 'my uncle', 'the newspaper', 'a rumour'), and (ii) the information that the chronicler is reporting on, which they have obtained from the media. these data are contained in the chronicles and traditionally close reading methods would be applied. however, in order to generalise across chroniclers, it is necessary to analyze as many chronicles as possible. since close reading is a time consuming process, computational methods should be applied to automatically find the data. this is the goal of this study. we focus on the first aspect (media) and present a computational approach to automatically extract media related information from dutch early modern chronicles. this has been possible thanks to the availability of a corpus of 350 dutch chronicles from the period 1500-1850 fully digitized. our research question is: to which extent is it possible to automatically extract media mentions from a corpus of early modern dutch chronicles? addressing this issue is relevant because having this information available would allow historians to find out which books chroniclers owned, which newspapers they read, and in what other ways they received new information. it also provides insights in how the media use of chroniclers differs among each other. it would be possible to investigate, for example, how the mediascape of a chronicler from a large city such as ghent, relates to that of a chronicler from a small provincial town such as hoorn. furthermore, it tells us something about how the preferences for certain media develop over time. the broader context of this study is to explore whether the use of computational methods can help historians to analyse the data concerning the mediascape of early modern chroniclers. 1 section 2 presents related work, section 3 describes the data, section 4 the methods, section 5 the experiments, and section 6 the results. finally, in section 7 we put forward some conclusions. ## related work historical text mining is an incipient field @xcite . regarding the dutch language, historical text mining studies are limited. stylometric methods have been applied in authorship verification tasks on medieval and early modern texts @xcite . furthermore, dutch early modern songs have been quantitatively researched @xcite . corpora of historical dutch newspapers have been subject to research in several studies, including @xcite , @xcite . this work is the first to investigate chronicles on a large scale and with the use of text mining methods. early modern dutch chronicles have only been qualitatively researched up until now @xcite @xcite . the same goes for chronicles in other languages, such as german, where only case studies on a handful chronicles have been published @xcite . word embedding models (or word vector mod-els) have recently become a popular way to describe a corpus of texts. well known implementations are word2vec and fasttext @xcite . regarding the dutch language, they have been used for relation evaluation and dialect identification @xcite , and to detect semantic change in dutch newspapers @xcite and parliamentary debates @xcite . word embeddings have been used as features in classification tasks (see for example @xcite ), but to the best of our knowledge, this has never been done for the historic dutch language. studies with corpora in other languages show that there are multiple ways to implement the word embeddings as features of a classifier. in @xcite , every one hot encoding word representation is simply replaced with its corresponding 50-dimensional embedding vector. the authors of @xcite compared three strategies for deriving distributed word representations in a corpus of english clinical texts. they base their method on earlier work @xcite , in which three different approaches for utilizing the embedding features are compared. in @xcite it is shown that the binarized word embedding features caused the largest improvement of the performance of the crf system, while @xcite concluded that a distributional prototype approach performed the best. @xcite perform agglomerative hierarchical clustering of the embedding vectors to take into account all dimensions of a vector simultaneously. dařena and süss (2020) use a neural network that is allowed to update the word vectors during training, since their czech corpus from which a word embedding model was build, was rather small. as far as we know, the current study is the first in which historic dutch word embeddings are used as features of a classifier. we also do not know of any studies in other historical languages, in which a similar classification task is performed. ## data in the context of the research project 'chronicling novelty. new knowledge in the netherlands, 1500-1850', led by judith pollmann (leiden university) and erika kuijpers (vu), a corpus of about 350 early modern dutch chronicles has been built over the last three years. 2 the texts included in the corpus have the following characteristics: they are written between 1500 and 1850, they are organised chronologically, they cover events that happened in the lifetime of the author, and they focus on local events more than national, individual or familial. about 100 of these chronicles had been published before as a contribution to a journal, on the initiative of an archive, or in the private domain, and were digitised by the dbnl (digital library for dutch literature). the other 250 chronicles were kept in libraries and archives throughout the netherlands and belgium. every manuscript page was scanned, and afterwards transcribed with both the handwritten text recognition tool transkribus @xcite , and the help of volunteers on the online crowd sourcing platform vele handen. currently, the corpus contains about 70,000 pages. in order to improve the searchability of the corpus of chronicles, a second group of volunteers is labeling useful information in the completely transcribed chronicles. they annotate both content and lay out in the chronicle pages, using labels including date, location, and person name, as well as page number, lists and tables, and copied text. besides, a select group of volunteers has labeled media mentions in the chronicles. these are the annotations that are used in this study, which will be discussed in more detail in the next subsection. ## methods the manually labeled data was used to train a crf-classifier that is able to recognize media mentions in unseen chronicles. we use the python library sklearn-crfsuite and the default parameters of crf @xcite . the classification task consists in labelling tokens with one of the following labels: o (no medium mention), source-b, source-i, receiver-b, receiver-i, perception-b, or perception-i. conditional random fields is a frequently used method in named entity recognition (ner) labeling tasks, because crf is able to deal with sequential data implicitly, without adding this information as features. since the media mentions are tagged on token level using the iob-tagging scheme, this task is also one in which labels of consecutive points can influence each other. there are, however, a few aspects in which this labeling task differs from a ner-task performed on modern texts, and complicates it. preprocessing of the data was limited: upper cases and punctuation were not removed, since these can be an indication of a media mention (consider the 'gentsche gazette'). since chronicles were handwritten by authors from different areas, different time periods, and with different levels of literacy and dialects, there is a large variation in the vocabulary used. as a result of that, features such as pos-tags -which are often used in classification tasks that are performed on modern textscan not be used in this task. previous studies have shown, though, how the use of word embeddings as features of a classifier can improve its quality @xcite @xcite . vectors of historical spelling variants are often each other's nearest neighbors. the hypothesis is therefore that using them as features will positively affect the quality of a model. ## conclusions in this paper we have presented experiments aimed at extracting media related information from early modern dutch chronicles with the purpose of investigating to which extent it is possible to perform this task automatically and to which extent are the results of automatic systems useful to facilitate history research. we have described the manual annotation task, which consisted in labeling sequences of tokens with the labels source, perception and receiver, considered by historians to be of relevance for their research. the iaa scores, with f-scores of around 0.73, show that the annotation task is well defined but that it is not easy. the label that produced more confusion was source, due to different interpretations of when to assign this label. four fully annotated chronicles were used to perform experiments with a crf algorithm and different types of features that included several models of raw and binary word embeddings. we performed evaluations on token and sequence level. both reveal that the systems obtain a much better precision than recall. in any case, the results of baseline experiments without word embeddings are only 0.04 f-score lower than the experiments with word embeddings, indicating that the word embedding models are not rich enough, probably due to the size of the corpus used to generate them. experiments with set 2, which contains more variety in authors, and therefore also in vocabulary, benefit more from word embedding features than experiments with set 1, where only one chronicle was used. the classifier with raw embeddings performs slightly better than the one with binary embeddings. as expected, the scores of the sequence level evaluation are lower than the scores of the token level evaluation. in this paper we aimed at determining whether it was possible to automatically extract media related information from chronicles and whether the extracted information can be used by historians to answer their research questions. since the sequence level partial match f-scores are not higher than 0.51 for receiver, 0.38 for perception and 0.44 for source, we conclude that the methods that we applied are not precise enough to extract the information and that it is not possible for historians to solely use such a system to obtain data to answer their research questions because too much information is missed. both the data used to train the classifiers and the methods need to be improved. for future work we aim at experimenting with more algorithms and with larger corpora that include more chronicles, both for creating the embedding models and to train the systems.