People and Places of the Past - Named Entity Recognition in S wedish Labour Movement Documents from Historical Sources (2024)

Abstract: Named Entity Recognition (NER) is an important step in many Natural Language Processing tasks. The existing state-of-the-art NER systems are however typically developed based on contemporary data, and not very well suited for analyzing historical text. In this paper, we present a comparative analysis of the performance of several language models when applied to Named Entity Recognition for historical Swedish text. The source texts we work with are documents from Swedish labour unions from the 19th and 20th century. We experiment with three off-the-shelf models for contemporary Swedish text, and one language model built on historical Swedish text that we fine-tune with labelled data for adaptation to the NER task. Lastly, we propose a hybrid approach by combining the results of two models in order to maximize usability. We show that, even though historical Swedish is a low-resource language with data sparsity issues affecting overall performance, historical language models still show very promising results. Further contributions of our paper are the release of our newly trained model for NER of historical Swedish text, along with a manually annotated corpus of over 650 named entities.

## introduction the swedish labour movement is strong by tradition and has played a crucial role in the development of the swedish welfare society and in shaping the structure of the labour market. the swedish trade union federations play an important role internationally, and their archives offer a unique possibility to study the development of the trade unions and their key topics over time, and thereby also the social development nationally and internationally. in the project labour's memory. digitization of annual and financial reports of blue-collar worker unions 1880-2020 , we aim to collect and digitize annual and financial reports from local, regional, national and international trade union organisations from 1880 onwards. the collection is to be stored in a database, and will be made searchable for people with an interest in diachronic labour movement documents through a user portal. this is achieved in collaboration between labour history experts, archivists, computational linguists and image analysis specialists. in this paper, we aim to investigate to which extent current state-of-the-art models for swedish can be used to extract named entities from historical sources, a key topic for enhanced searchability in the trade union documents. secondly, we focus on maximizing usability for the intended end product by combining the strengths of different models. last but not least, we evaluate the performance of these models in terms of accuracy, as well as f1 score. on a more practical level, we also release a new model that is fine-tuned for ner and trained on historical swedish text, as well as a manually annotated gold-corpus of named entities extracted from swedish labour union documents dated between 1892 and 1974. the choices that we made in order to optimize the results and usability of our system were made in consultations with a group of experts from the labour's memory project, whose competence overlaps with that of the intended user. ## background named entity recognition (ner) is the process of automatically identifying and classifying namelike entities in text, such as names of persons, organizations and locations @xcite . ner is an important subtask in many natural language processing (nlp) applications, e.g., in information extraction/retrieval (see for example @xcite ) and for anonymisation/pseudonymisation of sensitive personal data in a text (e.g. @xcite or @xcite ). for swedish, researchers have recently worked with developing a gold standard for swedish named entity recognition @xcite , trying to merge and accommodate previous ner annotation schemes used for swedish. there are also initiatives to adapt this standard to the task of annotating named entities in historical swedish text, where the needs and features to consider differ slightly @xcite . outside the topic of ner itself, it is important to acknowledge that swedish is still a low resource language, which does not have the same largescale nlp infrastructure as other high-resource languages such as english, spanish or chinese. this is evident in terms of data sets, language models and tools, and even more so in the case of historical swedish text. while there are efforts currently being made to build large language models for swedish by organizations such as ai sweden, foot_0 or historical resources from the side of swe-clarin (e.g. pettersson and borin (2022) ), it is still a tough undertaking to achieve high benchmark scores for nlp tasks on swedish. ## method the aim of our work is to perform named entity recognition (ner) for trade union documents from the late 1800s and onwards, with the goal of enhancing searchability in the documents by automatically extracting metadata on persons, locations, organisations, events etc. an important subtask is therefore to adapt our tools to handling historical text, which is further elaborated on in section 3.1. we move on by describing the different language models we use for the ner task in section 3.2. in section 3.3, we introduce the evaluation method that we use, and the gold standard created for this purpose. ## results in order to identify which approach gives us the best results, we calculate the accuracy of each model, and in doing so we also look at the count for each type of match per individual model. by accuracy we mean the percentage of entities from the gold standard which have a counterpart in the output predicted by the model, regardless of the type of match. the matches we look at are detailed in section 3.3, and we do not include the counts for fp and fn in calculating accuracy. we also calculate the (accuracy) count for all the different kinds of matches, as well as ner labels, which we present in figures 2 through 10. we calculate both these metrics first on the original text, then on the normalized version of the text in the gold standard. our goal is to reach as many exact matches as pos-sible (at best) and to minimize the number of cases where the model returns no match for a given entity in the gold standard (fn). as a secondary step, we provide calculations for f1 score as applied to our text, since f1 score is a widely used metric for ner, and which also accounts for the fp and fn tags in our case. we present our results for f1 score in section 4.4 ## discussion our study focuses on the application of ner to historical swedish text, specifically documents sourced from swedish labor unions dating back to the 19th and 20th centuries. the primary challenge lies in adapting contemporary state-of-the-art ner systems to effectively process and extract entities from historical text, which often differs significantly in linguistic norms, vocabulary, spelling, and syntactic structures from contemporary swedish. our research delves into a comparative analysis of multiple language models applied to ner for historical swedish text. three off-the-shelf models designed for contemporary swedish text were experimented with, alongside a custom-built language model trained on historical swedish text. this unique approach allowed us to explore the adaptability of existing models and assess the feasibility of fine-tuning historical language models for ner tasks. through our experiments, we show that current off-the-shelf models have the capability to extract named entities from historical text, but at the same time they can benefit from training on historical data, as shown by the high accuracy of our ra model. moreover, we believe that the inconsistent effect of the normalization rules could be partly due to the rather small amount of normalization rules, as well as the nature of a rule-based approach to spelling normalization, where it is hard to write efficient rules without risking overgeneration. another, more data-driven approach, might have given more consistent results. it is also important to keep in mind that our evaluation metrics were customized according to the needs of our future users. since our target groups are looking for as many named entities as possible, we attempt to adapt our approach in order to maximize the usability for the end product -which is the archival database of the labour's memory project, while at the same time maintaining a good level of quality for the automatically extracted named entities. for future work, we would like to investigate the way different data augmentation methods can improve our results, since previous work done on english text shows promising results when it comes to applications on pretrained transformer models (see, for example, dai and adel (2020)), such as the ra model we propose in this paper. moreover, given the fact that our source material comes from labour union documents, it could also be interesting to look at a more fine-grained analysis of the prs label in order to be able to identify potential biases in ner systems -similar to the work conducted by @xcite for danish text. ## conclusion in this paper, we show that current off-the-shelf models for swedish can perform ner on historical text, but using a historical language model shows more promising results. however, data from historical sources could also be beneficial for training in order to achieve better f1 score and reduce errors. an alternative path we would like to explore in the future is training on multilingual data from other scandinavian languages, given that multilingual models show great promise when it comes to crosslingual transfer learning (see, for exasmple, @xcite or @xcite ), with the added bonus that scandinavian languages have similar vocabulary and structure. a significant contribution of this study lies in the release of the newly trained ra model tailored for ner of historical swedish text. additionally, we introduce a manually annotated corpus comprising over 650 named entities, offering a valuable resource for future research endeavors. we also show that combining the strengths of multiple models can be beneficial to our ner task. in conclusion, while our study provides valuable insights and tools for ner in historical swedish text, it also underscores the necessity for further advancements and novel methodologies to address the challenges posed by data sparsity in low-resource languages.