To the Most Gracious Highness, from Your Humble Servant: Analysing S wedish 18th Century Petitions Using Text Classification (2022)

Abstract: Petitions are a rich historical source, yet they have been relatively little used in historical research. In this paper, we aim to analyse Swedish texts from around the 18th century, and petitions in particular, using automatic means of text classification. We also test how text pre-processing and different feature representations affect the result, and we examine feature importance for our main class of interest - petitions. Our experiments show that the statistical algorithms NB, RF, SVM, and kNN are indeed very able to classify different genres of historical text. Further, we find that normalisation has a positive impact on classification, and that content words are particularly informative for the traditional models. A fine-tuned BERT model, fed with normalised data, outperforms all other classification experiments with a macro average F1 score at 98.8. However, using less computationally expensive methods, including feature representation with word2vec, fastText embeddings or even TF-IDF values, with a SVM classifier also show good results for both unnormalise and normalised data. In the feature importance analysis, where we obtain the features most decisive for the classification models, we find highly relevant characteristics of the petitions, namely words expressing signs of someone inferior addressing someone superior.

## introduction in many pre-modern and pre-democratic societies, ordinary people had the right to address those in power through written petitions in order to ask for help or confirmation of existing rights. petitions usually addressed a social and economic superior, for example a court of law, a parliament, a landlord, or even the monarch @xcite . in other words -petitions allowed the powerless to speak to the powerful. petitions are a rich historical source that could answer questions about the everyday life of ordinary people in the past. even so, petitions have been relatively little used in historiography. for this reason, we are involved in an interdisciplinary research project at uppsala university, funded by the swedish research council, with the goal of enhancing accessibility to and knowledge of swedish 18th century petitions, and using this source to answer questions about people's ways of supporting themselves and claiming rights in the past. 1 this project, titled "speaking to one's superiors: petitions as cultural heritage and sources of knowledge", is coordinated by the gender and work (gaw) research project, conducted at the department of history, uppsala university. the gaw project studies how women and men sustained and provided for themselves in sweden in the period from 1550 to 1800. as part of the project, thousands of historical sources have been gathered, classified and stored in a unique database that has been made accessible for researchers, students, and the general public @xcite . the computational linguistic part of the project aims to contribute to the field of digital philology and the development of automatised historical text analysis. in this paper, we explore computational approaches, more specifically text classification and feature importance, as means to study petitions and other historical documents. firstly, we examine the possibility to distinguish petitions from other historical texts using different automatic classification methods. if possible, we also want to see what sort of features that characterise different genres of historical texts, and petitions in particular. due to the noisy nature of historical data, as well as generally limited resources, we are also interested in studying how much is gained when using different variants of pre-processing methods, and how to best represent our data for a classification task. we show that the different text genres in our data set are certainly possible to classify, using both more state-of-the art and traditional methods. we also find that our approach to feature importance analysis, where we obtain the features most decisive for some of the classification models, indeed finds highly relevant and interpretable characteristics of the petitions. as a third step, which we plan to proceed with in future work, we want to examine the possibility to distinguish different parts of the petitions. research suggests that petitions follow a certain structure, based on a classical rhetorical division @xcite . it would be interesting to investigate how informative specific parts of the petitions are to a classification task, or where the most relevant features are placed. we hope that our work can facilitate the task of information extraction for historians and other scholars interested in studying petitions further. ## related work text classification (tc), the task of assigning text documents to one or more predefined categories, has traditionally been solved by using supervised learning algorithms such as naive bayes (nb) @xcite , random forest (rf) @xcite , support vector machines (svm) @xcite and k-nearest neighbor (knn) @xcite . tc could be implemented either topicbased, paying attention to what the text is about, or stylistic, being more concerned with how a text is written. while topic-based categorisation often uses models based on "bags of content words", style is somewhat more elusive and can include, but is not limited to, the use of function words and syntactic structures @xcite . for historical texts, a common application for tc is automatic dating of documents @xcite . as with most nlp applications, the raw data used for tc typically undergoes several steps of text pre-processing, though the best pre-processing strategy might differ depending on the data set and the tc algorithm at hand @xcite . fewer pre-processing steps and less need of annotation could be particularly advantageous for historical text, since its spelling variations, possible ocr-errors and limited resources of (annotated) data pose challenges for nlp tools. a common approach to tackle spelling variations is to view it as a translation task, where character-based statistical machine translation (smt) @xcite and corresponding neural methods (nmt) @xcite have proven to work well. @xcite points out that while neural approaches have become popular for a variety of nlp tasks, there is no clear consensus about the state-of-the-art for the task of normalisation. to the best of our knowledge, no method yet has substantially outperformed a character smt-based approach for historical swedish. an important question in tc is how to represent the documents of interest as input to the machine learning algorithms, where common techniques include bag-of-words (bow) representation in the form of term frequencies or tf-idf values, or distributed representations of words in the form of word embeddings @xcite , such as word2vec @xcite or fasttext @xcite . more recent, deep neural language models such as bert @xcite produce contextualised word vectors that are sensitive to the context in which they appear. such a pretrained model is commonly fine-tuned to perform a specific task, such as text classification, simply by changing the final output layer. however, due to memory limitations, the maximum length for the input sequence is limited, which is problematic for long documents, although @xcite have shown that state-of-the-art results can be obtained with 512 tokens, by concatenating text from the head and tail of a document. another potential challenge when using large language models such as bert, especially relevant for historical data, are observed instabilities when fine-tuning with small data sets @xcite . instead of applying techniques to standardise variations in orthography, one could also develop tools that are trained on text more similar to the target data. @xcite have released a collection of swedish diachronic we models trained on historical newspaper data. their models include word2vec and fasttext models, trained on 20-year time bins from 1740 to 1880, with two temporal alignment strategies: independentlytrained models for post-hoc alignment, and incremental training. as described in section 1, the petition project seeks to use historical sources to study how ordinary people claimed their rights and what they did for a living. the latter has been approached by computational manners within the coordinating gaw project, using historical court records and church documents as a source, by implementing a verb-oriented approach to find text passages de-scribing work activities @xcite . this has also resulted in a web-based tool for automatic information extraction from historical text @xcite . though, to use petitions as a source of information has, to the best of our knowledge, not yet been approached by computational manners. ## data collection as part of our project, we make use of a transcribed collection of 18th century petitions submitted to the regional administration in örebro, sweden. in order to compare the petitions to other relevant historical documents, we select data from other genres based on the following criteria: (a) each genre should to be fairly easy to divide into smaller documents in an automatic or semi-automatic manner, (b) the selected genres, and each document within them, should be reasonably similar to the petitions in terms of size (number of tokens) and time period, and (c) the selected genres should vary in terms of similarity in content to the petitions (to the best of our knowledge), with the purpose of having some variety in challenge for the classification models. given that transcribed historical documents are a limited resource, it is not possible to meet all criteria for every genre, though we strive to come as close as possible. our selected genres, which will be referred to as classes from now on, can be viewed in ## conclusions in this paper, we present a study of text classification and feature importance applied to historical swedish text, with a special focus on petitions. we test the performance of both traditional and newer classification algorithms, and we examine how text pre-processing and different types of feature representation affect the result. we also analyze feature importance for our main class of interest: petitions. the text classification results show that the statistical classification algorithms nb, rf, svm, and knn are indeed very able to distinguish between our different classes of historical text. we also find that pre-processing in the form of normalisation has a positive impact on the classification models, and that content words are particularly informative. using a normalised and lemmatised version of our data set classified with an svm classifier achieves a macro-averaged f1-score at 96.5, and only targeting content words with a knn model pushes the score up to 97.0. we also test how to best represent our data for a classification task. using a more state-of-the-art method, a pre-trained bert model, fine-tuned for our classification task and fed with a normalised version of our data set outperforms all other classification experiments with a macro average f1 score at 98.8. however, using much less computationally expensive methods with an svm classifier also show quite good results for both a raw and a normalised version of our data set. for the raw data, using fasttext embeddings, trained on historical swedish texts, gave the best f1 score at 93.9. for the normalised data, fasttext embeddings trained on contemporary swedish resulted in a f1 score at 95.1. even using such a simple approach as tf-idf values in combination with an svm classifier gave quite good results, both for the raw and normalised data set, with f1 scores at 92.7 and 95.0, respectively. we believe that this could be explained by the small amount of data used, and also that the classes in our data set have characteristics that the classifiers are able to differentiate quite effectively. in the feature importance analysis, we make use of our text classification task and obtain the features most decisive for some of the classification models. we find that this method reveals features that are highly relevant and interpretable characteristics of the petitions, namely tokens that express signs of someone inferior addressing someone superior. for future work, we are interested in further exploring text classification and feature importance as methods to analyse petitions. as mentioned in section 1, research indicate that petitions follow a certain disposition. with this in mind, and given our results in this paper, we plan to investigate if petitions could be segmented by automatic means. if possible, we also want to examine where the most relevant features typically are placed. the main goal with these steps would be to facilitate and improve the task of information extraction for historians and other scholars interested in study-ing petitions further. another area of improvement would be to test if other methods of classification would work better for a small, historical data set such as ours, in particular additional deep learning techniques. it would be interesting to make use of new generations of language models adapted to historical texts, if available.