Can Stanza be Used for Part-of-Speech Tagging Historical P olish? (2024)

Abstract: The goal of this paper is to evaluate the performance of Stanza, a part-of-speech (POS) tagger developed for modern Polish, on historical text to assess its possible use for automating the annotation of other historical texts. While the issue of the reliability of utilizing POS taggers on historical data has been previously discussed, most of the research focuses on languages whose grammar differs from Polish, meaning that their results need not be fully applicable in this case. The evaluation of Stanza is conducted on two sets of 10286 and 3270 manually annotated tokens from a piece of historical Polish writing (1899), and the errors are analyzed qualitatively and quantitatively. The results show a good performance of the tagger, especially when it comes to Universal Part-of-Speech (UPOS) tags, which is promising for utilizing the tagger for automatic annotation in larger projects, and pinpoint some common features of misclassified tokens.

## introduction and background annotated data for historical or otherwise nonstandard variants of language can be difficult or resource-consuming to obtain but is nevertheless necessary for certain linguistic inquiries. one of the possible methods of alleviating this issue is attempting to use tools developed for a contemporary standard language for automated annotation. however, the data in question differing from the standard may pose problems. consider the example presented in ## materials and methods the tagger used in this project is that provided by stanza, a natural language processing (nlp) toolkit featuring models for a large number of languages @xcite . the default model for polish was trained and evaluated on the polish dependency bank treebank @xcite . it is also that corpus's test set that is used to exemplify the tool's performance on modern polish in this paper, although it represents genres different from the historical texts. the main reasons for selecting this tagger are its ease of use and high reported accuracy on modern data. the data used for testing the tagger in this project comes from the memoir of juliusz czermiński, who lived in the 19 th century in the area corresponding to nowadays eastern poland and western ukraine. the original manuscript was composed in 1889, retyped on a typewriter, and recently digitized. no intentional alterations were made to e.g. seemingly misspelled tokens. this data was first presented by szawerna (2023), where its divergence from modern polish was asserted, especially when it comes to features typical for the dialects of that region @xcite . according to @xcite , there was no singular universally accepted spelling convention around the time of the memoir's creation. therefore, the text should not be considered to be representative of historical polish of its time, both due to its dialectical features and spelling which is not representative of the bulk of the contemporaneous writing. in its entirety, the data consists of 37,405 tokens. out of those, the first 10286 tokens were manually annotated using universal dependencies' universal pos tags (upos tags). a subset of 3270 tokens was further annotated using xpos tags. both of these tagsets are utilized by stanza. the only changes to the original text include the splitting of the "mobile inflection" as per the ud guidelines and removing any punctuation from inside numbers (szawerna, 2023; universal dependencies, n.d.). this previously conducted manual annotation of the tokens has been reviewed, and a few corrections have been made. evaluation measures were calculated for both kinds of annotation. the results were also subjected to a qualitative analysis, the goal of which was to determine what kinds of errors are the most prevalent, which could give insights into what kinds of potential pre-processing could eliminate that problem. the misclassified examples were saved and manually annotated for the error type before being processed to obtain the relevant statistics. ## results stanza exhibits very good performance on modern polish data and relatively good performance on historical data. another method of inspecting the tagger's performance is investigating the erroneously labeled tokens. ## discussion the results of the quantitative evaluation show a good performance of the tagger, exceeding most of the previously reported ones, including the results reported for the same data and tagger by szawerna (2023), foot_3 possibly due to improvements that have been made to stanza's model. on the other hand, @xcite still achieve a better performance on xpos tags using a crf-based model. however, they use a more diverse and larger dataset which may consist of more standard polish than the data investigated in this paper. nevertheless, stanza's performance on this test data is only around 4 (upos) and 7 (xpos) percentage points below the accuracy it has shown on its own test set. interestingly enough, the performance on the pdb test set is slightly higher than reported by stanza @xcite , possibly due to the corpus being pre-tokenized before being fed to the model. a qualitative error analysis has approximated what the tagger struggles with when it comes to the test data. previous studies have shown that variations in spelling, capitalization, punctuation, differences in morphology and syntax, and semantic shifts are some of the factors that make accurate tagging of historical texts using modern taggers difficult @xcite @xcite . in the case of stanza, some of those issues, such as nonstandard capitalization, archaic vocabulary, and spelling have negatively impacted the tagger's performance. this is particularly prominent as far as upos tagging is concerned. as far as xpostagging goes, issues pertaining to the inflectional morphology have been highlighted, such as confusing word endings or problems with words the class of which is ambiguous. additionally, issues such as the possible underrepresentation of rarer classes in the training corpus could be noted, leading to biases concerning feminine pronouns and issues identifying the vocative case. ## conclusions and future work within this paper, a modern polish pos tagger, stanza, has been evaluated on historical and modern data, and some of the issues causing the drop in its performance on historical texts have been successfully identified. it has been shown that it can perform quite well on non-standard, historical polish data from the late 19 th century, and this can possibly be improved using some preprocessing methods, making it a promising candidate for at least assisting the annotation of historical texts, if not completely automating it. many of the misclassified tokens were problematic due to issues previously identified in the literature in the field; however, some problems seemed to stem from the inflectionality of the language or be inherent to the tagger itself. potential biases stemming from the under-representation of certain classes in the training data for the tagger have also been shown. in the future, it would be interesting to test the influence of various factors, such as e.g. punctuation or lowercasing, on the quality of tagging. another possibility could be comparing the performance of multiple different taggers or tagging architectures on the same data, or testing the same tagger on data from different periods. alternatively, one could juxtapose the results presented in this paper to those from tagging a very recent, nonstandard text, e.g. sourced from the web, to see to what extent the same issues are causing tagging problems. finally, developing some methods for the pre-processing of texts from this period for subsequent tagging could also be quite useful. it would also be interesting to compare how the models for other languages included in stanza perform on samples of historical texts from their respective languages. as far as the data itself is concerned, it would be interesting to complete and review the annotation of the entire memoir, and see how the results of an analysis such as the one presented in this paper would change; this would also open up the opportunity for different kinds of research on the text.