Modeling Moravian Memoirs: Ternary Sentiment Analysis in a Low Resource Setting (2024)

Abstract: The Moravians are a Christian group that has emerged from a 15th century movement. In this paper, we investigate how memoirs written by the devotees of this group can be analyzed with methods from computational linguistics, in particular sentiment analysis. To this end, we experiment with two different fine-tuning strategies and find that the best performance for ternary sentiment analysis (81% accuracy) is achieved by fine-tuning a German BERT model, outperforming in particular models trained on much larger German sentiment datasets. We further investigate the model(s) using SHAP scores and find that the best performing model struggles with multiple negations and mixed statements. Finally, we show two application scenarios motivated by research questions from religious studies.

## introduction while not entirely uncontroversial (cf. @xcite , ego-documents (i.e. documents in which humans write about themselves and their experiences) are an important source of historical research @xcite @xcite . in this paper, we focus on one specific kind of ego-document, often called memoir: semi-autobiographical records written by members of the moravian church in the 18th century. in line with general migration movements at that time, many moravians migrated from europe to america. the semi-autobiographical records we investigate are the result of a custom among moravians to document their lives in written form. as they were completed, compiled and collected by other members of the respective local church @xcite , we consider them semiautobiographical. religiously, the moravians are connected to the so-called "blood and wounds" theology @xcite , dating back to their founder, nikolaus ludwig von zinzendorf (1700-1760). next to blood, the "wounds jesus suffered on the cross became the main focus of this religious attention" @xcite . the memoirs are also believed to express a high degree of emotionality @xcite , which is why we focus on sentiment analysis in this paper, while also taking into account that emotionality found in the text is not necessarily only rooted in emotions of the person the memoir is about. from a linguistic standpoint, these sources exhibit regional variation and domain-specific terms, some of them connected to the "blood and wounds" theology. we therefore experiment with multiple ways of assigning sentiment scores, and explore the gain by adapting these systems to the specific domain and text genre. we also investigate how to visualize what these models actually have learned, and provide two application scenarios motivated by research interests from religious and historical studies. in the following sections, we outline connected fields of research first. then, we go into details about how we compiled our dataset and what kind of analyses we conducted before discussing our findings. ## related work this paper has links to multiple research areas from computational linguistics (cl) and digital humanities (dh). ## results we performed two types of model evaluations, namely looking at raw performance scores on the one hand and explainability attempts on the other. the following subsections present our findings. ## applications to illustrate some of the analyses made possible through sentiment assignments of the moravian memoirs, we showcase two content-wise analyses of interest to the moravian community. ## conclusion in this paper, we introduced a manually annotated dataset for ternary sentiment analysis of german memoirs of moravians that lived in the 18th century. the prevalence of non-neutral samples in it attest that sentiment in particular and emotions in general are important features of this domain which is in line with theological research @xcite . we also introduced bert models fine-tuned on this dataset that not only outperform existing transformer models and lexicon-based approaches but also reach or even surpass state-of-the-art results @xcite . this was not only true for performance statistics like accuracy and f1 scores but also for two research driven applications. here, we found that german memoirs of women tend to be more positive than those of men and that a positively framed "blood and wounds" theology can be observed empirically. both findings confirm results from various moravian research projects @xcite @xcite . they also show that the minimum performance level required may depend on the downstream task at hand. concerning model explainability, we showed that a deeper look at f1 score distributions and confusion matrices can already give some hints on the classification behavior and possible problems related to this. these results can be enriched by applying local explanation approaches like shap to a whole dataset. this revealed in our case that gbert actually learned sentiment bearing lemmata during fine-tuning and that a neutral class has to be inferred ex negativo. the base english bert model, by contrast, focused more on rather random imbalances in our test dataset like negations and punctuation marks. the latter was also observed in a related nlp task @xcite . we suspect that these differences might be due to the different tokenizers involved as the german one tends to split fewer word forms that may carry sentiment information. from these observations we draw the conclusion that state-of-the-art performances for ternary sentiment analysis can already be reached with less than 2k fine-tuning samples. this makes transformerbased approaches feasible in low resource settings even more so since individual model predictions can be explained to a certain degree.