Emotion Classification in G erman Plays with Transformer-based Language Models Pretrained on Historical and Contemporary Language (2021)

Abstract: We present results of a project on emotion classification on historical German plays of Enlightenment, Storm and Stress, and German Classicism. We have developed a hierarchical annotation scheme consisting of 13 sub-emotions like suffering, love and joy that sum up to 6 main and 2 polarity classes (positive/negative). We have conducted textual annotations on 11 German plays and have acquired over 13,000 emotion annotations by two annotators per play. We have evaluated multiple traditional machine learning approaches as well as transformer-based models pretrained on historical and contemporary language for a single-label text sequence emotion classification for the different emotion categories. The evaluation is carried out on three different instances of the corpus: (1) taking all annotations, (2) filtering overlapping annotations by annotators, (3) applying a heuristic for speech-based analysis. Best results are achieved on the filtered corpus with the best models being large transformer-based models pretrained on contemporary German language. For the polarity classification accuracies of up to 90% are achieved. The accuracies become lower for settings with a higher number of classes, achieving 66% for 13 sub-emotions. Further pretraining of a historical model with a corpus of dramatic texts led to no improvements.

## introduction transformer-based language models like bert @xcite and electra @xcite have recently gained a lot of attention and achieve state-of-the-art results for various tasks in natural language processing (nlp) @xcite . these language models are usually trained via deep learning on large amounts of texts acquired from the internet. unlike previous methods in nlp, these models use context-sensitive word representations and they can better deal with out-ofvocabulary words. these attributes are, of course, advantageous for various text sorts in digital humanities (dh) and computational literary studies (cls). furthermore, transformer-based language models can be adapted to specific domain texts by either training a model from scratch on large amounts of these texts or taking an existing model and further pretraining it with domain-specific texts @xcite @xcite . transformer-based models as well as these approaches have been successfully applied in dh contexts with historical or poetic german texts for named entity recognition (ner) @xcite and speech type recognition @xcite . we present a study for the task of textual emotion classification in the same line of research for the use case of german historical plays. emotion classification deals with the prediction of (multiple) emotion categories in text. its neighbouring field sentiment analysis primarily focuses on the prediction of the overall polarity (or valence) of a text, meaning if it is rather positive or negative @xcite . both methods have been explored in dh and cls to analyze emotion/sentiment distributions and progressions in social media @xcite or literary texts like plays @xcite @xcite , novels @xcite and fairy tales @xcite (see @xcite for an in-depth review of this research area). however, as the review of @xcite and recent tool developments in dh show @xcite , the application of rather basic lexiconbased methods is frequent although these methods are usually outperformed by more modern approaches in sentiment and emotion classification @xcite @xcite and are especially problematic for literary texts @xcite . furthermore, performance evaluation of computational approaches compared to human annotations ("gold standard") are rare. thus, we present an evaluation study for the use case of german historical plays (from enlightenment, storm and stress and german classicism) for emotion classification. our goal is to develop emotion classification algorithms with a satisfactory performance for the described use case to investigate in later stages of our research, for example, emotion progressions throughout time or genrebased differences concerning emotion distributions on a larger set of plays. we primarily focus on current state-of-the-art transformer-based language models. the main contributions of this paper are as follows: (1) the development of an emotion annotation scheme directed towards the interest of literary scholars for the time frame of our corpus, (2) the annotation results for the annotation of 11 plays by 2 annotators for each play, (3) a systematic evaluation of traditional textual machine learning (ml)-approaches, transformer-based models pretrained on contemporary and historical language and further pretrained on dramatic texts on different instances of the annotated corpus. the goal of this contribution is to work towards the development of emotion classification algorithms with a satisfactory performance for the described use case. ## training and evaluation corpus in the following section, we describe the conceptual framework and process for the acquisition of the annotated corpus that serves as training and evaluation corpus for the emotion classification ("gold standard"). ## emotion classification methods we regard the emotion classification as single-label classification on text sequences of varied lengths. the amount of classes differs depending on the hierarchical system: polarity (2 classes), triple valence (3 classes), main classes (6), sub-emotions (13). we have implemented reference baselines based on traditional ml-approaches but otherwise focus on transformer-based language models for german pretrained on contemporary and historical texts since transformer-based models have been shown to achieve state-of-the-art results for emotion classification @xcite @xcite and performed best in a pre-study @xcite . we also explore further fine-tuning/pretraining of a pretrained model with our domain texts since research suggests performance improvements for this method @xcite @xcite . ## results we report accuracies and f1-scores for all models and category systems as well as corpus manifestations in tables 3, 4 and 5. considering f1-scores, we report weighted f1 due to the imbalanced class distributions. in general, transformer-based models outperform traditional ml-approaches. for every corpus manifestation the performance of the different transformer-based models is rather similar regardless whether contemporary or historical language is the basis for the pretraining. the best models are the large contemporary models gbert-large and gelectra-large achieving up to 90% for polarity (2 classes), 85% for triple polarity (3 classes), 75% for main classes (6 classes) and 66% for subemotions (13 classes) on the filtered corpus. the historical models perform rather similar but consistently slightly below the large contemporary ones, but also slightly above the smaller contemporary model bert-base-german-europeana-cased. considering the different corpus manifestations, all models perform best on the filtered corpus and worst for the speech-based prediction. the difference becomes larger with increasing number of classes. for example, gbert-large achieves an accuracy of 75% for main class prediction on the filtered corpus which reduces to 51% on the speech corpus. as the analysis of recall and f1-macro statistics show, this is mostly due to the bad prediction accuracies for low-frequency classes. 8 further pretraining the model bert-base-germaneuropeana-cased with dramatic texts did not result in improvements. indeed, the accuracies become slightly worse and significantly lower looking at settings with multiple classes (e.g. 29% for subemotions on the filtered corpus). ## discussion and future work as the results show, we can confirm general findings of nlp-research for classification tasks for various text genres, in the sense that transformerbased models perform better than traditional textual ml-approaches in our setting with german historical plays. however, we cannot confirm our assumption that models pretrained on historical language achieve better results because they are closer to the language of our annotated material. indeed, the best performing models are gbert-large and gelectra-large by deepset @xcite . these are, to our knowledge, the largest german models trained on contemporary texts, primarly internet texts. the difference between historical and these contemporary models is however small. since the differences in the amount of text for the pretraining are significant (around 20 gb) it opens up the question if the performance of historical models improves with similarly large amounts of texts. considering the different corpus instances, we showed that filtering out overlapping annotations annotators disagree upon results into the strongest performance boost, although the training and test size become smaller. thus, it is crucial for our project to find ways to deal with disagreements among annotators. due to the varied and overlapping annotation lengths, we cannot rely on standard solutions like majority voting. furthermore, the inherent subjectivity of literary texts and the resulting low agreement among annotators is a specific feature of these kind of texts. we do however think that we can reduce disagreement with further training of the annotators and also by implementing a subsequent step after the first annotations of two annotators, in which a literary scholar expert creates a consensus annotation resolving disagreement. additionally, we intend to switch from single-label classification to multi-label emotion classification since this is more in line with the annotation process. this will open up further possibilities to deal with overlapping annotations and integrates this phenomenon into the classification task. applying a heuristic to map single emotion classes to entire speeches led to the models performing rather poorly compared to the other corpus manifestations. for sub-emotion prediction with 13 classes, accuracies became 25% worse for certain considering the results for filtered and full corpus, the transformer-based models achieve state-of-the-art accuracies for polarity classification (88-90%) compared to results with sentiment analysis with similar amounts of classes on contemporary german @xcite . the results achieved by the transformer-based models for polarity are also around 20% above results on german dramatic texts predicted by lexicon-based sentiment analysis, which yields results around 70% @xcite . for the main class and sub-emotion classification, results are, however, for the best models (75% for main classes, 66% for sub-emotions), below state-of-the-art results on emotion classification tasks with 4 or more classes for contemporary english texts for which accuracies of up to 86% are reported @xcite 2019; @xcite , however, for the most part, with larger training corpora and fewer classes as in our setting. we intend to improve the performance to satisfactory levels by hyperparameter-tuning and especially by exploring recommended ml-methods like over-and undersampling to deal with the class imbalances @xcite , which is one of the main problems of the main class and sub-emotion classification. among all transformer-based models, the berteuropeana-model further pretrained on dramatic texts yields the lowest accuracies. the performance becomes especially low for main classes and subemotions (see