Sentiment Classification of Historical D anish and N orwegian Literary Texts (2023)

Abstract: Sentiment classification is valuable for literary analysis, as sentiment is crucial in literary narratives. It can, for example, be used to investigate a hypothesis in the literary analysis of 19th-century Scandinavian novels that the writing of female authors in this period was characterized by negative sentiment, as this paper shows. In order to enable a data-driven analysis of this hypothesis, we create a manually annotated dataset of sentence-level sentiment annotations for novels from this period and use it to train and evaluate various sentiment classification methods. We find that pre-trained multilingual language models outperform models trained on modern Danish, as well as classifiers based on lexical resources. Finally, in classifier-assisted corpus analysis, we confirm the literary hypothesis regarding the author’s gender and further shed light on the temporal development of the trend. Our dataset and trained models will be useful for future analysis of historical Danish and Norwegian literary texts.

## introduction sentiment analysis, the computational study of emotions, opinions, and evaluations expressed in text, has become an important tool in natural language processing. it is based on the premise that words are associated with sentiments or valence, and that these associations can be quantified @xcite . however, its application in literary studies has been limited. literary texts, such as novels, poems, and plays, provide a unique cultural window into past attitudes, beliefs, and emotions. by analyzing the sentiment expressed in these texts, researchers can gain a deeper understanding of the cultural and societal attitudes of the past and how they have shaped our present understanding of the world. however, traditional sentiment analysis techniques in the investigation of literary texts may be less effective due to the use of an archaic vocabulary, literary ambiguity, and figurative language, as well as the limited training data, the difficulties of generalizing models trained on modern text, and the challenges of annotation. therefore, there is a need for new methods and approaches to analyze sentiment in historical and literary texts to uncover the valuable insights they can provide. however, there is no existing benchmark for sentiment analysis of the texts we are interested in, and it is not known how well existing, general methods perform on them, and whether they can be used for a meaningful analysis of literary hypotheses. in this paper, we (1) introduce an annotated sentiment dataset of historical literary danish and norwegian, (2) evaluate various classification models on the dataset, and (3) use an accurate classifier to analyze a large historical literary corpus and provide initial evidence for a literary hypothesis as outlined in the following section. our code and data are available at https: //github.com/mime-memo/unhappy . ## literary context: the unhappy texts in our experiments, we focus on literature from the so-called 'modern breakthrough,' which denotes the literary currents embedded in the new social realist and naturalist literature which blossomed in scandinavia during the 1870s and the 1880s as well as a cultural and societal transformation, encompassing politics, morals, gender roles, etc @xcite . on the whole, the historiography of the breakthrough, largely concentrated on a few canonical male authors, does not reflect the diversity of the period's literary production, especially in terms of gender. women were most noticeable among the new groups of authors stepping forward from 1870-1900. as a rule, female authors were not recognized by contemporary (male) criticism; the famous critic georg brandes introduced the term breakthrough in a book grouping only male scandinavian authors, the men of the modern breakthrough @xcite . and, a generation later, the influential literary historian vilhelm andersen dubbed the period's female literary production as an 'odd province' of literature @xcite . while andersen recognized the period's literary production by danish female authors, a much later literary handbook hovedsporet @xcite does not. its chapter on the debates on gender in the period's literature has the heading 'women's literature, written by men'. in fact, for many decades of the 20 th century, the female breakthrough authors were overlooked and mostly left out of literary histories, but in the 1970's and 80's, as a part of a greater feminist wave in scandinavian literary studies, scholars made a great effort to reinterpret the period. a result of this was the prevalent hypothesis about the works of the female authorships from the period being characterized as 'unhappy texts' @xcite . the argument is unfolded in greatest nuance in pil dahlerup's doctoral dissertation, the women of the modern breakthrough @xcite . dahlerup argues that because the women of the period were restrained by the patriarchal society, they also wrote fictional characters who lacked agency, were unhappy and disillusioned. meanwhile the scholars highlight that the emancipated female characters of the period are first and foremost present in the works of male authors, arguing that the male authors, who enjoyed both private and public liberty, had the capacity to portray fictional characters with the same liberties @xcite . our investigation will revisit this hypothesis with two points of departure, one regarding empirical selection and scale, and another regarding methodology. the hypothesis of the unhappy texts is based solely on texts by female authors and often on relatively few texts. the first point of departure is thus to revisit the hypothesis with a quantitative perspective, also including the works authored by men. our second point of departure, in revisiting the hypothesis of the unhappy texts, is based on a theoretical framework from the field of gender studies and feminist literary studies. central to our inquiry are new insights from affect theory and the notion of sentiment and affects as something humans 'do' rather than an underlying a priori structure as is the case in the psychoanalytic assumptions that the hypothesis of the unhappy texts is based on. we thus set out to find new methodologies to test the hypothesis of unhappy texts. in this experimental methodological phase, we find that sentiment analysis, analyzing the valence associated with a given word, sentence, or text, is a meaningful starting point because of its juxtaposition of affect theoretical and quantitative perspectives. ## related work in this section, we present sentiment-related works in both literature and historical corpora. ## dataset of historical literary text this section describes our main corpus of historical literary text and a sub-corpus annotated for sentiment. ## experiments the dataset is split into three sets: training, validation, and testing to facilitate the development and evaluation of our models. the training set comprises 2376 examples, accounting for approximately 86% of the dataset. the validation set used for hyperparameter selection consists of 272 samples, representing about 10% of the entire dataset. lastly, the testing set, which is utilized to evaluate the final performance of the model, contains 100 examples, representing approximately 4% of the total dataset. in the case of the training and validation sets, annotations were performed by a single expert. however, for the testing set, samples were annotated by all three experts, and the final label was determined through a majority vote. we use f1-score as our evaluation metric. ## classifier-assisted corpus analysis we employ the 'danish sentiment' bert model, which has shown to be the top-performing model, for predicting the sentiment of all sentences in the main corpus. we align the sentiment with the author's gender and the novel's year of publication. in figure 1 , the distribution of sentiment levels is depicted in relation to the author's gender and the percentage of sentences. notably, female authors tend to exhibit a higher proportion of both negative and positive sentiments compared to male authors, on average. these results provide insights into potential differences in sentiment expression between male and female authors in the analyzed data. the literary hypothesis about female authorships being 'unhappy' is partly confirmed by our analysis. the female authors did express more negative sentiment, but also more positive sentiment than the male authors. thus the implied positivity of the male authors, which is also part of the literary hypothesis, is not confirmed. a preliminary new hypothesis could therefore be that the female authors of the period wrote with a more expressed sentiment, whilst the male authors had a tendency towards a more unaffective style of writing. figure 2 provides a detailed analysis of the sentiment distribution over time concerning the author's gender. this figure shows the changes in sentiment tendencies across different gender groups and how these sentiments evolve over a given period. the results provide valuable insights into the dynamic nature of sentiments based on gender and can aid in understanding the evolving nature of sentiments and gender influences. the figure demonstrates that female authors exhibit more negative and positive sentiments over time than male authors. additionally, we calculated the overall sentiment by using a weighted average approach that considered both the sentiment distribution and weights of positive, neutral, and negative sentiments. specifically, we assigned a weight of 3 to negative sentiment, 2 to neutral sentiment, and 1 to positive sentiment. to compare the sentiment of male and female authors over time, we computed the average sentiment scores over a period of 29 years and plotted them in figure 3 . higher scores indicated more negative sentiment, while lower scores indicated more neutral or positive sentiment. the results in the figure show that, on average, the negative sentiment scores of female authors are higher than those of male authors in 16 out of the 29 years analyzed. the significance of our findings was assessed by performing a t-test for each sentiment category, with the null hypothesis that the mean number of sentences with that category across the years is the same for sentences by male and female authors. in statistical analysis, a p-value lower than 0.05 is generally considered statistically significant and enables us to reject the null hypothesis, indicating the counts are statistically different. ## discussion these initial analyses of a literary corpus with the newly developed sentiment classifier both confirm and contest the thesis of the 'unhappy' female texts. on the face of it, it seems that the female authors in the corpus express not only negative but also positive sentiments. at the same time, the second (implicit) part of the thesis is refuted, as the male authors do not write more positively. clearly, this is reopening the literary discussion of the unhappy text with a level of complexity that these results cannot account for alone. further work is needed. in terms of computational interventions, a more in-depth analysis would call for us to investigate polarity in greater detail (how positive or negative are the novels?), while also paying more attention to narrative structures (how does polarity relate to plot?). more trivial analytical next steps involve employing segmentation of data and comparisons of sub-corpora. are there, for example, certain works or authors that are outliers in terms of gender or valence? are there any correlations when comparing canonized works and popular literature? by zooming in on either part of the novels (the endings, for example) or parts of the corpus (with regards to time slots or subgroups of producers/individual authors), specific strong trends may stand out. also, by working with smaller subsets of the corpus, we could perhaps challenge the essentialist idea of an ' écriture féminine', a unique feminine style of writing. indeed, instead of simply asking whether there is a difference between male and female authorship in the period, we would also be able to explore how this difference is created in the corpus and possibly cultural, historical, gendered, and narratological reasons for it. in this connection, the fact that male authors seemingly write more 'neutrally' while female authors write with a higher degree of valence might be put into perspective through sara ahmed's theory of the 'stickiness' of emotions. inspired by performativity theory, sara ahmed is interested in what emotions or affects do, rather than what they are @xcite . both 'positive' affects, such as (naïve) joy, and 'negative' affects, such as shame have, especially during the period in question but also still today, clung to the feminine, while hegemonic masculinity has been and is associated with the rational and unsentimental @xcite . in other words, a further refinement of the analytical steps goes hand in hand with critical interactions with the theoretical framework and specific historical contexts concerning the concepts of gender and emotion. ## conclusion in this work, we used the memo corpus to create a high-quality human-annotated sentiment dataset for historical literary danish and norwegian. despite multiple challenges, we showed that the task is feasible and that inter-annotator agreement is sufficiently high to warrant the use of the dataset for the training and evaluation of sentiment classifiers. in an extensive evaluation of such models, we found that the best performance is obtained with xlm-roberta fine-tuned on senti-ment analysis of modern danish tweets and then further on the training set from our dataset. using this model to annotate the whole memo corpus automatically, we observe that, as the literary theory predicts, female authors expressed more negative sentiments than male authors in the period. however, the hypothesis of the unhappy texts is only partially confirmed by our analysis as the numbers also suggest that the female authors, in contrary to the hypothesis, expressed more positive sentiments than the male authors who collectively expressed more neutral sentiments than their female counterparts. in future work, we intend to experiment with further improvements of the classifier, including pre-training a language model on the whole corpus as a basis for fine-tuning the sentiment dataset. furthermore, we intend to complement the classifier with topic models to investigate the evolution of sentiment towards specific topics (cultural change) over time and the evolution of words used to express a sentiment (language change). from a literary perspective, a sentiment classifier trained on scandinavian novels from the end of the 19 th century holds great potential outside of the context of our specific test case. to give just one example, it would be interesting to study the so-called 'naturalism' from the time of the modern breakthrough. this literary movement or current is often said to be characterized by a pronounced pessimistic worldview. one can assume that negative segments predominate naturalist writings, and if that is in fact the case, the numbers would be interesting to compare with the rest of the literature from the period, i.e., canonical, realistic novels and this period's many forgotten texts. from a literary theoretical perspective, a classifier such as this can be used to shed light on the importance of narratological layers at a macro level. although the human reader is far superior to algorithms in separating character, narrator, and (implied) author, the numerical output of the computer is very interesting for existing research in the period. questions like 'does the sentiment distribution correspond to the overall assertions of the individual works?' and 'how does sentimental quantity relate to literary expression?' become possible to investigate.