Constructing a Sentiment-Annotated Corpus of A ustrian Historical Newspapers: Challenges, Tools, and Annotator Experience (2024)

Abstract: This study presents the development of a sentiment-annotated corpus of historical newspaper texts in Austrian German, addressing a gap in annotated corpora for Natural Language Processing in the field of Digital Humanities. Three annotators categorised 1005 sentences from two 19th-century periodicals into four sentiment categories: positive, negative, neutral, and mixed. The annotators, Masters and PhD students in Linguistics and Digital Humanities, are considered semi-experts and have received substantial training during this annotation study. Three tools were used and compared in the annotation process: Google Sheets, Google Forms and Doccano, and resulted in a gold standard corpus. The analysis revealed a fair to moderate inter-rater agreement (Fleiss’ kappa = 0.405) and an average percentage agreement of 45.7% for full consensus and 92.5% for majority vote. As majority vote is needed for the creation of a gold standard corpus, these results are considered sufficient, and the annotations reliable. The study also introduced comprehensive guidelines for sentiment annotation, which were essential to overcome the challenges posed by historical language and context. The annotators’ experience was assessed through a combination of standardised usability tests (NASA-TLX and UEQ-S) and a detailed custom-made user experience questionnaire, which provided qualitative insights into the difficulties and usability of the tools used. The questionnaire is an additional resource that can be used to assess usability and user experience assessments in future annotation studies. The findings demonstrate the effectiveness of semi-expert annotators and dedicated tools in producing reliable annotations and provide valuable resources, including the annotated corpus, guidelines, and a user experience questionnaire, for future sentiment analysis and annotation of Austrian historical texts. The sentiment-annotated corpus will be used as the gold standard for fine-tuning and evaluating machine learning models for sentiment analysis of Austrian historical newspapers with the topic of migration and minorities in a subsequent study.

## introduction sentiment analysis (sa), the automatic identification of attitudes, opinions, and emotions in textual data, has been popular since the early 2000s @xcite . deriving from natural language processing (nlp), it was initially used to study contemporary data, including reviews and microblog posts. recently, it has gained prominence in digital humanities (dh), expanding beyond contemporary texts to historical and literary texts @xcite @xcite . however, texts such as newspaper articles, novels, letters, and poetry, which are commonly studied in dh, pose a challenge due to their formal structures and historical nuances, making sentiment analysis difficult @xcite . traditional dictionary-based sa methods, heavily relied upon in dh, involve annotating words and phrases with sentiment values. although this method is easily interpretable and transparent, sentiment dictionaries suffer from low reusability and do not consider word context, missing nuances such as sarcasm or negation @xcite . to address these limitations, context-sensitive transformerbased machine learning models such as bert have been developed @xcite . these models require less annotated data data than traditional ml algorithms (such as bow or tf-idf), since they can be pre-trained on large, unannotated datasets. this means that pre-training is usually done once and the model can then be further fine-tuned for various specific purposes and tasks, such as e.g. named-entity recognition or sentiment analysis, using a smaller annotated corpus (of e.g. sentences, plays or verses with corresponding sentiment annotation). this is particularly use-ful in dh, where the annotation process is often tedious and time-consuming due to the complexity of literary and historical texts. however, good quality annotations are crucial for the accuracy of the models with which they are fine-tuned. to better understand the conditions necessary for creating a high-quality sentimentannotated fine-tuning corpus for texts in dh, recent studies have focused on the annotation process @xcite , annotator behavior @xcite , and annotation tools @xcite . these studies discuss the relevance of expert versus non-expert annotators, the optimal tool for sentiment annotation, and the importance of guidelines. these insights informed the approach taken in this study, as will be shown in the next chapters. there is still a gap in sentiment-annotated dh corpora that could be used for fine-tuning machine learning models, one such model being presented in @xcite . this model was trained on non-annotated historical newspapers and offers the possibility of further fine-tuning with an annotated corpus, for a specific task -such as named-entity recognition or sentiment analysis. however, newspaper texts pose various annotation challenges: historical language and context, discriminatory language, sarcasm and metaphors. the following sections present the current state of sentiment annotation in digital humanities and describe the creation of a sentiment-annotated corpus of austrian historical newspapers through an annotation study. the annotations, the annotation process and annotation tools are evaluated quantitatively and qualitatively. this evaluation identifies key challenges and provides guidelines for annotating historical newspaper texts. these initial results will guide future refinements of the corpus, which will be openly accessible on zenodo in accordance with the fair principles ## sentiment annotation for the digital humanities sentiment analysis (sa), in the context of digital humanities (dh), has often been used to answer specific research questions related to literary or historical studies. it has been used to analyse german plays @xcite , spanish enlightenment periodicals @xcite , spanish song lyrics @xcite and conflict in german novels @xcite . texts investigated by dh, such as newspapers, novels, poetry, and drama, present unique challenges due to their formal structures and historical linguistic nuances, making sa and annotation particularly complex @xcite . @xcite lists several aspects of the annotation process that are to be considered when constructing an annotation study: classification granularity, type of annotator (expert, nonexpert or crowd workers), perspective, unit of annotation, and language of annotation unit. a further consideration @xcite is the choice of annotation collection tool, which can have an effect on the annotation experience. these factors can impact the inter-rater agreement, which informs about the quality of annotations and is crucial in the development of a gold standard corpus. in sa, two primary classification tasks are typically addressed: polarity and emotion analysis. polarity analysis focuses on determining the direction of the sentiment within the text, often classified into categories such as positive, negative, and neutral @xcite . for more complex analyses, polarity classification may involve finer distinctions, such as differentiating between highly negative and highly positive sentiments, often using a numerical scale or additional categories. @xcite annotate four categories: positive, negative, neutral, and mixed. on the other hand, emotion analysis refers to the classification into emotional categories, often following ekman's theory of basic emotions @xcite ) or russell's circumplex model @xcite . @xcite conduct a polarity annotation study of german historical plays by g.e. lessing, using extended polarity categories -negative, positive, neutral, mixed, uncertain, and other. in a subsequent study @xcite , they conducted an additional annotation study with emotion categories. in these studies, inter-rater agreement, measured by statistical measures of cohen's or fleiss' kappa and krippendorff's alpha, decreases with the higher number of categories that need to be annotated @xcite . however, these studies show that the agreement on literary and historical texts ranges from poor to moderate agreement, due to subjectivity and difficulty of the annotation process. when conducting sentiment annotation, experts are preferred annotators, due to their accuracy and deep understanding of complex texts @xcite . but, they are scarce and expensive @xcite . semi-experts, such as advanced students, provide a more accessible alternative with reasonably reliable results @xcite , while nonexperts (e.g. obtained through crowd-sourcing), though less accurate, can be effectively utilised in large-scale projects with appropriate guidance and annotation schemes @xcite ). sentiment annotation can focus on two perspectives: the emotions the author intended to convey, or the emotions perceived by the reader @xcite . most studies focus on sentiments as intended by the author of the text @xcite @xcite , as the annotation from the perspective of the reader can lead to low inter-rater agreement, due to subjectivity of the task. the unit of annotation is also significant, and highly dependent on the type of text. @xcite and @xcite annotate sentences, while @xcite annotate speeches in a larger play. annotating a shorter unit can be beneficial, as it minimises the change in sentiment shifting within the annotation unit. traditionally, spreadsheets and word have often been used as the main tool for annotation collection @xcite @xcite . @xcite compare various annotation tools, such as word, webanno, catma, emargin and sentimentator. they report using sentimentator (a dedicated annotation tool) and word increases annotator levels of certainty, thus, making the choice of annotation tool important for obtaining high-quality annotations. they employ standard usability and user experience questionnaires, nasa-tlx @xcite and user experience questionnaire (ueq-s) @xcite , to assess user experience and perceived annotator workload. with respect to the previous work presented in this section, the annotators in this study are semiexperts who have received extensive training in sentiment annotation. they annotate sentences in four categories: positive, negative, neutral and mixed, focusing on the sentiment intended by the writer. furthermore, google forms (an online survey tool), google sheets (an online spreadsheet tool) and doccano @xcite are compared for the annotation process in order to establish an optimal tool for future annotation processes. to assess the usability of different annotation tools, google forms, google sheets, and doccano @xcite were compared. the findings from this comparison, along with insights into annotator experiences, will inform the choice of tools for future sentiment annotation projects. additionally, user experience was assessed using a combination of nasa-tlx, ueq-s, and custom questions targeting the specific challenges of annotating historical texts. ## aims and research questions the main aim of this study is establishing the optimal conditions for sentiment annotation of austrian historical newspapers, with the goal of creating a reliable gold standard corpus for fine-tuning of ml models for sentiment analysis. this study aims to answer the following questions: rq1 is using semi-expert annotators appropriate for the task of annotation of historical newspapers? rq2 how does the historical language and context of the texts influence the annotation process? rq3 how do the annotators perceive the difficulty of the annotation task? rq4 which tool is most optimal for sentiment annotation of historical newspapers? ## annotation evaluation the inter-rater agreement, as measured by fleiss' kappa, indicates a fair to moderate level of agreement (0.405), which is in line with expectations. the average percentage agreement (apa) is 45.7% when all three annotators concur. however, for a sentence to be included in the gold standard, a majority vote -where two out of three annotators agree -is deemed sufficient. this resulted in a 92.5% agreement among at least two annotators across all annotation items in our corpus, which was used to create a gold standard of 930 items. of the total number of annotations, 447 were classified as negative, 345 as neutral, 81 as positive, and 56 as mixed. the results indicate that the token length may have an impact on the annotations. items rated as 'mixed' had a mean of 53 tokens, compared to means of 31, 29, and 38 tokens for 'positive', 'neutral', and 'negative' annotations, respectively. this indicates that additional splitting of the annotation items may be required to eliminate ambiguity. notably, two annotators classified approximately half of the units as negative, while the third annotator rated 34% as negative and 45% as neutral, indicating that the most predominant sentiments were negative and neutral (see by closely examining the sentences in the annotation study, we can identify the causes of disagreement. for instance, the sentence "in den beziehungen zwischen polen und czechen ist, trotz der vielen gegenseitigen versicherungen brüderlicher freundschaft, in jüngster zeit -wie fast regelmäßig vor jedem wiederzusammentritte des reichsrathes -eine spannung eingetreten." (english translation: "in the relations between poles and czechs, despite the many mutual assurances of brotherly friendship, a tension has recently arisen-almost always before each reconvening of the reichsrat.") is highly ambiguous. it conveys both positive and negative sentiments between the subjects (czechs and poles) and suggests a possible disdain by the author ("almost always"). in contrast, sentences with complete agreement, such as "treibt auch noth einen serben oder walachen zur arbeit, so strengt er sich durchaus nicht an." (english translation: "even if 1 a jupyter notebook outlining the evaluation can be found at https://github.com/lucijakrusic/sentianno/ necessity drives a serb or wallachian to work, they do not exert themselves at all."), present a clearer sentiment and a more evident object of that sentiment. ## evaluation of annotator experience and annotation tools the questionnaire yielded valuable insights into the attitudes and perceptions of the annotators regarding the annotation process. in terms of the complexity of the text annotation process, two participants rated it as "challenging" (4 on a 5-point scale), while one rated it as "moderate" (3 on a 5-point scale). the estimated time required for one annotation unit is between one and four minutes, with one annotation round taking between three and over five hours. two annotators reported needing to take regular breaks from the annotation process. however, two out of three annotators report having high confidence (4 out of 5 on a likert scale) in their annotations, while one reports moderate confidence (3 out of 5). in terms of specific difficulties, annotating longer texts was identified as particularly challenging, particularly when sentences were complex and required close attention. another challenge identified was the need to remain objective, as one participant mentioned the difficulty of not letting personal beliefs influence the annotation process. the historical context and language of the texts had a notable impact on the annotation process. the participants indicated that the historical context affected their ability to annotate, with one annotator finding it particularly challenging. this emphasizes the importance of familiarity with the historical background when conducting sentiment analysis. moreover, all participants felt that the historical context significantly influenced their ability to annotate sentiment, reporting the need to independently research the historical background using resources like wikipedia, the university library catalogue, britannica, and the anno repository. the historical language, including vocabulary, grammar, and phrasing, also posed challenges, similarly influencing their ability to annotate sentiment. participants indicated that the clarity of what should be annotated-whether it was the sentiment of the language, sentiment towards a group, or the emotional state of the speaker-was not always clear. they reported that regular discussions and feedback sessions were useful for overcoming these challenges. the use of sarcasm and metaphors in the texts presented a challenge for most participants. two respondents found metaphors challenging, while one was affected by sarcasm, indicating a need for additional training or guidelines on handling figurative language in sentiment analysis. access to the previous and following sentences (context) was generally seen as helpful, with all participants agreeing that it aided in making more accurate annotations. however, opinions were divided on whether more context was necessary, with one participant suggesting that additional context could clarify ambiguous sentiments. the overall nasa-tlx score was calculated by averaging the scores on six dimensions: mental, physical, temporal demand, performance, effort and frustration level. this equates to a score of 3.09 out of 5, indicating that the perceived workload is slightly above neutral. the annotators indicated that the level of workload was moderate overall, with some dimensions (such as mental demand, effort, and performance) rated higher than others. in terms of the ueq-s, the pragmatic quality (pq) achieved a score of 4.97 on a scale from 1 to 7 (1 reflects a poor experience in terms of usability, and a higher score of 7 reflects a good experience). this indicates that the task was moderately usable and clear. the hedonic quality (hq) was rated at 4.45 out of 7 (with 1 indicating a less enjoyable experience and 7 indicating an emotionally satisfying one), indicating that the task was perceived as somewhat enjoyable and not unpleasant. the questionnaire also assessed the usability of different annotation tools, with participants evaluating google forms, google sheets, and doccano. all three annotators identified doccano as the most intuitive tool, citing its clear layout, ease of navigation, and effective display of context as key factors. additionally, the ability to leave comments and track progress was identified as a valuable feature. however, both google forms and google sheets are also considered relatively straightforward to use and easily adaptable (see figures 1 and 2 ). the annotators found google sheets less practical for navigating between the annotation units and viewing the full sentences. this is reflected in figure 1 , where one annotator noted that google sheets were difficult to use. lastly, annotators provided constructive feedback on how to improve the annotation process. one suggestion was to standardise the token length of context provided for each sentence, as inconsistencies sometimes made interpretation difficult. another recommendation was to allow annotators to "correct" incomplete sentences by adding parts from adjacent sentences. these insights will be implemented in the following rounds of annotation. ## discussion in this study, a sentiment-annotated corpus of austrian historical newspaper texts was developed, with three semi-expert annotators categorizing sentences into four sentiment classes: positive, negative, neutral, and mixed. the inter-rater agreement, measured using fleiss' kappa, resulted in a score of 0.405, indicating fair to moderate agreement and reflecting the inherent challenges in annotating historical corpora. this aligns with previous research @xcite @xcite ) and highlights the difficulty of classifying complex texts, particularly those with mixed sentiments. this category, indicating both positive and negative sentiments within a single sentence, was the most challenging due to its higher token count, and presumably as a result, higher content ambiguity. this finding underscores the complexity of historical texts, where sentiments can shift within the same sentence or be expressed through nuanced language, including sarcasm and metaphors, that is difficult to categorise definitively. the study also observed a notable imbalance in sentiment categories, with the majority of annotations marked as negative or neutral. this distribution mirrors the historical context of the periodicals, which frequently adopted a critical stance toward migration and minorities. despite these challenges, the annotation process achieved an average percentage agreement (apa) of 45.7% for full agreement and 92.5% for majority vote (two out of three annotators). these results validate the reliability of the annotation process, allowing the creation of a gold standard corpus comprising 930 sentences, which will be extended in future annotation rounds. notably, the successful use of semi-expert an-notators-advanced students-demonstrates that it is possible to achieve reliable annotations without relying on fully trained experts. this finding corroborates previous studies @xcite , reinforcing the notion that semi-experts, can serve as an accessible yet effective alternative for similar tasks. the annotation process was systematically supported by providing essential contextual informa-tion, including references to the previous and following sentences, newspaper name, and date, which helped annotators interpret complex historical texts more accurately. such contextual cues were particularly important given the nuanced language found in historical materials, where sentiments often shift within a sentence. the user experience questionnaire (appendix a) revealed the cognitive and emotional demands on annotators, especially when dealing with historical language, figurative expressions like metaphors, and sarcasm. this feedback is valuable for improving future annotation workflows and provides a basis for comparing annotator experiences in similar tasks. reusing the questionnaire, particularly its sections on historical language and context, could further enhance the systematic evaluation of annotation processes within digital humanities projects. the standardized usability assessments, nasa-tlx and ueq-s, highlighted the need to consider both cognitive workload and user engagement when designing annotation tasks. the nasa-tlx results showed that while the task was manageable, it required significant cognitive effort, particularly for complex, sentiment-laden historical texts. this finding aligns with @xcite and highlights the importance of considering workload when designing annotation tasks, particularly for complex historical texts. the ueq-s results reveal a clear process (pragmatic quality) but suggest the task could be more engaging (hedonic quality). while doccano proved to be the most user-friendly tool, with a positive impact on annotator efficiency and accuracy, there is room for improvement in user experience, particularly regarding task engagement. these results reinforce the need for comprehensive guidelines and tool evaluations, as well as attention to annotator workload, to ensure efficient and accurate sentiment annotation in digital humanities. ## conclusion this study contributes to the field of digital humanities by presenting the first sentiment-annotated corpus of austrian historical newspaper texts in austrian german. through the collaboration of three semi-expert annotators, 930 sentences were annotated for sentiment using a carefully designed process supported by tools like doccano, google sheets, and google forms. the fair-to-moderate inter-rater agreement (fleiss' kappa of 0.405) reflects the challenges of annotating historical texts, where sentiment is often complex and contextually dependent. a contribution of this study is the user experience questionnaire, which were specifically designed to assess the cognitive and emotional challenges encountered during the annotation process. the bespoke sections of the questionnaire not only provided valuable insights for improving subsequent annotation rounds but also offer a reusable framework for evaluating annotator experiences in other historical and literary annotation projects. furthermore, this study highlights the feasibility of employing semi-expert annotators in sentiment annotation, achieving reliable results through thorough guidelines and iterative feedback. standardized assessments of usability and user experience, combined with the custom questionnaire, provided critical insights into annotators' cognitive demands and areas where the task could be improved. by making the corpus openly available, this research offers a valuable resource for further sentiment analysis in austrian german, particularly on topics such as migration, minorities, and labor rights. the findings and methodology outlined here will serve as a basis for future annotation projects, contributing to more nuanced and accessible sentiment analysis in historical and literary contexts.