C hat GPT as Your n-th Annotator: Experiments in Leveraging Large Language Models for Social Science Text Annotation in S lovak Language (2024)

Abstract: Large Language Models (LLMs) are increasingly influential in Computational Social Science, offering new methods for processing and analyzing data, particularly in lower-resource language contexts. This study explores the use of OpenAI’s GPT-3.5 Turbo and GPT-4 for automating annotations for a unique news media dataset in a lower resourced language, focusing on stance classification tasks. Our results reveal that prompting in the native language, explanation generation, and advanced prompting strategies like Retrieval Augmented Generation and Chain of Thought prompting enhance LLM performance, particularly noting GPT-4’s superiority in predicting stance. Further evaluation indicates that LLMs can serve as a useful tool for social science text annotation in lower resourced languages, notably in identifying inconsistencies in annotation guidelines and annotated datasets.

## introduction the emergence of large language models (llms) has not only revolutionized the field of natural language processing (nlp) @xcite but also significantly impacted social sciences @xcite . these models' ability to understand and generate human-like text has opened new avenues for analyzing complex social phenomena such as political discourse @xcite , public opinion @xcite , and media analysis @xcite with unprecedented precision. this progress has set the stage for augmenting, or even substituting, human annotators in tasks demanding profound linguistic and semantic insights @xcite @xcite . our research explores the application of llms, coupled with sophisticated prompting strategies, to scrutinize slovak news media content on migration, a topic figure 1 : prompting strategy grid search: slovak prompts exhibit the lowest effectiveness with gpt-3.5 but secure the highest performance with gpt-4. notably, incorporating explanations within prompts significantly enhances effectiveness across models. this boost is particularly pronounced for slovak prompts used with gpt-3.5 and english prompts with gpt-4. the red dashed line represents the 'zero shift' from the average performance, illustrating changes in f1 score relative to the average across all parameter combinations. with deep societal and political ramifications. the dataset, created for project mimedis foot_0 , seeks to unravel how media shapes public migration viewpoints, integrating computational and manual analysis. our investigation pivots on employing llms for annotating social science materials in less commonly used languages, revealing that advanced prompting methods can position llms as viable alternatives or complements to traditional supervised fine-tuning. our findings also suggest that non-english instructions could enhance llm performance, as outlined in figure 1 . we hope that this study will help highlight the llms' potential as a helpful tool in annotating social science texts in lower resourced languages and instigate further development in this area. ## related work large language models (llms) have garnered a significant amount of interest over the past few years, especially due to their unprecedented ability to generalize based just on zero-shot input, or from just a handful of examples, also known as few-shot learning. when combined with advanced prompting strategies such as retrieval augmented generation (rag) @xcite and chain of thought (cot) prompting @xcite , this makes llms state-of-the-art methods for various nlp and text understanding tasks @xcite . among other achievements, this has led to research that suggests that llms such as gpt-3.5-turbo and gpt-4 can be adapted for annotation @xcite @xcite and in some cases even as a potential replacement for human annotation @xcite as it was able to perform on-par or better than a human annotator @xcite . on the other hand, closer inspection by @xcite has found that "fewshot learners offer enticing, yet mixed results on text annotation tasks", suggesting that evidence for aforementioned claims is only partial at best. despite the partial evidence, llms still present an interesting option, particularly for languages which lack large-scale data resources and for which the cost of annotation is often significant due to the low number of native speakers and/or experts available, which is the case in our situation as well. perhaps the most similar work to ours would be @xcite in which the authors evaluate stance of sentences in estonian news articles about immigration and compare the performance of supervised models with chatgpt, finding that chatgpt obtains similar performance. in contrast, in our work we explore a problem that can be viewed as multi-target and multi-class, we further consider the article-level as opposed to sentence-level stance, employ multiple llms (gpt-3.5-turbo and gpt-4) and a number of advanced prompting strategies such as rag and cot prompting, which make our best performing llms capable of performing better than supervised models. ## dataset to evaluate our models we utilize a specific slovak dataset annotated for classification across various dimensions. the dataset aims to understand migration representation in slovak media spanning from 2003 to 2022, targeting individual media outputs like articles and debate transcripts. we briefly outline the specific dimensions below. thematic relevance articles are classified based on relevance to human migration within the study period, marked as strong, weak, or irrelevant. geographical relevance this categorization differentiates between articles related to slovakia and those not. ## methods our experimental methodology was informed by three key guidelines to ensure uniformity and comparability of results. to standardize input data, articles were cut to 2500 characters. this length not only aligns with the maximum input capacity of our baseline models but also helps in managing openai credit usage efficiently. the evaluation of annotations was carried out on the test split of the pre-annotated dataset. the experiments were structured with a single-task focus, i.e. there was a dedicated model for each task. each model was tasked to predict a single label. ## discussion the results in ## conclusion this study evaluates the performance of llms in automating stance classification tasks within a slovak news media dataset, emphasizing the impact of advanced prompting strategies and native language instructions. the results indicate that while large language models (llms), particularly gpt-4, significantly outperform bert-based baseline models, they still lack the ability to fully replace human annotators in complex tasks such as stance classification in political texts under the conditions of our experiments. however, their ability to uncover inconsistencies in annotation guidelines and datasets highlights their potential as valuable tools in social science research. the findings from this study have enabled the mimedis project team to refine the annotation manual and to distinguish between inherently difficult tasks and those that are simply underdefined. just like if chatgpt was our n-th expert annotator.