Introducing C hat GPT to a researcher’s toolkit: An empirical comparison between rule-based and large language model approach in the context of qualitative content analysis of political texts in F innish (2023)

Abstract: Large Language Models, such as ChatGPT, offer numerous possibilities and prospects for academic research. However, there has been a gap in empirical research regarding their utilisation as keyword extraction and classification tools in qualitative research; perspectives from the social sciences and humanities have been notably limited. Moreover, Finnish-language data have not been used in previous studies. In this article, I aim to address these gaps by providing insights into the utilisation of ChatGPT and drawing comparisons with a rule-based Natural Language Processing method called Etuma. I will focus on assessing the effectiveness of classification and the methods’ adherence to scientific principles. The findings of the study indicate that the classic recall and precision trade-off applies to the methods: ChatGPT’s precision is high, but its recall is comparatively low, while the results are the opposite for Etuma. I also discuss the implications of the results and outline ideas for leveraging the strengths of both methods in future studies.

## introduction the field of natural language processing (nlp) has recently undergone a significant transformation, largely driven by the widespread adoption and popularity of large language models (llms). llms, such as chatgpt, offer numerous possibilities and prospects for academic research as well. many researchers who have previously relied on traditional nlp methods are now considering the future trajectory of the field. the question arises: will other nlp methods become obsolete, with llm applications replacing them in research? since the launch of chatgpt in november 2022, there has been extensive discussion within the scientific community, and research articles have been published at an accelerated pace. many of these studies have demonstrated that chatgpt's performance in various tasks is comparable to that of humans in terms of quality. in a study by @xcite , chatgpt was able to identify implicit hate speech well compared to humans. @xcite found that chatgpt's capabilities to answer questions from several domains including finance, medicine, law, and psychology, were on par with those of human experts. @xcite reported that chatgpt even outperformed humans in annotation tasks including relevance, stance, topics, and frames detection. on the other hand, chatgpt's ability to produce consistent results has been questioned and caution has been advised regarding its application to text classification @xcite . some studies have found chatgpt's zero-shot performance to be lacking, although prompt engineering and additional training have been shown to improve results @xcite . while chatgpt has been extensively examined for a diverse range of tasks, there remains a gap in empirical research regarding its utilisation as a classification tool in qualitative research. furthermore, perspectives from the social sciences and humanities have been notably limited thus far. in addition, finnish-language data has not been used as research material. in this article, i will also introduce etuma, an nlp tool that represents a traditional rule-based approach based on supervised learning methods, dictionaries, and grammar rules. the aim is to highlight the distinctive features of these two different approaches in one of the most common nlp tasks: text classification. i will focus on the qualitative content analysis of extensive datasets in the field of digital humanities, with a particular emphasis on topic classification, a central aspect of qualitative content analysis. ## data and methods the context of this study is an ongoing research project focusing on political energy discourse in finland. in the project, my goal is to analyse the public political debate, specifically examining the comments made by citizens and politicians. the objective is to gain a comprehensive understanding of the issues underpinning the energy debate and to explore the various themes that emerge from the collected material. given the large volume of the research material, i will employ a combination of automatic text analysis and qualitative methods @xcite @xcite . in research work, several important criteria should be considered when selecting a method or a research tool. the tool should be suitable for scientific research in general, adhering to rigorous standards of reliability, validity, and ethical considerations. in this case, an important feature was also the tool's proficiency in the finnish language, to ensure its ability to process and analyse finnish texts. additionally, the tool needed to provide a comprehensive overview of large volumes of research material by effectively categorising it into relevant topics. with these criteria in mind, i sought to explore whether the widely popular chatgpt could be a suitable method for conducting the analysis required for the project. in my previous work, i have utilised the etuma tool for keyword extraction and topic classification. therefore, i decided to compare the two approaches to delineate the strengths and weaknesses of each method. ## empirical analysis in this section, i will present the key findings obtained from the analysis conducted by etuma and chatgpt on the corpora. firstly, i will describe the characteristics of keyword extraction and topic classification for both methods, along with relevant examples. it is important to note that the purpose of these key figures is to compare the classifications, without taking a stance on what constitutes the ideal classification. secondly, i will present a comparison of the methods using a smaller sample, employing traditional metrics such as recall, precision, and f1 score. this analysis will provide a quantitative evaluation of the performance of each method. additionally, appendix 1 contains a list of the most frequently occurring topics and keywords identified during the analysis. ## discussion in this section, i revisit the research questions i presented in the introduction. firstly, i discuss the effectiveness of the classification in qualitative content analysis from the perspectives of key aspects such as recall, granularity, precision, and refinement. secondly, i assess the alignment of the methods with scientific principles, specifically focusing on repeatability, transparency, and research integrity. ## conclusions to summarise my findings, the utilisation of chatgpt as a research tool poses challenges to the reliability of the research due to issues of repeatability and transparency. in the context of result usability, challenges arise from the occurrence of hallucinations and potentially from low recall. a major limitation of a low recall rate is that it excessively restricts the researcher's autonomy in decision-making. in this project, my aim was to conduct a comprehensive classification that allows for a qualitative analysis of the material from various perspectives. hence, i do not want the tool to determine what is important or interesting in the text on my behalf. on the other hand, a drawback of the rulebased approach often lies in its lack of semantic meaning and context. nevertheless, this deficiency can be addressed through refining, which, at least with the tools employed in this study, proves to be more straightforward with a rule-based method.