Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding (2023)

Abstract: Qualitative coding is a content analysis method in which researchers read through a text corpus and assign descriptive labels or qualitative codes to passages. It is an arduous and manual process which human-computer interaction (HCI) studies have shown could greatly benefit from NLP techniques to assist qualitative coders. Yet, previous attempts at leveraging language technologies have set up qualitative coding as a fully automatable classification problem. In this work, we take a more assistive approach by defining the task of qualitative code suggestion (QCS) in which a ranked list of previously assigned qualitative codes is suggested from an identified passage. In addition to being user-motivated, QCS integrates previously ignored properties of qualitative coding such as the sequence in which passages are annotated, the importance of rare codes and the differences in annotation styles between coders. We investigate the QCS task by releasing the first publicly available qualitative coding dataset, CVDQuoding, consisting of interviews conducted with women at risk of cardiovascular disease. In addition, we conduct a human evaluation which shows that our systems consistently make relevant code suggestions.

## introduction in qualitative research, qualitative coding is a content analysis method that is used to analyze textual corpora such as cultural and political texts, questionnaires and interview transcripts. first used in social science studies, qualitative coding has become a ubiquitous and universal method of analysis geared at providing researchers with an indepth understanding of their studied corpus @xcite . it is now employed in several research disciplines including medicine, education, and psychology @xcite @xcite . during the coding process, a researcher carefully scans each document in their corpus, identifies passages that are associated with their research question, and assigns these passages descriptive labels or qualitative codes (figure 1 ). despite its widespread use, coding remains an arduous and time-consuming process as it requires manually annotating a studied corpus line by line. for instance, studies have shown that coding a 1hour interview transcript will typically take an experienced researcher between 4 to 8 hours of annotation @xcite . to increase coding efficiency and to reduce its cognitive load on researchers, nlp practitioners have attempted to insert language technologies into the coding process by formulating it as a text classification task. however, as human-computer interaction (hci) studies have shown @xcite , qualitative coders prefer receiving suggestions for identified passages as they code rather than having yet-to-be-seen passages automatically coded for them. this aversion towards a fully automatic coding system may explain why, to this day, all available qualitative coding tools are used solely as "electronic filing cabinets" @xcite . that is, if automatic coding is not approached in a user-centered manner, despite a desire for automated assistance, coding tools such as nvivo 1 and maxqda foot_1 will continue to be used for their bookkeeping features only @xcite . in this work, we take a more user-centered approach to qualitative coding by proposing the task of qualitative code suggestion (qcs) in which a ranked list of previously assigned codes is suggested from passages identified by a qualitative coder. our framing of qualitative coding reveals technical subtleties which have not been addressed in previous work. for instance, our task definition exposes the necessity to evaluate code suggestions based on the sequence in which documents are coded as well as the importance, and difficulty, of detecting rare codes. in addition, unlike other nlp tasks that use human-sourced annotations, qcs expects differences in annotation styles and avoids attempts to correct or homogenize these differences. although more challenging, we believe that this human-centric approach is appropriate given that qualitative coding is meant to aid a researcher in their personal understanding of a corpus. to summarize, the research questions we address in our work are: 1) what previously unexplored technical challenges arise from approaching qualitative coding in a user-centered manner? and 2) how effective are current nlp methods at handling these challenges? to this end, the contributions of our work are as follows: ## related work in qualitative research, content analysis is defined as "a family of research techniques for making systematic, credible, or valid and replicable inferences from texts and other forms of communications" @xcite . these techniques rely heavily on qualitative coding which has been used extensively throughout different areas of research, first in the social sciences and, now, across most fields @xcite . there are two types of qualitative coding: deductive and inductive @xcite . in deductive coding, all codes are predefined and the coding process serves as a way of validating or invalidating a hypothesis. in inductive coding, codes are defined ad hoc as documents are analyzed and, thus, the coding process is used to discover new phenomena underlying a corpus. in terms of assistive technologies for qualitative coding, studies have investigated various computational approaches to automate parts of the deductive qualitative coding process. efforts by @xcite ; @xcite ; @xcite have investigated the use of rule-based learning and supervised learning to automate the assignment of codes to passages. more recently, @xcite ; @xcite have explored using bert-based models and gpt3 respectively to predict a previously assigned code from a passage. in all cases, deductive coding was cast as a fully automatable classification problem and metrics such as f1 and annotatorsystem agreement scores were used to measure performance. at the same time, there has been extensive work from the hci community on how qualitative coders would like automation to assist them during the coding process @xcite @xcite . these studies show that while researchers desire automation, they believe the coding agency should remain in their hands. thus, methods that introduce language technologies in the qualitative coding process should favour assisting coders by highlighting ambiguous passages @xcite , by helping them remain consistent in their annotation style @xcite , or by interactively refining the definition of codes @xcite . these assistive approaches differ from previous human-in-the-loop annotation studies (montani and honnibal, 2018) since, as jiang et al. (2021a) point out, "qualitative analysis is a unique case for human-ai collaboration because uncertainty is essential to qualitative analysis itself." in other words, qualitative coding is unique because the act of coding data, and the uncertainty that comes with it, is valued by researchers as it cultivates their understanding of their corpus. ## qualitative code suggestion we present properties of qualitative coding which have surfaced from the user-centered perspective we take in this work. we show that these properties warrant a new task definition that centers around suggestions and rare codes. ## the cvdquoding dataset to investigate the qcs task, we release the first publicly available qualitative coding dataset which we name cvdquoding. in this section, we describe the dataset and how it was created. in addition, we draw parallels between cvdquoding and the properties of qualitative coding we described in section 3.1. cvdquoding is a dataset consisting of 15 transcripts of interviews conducted with women at risk of cardiovascular diseases (cvds) as well as annotations carried out by two qualitative coders. these qualitative coding annotations consist of text spans identified by the qualitative coders, as well as the codes that were assigned to each text span. additional details about the dataset as well as comparisons with previous qualitative coding datasets, which have all been closed-source, can be found in the annotations in cvdquoding were originally created as part of a larger project to investigate the interest of women at risk of cvds in using ai tools @xcite and were subsequently adapted for our purposes. during the interviews, women were asked about the health challenges they face and their opinions about using an ai tool to help them manage their risk of cvds. for this work, we extracted the 15 interview transcripts and tagged the text spans coded by each coder and the questions asked by the interviewer using xml. readers may refer to cvdquoding exhibits the properties of qualitative coding presented in section 3.1. in terms of data saturation, both coders create many new codes in the first five interviews, and this trend tapers off as they code more transcripts (figure 2 ). in addition, coder 1 finds new rare codes in every transcript whereas coder 2 ceases to create new codes by interview 7. moreover, both coders clearly exhibit significantly different annotation styles with coder 1 creating 207 codes with an average of 4 text spans per code and coder 2 creating 23 codes with an average of 19 text spans per code. this difference can be attributed to the specificity of coder-2's codes, which tend to be more abstract (e.g., "health measures taken") than coder-1's codes (e.g., "cvd-related examinations, tests or measures"). ## modeling approaches we investigate the ability of current nlp methods to rank lists of codes by relevance as well as detect novel codes from text spans. to this end, we describe our data formatting decisions and the modeling paradigms we explore. we choose to explore three distinct modeling paradigms, classification, information-retrieval and zero-shot prompting, based on the following intuitions. firstly, the classification paradigm assumes that the confidence scores of a multi-label classifier can be used to rank codes. secondly, the information-retrieval paradigm assumes that codes can be treated as documents and that text spans are the queries used to rank them. finally, the zeroshot prompting paradigm assumes that the descriptiveness of codes is sufficient for a large language model (llm) to rank them based on the text span. ## experiments we investigate our approaches to modeling the qcs task in order to determine the ability of current nlp-based methods to allow for a more usercentered approach to qualitative coding. we discuss the experimental setup to test our modeling paradigms as well as our evaluation methodology. ## results we present the results of our experiments with respect to both the original annotations and the human evaluation annotations. ## analysis we discuss the implications of our results in relation to the three properties of qualitative coding which we uncovered through our human-centric ap- in addition, the ranked-based metrics computed using human judgments point to a promising future for the downstream viability of qcs systems. the recomputed mrr and the additional results from the p@4 show that not only are our systems able to correctly rank relevant codes, but they are also likely to highly rank more than one relevant code, potentially forcing the coder to further reflect on their coding decisions without needing to consider irrelevant codes. ## conclusion in this work, we approached qualitative coding in a novel user-centered manner and showed that this approach gives rise to technical subtleties related to qualitative coding which have not been previously investigated. these subtleties include the importance of considering a coder's personal point of data saturation, the difficulty of detecting rare codes and the necessity to avoid homogenizing different annotation styles. we showed that each of these properties is reflected in both the cvdquoding dataset we introduced and the experimental results of our modeling of qcs. lastly, our human evaluation showed that solely relying on automatic metrics computed with respect to the original annotations made by the qualitative coders is insufficient to describe the performance of qcs systems. the results derived from our human evaluation show that qcs systems can consistently provide several relevant code suggestions and, thus, that this human-centric approach to qualitative coding may be able to truly assist researchers in their study of textual corpora.