Coding Open-Ended Responses using Pseudo Response Generation by Large Language Models (2024)

Abstract: Survey research using open-ended responses is an important method thatcontributes to the discovery of unknown issues and new needs. However,survey research generally requires time and cost-consuming manual dataprocessing, indicating that it is difficult to analyze large dataset.To address this issue, we propose an LLM-based method to automate partsof the grounded theory approach (GTA), a representative approach of thequalitative data analysis. We generated and annotated pseudo open-endedresponses, and used them as the training data for the coding proceduresof GTA. Through evaluations, we showed that the models trained withpseudo open-ended responses are quite effective compared with thosetrained with manually annotated open-ended responses. We alsodemonstrate that the LLM-based approach is highly efficient andcost-saving compared to human-based approach.

## introduction in the qualitative data analysis (qda) @xcite , survey research basedon open-ended questionnaire responses is an essential method for the discovery of unknown issues and new needs. the grounded theory approach (gta) @xcite , a representative method of the qda, requires several manual complex procedures, referred to as "coding." as a result, analyzing large qualitative data becomes exceptionally challenging. in addition, for most confidential information such as survey responses, the use of external apis like chatgpt (openai, 2023) is prohibited by terms of service. therefore, we propose a pipeline that can automate parts of the grounded theory approach @xcite . to avoid the obstacle of not being able to use external apis such as chatgpt, we firstly generated and annotated pseudo open-ended responses, and used them as the training data for the coding procedures of gta as shown in figure 1 . through evaluations, we showed that the grounded theory approach pipeline trained with pseudo open-ended responses are quite effective compared with those trained with manually annotated open-ended responses. we also demonstrate that the proposed pipeline is highly efficient and cost-saving compared to human-based approach. the code is available at https://github.com/ zeni-y/naacl2024-coding-open-ended 2 related work initial studies on coding automation proposed symbolic approaches based on rules created by researchers or statistical approaches based on corpora @xcite . however, these approaches required considerable effort to develop rules, and such rules were not adoptable to different domains. to address these issues, more versatile methods using supervised learning have been proposed to label and cluster qualitative data @xcite @xcite . additionally, approaches that combine a rule-based method with a machine learning method to assist human coding tasks have also been proposed @xcite : coding an open-ended response on covid-19: comparison between by gta and by an llm ("property", "dimension", "label", and "category" are terminology of gta, where "property" represents the various important aspects in the raw data, "dimension" the variation of the property, "label" the summary of the response, and "category" a small number of classes that aggregate all the set of labels.) 2021; gebreegziabher et al., 2023). nevertheless, researchers still had to design the types and definitions of labels and clusters. in terms of automatically determining labels and clusters, there exists research that uses the lda topic model @xcite to generate preliminary category suggestions and support human qda procedures @xcite . however, as is known in gta, which involves properties, dimensions, labels, and categories, coding tasks that require aggregating information from bottom up demand an advanced linguistic interpretation ability to capture the various aspects of the data. thus, simply using bert or the lda topic model is insufficient to conduct coding tasks. large language models (llms) have achieved high performance in tasks similar to various steps in qualitative analysis, such as text clustering @xcite ), text summarization @xcite , aspectbased sentiment analysis @xcite . additionally, numerous comparative experiments between manual annotations and llmgenerated annotations have been conducted, and it has been shown that the llms can annotate with an accuracy comparable to that of humans @xcite @xcite . therefore, it can be inferred that llms have the potential for automating qualitative analysis. however, there has not been any research on using an llm to automate qualitative analysis approaches. ## experiments in this section, we described the each step in the proposed pipeline. the flow of this pipeline is shown in figure 3 . ## conclusion this study proposed a pipeline for automating parts of grounded theory approach, which typically requires many complex manual tasks. it also demonstrated that the proposed pipeline is significantly superior in terms of time and cost when compared to manual analysis. by employing the proposed method, it is possible to easily generate statistics, from a state where it is unclear what kind of content exists and to what extent throughout the data. future work will aim to improve the pseudo response generation method to enhance the performance of both the segmentation and category classification models. the performances of the segmentation model and classification model using pseudo responses were lower than that of models using real responses. refining the linguistic features of the pseudo-responses to more closely resemble real responses could potentially improve model performance since the performance of each model heavily depends on the quality of the pseudo responses generated by chatgpt. furthermore, this study has only automated the segmentation and category classification within the grounded theory approach. future efforts should aim to automate the extraction of properties and dimensions, label generation, and the unification of categories to automate all tasks.