RACER : An LLM -powered Methodology for Scalable Analysis of Semi-structured Mental Health Interviews (2024)

Abstract: Semi-structured interviews (SSIs) are a commonly employed data-collection method in healthcare research, offering in-depth qualitative insights into subject experiences. Despite their value, manual analysis of SSIs is notoriously time-consuming and labor-intensive, in part due to the difficulty of extracting and categorizing emotional responses, and challenges in scaling human evaluation for large populations. In this study, we develop RACER, a Large Language Model (LLM) based expert-guided automated pipeline that efficiently converts raw interview transcripts into insightful domain-relevant themes and sub-themes. We used RACER to analyze SSIs conducted with 93 healthcare professionals and trainees to assess the broad personal and professional mental health impacts of the COVID-19 crisis. RACER achieves moderately high agreement with two human evaluators (72%), which approaches the human inter-rater agreement (77%). Interestingly, LLMs and humans struggle with similar content involving nuanced emotional, ambivalent/dialectical, and psychological statements. Our study highlights the opportunities and challenges in using LLMs to improve research efficiency and opens new avenues for scalable analysis of SSIs in healthcare research.

## introduction semi-structured interviews (ssis) are a widely used qualitative research method in healthcare research that provide an in-depth understanding of subjects' experiences in their own words @xcite . ssis require interviewers to ask pre-specified 'root' questions, along with the option to ask follow-up questions to gain clarity on the interviewee's responses. this flexibility is a key characteristic of ssis, allowing for a more dynamic and responsive data collection process, especially in areas where exploratory forays are needed. the adaptability of ssis is particularly beneficial in exploring com-plex or sensitive topics such as mental health. ssis allow rapport building between interviewer and subject and facilitate candid responses on sensitive matters. the open-ended nature of follow-up questions gives subjects the freedom to reflect on experiences and articulate thoughts without judgement. this helps reveal the nuances, contradictions, and diversity of perspectives that traditional fixed quantitative surveys may overlook. however, the traditional manual analysis of these interviews is a time-consuming and resource-intensive process. the advent of large language models (llms), such as gpt-4 @xcite , offers a novel and efficient method to extract and interpret data from such text corpora. yet, the validity of llms in analyzing emotional states may be limited in circumstances where participants express multiple emotions or conflicting (dialectical) states. as a case-study, we leveraged data from ssis, conducted during the peak of the covid-19 crisis in 2020, to understand the mental well-being of 93 healthcare professionals and trainees. the covid-19 pandemic brought to the forefront profound personal and professional challenges experienced by healthcare workers. fear of infecting family members, grief over patient deaths, moral dilemmas in resource allocation, and anxieties about professional preparedness collectively introduced a heightened level of psychological complexity and stress in the lives of healthcare professionals. the stigma surrounding the pursuit of mental health support exacerbated these challenges, leaving healthcare workers hesitant to openly discuss their difficulties or seek assistance. in this paper, we developed racer, an expertguided automated pipeline that retrieved responses to about 40 questions per ssi, aggregated responses to each question across all subjects, clustered these responses for each question into insightful domain-relevant expert-guided themes @xcite , and finally re-clustered responses to produce a robust result. human evaluation on a subset of the total population revealed moderately high agreement @xcite between humans and racer outputs, and similarities between inter-human disagreement and humanmachine disagreement. we summarize our findings from applying racer to our ssi-survey on the experiences of healthcare professionals and trainees during covid-19, to reveal the power of this approach. our results demonstrate both the capabilities and the limitations leveraging llms to efficiently process and extract insights from a large corpus of ssis. ## results the study population of healthcare professionals and trainees consisted of 93 subjects (51 male, 42 female) with diverse demographics ( ssis were conducted over videoconferencing using a standard template consisting of a total of 41 questions, including four questions that were only asked to students, and seven questions that were asked to only non-students. questions were either factual, concerning demographics and personal and professional background, or open ended, where interviewees were asked to talk about their experiences during the covid-19 pandemic, focusing on their exposure to the virus, work impacts, emotional responses, future outlooks, and coping strategies. interviewees discussed how they had practiced in high-risk areas, their concerns for personal and family safety, and modifications made to their routines. they also reflected on the physical toll the crisis had taken. the impact on their work included changes in working hours, shifts in patient care quality, and altered management approaches. emotional and psychological questions revealed how the crisis affected them emotionally, the level of support they received, family dynamics, and changes in burnout levels. looking ahead, they pondered the crisis's short-term and long-term impacts on their careers and specialty choices. finally, they shared their openness to seeking help for burnout or mental overwhelm and identified potential obstacles in obtaining this help. students were not asked clinical-practice related questions, and were instead asked about how their training was being affected by pandemic-related changes. interviews lasted on average 26.7 +/-8.9 s.d. minutes. when transcribed from raw interview audio into text transcripts (using otter.ai(otter.ai, 2023)), were on average 4044.30 +/-1348.34 s.d. words long. ## insights using racer on healthcare worker experience during covid-19 we summarize racer-derived insights from analyzing our 93-subject ssi corpus in appendix b.