AI Conversational Interviewing: Transforming Surveys with LLM s as Adaptive Interviewers (2025)

Abstract: Traditional methods for eliciting people’s opinions face a trade-off between depth and scale: structured surveys enable large-scale data collection but limit respondents’ ability to voice their opinions in their own words, while conversational interviews provide deeper insights but are resource-intensive. This study explores the potential of replacing human interviewers with large language models (LLMs) to conduct scalable conversational interviews. Our goal is to assess the performance of AI Conversational Interviewing and to identify opportunities for improvement in a controlled environment. We conducted a small-scale, in-depth study with university students who were randomly assigned to a conversational interview by either AI or human interviewers, both employing identical questionnaires on political topics. Various quantitative and qualitative measures assessed interviewer adherence to guidelines, response quality, participant engagement, and overall interview efficacy. The findings indicate the viability of AI Conversational Interviewing in producing quality data comparable to traditional methods, with the added benefit of scalability. We publish our data and materials for re-use and present specific recommendations for effective implementation.

## introduction structured surveys are popular tools to assess public opinion @xcite @xcite . these surveys typically gather individual orientations through self-reports, asking respondents to select from predefined options on fixed questions. this method allows for efficient data collection across large populations, producing structured, tabular data that is straightforward to analyze and comparable across respondents @xcite . due to these benefits, structured surveys hold a prominent position in both academic and commercial research. despite their established utility, structured surveys with predefined response options have significant limitations @xcite . their static and impersonal nature often leads to respondent fatigue, which can diminish engagement and, consequently, the quality of responses @xcite . more critically, the rigid format of these surveys constrains respondents from fully expressing their thoughts, restricting them from offering responses that researchers may not have anticipated @xcite @xcite @xcite . this limitation hampers the discovery of new phenomena and prevents a comprehensive understanding of the full spectrum of people's attitudes. an alternative to structured surveys is conversational interviewing, sometimes called indepth or semi-structured or qualitative interviewing @xcite @xcite . it involves interviewers engaging with respondents in a more open-ended format, allowing them to freely express their thoughts on topics of interest. the dynamic nature of conversational interviews helps alleviate respondent fatigue and permits the exploration of opinions beyond predefined response options. however, this approach requires skilled interviewers capable of conducting nuanced conversations, which limits its application to small sample sizes due to the associated costs. so, survey research faces a trade-off between depth and scale: researchers must choose between conducting in-depth explorations with small groups through or large-scale but rigid surveys. however, recent advances in natural language processing @xcite @xcite present new possibilities for addressing this dilemma. the conversational capabilities of instruction-finetuned large language models @xcite have made them applicable across various academic and industrial domains. because llms can engage in human-like conversations @xcite @xcite , they have the potential to assist or even replace human interviewers in conducting conversational interviews. by eliminating the costly need for human interviewers, llms could enable scalable in-depth conversations, potentially resolving the trade-off between depth and scale. contributions we contribute to the emerging paradigm of ai conversational interviewing by conducting the first close-up investigation of its practical implementation and performance (cf. figure 1 ): ## related work to implement and evaluate ai conversational interviews this study combines insights from three distinct lines of work that have rarely been combined. advances in ai research have facilitated multiple ongoing commercial and academic projects that use llm-powered chatbots for in-depth, qualitative, or semi-structured interviews, as they are interchangeably called @xcite . although implementations vary, the studies collectively highlight the potential of llms for conducting conversational interviews. yet, critical questions regarding the implementation remain unresolved and little is known about the relative performance compared with human-led interviews. qualitative studies have extensively explored best practices for conducting in-person interviews @xcite . our approach is to build on these insights when implementing ai conversational interviewing. studies in survey methodology have extensively examined how different interview implementations influence responses. one line of research has focused on interviewer and mode effects @xcite . the presence of an interviewer significantly impacts respondents, often leading to greater engagement but also increasing the likelihood of socially desirable responses @xcite . in this vein, studies on conversational interviewing has shown that a more active and flexible interviewer who engages with questions from respondents can improve data quality @xcite @xcite . 1 another important factor is the input mode. responses to open-ended questions vary depending on whether they are submitted via text or speech. text input typically requires more effort, which can result in shorter but more carefully considered responses @xcite . so, the responses will not necessarily be better or worse depending on input mode, but they will differ predictably, as text-and speech-based interviews elicit distinct psychological reactions from participants @xcite . ## study design and implementation our study pursues two goals: (a) assess the performance of ai conversational interviewing (in comparison to human-led interviewing) and (b) identify problems and opportunities for improvement of ai conversational interviewing. we conducted a small-n study among university students in a controlled environment. ahead of data collection, we pre-registered our research questions, research design, and evaluation metrics (cf. osf registry). we conducted both ai-led and human-led interviews as part of a class activity, where students were randomly assigned to serve as either inter-viewers or respondents in the respective conditions. identical questionnaires were used in both interview settings. after the interview, respondents filled out a structured questionnaire to evaluate their interview experience. in the ai interview condition, students monitored the interviewees in real-time to identify any technical issues. ## findings we collected data on six human-led and five aiconducted interviews. human-led interviews were audio-recorded and then transcribed. figure 2 presents an example snippet from an ai conversational interview, showcasing how the interviewer engages in active listening by occasionally repeating the preceding answer, as instructed. qualitative inspection of the transcribed data shows that both the ai and human interviewers faithfully followed the provided questionnaire. manual coding of all interviewer behavior shows that neither humans nor ai always acted in full accordance with the interview guidelines (figure 6 ). summarizing across all coded categories, we counted 72 violations per ai interview and 64 violations (↓ -11.11%) per human interview, on average. while error rates of human and ai interviewers were at similar levels, the nature of the errors differed. contrary to instructions, human interviewers often failed to engage in active listening, which involves restating the respondent's answer to ensure proper understanding. specifically, 94 percent of guideline violations related to active listening were committed by human interviewers, compared to only 6 percent by the ai interviewer (cf. appendix i)). conversely, and in contrast to internal pre-tests, the ai interviewer predominantly failed to follow the instruction to 'ask follow-up questions when a respondent gives a surprising, unexpected, or unclear answer,' with 88 percent of violations of this rule attributed to the ai interviewer. these findings highlight the challenge of finding the right balance between asking too many and too few follow-up questions in any in-depth interviewing setting. moreover, the fact that the interviewer model had previously succeeded in asking appropriate follow-up questions during internal tests serves as a reminder that even minor modifications to prompts can lead to unintended side effects. another guideline was to avoid any behavior that could bias the respondents' answers. however, despite the instruction to 'not take a position on whether their answers are right or wrong,' the ai interviewer occasionally judged the respondent, typically in an encouraging manner (e.g., 'your definition of politics is quite insightful', 67 percent attributed to the ai interviewer). in contrast, human interviewers sometimes erred by guiding respondents through associations or suggestions for their answers, accounting for 75 percent of such violations. overall, while no interviewer setting perfectly adhered to the guidelines, these findings suggest that ai interviewers demonstrate a similar level of effectiveness to human student interviewers in following instructions for in-depth interviewing. however, achieving optimal performance relies on fine-tuning and thoroughly testing model instructions. turning from the interviewer's behavior to the participants' responses, we see that both interviewing settings succeeded in eliciting answers from respondents at substantial lengths. in the ai interviewer setting, the average response length was 52.39 words. in the human interview setting, the average response length was 32.81 words one particularly important outcome is the assessed relevance of the responses-whether they are useful and directly related to the question asked. once again, no substantial differences in relevance were observed between ai and human interviews. while these estimates should be interpreted with caution due to the considerable imprecision associated with the small sample size, the findings suggest that engaging with an ai interviewer does not lead to a significant decline in response quality compared to a human interviewer. we interpret this as a proof-of-concept, underscoring the general viability of ai conversational interviewing. our setup allowed for a close-up investigation of how our ai interviews unfolded in practice. real-time problem recording during ai interviews showed that respondents interacted seamlessly with our user interface, which resembled familiar chat interfaces, indicating that no learning curve was necessary. yet, occasionally, the latency of the gpt responses was criticized (e.g. "sometimes the time it takes to produce an answer is unexpectedly long. but it is not really off putting.", "run time is quite slow, it takes a couple (>5 seconds)"). while this latency may reflect similar reaction times in human-to-human chat interactions, participants appeared to prefer shorter waiting times when they were aware they were interacting with an ai interviewer. our implementation was voice-assisted, allowing respondents to choose between text and speech for both the interviewer's output and their own input. while no issues were reported with the voice output of the interview questions, the realtime problem recording noted several instances where respondents reported technical issues with audio recording and transcription ("some problems with the microphone: sometimes does not record, speech recognition sometimes recognises words incorrectly", "small recurring problems with audio recording (not sure if it already runs, accidently stop in recording early", "recording just stopped completely for a couple seconds and interviewee was kinda mad about it"). our post-interview survey confirmed these issues. although five ai interview participants reported trying the audio recording function, only one found it to work sufficiently well to rely on it primarily during the interview. the remaining respondents either partly or primarily preferred to provide written answers to the ai interviewer. although unintended, this presents an analytical opportunity to explore differences between written and audio-recorded responses in the ai interviewer setting. as the survey-methodological literature suggests, the answers of respondents who relied on text input were significantly shorter (on average, 21 tokens per answer) than the answers by respondents who used audio-recorded throughout the ai interview (63 tokens per answer (↑ +67%). so, response length markedly varied with input mode. however, the survey-methodological literature indicates that audio-recorded responses should not be considered inherently superior but rather qualitatively different from written responses. one stu-dent observing a respondent providing written input noted that "the respondent does not have the opportunity to elaborate in a free way in the written answers. she was very focused on writing good sentences which hindered her in her elaboration", highlighting the distinct psychological processes associated with each input mode. further qualitative observation indeed suggests that text-based inputs encourage respondents to think before writing, whereas audio recording tends to prompt respondents to "think out loud", allowing them to develop their thoughts while speaking (see appendix f for an example). the response styles associated with audio-and text-based input modes are also reflected in objective measures we extracted from the transcribed interview data. textbased ai interviews achieved a flesch reading ease score of 77.66 while the fully audio-based ai interview scored at 48.32 (↑ +62,22%) @xcite ease score for human interviews: 62; higher values indicate higher readability). hence, compared to text input, audio input in ai interviews may be associated with longer but less elaborate answers. how did respondents experience the interviews? participants felt that both the human and ai interviewers were clear in their questions and that each understood their responses (figure 3 ). respondents in both settings left the interview satisfied. however, participants found the ai interview less interesting and were less likely to repeat it, possibly due to the technical problems with the audio recording. while emphasizing that a satisfactory interview hinges on a flawless technical implementation of the interview process, these findings suggest that the absence of a human interviewer does not necessarily need to go along with a deteriorated interview experience for the respondents. ## discussion and recommendations applying the questionnaire to a student sample with both human and ai interviewers demonstrates the general viability of ai conversational interviewing. when properly implemented, ai conversational interviewing can collect high-quality data. a comprehensive set of qualitative and quantitative metrics suggests that ai interviewing maintains quality comparable to that of human interviewing, but at significantly lower costs, thereby making in-depth interviews more scalable. although these findings highlight the potential of ai conversational interviewing, the success of the method depends on its precise implementation. based on our comprehensive analysis, we present five recommendations for the future development and employment of ai-driven in-depth interviews: leverage existing knowledge. when specifying desired interviewer behavior, it is crucial to draw on established principles from survey methodology. these practices, developed through extensive research and practical experience, offer proven guidelines for effective implementation. context-specific definition of desired interviewer behavior. it is crucial to make deliberate judgment calls to tailor the desired interviewer behavior to your specific research context. this may involve decisions on aspects such as the importance or frequency of follow-up questions, the depth of probing on certain topics, or the level of formality in the interview tone (for example, @xcite demonstrated the effectiveness of empathy prompting). each research project may require a unique approach to ai interviewer behavior to ensure the collection of appropriate data. consider user experience. the interface through which participants interact with the ai interviewer is crucial to the success of the interview. it is essential to rely on familiar and intuitive user interfaces that minimize cognitive load and technical barriers. well-designed interfaces enable participants to focus on providing thoughtful responses rather than being distracted by technical difficulties. careful prompting. the prompts provided to the ai interviewer are crucial to its performance. conduct thorough pre-testing to ensure that the ai's behavior aligns with your established guidelines. it is important to consider the potential unintended side-effects of modifying prompts, as even minor adjustments can lead to significant changes in interviewer behavior or question interpretation @xcite @xcite . ## conclusion our research contributes to the growing field of aisupported interviewing by offering initial insights through an in-depth evaluation process. we assessed ai performance using a variety of quantitative and qualitative evaluation methods, documenting the challenges participants faced and comparing ai-conducted interviews with human-led ones. to ensure transparency, we have made our pipeline, questions, and data publicly available. based on our preliminary findings, we propose five areas for consideration in future implementations: integrating established survey methodology principles, adapting ai behavior to different contexts, designing user-friendly interfaces, conducting comprehensive pre-testing, and being aware of input mode effects. while our results highlight the potential of ai conversational interviewing, it is important to recognize that outcomes are heavily dependent on the specific implementation methods used.