LLM -as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation (2025)

Abstract: We introduce LLM-as-an-Interviewer, a novel paradigm for evaluating large language models (LLMs). This approach leverages multi-turn interactions where the LLM interviewer actively provides feedback on responses and poses follow-up questions to the evaluated LLM. At the start of the interview, the LLM interviewer dynamically modifies datasets to generate initial questions, mitigating data contamination. We apply the LLM-as-an-Interviewer framework to evaluate six models on the reasoning, factuality and instruction-following tasks. Our results show that the framework effectively provides insights into LLM performance, including the quality of initial responses, adaptability to feedback, and ability to address follow-up queries like clarification or additional knowledge requests. The framework also addresses key limitations of conventional methods like LLM-as-a-Judge, including verbosity bias and inconsistency across runs. Finally, we propose the Interview Report, which aggregates insights from the interview process, providing examples and a comprehensive analysis of the LLM’s strengths and weaknesses. This report offers a detailed snapshot of the model’s real-world applicability.

## introduction with large language models (llms) becoming increasingly proficient in generating fluent free-form responses, it has become crucial to properly assess their capabilities and limitations @xcite . recently, llm-as-ajudge has emerged as a promising framework for automatic free-form response evaluation. compared to traditional lexical matching-based metrics (e.g., rouge @xcite , bleu @xcite ) or embedding-based metrics (e.g., bertscore @xcite ), previous works on llm-as-a-judge have reported higher correlations with human judgments @xcite @xcite . despite its potential, the llm-as-a-judge framework faces several practical limitations that hinder its widespread adoption, primarily due to its static nature @xcite . first, using a fixed set of test inputs raises concerns about data contamination @xcite @xcite , where the evaluated models may achieve high scores on instances encountered during training. second, single-turn interactions fail to thoroughly probe a model's true comprehension @xcite @xcite . for instance, the judge model may assess the confined performance of llms distant from the use case, be influenced by superficial factors (e.g., favoring longer responses), and exhibit high variance across runs. in this study, we propose llm-as-an-interviewer, a new paradigm for evaluating llms. inspired by human interviews, this approach starts with general questions but dynamically adapts by posing different types of questions based on the model's responses. as shown in figure 1 , the llm interviewer plays three key roles: (1) question modification, adapting benchmark datasets to generate diverse and challenging initial interview questions; (2) providing feedback, guiding the model to refine its responses; and we demonstrate the efficacy of our framework through experiments on three tasks: reasoning, llm-as-an-interviewer if a recipe for a two-pound cake requires how many cups are needed for 2 five-pound cakes? y cups of flour, clarifying question to solve this problem, you need the of flour for a two-pound cake. exact value of y, which is 1.5 cups it appears there was a misinterpretation of the question. the problem asks for ... we first find that … cups of flour will make a ten pounds cake. we can rearrange our equation to solve for and find that ... factuality, and instruction following. in §5, we evaluate six different models using gpt-4o as the interviewer. we demonstrate the impact of providing feedback on the model's performance and suggest that follow-up questions can offer deeper insights into its behavior. these follow-ups can help uncover failure reasons or assess performance on additional requests. furthermore, in § 6 and § 7, we show that our interactive and dynamic framework addresses key limitations of static evaluations, such as data contamination, verbosity bias in llm evaluators, and high variance across multiple runs. moreover, we find that the extended interaction between the llm interviewer and interviewee contains rich information that reveals the llm interviewee's strengths and limitations. based on this, we introduce a new evaluation protocol, interview report, which summarizes the interaction into a structured format. our contributions are as follows: ## llm-as-an-interviewer we introduce the llm-as-an-interviewer, which simulates an interview where one llm dynamically evaluates another. the llm being evaluated is referred to as the interviewee, while the llm conducting the evaluation is referred to as the interviewer. in this section, we describe (1) the overall interview framework ( § 3.1) and (2) the interview report, which summarizes and presents the results of the interview ( § 3.2). we design an interview framework for llms as illustrated in figure 2 . the interviewer plays three main roles throughout the interview process: (1) question modification, (2) providing feedback, and (3) generating follow-up questions. this structure ensures a comprehensive evaluation of the interviewee's capabilities. these roles are modular and pluggable, allowing users to enable or disable specific roles based on their evaluation needs. algorithm 1 provides the implementation of the interview process. ## experimental setup in this section, we describe the overall experimental setup used in § 5-6, as well as the design of the llm-as-an-interviewer framework. in our experiment, we use three tasks: reasoning, knowledge, and instruction following. for reasoning, we use math @xcite and the reasoning subset of mint @xcite . for knowledge, we use depthqa @xcite , a dataset of real-world stem-related questions assessing factuality, rebuilt from the tutoreval dataset. for instruction following, we leverage mt-bench @xcite , which evaluates an ability to follow multi-turn instructions. follow-up question type and example knowledge/factuality additional facts can you provide an example of determining the 6th roots of unity and specifying their arguments in radians? reasoning clarification what does it mean for a and b to be consecutive integers? rationale how did you determine jasmine's rate of water consumption per mile? instruction following modification of conditions if we change the previous question and assume that each sister of david has two brothers, how many brothers would david have? additional explanation can you provide a specific example of how you would address a potential objection your friend might have about public speaking in your persuasive email? correction can you now reformulate your earlier reply, outputting it in json format and only including books published after 1980? ## evaluating llms with llm-as-an-interviewer we simulate the interview process with various interviewee models. we present insights from interviews, focusing on the seed question-solving phase in § 5.1, and discuss the follow-up questions phase in § 5.2. interviewee models we use 6 models for the interviewee, gpt-4o, gpt-3.5, llama-3.1-{8,70}b, deepseek-math-7b, qwen2.5-math-7b. ## can llm-as-an-interviewer mitigate data contamination issue? this section demonstrates the efficiency of the modification strategy in generating questions based on established benchmarks. ## discussion we assess whether llm-as-an-interviewer exhibits verbosity bias and maintains robustness across multiple runs, a common consideration in model-based evaluation (zheng et al., 2023b). 7 interact. ## conclusion in this paper, we introduce llm-as-an-interviewer, a novel approach for evaluating llms through interviews. our multi-turn evaluation framework with interview report reveals the abilities of llms, such as revising answers based on feedback or handling follow-up questions, while providing comprehensive insights into the task. our analysis demonstrates that our framework mitigates common issues in static evaluation methods, including verbosity bias and data contamination. we hope our framework is widely adopted and brings a paradigm shift in llm evaluation.