Single- vs. Dual-Prompt Dialogue Generation with LLM s for Job Interviews in Human Resources (2025)

Abstract: Optimizing language models for use in conversational agents requires large quantities of example dialogues. Increasingly, these dialogues are synthetically generated by using powerful large language models (LLMs), especially in domains where obtaining authentic human data is challenging. One such domain is human resources (HR). In this context, we compare two LLM-based dialogue generation methods for producing HR job interviews, and assess which method generates higher-quality dialogues, i.e., those more difficult to distinguish from genuine human discourse. The first method uses a single prompt to generate the complete interview dialog. The second method uses two agents that converse with each other. To evaluate dialogue quality under each method, we ask a judge LLM to determine whether AI was used for interview generation, using pairwise interview comparisons. We empirically find that, at the expense of a sixfold increase in token count, interviews generated with the dual-prompt method achieve a win rate 2 to 10 times higher than those generated with the single-prompt method. This difference remains consistent regardless of whether GPT-4o or Llama 3.3 70B is used for either interview generation or quality judging.

## introduction a critical challenge for the development of conversational agents remains collecting sufficient amounts of data @xcite to be used for supervised fine-tuning or direct preference optimization @xcite . collecting such dialogue data can be done with crowd-sourced human workers, but this process is time-consuming and labor-intensive @xcite . as an alternative, the generation of synthetic dialogue data has emerged @xcite . furthermore, llms are not only used to develop synthetic dialogues but also to automatically evaluate the quality of the dialogues once they are generated @xcite . in our paper, we focus on generating high-quality job interview data. such data can be used to finetune or preference-optimize task-oriented dialogue systems for conducting job interviews with job candidates in various human resources (hr) contexts. following @xcite , we define a highquality dialogue as a dialogue that is indistinguishable from authentic human discourse. to generate the dialogues, we compare two different methods. recent works (e.g., @xcite and @xcite ) use a single prompt to generate the complete dialogue. others (e.g., duan et al. ( 2024 )) use two prompts, instructing llms to assume roles and carry out a conversation. in the case of a job interview, such roles typically comprise an interviewer and a candidate. we investigate the following research questions: to the best of our knowledge, our study is the first to rigorously conduct this comparison, providing a comprehensive evaluation of these dialogue generation methods. this analysis is particularly important due to the substantial cost disparities between the methods, with significant implications for research and real-world (e.g., hr) applications. for the remainder of this paper, "llama 3.3" refers to the 70b model, unless otherwise specified. our code and accompanying dataset are publicly available at: https://github.com/jdebaer/ dual-vs-single-prompt-hr-interviews . ## related work in this section, we examine the existing dialogue generation strategies and explore the role of llms as human-like evaluators of generated dialogues. ## methodology our objectives are (1) to compare single-prompt vs. dual-prompt job interview generation on dialogue quality using a judge llm, and (2) to examine if results are consistent across gpt-4o and llama 3.3 for dialogue generation and judging. to realize this, we first create interview seeds and then build a dialogue generation pipeline that uses those seeds to construct interviews. finally, we devise a strategy to rate interviews. to create the interview seeds, following the methodology of @xcite , we start with constructing a dataset of 100 summarized anonymous job histories, randomly selected from a larger job history dataset. 1 we summarize the job histories with gpt-4t. these summaries are then used as input seeds to generate job interviews, inspired by how @xcite use knowledge-based narratives to generate opendomain dialogues in the context of those narratives. for each summarized job history, we generate a set of four interviews by systematically varying the use of a single-prompt or dual-prompt generation strategy in combination with gpt-4o and llama 3.3. this ensures that each model is employed for both prompt strategies, ultimately yielding four distinct interviews. for llama 3.3 we invoke the llama-3.3-70b-versatile model via groq. 2 we consistently use a default temperature of 1 for all generating llms, to obtain a balance between creativity and coherence. for the dual-prompt strategy, we implement an interviewer and a candidate agent. each agent has a dedicated prompt, in which we specify its role, an expectation to pass the turing test, and an expected number of turns in the conversation that is going to follow @xcite . for the candidate agent, we also feed in a summarized job history. complete prompts are listed in appendix a.1. for the single-prompt strategy, we ask an llm to generate a complete interview, based on the same summarized job history that is used for the dualprompt strategy above. the complete prompt is listed in appendix a.2. after generating the interviews, we normalize them by removing double newlines and standardizing speaker labels. the normalized interviews have a moderate length difference across generation methods, which is nevertheless statistically significant, as verified by a kruskal-wallis h test @xcite . length difference can introduce bias when using an llm to judge texts, where longer texts usually get systematically preferred @xcite . we address this issue below. for each set of four normalized interviews (with each interview generated from the same seed but with a different prompt strategy and different llm), we perform pairwise comparisons using both gpt-4o and llama 3.3 (llama-3.3-70bversatile model via groq) as the judge llm. following @xcite , we set the temperature to 0 for our judge llms to favor reproducible results. following paireval from duan et al. ( 2024 ), we ask our judge llm to detect ai generation for a pair of provided interviews. the winning interview is the interview for which it judges ai generation to be less likely. in line with paireval, we allow the judge llm to also cast a tie, indicating that it considers both interviews to be equivalent. to avoid any bias based on the order in which interviews are presented in the prompt of the judge llm @xcite , we perform each pairwise comparison twice, alternating which interview comes first. following @xcite , we use all scores for our win rate calculation. the prompt of our judge llm is similar to the one used in duan et al. (2024)'s paireval, with three differences. first, we ask the llm to first provide its rationale and then its decision. using this order has been shown to create a more consistent alignment between rationale and decision @xcite . second, to streamline coding, we instruct the llm to generate responses in json format, a constraint that large models have been shown to handle robustly @xcite . third, we add "do not consider conversation length as a factor" to the prompt to eliminate the aforementioned potential interview length bias. the complete prompt for our judge llms is listed in appendix a.3. we calculate the win rate for each interview generation method m i using eq. ( 1 ). when calculating the win rate for a method, the denominator only contains the results from the pairwise comparisons in which that particular method participates. we explicitly include ties in our win rate calculation, as they are a non-negligible outcome category when using llms as judges @xcite . gpt-4o and llama 3.3 yield similar win rates when generating interviews (right column, labeled "both" in the almost identical win rate for gpt-4o and llama 3.3 as dialogue generators in our experiment is surprising, given that llm judges tend to favor their own generations @xcite . ## measuring the impact of length to assess the impact of conversation length on win rates, we use ordinal logistic regression @xcite . per interview, we subtract losses from wins and divide the resulting scores in 3 ranked buckets of equal range. the regression checks if the independent variable (interview length) has a statistically significant effect on the ranking outcome. for both the gpt-4o and llama 3.3 judges, character-based and word-based lengths have a statistically significant (negative) impact on ranking, but the impact is minimal. we examine whether our inclusion of the instruction "do not consider conversation length as a factor" in the llm judges' prompt influences the observed regression outcome by rerunning the experiment with this instruction removed and gpt-4o as the judge. interestingly, the regression fit still only displays a very minimal impact of length on win rate. we hypothesize that this may be due to our evaluation setup, which requires the llm to first generate a rationale before providing its final decision, deviating from @xcite . this deviation potentially encourages the model to ground its judgment more in the rationale it constructs rather than the dialogue itself, thereby diminishing the influence of superficial features such as length. ## agreement between llm judges when investigating whether the judgments of the gpt-4o and llama 3.3 llms correspond, initial results exhibit no discernible trend ( in summary, there is high agreement between the gpt-4o and llama 3.3 judge llms when we allow for flexibility in handling uncertainty. ## token counts we provide the average token counts for the interview generations in figure 1 . while the singleprompt strategy demands only one api call per interview, the dual-prompt strategy requires an api call per utterance, with the dialogue history provided as input. as a result, the token count of the dual-prompt approach increases quadratically with the number of utterances in a conversation (see appendix b). for the job interviews in our dataset, we observe an average sixfold increase in token count. ## conclusion and future work to generate job interview dialogues that are indistinguishable from authentic human discourse, a dual-prompt dialogue generation method achieves a win rate 2 to 10 times higher than when a single prompt is used, but with a sixfold increase in token count. the win rate is derived from pairwise interview comparisons, where a judge llm evaluates dialogue authenticity. the quality difference remains consistent regardless of whether gpt-4o or llama 3.3 70b is used for the dialogue generation. additionally, both models provide consistent evaluations when serving as the judge llm. assuming that llama 3.3 70b is available at a lower price point than gpt-4o, using llama can help mitigate the additional costs associated with the dual-prompt strategy. consequently, we consider the integration of llama 3.3 70b with the dual-prompt approach to be the optimal solution for generating synthetic job interviews. in future work, we aim to expand our quality criteria beyond assessing whether a dialogue reflects human-like interaction. specifically, we plan to incorporate an additional dimension that evaluates whether llm-generated interview questions align with best practices for job interviews in hr. to ensure adherence to industry standards, we will collaborate with hr professionals.