G uide LLM : Exploring LLM -Guided Conversation with Applications in Autobiography Interviewing (2025)

Abstract: Although Large Language Models (LLMs) succeed in human-guided conversations such as instruction following and question answering, the potential of LLM-guided conversations—where LLMs direct the discourse and steer the conversation’s objectives—remains under-explored. In this study, we first characterize LLM-guided conversation into three fundamental components: (i) Goal Navigation; (ii) Context Management; (iii) Empathetic Engagement, and propose GuideLLM as an installation. We then implement an interviewing environment for the evaluation of LLM-guided conversation. Specifically, various topics are involved in this environment for comprehensive interviewing evaluation, resulting in around 1.4k turns of utterances, 184k tokens, and over 200 events mentioned during the interviewing for each chatbot evaluation. We compare GuideLLM with 6 state-of-the-art LLMs such as GPT-4o and Llama-3-70b-Instruct, from the perspective of interviewing quality, and autobiography generation quality. For automatic evaluation, we derive user proxies from multiple autobiographies and employ LLM-as-a-judge to score LLM behaviors. We further conduct a human-involved experiment by employing 45 human participants to chat with GuideLLM and baselines. We then collect human feedback, preferences, and ratings regarding the qualities of conversation and autobiography. Experimental results indicate that GuideLLM significantly outperforms baseline LLMs in automatic evaluation and achieves consistent leading performances in human ratings.

## introduction large language models (llms) have demonstrated their effectiveness in human-guided dialogue, in which llms are tasked with producing responses according to specific commands from human operators, such as instruction following @xcite ) and question answering @xcite . in this type of task, the primary duty of llms is to adhere to the instructions given by humans to ensure the generated output is accurate and close to human expectations, as shown in figure @xcite . however, tasks in the real world are more complex, necessitating greater autonomy from llms @xcite @xcite . for example, tasks such as interviewing are dramatically different from traditional tasks as interviewing is open-ended, without definitive or "perfect" outcomes. interviewing tasks demand that llms plan the interview procedure, manage the objectives, e.g., exploring the user's memory and life experiences in autobiography interviewing, and offer adaptive and personalized inquiries based on the users' responses. this conversation paradigm requiring llms to guide and manage the conversation, ensuring the conversation flows smoothly and the objectives are met, is termed as llm-guided conversation ( figure 1 ## llm-guided autobiography interview in this section, we first introduce llm-guided conversation and its general framework. then, we introduce guidellm as an implementation of llm-guided conversation, centering on interviewing. ## human subject experiments experimental configuration. a within-subject study with 45 participants was conducted at a large urban university campus in the us. participants interact with the interviewing agents powered by gpt-4o and ours guidellm, discussing the topic key scenes in the life story: positive childhood memory (appendix a) with each chatbot. to remove any biased factors, we use the nickname breeze and echo for the gpt-4o baseline and guidellm to make sure participants are unaware of the identity of the chatbot. the order to interact with chatbots is also randomized for each participant. due to resource constraints, each participant spent only 8 minutes chatting with each chatbot on a single topic. this differs significantly from the automatic evaluation protocols (section 4). as a result, the capabilities of guidellm, such as context management, goal navigation, and evaluation metrics, will be substantially affected and limited. in a follow-up survey (appendix j), participants indicated which model performed better, or it was a tie, and provided reasonings. participation is voluntary, with informed consent obtained online, and participants are compensated with a cookie. the study received irb approval from the university where the study was conducted. findings. overall, guidellm was preferred for conversation quality (figure 5a ), particularly in fluency and question identification (fluency: guidellm win @xmath0, baseline win rate≈33%; identification: guidellm ≈53.3%, gpt-4o≈42.2%). however, in terms of comfort, guidellm had a 31.1% win rate, while baseline had a 40% win rate. in autobiography quality, we do not observe significant differences emerged, possibly because this study used one topic and allowed a short interaction. since the guidellm uses modules such as context management and goal navigation for insightful and consistent narratives, longer engagement across multiple topics might have better highlighted the differences in autobiography generation. we further conduct llm-as-ajudge evaluation by prompting llms to compare the two human-interviewed autobiographies. as shown in figure 5b , we obtained consistent results as in section 4.4: guidellm achieves higher autobiography quality in general. previous ai experience affects participants' perceptions of the models: participants who frequently used ai (4-7 days weekly) tended to prefer guidellm for overall conversation quality (chi-@xmath1.56, @xmath2, p-@xmath3.03). open-ended responses indicated that daily ai users felt guidellm "asked questions to get a better understanding", "made more sense", was "more interactive", and asked "more personal questions". one user appreciated that "guidellm is more 0 20 40 60 80 100 fluency identification comfort 40% 26.67% 33.33% 53.33% 4.44% 42.22% 31.11% 28.89% 40.0% guidellm wr tie baseline wr (a) win rate of human preference on conversation quality. 0 20 40 60 80 100 percentage (%) insightfulness narrativity emotional impact 46.67% 4.44% 48.89% 60.0% 4.44% 35.56% 51.11% 4.44% 44.44% guidellm wr tie baseline wr (b) win rate of llm-as-a-judge results on human-interviewed autobiographies. in-depth as it tries to connect my past experience to my current life". overall, frequent ai users perceived guidellm as asking more focused and personal questions to explore in-depth childhood experiences. this aligns with guidellm 's memory graph extrapolation (mge), which allows it to explore unique properties and offer adaptive, personalized interview questions. frequent ai users favored guidellm for its emotional impact on autobiography (chi-@xmath4.24, @xmath5, p-@xmath6.07). one user noted that guidellm "truly laid out what it felt like to be at a dance competition pretty much just as i remembered it", while another mentioned it "described in immense detail my exact experience and made me feel like i was reliving it". this suggests how frequently ai users recognized guidellm 's training with the emotion detection module, which analyzes the emotions in user responses and assigns emotion categories and strengths to their utterances. ## conclusion in our study, we introduce guidellm, an llm-guided conversation framework that offers a promising shift from the commonly used userguided paradigm. guidellm's ability to facil-itate informative and creative dialogues through goal navigation, context management, and empathetic engagement proves effective, particularly in challenging tasks like autobiography interviewing. our assessments on event extraction correctness, conversation, and autobiography quality show guidellm's distinct edge over baseline llms.