“What’s Up, Doc?”: Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets (2025)

Abstract: People are increasingly seeking healthcare information from large language models (LLMs) via interactive chatbots, yet the nature and inherent risks of these conversations remain largely unexplored. In this paper, we filter large-scale conversational AI datasets to achieve HealthChat-11K, a curated dataset of 11K real-world conversations composed of 25K user messages. We use HealthChat-11K and a clinician-driven taxonomy for how users interact with LLMs when seeking healthcare information in order to systematically study user interactions across 21 distinct health specialties. Our analysis reveals insights into the nature of how and why users seek health information, such as common interactions, instances of incomplete context, affective behaviors, and interactions (e.g., leading questions) that can induce sycophancy, underscoring the need for improvements in the healthcare support capabilities of LLMs deployed as conversational AI. We release code and artifacts to retrieve our analyses and combine them into a curated dataset for further research.

## introduction large language models (llms) have demonstrated significant medical knowledge which has translated to proficiency at a range of clinical tasks including differential diagnosis, interpretation of health records, and medical summarization @xcite @xcite . these impressive capabilities, coupled with the costs and inaccessibility of traditional medical care, have led a growing number of people to turn to llm-based chatbots to seek healthcare information. one 2024 survey found that 31% of us adults were turning to generative ai for health requests, seeking help with self-diagnosis, treatment management, and support needs @xcite . despite this surge in public use, the vast majority of evaluations of llms in medicine rely on benchmarks that focus on clinician-or researcheroriented tasks @xcite . these benchmarks often assume a structured, professional context that differs substantially from how patients engage with chatbots in the real world. while public datasets focused on consumer health queries do exist, they take the form of single-turn health queries or synthetic interactions @xcite @xcite . unfortunately, these existing benchmarks provide a suboptimal proxy for the open-ended, often ambiguous questions posed by lay users over the course of multi-turn conversations. a recent largescale user study found that the clinical knowledge measured by current synthetic benchmarks is insufficient to account for the failure modes surfaced via real human interactions @xcite . additional recent work has shown that llms are suboptimal at soliciting further details when only partial information is provided for differential diagnosis @xcite . furthermore, llms are known to exhibit problematic tendencies such as sycophancy, overconfidence, and hedging, which can seriously impact the quality and reliability of healthcare information given to users @xcite @xcite . given these known limitations of llms, there is an urgent need to characterize real-world communication patterns of people querying chatbots for healthcare information to understand these behaviors (and their corresponding risks) under realistic conditions. in this work, we investigate and systematically analyze common interactions, instances of providing incomplete context, affective behaviors, and interactions (e.g., leading questions) that can induce sycophancy by real-world users engaging with llm-based chatbots for healthcare information. in collaboration with clinicians, we (a) develop a clinician-driven healthcare interaction taxonomy in order to (b) classify and annotate a curated dataset of conversations filtered from large-scale conversational datasets such as lmsys-chat-1m @xcite and wildchat-1m @xcite ). an llm is used to apply conversation-level specialty annotations and message-level taxonomic code annotations to the curated dataset. subsequently, we leverage our annotations to (c) analyze users' common interactions, instances of providing incomplete context, affective behaviors, and interactions (e.g., leading questions) that can induce sycophancy. our contributions are as follows: to foster future research, we have released code and artifacts to retrieve our analyses and combine them into a curated dataset here: https://github. com/yahskapar/healthchat 2 related work online health information searching there is a significant imbalance between clinician availabil-ity and patient needs (for health workforce analysis, 2024). in response, patients have long turned to the internet, e.g. search engines and symptom checkers, to independently seek health information @xcite . although these tools can be useful, they can also misinform patients (even unintentionally), with some patients placing greater trust in online information than their providers @xcite . more recently, generalpurpose generative ai tools, such as chatbots, have emerged as a new avenue for seeking health information @xcite . however, beyond survey-level insights, we know little about how patients engage with these ai systems in multi-turn interactions or how such behavior compares to earlier modes of online information search @xcite . the potential for llms to dramatically improve information accessibility coupled with their potential harms highlight an urgent need to better understand the nature of these interactions, as we do through our clinician-driven healthcare interaction taxonomy. ## dataset creation we first curate a dataset of health information seeking conversations and then apply conversation-level and message-level taxonomic annotations in a scalable manner. our dataset curation process consists of several distinct steps -1) we use labels from both wildchat-1m and lmsys-chat-1m to filter out any non-english, toxic conversations. then, 2) we create and apply llm-based filtering prompts to filter out non-health conversations using gemini 1.5 pro. 20 conversations are evaluated at a time and subsequent refinement of the prompt occurs after manual inspection of filtered conversations. the initial, llm-based filtering prompt targeting the removal of non-health, academic, fictional, and notsafe-for-work (nsfw) conversations brings us to approx. 37k conversations, after which 3) another, llm-based filtering prompt with few-shot examples is applied in order to more explicitly reject conversations that were observed to be undesirable through empirical observation. following the aforementioned llm-based filtering prompt with few-shot examples, we end up with approx. 16k conversations. we additionally 4) manually mark and filter out approx. 400 conversations and utilize a pre-trained sentence transformer model @xcite , all-minilm-l6-v2, to filter out approx. 100 additional conversations that had too high of an embedding similarity (≥ 0.9) to undesirable conversation examples. then, 5) we perform de-duplication to remove an additional 2000 conversations. after proceeding with taxonomic annotation that is further detailed in section 4.3, we also take one final step to curate our dataset: 6) utilizing d (out-of-scope) codes to filter our dataset. after dropping any conversations where more than half of the user messages include d codes, we end up with a final dataset of 11k real-world conversations and 25k user messages. further details (e.g., prompts, hyperparameters) of our dataset curation process can be found in appendix a. our final dataset statistics can be found in ## a taxonomy for conversations seeking health information we develop and apply a taxonomy to (i) allow for more robust identification and categorization of conversations where users are genuinely seeking healthcare advice amidst broader, open-domain interactions, and (ii) enable fine-grained analysis of users' interactions with llms for health-related concerns. our healthcare interaction taxonomy was built in collaboration with two clinicians; an abridged version can be found in figure 1 and the full version can be found in appendix b.1. ## taxonomic analysis the creation of our taxonomy in section 4 in collaboration with clinicians and subsequent annotation (section 4.3) enables us to analyze common user interactions while seeking health information. additionally, we analyze the long tail of health information seeking interactions through b10 -a taxonomy code that targets health-related inquiries not covered by other information request (b) codes. ## case studies alongside our taxonomic analysis, we perform three preliminary case studies that investigate instances of incomplete context, affective behaviors, and interactions (e.g., leading questions) that can induce sycophancy. de-identified examples of conversations corresponding to these case studies, with taxonomy code annotations, are provided in figure 4. ## conclusion our work systematically analyzes how users seek health information from llms, using healthchat-11k, a new large-scale dataset of real-world conversations annotated with a clinician-driven taxonomy. we investigate common interaction patterns alongside case studies of incomplete context, affective behaviors, and interactions (e.g., leading questions) that can induce sycophancy, revealing key gaps in the capabilities of current healthcare chatbots. our findings show that real-world user interactions are often non-intuitive: users frequently engage in repetitive information-seeking loops, especially for ambiguous problems (e.g., mental health), and may unintentionally omit context, leading to potentially misleading responses. analysis of negative interactions (disagreement, distress) and leading questions that can induce sycophancy reveals patterns that represent critical moments for chatbot intervention. understanding these behaviors is paramount for developing llm-based chatbots that can cautiously handle user-led suggestions and provide safe, effective healthcare support. to this end, our annotated dataset provides a vital resource for the community to further investigate and benchmark real-world conversational dynamics. future work. future work could focus on expanding the linguistic and topical scope of interaction analysis and moving beyond english-only conversations. a crucial next step involves analyzing the llm's contributions to dialogue, assessing how different model response strategies impact user behavior, an aspect our current study did not cover. this could lead to a more comprehensive understanding of the dyadic nature of health information seeking conversations. further research should also aim to develop and evaluate llm interventions designed to strategically address common user interactions, including contextualized, negative ones such as disagreement or the propensity to ask leading questions. this could involve creating adaptive dialogue strategies that enhance user understanding and mitigate risks like sycophancy. our annotated dataset can also lead to the development of new benchmarks that assess the quality of user-llm health interactions across a diverse variety of real-world interaction modes.