Empathy vs Neutrality: Designing and Evaluating a Natural Chatbot for the Healthcare Domain (2025)

Abstract: As lifestyle-related diseases rise due to unhealthy habits such as smoking, poor diet, lack of exercise, and alcohol consumption, the role of Conversational AI in healthcare is increasingly significant. This study provides an empirical study on the design and evaluation of a natural and intuitive healthcare chatbot, specifically focusing on the impact of empathetic responses on user experience regarding lifestyle changes. Findings reveal a strong preference for the empathetic chatbot, with results showing statistical significance (p <0.001), highlighting the importance of empathy in enhancing user interaction with healthcare chatbots.

## introduction in our contemporary healthcare situation, lifestyle-related diseases are increasing, primarily influenced by unhealthy habits such as smoking, poor diet, lack of exercise, and alcohol consumption @xcite . simultaneously, conversational ai, or chatbots, have gained popularity and emerged as powerful tools, particularly in the healthcare sector @xcite . nevertheless, the use of conversational agents in the healthcare domain is not too widespread, especially when compared to other industries such as travel and hospitality @xcite . furthermore, very little is known about how the linguistic design of a medical conversational agent can impact the users' likelihood to employ it for their healthcare queries @xcite . this paper focuses on the design and evaluate a natural and intuitive chatbot for the healthcare domain, including an empirical analysis of the results. more specifically, we investigate how the use of empathy in generated messages can affect user experience during queries about lifestyle changes, hence influencing the likelihood to incorporate a healthcare conversational agent in their daily lives @xcite . the two primary contributions of this study are: ## conversational agent design the decision to use a llm-powered chatbot was driven by the need for a system capable of understanding and generating natural language with a high degree of fluency and contextual awareness. unlike traditional rule-based or retrievalbased chatbots, which rely on predefined scripts or a database of responses, an llm-powered chatbot can generate nuanced, contextually appropriate responses based on the specific needs of the user at any given moment. one of the primary advantages of llms is their ability to process complex language inputs, making them well-suited for conversations that require deep contextual understanding, such as those in the healthcare domain. given the nature of healthcare queries, which often involve detailed and sensitive information, it was essential to implement a system that could handle such complexities with a high degree of accuracy and flexibility. the main objective of the chatbot is generating responses to user queries in a manner that is both informative and aligned with the specific version (empathetic or neutral) being tested. the implementation of both the empathetic and neutral version is the same, except for the specific prompt used. in our first experiment, we evaluate different llms to decide on the most suitable for our task. the implementation of the chatbots involved using the autotokenizer from hugging face to preprocess and tokenise input data, ensuring compatibility with the model and efficient handling of user queries. the chatbot's llm was run locally using a gpu cluster, which was crucial for managing the computational demands of real-time text generation during user interactions. the web component of the chatbot was built using flask, a lightweight web framework for python, chosen for its simplicity and effectiveness in developing web-based applications. ## experiment i: llm evaluation the first experiment consisted of an evaluation of the responses generated by different llms, where the best-performing llm was used as the basis of the chatbot in the user experiment (experiment 2). ## experiment ii: user experiment we further compared the user experience with the neutral and empathetic conversational agents based on medalpaca. ## conclusion this paper aimed to investigate how the use of empathy in generated messages can affect user experience during queries about lifestyle changes. the two primary contributions of this study are to provide insight in the impact of empathetic versus neutral tones in messages in a llm based chatbot, and to understand user expectations in human-computer interactions -using chatbotsin the healthcare domain, especially on lifestyle changes. the results of the first experiment show the differences between different llms, specifically two domain-specific and two general ones, on the different metrics fluency, coherence, groundedness and naturalness. these differences are not big, and the model with the most naturalness on the mash-qa dataset concerning lifestyle questions -medalpaca -is chosen as the model to use in the second experiment. the results of the second experiment show that empathy plays a crucial role in enhancing user satisfaction. the empathetic chatbot significantly outperformed the neutral chatbot across all dimensions measured by the chatbot usability questionnaire, including overall user experience, linguistic perception, and usability (@xmath0.001). this outcome highlights the importance of empathy in chatbot communication, especially in healthcare settings where users are likely to seek comfort and understanding. beyond just evaluating chatbot performance, it was essential to analyse what users expect from these interactions and how these expectations shape their experience. results revealed that users expect healthcare chatbots to offer more than just accurate and relevant information -they expect to participate in an interaction that mirrors human conversation. the high cuq scores for the empathetic chatbot suggest that when these expectations are met, users are more satisfied and more likely to view the chatbot as a trustworthy and effective tool for asking about health advice. some of the limitations of this work include that the user experiment was specifically set to 4 scenarios and the participants recruited were not searching actively for lifestyle change. although the participants were free to use their wording, the scenarios were quite restricted. in future work, it would be nice to conduct the experiment in a more realistic setting with more participants to verify our findings. additionally, the sample size was relatively small and homogeneous, which hinders the generalisation of the results to a broader population. for example, individuals from different educational backgrounds or age groups might prioritise straightforwardness over empathy, which could yield slightly different results over the preferred tone in messages. future work could replicate the experiment with a larger, more diverse sample to verify whether these preferences could be applied universally or are influenced by specific demographic factors. in summary, it is evident that the most effective healthcare chatbots are those that balance generating accurate medical information with an empathetic dialogue style. while other general linguistic capabilities are important, the success of a healthcare chatbot heavily relies on its ability to communicate empathetically and align with human conversational patterns. this project has demonstrated that incorporating empathy into chatbot design can significantly improve user experience, making these tools more appealing and effective in supporting lifestyle changes and health-related decision-making.