Design, Generation and Evaluation of a Synthetic Dialogue Dataset for Contextually Aware Chatbots in Art Museums (2025)

Abstract: This paper presents the design, synthetic generation, and automated evaluation of ArtGenEval-GPT++, an advanced dataset for training and fine-tuning conversational agents with artificial awareness capabilities targeting to the art domain. Building on the foundation of a previously released dataset (ArtGenEval-GPT), the new version introduces enhancements for greater personalization (e.g., gender, ethnicity, age, and knowledge) while addressing prior limitations, including low-quality dialogues and hallucinations. The dataset comprises approximately 12,500 dyadic, multi-turn dialogues generated using state-of-the-art large language models (LLMs). These dialogues span diverse museum scenarios, incorporating varied visitor profiles, emotional states, interruptions, and chatbot behaviors. Objective evaluations confirm the dataset’s quality and contextual coherence. Ethical considerations, including biases and hallucinations, are analyzed, with proposed directions for improving the dataset utility. This work contributes to the development of personalized, context-aware conversational agents capable of navigating complex, real-world environments, such as museums, to enhance visitor engagement and satisfaction.

## introduction recent progresses in conversational ai have been achieved thanks to large language models (llms), which excel in generating high-quality responses following carefully designed prompt instructions @xcite . then, these same llms can be improved further by fine-tuning them on diverse human and synthetic instruction-based datasets and enriched with techniques like reinforcement learning from human feedback (rlhf), showcasing remarkable in-context learning capabilities @xcite @xcite . this has led to breakthroughs in natural language understanding, dialogue coherence, and adaptability across various domains. despite these advancements, there remain significant challenges in creating conversational agents that are not only knowledgeable but also socially competent and context-aware. specifically, we are interested in the incorporation of artificial consciousness features, situational awareness, or user profiling into conversational systems which are relevant characteristics for improving user experience. addressing these gaps can enhance user engagement, improve adaptability to dynamic interactions, and enable the development of more intuitive and meaningful dialogues @xcite . in this paper we focus on generating and enhancing the quality and scope of datasets for training such systems. building on prior work, particularly the artgeneval-gpt dataset released by @xcite foot_0 and the methodology described in @xcite , we introduce artgeneval-gpt++. this new dataset is specifically designed for training conversational agents in a museum context, where the chatbot acts as a tour guide, tutor, or art expert that adapts to the knowledge level of the visitor, age, ethnicity and even physical appeareance of the visitor. the dataset also includes external situations or interruptions that could happen in a museum. the updated version also includes improved emotional balance, refined and less hallucination dialogue scenarios, and enhanced personalization features such as chatbot tone, multiple visitors and response strategies. key contributions of this work include: by focusing on these enhancements, this work contributes to advancing the state-of-the-art in conversational agents, paving the way for more adaptive and socially aware ai systems.). the paper is structured as follows: section 2 review the seed datasets used in this work. section 3 details the methodology for dataset design, dialogue generation and automatic evaluation. section 4 presents the results and examples of the generated dialogues. finally, section 5 discusses conclusions and future work. ## methodology after inspecting the artgeneval-gpt dataset and using it for training our own chatbot, we found the need to improve the diversity of the dataset, and at the same time replace low-quality dialogues (primarily due to limitations of chatgpt-3.5) and incorporate additional mechanisms to mitigate hallucinations by performing additional preprocessing to the artworks selected for generating the new synthetic dialogues. in addition, with the goal of increasing the capabilities of the chatbot to showcase awareness and extended social competences, we incorporated new ideas and situations. this section describes the design process. in first place, the new version uses the more powerful gpt-4 model (specifically, gpt4o-mini vs 2024-07-18) and introduce refined preprocessing to improve information accuracy and dialogue consistency. these updates provide new dialogue scenarios with nearly 12,500 new dialogue interactions. 3 summarizes the methodology consisting on three phases: pre-processing, generation, and evaluation. note: the indicated csv files are used to record the output of the different methodology steps for logging purposes. ## conclusions and future works this paper presents a comprehensive approach to designing, generating, and automatically evaluating synthetic dialogue datasets tailored for training aware and socially improved chatbot applications in art museums. through the refinement of the previous artgeneval-gpt dataset, the new version provides a better and extended dataset comprising dialogues over 821 artworks from 384 artists across 26 art styles, ensuring familiarity with gpt models (less hallucinations). by expanding dialogue scenarios to include diverse visitor attributes (including groups, ethnicity, age, physical appearance) and interaction contexts, the new dataset that can be used for training chatbots to engage effectively with varied museum audiences. approximately 12,500 dialogues were generated using the latest gpt-4-turbo model, simulating realistic visitor interactions and demonstrating the chatbot's capacity for meaningful engagement. to automatically assess the quality of the generated dialogues, an automated framework was included that relies on objective metrics such as wer, bleu, and accuracy using llm-as-judge, achieving highquality scores (85.5% ). notable challenges in sentiment prediction and ethnicity detection indicate areas for further improvement, highlighting the complexity of emotional and contextual nuances in dialogue systems. thus, the dataset provides a valuable resource to advance conversational ai in cultural heritage settings. future work will focus on several key areas to enhance unconscious and conscious capabilities. first, expanding the dataset by incorporating a larger number of artworks (including well-known and more emotionally compelling paintings), multiple languages (beyond english, e.g., spanish, french, german, italian, portuguese, japanese or chinese), and including the actual image data during the generation to reduce hallucinations and improve contextual understanding. methodological improvements are also needed by evaluating the tutor scenarios, where the simulated visitors make mistakes in 30% of interactions. this is critical to measuring the efficacy of the chatbot in classroomlike settings for children and teenagers. we are also considering performing human subjective evaluations for evaluating the quality of the dataset and a fine-tuned version of the chatbot. lastly, based on the attention schema theory (graziano, 2022), we would like to explore extended reasoning capabilities to model the mind of the visitors, as well as enhancing explainability through tailored prompts. these directions promise to advance the personalization, contextualization, and consciousness of chatbots in our case for cultural heritage contexts.