Dialogue Scaffolding: Producing a Realistic Corpus of Human-Computer Open-Domain Dialogues Using a Spoken Dialogue System and C hat GPT (2025)

Abstract: Researchers in dialogue interaction have had a long-term interest in multi-domain human-computer conversations and how they differ from human-human conversations. Recently, research on dialogue has begun to rely more and more on corpus-based training of neural conversational models, and conversational LLMs such as ChatGPT. However, existing large open-domain dialogue corpora do not accurately capture the characteristics of social human-computer dialogue. This paper addresses this gap by synthesizing a new corpus of 4000 long social dialogues on 200 user-model based topics that we call User-Centric SocialChat (UCSC). We create UCSC with a novel method called Dialogue Scaffolding, where a real dialogue system, that competed successfully in the Alexa Prize, interacts with ChatGPT to generate conversations. The Dialogue Scaffolding method ensures that the dialogues closely resemble the social chat genre of human-computer dialogues. We evaluate UCSC to ensure quality and safety, and we measure lexical diversity and topic consistency to show that the conversations are not repetitive and stay on topic. We evaluate the utility of UCSC by fine-tuning a compact dialogue-level model, PerQy-DLM, and showing that it outperforms competitive fine-tuned models like COSMO, Vicuna, and RedPajama-Chat-3B.

## introduction researchers in dialogue interaction have long been interested in human-computer (hc) conversations and how they differ from human-human (hh) conversations, and there has been a concentrated longterm effort to create both types of dialogue corpora for research use @xcite @xcite @xcite @xcite inter alia. recently, there has been an upsurge of interest in hc social chat dialogue, where informa- @xcite i love reading about superheroes! i am familiar with dc and marvel. who is your favorite character? [u1] batman is my favorite. thank you for telling me about batman, i love talking about amazing characters. no lie, this topic is seriously awesome. who needs superman when you've got iron man, captain america, and the avengers? they're the ultimate heroes who save the day while looking cool as heck! [u3] true, avengers are impressive. figure 1 : part of a scaffolded social chat dialogue (full dialogue in figure 11 ). chatgpt has been assigned the user role (u), with specific instructions that responses model the statistical properties of real open-domain social sds users @xcite . a text-based version of the athena sds takes chatgpt's input as if it were a real user and produces the next response (a). we create ucsc with a novel method called dialogue scaffolding, where athena, an alexa prize spoken dialogue system (sds), that was a finalist in the alexa prize social bot competition, interacts with chatgpt to generate an initial set of seed dialogues over 17 topics that represent athena's policies @xcite @xcite . 2 these seed dialogues are then used in prompts to chatgpt to condition the synthesis of a corpus of 4000 social chat dialogues covering 200 user-model based topics. the prompts utilize a separate corpus of questions, answers, statements, and opinions covering those topics used in the athena system @xcite . the dialogue scaffolding method automatically ensures that the synthesized dialogues follow the policy and thus closely resemble the social chat seed dialogues, because each dialogue turn by chat-gpt is conditioned on the conversational context and the system's turns are controlled by the dialogue policy. subsequently, the resulting corpus is distinctly different from other corpora that represent the social chat genre, such as persona 1 alexa prize dialogues with alexa users are not publicly available because of user privacy concerns. amazon has also required conversations collected in the lab with paid subjects to be removed from github @xcite . chat, topical-chat, realpersonachat, or soda, among others @xcite @xcite . figure 1 provides an example of a seed sds dialogue about superheroes. we qualitatively evaluate ucsc to ensure quality and safety, and we measure lexical diversity to show that the conversations are diverse. measuring diversity helps ensure that ucsc conversations do not continuously repeat the same slot-filled template, and because llm-generations may be generic, bland, or lacking in specificity @xcite . we then evaluate the utility of ucsc by fine-tuning a compact 3b dialogue-level model, perqy-dlm, and show that it outperforms competitive conversational models such as cosmo 3b, gpt 3.5, vicuna 33b, and redpajama-chat-3b. our contributions include the method, the user-centric socialchat corpus, and the 3b perqy-dlm model. 3 2 related work there are many dialogue corpora available, and there are several survey papers or websites that aim to summarize them @xcite . there are other corpora of dialogue acts intended to be used to train natural language generators that can work well within a dialogue system @xcite @xcite , some of which cover a large number of domains. social chat can be oriented towards particular topics @xcite @xcite , may be knowledge grounded @xcite , or persona-grounded, using either an invented persona @xcite or the actual personas of the human conversants @xcite . the number of topics vary widely, over common topics such as music and movies, the sport teams the conversants support, their pets, or hobbies such as gardening. additionally, shared corpora of real human-gpt conversations that cover many different topics have recently become available, such as the 90k conversation sharegpt @xcite and the 1m lm-sys chat gpt @xcite . these conversations reflect what actual gpt users want them to do and are not necessarily social chat @xcite . the switchboard corpus (row 1 of before llms, most corpora aimed at hc social dialogue were crowdsourced. the self-dialogue corpus (row 2 of dailydialog (row 7 of the empathetic dialogues corpus (row 9 of prosocial dialog (row 11 of there is also a growing number of dialogue corpora generated synthetically using llms. carecall (row 12 of ucsc's statistics in the row 16 of ucsc is the only corpus designed with longer system turns than user, reflecting reported statistics of alexa prize dialogues @xcite . we report an mtld (mccarthy and jarvis, 2010) lexical diversity of 69.0, to confirm that the dialogues cover a diverse range of content and avoid repetition. these conversations are also topically consistent (see section 3.3), because they follow the dialogue policies of athena, whereas several crowdsourced conversational corpora have short topical segments with abrupt topic switches @xcite . ## generating synthetic social dialogues figure 2 shows ucsc's generation pipeline. we first collect a seed set of topic-focused hc conversations using dialogue scaffolding, i.e., the interaction between an sds and a conversational llm. in this work, we pair the athena alexa prize so-cialbot with chatgpt (gpt-3.5) @xcite , 2020; @xcite @xcite , but in principle, any dialogue system could be used. we collect a small set of seed dialogues covering the 17 topics for which athena had a rich set of knowledge and topical depth. because athena uses specifically programmed dialogue policies, chatgpt's responses are highly conditioned on a policy-specific trajectory, hence the final dialogue resembles conversations with athena. since athena was a finalist in the alexa prize @xcite , its dialogue policies were optimized for hc interaction, guaranteeing that the resulting corpus actually models policies of which an sds is capable. examples of these policies and a scaffolded dialogue are in appendix a. ## applying user-centric socialchat to evaluate ucsc in a realistic dialogue setting, we fine-tune a compact model and compare its performance against competitive fine-tuned dialogueoriented llms. to create shorter, more manageable contexts for fine-tuning and crowdsourced evaluation, we segment the long dialogues into sub-topic sub-dialogues using gpt-3.5 (prompt in figure 25 ). a second part of this process also identifies keywords associated with each sub-topic (prompt in figure 26 ). figure 8 illustrates two sub-dialogue segments from a question-oriented conversation about gardening. ## conclusion existing publicly available large-scale opendomain social dialogue corpora do not accurately capture the characteristics of human-computer interaction found in systems like those from the alexa prize @xcite . to address this, we introduce user-centric socialchat (ucsc), a dataset generated via a novel method called dialogue scaffolding, in which an sds interacts with chatgpt to produce seed conversations that serve as context for generating 4,000 extended social dialogues across 200 user-targeted topics. we evaluate ucsc for quality and safety, and we measure lexical diversity and topic consistency to confirm conversational variety and topic adherence. we demonstrate its utility by fine-tuning a compact dialogue-level model, perqy-dlm, which outperforms competitive 3b conversational models such as the fine-tuned redpajama chat, as well as larger models like gpt-3.5 for some metrics.