G roun D ialog: A Dataset for Repair and Grounding in Task-oriented Spoken Dialogues for Language Learning (2023)

Abstract: Improving conversational proficiency is a key target for students learning a new language. While acquiring conversational proficiency, students must learn the linguistic mechanisms of Repair and Grounding (R\&amp;G) to negotiate meaning and find common ground with their interlocutor so conversational breakdowns can be resolved. Task-oriented Spoken Dialogue Systems (SDS) have long been sought as a tool to hone conversational proficiency. However, the R&amp;G patterns for language learners interacting with a task-oriented spoken dialogue system are not reflected explicitly in any existing datasets. Therefore, to move the needle in Spoken Dialogue Systems for language learning we present GrounDialog: an annotated dataset of spoken conversations where we elicit a rich set of R&amp;G patterns.

## introduction and motivation many conversations are impromptu back-and-forth interactions that often have no prior preparation or review. as a result, conversational breakdowns @xcite may occur due to minor misinterpretation, mishearing, misspeaking, or a general lack of common ground @xcite . interlocutors use repair mechanisms (albert and de ruiter, 2018) to detect and resolve communicative problems during conversations; and grounding mechanisms to establish common ground. for example, we often ask our interlocutors to repeat what they said, explain themselves, request clarifications, etc. such processes arise proactively or when the initial communication attempt has failed, during which modification and revision to the previous utterances are needed to proceed the conversations naturally. according to @xcite , r&g is meaningful in the following perspectives: 1) repair the discourse when breakdown occurs and 2) avoid conversational breakdowns. based on @xcite , there are plenty of strategies they can adopt to prevent the breakdowns during communications: 1) relinquish topic control; 2) simplify topic by asking "yes-no" questions; 3) confirm comprehensions of speakers before proceeding, etc. from the perspective of a language learner, dialogues serve as important media in language acquisition and learning @xcite . when language learners chat with highproficiency speakers, language learners make considerable efforts to ground what they have to say @xcite . more specifically, the low-proficiency speakers (lps) attempt to negotiate the meanings of conversations with high-proficiency speakers (hps). according to @xcite and @xcite , interactional processes including negotiation for meaning and various kinds of repair and grounding are among the many ways learners gain access to the second language acquisition. besides, lps can also en-hance their language skills, general communication skills and cultural knowledge during the conversations with hps @xcite . while r&g is common in nearly all conversations, it is particularly important for language learners as learners are still building up the full understanding of the language. they may also bring r&g influences of their primary language into the language they are learning. it is also possible that low-proficiency speakers (or language learners) employ additional or different r&g mechanisms than high-proficiency speakers of a language. therefore, there is a lot to know about r&g mechanisms from low-proficiency speakers. in this paper, we present a dataset that can help linguists and other researchers with several novel linguistic tasks such as identifying r&g patterns. further, while repair and grounding is an important linguistic mechanism, it is rarely reflected explicitly in the design of spoken dialogue systems that aim to help people learn a new language. our dataset can fill this gap by allowing researchers to model dialogue state tracking with r&g, generating responses with r&g turns, etc. we collected this dataset by connecting a high-proficiency speaker and a low-proficiency speaker on a crowd sourcing platform. the highproficiency speaker played the role of a human resources (hr) assistant in a wizard-of-oz style and was tasked to convey information about an interview. the low-proficiency speaker played the role of an interviewee and was tasked with finding specific information about the same interview through their conversation with the highproficiency speaker. while r&g may occur as a course of natural conversation, we further induced it by giving the interlocutors some conflicting and incomplete pieces of information. we collected the voice of the low-proficiency speaker and the text responses of the wizard. to the best of our knowledge, groundialog dataset is the first task-oriented dialogue dataset specifically tailored for repair and grounding in spoken conversations between high-proficiency and low-proficiency speakers. each dialogue in the dataset is transcribed by human experts and contains vocal markers and disfluencies, such as "uh" and "um". it is annotated with r&g types, intents, and slots that are relevant to dialogue state mapping tasks. hence, groundialog can be used to develop a task-oriented conversational agent, equipped with the r&g ability to detect communicative trouble, and adopt certain strategies to repair the discourse when trouble occurs. the rest of the paper presents related work, details of the data collection process, the data annotation scheme, analyses of the data, and initial model benchmarks. ## related work as indicated in @xcite , task-based language teaching (tblt) puts emphasis on the utilization of tasks as the critical element in the language classroom given that tasks can offer better contexts for active language acquisition and second language promotion. from the perspective of dialogue systems, it is the task-oriented dialogue (tod) that can help language learners achieve their proficiency goals through task completion. previous dialogue systems have shown great promise in increasing second language acquisition proficiency. @xcite provide an overview of all spoken dialogue systems for language learning. timpe-laughlin et al. (2022) have compared learning language via role-play with a spoken dialogue system versus human, and found that spoken dialogue systems are a feasible alternative to human interaction in the role-playing context. @xcite have found that interaction with spoken dialogue systems in immersive contexts improved students proficiency and decreased their anxiousness while using a foreign language thereby indicating there may be increased willingness to communicate with automated humanoid interlocutors. all this points to evidence that spoken dialogue systems are an effective tool for language acquisition. many spoken dialogue systems for the use of language learning have been built using off-the-shelf intent and slot detectors, and dialogue state managers @xcite . @xcite have found some repair and grounding mechanisms in their dialogue system for language learning such as systems being able to respond to learners' questions like "what do you mean" or "what can i say next" in a rule-based system. however, quick scaling up for such systems can only come with datasets. several datasets exist to help build task-oriented dialogues such as schema-guided-dialogue (sgd) @xcite , multiwoz @xcite , dialogue state tracking challenges (dstc) 1-3 @xcite and dstc 4-5 @xcite . besides, there are other frequently used speech-based tod data, including fluent speech commands (fsc) 1 , audio-snips @xcite , carnegie mellon communicator corpus (cmcc) @xcite ) and let's go dataset 2 . however, existing task-oriented dialogue datasets do not reflect the language learning perspective as there are no constraints in their collection process that one interlocutor must be a low-proficiency speaker. moreover, most datasets are also a result of a text-based interaction @xcite @xcite . this also means that the existing datasets will not contain r&g patterns specific for language learners interacting with a task-oriented spoken dialogue systems. therefore, we present a new dataset, namely groundialog, which will be the first dedicated tod dataset specifically tailored for r&g in hps-lps conversations. besides, the dataset can address the need for r&g in spoken form in specific scenarios that do not exist in the text-based exchange. ## data collection set-up our goal was to collect conversations between highproficiency (hps) and low-proficiency speakers (lps). to accomplish this, we use amazon mechanical turk (amturk) to recruit and connect pairs of hps and lps for our study. to identify whether a participant is hps or lps, we provided the participants descriptions of cefr levels @xcite and asked them to self-identify their proficiency level 3 . for the purposes of this study, turkers who identify themselves as beginner, elementary, intermediate, and upper intermediate i.e., a1-b2 levels were regarded as lps; whereas those selecting advanced and proficient i.e., c1-c2 are considered as hps. an assumption of our study is that we draw the line between hps and lps arbitrarily at b2 and trust the turker's self-reported proficiency to be accurate. with this setup, we can end up with nearly equal size of hps and lps, which can ease the turker-pairing process for our data collection. a detailed explanation of the data collection process and conversational task for both hps and lps is shown below. subsequently, we 1 https://fluent.ai/fluent-speech-commands-a-dataset-forspoken-language-understanding-research/ 2 https://dialrc.github.io/letsgodataset/ 3 the complete pre-chat survey form is shown in appendix a will present the general statistics of the collected dialogues and users. the study was approved by the irb of the institute conducting this research. all participants were adults and provided consent before starting data collection. all collected data released with the paper is anonymized to our best abilities. ## groundialog corpus the primary goal of the data collection was to gather free-form conversations with repair and grounding (r&g) patterns, between highproficiency (hps) and low-proficiency (lps) english speakers. for this work, we constrain ourselves to the domain of job interviews, where an hr hiring manager attempts to schedule an upcoming interview with an interviewee candidate and answers any related questions. we leave the conversations in other domains to our future work. to analyse the r&g patterns in the collected data from mturk, we inherit r&g types from previous studies (dobao and @xcite @xcite @xcite . the complete list of r&g types is shown in ## conclusion and future work in this paper, we collected and annotated a new dataset groundialog, which is the first dedicated task-oriented dialogue dataset specifically designed for studying repair and grounding in spoken conversations between high-proficiency and lowproficiency speakers. we described the data collection procedure, annotation schemes, and presented a series analysis over the data. in addition, we demonstrated the potential and utility of groundialog by performing two tasks: r&g detection and dialogue state tracking. the results showed that groundialog can be used to train a conversational agent with the r&g capability. it could be further used to detect communicative gaps, which can be addressed in dialogue design. in future, we plan to extend groundialog to a much larger dataset potentially covering multiple domains other than job interviews. besides, we will use groundialog as a benchmark for a shared task to build task-oriented dialog agent with r&g ability. we will also conduct comprehensive user studies to determine the r&g patterns that are most useful in improving learner's conversational proficiency during language learning. further, we plan to present findings from the speech data so researchers can use speech signals along with text to identify repair and grounding related turns.