H ello T here: A Corpus of Annotated Dialogues and Knowledge Bases of Time-Offset Avatars (2024)

Abstract: A Time-Offset Interaction Application (TOIA) is a software system that allows people to engage in face-to-face dialogue with previously recorded videos of other people. There are two TOIA usage modes: (a) creation mode, where users pre-record video snippets of themselves representing their answers to possible questions someone may ask them, and (b) interaction mode, where other users of the system can choose to interact with created avatars. This paper presents the HelloThere corpus that has been collected from two user studies involving several people who recorded avatars and many more who engaged in dialogues with them. The interactions with avatars are annotated by people asking them questions through three modes (card selection, text search, and voice input) and rating the appropriateness of their answers on a 1 to 5 scale. The corpus, made available to the research community, comprises 26 avatars’ knowledge bases and 317 dialogues between 64 interrogators and the avatars in text format.

## introduction time-offset interaction applications (toias) have evolved as an innovative dialogue system, bridging the interaction between individuals and prerecorded video representations of others, hence enabling users to hold conversations outside realtime constraints @xcite @xcite . we built on an opensource project's application, offering a dual interface targeting two distinct user groups: (a) avatar creators, individuals interested in generating their time-offset personas, and (b) interactors, those who engage with these avatars. however, designing a robust toia is a challenging endeavor. the goal is to mirror humanto-human interactions as authentically as possible. this demands seamless integration from an engineering standpoint, such as flawless video clip transitions and numerous linguistic and dialogueturns complexities that intrigue dialogue system researchers. central to a toia's functionality are the avatar's knowledge bases (kbs), repositories of questions paired with corresponding video responses and their transcriptions. one of the inherent challenges is devising an optimal strategy for populating this kb. should it be intuition-driven, or should it stem from authentic dialogue transcripts? furthermore, what data sets can be useful for training models to retrieve the right answer for an interrogator interacting with the avatars? while we explored such questions in other research @xcite @xcite , here we focus on building on such body of work and present the language resources generated in the process. we explored kbs created in three distinct ways: intuition-guided (brainstormed), led by automatic suggestions (generated by gpt-3), and led by human suggestions. we used gpt-3 because our software and study were designed and set up between 2022 and 2023 before newer versions were available. the hellothere corpus offers a unique resource for dialogue researchers, enabling studies on multi-modal interactions, user engagement patterns, and the effectiveness of timeoffset avatar responses. by providing annotated dialogues across different interaction modes, this corpus supports research into natural language understanding, response retrieval and generation, and user experience in asynchronous communication systems. ## related work we categorize pertinent literature on time-offset interaction applications (toia) into three primary areas: system approaches, data sources, and evaluation methodologies. ## data acquisition and annotation our work resulted in collecting and annotating dialogue data comprising 2.2 million words. this effort was part of a large user study involving 90 individuals, some who built the avatars, and others evaluated their interaction quality, along with testing and evaluating a few software features and related research questions discussed in @xcite . ethical considerations were upheld as our institution's institutional review board approved the experiments, and participants consented to release data transcriptions, annotations, and video recordings for research purposes only. in both parts of the study, participants were university students recruited via an online form that included informed consent and details about the study. in the first part, 26 individuals aged 18-24 participated, with 14 females and various international provenance. they are fluent in english and major in various fields, mostly science. in the second part, 64 people participated. they were mostly between the ages of 18-23, and 35 were female. all are also fluent in english, though 80% consider it their second language. the majority were science majors, and a subset of 16 had participated in the previous part of the study. to clarify how data is collected, we describe the user interface used in the extensive user study that generated the corpora. ## conclusion and further work in this paper, we presented the hellothere corpus, which includes two main categories of datasets: 26 single-turn knowledge bases and multi-turn dialogue corpora featuring annotated chat logs. to ensure consistency, we have standardized our terminology throughout, using "q-a pairs" to refer to question-answer pairs in the knowledge bases and dialogues. all q-a pairs are rated by human interactors and benchmarked for answer retrieval. the hellothere corpus offers a multifaceted resource for the sigdial community. it is beneficial for benchmarking conversational agents, studying user behavior, and conducting multimodal analysis. it allows for focused studies on dialogue complexity, retrieval failures, and localized or general user interests, providing a comprehensive foundation for future research in natural language interactions. the key future directions we plan to work on include: (a) expanding the corpus with more data to support diverse research applications; (b) refining models to enhance answer retrieval efficiency and engagement in multi-turn dialogues; and (c) providing and evaluating model performance under multilingual conditions.