A Cloud-based User-Centered Time-Offset Interaction Application (2021)

Abstract: Time-offset interaction applications (TOIA) allow simulating conversations with people who have previously recorded relevant video utterances, which are played in response to their interacting user. TOIAs have great potential for preserving cross-generational and cross-cultural histories, online teaching, simulated interviews, etc. Current TOIAs exist in niche contexts involving high production costs. Democratizing TOIA presents different challenges when creating appropriate pre-recordings, designing different user stories, and creating simple online interfaces for experimentation. We open-source TOIA 2.0, a user-centered time-offset interaction application, and make it available for everyone who wants to interact with people’s pre-recordings, or create their pre-recordings.

## introduction stories are the most pervasive medium people use to make sense of themselves and the surrounding world. in the past twenty years, social media development has enabled people to document narratives of their daily experiences online in unprecedented measure @xcite . toias explore the next evolution of narrative sharing devices exploiting advances in artificial intelligence, as well as peoples' increasing comfort with virtual communication methods, to simulate face-to-face interactions between humans and previously-recorded videos of other humans. democratizing time-offset interaction has many challenges, including but not limited to: creating appropriate videos, robust human-computer interaction (hci) design, and lack of intuitive online interfaces. we open-source toia 2.0, a user-centered toia, and make it available for everyone who wants to interact with people's pre-recordings, create their own, and researchers in dialogue, machine learning, and hci to gather original datasets. ## related work the usefulness of toias has been demonstrated in many practical scenarios. for example, for keeping historical memories @xcite , job interview practice for young adults with developmental disabilities, foot_0 and building digital humans across different industries. storyfile, typeform's videoask are some examples of commercial applications. 2 toias may also be reminiscent of virtual assistants like siri and alexa and digitally animated characters like digital humans, foot_2 and soul machines; foot_3 however, these are not authentic representations of human beings which is toias' goal. the general public cannot afford current toia deployments due to their high production costs: creating a character (aka avatar) may require prerecording about 2,000 video answers @xcite . @xcite proposed a more streamlined avatar development process, but it is still impractical for the everyday user: it involves transcribing and recording conversations based on brainstormed plausible utterances. their work resulted in creating more than 400 prerecordings and manual annotations that took several days. research into time-offset interactions needs to generalize and to streamline the avatar development process to make a mass use system. a first attempt made by @xcite goes towards this direction and includes the possibility to chat with the avatars in different languages. their system implementation is not simple to use because it has two separate, non-communicating components for recording videos and interacting with them. it also requires local installation and does not support multiple users. ## design principles we designed toia 2.0 so that the time-offset interaction feels like a natural interaction to all users: the user who interacts with other people's prerecordings (henceforth, interactor), and the user who records their own narratives (henceforth toia maker). as per best practices in ux design, each step of interaction with the interface should minimize the user's cognitive load and be psychologically satisfying. we followed the human-computer interface heuristics established by @xcite . some design decisions to reach this goal included: the creation of a common visual framework across the whole system for all types of users; use of familiar layouts and vocabulary; creation of affordances by auto-generating suggested questions to answer and to suggest particular kinds of interactions; providing psychological satisfaction with profiles that emphasize social interaction; giving users agency and autonomy over aspects of their toia experience; and providing reassurance with repeated language and layouts. an important extension to the previous work by abu @xcite is the introduction of a social network aspect. this does not just serve the goal of creating a community of toia makers, but also provides helpful feedback to them including what additional questions are asked, what answers are liked or disliked. another extension is giving the toia maker control over which recordings are playable individually, and as part of streams (i.e. collections or albums) that define different contexts and intentions for the interactions. ## toia 2.0 system architecture the cloud-based centralized toia 2.0 system consists of six components, broadly speaking, that are interconnected via a web server and component apis (figure 1 ). in the rest of this section, we discuss the back-end components of toia 2.0. we then proceed to discussing the user interface which is the highlight of this demo paper in section 5. ## conclusion and future work we presented toia 2.0, a user-centered time-offset interaction application system, which we plan to make publicly available to users -toia makers and interactors -as well as researchers interested in toia systems. in the future, we plan to increase the robustness of the system and its membership. we are also considering additional enhancements to both the toia maker and interactor's experience, e.g., providing semi-automatically generated videos or guessing answers on behalf of a toia maker given their history. in all cases, we keep the toia maker in complete control of their recordings and streams.