EMISSOR : A platform for capturing multimodal interactions as Episodic Memories and Interpretations with Situated Scenario-based Ontological References (2021)

Abstract: We present EMISSOR: a platform to capture multimodal interactions as recordings of episodic experiences with explicit referential interpretations that also yield an episodic Knowledge Graph (eKG). The platform stores streams of multiple modalities as parallel signals. Each signal is segmented and annotated independently with interpretation. Annotations are eventually mapped to explicit identities and relations in the eKG. As we ground signal segments from different modalities to the same instance representations, we also ground different modalities across each other. Unique to our eKG is that it accepts different interpretations across modalities, sources and experiences and supports reasoning over conflicting information and uncertainties that may result from multimodal experiences. EMISSOR can record and annotate experiments in virtual and real-world, combine data, evaluate system behavior and their performance for preset goals but also model the accumulation of knowledge and interpretations in the Knowledge Graph as a result of these episodic experiences.

## introduction multimodal interaction in real-world settings using sensors between humans and agents is a complex process. furthermore, it typically evolves over time and within a shared (physical) space, being bound yet remaining continuously dynamic. the fact that certain contextual factors are not physically present, such as past episodic encounters, background knowledge and intentions, adds to this complexity. agents designed to behave intelligently need to handle this complexity and form teams with people to collaborate and achieve shared goals. within the hybrid intelligence framework, 1 we are specifically interested in such collaborative set-1 www.hybrid-intelligence-centre.nl tings and focus on analysing what causes such systems to succeed or fail. collaboration requires shared grounding and partially shared understanding of situations, communications, and references across modalities. as humans and agents may have different beliefs and perceptions of these situations, we argued in previous work @xcite that agents need a theory-of-mind (tom) @xcite to handle conflicts, miscommunication and errors in referential grounding and interpretations. 2 although there are many initiatives for representing multimodal interactions, referential grounding is hardly handled in its full complexity. most approaches to multimodal interaction data either label media such as video, images, or audio with annotations or simply present situated agent interactions in the form of dialogues or actions without labeling. in these approaches, annotations may be seen as interpretations of direct "behavioral" responses (utterances or actions) to the preceding signals. however, they lack a formalization of these interpretations into an explicit model that supports transparent reasoning. such multimodal data sets can be seen as episodic experiences but not yet as knowledge-aware episodic memories that reflect the cumulative result. the latter requires interpretations of different multimodal signals to be combined in an explicit knowledge structure according to an ontological model that reflects our conceptualization of the world. in addition, this model needs to handle alternative interpretations, uncertainties and conflicts as the interpretations are not always correct or consistent. we propose a generic model that can capture multimodal interactions as recordings of episodic experiences with explicit referential interpretations that also yield an episodic knowl-2 makerobotstalk.nl edge graph (ekg) as a tom . emissor stands for episodic memories and interpretations with situated scenario-based ontological references. the platform stores multiple streams of modalities as parallel signals. each signal can be segmented and annotated independently with interpretation, providing robustness and simplicity. signals can thus represent natural conversations in situated contexts in which (visual) actions and (verbal) utterances can be responses to each other but can also happen independently. emissor can represent any (multimodal) interaction that takes place either in virtual or real-world settings, involving any virtual or real-world agent. annotated signals do not necessarily stand on their own, but can be mapped to explicit identities, relations, and properties in an ekg for capturing time-series of instances of situations. these mappings ground signal segments to formal instance representations and ground different modalities across each other. time-bound experiences are thus captured as episodic experiences, i.e. as an explicit cumulative interpretation of streams of signals. the ekg models knowledge and interpretation shifts over time and supports reasoning over the interpretation. by keeping track of the provenance of signals and interpretations, our model reflects alternative tom interpretations from different sources, modalities and experiences. in the current paper, we set out the basic design and structure of our representation and our motivation. we first discuss in section 2 representations of multimodal interaction proposed in various paradigms such as virtual games, agent interactions, multimodal dialogue systems. in section 3, we describe the desiderata for our proposal for representing interactions, which combines aspects from the different approaches in the related work but adds a kg as the host of such transparent episodic memories for situated experiences. we elaborate on the different data layers and relations in our proposal. we discuss how different types of data sets can be converted, aligned and annotated such that segments get grounded to identities in a knowledge graph using an annotation tool. future research and conclusions are presented in the final section 4. ## related work interaction data takes many forms. not only can it come in different modalities @xcite dio, text, action), but we can also have different types of interactions, e.g., search, questionanswering, command-action sequences, (taskbased) dialogues, navigation tasks, games, graphical interfaces, plain video, and audio recordings. it is impossible to provide a comprehensive overview of representations in each separate modality and interaction type in this paper. for our research on social communicating robots, we are interested in representations of multimodal interactions with referential grounding across modalities and the representations of these modalities as such. therefore, we discuss mainly works on modalities aligned in time series representing interactions. this excludes data in single modalities, such as plain text corpora with dialogues (spoken or text) and image or video data without dialogues. it also excludes static data that does not represent temporal sequences of data. for example, visual data labeled with textual descriptions and textual data augmented with visual scenes do not necessarily represent interactions. in interactions, modalities partially complement each other, such as speech responding to speech or to scenes and actions following speech. such sequences often react to and complement each other and reflect some degree of causality and coherence, but not entirely. augmented modalities, on the other hand, mainly represent paired data where one modality describes or illustrates the other. collections of augmented data, e.g., labeled flickr images, do not exhibit coherence across data points and do not reflect causal interaction. nonetheless, single modality data and non-interactive data can still aid in processing multimodal interaction. models and classifiers trained on such data can automatically annotate scenes in a time series of multimodal interaction data. an interesting research question would be whether static or single modality annotations also model collaborative interactions without considering temporal, causal, and coherence relations across data. a recent survey of interactive dialogue datasets is given in @xcite . @xcite differentiate dialogue systems by the types of interaction (goal(s), non-goal chit-chat, topical); their modalities (written, spoken, video); the participants (human, agent); being constrained, spontaneous, scripted, fictional; being goal-oriented, domainspecific or open. most of the data described is, however, non-situated or not situation-grounded. there is hardly any reference to and interaction with phys-ical contexts. many of these datasets and tasks have been developed and presented in sigdial foot_0 , the acl special interest group, for research on dialogues structures and models. a more applied perspective is taken by the dialogue system technology challenge (dstc 4 ). dstc provides a platform for researchers and industry to develop and evaluate agent-interaction systems. older datasets mostly contain conversational data for chatbots. a more recent challenge, audio visual scene-aware dialog (avsd, @xcite ), contains short video clips with audio, descriptive captions, and a dialogue history. participating systems need to answer a follow-up question or ground an answer to an image from the video or an audio fragment. this challenge represents situated-references and grounding across modalities, but the conversation is very descriptive: the conversations describe situations rather than being embedded in them. below, we briefly describe some well-known datasets and challenges that represent various types of interaction data. parlai foot_2 released more than a hundred conversational datasets covering a wide range of topics, but most are single modality chat @xcite . simmc foot_3 is facebook's sequel to parlai with situated and interactive multi-modal conversation @xcite . it consists of taskoriented dialogues in multimodal contexts represented by collections of images. the data contains referential relations between the dialogues and situations, but the current challenge is restricted to the e-commerce contexts of buying furniture and fashion items. there is no grounding to complex situations but intentions and goals are explicitly represented. facebook-research also launched various other related tasks, among which reccon foot_4 : recognizing emotion cause in conversations @xcite , and minirts: hierarchical decision making by generating and following natural language instructions in real-time strategy game environments 8 @xcite . the former grounds dialogues to emotions and their causes but not to visual or audio data. the latter grounds language to a closed virtual world by references to objects, agents, and actions. the dialogues are limited to commands and instructions to operate the game. google developed schema-guided-dialogue (sgd foot_6 ) for task-oriented conversational agents @xcite . in addition to e-commerce services, the tasks involve intent prediction, slot filling, dialogue state tracking, policy imitation learning, language generation, and user simulation learning. the goals are defined, but there is no multimodal situational grounding. an older comprehensive robot platform is provided by openease 10 , which is a web-based knowledge service providing robot and human activity data constituting episodic memories @xcite . it produces semantically annotated data of manipulation actions, including the agent's environment, the objects it manipulates, the task it performs, and the behavior it generates. ease uses so-called neems (narrative enabled episodic memories) as episodic memories. neems consist of a video recording by the agent of the ongoing activity. these videos are enriched with a story about the actions, motion, their purposes, effects, and the agent's sensor information during the activity.ease is not data-centric but a service platform that uses a knowledge database as a back-end. the database can be explored through prolog queries. the focus of openease is on physical interactions and not on conversations with complex referential relations between expressions and situations. microsoft created a platform for situated intelligence, psi foot_8 @xcite . psi offers multimodal data visualization and annotation tools, as well as processing components for various sensors, processing technologies, and platforms for multimodal interaction. psi models multimodal situations and interactions within and comes close to a comprehensive solution. however, psi is a software integration platform through which developers can share modules using a streaming architecture for signal annotation. interactions are not stored in a shared representation and the platform cannot be used for sharing experimental data independently. action learning from realistic environments and directives (alfred 12 ) is a recent benchmark for mapping natural language instructions and ego-centric vision to sequences of actions for household tasks @xcite . alfred releases challenges and leader boards for trajectory tasks in which a human instructs an agent through natural language to carry out specific household tasks in a virtual world. the task is grounded in situations and combines video, audio, and text for clear goals. there is no natural dialogue that is independent of the task. the previous datasets all involve agents. video recordings of interacting people can be seen as another source of data. meld foot_10 and cosmic foot_11 represent the friends sitcom through videos, timestamped dialogues and emotion annotations @xcite . two seasons from the friends dialogues were also annotated with person references and identities by (choi and chen, 2018) for the semeval2018-task4 on and for q&a on open dialogues (yang and choi, 2019). iemocap foot_12 created detailed multimodal recordings of scripted human-human conversations with annotations of participants, gaze, gestures, etc. @xcite . both datasets do not provide any further situated-references, and there is no specific goal set for the conversation. other similar smaller datasets with audio-visual emotion expression are ravdess (livingstone and russo, 2018), tess @xcite and savee @xcite . ## emissor: design and specification in this section, we describe the different data layers of our model and their interrelations. we summarise the design desiderata for our model as follows: to meet these requirements, we use grasp as a referential framework to connect segments in signals to identities in an ekg. these identities are individual people, objects, and places, or relations and properties of these. in the former case, identities are represented as instances through their uris, possibly with names as labels and instances of a particular type. in the latter case, constellations of instances are interpreted as relations or properties represented by rdf triples. in addition, media streams are stored as separate data files for each modality. separate json-ld files are provided for each modality that 1) divide this modality signal into segments grounded in time and space and 2) annotate these segments as representing identities and relations registered in the ekg. a stream of media interpretations emitting triples over time then results in the cumulative growth of an ekg representing an episodic memory. with the use of json-ld elements, signal metadata can be mapped to our referential framework and included in the ekg. motivating example let us consider a simple example (1). a face detector module detects a human face in a video frame at time t1 which results in a box segment. face recognition cannot recognize this person, so it is identified as a new instance of the type person. next, at t2, another face is detected as a box segment. this face is identified as a known person in the ekg: a uri with the name 'carl'. at t3 a speaker is detected whose voice is mapped to the same identity of "carl". the speaker says "do you know my daughter carla?". a text understanding module processes the text representation of the audio at t3. words such as "i" and "me" are linked to the same uri as the speaker, while "carla" will be mapped to another identity (not yet visually grounded) and related to "carl" as his daughter. through contextual reasoning, the model may ground the first unidentified face perception to this newly created identity of "carla" in hindsight. let us consider an alternative variation on this scenario in which the detected speaker at t3 is mapped to "alice" rather than "carl". alice says, "that is carl and his daughter carla". the deictic references to "carl" and "carla" and the pronominal reference "his" can only be resolved by combining the earlier perceptions with the semantics of this utterance. on the other hand, in both scenarios, "being a daughter" is knowledge that cannot be concluded from visual and audio signals and is solely conveyed by interpreting the semantics of the utterance. this demonstrates that neither audio-visual nor textual segments contain all the information needed to come to the correct interpretation: language utterances due to their referential nature rather than being descriptive, audio-visual segments due to their limitation to signal social, conceptual, and cultural framing. ## conclusions in this paper, we described eight desiderata for the representation of multimodal interactions in collaborative contexts. we argued that existing representations do not satisfy all these desiderata and therefore presented emissor for referential interpretations of multimodal interactions to yield a knowledge graph as an explicit episodic memory of the experiences (ekg). emissor combines light-weight json-ld representations for sequential media with semantic web-based rdf models of interpreted worlds. through this we model cumulative growth of knowledge and information in the ekg as a result of processing multimedia streams over time. emissor is designed to address all eight desiderata. it enables to create and compare recordings, annotations and interpretations of interactions in real-world contexts. this allows researchers to more easily share experiments and compare the interactions across different experiments, regardless of the specifics of agent systems or humans that participate in the experiment. our model, software and converted data sets are available foot_14 according to the apache open source license. our release includes an annotation tool to create scenarios manually or inspect and annotate any recording of data. as next steps, we develop more tooling and automatic linking of multimodal segments to identities. we will also provide more data either by converting existing public data to our framework or by rendering data through our robot platform. so far we focused on grounding segments in temporal containers but not yet in spatial containers. our interactions do not include motion and navigation. in future work, we hope to include spatial grounding and reasoning. finally, we will include an evaluation framework for analysing system performance in relation to 1) qualitative properties of the interaction, 2) goals and intentions following a reinforcement learning approach and 3) by evaluating the resulting ekg.