Incremental Unit Networks for Multimodal, Fine-grained Information State Representation (2021)

Abstract: We offer a fine-grained information state annotation scheme that follows directly from the Incremental Unit abstract model of dialogue processing when used within a multimodal, co-located, interactive setting. We explain the Incremental Unit model and give an example application using the Localized Narratives dataset, then offer avenues for future research.

## introduction human experience is profoundly multimodal. as people explore the world they are organizing perception, action, and thought in a complex social environment @xcite . tied directly to this multimodal experience is human language, primarily spoken language @xcite , and a growing body of literature across several disciplines make a strong case that language learning and language meaning is grounded in rich multimodal (even embodied), interactive, and enactive experience @xcite @xcite @xcite . despite this, current state-ofthe-art language models such as bert @xcite are trained only using static text, and while it is clear that such models are powerful and useful for many tasks, they are clearly missing important multimodal semantic knowledge @xcite ). 1 we argue that what is needed is a semantic model that is learned not only from text, but has knowledge of multiple modalities and that the model operates in a setting similar to how language is acquired for humans: multimodal, co-located, interactive spoken dialogue: multimodality: a model of semantic meaning of language must ground into not just vision, but other modalities such as taste, touch, smell, proprioperception, and even affect. this is as much a modeling challenge as an engineering challenge, because each modality requires sensor hardware (e.g., cameras for vision) and methods for fusing the sensor information from different modalities. co-location: multimodal systems have multiple sensors that sense things like objects, events, and the interlocutor who has knowledge about the environment, language used to denote objects, and uses cues such as gaze and gestures in communication. spoken interaction: semantic meaning is learned and used in coordination with members of a particular language community (clark, 1996) and spoken interaction is the setting where children learn language. moreover, spoken language differs dramatically from written text in that spoken language contains communicative artifacts such as hesitations, false starts, repetitions, repairs, and coordination of turn-taking. furthermore, people produce and understand language sequentially, not as complete and fully grammatical units @xcite . taken together, these requirements imply technical and modeling challenges. technical challenges include using multiple sensors and articulators, fusing their information streams, temporally aligning input and output. modeling challenges include binding information from the sensors, learning meaningful patterns in a noisy setting, and representing the states of the sensors and unfolding interaction. in this paper, we don't formulate a semantic model, but focus rather on a representation with a fine-grained information state update approach using the incremental unit abstract model of spoken dialogue. we explain the incremental unit model in the next section, including how multi-modal how information is represented, then offer a simple scheme for using incremental units as a basis for developing multimodal semantic models. ## the incremental unit framework the incremental unit (iu) framework @xcite is an abstract, conceptual approach for incremental processing for spoken dialogue. the iu framework consists of a network of processing modules, each of which play a different role in an unfolding dialogue, all of which work together to create the fine-grained information state. modules take input data on their left buffers, process the input, then produce output on their right buffers. a critical part of the iu framework is how the data are packaged and processed. the data are packaged as the payload of incremental units (ius) which are passed between modules-each iu holds a discrete amount of information. another critical part of the framework is that the ius themselves are interconnected via same level links (sll)-allowing the linking of ius as a growing sequence-and grounded-in links (grin) which allow that sequence to convey what ius directly affect another iu. ideally, ius (e.g., produced from a sensor or processing module) can be guaranteed to be correct, but often an iu that has been outputted to the next module needs to be updated in light of new information. to make this possible, the framework makes use of three operations: ius can be added to the iu network, but can be later revoked, and also committed when a module can guarantee that an added iu will not be revoked. figure 1 shows an example of how a speech recognition module would process incrementally, typically word-by-word. it takes a continuous audio signal as input from a microphone and produces discrete word ius as output. as the utterance i will leave now is uttered, the speech recognizer outputs words as they are recognized at the word level and adds them to the iu network. the recognizer mis-recognized the word live, but in light of new information from the unfolding utterance, revoked live and replaced it with leave. horizontal arrows show slls; i.e., how the ius are related to each other temporally, and at the end of the utterance when the recognizer knows it will no longer revoke, it marks all of the ius as committed. iualso contain information about their creation time. it's important to distinguish at this point the networked iu modules or processors that pass ius to each other and the network of ius themselves. for example, a speech recognizer might pass its transcribed speech as ius with payloads of word strings to a part-of-speech module that produces a part-ofspeech for each word as payloads of part-of-speech strings, which are then the input of a language understanding component that operates on both the words and parts-of-speech to produce some kind of semantic abstraction of the unfolding utterance. thus the three processors-speech recognizer, partof-speech tagger, and language understanding-are separate modules, but each use the add, revoke, commit operations to alter the shared network of ius. the iu framework, including the operations, can be used as a fine-grained model of the dynamics of the creation of the information state of an agent in a situated interaction, comprising both its world model and its discourse model, and the interaction between them. ## the iu framework for fine-grained information state representation in this section, we sketch a scheme for the iu network as a representation of a fine-grained information state. the scheme follows the iu approach to processing live speech; all annotations are packaged as ius with links between them, all add operations are accounted for (and revoke operations under live annotation conditions), each operation is timestamped, and the creation time of each iu is timestamped. we don't specify how the modalities or modules interact with each other, the goal here is to focus on the information state. we give an example in figure 3 using a sample from the localized narratives dataset @xcite . the dataset consists of images described by annotators. descriptions have speech and mouse pointer modalities that are later temporally aligned. speech is automatically transcribed as the annotators speak, but annotators are tasked with hand-transcribing their descriptions after they are complete. the dataset on its own has multimodal annotations, though it's unclear how they would work in a live interaction with a system. the iu network annotation in figure 3 shows locations of mouse pointer (x,y coordinates), words, and added part-of-speech tags and semantic abstraction similar to that in figure 2 . the sll and grin links are also present, and additional links between the speech and pointer modalities are depicted. what is not depicted in the figure are the add and revoke operations that enable the network to grow as an interaction unfolds in real time, though it is obvious that all ius in the figure were created through an add operation. in the case where a perfect transcription exists, only add operations are necessary, but a live interaction would require the ability to revoke erroneous words then add correct ones in real time, in alignment with the movements of the mouse pointer. timestamp information is not present in the figure; time generally flows downward as ius are added to the network. the scheme can be applied during the data collection process. this requires some up-front effort to setup each individual module to operate incrementally. for the localized narratives dataset, incremental text can come from asr or typed text, and the other annotations from respective modules. annotated data can be represented in any format, e.g., json. this scheme highlights the importance of annotating data that is representing a fine-grained information state collected in a multimodal, co-located, and spoken interactive task. such a representation is potentially useful for a formal representation of situated conversation and embodiment. ## conclusion and future work in this paper, we outlined an iu network-based approach to representing multimodal states within the requirements of multimodality, co-location, and interactive speech. implicit in this representation is the requirement that the system is modular, though it is potentially possible to represent the iu network in an end-to-end neural architecture. the modalities explored here were only a minimal example of what the network could potentially handleadded modalities enrich the semantic representation. for example, we have used the iu framework to represent audio, visual, and internal robot state modalities in prior work @xcite . we leave formalizing semantic operations, such as compositionality, meaning derived from handling uncertainty or requests for clarification, and global decoding strategies in the iu network semantic representation for future work.