Studying Common Ground Instantiation Using Audio, Video and Brain Behaviours: The B rain KT Corpus (2023)

Abstract: An increasing amount of multimodal recordings has been paving the way for the development of a more automatic way to study language and conversational interactions. However this data largely comprises of audio and video recordings, leaving aside other modalities that might complement this external view of the conversation but might be more difficult to collect in naturalistic setups, such as participants brain activity. In this context, we present BrainKT, a natural conversational corpus with audio, video and neuro-physiological signals, collected with the aim of studying information exchanges and common ground instantiation in conversation in a new, more in-depth way. We recorded conversations from 28 dyads (56 participants) during 30 minutes experiments where subjects were first tasked to collaborate on a joint information game, then freely drifted to the topic of their choice. During each session, audio and video were captured, along with the participants’ neural signal (EEG with Biosemi 64) and their electro-physiological activity (with Empatica-E4). The paper situates this new type of resources in the literature, presents the experimental setup and describes the different kinds of annotations considered for the corpus.

## introduction language processing in a natural context is inherently multimodal, and many studies have been devoted to better understanding how the interactions between the different channels leads to a better understanding between participants of a conversation. interaction theories @xcite postulate that this understanding is based on an operation of information transfer between participants, leading to the establishment of a common ground of knowledge. these processes happen at different levels, and the encoding and transmitting of information can be manifested through various cues for the different sources. these include feedback from gestures, gaze and facial expressions @xcite and are also manifested at the physiological level with variations in respiratory rate, heart rate, skin temperature, etc. @xcite . less perceivable to the other speaker but not less interesting for the understanding of their behavior, the brain activity denote of specific rhythmic activity when alignment between speakers occurs in a conversation, with the 10-12hz (mu) frequency band presenting a specific pattern in the integration of mutual information during an interaction as well as in the coordination between speakers @xcite @xcite . these new perspectives have laid the ground for investigations of natural conversations using neurophysiological elicited; however enterprises into this domain remain few in number, for various reasons. the main constraint indeed remains the technical difficulty to create a corpus of natural conversations condensing all of the aforementioned information sources, as movement inherent to speech might impede on the quality of the measured brain activity. furthermore, though models of how the different sources of information interact during a conversation might exist for subsets of the modalities, there is to date no existing global view detailing how audio, video and neurophysiological features in a conversation interact to build and exchange meaning. furthermore, information can be transmitted and integrated in a local time frame (at the given moment when it appears in conversation) but also with delay, impacting the conversation as a whole. finally, the question of which experimental design to use to capture the progressive building of the common ground in the conversation needs to be resolved, as conversational tasks might be too constrained to correctly explain conversational behavior in the wild, and on the other hand free conversation being too reliant on external existing internalised world representations to accurately model and label the different types of information received. we aim with this paper at addressing some of these questions and presenting a new, original resource for language processing studies. we describe below in greater detail our methodological approach to setting up an adequate experiment for acquiring synchronised multimodal natural conversation recounting of the progressive building of a shared knowledge base, the first steps of preprocessing techniques applied for data cleaning and the results we obtained from the early analyses. the originality of this project lies in two aspects. first, we combined existing conversational tasks to induce a discussion where information transfers could be observed and common ground build on gradually, starting from a very controlled environment and increasingly releasing the constraints on conversational vocabulary and topics. seconds, we recorded various types of data, namely audio, video, physiological and brain activity, all of which are crucial when studying natural conversation. compared with recent research which adopted light eeg headset that traded off commodity for recording quality @xcite , we aimed at developing a protocol for recording every modality with great quality. sections 2 and 3 detail our goals for setting up such an experiment and the context in which it is set. next, we outline in sections 4 and 5 our experimental protocol. section 6 describes the processing steps realised on the data to ensure quality and synchronisation between the different modalities recorded, and section 7 presents the first few analyses we ran on the corpus. ## scientific goals unlike language models that learn from massive amounts of data from data sources of various qualities, simulating good language capabilities but failing at delivering a precise description of human language processing, models aiming to better understand language capabilities usually focus on smaller and well-curated datasets. acquiring data for studying conversation in a natural context remains complex because of the heterogeneous nature of the different sources of information that can be collected and analysed. if audio/video record-ings are quite widespread, this is not the case of neuro-physiological recordings which, when they exist, are in limited quantity. a dataset allowing for the extensive study of conversational markers concurrently using audio, video and neurophysiological modalities does not currently exist. with this work, we aim for two goals: first, developing a protocol for acquiring adequate resources for the neuro-physiological study of conversational behaviours in a natural setting; and secondly, designing new resources for the study of information transfer and common ground instantiation in free conversation. with these research questions, an important feature for designing experimental protocols is balancing the conversation environment. constrained experimental tasks such as the maptask @xcite are indeed great at generating conversational attempts, measuring task successes and failures and linguistic alignment; information transfers are clearly identifiable and conversation evolution can be parameterized. they are however restrictive and not representative of most conversational behaviors, which can cover a wide range of topics and usually rely on knowledge far from the experimental context, conversational schemes and experience specific to a speaker. for these reasons, information transfers are more difficult to study in natural conversation, as they can take a larger range of shapes and intensities. with this in mind, we recorded participants through a several tasks experiment, designed so as to progressively release the constraints on conversational topics and gradually allow for the introduction of new vocabulary, concepts and knowledge to the conversation. each 30 minutes experiment starts with a 15min collaborative video game where one player possesses all information relative to solving the game and must instruct the other player who operates the game. once this controlled task completed the experiment then moves on to the discussion of personal views, with a moral dilemma that participants have to discuss and agree on, before finally moving on to the topics of their choice and a freer conversation. participants familiarity and mutual knowledge progressively increase throughout the course of these experiments (dyads were not acquainted before the experiment), offering a way into the study of their progressive alignment. the combination of these very different tasks also allows for the comparison of communication strategies and efficiencies be-tween very specific contexts and completely free conversation. we collected 28 such interactions (∼14 hours) between french speakers, complete with the recordings of their verbal, behavioral, physiological and neurological activities and later enriched with various annotations and descriptors for the different modalities (transcription and morpho-syntactic labeling, facial landmarks and movement annotations, moments corresponding to information exchanges...). when collecting such corpora, a specific attention must be paid to the technical difficulties that arise, namely the synchronisation between all modalities and how behaviors in one modality might affect the collected quality of another. it is for instance necessary to find a good tradeoff between eeg signal quality, which can be very affected by sources of noise such as gestures and speech, and the degree of freedom given to participants for the experiment be considered naturalistic. the corpus will then be used to study conversational patterns across all collected modalities, as the progressive alignment of speakers in conversation can be observed in their verbal (reuse of lexical terms, prosodic similarity), behavioral, physiological (respiratory, heart rate etc) but also neurological activity @xcite . physiological and neurological correlates for information transfers, speakers alignment, parameters for the success of an interaction will be investigated, both at local and larger scales. despite the focus of the experimental design on generating information transfer between participants, the inclusion of a free conversation task will allow for the wider reusability of the rare corpora for other research questions which might benefit from any kind of multimodal setups. finally, increasing our understanding of human linguistic behaviors might find applications for the improvement of human-machine interfaces. ## dataset analysis a first analysis of the corpus can be done based on experimental videos, transcripts and questionnaire answers (see ## conclusion and perspectives in this paper, we presented a procedure for collecting new types of naturalistic corpora including a larger number of sources of information (audio, video but also physiological and neural signals) and the dataset collected as a result. the perspectives of use of this data are numerous: as a language resource, this dataset can be used in the study of convergence and alignment between participants in a conversation, through its tasks gradually releasing the constraints on conversation. the neurological part of the data can be used to further the research into natural conversation procedures and how to deal with noise and movement when running such experiments. but most interestingly, this new kind of corpora opens the way to the possibility of multimodal models complementing audio-video analysis with neurophysiological cues. future works will focus on enhancing the dataset with additional annotations and a more in-depth analysis of the corpus. the dataset is being made available through the ortolang repository.