More Than Spoken Words: Nonverbal Message Extraction and Generation (2023)

Abstract: Nonverbal messages (NM) such as speakers’ facial expressions and speed of speech are essential for face-to-face communication, and they can be regarded as implicit knowledge as they are usually not included in existing dialogue understanding or generation tasks. This paper introduces the task of extracting NMs in written text and generating NMs for spoken text. Previous studies merely focus on extracting NMs from relatively small-scale well-structured corpora such as movie scripts wherein NMs are enclosed in parentheses by scriptwriters, which greatly decreases the difficulty of extraction. To enable extracting NMs from unstructured corpora, we annotate the first NM extraction dataset for Chinese based on novels and develop three baselines to extract single-span or multi-span NM of a target utterance from its surrounding context. Furthermore, we use the extractors to extract 749K (context, utterance, NM) triples from Chinese novels and investigate whether we can use them to improve NM generation via semi-supervised learning. Experimental results demonstrate that the automatically extracted triples can serve as high-quality augmentation data of clean triples extracted from scripts to generate more relevant, fluent, valid, and factually consistent NMs than the purely supervised generator, and the resulting generator can in turn help Chinese dialogue understanding tasks such as dialogue machine reading comprehension and emotion classification by simply adding the predicted “unspoken” NM to each utterance or narrative in inputs.

## introduction nonverbal messages (nm), such as facial expressions, body movements, and tones of voice, can complement or modify verbal messages as well as improve the teamwork efficiency @xcite and effectiveness of face-to-face communication @xcite . these messages are usually not explicitly mentioned in the transcribed verbal messages as the dialogue participants share most of the nms via other modalities, and nms are also seldom included in existing text-based dialogue tasks that mainly focus on verbal messages @xcite . though human readers can infer missing nms based on their own knowledge, machines still have difficulty understanding the meanings behind and beyond the words @xcite and automatically decide what nonverbal behaviors they should display in interactions @xcite . we focus on text-based nm extraction and generation, an important step towards reaching the ultimate goal of bridging the human-machine implicit knowledge gap. one of the most relevant text resources for nonverbal messages is tv and movie scripts. generally, scripts are written in a standard format: for example, nms of their corresponding utterances are enclosed in parentheses (e.g., elizabeth (ironically) "with five thousand a year, it would not matter if he had a big pink face." and mr darcy (shakes his head) "you know how i detest it."), which usually describe what can be seen or heard by the audience beyond the verbal messages. based on the well-defined screenplay structures, it is relatively easy to use heuristics to extract utterances and their nms from scripts @xcite . however, in scripts, only a small percentage (≈ 10.5% (section 6.2)) of utterances are followed by nms, and existing public script corpora are usually small-scale even for resource-rich english (e.g., 1,276 movies @xcite and 917 movies @xcite ). in contrast, novels also contain rich nms via the words of the writers alongside what their characters speak, and thousands of novels have already been adapted into scripts (mainly by professional scriptwriters). besides, we observe that the density of nms in novels is higher than that of scripts (≈ 67.4% based on the annotated corpus (sec-tion 3.2)), indicating the potential of leveraging novels for nm extraction. therefore, we are interested in whether we can use this unstructured resource to alleviate the nm data scarcity problem, which hinders the full utilization of deep neural models. as this direction is still unexplored, we first define the task as extracting one or multiple spans from the surrounding context of the target utterance and annotate nme, the first nonverbal message extraction dataset based on three chinese novels @xcite containing 4k (context, utterance, nm) triples. furthermore, we design three baselines (pattern, extractive, and generative) to extract nms and evaluate them on nme. another question is whether we can leverage unlabeled novel corpora to automatically construct data for improving nm generation. to investigate this question, we first use the trained extractors to extract 749k pseudo-labeled triples from several hundreds of chinese novels and train generators based on different backbone models to generate a nonverbal message given one target utterance and its context. experiments show that these triples can serve as high-quality augmentation data of clean triples extracted from well-structured scripts to generate more relevant, fluent, valid, and factually consistent nms. furthermore, our semi-supervised generators can in turn help chinese dialogue and narrative understanding tasks that lack nms such as the dialogue subset of a machine reading comprehension dataset c 3 @xcite and emotion classification ewect by simply adding the generated "unspoken" nms to each utterance or narrative in inputs, showing their usefulness. the contributions of this paper are as follows. ## nm extraction data construction as there is no public dataset in any language for identifying nms in unstructured corpora, we first introduce our nm annotation guidelines and analysis of our annotated dataset nme. ## nonverbal message extraction this section introduces three nm extraction baselines (pattern-based, extractive, and generative) and how to use the extractors and unstructured corpora to construct large-scale pseudo-labeled nm data. ## experiment 6.1 implementations for the extractive baselines, we consider both encoder-only (roberta-wwm-ext-large and macbert, both released by @xcite ) and encoder-decoder models (t5 base @xcite , bart large @xcite , and dialbart large ). note that dialbart large (hit-tmg, 2022) is obtained by fine-tuning bart large @xcite on luge dialogue (section 6.2). the above encoder-decoder models are also used for generative nm extractors and nm generators. we discard the tokens from the bottom of the input if its length exceeds the maximum model sequence length. we conduct experiments on eight nvidia-v100 32gb gpus. appendix a.8 introduces the models' details ( commonsense knowledge (csk): a contextutterance-nm triple can somehow be regarded as a piece of commonsense knowledge, as they both are usually ignored and assumed to be known without being told. thus, we use the human-annotated argument pairs such as ("someone holds an umbrella", "it is raining") in the chinese set of commonsense knowledge conceptnet @xcite by regarding the two arguments as the input and the nm, respectively. the context is left empty. luge dialogue : it contains four chinese dialogue datasets: chinese persona chat (cpc) foot_4 , lccc @xcite , emotional stc (estc) @xcite , and kdconv @xcite . we indirectly use luge dialogue by using dialbart large that is fine-tuned on this dataset as the backbone model to study the usefulness of dialogue generation datasets for nm generation. c 3 d and ewect foot_5 : for dialogue/narrative understanding tasks, we consider a multiple-choice mrc dataset c 3 d and an emotion classification dataset ewect (the general-domain version) to investigate the impact of introducing generated nms into dialogue tasks without nms. we provide the detailed data statistics in