MTP : A Dataset for Multi-Modal Turning Points in Casual Conversations (2024)

Abstract: Detecting critical moments, such as emotional outbursts or changes in decisions during conversations, is crucial for understanding shifts in human behavior and their consequences. Our work introduces a novel problem setting focusing on these moments as turning points (TPs), accompanied by a meticulously curated, high-consensus, human-annotated multi-modal dataset. We provide precise timestamps, descriptions, and visual-textual evidence high-lighting changes in emotions, behaviors, perspectives, and decisions at these turning points. We also propose a framework, TPMaven, utilizing state-of-the-art vision-language models to construct a narrative from the videos and large language models to classify and detect turning points in our multi-modal dataset. Evaluation results show that TPMaven achieves an F1-score of 0.88 in classification and 0.61 in detection, with additional explanations aligning with human expectations.

## introduction identifying key moments in videos, like highlight detection or moment retrieval, is crucial. this involves pinpointing moments through scene changes or specific descriptions using matching and strategic comparison processes. turning point (tp) classification and detection enhance this by incorporating reasoning to identify significant conversational shifts. the challenge lies in the complex reasoning needed, evident in our data annotation where even human annotators require group discussions. detecting these turning points is vital for post-analysis of conversations, recognizing moments that impact speakers' reactions. understanding these moments enhances future interactions, particularly valuable in new or unfamiliar settings like therapy or negotiation, and offers strategies for successful outcomes. given limitations in existing multi-modal datasets and the novelty of our research, we aim to pioneer the creation of a novel high-quality dataset with turning points. collecting four seasons of the big bang theory tv series, with its eccentric characters likely causing turning points, we focus on 40 episodes from seasons 1 to 4, specifically on conversations. this study makes several contributions: (1) introducing multi-modal turning point classification (mtpc), multi-modal turning point detection (mtpd), and multi-modal turning point reasoning (mtpr) tasks in human casual conversation. (2) curated a human-annotated multimodal turning points (mtp) dataset for casual conversation, enriched with textual and visual cues depicting subjective personal states. (3) proposing a novel framework for mtpc and mtpd, utilizing vision language models (vlms) for narrative construction and large language models (llms) for effective reasoning in turning point detection. (4) the code and data are publicly available. 1 2 related work multi-modal datasets have been developed for understanding human conversations @xcite @xcite @xcite @xcite . each of them having limitations such as missing visual data, or providing just extracted features from it, missing context on shorter sequences, alignment issues and so forth. to address these gaps, we developed a multi-modal conversational dataset from tv series episodes, featuring video content with timestamp annotations, aligned transcripts, and video frames, with annotations for turning points. turning points are a special case of change points @xcite sometimes indicating a trend change direction or substantial change in intent for human data. tps in tp cause: penny suddenly mentions her ex utterance i utterance i+1 utterance i+2 utterance i+3 utterance i+5 utterance i+4 ... ... penny: um, i guess that's about it penny: that is the story of penny leonard: well, it sounds wonderful penny: it was penny: until i fell in love with a jerk penny: god, you know, four years i lived with him. narrative analysis, as described by @xcite @xcite , denote critical moments that shape the plot and segment narratives into thematic units. in psychology and social sciences, tps are moments of significant change in individuals' perceptions, feelings, or life circumstances (florida association for women lawyers, 2003; wieslander and löfgren, 2023). our research adopts the tp definition from @xcite and @xcite , focusing on crucial moments within conversations that significantly impact discourse elements in human-simulated dialogues from a tv series. @xcite introduces emotion-flip reasoning (efr), which is the task of identifying past utterances in a conversation that triggered a speaker's emotional state to change, aiming to explain emotional shifts during dialogue. for clarification regarding the differences, we not only provide information on emotional changes but also on the causes behind those changes. we specifically focus on significant emotional shifts. moreover, we consider changes in decisions, perspectives, and behaviors as they are deemed significant. additionally, we provide visual-textual evidence for these changes. ## problem formulation the context of a casual conversation is denoted as c, comprising m utterance-level videos @xmath0, . . . , u m }. each utterance video u i is associated with a corresponding text transcript and a speaker name {t i , s i }. we consider turning points within the conversation, in accordance with definition 1. definition 1 a turning point in this context is a moment that belongs to an utterance in a conversation, triggered by an identifiable event (that is called the turning point cause). this moment marks the beginning of unexpected or significant changes in the subjective personal states of at least one participant (such as decisions, behaviors, perspectives, and feelings) foot_1 . we have annotated it with a timestamp and a textual explanation of its cause (further elaboration on the definition is in appendix b.1). our proposed problem inputs consist of utterance-level videos with corresponding transcriptions, speaker names, and timestamps bound to the transcript. the problem can be divided into three tasks. the first task, referred to as mtpc (multi-modal turning point classification), involves determining if a conversation includes any turning points (tp). the second task, mtpd (multi-modal turning point detection), focuses on pinpointing the timestamps of these turning points in the conversations. a correct turning point is identified when the predicted timestamp falls within a time window threshold δ t relative to the ground truth. the third task, mtpr (multi-modal turning point reasoning), aims to discern the reasons behind each turning point, presented as a textual description. this task is crucial for formulating potential solutions to address negative turning points and gaining insights into cultural norms. regarding evaluation, the model's timestamp predictions can be assessed qualitatively. however, we believe that the textual causes should be evaluated by human experts. currently, we have not identified a qualitative method for evaluating textual causes, considering it as a potential avenue for future research. total number of conversation videos 340 total duration (h) 13.3 total number of utterance-level videos 12351 total number of words in all transcripts 81909 average length of conversation transcripts 241.5 maximum length of conversation transcripts 460 average length of conversation videos (s) 1.9 maximum length of conversation videos (m) 2.5 total number of tps videos 214 ## tpmaven framework we present tpmaven, a language model prompting framework engineered to identify and ground turning points in casual conversational videos. the framework comprises two key components: 1) a scene describer that captures the visual information and articulates the essence of each utterance; and 2) a robust reasoner that interprets instructions, locating and elucidating turning points. for the first component, we prompt the llava model @xcite as our scene describer to get the relevant visual description of the scenes (frames) in the conversations. for the second, various chat-gpt models are prompted with a system prompt, including the definition of tp and three prompts for turning point identification: a describing instruction, the conversation @xmath1, v 1 , s 1 > , . . . , < t m , v m , s @xmath2, with v being the visual description, an optional tracking prompt to direct chatgpt to track individual in the conversation, and a command prompt. further details on the prompting templates for both components can be found in appendix c. ## experiments we use llava-7b @xcite to extract visual information in scene descriptions. gpt-3.5-1106 (a version of gpt-3.5 (openai, 2022)) and gpt-4-1106 identify turning points, addressing context length issues. for assessing turning point localization, we focus on the positive set with 214 conversations. true positives are determined when predicted timestamps fall within δ @xmath3. during segmentation, we map gpt model outputs (utterance indices) back to timestamps for comparison (see more details in appendix d). the performance metrics, including precision, recall, f1 and area under the curve (auc) are reported for each method in ## conclusion in conclusion, our research addresses the crucial task of recognizing pivotal moments in conversa-tions, presenting a detailed taxonomy and a curated dataset called mtp. our baseline framework, tp-maven, utilizes vision-language and gpt models for classification and detection, demonstrating its performance across various metrics. while tp-maven provides explainable predictions for sensitive moments, experimental results highlight the need to discern conversations with and without turning points. future directions are in appendix e.