Werewolf Among Us: Multimodal Resources for Modeling Persuasion Behaviors in Social Deduction Games (2023)

Abstract: Persuasion modeling is a key building block for conversational agents. Existing works in this direction are limited to analyzing textual dialogue corpus. We argue that visual signals also play an important role in understanding human persuasive behaviors. In this paper, we introduce the first multimodal dataset for modeling persuasion behaviors. Our dataset includes 199 dialogue transcriptions and videos captured in a multi-player social deduction game setting, 26,647 utterance level annotations of persuasion strategy, and game level annotations of deduction game outcomes. We provide extensive experiments to show how dialogue context and visual signals benefit persuasion strategy prediction. We also explore the generalization ability of language models for persuasion modeling and the role of persuasion strategies in predicting social deduction game outcomes. Our dataset can be found at https://persuasion-deductiongame. socialai-data.org. The codes and models are available at https://github.com/SALT-NLP/PersuationGames .

## introduction as humans, from childhood, we develop the ability to attribute mental belief states to ourselves and others @xcite . moreover, we constantly exhibit persuasive behaviors to influence and even reshape the belief states of others during our daily social interactions @xcite ). an automatic system with the ability to understand human persuasion strategies and deduce human belief states may enable more proactive humancomputer interaction, and facilitate collaborative decision-making processes. * denotes equal contribution. prior works targeted at understanding the persuasion strategies utilized on online forums like reddit, crowd-funding platforms @xcite @xcite , and in 1-on-1 dialogues under simulated scenarios through the amazon mechanical turk platform @xcite . however, the persuasive behaviors during naturalistic group discussions with face-to-face conversation remain unexplored. more importantly, daily human social interaction is multimodal by nature. both verbal communication (e.g. language and audio) and non-verbal communication (e.g. gesture and gaze behavior) are essential for analyzing persuasive behavior. moreover, resources for understanding how persuasion strategies affect decision and deduction outcomes during social interactions are missing from the language technologies community. to bridge these gaps, we introduce the first multimodal benchmark dataset for modeling persuasive behaviors during multi-player social deduction games. as shown in fig. 1 , our dataset is captured in a naturalistic setting where groups of participants play social deduction games 1 . our dataset contains both video recordings and the corresponding dialogue transcriptions. the video data is sourced from both the ego4d social dataset @xcite and youtube videos. our dataset also has annotations for persuasion strategy at the utterance level and the voting outcome of each participant during the social deduction game. we benchmark our dataset by providing comprehensive experimental results and analyzing the role figure 1 : demonstration of the six persuasion strategies included in our dataset and the corresponding video. the players are numbered as 1,2,3,4 from left to right. players' roles might be changed during the game. in this example, player1's and player2's cards were swapped by the troublemaker, and player3's and player4's cards were swapped by the robber. player 2 voted for player 3 at the end while the others voted for player 2. ## related work persuasive behaviors understanding. a few previous works introduce datasets for the computational modeling of persuasion @xcite @xcite @xcite . ## strategy prediction given an utterance and its corresponding video segment, we seek to predict the persuasion strategies adopted in the utterance. we first leverage a pretrained language model @xcite as the text encoder to obtain the utterance embedding, and a vision transformer @xcite to obtain the visual embedding. we then concatenate the textual and visual features to predict the persuasion strategy. additionally, we study the impact of textual context by including prior utterances as input. figure 2: architecture of the independent model for each strategy. we fix the parameters in the video encoder and train the other modules end-to-end. ⊕ denotes the concatenation of two feature representations ## game outcome deduction in addition to predicting persuasion strategies, we further model the human deduction process by predicting the voting outcomes of each pair of players, i.e., whether player a (voter) votes for player b (candidate). therefore, in a game of n players, there are c 2 n (combinations) of player pairs, corresponding to p 2 n (permutations) data points. we merge all data points from the ego4d werewolf data and youtube data to enlarge the dataset size, and split the resulting data into 2741/427/827 samples for train/val/test sets. since each player is only allowed to vote for one player, the resulting data has an imbalanced distribution, with 20.4% of the samples being positive (positive indicates the voter votes for the candidate). ## conclusion and future work in this work, we introduce the first persuasion modeling dataset with multiple modalities and rich utterance-level persuasion strategy annotations. we design a computational model that leverages both textual and visual representations for understanding persuasion behaviors in social deduction games. our experiments show that visual cues benefit model performance on persuasion strategy prediction. we encourage future work to explore the role of the audio modality in persuasion modeling and to investigate joint learning of multimodal representations for the social persuasion setting.