D ecep B ench: Benchmarking Multimodal Deception Detection (2025)

Abstract: Deception detection is crucial in domains such as security, forensics, and legal proceedings, as well as to ensure the reliability of AI systems. However, current approaches are limited by the lack of generalizable and interpretable benchmarks built on large and diverse datasets. To address this gap, we introduce DecepBench, a comprehensive and robust benchmark for multimodal deception detection. DecepBench includes an enhanced version of the DOLOS dataset, the largest game-show deception dataset (1,700 labeled video clips with audio). We augment each video clip with transcripts, introducing a third modality (text) and incorporating deception-related features identified in psychological research. We employ explainable methods to evaluate the relevance of key deception cues, providing insights into model limitations and guiding future improvements. Our enhancements to DOLOS, combined with these interpretable analyses, yield improved performance and a deeper understanding of multimodal deception detection.

## introduction generalizable deception detection systems, which emerge in critical areas of psychology, computational linguistics, and criminology, remain a significant challenge due to the lack of standardized benchmarks for evaluating performance across diverse datasets. for example, @xcite demonstrated that while syntactic and lexical characteristics can effectively detect deception in specific domains, such as fake online reviews, these features often fail to generalize in different contexts, highlighting the need for universal evaluation frameworks. existing datasets, such as dolos, provide resources for studying deceptive behavior. however, they often vary in terms of context, modality, and annotation quality, making it difficult * equal contribution ‡ corresponding author to compare results or assess the generalizability of detection models. this inconsistency has led to a fragmented understanding of deceptive behavior, with many studies relying on small or limited datasets that do not capture the complexity of realworld deception @xcite . to address this gap in the generalizability of deception detection models, we propose the creation of a novel and comprehensive deception detection benchmark tailored for the dolos dataset. a central research question that this benchmark will address is: to what extent can models trained on specific deceptive contexts generalize to new, unseen contexts, and how can we improve this generalizability? this benchmark, decepbench, aims to establish a unified framework for evaluating deception detection algorithms, allowing researchers to systematically assess model performance, identify strengths and weaknesses, and foster advancements in the field. while most benchmarks (e.g., fakeddit @xcite , spotfake @xcite ) focus on black-box multimodal fusion, we prioritize interpretable features validated by professionals (e.g., forensic linguists, psychologists). this ensures that the features align with real-world deceptive behaviors, as supported by psychological research @xcite and interdisciplinary studies that emphasize the importance of expert validation in deception detection @xcite @xcite @xcite . by incorporating diverse datasets, multimodal features, and standardized evaluation metrics, decepbench will provide a rigorous and reproducible foundation for future research. in summary, our contributions are as follows. deception detection is a diverse field that intersects psychology, linguistics, and artificial intelligence. early works in the field relied on textbased datasets such as liar @xcite and fakenewsnet @xcite , which focused on linguistic cues to identify deception in news articles and social media posts. however, these datasets were limited in capturing the multimodal nature of deception, such as tone, facial expressions, and physiological responses. ## results on the dolos dataset, the imagebind @xcite model achieved 85.3% accuracy and an f1-score of 0.83, outperforming prior baselines. the imagebind model is a multimodal model developed by meta ai, which is capable of learning a joint embedding space across multiple modalities such as text, audio, and visual data. the auc-roc was 0.91, demonstrating robust discriminative power in classifying truthful and deceptive clips. shap analysis highlighted the two most important features, which are microexpressions (e.g., fleeting eyebrow raises, contributing 35%) and pitch variation (e.g., deviations in vocal frequency, contributing to 28%). the model accurately detected deception in high-stakes scenarios, such as courtroom testimonies, where rapid gaze shifts and vocal hesitations aligned with untruthful labels. common failure patterns include false positives: sarcastic remarks and stress responses were misclassified as deceptive. false negatives include natural liars and suppressed microexpressions that evaded detection (they were misclassified as true). these results indicate that combined verbal, nonverbal, and vocal cues significantly improve deception detection. compared to unimodal baselines, includ-ing multimodal features helped yield performance gains, as shown in furthermore, we implemented domain-specific prepossessing and targeted fine-tuning to ensure robustness across datasets like bag of lies (80.4%) and real-life legal deception (78.1%). domainspecific preprocessing includes a bert-based token classifier to identify repetitive phrases, as well as nlp used to tag interviewer prompts and align them with candidate responses and flagging mismatched timelines. we re-trained imagebind's text encoder on interview transcripts to help prioritize lexical patterns over vocal and visual cues, which improved accuracy by 9%. when tested in real-life legal deception, the interview-adapted model retained moderate performance by leveraging shared verbal cues but struggled with highstakes micro-expressions. regarding model size concerns, we performed parameter-matched experiments. a 300m-parameter unimodal bert baseline achieved 68.2% accuracy. our trimmed 300mparameter imagebind (multimodal) reached 79.4%. this 11.2% gap persists even with equalized parameters, demonstrating that multimodal integration, not just capacity, drives improvements. ## limitations deception detection is held back by the limitation of diverse, robust, generalizable data, making it challenging to develop models that can perform well across domains. dolos is among the most comprehensive datasets available, yet it still lacks the complexity and diversity of real-world contexts. additionally, identifying the most relevant deception cues remains difficult, not only for models but for humans as well. deception is context-dependent and is not reliably or consistently shown through any indicators. cues, including facial expressions, speech patterns, and body language, can vary significantly depending on the individual. features extracted via openface and liwc were validated on dolos but may not generalize to domains with differing cultural norms (e.g., gaze aversion in some cultures signals respect, not deception). future work should calibrate thresholds per domain. also, while our system outperformed gpt-4/gemini overall, the llms achieved higher accuracy (78 vs our 73) on the textonly deceptive opinion spam subset, suggesting their pretraining gives them an advantage in narrow textual domains. despite limitations, our work still contributes to advancements in this field and furthers the development of accurate classification in deception detection.