UNIDECOR : A Unified Deception Corpus for Cross-Corpus Deception Detection (2023)

Abstract: Verbal deception has been studied in psychology, forensics, and computational linguistics for a variety of reasons, like understanding behaviour patterns, identifying false testimonies, and detecting deception in online communication. Varying motivations across research fields lead to differences in the domain choices to study and in the conceptualization of deception, making it hard to compare models and build robust deception detection systems for a given language. With this paper, we improve this situation by surveying available English deception datasets which include domains like social media reviews, court testimonials, opinion statements on specific topics, and deceptive dialogues from online strategy games. We consolidate these datasets into a single unified corpus. Based on this resource, we conduct a correlation analysis of linguistic cues of deception across datasets to understand the differences and perform cross-corpus modeling experiments which show that a cross-domain generalization is challenging to achieve. The unified deception corpus (UNIDECOR) can be obtained from https://www.ims.uni-stuttgart.de/data/unidecor .

## introduction deception detection has remained an area of vested interest in fields like psychology, forensics, law, and computational linguistics for a myriad of reasons like understanding behavioral patterns of lying @xcite , identifying fabricated information @xcite , distinguishing false statements or testimonies @xcite and detecting deception in online communication @xcite . these are relevant tasks because of the truth bias, which is the inherent inclination of humans to actively believe or passively presume that a statement made by another person is true and accurate by default, without the need for evidence to substantiate this belief @xcite . while this facilitates efficient communication, it also makes people susceptible to deception, especially in online media where digital deception @xcite manifests in many forms like fake news, misleading advertisements, impersonation and scams. this warrants automatic deception detection systems that can accurately distinguish between truthful and deceptive discourse solely from textual data. the task of automatic deception detection comes with several challenges. deception or lying is a complex human behavior and its signals are faint in text. moreover, it is sensitive to the communication context, interlocutors, and the stake involved @xcite . most importantly, acquiring annotated data proves to be one of the major hurdles for deception studies. traditional data annotation methods cannot be employed because human performance is shown to be worse than machines in differentiating truths and lies @xcite . one way to collect accurate data is to get the labels at source by the person producing the text. alternatively, they can be collected using the acquired knowledge that certain types of contents are deceptive. across the literature, different strategies like crawling fake reviews @xcite , collecting text from users identified as suspicious @xcite , using non-linguistic deception cues @xcite and soliciting through crowd-sourcing @xcite @xcite have been employed to obtain reliable annotations. the diversity in the domains of interest, the medium of deceptive communication (spoken vs. written) and dataset creation procedures make it difficult to compare cues of deception across datasets and to understand their generalizability across different domains. with this study, we aim at mitigating this situation by conducting a comparative survey of publicly available textual deception datasets. we contribute (1) a consolidated corpus in a unified format and (2) conduct experiments in which we evaluate models trained on one data set on all others. our (3) results show that cross-corpus, particularly cross-domain, generalizability is limited, which motivates future work to develop robust deception detectors. we lay the foundation for such work with (4) additional correlation analyses of the linguistic cues of deception across datasets and verify their generalizability across domains. ## background & related work deception in communication is the act of intentionally causing another person to have a false belief that the deceiver knows or believes to be false @xcite @xcite . lies, exaggerations, omissions, and distortions are all different forms of deception @xcite . while the definition of deception varies across literature, they concur that it is intentional or deliberate @xcite . ## unified deception dataset as preparation for cross-corpus analysis of the concept of deception, we consolidate publicly available textual deception datasets into a unified format. 3 we now describe the included datasets. deceptive opinion spam (opspam). @xcite describes deceptive opinion spam as fraudulent reviews written to sound authentic with the goal to deceive the reader. to study the nature of such reviews, they collected truthful reviews by crawling online review platforms like tripadvisor foot_2 and crowd-sourced deceptive reviews via amazon's mechanical turk (amt). the initial opspam dataset published by @xcite contains 400 truthful and 400 deceptive reviews with positive sentiments. @xcite extended the dataset to include reviews with negative sentiments. the complete opspam dataset contains 1600 instances labeled for veracity and sentiment. it is available publicly with a creative commons attribution-noncommercial-sharealike license. 5 cross-cultural deception (crosscultde). @xcite collected the crosscultde dataset to investigate deception in a cross-cultural setting. it consists of short essays on the topics of abortion, death penalty, and feelings about a best friend, collected from the united states, india, and mexico. we take into account the data collected from the united states and india which are in english and consist of 100 deceptive and 100 truthful essays per topic per geographical region adding up to 1200 labeled instances. the dataset is available for download without mentioning any usage restrictions. 6 deception in reviews (derev2014/2018). to investigate the phenomenon of sock puppetry, fornaciari and poesio (2014) collected derev2014, containing book reviews from amazon.com that were identified as authentic or fake using predefined linguistic cues. to overcome the shortcoming that these cues cannot be used while developing a deception classifier, @xcite released the derev2018 dataset, in which they collect deceptive reviews based on a priori knowledge about authors who solicited fake reviews. additionally, the authors crowd-sourced both truthful and deceptive reviews for the same books. the derev2014 dataset contains 118 reviews each with a truthful label and a deceptive label, while the derev2018 dataset includes 1552 reviews each collected from amazon.com and through crowdsourcing with a balanced distribution of truthful and deceptive reviews. the datasets overlap by 62 reviews. both corpora are available for download. 7 open domain deception (opendomain). @xcite dataset acquired via amt. workers were asked to contribute seven true and seven plausible deceptive statements without a restriction of domain, each in a single sentence. the balanced dataset consists of 7168 annotated instances with additional demographic information. the data set is made available without specifying usage restrictions. 6 real-life trial data (trial). to study real-life high-stake deception scenarios, @xcite collected videos of trial hearings from publicly available sources like "the innocence project" website foot_6 . the dataset contains multimodal information with annotations for non-verbal behavior like facial displays and gestures in addition to crowdsourced transcriptions. it contains 60 truthful and 61 deceptive reviews.this corpus is made available without specifying any usage restrictions. 6 boulder lies and truth corpus (bltc). @xcite built a balanced dataset containing reviews elicited via amt for the domains of electronic appliances and hotels. the crowdworkers were instructed to write fake or real reviews, with positive or negative sentiment, about objects that they were familiar with or not. unlike other datasets which limited the labeling to truthful vs. deceptive, this dataset distinguished between fake and deceptive reviews, where the former are fabricated opinions about an unknown object while the latter was a false review of a known object. the corpus contains 1492 reviews, out of which 451 are truthful and the rest is labeled as fake or deceptive. it is available through the ldc. foot_7 online deceptive reviews (onlinede). to address the bottleneck that large realistic data for deception detection do not exist, @xcite created the onlinede corpus containing manipulated reviews posted online. they employed the automatic deception detection framework outlined by @xcite to identify deceptive reviewers and reviews from social media manipulation campaigns. it contains more than 100k labeled reviews with ≈10000 deceptive instances, covering more than 30 domains. the dataset is available for research purposes from the authors. ## similarity analysis the datasets included in unidecor come from a variety of domains and differ markedly in terms of the method of collection. at the same time, datasets from the same domains also have differences (e.g., solicited reviews vs. actual reviews). to understand the differences of datasets better, we explore the similarity values between these datasets figure 1 : similarity values, according to the measure proposed by @xcite , between all pairs of datasets. the heatmap reflects the domains of datasets. for instance, bltc, opspam, as well as derev2014 and derev2018 from the review domain have similarity scores of 0.89 and .87, respectively. the opinion statement datasets cross-cultde and decop exhibit a high similarity score of 0.83. similarly, mafiascum and diplo-macy show relatively high similarity (0.77), despite differences in the game rules. datasets obtained under specific conditions within the same domain are assigned a lower similarity score. for instance, boxoflies, which is a game that takes place in an in-person setting, differs from the online game datasets (.42 with diplomacy and mafiascum). we also observe similarity across domains, e.g., bluff is more similar to reviews than games, presumably due to its monologue setting instead of dialogue. ## linguistic correlation analysis to understand the generalizability of linguistic cues across different dataset, we conduct a correlation analysis, similar to previous studies that focused on isolated or smaller numbers of corpora @xcite ## deception detection experiments the correlation analysis in the previous section showed that deception cues do barely generalize across domains. this analysis might be limited by the choice of categories, which motivates us to conduct cross-corpus modeling experiments. ## conclusion & future work different scientific disciplines have contributed to the creation of deception datasets for textual communication in a variety of domains. in this study, we present a comprehensive survey of deception datasets in english available for research and compile them into a unified deception dataset. we are not aware of any previous work that considered a comparably large amount of corpora and evaluated models between all of them. some of the evaluation results are encouraging, but particularly between dissimilar domains, the generalization is limited and requires future research. the roberta-based classification experiments and linguistic correlation analysis of deception cues demonstrate that it is indeed challenging to generalize the concept of deception across datasets, or domains. in the classification experiment results, the wildly diverging f 1 scores can be attributed to the complexity of the task as well as to the limitations of the approach employed. in future work, we plan to explore the reasons for this variability across datasets further. additionally, we acknowledge the need to address the issue of biased models, such as the ones trained on mafiascum, onlinede, and diplo-macy, which tends to favor truthful labels owing to the label imbalance in these datasets, resulting in an f 1 score of 0. to overcome this challenge, we could employ techniques like oversampling to rectify the class imbalance and improve the reliability and effectiveness of our approach. the goal of our future work is to create robust deception detection models that work reliably across corpora and domains. this includes understanding differences in the concept as it represents itself in these data and understanding differences in linguistic realization. our unidecor dataset serves as a valuable resource for future research enabling standardized data comparison, transfer learning, and domain adaptation experiments.