Improving Cross-domain, Cross-lingual and Multi-modal Deception Detection (2022)

Abstract: With the increase of deception and misinformation especially in social media, it has become crucial to be able to develop machine learning methods to automatically identify deceptive language. In this proposal, we identify key challenges underlying deception detection in cross-domain, cross-lingual and multi-modal settings. To improve cross-domain deception classification, we propose to use inter-domain distance to identify a suitable source domain for a given target domain. We propose to study the efficacy of multilingual classification models vs translation for cross-lingual deception classification. Finally, we propose to better understand multi-modal deception detection and explore methods to weight and combine information from multiple modalities to improve multi-modal deception classification.

## introduction deception detection is a deliberate choice to mislead to gain some advantage or avoid some penalty @xcite . deception detection is an important goal of law enforcement, military and intelligence agencies, as well as commercial organizations. in recent years, automatic deception detection in text has gained popularity in the natural language processing (nlp) community, and researchers have studied cues to deception in a diverse set of domains. these include detecting deception in news @xcite , online reviews @xcite , interview dialogues @xcite , trial testimonies @xcite , and in games @xcite . these studies have been useful for identifying linguistic characteristics of deception, and for developing machine learning techniques to automatically detect deceptive language. however, we are still a long way from applying these state-of-the-art deception detection models in real-world deception scenarios. we currently lack information about how deception detection models perform across domains, languages, and in multiple modalities. in this proposal we outline current limitations in these three areas of deception detection: across domains, across languages and in multiple modalities. we propose work to address these limitations, with the goal of developing robust deception detection models that can generalize from lab-based datasets to real-world deception. for each of the three topics of deception detection, we discuss current limitations, formulate research questions, and state proposed work to address the research questions. for some of our research questions, we present completed or ongoing work to answer the questions. for cross-domain deception classification, we first establish baseline performance at within and cross-domain deception classification using the well-established nlp model bert. we identify major performance gaps between within and cross-domain deception classification. to understand the cross-domain performances, we formulate distance metrics and propose a cross-domain classification model that does not require target domain labeled training data and outperforms several baseline models. we also discuss ongoing and future research to further our understanding of and further imrpove cross-domain deception detection. for cross-lingual deception classification, we formulate the task for deception detection in two non-english languages: bulgarian and arabic. we discuss the effectiveness of using a wide range of classifiers including multilingual bert @xcite , and propose additional experiments to further understand and improve cross-lingual deception detection. finally, we present proposed work in deception detection in a multi-modal setting from text and image features. learning to identify deception is a challenging task, especially when there is one modality, and we propose to dynamically fuse in-formation from multiple modalities. the thorough experiments in the proposed work will contribute substantially to our understanding of cross-domain, cross-lingual, and multi-modal deception detection, and to the development of robust deception detection technologies. ## proposed work and preliminary exploration to address the limitations discussed above, we formulate concrete research questions that would help study deception classification across domains, languages and modalities. we now discuss proposed work and findings from initial experiments for each research question. ## ethical considerations although automatic deception detection has the potential to benefit society, there are several ethical concerns within this line of research. automatic deception detection has varying degrees of severity depending on the application area. the impact of a false positive is substantially lower when detecting deceit in informal activities such as gaming. however, when detecting dishonesty in a criminal investigation, a false positive can have serious implications. in general, automatic deception detection should be employed with caution, especially when there is no manual human verification involved. for the case of cross-domain deception detection applications, it is important to test the model on the target domain before deploying it, as mentioned in section 1. to understand the differences between deception domains, a linguistic feature analysis should be performed, as we mention in section 3.1. finally, to increase transparency in multi-modal deception detection, it is critical to compute importance scores for each modality as mentioned in section 3.3. as automatic deception detection across domains, languages and modalities becomes a more widely studied subject, it is important to be aware of the ethical considerations and also take the necessary precautions to avoid harm to society. ## conclusion we identify key challenges in deception detection in cross-domain, cross-lingual and multi-modal scenarios. for cross-domain deception classification, we quantified the gap between in-domain and crossdomain accuracies. our proposed recommender based on distance measures improves cross-domain performance over two baselines. we plan to extend the completed work by improving the recommendation process by recommending at the sample level instead of the domain level. we also plan to analyze the cross-domain results using linguistic features. for cross-lingual deception classification, we discuss the challenges in predicting deception in a target language with no or little training data. we propose to study the effectiveness of using translation text for training and testing. for multi-modal deception classification, we discuss the merits and limitations of the current state-of-the-art models. we propose to dynamically fuse the text and image feature vectors using an attention module to better understand the importance of each modality.