Acoustic-Prosodic and Lexical Cues to Deception and Trust: Deciphering How People Detect Lies (2020)

Abstract: Humans rarely perform better than chance at lie detection. To better understand human perception of deception, we created a game framework, LieCatcher, to collect ratings of perceived deception using a large corpus of deceptive and truthful interviews. We analyzed the acoustic-prosodic and linguistic characteristics of language trusted and mistrusted by raters and compared these to characteristics of actual truthful and deceptive language to understand how perception aligns with reality. With this data we built classifiers to automatically distinguish trusted from mistrusted speech, achieving an F1 of 66.1%. We next evaluated whether the strategies raters said they used to discriminate between truthful and deceptive responses were in fact useful. Our results show that, although several prosodic and lexical features were consistently perceived as trustworthy, they were not reliable cues. Also, the strategies that judges reported using in deception detection were not helpful for the task. Our work sheds light on the nature of trusted language and provides insight into the challenging problem of human deception detection.

## introduction humans are notoriously poor lie detectors, most performing at chance level or worse @xcite . this result has been found across a wide variety of deception detection tasks, in multiple modalities, and in different cultures. although poor performance has been well-attested, very little work has been done to understand why humans perform so poorly at detecting lies. because humans are so poor at deception detection, there have been many efforts to develop automated methods to detect deception in multiple modalities. biometric indicators, typically measured by the polygraph (a device used to detect lies by measuring blood pressure, pulse, respiration, and skin conductivity), have been shown to perform poorly at deception detection @xcite . facial expressions @xcite , gestures and body posture @xcite , and even brain imaging (meijer and verschuere, 2017) have been explored as potential indicators of deception. some of these features are difficult or expensive to capture automatically, or are too invasive to be practical for general use. in recent years, automatic deception detection has gained popularity in the speech and nlp communities. language cues have the advantage of being inexpensive, non-invasive, and easy to collect automatically. more importantly, prior research examining linguistic cues to deception has been promising. researchers have used machine learning to identify deceptive language in various domains, including court testimonies @xcite , hotel reviews @xcite , and interview dialogues @xcite . these automated methods have demonstrated that machine learning classifiers can indeed identify deceptive language with accuracy between 70% and 90%, depending on the task-much better than human performance on the same task. these studies have also identified specific characteristics of deceptive language. despite these important advances in understanding and automatically identifying deception, there has been little work investigating human perception of deception. what linguistic and prosodic characteristics of an utterance lead listeners to believe that it is true-to trust it-regardless of whether it is true or not? why do people frequently believe lies? how do the strategies humans use in lie detection align with actual indicators of deception and how do they relate to people's performance in lie detection? can we in fact train machine learning classifiers to automatically identify speech that will be perceived as truth (trusted) or lie (mistrusted) by humans? to investigate these questions, we created a lie detection game, liecatcher, to conduct a large-scale study of human perception of deception. the stimuli for this game were drawn from a large corpus of previously collected truthful and deceptive dialogues; players were asked to judge whether single utterance spoken responses to written questions were truthful or deceptive. we distributed the game on amazon's mechanical turk crowd-sourcing platform to collect large scale judgments of deceptive or true responses to a set of biographical questions. we systematically analyzed a number of linguistic and prosodic features in the rated responses to understand the characteristics of trusted vs. mistrusted speech. we compared these features to the actual characteristics of truthful and deceptive responses presented in the game to identify the similarities and differences between human perception of deception and the actual production of deception. we also examined player-reported strategies to discover which the raters believed to be useful and which were in fact useful or not useful for detecting deception. finally, we trained machine learning classifiers using a large set of lexical and speech features to automatically identify human-trusted speech. the contributions of this paper include: 1) a large-scale analysis of linguistic and prosodic cues to trust compared with cues to deception; this adds considerably to our scientific understanding of human perception of deception. our results show that there are several prosodic and lexical features that were consistently perceived as trustworthy, but that these were not reliable cues to deceptive speech. 2) a game framework for studying deception perception, which can be extended to other speech and language perception studies. 3) a classifier that uses lexical and acoustic-prosodic features to identify speech that was trusted by humans, achieving an f1 of 66.1%. 4) an analysis of successful and unsuccessful human strategies for de-tecting deception, showing that strategies that judges reported using in deception detection were not helpful for the task. we further believe that this latter analysis may be useful for training humans to detect lies more successfully. ## related work previous studies have examined deceptive language in various domains, including fake reviews @xcite , public trials @xcite , tv shows @xcite , twitter @xcite , opinions on controversial topics @xcite , online games @xcite , and interviews @xcite . machine learning classifiers have been shown to outperform human judges by a large margin. for example, @xcite trained a deception classifier that achieved nearly 90% accuracy on a corpus of fake hotel reviews, whereas human accuracy was about 60%. researchers have also examined various features that are characteristic of truthful vs. deceptive language. a meta-study by bond @xcite highlighted several patterns of deceptive language found in multiple studies, such as shorter responses, fewer details, and more negative emotions. other cues to deception that have been identified include language that is less sensory or concrete @xcite . truthful language has been found to contain more linguistic markers of certainty @xcite . syntactic features such as lexicalized production rules and part of speech tags have also been shown to be useful in predicting deception @xcite . linguistic inquiry and word count (liwc) @xcite , which groups words into psychologically meaningful dimensions, has also been used extensively in deception studies @xcite @xcite . prosodic cues to deception have also been identified; for example, @xcite found increased pitch maximum and intensity maximum are indicators of deception. though these studies are critical for advancing the state of machine deception detection and for understanding the nature of deceptive language, they do not address the question of human perception of deception, which is the focus of this work. we aim to gain insight into why humans are poor judges t my dad works i don't i never really know how to say it he works with computers um in information technology t he's a technical engineer at draper laboratory f he works for um it's like um a subsidiary of walgreens kind of it's very it's very corporate it's like a big big very impersonal company which is i think he doesn't like about it f uh my dad is an official in uh in the government system psychology research of human deception detection has traditionally focused on facial expression cues @xcite and personal beliefs about what characterizes deceptive behavior @xcite @xcite . based on worldwide survey studies, the global deception research @xcite found pan-cultural deception stereotypes that liars tend to be nervous with flawed speech. however, @xcite pointed out the methodological limitation of such studies: we cannot be certain that what people report reflects their actual decision process @xcite . our work attempts to decipher the cues people actually use to detect lies by examining features of utterances that are labeled as true by participants, compared with features of utterances rated as lies. ## cxd corpus we used deceptive and truthful utterances from the columbia x-cultural (cxd) corpus for our deception perception study @xcite . the cxd corpus is a collection of interviews between native speakers of standard american english and mandarin chinese, all speaking in english. it contains 122 hours of conversational speech between 340 individuals. previously unacquainted pairs of participants were brought into the lab to interview one another. they were first surveyed for gender and native language and asked to complete the neo-ffi personality inventory @xcite . they were then asked to provide true answers to a set of 24 biographical questions and then to provide false answers for a random half we chose. interviews took place in a soundproof booth and each pair of participants took turns playing the role of interviewer and interviewee. during the game, the interviewer asked the 24 questions in any order and was encouraged to ask follow-up questions to help determine whether the interviewee was lying or telling the truth about each question. participants were financially compensated for both successful deception and successful deception detection. the recorded interviews were orthographically transcribed using amazon mechanical turk (amt) crowd-sourcing and the transcripts were force-aligned with the audio recordings using the kaldi speech recognition toolkit @xcite . the interviews were segmented using a question identification classifier @xcite . all interviewee turns were automatically identified using the question identification system and subsequently hand-corrected. the corpus was segmented into: 1) question responses: the single interviewee turn directly following the question; 2) question chunks: all interviewee turns in (1) plus answers to subsequent follow-up questions. we used the single turn question response segmentation for our deception perception study, so as not to influence raters' responses with interviewers' follow-up questions. ## liecatcher using the data described in section 3, we created a lie detection game called liecatcher. 1 liecatcher is a game with a purpose that allows players to assess their overall ability to detect lies, while simultaneously providing deceptive speech judgments that we then use to study deception perception. we developed liecatcher in unity foot_1 and hosted the game on the web. figure 1 shows a screenshot from the game. in the game, players are shown a text version of a question asked by one interviewer and then listen to the singleturn spoken interviewee response. after listening, the player selects a ''truth'' or ''lie'' button, indicating their perception of the speech sample as truthful or deceptive. the game was designed so that players could submit their decision only after the audio had finished playing, so they could not make a judgment without hearing the full response. this feature of the game also provided information about raters' behavior when making judgments, as we recorded the time interval between the end of the audio clip and the time that the player entered their response for each decision. after the gameplay, a score report is displayed summarizing all their judgments for that task, giving players feedback about their performance at the end of each multi-question task. ## textual and prosodic indicators of trust and deception in this section, we consider the following questions: what are the characteristics of trusted and mistrusted speech? how do these compare with the characteristics of truthful and deceptive speech? of all features raters believed to indicate lies, which are valid cues and which not? also, what are the deceptive cues that raters failed to perceive? we compared features of trusted and mistrusted utterances and features of truthful and deceptive utterances using paired t-tests. labels for trust were computed using majority vote (i.e., an utterance is considered trusted if at least 2 annotators believed it is true, otherwise mistrusted). we also compared utterances trusted by all and mistrusted by all, observing differences in complexity and prosodic features. for complexity features, #verbs, #nouns, #num, concreteness ceased to be significant, and type-token became significant with @xmath0.05. for prosodic features, pitch mean and pitch std ceased to be significant, and intensity max became significant with @xmath1.05. notice that all features that differ had relative small significance levels @xmath2.001. to prevent the inflation of false positive errors caused by conducting multiple comparisons, we present only features that are statistically significant after benjamini-hochberg correction @xcite . ## characterizing strategies for detecting lies we summarized annotator-provided strategies for detecting lies based on previous work in deception detection @xcite @xcite @xcite and annotators' responses. the strategies were manually labeled by domain experts with previous research experience in deception detection and all ambiguities were discussed by three people. for each strategy, we computed the average percentage of utterances judged correctly by annotators who reported using this strategy and compared it with the average percentage of utterances judged correctly across all annotators. we performed the same analysis for the percent of utterances the raters believed to be true (trusted). in addition, we reported the percentage of annotators who claimed to have used these strategies in which strategies are reported by raters who are more or less trusting over all? we found that people who reported using response latency, pauses, and disfluency when judging deception trusted a smaller percentage of utterances. this could be because of the high prevalence of disfluencies in spontaneous speech, regardless of whether the utterance was deceptive or not. we also found that raters who used the level of detail in a response as a cue were more mistrusting in their judgments. conversely, those who used clarity of a response and prior domain knowledge were more trusting. does complex reasoning correlate with accuracy in lie detection or trust level? we examined two measures of player behavior that approximate complex reasoning: how long do people take to make judgments? how many strategies do they report in total? we measured how long people took to judge responses using the time interval between the end of the audio clip and the time that annotator entered his/her response. we found no correlation between response time and the percentage of strategy %correct %trust %used example prosody -0.25 -0.17 45.74% voice tone and pattern response latency +0.11 -2.13 * * 30.71% listened for delays in the speakers response pauses -0.52 -2.95 * * 24.66% i listened for pauses to see... disfluency -0.59 -1.88 * 22.87% if they said ''um'' i thought they were lying intuition +1.09 +0.52 22.87% my gut instinct... details +0.81 -2.95 * 17.26% ...how much or how little detail they used... prior +1.95 +2.85 * 13.90% how realistic the answers were style -0.65 +0.86 11.88% anxiety in voice confidence -2.83 * -1.60 11.21% paying attention to the person's confidence.. duration -0.94 -2.80 9.41% length of answer speaking rate +0.39 -0.64 6.72% speed of answer speaker traits +0.07 -0.00 6.05% how relaxed they were lexical +1.53 +1.00 5.16% look for context around the words laughter +1.04 +0.40 1.79% if they laugh its false clarity +2.52 +9.71 * 1.35% people usually give more and clearer details... breathing +5.33 -2.73 1.12% i tried to notice when they breathe so deeply.. repeat question +0.36 +6.10 0.67% i id notice one person repeat the question.. contradictions +0.04 +1.24 0.67% ...the person blatantly contradicted themselves... repetition +1.24 +6.52 0.44% repetition when lying ## conclusion in this paper we presented a framework for understanding human deception perception. we created a lie detection game, liecatcher, and used it to collect large-scale judgments of deceptive speech. we analyzed a large set of linguistic and prosodic cues to deception and identified some mismatches between the responses people perceived as deceptive and those that were actual deceptive responses. particularly notable in these mismatches were prosodic features, suggesting that humans have difficulty interpreting prosodic cues to deception. we built a predictive model of trust with a macro-f1 score of 66.1%, and showed that disfluencies and prosody were most useful for predicting trust. we summarized and manually annotated annotator-provided strategies and found that none of them were associated with an improvement in lie detection ability; however, some were associated with raters' tendency to trust. the identified mismatches between features of trusted vs. deceptive speech, as well as the lack of useful strategies reported by raters, shed new light on the poor performance of humans at deception detection. in addition, we showed that complex reasoning did not correlate with accuracy in deception detection but negatively correlated with trust level. this work has implications for several applications in multiple disciplines. in business, politics, and interpersonal relationships, it is critical to cultivate the trust of others. our empirically identified characteristics of trusted language provide useful information for training individuals who want to speak in a more trustworthy manner. furthermore, we are interested in using these findings to synthesize voices that are likely to be trusted by others and we have already begun that process. potential applications that can benefit from trustworthy voices include dialogue systems and robots, especially for assistive technologies (e.g., for individuals with disabilities, elderly individuals) where trust is crucial for successful interactions. our liecatcher game was a useful framework for studying perceived deception in an engaging format. the experiments presented in this paper were conducted using stimuli from the cxd corpus. in future experiments, we plan to conduct a cross domain analysis to see if these findings generalize to other domains and corpora. because of the lack of corpora with annotations of human perceptions of deception, we plan to conduct similar perception studies using the liecatcher framework to enable this cross-domain analysis. in addition, the liecatcher game can be extended to explore the perception of other aspects of spoken language. we are currently exploring its use for training purposes. in the future, we will provide more immediate feedback to players about their judgments for each response, with the goal of training practitioners to improve their performance at deception detection.