M ental M anip: A Dataset For Fine-grained Analysis of Mental Manipulation in Conversations (2024)

Abstract: Mental manipulation, a significant form of abuse in interpersonal conversations, presents a challenge to identify due to its context-dependent and often subtle nature. The detection of manipulative language is essential for protecting potential victims, yet the field of Natural Language Processing (NLP) currently faces a scarcity of resources and research on this topic. Our study addresses this gap by introducing a new dataset, named \textsc{MentalManip} , which consists of 4,000 annotated fictional dialogues. This dataset enables a comprehensive analysis of mental manipulation, pinpointing both the techniques utilized for manipulation and the vulnerabilities targeted in victims. Our research further explores the effectiveness of leading-edge models in recognizing manipulative dialogue and its components through a series of experiments with various configurations. The results demonstrate that these models inadequately identify and categorize manipulative content. Attempts to improve their performance by fine-tuning with existing datasets on mental health and toxicity have not overcome these limitations. We anticipate that \textsc{MentalManip} will stimulate further research, leading to progress in both understanding and mitigating the impact of mental manipulation in conversations.

## introduction mental manipulation is a deceptive strategy aimed at controlling or altering someone's thoughts and feelings to serve personal objectives @xcite . facilitated by digital technologies, mental manipulation has gained unprecedented capability to target and influence individuals via interpersonal interactions and public dissemination of information @xcite , causing significant mental health distress to victims @xcite . compared to overt verbal toxicity and abuse, such user input: does this dialogue contain elements of mental manipulation? just answer "yes" or "no". <insert the above @xmath0: no. as hate speech, manipulation is more insidious, deceitful, and context-dependent, posing challenges for individuals and automatic moderation tools to discern. for years, the nlp community has witnessed advancements in verbal toxicity and abuse detection, but most of those works focus on contextfree content and face challenges in identifying implicit toxicity @xcite @xcite . existing works in dialogue systems have targeted context-aware toxicity, but are limited to direct toxicity, such as profanity, condescension and forms of hate speech @xcite @xcite . we argue that existing toxicity detection resources are insufficient for developing automatic systems to detect and properly handle verbal mental manipulation. additionally, current state-of-the-art large language models (llms) are not well-positioned to address this issue, as demonstrated in figure 1 . this paper introduces mentalmanip, a dataset with multi-level annotations for mental manipulation detection and classification. we define mental manipulation as using language to influence, alter, or control an individual's psychological state or perception for the manipulator's benefit. detexd @xcite delicate text no 1, 023 1 online forums multi-level fox news @xcite hate speech yes 814 2.0 (±0.1) news comments binary talkdown @xcite this definition aligns with one of the explanations of manipulative influence in barnhill's work on the philosophy of online manipulation @xcite . mentalmanip dataset contains 4, 000 multi-turn fictional dialogues between two characters extracted from online movie scripts. to enable fine-grained analysis, we proposed a labeling taxonomy covering three dimensions: presence of manipulation, manipulation technique, and targeted vulnerability, which are illustrated in figure 2 . our taxonomy aids in the precise detection of mental manipulation and facilitates a nuanced classification of the techniques used by manipulators, as well as the vulnerabilities targeted in victims. we conducted three classification tasks on mentalmanip to detect the existence of mental manipulation and its elements. our experiments spanned state-of-the-art generative and discriminative llms across multiple settings. to investigate the effectiveness of existing datasets in toxic speech and mental health for our objectives, we fine-tuned llms with seven relevant datasets and conducted evaluations. experimental results reveal that these llms are limited in understanding mental manipulation, as shown by a significant number of misclassified "manipulative" dialogues. moreover, fine-tuning llms with these relevant datasets did not enhance their detection and classification capabilities on manipulative language. these findings highlight the importance of our dataset, and suggest an avenue for future studies in mental manipulation detection and analysis. our mentalmanip dataset, along with the code for statistical analysis and experiments in this paper, is available in our github repository: https: //github.com/audreycs/mentalmanip . ## conclusion and future studies