Beyond Emotion: A Multi-Modal Dataset for Human Desire Understanding (2022)

Abstract: Desire is a strong wish to do or have something, which involves not only a linguistic expression, but also underlying cognitive phenomena driving human feelings. As the most primitive and basic human instinct, conscious desire is often accompanied by a range of emotional responses. As a strikingly understudied task, it is difficult for machines to model and understand desire due to the unavailability of benchmarking datasets with desire and emotion labels. To bridge this gap, we present MSED, the first multi-modal and multi-task sentiment, emotion and desire dataset, which contains 9,190 text-image pairs, with English text. Each multi-modal sample is annotated with six desires, three sentiments and six emotions. We also propose the state-of-the-art baselines to evaluate the potential of MSED and show the importance of multi-task and multi-modal clues for desire understanding. We hope this study provides a benchmark for human desire analysis. MSED will be publicly available for research.

## introduction multi-modal sentiment and emotion analysis has immense potential in dialogue analysis and generation, emotion communication, etc., which has been an active field of research in natural language processing (nlp) @xcite . although numerous advanced models and datasets have been proposed, covering different levels of granularity, such as sentence, aspect, conversation, human desire behind emotions is still relatively unexplored. human desire understanding models and datasets can benefit different areas of nlp and ai. research in ai is a step closer to recognizing human emotional intelligence if a machine is able to achieve a deeper understanding of human desires and even make reasonable desire-aware responses @xcite . with researchers' increasing understanding of emotional intelligence and advancements in multi-modal language analysis, desire understanding and analysis comes into view @xcite . desire is a primitive instinct and a basic need for strongly expressing human wants to get or possess something, where its endless and insatiable attributes distinguish human beings from other animals @xcite . it involves not only a linguistic expression, but also has underlying cognitive phenomena driving human sentiments and emotions @xcite . hence, we argue that there is a close relationship between human desire, sentiment and emotion, where desire stealthily dominates sentiment and emotion while sentiment and emotion also have influence on desire. such three tasks are complementary in that desire analysis helps the understanding of the other two. for example, in fig. 1 (a), three kids with a magnifying glass are smiling and observing something interesting. the positive sentiment and happy emotion are judged by means of the desire curiosity. fig. 1 (b) depicts that a young lady and her two children are walking at a leisurely rate along a winding road. their smiles in the image and the words in its text counterpart convey joyful emotion. such feelings explains the lady's strong need to be in the company of the children, i.e., family desire. we also check whether our hypothesis is tenable in the experiments (c.f. sec. 5.5). given the importance of desire understanding, numerous research results in psychology and philosophy have been proposed and are being actively studied to explain and analyze human desire, e.g., desire inference @xcite , the correlation between desire and love @xcite , desire diagnosis @xcite . however, it is still an understudied new task in nlp and multi-modal affective computing. the lack of publicly available desire datasets has been the main issue in advancing multi-modal desire analysis models. in this paper, we take the first step to overcome this limitation by presenting msed, a novel multimodal dataset manually annotated with sentiment, emotion and desire labels. msed consists of 9,190 text-image pairs collected from a wide range of social media resources, e.g., twitter, getty image, flickr. it aims to extend the goal of human desire understanding within other disciplines and bring it to the nlp community. this dataset also facilitates the study of desire detection models by investigating both multi-task and multi-modal clues. besides, msed is also valuable for other nlp domains such as multi-modal language analysis, multi-task learning. in summary, the major contributions of the work are: multi-modal emotion recognition in conversation (erc) has increasingly become an active research topic. the community also established iemocap @xcite meld @xcite , scenariosa @xcite and mus-tard @xcite , to show the impact of social interaction on human emotion evolution. however, the existing datasets only contain sentiment and emotion annotations. there is a lack of a dataset which provides insights into the desire be-hind human emotions. in contrast, msed contains all of sentiment, emotion and desire multi-modal annotations to support and encourage future studies on the correlation between desire, sentiment and emotion. ## the msed dataset the process of creating msed, the annotation procedure and the basic features are detailed. ## dataset analysis desire analysis. we present the distribution of desire labels in msed, as shown in fig. 4 . from fig. 4 (a), we observe that desire and non-desire samples account for 51% and 49% respectively. this shows that msed is a well-proportioned and balanced dataset, which is suitable for machine or deep learning based analysis. specially, the proportions of curiosity, family and romance are 11%, 14% and 11%. which are much larger than the proportions of vengeance and tranquility (i.e., 4%, 4%). this is also in line with our actual life where fewer people are ready to share their dark sides and flurried attitudes on social media platforms. more people are likely to publish tweets about family life, romantic love, etc. sentiment analysis. fig. 4 (b) shows how sentiment is entangled with desire and non-desire. we can see that positive sentiment accounts for the largest proportion of 53% in desire samples while negative sentiment is not far behind, i.e., 33%. neutral polarity has the smallest proportion of 14%. the proportion of non-neutral sentiment towers over that of neutral polarity. in non-desire data, the proportion of neutral polarity (i.e., 42%) is more than the proportion of positive and negative sentiments (i.e., 28% and 30%). but the proportions of neutral and non-neutral sentiments turn out very close, which indicates that there is a poor correlation between non-desire and the different sentiment classes. these results have verified our previous arguments: (1) human desire is often accompanied by a range of sentiment responses; and (2) desire stealthily dominates emotion. ## conclusions and future work human desire understanding is a relatively unexplored task in nlp. to fill this gap, we expand desire research from psychology to multi-modal language analysis, and thus propose the first multimodal multi-task dataset for desire, sentiment and emotion detection, msed. each sample is annotated with six basic desires, three sentiments and six emotions. in addition, qualitative and quantitative studies are performed for analyzing the dataset. we also present a range of baselines to evaluate the potential of msed. the comparative and human evaluation results demonstrate the need of new desire analysis models and the potential of msed to facilitate the development of such models. our work has also a few limitations. the images available in platforms like flickr and getty may not express spontaneous human desire, as many of them are purposefully designed by professional photographers. the current dataset only collects static images and texts, the conversational samples might be considered. moreover, a larger scale multi-modal dataset with more desire categories is left to our future work. the technique of human desire analysis based on online data also has the potential to be misused, e.g. by integrating them with facial recognition techniques to make interventions or decisions for humans. in summary, we hope that the creation of msed will provide a new perspective in nlp for research on human desire analysis. the dataset will be publicly available for research. given the close relationship between desire, sentiment and emotion, a refined multi-modal multi-task learning framework is left to our future work.