Automatic Annotation of Dream Report’s Emotional Content with Large Language Models (2024)

Abstract: In the field of dream research, the study of dream content typically relies on the analysis of verbal reports provided by dreamers upon awakening from their sleep. This task is classically performed through manual scoring provided by trained annotators, at a great time expense. While a consistent body of work suggests that natural language processing (NLP) tools can support the automatic analysis of dream reports, proposed methods lacked the ability to reason over a report’s full context and required extensive data pre-processing. Furthermore, in most cases, these methods were not validated against standard manual scoring approaches. In this work, we address these limitations by adopting large language models (LLMs) to study and replicate the manual annotation of dream reports, using a mixture of off-the-shelf and bespoke approaches, with a focus on references to reports’ emotions. Our results show that the off-the-shelf method achieves a low performance probably in light of inherent linguistic differences between reports collected in different (groups of) individuals. On the other hand, the proposed bespoke text classification method achieves a high performance, which is robust against potential biases. Overall, these observations indicate that our approach could find application in the analysis of large dream datasets and may favour reproducibility and comparability of results across studies.

## introduction dreams have fascinated humans since the dawn of time, and their scientific study in the last decades even increased attention and interest towards this peculiar phenomenon. indeed, available evidence suggests that dreams may be related to psychophysical well-being, and may be involved in or repre-sent a window on sleep-dependent processes affecting the consolidation and integration of new memories @xcite @xcite . moreover, given their nature of internally generated conscious experiences, dreams are regarded as a fundamental model to study and understand human consciousness @xcite . in spite of this, the mechanisms that lead to dream generation and development, and the possible functions of dreams still remain poorly understood to this day. among the factors that limit and slow down research on dreams is the fact that the content of dreams is difficult to assess quantitatively and in a reproducible way @xcite . automating and standardising the scoring of dream reports' emotional dimensions is paramount for health and psychophysiological well-being as it can uncover valuable insights into an individual's mental states during sleep. as stated by the established continuity-hypothesis @xcite , elements in dream scenarios mirror someone's waking states and concerns @xcite @xcite . nightmares have a particular potential to disrupt everyday life as they are linked to high levels of psychological distress, self-harm, and suicidal tendencies @xcite . their frequency may serve as a promising early indicator of psychiatric and sleep disorders @xcite . in line with this, dream content was reported to change in several pathological conditions, including for instance eating disorders and depression @xcite . typically, the assessment of dream content -including the presence of specific emotions -is performed manually, by trained annotators, by applying particular scales or scoring systems. while multiple scoring approaches exist to annotate and analyse dream reports, such as the scale by hauri and colleagues @xcite or the rating system developed by schredl @xcite , the hall and van de castle (hvdc) coding system @xcite remains the most popular and widely adopted @xcite . a growing body of evidence has shown that nlp methods can support the automatic analysis of dream reports. so far, efforts have mainly focused on investigating different implicit structures, such as speech or syntactic graphs @xcite , and/or analysing the semantic content of dream experiences @xcite @xcite (see @xcite for an extensive review). of more relevance for this work are those studies that focused on dream reports' semantic content using dictionary-based linguistic analysis @xcite @xcite and distributional semantic models @xcite @xcite . while notably different, both approaches cannot fully and coherently manipulate a report's full content and context. this shared limitation is of great relevance, as the correct identification of an emotional state may rely on complex constructions and more implicit information, as well as a combination of the two. in recent years, these and similar issues were largely overcome by pre-trained large language models (llms) based on transformer architectures @xcite . over the last years, llms pre-trained on self-supervised tasks like masked language models have shown strong performance on down-stream tasks like sentiment analysis @xcite ), text summarisation @xcite , question answering @xcite , and machine translation @xcite . given their success, in this work, we propose to address the issues identified with existing approaches to automatically analyse dream reports analysis by leveraging pre-trained llms. specifically, we investigate whether and how llms can support the detection and analysis of emotions expressed in dream reports, as defined in accordance with the hvdc coding framework. more specifi-cally, we study the ability of a bespoke multi-label text classifier, based on a pre-trained llm tuned using dream reports previously scored by expert annotators, and propose a set of experiments and analyses to test the robustness of this solution to different potential biases in the dataset and out-ofdistribution applications. to the best of our knowledge, our work represents the first attempt to analyse and reproduce gold-standard hvdc annotations of dream reports with llms, and makes two main contributions. first, we show how, despite the limited amount of training data, a fully-supervised approach based on multi-label text classification yields good and stable performance. two, we provide follow-up experiments and analysis showing how the strategies learned by the model are robust with respect to out-of-distribution data, as well as biases and spurious correlations present in the dataset. ## related work as summarised by @xcite , a growing body of research is adopting nlp methods to automatically analyse dream reports. yet, while emotions represent a fundamental component of oneiric experiences, only a fraction of published studies based on nlp methods have explicitly focused on the emotional aspects of dream reports @xcite @xcite @xcite @xcite . moreover, most of these investigations did not include a direct nor transparent comparison with widely adopted report annotation approaches such as the hvdc coding system. in terms of implementation, adopted nlp methods include three main solutions: graphbased approaches @xcite @xcite , dictionary-based linguistic analysis @xcite @xcite , and distributional semantic models @xcite @xcite . dictionary-based methods analyse data word by word, comparing each item to a dictionary file that is structured as a collection of words defining different semantic categories. an example could be the 'positive emotion' category, containing words such as "joy", "happiness", and "smiling". approaches based on these methods @xcite @xcite @xcite are mainly used to determine the relative frequency of references to specific content words, and can hence be inherently misleading, as they generally cannot interpret contextual information and syntactic structures. syntax-like structures are used by graph-based approaches, which cannot access semantics but have successfully been adopted to classify populations of participants (e.g., healthy and psychotic subjects) @xcite @xcite ). an exception is the work by @xcite , which proposed a solution based on a combination of dictionary and graph models, able to extract information about the content of dream reports, including their emotions. however, the evaluation was reframed in binary ("positive", vs "negative") terms. solutions based on distributional semantics @xcite @xcite @xcite were largely based on word-level representation obtained using models like word2vec @xcite or glove @xcite . in such cases, encodings of full reports were generated by adding or averaging word embeddings, losing access to syntactic structure and more contextual understanding (klafka and ettinger, 2020). a niche of previous work applied nlp and machine learning methods specifically to assess emotional aspects of dream reports @xcite @xcite . @xcite added word associations to improve the performance of a machine learning model that was trained to automatically score dream reports' emotional tone based on human judgements, resulting in increased machine-human agreement. yet, relying on predefined word associations may oversimplify the nuances of emotional content unique to dreamers. @xcite combined ad-hoc classifiers with a distributional approach to detect potential shifts in sentiment within each report. evaluated reports were extracted from dreambank, a large public database, and were (re-)scored by the authors using a four-level emotional rating system ("very-negative" to "very-positive"). despite relying on a mixture of local (word-toword) and general (sentence-to-sentence) occurrences, the adopted approach strongly relies on extensive data pre-processing, as well as composition by averaging, hence losing access to structural and deeper semantic information. @xcite used a pre-trained agent to detect recurrent themes in a series of reports and found a partial match in the retrieved themes with aspects of the hvdc system, as well as significant differences in how themes occurred in male vs. female dreamers. the distribution of these themes was then used to assess the "mood" of each report. yu (2022) combined a dictionary-based method with support vector machines @xcite to asses the general sentiment of dream reports in multiple languages. overall, the described studies present two main differences from our work. first, the models lacked access to the global context of each report. in our work, we use pre-trained large language models (llms) to encode dream reports and thus allow a model to have full access to the context. second, the annotations and evaluations of emotional states were not directly compared against widely used coding systems such as the hvdc. that is, whenever human annotations were considered for the evaluation of a system, scores or labels were largely (and generally non-transparently) re-framed to be comparable with the output produced by the system of choice. in this work, we propose a solution to adapt a model to produce interpretable labels, that can be directly compared against human-produced hvdc annotations. furthermore, evaluations will take into account the fact that labels could be associated with different characters, thus further highlighting the possible value of our approach as fully automatic and reliable support for manual annotations in dream research. ## dataset for our experiments, we use a subset of reports extracted from the dreambank database foot_0 (domhoff and @xcite , pre-annotated according to the hall and van de castle (hvdc) coding system @xcite . dreambank.net consists of a collection of over 20k dream reports gathered from different sources and organised in series, either provided by single individuals or groups of people, such as college students, teenagers, and blind adults. while dreambank.net can be freely explored, the reports and the hvdc scores adopted in the current work are made available upon direct request to the researchers who maintain the dreambank website. among the approximately 1.8k labelled dream reports, all in the english language, 922 contained at least one emotion associated with either the dreamer or another character. considering that no guidelines or metadata are available to demonstrate that the absence of emotion labels reflects the actual absence of emotions in a report, we focus our experiments on those reports containing at least one emotion (@xmath0. the dataset is further divided into six series: bea 1: a high school student (@xmath1; total number of reports/reports including at least one emotion), ed: dreams of his late wife (@xmath2, emma: 48 years of dreams (@xmath3, hall/vdc norms: female (@xmath4, hall/vdc norms: male (@xmath5, barb sanders: baseline (@xmath6. the hvdc coding system examines ten categories of elements appearing in dream reports (characters, interactions, emotions, activities, striving, (mis)fortunes, settings and objects, descriptive elements, food and eating, and elements from the past). within this study, we focused only on the annotation of the emotions feature. in the hvdc coding system, emotions are divided into 5 classes, that are anger (an), sadness (sd), apprehension (ap), confusion (co), and happiness (ha). emotions might be assigned either to the dreamer or to other dream characters. we analyse both the emotions scored as experienced by the dreamer (dreamer emotions) and the overall occurrence of emotions in the dreams regardless of the dream characters they are associated with (general emotions). ## multi-label text classification a set of preliminary experiments (see appendix b) showed that an off-the-shelf sentiment analysis llm cannot coherently solve the task when framed, similarly to previous work, as a binary positive -negative classification. hence, we investigate whether the human annotation of dream reports can be reproduced with supervision, via a bespoke text classification model, trained on gold-standard hvdc labelled data. contrary to all previous work, that reframed hvdc labels to fit binary classification and classifiers, we perform a fine-grained classification aimed at determining the presence (1) or absence (0) of each hvdc emotion (i.e., anger (an), sadness (sd), apprehension (ap), confusion (co), happiness (ha)), regardless of the number of times they appear in a given report. moreover, we experiment with both the sets of emotions described in section 2: dreamer emotions and general emotions. ## discussion in the field of dream research, the assessment of a report's emotional content is typically based on time-consuming, annotator-dependent procedures. throughout the years, only a few studies employed automated approaches based on nlp techniques, including dictionary-based and distributional semantics methods. however, these approaches have very limited access to the syntax and semantics of a report's content, and may thus fail to correctly and fully capture emotions described in dream reports. in this work, we tested whether a transformer-based large language model (llm) could be used to overcome such limitations and reproduce human-based scoring. specifically, we trained a model end-to-end, using pre-annotated data to predict if and which emotions were present or absent in a given dream report. the obtained results showed that the model was able to learn reliable and stable classification rules. follow-up experiments further confirmed two important aspects of our solution. first, via an ablation experiment, we showed that the ability of the model to solve the task is only marginally affected by differences between distinct subsets of the training data. second, such generalisation holds also for instances that significantly deviate from the training data, as shown by the experiment with reports from a ptsd patient. our findings suggest that what is more likely to impact the model performance is the vocabulary used to describe specific emotions across different series. indeed, variability in the used vocabulary may be explained by the fact that the series included in the present work were collected from different individuals or groups of individuals, with relevant differences in demographic, psychological, and behavioural characteristics. should this be the case, it would be yet another reason to support the use of tools that are able to reason over the full content of a report, and have access to a large and dynamic vocabulary, already have significant information about a large set of lexemes, and can be easily adaptable to new words and languages. current pre-trained transformer-based llms satisfy all these requirements. given the current state of nlp resources, our proposed architecture can be easily adapted to be used with llms pre-trained on different languages or tasks. moreover, trained models like ours are fully open-source and can be easily adopted by researchers and practitioners in their pipeline, making these results and the framework extremely replicable and widely standardised. ## conclusion in this work, we tested the feasibility of using transformer-based large language models (llm) to annotate dream reports with respect to emotions expressed in a given report. our results show that our llm-based solution using multi-label classification yields a strong performance, which was found to be robust to correlational biases and out-ofdistribution data. such approaches have the potential to significantly accelerate research investigating the origin, meaning, and functions of dreams, and might present a valuable and efficient support or alternative to human-based procedures involving the analysis of large datasets, ensuring at the same time reproducibility of the obtained results through the sharing of adopted models.