The PEACE -Reviews dataset: Modeling Cognitive Appraisals in Emotion Text Analysis (2023)

Abstract: Cognitive appraisal plays a pivotal role in deciphering emotions. Recent studies have delved into its significance, yet the interplay between various forms of cognitive appraisal and specific emotions, such as joy and anger, remains an area of exploration in consumption contexts. Our research introduces the PEACE-Reviews dataset, a unique compilation of annotated autobiographical accounts where individuals detail their emotional and appraisal experiences during interactions with personally significant products or services. Focusing on the inherent variability in consumer experiences, this dataset offers an in-depth analysis of participants’ psychological traits, their evaluative feedback on purchases, and the resultant emotions. Notably, the PEACE-Reviews dataset encompasses emotion, cognition, individual traits, and demographic data. We also introduce preliminary models that predict certain features based on the autobiographical narratives.

## introduction language modeling provides insights into the linguistic features associated with how people think, feel, and behave. some studies have modeled various psychological constructs in online text such as world beliefs @xcite , personality @xcite , stress @xcite , locus of control @xcite , and even psychological disorders such as depression @xcite . there is also work on modeling emotions using language models @xcite . however, to understand emotional experiences of individuals comprehensively, what needs to be added in the current literature is understanding the cognitive antecedents of emotionshow people evaluate their situations that results in specific emotional experiences. these evaluations are known as cognitive appraisals @xcite , and this study focuses on constructing a dataset to model such evaluations that people make about their situations. cognitive appraisal theories proposed that emotions are elicited from the evaluation, or appraisal, of situations along a set of motivationallyrelevant dimensions @xcite . consider the following scenario. suppose andy encountered a rude server in a restaurant that only attends politely to other customers except him. andy might interpret this personal situation in the following way-a) the rude behavior of the server is unexpected as he expects that servers should always be polite to their customers, b) the rude server is responsible for him having such an unpleasant experience in the restaurant, and c) he finds that the rude behavior directed only to him and not to other customers is unfair. by interpreting the situation in this manner, andy might experience an emotion like anger. specific emotions are characterized by profiles of cognitive appraisals that represent the evaluation of the person-situation relationship that is important to an individual @xcite . emotions with a certain appraisal profile have been shown to influence decision-making and behavioral processes @xcite . for example, individuals experiencing fear (an emotion that is characterized by the appraisals of high threat and uncertainty) made more risk-averse decisions about an event @xcite . the effect of emotions on decision-making processes and subsequent behaviors is also found in the domain of consumption (buying or using a product). for instance, emotions affect various processes such as purchase decisions @xcite , recommendations @xcite , and brand-switch intentions @xcite . moreover, recent studies have also shown that cognitive appraisals directly and indirectly (via emotions) influence pre-purchase intentions and post-consumption behaviors @xcite . considering emotions and their antecedents (i.e., cognitive appraisals) can offer more holistic models of consumers' thoughts, emotions, and behaviors. online product reviews provide implicit evidence of consumer behavior. they include descriptions of subjective feelings and non-emotional factual recounts @xcite . analyzing the emotional content of reviews can infer reviewer behaviors. however, these reviews may not reflect the necessary emotions and cognitive appraisals to study the psychological link between the two. public datasets of product reviews do not consist of rich emotional labels (emotions and cognitive appraisals) to model the associations between consumer language and emotional constructs. furthermore, given that most datasets comprise annotations inferred by independent annotators, it is worthwhile to pause and consider whether they offer an accurate first-hand representation of consumer appraisal. in other words, are public datasets of product reviews suitable to study emotions, cognitive appraisals, and subsequent consumer behavior in the context of consumption? to address these research gaps, we conducted a pilot study of which kinds of writing prompts elicit high-quality text of emotions and cognitive appraisals and subsequently used the writing prompts derived from the pilot to collect the full dataset. our dataset, called perception and experience of appraisals and consumption emotions in reviews (peace-reviews). peace-reviews offers review texts annotated with rich first-person emotional variables such as cognitive appraisal dimensions and emotional intensity. this work contributes to the extant literature that overlaps between the disciplines of emotion science and natural language processing by enhancing the understanding of linguistic features of appraisals reflected in the emotional expression of autobiographical review text. the key contributions of our work are as follows: ## theoretical framework one of the primary theoretical perspectives of understanding and predicting emotions in people is cognitive appraisal theory, as studied in psychology. this theory posits that emotions are elicited from one's evaluation of situations that are of personal significance to one's well-being @xcite @xcite @xcite @xcite . these evaluations, or appraisals, are done along a set of motivationally relevant variables. for example, if a person appraises a situation as 1) not consistent with their goals (goal conduciveness), 2) some other person is responsible for causing the situation (other-accountability), and 3) unfair (fairness), they will feel an emotion like anger. therefore, instead of just knowing what emotion a person is feeling, these appraisal dimensions are essential in understanding why such an emotion is felt by the person. thus, according to cognitive appraisal theories, how people appraise their situations results in specific emotions experienced. moreover, appraisals are proposed to be central to the emotion process and are considered to be antecedents of emotions @xcite . therefore, analyzing appraisals is essential in understanding what specific emotions get elicited in a situation and can also answer why a person is feeling the way he or she is feeling @xcite . autobiographical writing, such as reviews, reflects and communicates psychological aspects about the writer such as one's thoughts and feelings @xcite . words or phrases used in such writings can suggest cognitive processes (e.g. how people think about themselves or objects, their goals, and intentions), affective processes (e.g. how people are feeling), and even personality and traits @xcite . moreover, previous research has found that stronger emotional responses experienced in such consumption situations often lead to the posting of product/service reviews @xcite . thus, reviews are typically informative enough to reflect features of emotions and appraisals that underlie the text and therefore provide a suitable context for modeling consumer behavior. in summary, motivated by how emotions and cognitive appraisals are critical in understanding consumer behavior, our broad aim is to analyze these emotional constructs in review text. the following research questions guided our dataset curation: although there are some annotated emotional textual datasets with third-person appraisal dimensions @xcite @xcite , these are perceived appraisal ratings by third-person readers rather than first-person appraisals that are personally evaluated by the writer who experienced the situation. thirdperson ratings are conceptually different from firstperson ratings as they measure emotional perception rather than experience. the former ratings are indications of how a third-person (i.e. a reader) interprets a situation that someone else (i.e. a writer) is experiencing, and then from these interpretations, reason how the first-person writer is feeling @xcite . these thirdperson appraisal ratings might differ from how the writers personally appraise the situation in terms of the kinds of appraisal dimensions used and the intensity of the appraisal ratings. thus, the existing datasets cannot address questions relating to first-person emotional experiences. datasets of products/service reviews are primarily investigated in sentiment analysis where the outcome labels are either 1) positive, negative, or neutral, or 2) an integer rating scale of 1 to 5. the amazon product reviews dataset @xcite , and the yelp restaurant review dataset @xcite are widely utilized in studies that analyzed sentiments of product/services. other review datasets include restaurant @xcite @xcite @xcite , and hotel reviews @xcite . therefore, the lack of a product/service review dataset containing emotionally rich variables such as cognitive appraisals and emotional intensity is a primary motivation for the proposed dataset. such a dataset can be useful not only for researchers from computational linguistics and consumer psychology to model linguistic correlates of emotional constructs and behaviors in the domain of consumption but also for practical applications in the fields of marketing, advertising, and business analytics to understand how specific product/service designs can be improved to result in targeted emotions. ## research design this study aimed to curate a dataset of text responses labeled with rich emotional variables such as appraisal and emotional intensity. therefore, we conducted a pilot study to evaluate which kinds of writing prompts elicit high-quality text of emotions and cognitive appraisals. in an online panel, qualified participants were assigned to one of four conditions eliciting different kinds of detail about their cognitive appraisal of a product they purchased. after evaluating the data collected for its emotional and cognitive richness through automatic content analysis methods, we subsequently used the most effective writing prompts to collect the full dataset. ## pilot study the purpose of the pilot study was to evaluate the quality of emotional text responses elicited by different writing prompts. its protocol design is illustrated in figure 1 . therefore, we employed a 2 x 2 factorial between-subjects design to examine whether a questionnaire format might yield more detailed emotional responses or whether a review format is as emotionally detailed as the questionnaire format. the two factors that are manipulated are the presence of emotion prompts (emotion prompts or no emotion prompts), and 2) response format (review or questionnaire). this design was motivated by the concern that the review format responses might emphasize product-specific details instead of emotional reactions even when we ask participants to write about the emotional aspect of it. another primary concern is that the review format responses might not contain text pertaining to appraisal dimensions. therefore, the four conditions are: in the emotion treatment, participants were tasked to recall when they felt either the emotion of joy or anger when they used or experienced an expensive product/service they purchased. these emotions are chosen based on the literature on consumption emotions commonly experienced during the usage of products @xcite . therefore, the 'presence of emotion prompts' condition asks the participants to think about an experience that involves a particular emotion. in the no-emotion treatment, participants were tasked to think about a time that they have used or experienced an expensive product/service that they purchased without any emotional prompts. next, we manipulate how participants are required to express themselves. in the questionnaire treatment, participants will answer a series of open-ended questions about their experience with the product that they have thought about. we only included four specific open-ended appraisal questions (goal relevance, goal conduciveness, expectedness, and pleasantness) due to the feasibility of data collection (i.e. eliciting responses for a large number of appraisals might result in poor response due to fatigue) although we think that other appraisal dimensions might be relevant in the context of consumption (see finally, in the review treatment, participants were tasked to write a review about their experience of using the product that they have thought about as if they were going to publish online. after completing one of the four conditions, participants will also complete a validated cognitive appraisal and emotion questionnaire that was adapted from previous studies @xcite @xcite , where they will be asked to rate how they appraised and felt when describing the experiences on a set of 20 appraisal (see we created a qualtrics survey that reflects these conditions with the respective questions and prompts. once participants have consented to participate, they will be randomly assigned to one of the conditions. ## main study based on the results of the pilot study, we observed that indeed the structured nature of the questionnaire format elicits richer emotional responses that contain appraisal dimensions that could be used to analyze such text and so we chose the questionnaire format as the writing prompts to elicit appraisal and emotional text responses in the main study. in terms of the presence of emotion prompts, although there were generally no significant differences between questionnaire conditions with or without emotion prompts (with the exception of the cognition dimension of liwc22), we decided to go along with the presence of emotion prompts as it allows us to determine the ground truth emotion labels for each participant's experiences. prior to the main data collection, we conducted a screening study that asked one questionwhether participants have purchased expensive products/services recently. participants who answered yes are then eligible to participate in our main study. more broadly, this question serves as a proxy for products/services that are personally important to the participants, which we are interested in as this class of products has the highest likelihood of experiencing an emotion while using it @xcite . during the main data collection, participants followed the framework of the presence of emotion prompts and questionnaire format. that is, participants were tasked to state the dominant emotion experienced when they were using the expensive product/service. unlike the pilot study where the emotion prompts involved only the emotion of joy or anger, participants were instead asked to state the dominant emotion (amongst the eight emotions listed and an 'other' option). following this, they answer the same set of questions as in the pilot study in the text regarding their experiences and appraisals with respect to the dominant emotion that they have stated. additionally, we also added a trait measure called preference for intuition and deliberation scale @xcite to measure individual differences in the tendency to utilize a particular decision mode. this trait measure is highly relevant in the context of consumer behavior to understand the decision-making process underlying the consumers' experiences. finally, they answered the same validated 8-emotions and 20-appraisals questionnaires in the pilot study. we also collected other responses such as the product/service that the participants have recalled and the associated cost, intention to buy the product again, and intention to recommend the product to others. this process was then repeated again and participants were tasked to recall another experience of using an expensive product/service that is opposite in terms of valence (positive or negative) to the experience that they have previously stated. that is, if the participants recalled a positive experience previously, they were now tasked to recall and write about a negative experience. this is typically done in appraisal studies and allows us to obtain another set of responses @xcite . ## conclusion we curated a text dataset that described autobiographical expressions of participants' experiences when using an expensive product/service, annotated with a large number of self-reported psychological variables such as cognitive appraisals, emotions, traits, and future consumer behavior. we experimented with various writing prompts in the pilot study so as to determine which method elicits the best expression of emotional and cognitive appraisals of consumption experiences. furthermore, we conducted experiments with state-of-theart language models to provide baseline evaluations of appraisal and emotion classifications. although we primarily examined appraisals and emotions in this paper, future research could utilize the other variables in this dataset such as decisionmaking traits, and post-purchase consumer behavior (e.g. intention to purchase and product again, and recommending the product to someone else) to model how appraisals and emotions interact with these variables. in summary, the peace-reviews dataset enables researchers to study the correlates between expressed linguistic features and cognitiveemotional processes and affords a comprehensive understanding of emotional experiences in the domain of consumption and product/service reviews.