Beyond Text: Leveraging Multi-Task Learning and Cognitive Appraisal Theory for Post-Purchase Intention Analysis (2024)

Abstract: Supervised machine-learning models for predicting user behavior offer a challenging classification problem with lower average prediction performance scores than other text classification tasks. This study evaluates multi-task learning frameworks grounded in Cognitive Appraisal Theory to predict user behavior as a function of users’ self-expression and psychological attributes. Our experiments show that users’ language and traits improve predictions above and beyond models predicting only from text. Our findings highlight the importance of integrating psychological constructs into NLP to enhance the understanding and prediction of user actions. We close with a discussion of the implications for future applications of large language models for computational psychology.

## introduction natural language processing (nlp) tasks involve predicting outcomes from text, ranging from the implicit attributes of text to the subsequent behavior of the author or the reader. recent research suggests that user-level features can carry more task-related information than the text itself @xcite , but these experiments have been conducted in a limited scope. other studies have explored how the linguistic characteristics of text, such as its politeness or the use of discursive markers, may predict subsequent user behavior @xcite . yet, these studies offer unimodal perspectives of users through the text they author and lack rich annotations of other user attributes, such as their cognitive and psychological traits. such data would be especially useful in applied nlp tasks, such as in the context of online reviews, to better contextualize and predict outcomes related to purchase behavior and product recommendations. in this study, we focus on cognitive appraisal theory, one of the primary theoretical frameworks in psychology to understand emotional experiences and how they are elicited (antecedents). central to cognitive appraisal theory is the proposition that emotions are not merely spontaneous reactions but are the result of intricate cognitive evaluations conducted across multiple dimensions of psychological motivation that are of personal significance to one's well-being, as discussed by seminal works in the field @xcite @xcite . people interpret-or appraise-situations along various dimensions, and the specific manner in which people appraise their situations characterizes the particular emotions they feel. for example, if a consumer evaluates a restaurant experience as slow (goal inconduciveness), the server was specifically being rude to them (unfair), and blames the waiter for such an experience (accountabilityother), then the consumer might feel an emotion like anger. our empirical investigation specifically targets the nuances of purchase behavior, guided by a focus on two critical dimensions as illuminated by cognitive appraisal theory: • cognitive appraisals: the multifaceted evaluative processes through which consumers engage with and interpret their interactions with products, including, but not limited to, the novelty and pleasantness of the consumer-product encounter @xcite ). ## dataset and variables we used the peace-reviews dataset @xcite , a dataset of 1,400 author-annotated product reviews describing people's emotional experiences of using an expensive product/service. to our knowledge, this is the only reviews dataset annotated with a large number of first-person emotional and behavioral intentions variables. existing emotion text datasets are usually annotated with only a subset of these variables, without the inclusion of any behavioral intentions ratings @xcite . most importantly, existing emotion text datasets are typically annotated with third-person annotations (i.e., raters rate text written by other people), where such annotations might not correspond to the writers' first-hand experiences @xcite . in the peace-reviews dataset, each review was annotated with first-person emotions, cognitive appraisals, and pcb ratings, which makes the dataset exceptionally relevant in comprehensively modeling consumers' first-hand emotional experiences and behavior intentions. our multi-task framework incorporates the following inputs: ## experiments see figure 1 for a visual representation of all models. we fine-tuned the bert-base model @xcite for models requiring input text. we trained feed-forward neural networks (ffnn) for models that require appraisal and emotion ratings as inputs. since pcbs are rated on a 7-point likert scale, we segment each rating into low (1-2), moderate (3-5), and high (6-7) and define it as a three-way classification task (see appendix b for the distribution of classes). for multi-task models where appraisals and emotions are outcome variables, we defined a multi-label binary classification task for emotion ratings, where we segment each rating into low (1-4) and high (5-7). this segmentation represents the presence or absence of emotions experienced by the participant in the situation, where only emotions that are felt with high intensity are considered to be present. we define a multi-output classification task for appraisals where we segment each rating into low (1-2), moderate (3-5), and high (6-7). the segmentation of appraisal ratings in this manner is typical in emotion research @xcite . we conducted 5 repetitions for each model and obtained the means and standard deviations of the accuracy and f1 scores. implementation details are in the appendix c. baseline models. three models serve as the baselines. we run separate models to predict pcbs for each modality m i , where @xmath0, appraisals, emotions]. we would like to observe which modality performs best in predicting pcbs. constrained models. we implemented three models. the first two models use the bert model fine-tuned on the reviews to predict either the ap-praisal or emotion ratings, and the resulting embeddings are then used to predict pcbs. the third model uses the bert model fine-tuned on the reviews to predict appraisals, subsequently uses these appraisal embeddings to predict emotions, and finally uses the resulting emotion embeddings to predict pcbs. according to emotion theory, this follows where appraisals are deemed to be antecedents to emotions, resulting in behaviors @xcite . they are termed constrained because the intermediate variable (appraisals or/and emotions) serves as a bottleneck that reduces the textual dimensions to a much lower dimension in predicting pcbs, compared to directly predicting pcbs from text. multi-modal models. we implemented three models. the first two models predicted pcbs using review text + m i , where @xmath1, emotions]. the third model predicted pbs from all three modalities. the embeddings of the modalities are concatenated to predict pcbs. this modeling approach is chosen for its capacity to assimilate psychological variables alongside linguistic features. the results allow us to compare whether ratings combined with review text help improve performance predicting pcbs. multi-task models. we implemented three models. for the first two models, the review texts are used to predict the pcbs and r i , where @xmath2, emotions], simultaneously. moreover, the embeddings of r i are used to predict pcbs by concatenating with the text embeddings. the final model, termed 'theoretical model', uses the review text to predict appraisals, emotions, and pcbs. the resulting embeddings from each modality are then concatenated to predict pcbs. additionally, we also used the appraisal embeddings to predict emotions. overall, this model is based on consumer and psychological theories. we would like to validate whether such a computational model consisting of the variables and their theoretical links has predictive utility in the context of language. the poorest results came from constrained models, likely due to the reduction of text embeddings to a lower dimensional appraisal and emotion feature space and resulted in the lost of information in predicting pcb. the integration of different modalities (multimodal models) did not enhance the performance as expected, indicating that unique information from each modality may not be additive for pcb prediction. nevertheless, some multi-modal models offer a slight edge in accuracy and f1 scores compared to the baseline text -> pcb model as observed for the results of the text + appraisals -> pcb model in predicting intention to promote, and text + emotions -> pcb and text + appraisals + emotions models in predicting intent to repurchase. for the multi-tasks and theoretical models, for the prediction of intent to promote, the theoretical model and the text -> pcb + emotions models outperformed the constrained, multi-modal and baseline text -> pcb models. in predicting intent to repurchase, the performances of the multi-tasks and theoretical models are similar to the constrained and the baseline text -> pcb models but did not perform better than the multi-modal models. this suggests that combining appraisals and emotions based on theory might not be optimal in predicting intention to repurchase compared to merely combining the features of the text, appraisals, and emotions. in general, for intention to promote, the multitask and theory-informed models performed modestly better than the rest of the models (except for the appraisals -> pcb model), likely due to their structured integration of appraisal and emotional constructs. however, for intention to repurchase, the two multi-modal models (excluding text + appraisals -> pcb) performed the best (except for the appraisals -> pcb model). overall, our results affirm that incorporating appraisal and emotional considerations generally enhances pcb prediction and supports the validity of cognitive appraisal theory in informing multi-task learning approaches. word attributions and explainability. we implemented the integrated gradients method to obtain the word attributions to explain the predictions @xcite . the visual depictions in figure 2 showcase word attributions corresponding to high and low instances of intentions to promote or repurchase, respectively, predicated upon our baseline text -> pcb model. the word attributions underscore the integral role of the emotionallycharged lexicon -'enjoyment,' 'disappointing' -and cognitive appraisal terms -'unexpected,' 'important,' and 'consistent' -in influencing the predictive outcomes of our bert-based model. the first two rows indicate that the model's reliance on affective language is pronounced, indicating a robust association between sentiment-laden words and positive intention to promote. in contrast, the word-and phrase-associations with intention to purchase illustrate a less pronounced correlation. we can infer that emotionally reso- negative intention positive intention intention to repurchase intention to recommend nant words seem more decisive in predicting the intention to promote, while a blend of cognitive appraisal and emotional language informs purchase intentions. this distinction may be crucial for refining the predictive efficacy of sentiment analysis models in consumer behavior contexts. finally, the figures highlight the errors in how non-cognitive, non-emotional words (e.g., 'florida,' and 'hotel') are correlated with pcbs. overall, our results are consistent with the findings that emotions and appraisals have significant links to pcbs @xcite . the analysis of word attributions in our models sheds light on the cognitive processes underpinning specific emotional reactions and behavioral tendencies. therefore, fine-tuning transformer models with appraisal and emotional variables and identifying linguistic features of such variables can potentially improve the prediction of pcbs. future studies could implement models that learn these variables simultaneously in a multi-task framework, thereby predicting pcbs. ## conclusion many nlp tasks focus on predicting user behavior, and enriching text-based models with user and social contexts is increasingly necessary. this work emphasizes the increasingly prominent role of cognitive and emotional signals in behavioral prediction. consumption emotions act as adaptive signals of how we evaluate how the use of products/services affects our well-being, which subsequently triggers future actions to either promote positive emotions (e.g., repurchasing or promoting to others) (white, 2010) or reduce negative emotions (e.g., complaint behaviors) @xcite . to our knowledge, the current work is the first to construct models grounded on psychological theory to model real post-consumption decision-making processes, and we find empirical support for these associations. more broadly, our study offers a novel methodological approach to study psychological variables in the context of empirically validating theoretical relationships within review texts-a domain previously unexplored beyond the confines of traditional survey methods. our work finds variance in the importance of these appraisals across tasks, raising important practical considerations for designing future approaches to behavioral prediction.