Predicting Empathic Accuracy from User-Designer Interviews (2023)

Abstract: Measuring empathy as a natural language processing task has often been limited to a subjective measure of how well individuals respond to each other in emotive situations. Cognitive empathy, or an individual’s ability to accurately assess another individual’s thoughts, remains a more novel task. In this paper, we explore natural language processing techniques to measure cognitive empathy using paired sentence data from design interviews. Our findings show that an unsupervised approach based on similarity of vectors from a Large Language Model is surprisingly promising, while adding supervision does not necessarily improve the performance. An analysis of the results highlights potential reasons for this behaviour and gives directions for future work in this space.

## introduction user interviews are an important part of modern product development frameworks as meeting user needs defines success in engineering design. typically these interviews, conducted between a potential user and a designer, are used to either gather knowledge about the user's problem or their experiences with current products, or to gain feedback on the product as it is being developed. however it remains a question as to whether these processes improve user understanding and lead to good outcomes, and the factors which contribute to these. one such factor regards whether or not designers are able to understand the user during these interviews -this is referred to as 'empathic understanding' @xcite . if a designer is able to grasp the user's experiences and thoughts, does this necessarily lead to better outcomes? to answer this question, chang-arana et al. in this paper, we propose to use natural language processing (nlp) approaches to automate the measurement of empathic understanding in interviews, especially due to the advent of out-of-thebox large language models (llms). this can not only streamline the process of analysing interviews in engineering design but also provide a test bed for automatically measuring empathy in conversations, an open problem in nlp. automated evaluation in this way may be useful more broadly in other fields, where empathy is highly valued, such as teaching. ## background and related work work measuring empathy in nlp has been explored, with open domain dialogue data such as @xcite existing as benchmarks for the task. much work has been done detecting how empathy is expressed in dialogues in a variety of contexts from healthcare @xcite in both speech and text, as well as in online communities @xcite . however the theme of these works is primarily focused on empathy in the emotional sense. that is, there is a large focus on studying how individual express empathy towards others through dialogues. a common example is choosing the 'right' emotional words to comfort another individual in distress, guiding work in generating empathetic responses @xcite . on the other hand, in a review, @xcite points out that tasks revolving around cognitive empathy are not as prevalent in the nlp literature. while empathy is a complex concept, loosely we can distinguish between emotional empathy as processing and responding to another's emotions effectively, and cognitive empathy as being able to infer their thoughts in a broader sense @xcite . one may be able to identify how another person is feeling and act appropriately, but may not necessarily know what the other person is thinking. a key distinction between our work and more common tasks involving 'empathy' in nlp is that we primarily try to measure cognitive empathy from pairs of thoughts. ## data the dataset was collected from user-designer interview experiments in @xcite . figure 1 gives an overview of the annotation process for empathic accuracy ratings. each interview was recorded in video format. interviewees were played back the recording and were asked at any time to pause the video and write down their thoughts. the same recording was played back to the interviewer, where they were tasked with guessing the user's thoughts in those moments. in total, 46 users were interviewed by 3 designers, although not every user and designer were paired. each instance of the dataset is indexed by a (user, designer) pair and contains a timestamped sentence pair -one being the user's thoughts at that particular moment, and the other being a guess of the user's thoughts by the designer at the same moment. each pair is rated by 3 judges with a three-level likert scale ({0, 1, 2}), with the average taken as a score indicating the accuracy of the designer's prediction. the designer is also tasked with predicting the user's self-evaluated tone of speech at that moment. in this work, we focus on automating the rating stage (stage 4 in figure 1 ). each instance contains a sentence pair (a user thought paired with a designer guess) as the input and the averaged rate given by the judges as the output. figure 2 shows two such instances as an example. the inputs were preprocessed by removing text indicating the subject ("he/she/i was:") at the start of the string. we also rescaled the ratings to the unit interval. of user thought and designer guess. experiments with in-domain supervised models were performed using 10-fold cross validation. for a sound comparison, we use the same 10-fold setup for the unsupervised and out-of-domain models, using only the testing folds for evaluation. unsupervised. our first approach does not employ any training: we calculate the cosine similarity between the two embedding vectors and report the result as the rating. in-domain supervised. here we employ a standard cross-validation procedure, using 9 folds as training data. our main approach finetunes a sbert regression model following the original "siamese" method from @xcite , which uses the cosine similarity between the embedding vectors as the regression output. in addition, we also employed sbert as a feature extractor and two off-the-shelf regressors as additional models: a gaussian process (rasmussen and williams, 2006, gp) with an rbf kernel and a multilayer perceptron (mlp). each input uses the concatenation of the sbert vectors obtained from the user thought and the designer guess, plus the vector obtained from their absolute difference. on average, each training set contains 400 pairs. out-of-domain supervised. finally, we also tested with a supervised approach trained on outof-domain data. the rationale is that the rating can be framed as a semantic textual similarity problem @xcite . this raises the question of whether we can employ existing sts data to create a good regressor without requiring any initial ratings for training. for these experiments, we used the widely available sts-b @xcite dataset, containing approximately 6000 pairs. we used the same models as in the in-domain experiments. ## results our main results are shown in the models trainedd on the out-of-domain sts-b data did not outperform the unsupervised approach for any regressors. we believe this is due to significant differences in the sts-b and the interview data. while both can be interpreted as sentence similarity, the pairs present in sts-b are much shorter and use simpler language, compared to the more complex sentences present in our dataset. while we were aware of this important domain difference, we still expected the performance to be better than the unsupervised approach, but our findings showed otherwise. it is important to note that a pearson score of 0.66 already demonstrates good prediction performance. performance improvements could be obtained by adding in-domain training data and further model tuning. however, these results are already promising from an application perspective and could potentially lead to a reduction in human labour for obtaining empathic accuracy scores. ## qualitative analysis here we will conduct further analysis on our data to understand the performance of our models under our task. we summarise three findings that could lead to further improvements in the prediction task. lack of non-textual context textual similarity tasks rely on the meaning and context within the sentence itself, but in our case did not contain the extra information that raters may have when scoring pairs of text. the thoughts are often written down in an ad-hoc and conversational manner, containing implied information around the topic or interview itself that is able to be inferred by the raters, but which models which rely on complete information fail to do. this causes a mismatch between true scores and predicted scores. our first instance in inconsistent points of view within our data it is often the case that the two pairs of text are written from two different points of view, resulting in sentences that may have similar content, but have different meaning. however they may still be rated highly because the designer, in their own writing, has effectively guessed the user's thoughts, even if they are not writing the thoughts from the perspective of the user. judge scoring our methods also tend to overestimate the scores in cases where the context or topic that both the designer and user are thinking of are the same, but the actual user text was different. for example, because the interviews were related to driving, both the user and designer wrote down thoughts related to driving, but these thoughts did not necessarily contain the same idea. in these cases, the human judges tended to more harshly assign scores of 0 whereas our system tended to provide a more soft assignment. this is a common problem of standard regression models, which are unable to predict extreme values outside a certain range. future work should carefully consider how to penalise the scores based on how the two sentences diverge in actual meaning. ## conclusion we introduce a novel task of predicting an individual's cognitive empathy as scored by their ability to predict, in text, the thoughts of another individual using a dataset from design engineering interviews. using this data we demonstrate the performance and limitations of current state of the art models on our task. our analysis shows that this problem poses unique challenges due to the unique structure and missing context of user written thoughts. initial directions for future work are based on our analysis in section 6. incorporating context from interview transcripts is an important direction, as well as improved regression models that can better predict extreme values. a more challenging, longer term goal is the prediction of empathic accuracy directly from interviews, without requiring user thoughts and designer guesses. this would effectively bypass stages 2 and 3 in figure 1 , drastically reducing annotation costs and potentially enabling real-time empathy feedback during an interview. we believe this is a much harder problem, but that nevertheless would lead to benefits to not just our task in engineering design, but lead to novel advances in other tasks in nlp.