Leveraging a Cognitive Model to Measure Subjective Similarity of Human and GPT -4 Written Content (2024)

Abstract: Cosine similarity between two documents can be computed using token embeddings formed by Large Language Models (LLMs) such as GPT-4, and used to categorize those documents across a range of uses. However, these similarities are ultimately dependent on the corpora used to train these LLMs, and may not reflect subjective similarity of individuals or how their biases and constraints impact similarity metrics. This lack of cognitively-aware personalization of similarity metrics can be particularly problematic in educational and recommendation settings where there is a limited number of individual judgements of category or preference, and biases can be particularly relevant. To address this, we rely on an integration of an Instance-Based Learning (IBL) cognitive model with LLM embeddings to develop the Instance-Based Individualized Similarity (IBIS) metric. This similarity metric is beneficial in that it takes into account individual biases and constraints in a manner that is grounded in the cognitive mechanisms of decision making. To evaluate the IBIS metric, we also introduce a dataset of human categorizations of emails as being either dangerous (phishing) or safe (ham). This dataset is used to demonstrate the benefits of leveraging a cognitive model to measure the subjective similarity of human participants in an educational setting.

## introduction when humans categorize textual information, such as when giving recommendations or learning to categorize documents, we often use our personal subjective concepts to complete the task. one example of this is giving a recommendation of a funny book to a fiend, which requires not only our own subjective conceptualization of humor, but also an understanding of the similarities and differences between ourselves and our friends. while humans perform this task with relative ease, recommendation systems @xcite and educational tools @xcite typically do not have personalized measurements of subjective concepts @xcite , potentially hindering their efficacy @xcite . when these systems incorporate data from human judgements to determine subjective similarity, they typically do so by pooling together as many judgements from different people as they can, and aggregate their measurement @xcite . this approach relies on machine learning based methods @xcite , which can be effective from a machine learning perspective, since more data can mean improved document similarity metrics on average over large datasets @xcite . focusing on individuals annotations of documents has been explored in the context of domain specific knowledge such as biomedical research papers @xcite , or for specific context like document summarizing @xcite . however to date little attention has been given to the notion of individualized metrics of similarity that account for biases and constraints specifically, which are highly relevant for educational contexts @xcite . related to the domain of this work in particular, recent work has demonstrated a broad range of human opinions and levels of trust associated with cybersecurity concepts such as trusted execution environments @xcite . this highlights the need for individualized metrics that take into account experience in training tasks such as the anti-phishing training dataset used in this work. in this work, we propose a method for providing personalized metrics of subjective concepts that can determine the similarity between sets of text, with additional applications in selecting educational examples and providing natural language feedback. this is done by leveraging a cognitive model of human learning and decision making that can act as a digital twin to individuals, and predict their behav-ior and opinions on a wider set of stimuli. we focus specifically on students categorizing emails as being safe (ham) or dangerous (phishing) in a training setting to help users identify and defend against phishing email attacks. our proposed method for providing personalized similarity metrics of documents is compared to alternative methods using a dataset of a phishing education task experiment that we additionally present in this work. the dataset of human annotations of emails as being either ham or phishing is described in @xcite and was made publicly available on osf foot_0 . this dataset consists of human annotations of email documents that are either written by cybersecurity experts or a gpt-4 model, the emails shown to participants, and conversations between human participants and a gpt-4o model providing feedback to students. in total this dataset represents 39230 human judgements from 433 participants making decisions while observing a set from 1440 gpt-4 or human generated emails, as well as 20487 messages between human participants and the gpt-4o teacher model. this type of learning task represents a serious challenge for traditional methods of adjusting document embedding similarity metrics to conform to human behavior, such as embedding pruning @xcite or embedding weighting @xcite . this is because these approaches typically rely on a large amount of annotations collected from many participants who are expected to have the same knowledge level throughout the annotation process. instead, we are interested in measuring the subjective similarity of documents as participants learn the document annotation task in a training setting. to do this, we employ a cognitive model that can predict the learning trajectory of each individual participant as they learn to correctly annotate these documents. ## background: cognitive model the cognitive model used in this work to predict the subjective similarity of human participants decisions on unseen emails relies on instance based learning theory (iblt) @xcite . one of the benefits of employing ibl models over alternatives like reinforcement learning is that they base their predictions on the full history of participant experience as well as the impact that limitations like memory size and decay can have on decision making. ibl models have been applied onto predicting human behavior in dynamic decision making tasks, including binary choice tasks @xcite , theory of mind applications (nguyen and gonzalez, 2022), and practical applications such as identifying phishing emails @xcite , cyber defense @xcite , and cyber attack decision-making @xcite . ## phishing email categorization dataset the first component of this dataset is human behavioral experiment data from a study on human categorization of emails. this experiment compared human document annotation when categorizing emails as phishing (dangerous) or ham (safe). the conditions of this experiment varied depending on the email author (human or gpt-4) and style (plain-text or gpt-4 stylized). there was also a comparison of the method of selecting emails to show to participants, either randomly selected, or chosen using an ibl model (ibl or random). finally, we compared the type of feedback given to participants between positive and negative point feedback and a natural language conversation with an gpt-4o chat-bot (points or written). this experiment included 10 pre-training trials without feedback, 40 training trials with feedback, and 10 post-training trials without feedback. during all trials, participants made judgments of emails as phishing or ham and indicated their confidence in their judgment as well as which action out of 6 possibilities they would select after receiving the email. we recruited 433 participants online through the amazon mechanical turk (amt) platform. participants (150 female, 280 male, 3 nonbinary) had an average age of 40.3 with a standard deviation of 11.02 years. participants were compensated with a base payment of @xmath012-15 bonus payment depending on performance and the length of the experiment. this experiment was approved by the carnegie mellon university institutional review board, and the study was pre-registered on osf. the second component of this dataset is the emails shown to participants, which were either written by human cybersecurity experts, a gpt-4 model working alone, or a combination of human and gpt-4 model work. 360 base emails written by human experts were used to form three additional versions of these base emails. these alternative versions included a 'human-written gpt4styled' version that used the email body written by human experts, the 'gpt4-written and gpt4-styled' version that was fully rewritten by gpt-4, and the 'gpt4-written plaintext-styled' version that stripped the html and css styling applied by the gpt-4 model. these emails as well as the original prompts to generate them are included in dataset on osf. 4 methods of measuring similarity 4.1 human subjective similarity llm embeddings have been suggested as a method of measuring human similarity judgements (bhatia and aka, 2022), while also capturing the wide range of individuals similarity measures. additionally, comparisons of llm behavior have also demonstrated human-like variability @xcite , suggesting these embeddings could be useful for capturing the variety of human similarity judgements. cognitive models that rely on representations of information from gai models have been shown to adequately account for the wide range of human behavior @xcite . however, for these methods to function properly there must be a connection between the way that similarity is measured in humans and gai models. previous applications in applying visual gai models onto representing decision-making tasks in humans relied on the close connection to these model representations and human representations @xcite . for this reason, we devised a metric of human subjective similarity that takes into account the confidence of document categorization as well as the time it takes participants to categorize documents. to determine the human subjective similarity measure, we use the category of human participant annotations, their annotation confidence, and the speed of their annotation. for accuracy and confi-dence, a higher value in our human subjective similarity metric signifies that participants were more likely to categorize an emails as being a member of that group, and more confident in their categorization. for reaction time, a lower value indicates that the document is more immediately obviously a member of a group and thus has a higher similarity to other members of that group. the result is a value that is difficult for a standard similarity metric to account for, as the annotations made in this dataset occurred in a learning setting where earlier trials had less accuracy, which also impacted reaction time and confidence. the reaction time and confidence weighted subjective similarity of an email x is given by multiplying the probability of a human participant categorizing that email as category c giving cs(x|c) = p(c|x)r(c|x)c(c|x). where p(c|x) is the probability of categorization, r(c|x) is the reaction time normalized to between 0 and 1, and c(c|x) is the confidence additionally normalized to between 0 and 1. the soft-max of this cs(x|c) value is the resulting similarity metric, with the equation shown in the supplementary materials foot_1 . ## ensemble similarity the final comparison method is based on using an ensemble of each of the previous similarity metrics, weighted to maximize the similarity to the average of the human subjective similarity metrics. this approach has been applied to document matching for patent documents @xcite , which requires the similarity of document embeddings be calculated to determine a match. this ensemble approach has the highest kde log probability score of any individual method by itself, at a value of -812.23. looking at the kde distributions above and to the right of the scatter plot in 6 demonstrates the high similarity of the ensemble similarity metric (light blue and light orange) and the human participant similarity metric (blue and orange). while this method is effective at resulting in a similarity metric that closely matches the average over all participants, it still does not fit as well to individual participants, as will be shown in our proposed model. ## instance-based individualized similarity (ibis) to determine an individual participant's metric of similarity, we employ an ibl model that is serving as a digital twin of the participant. the result in an instance-based individualized similarity (ibis) metric. the benefits of ibis are in the ability to predict human judgements on unseen documents or feedback from recommendations, and enhance measurements of subjective similarity. importantly, these predictions of human behavior are not merely relying on a separate machine learning based technique, but rather a cognitive model that is inspired by the human cognitive mechanisms underlying decision making and thus able to account for natural biases and constraints in humans. predictions of instance-bases individual similarity are done using an ibl model that is currently serving as a digital twin with the same experience as an individual participant. using this we determine the value that the ibl model assigns to predicting a category c as v k (c|x), or the value the ibl model assigns to choosing option c as the category of document x. then, we can divide this value by the same categorization value assigned to each alternative categorization of the same document. this results in the ibis metric which can be calculated after each decision is made by a participant, pseudocode for the ibis algorithm, the code-base for the ibis method including all comparison methods, data, and scripts to generate similarity measures and figures is made available foot_2 . ## case study of ibis: individuals in phishing email education dataset previous comparisons of similarity metrics and human participant behavior compared the average of human performance. to highlight the benefits of the ibis method, we replicate these calculations with one individual from the experiment. here, the individual similarity of phishing and ham emails is based only on a single individuals categorization, confidence, and reaction time in their judgement. these graphs are shown for illustration in figure 8 , with the average accuracy of logistic regression of similarity metrics predicting individual participant similarity metrics reported in an important aspect of individual similarity comparisons of the ibis method is that it can compare emails that were not originally presented to an individual, meaning there are more embedding similarities used in the logistic regression and kde score calculation. this comparison demonstrates the benefits of using a cognitively inspired method of modeling human participant decisions making that takes into account biases and cognitive constraints. the results is a prediction of behavior that can accurately fill in the gaps of unseen elements of the dataset that have not been observed by a participant. this method more accurately predicts the subjective similarity of participants. importantly, this is done while initially limiting the cognitive model to observing a single decision made by these participants, and increasing this data as the participant makes more decisions. this is important for the functioning of the ibl model as using too many instances in memory can slow compute performance. the final comparison shown in the right most columns of ## discussion many applications of llms are interested in tailoring use cases to individuals, even when little information is known about that individual. while many approaches of individualization exist but have typically relied on advanced machine learning techniques. the method proposed in this work is relatively simple from a mathematical perspective, though there is a strength in its reliance on theories of cognition that underlie human learning and decision making. the result is a simple to understand and easy to implement method of calculating similarities of unseen documents using a cognitive model, which can augment datasets that contain only a small number of decisions. the general method described here, of augmenting subjective similarity metrics with predicted decisions from a cognitive model, could be applied onto various other scenarios. this includes settings that leverage representations formed of visual information such as β-variational autoencoders @xcite , which have been related to biological representation formation @xcite . overall, we believe that this method is useful for any application where the experience of end-users impacts future decisions. for instance, in visual learning settings vaes have been integrated with cognitive models to predict human utility learning of abstract visual in-similarity kde score kde score regression metric average participants individuals accuracy semantic similarity @xcite -1127.69 -37.69±1.19 0.46±0.11 cosine similarity @xcite -2097.40 -47.26±2.27 0.52±0.10 embedding weighting @xcite -847.56 -29.28±2.32 0.86±0.14 embedding pruning @xcite -846.20 -30.39±2.76 0.86±0.04 ensemble similarity @xcite -812.23 -28.64±3.28 0.89±0.12 ibis (proposed) in particular, conceptnet @xcite has previously been integrated into a cognitive model for question answering @xcite . future research should investigate how additional uses of llms in integrations of cognitive models can aid in educational settings. overall, the results in this work demonstrate the usefulness of cognitive models in serving as digital twins to human participants. leveraging these models and integrating their results into large language model techniques has the potential to make measurements from these models more cognitively grounded. while there are existing methods of incorporating human behavior through the use of large datasets collected from many participants, these do not necessarily account for biases and constraints. the method proposed in this work takes these features of human learning and decision making into account in developing a similarity metric. ## limitations the semantic similarity metric suffered from the sparsity of semantic categories in ham emails, additional annotations could raise the performance of this metric and can be explored in future work. however, this ensemble method was partially responsible for the high kde score of the ensemble method, as it allowed for an integration of both semantic information and embedding similarity. our ibis method still outperformed the ensemble method suggesting that this ensemble alone does not address the issues of alternative methods. one limitation inherent in ibl cognitive models is the time requirements to compare the current instance to all instances in memory. this may make the proposed model unsuitable for applications that rely on large datasets of individual behavior. however, methods in instance compression exist for ibl models @xcite . in this setting, we were able to predict individual participant's decisions fast enough that this was unnecessary. the specific application we investigated is somewhat unique in that it is based on training human participants to make categorization judgements of textual information of one of two categories. additionally, the task of annotating whether an email is phishing or ham relies heavily on a small number of features within the email. namely, if an email contains a link that redirects to a nefarious website, or requests personal information, then it should be labelled as phishing. while students rely on many queues to make their judgements, the annotation is in reality simple. future work in the area of learning subjective similarity metrics should expand into more complex domains.