Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners (2024)

Abstract: Unsupervised clustering of student responses to open-ended questions into behavioral and cognitive profiles using pre-trained LLM embeddings is an emerging technique, but little is known about how well this captures pedagogically meaningful information. We investigate this in the context of student responses to open-ended questions in biology, which were previously analyzed and clustered by experts into theory-driven Knowledge Profiles (KPs).Comparing these KPs to ones discovered by purely data-driven clustering techniques, we report poor discoverability of most KPs, except for the ones including the correct answers. We trace this ‘discoverability bias’ to the representations of KPs in the pre-trained LLM embeddings space.

## introduction classifying students into behavioral or cognitive profiles using unsupervised cluster analysis techniques is a common application of machine learning to educational data @xcite @xcite @xcite . recently, there has been a growing interest in applying this methodology to textual student responses that are decoded using pre-trained large language models into vectorized embeddings in semantic spaces @xcite @xcite . the operational appeal of this approach is that it minimizes the need for expert knowledge, which is costly to inject through human labeling procedures @xcite @xcite . however, the validity of patterns discovered this way depends on the ability of the embeddings to maintain the pedagogically meaningful information that existed in the original, textual representations of responses @xcite and of the algorithmic method to discover them. evaluation of emergent profiles is often done in terms of the internal quality of the clustering, as data is usually not available to estimate the extent to which the discovered profiles align with a pedagogically meaningful representation of the responses. without such an evaluation, a loss of important information can be overlooked, potentially leading to sub-optimal educational decisions that rely on this analysis @xcite . to investigate whether this hypothesized risk manifests in real-life educational context, we utilize student answers to two constructed response questions in high school biology. the data was previously analyzed by a team of biology education researchers and experienced teachers, and graded according to a theory-driven detailed analytic rubric that is based upon the causal-mechanical explanation framework @xcite . the rubric contained 10 (item 1) or 11 (item 2) binary categories, each checking for the occurrence of a specific key piece of information in the response. using these human-generated binary vectors of length 10 (11), the responses were clustered using a kmeans algorithm into a set of 6 (7) knowledge profiles (kps) that were found by teachers to encapsulate specific patterns of errors. the validity of the kps was evaluated in several ways. first, human experts conducted a qualitative analysis to assess whether each kp captures a specific and distinct pattern of errors. second, we analyzed the results computationally, showing that i) the kps were consistent across the two items, namely, revealing the same type of conceptual errors; and ii) the learners tended to exhibit the same type of conceptual error (kp) in both items. third, we conducted an in-class formative assessment intervention study that provided automated guidance to students based on their kp, and showed significant improvement in their performance on a different prompt that measures the same conceptual knowledge. these analyses provided strong evi-dence that the kps capture pedagogically meaningful information (for full details, see @xcite . using these data, we are in a position to answer two research questions: rq1 what is the correspondence between clusters that are computed from pre-trained llm embeddings of student responses and theorybased kps? to preview the result, we find that two clustering techniques that are commonly used for such tasks (le quy et al., 2023) -kmeans @xcite and @xcite -largely fail to discover the kps though retrieval is somewhat better for the profile containing the correct responses. following up on this finding, we go 'upstream', to the pre-trained embeddings, and investigate: rq2 how well are the kps represented in the pretrained embeddings space? our results reveal a strong relationship between the quality of the responses in the profile (correct or various degrees of incorrect) and the shape and density of its embeddings-based representation. we refer to this relationship as an 'anna karenina principle' and tie it to the profile discovery failure we observed in rq1. the contribution of this work is twofold. first, it is the first to demonstrate the anna karenina principle in the context of pre-trained representation of student responses to open-ended questions. second, our results suggest that, in some cases, outof-the-box pre-trained llm embeddings may be a pedagogically unsound basis for profile discovery. ## data the data consists of 669 student responses to two open-ended items in high-school biology, collected anonymously from students in grades 10-12 from about 25 high schools of varied demographics and socioeconomic status (based on location) across israel. gender distribution was 70% females (typical to the gender distribution among high-school biology majors in israel). the items deal with the connection between respiration and energy in physical activity in the context of smoking (q1) and anemia (q2), taught as part of the core topic "the human body". the items were human-scored using a similar analytic rubric containing 10 (q1) or 11 (q2) categories @xcite . all rubric categories are binary, each targeting specific information that needs to be mentioned in a correct response, such as "the role of hemoglobin in oxygen transportation" or "changes in cellular respiration rate". the resulting binary vectors were clustered using kmeans; the clusters were analyzed by experienced teachers and ranked from 1 to 6 (q1) or 7 (q2) with larger numbers corresponding to clusters with more severe errors. we denote these clusters knowledge profiles, and index them from 1 (kp1) to 7 (kp7). see @xcite for a full description of the items and the assessment framework. the items and examples of student responses and their mapping into kps can be found in appendix 1. for the purposes of the analysis presented in this paper, all responses were represented using rich contextualized vectors -embeddings produced by a pre-trained large language model (llm). the llm being used, alephbert @xcite , is state-of-the-art for hebrew. it was trained on a large corpus of the hebrew language, including: twitter tweets, hebrew wikipedia, and the hebrew subset of the oscar @xcite dataset. alephbert has the same architecture as bert @xcite : 12 layers, 110m parameters, and 12 attention heads. it was trained on a 52k-word hebrew vocabulary on masked token prediction task, and on the hebrew language tasks: word segmentation, part-of-speech tagging, and full morphological tagging. it was further trained on the tasks of sentiment analysis and named entity recognition. ## methods to evaluate whether raw llm embeddings carry useful knowledge for unsupervised profiling of responses, we experimented with two common clustering approaches (le quy et al., 2023), kmeans @xcite and @xcite , which implement different clustering mechanisms. the first discovers convex-shaped clusters; its mechanism is centroid-based and applies an euclidean distance function. the second is densitybased and can be applied with various distance metrics, e.g., a metric induced by cosine-similarity, and the clusters may have various shapes. both approaches were used previously for profile discovery in constructed response data @xcite @xcite . experiments were conducted in python, using scikitlearn @xcite , sbert @xcite and pytorch @xcite . ## discussion and conclusion our data consists of 669 high-school student responses to two typical constructed response items in high-school biology. the responses were human graded according to an analytic rubric that is based on the causal-mechanical explanation framework @xcite , transforming each response to a binary vector that encodes the grading according to the rubric categories. previous work demonstrated that applying cluster analysis (kmeans) to these vectors, which result from a process that applies a theoretical assessment framework to concrete context by human experts, yields stable clusters that reveal pedagogically meaningful knowledge profiles, which were validated in several ways @xcite . (for more details, see section 3.) we reasoned that given the successful performance of pre-trained llms on a variety of education-related meaning-intensive tasks @xcite @xcite , and previous work that applied this specifically to profile discovery @xcite , we want to evaluate whether unsupervised profile discovery that is not aided by human knowledge works sufficiently well to be applied out-of the results of rq1 reveal that two distinct common unsupervised clustering techniques largely failed to discover the 'gold' kps from the pretrained llm embeddings. inasmuch as a weak relationship with the knowledge profiles was exhibited by kmeans clusters (ari of 0.12-0.19), our retrieval-based analysis per profile showed that kp1, the profile that captures the correct responses, was the most discoverable profile, with f1-scores of 0.60/0.67 (on q1/q2) for retrieving members of kp1 using the best-aligned emergent cluster. thus, had the emergent clusters been used as a basis for feedback, only the correct responses would have received pedagogically cogent feedback, since responses belonging to low-knowledge kps are all intermixed in the emergent clusters. this phenomenon was consistent across two items -q1 and q2 -that were analyzed separately. in an attempt to account for both the failure of overall profile discovery based on pre-trained llm embeddings and for the bias towards the correct responses exhibited by the emergent clusters, we turned 'upstream' to inspect how the kps are represented by the embeddings. we found that the lower the knowledge level of the profile, the less similar to each other its members are in the embeddings space. it is this property that we refer to as the anna karenina principle: analogously to tolstoy's observation that happy families are similar to each other whereas each unhappy family is unhappy in its own way, we see that the correct responses are similar to each other, whereas incorrect responses differ more from each other the more incorrect ('unhappy') they are (strong correlation of r 1 = -0.686, r 2 = -0.633 for q1/q2). one could say that tolstoy considered all unhappy families as an undifferentiated mass; presumably, if classified by their specific source of unhappiness (by family therapists, say), profiles would have probably emerged. in our case, the incorrect responses are grouped by teachers according to the type of problem they exhibit; however, within-profile similarities are still lower than those of correct responses and drift further apart the bigger the problems. the lower density of the poor-knowledge profiles may be one reason that inhibits their downstream discovery. further analysis suggests that the privileged status of 'happy families' (correct responses) extends beyond their higher density. we also found that while an average correct response is most similar to another correct response, an average incorrect response is closer in the embeddings space to a correct response than to a member of its own profile (tables 9 and 10 ). that is, in some sense, the correct responses are the center of the universe, whereas the incorrect responses drift around them in non-convex formations. the non-convexity of the lower-knowledge profiles may be another inhibitor of their downstream discoverability. taken together, the 'classic' anna karenina property and the strong version that puts the correct responses in the center suggest an explanation for both the overall discovery failure observed downstream and for the bias in favor of correct responses exhibited by the emergent clusters. based on our results, pre-trained representations may not lend themselves to making the necessary distinctions to support pedagogical decisions such as providing formative feedback that targets specific errors in student reasoning. in particular, our results show a case where the representations are not sufficiently nuanced to allow commonly used clustering methods to identify any error-based profiles, only the profile of the correct responses. since it is the students who gave the incorrect responses who are in most need of targeted formative feedback, the bias in favor of correct responses is especially counter-productive. thus, our results tell a cautionary tale about using emergent properties of student response data built over pre-trained embeddings without domain-and task-specific tuning, and without human supervision.