Cluster-Norm for Unsupervised Probing of Knowledge (2024)

Abstract: The deployment of language models brings challenges in generating reliable text, especially when these models are fine-tuned with human preferences. To extract the encoded knowledge in these models without (potentially) biased human labels, unsupervised probing techniques like Contrast-Consistent Search (CCS) have been developed (Burns et al., 2022). However, salient but unrelated features in activation space can mislead these probes (Farquhar et al., 2023). Addressing this, we propose a cluster-normalization method to minimize the impact of such features by clustering and normalizing activations of contrast pairs before applying unsupervised probing techniques. While this approach does not address the issue of distinguishing between latent knowledge and that portrayed by a simulated agent—a major issue in the literature of eliciting latent knowledge (Paul Christiano and Xu, 2021)—it still significantly improves the accuracy of probes in identifying the intended knowledge amidst distractions.

## introduction the deployment of language models for practical applications introduces novel challenges, including the potential creation of untrustworthy or incorrect text @xcite @xcite . specifically, models that are fine-tuned using human preferences may amplify existing human biases or generate persuasive yet deceptive outputs @xcite . empirical evidence suggests that simulated internal beliefs or knowledge can be extracted from language model activations @xcite @xcite . supervised probing methods can be employed to extract this knowledge @xcite but such methods require labels, which in some domains may not be readily provided due to human biases or because humans simply do not know the correct label. it may even be critical to avoid the use of human labels to differentiate between a model's true knowledge and its representation of human knowledge. motivated by these ideas, unsupervised probing techniques like contrast-consistent search (ccs) have been developed to extract the knowledge embedded in a language model without the need for ground truth labels @xcite . @xcite outline current limitations of these approaches, demonstrating that these unsupervised probes tend to identify the most salient binary feature, which may not always correspond to the specific knowledge feature we seek. for example, in one experiment, one of a pair of distracting random words is added to each prompt in a text dataset. after training, unsupervised ccs probes often function as classifiers for these random words, rather than the intended knowledge feature. in practice, there may be numerous salient features of which we are unaware, which can divert an unsupervised probe from identifying the target feature, regardless of whether they are correlated or uncorrelated with the target. to tackle this issue, we propose a cluster normalization method. our method starts by following the usual initial approach of unsupervised probing of harvesting contrast pair activations. we then cluster similar activations and normalize them separately, thereby eliminating the effect of distracting salient features. we can then apply any unsupervised probing method, such as ccs or crc-tpc @xcite , to train a probe on these normalized activations. it is of course crucial to ensure this approach does not inadvertently eliminate the knowledge feature itself. to prevent this, we utilize contrast pairs, performing the clustering on the average embedding of each pair. further details on figure 1 : under the standard ccs approach (left; burns-norm), a modified prompt including distracting random words (red) causes all ccs probes to achieve random accuracy against ground truth labels (gt), and high accuracy against these random word labels (random). when using our approach of cluster normalization (right; cluster-norm), the average accuracy of ccs probes for the desired feature increases significantly. probes trained with the original ccs approach achieve an average accuracy of approximately 0.5 on prompt datasets with distracting random word features @xcite . in contrast, our clustering method significantly improves this average accuracy to about 0.77, and to 0.81 for crc-tpc when tested with mistral-7b. details of our method are described in section 3. ## method for each pair, we harvest the intermediate activations of a language model m, specifically the state of the residual stream at the final token position at a specific layer, which we denote as m(@xmath0. we average these activations for each contrast pair m( following the notation in section 2.3, if x i belongs to cluster @xmath1, 2}, a successful cluster normalization will leave: this follows from equation 1, however in this case normalization is performed over c only as opposed to the whole dataset. a key element to the effectiveness of our method is that our clustering approach does not erase the effect of the desired knowledge feature. this is achieved by clustering the averages of each contrast pair, m(x i ). as a result, clustering only isolates salient non-contrastive features, and is effectively blind to ⃗ @xmath2. normalizing positive and negative samples separately per cluster aims to ensure that all contrastive features ⃗ f ′ related to ⃗ @xmath3, as explained in section 2.3. note, we do not normalize out similar ⃗ f ′ resulting from the mixing of ⃗ f with ⃗ f ⊤/⊥ . eventually, only contrastive features related to knowledge are kept. ## experiments in our experiments, we utilize mistral-7b as our main language model, harvesting activations (using the libraries from @xcite and @xcite ) at the 75th percentile layer (layer 24 for mistral-7b) since, from preliminary experiments, we find probes achieve higher accuracies using the 50th to 90th percentile layers. we also report results using different language models (phi-2 and 3, gemma-7b, llama-3-8b, pythia-6.9b) and layers in the following subsections and appendices to verify the efficacy of our method. our experiments follow the same general approach as those reported in @xcite , as each of these original experiments set out to demonstrate the limitations of current unsupervised probing techniques. individual results for each model can be found in appendix b.1. we present results for three experiments below. for the first and second, we create prompt datasets based on the imdb dataset @xcite , while for the third we use the com-monclaim @xcite dataset. we report results on a fourth experiment utilizing the dbpedia dataset @xcite in appendix a; this experiment follows on from results reported in @xcite , however, we find we are unable to replicate these results (on three different models) and instead obtain high accuracies for both the original method of ccs and our approach using cluster normalization. activation clustering is performed using hdb-scan, implemented in the scikit-learn library @xcite , setting a minimum number of elements in each cluster to 5 and using the euclidean distance metric. one advantage of hdbscan over other clustering algorithms (e.g., kmeans) is that the number of clusters does not need to be specified in advance. in order to examine the variance in probe performance, we report summary statistics of 50 probe fits in each experiment, and visualize the results from all. the following experiments generally involve a comparison between an original prompt and a modified one, to attempt to induce a bias in an unsupervised probe. hereon, we refer to these original prompts as unbiased or non-biased and modified prompts as biased. we also refer to normalization over an entire dataset, as burns normalization or burns-norm (see 2.3 for more details). we refer to our alternative approach through clustering as cluster normalization or cluster-norm. unlike the approach in @xcite , where multiple prompt templates were used, the study in @xcite utilized only one prompt template per dataset. our method follows the prompt-template setup from @xcite . each experiment utilizes a train-test split of 70% for training and 30% for testing. importantly, we evaluate our unsupervised probes on a test set where burns-norm is applied to the test set as it was done in @xcite , and not our cluster normalization. this is because we want probes to generalize, so if during evaluation they are fed with a contrast pair that belongs to an entirely different dataset, it is out of distribution for the clusters found during training. the probe should be a feature in the unaltered latent space. although we do not use cluster normalization on the test set for the aforementioned reason, we do use burns-norm for being able to compare our results with @xcite as this is what they do for the test set. farquhar et al. (2023) likely follow a similar approach, as they mention utilizing normalization but do not provide any details regarding a train-test split. for each experiment, we also report results using the crc-tpc method as an alternative unsupervised probing technique to ccs. finally, we report an upper-bound by using the results of the supervised method of logistic regression, similar as done by @xcite and later also by @xcite . for additional experiment details see also appendix b.1. ## related work it has been shown that language models develop internal representations of the world @xcite , with individual concepts often encoded as linear directions in activation space @xcite @xcite . language models can also output false information, even if the encoded knowledge in the activations seems to indicate a correct internal representation of the information @xcite @xcite . we seek to elicit this latent knowledge @xcite in an unsupervised manner. in recent years several methods have been proposed @xcite @xcite @xcite , although unsupervised methods can be subject to undesirable biases, as shown by @xcite . they demonstrate that unsupervised probing techniques, such as those developed in @xcite , often identify the most salient features in a dataset, as opposed to knowledge only. these features may not always align with the specific knowledge feature of interest, as described in section 3. we provide theoretical explanations for some of these issues, and propose a method to eliminate them. the work we cite in the introduction and background sections focuses on finding a general linear representation of knowledge in the latent space of a language model. while we focus on unsupervised approaches, most work concentrates on supervised ones @xcite . this body of work is part of a more general field of research that aims at ensuring truthfulness of language models, by making sure that what they answer is actually what they believe or follows from reasoning e.g., working with quirky language models or using chain-of-thought reasoning @xcite @xcite . ## discussion and conclusion in this study, we address significant challenges associated with the unsupervised probing of knowledge in language models. the primary issue tackled is that of distracting salient features that can mislead the probing process. our cluster normalization technique shows promising results in effectively isolating and minimizing the impact of such distractions, thereby enhancing the performance of unsupervised probes. our results demonstrate that without proper normalization, probes tend to align with the most salient features present in the dataset, which are not necessarily related to the target knowledge feature. this observation mostly aligns with findings from previous studies @xcite , which showed that unsupervised probes are prone to capturing irrelevant features when such features are salient. however, in general, our results do not show as pronounced an effect as @xcite suggested for the standard ccs method. this observation is especially true for the experiments detailed in section 4.2 and appendix a). nonetheless, through cluster normalization, we provide a promising method to mitigate the issue of distracting salient features by identifying these features and ensuring that they are canceled out during the training of the probe. this normalization allows the probe to focus more accurately on the intended knowledge feature.