Scaling up Discovery of Latent Concepts in Deep NLP Models (2024)

Abstract: Despite the revolution caused by deep NLP models, they remain black boxes, necessitating research to understand their decision-making processes. A recent work by Dalvi et al. (2022) carried out representation analysis through the lens of clustering latent spaces within pre-trained models (PLMs), but that approach is limited to small scale due to the high cost of running Agglomerative hierarchical clustering. This paper studies clustering algorithms in order to scale the discovery of encoded concepts in PLM representations to larger datasets and models. We propose metrics for assessing the quality of discovered latent concepts and use them to compare the studied clustering algorithms. We found that K-Means-based concept discovery significantly enhances efficiency while maintaining the quality of the obtained concepts. Furthermore, we demonstrate the practicality of this newfound efficiency by scaling latent concept discovery to LLMs and phrasal concepts.

## introduction transformer-based language models excel at revealing intricate patterns, semantic relationships, and nuanced linguistic dependencies concealed within vast textual datasets through unsupervised learning. their capability to encode complex abstractions, surpassing surface-level word meanings, has resulted in significant advancements across various natural language understanding tasks. a considerable body of research dedicated to interpreting pre-trained language models (e.g. @xcite ; @xcite ; @xcite ; @xcite among others) seeks to answer the question: what knowledge is learned within these models? researchers have delved into the concepts encoded in pre-trained language models by probing them against various linguistic properties. our work facilitates this line of work in interpretability by scaling up discovery of latent concepts learned within pre-trained language models. @xcite demonstrated that words exhibit a tendency to form clusters in highdimensional spaces, reflecting their morphological, syntactic, and semantic relationships. building upon this foundational insight, recent studies @xcite @xcite delve into representation analysis by exploring latent spaces within pre-trained models. @xcite discovered encoded concepts in pre-trained models by employing agglomerative hierarchical clustering @xcite on the contextualized representations in the bert model @xcite . however, a fundamental limitation of their work is the computational expense of the underlying methodology. since contextual representations are high-dimensional, only a limited amount of data can be clustered to extract the latent concepts. this significantly undermines the purpose of concept discovery, providing only a limited perspective on the spectrum of concepts that might be learned within the model and significantly limiting the scalability of the approach. in this work, we aim to address this shortcoming by employing computationally-cheaper clustering algorithms, specifically comparing three algorithms in quality and computational efficiency: agglomerative hierarchical clustering, leaders algorithm, and k-means clustering. as there is no inherent groundtruth clustering we can rely on to measure the quality of the algorithms in the latent space, we introduce a metric with two dimensions, alignment and coverage, to measure the "goodness" of a clustering. we also show that scaling the underlying data for concept discovery results in significantly better results, as well as enables new directions that were previously unexplored. in summary we make the following contributions: • we present a comprehensive comparison between various clustering techniques regarding their quality and efficiency for the task of latent concept discovery. ## concept discovery our investigation builds upon the work of discovering latent ontologies in contextualized representations @xcite . at a high level, feature vectors (contextualized representations) are initially generated by performing a forward pass on a pre-trained language model. the representations are then clustered to uncover the encoded concepts of the model (see figure 1 for illustration). a concept, in this context, can be understood as a collection of words grouped together based on some linguistic relationship, such as lexical, semantic, syntactic, or morphological connections. figure 2 showcases concepts within the latent space of the bert model, wherein word representations are arranged based on distinct linguistic concepts. formally, consider a pre-trained model m with l layers: l 1 , l 2 , . . . , l l . using a dataset of s sentences totaling n tokens, @xmath0, w 2 , . . . , w n ], we generate feature vectors: ## assessing quality of concept discovery with the unsupervised identification of encoded concepts by a clustering algorithm, a question arises: how can we effectively compare various clustering algorithms in relation to their ability to uncover these concepts? we introduce a measure that evaluates the alignment of encoded concepts in light of linguistic ontologies (e.g. parts-of-speech tagging). previous research @xcite @xcite showed that higher layers of plms get optimized for the task that the plm is trained for, and that tuning a plm for any task results in its latent space being skewed towards the output classes of the target task. 2 we use this finding to compare the concepts discovered via different algorithms, by measuring the alignment between the encoded concepts learned by a fine-tuned model to the humandefined concepts of the underlying task. although we here employ a fine-tuned model to assess quality, it is crucial to emphasize that this is solely for evaluation purposes. the selected algorithm can subsequently be applied to any generic pre-trained language model to uncover its latent concepts as well. formally, consider a downstream task (e.g. pos tagging @xcite ), for which we possess true class annotations for the input data d (i.e., per-word pos labels). for each tag, we construct a human-defined concept using the annotated data. for instance, c h (vbd) = {died, smiled, explored, . . . } defines a human concept comprising past-tense verbs, while c h (nns) = {boys, girls, rackets, . . . } outlines a concept of plural nouns. let c @xmath1, c h 2 , . . . , c h n } be the set of all humandefined concepts for the task, and c @xmath2, c e 2 , . . . , c e m } be the set of discovered encoded concepts within the latent space of the finetuned plm. we define their θ-alignment as a function λ θ (e, h): the first term computes the ratio of discovered concepts that are aligned (up to θ ∈ [0, 1]) to the human-defined concepts (alignment), while the second measures how many unique concepts within the human-defined ontology were recovered within the latent space (coverage). this latter term demarcates our metric from that of @xcite , and we use a high threshold θ = 0.95 in our experiments. figure 1 shows a visual representation of the two terms. ## results in the following subsections, we present our comparison of the three clustering algorithms in various data and model regimes to identify the strengths and weaknesses of the underlying methods. ## applications given the larger scale that k-means clustering offers, we present preliminary results on potential future directions in latent concept discovery that were previously not possible. specifically, we look at concepts beyond the level of individual words, and also present results on two large language models, which have now become the foundation for many important leaps in nlp. ## conclusion concluding this study, our exploration into uncovering latent concepts within the embedding space of pre-trained language models represents just the initial phase towards comprehending these models and establishing trust in their functionality. our findings underscore the effectiveness of employing clustering of contextualized representations to unveil meaningful concepts that resonate with human comprehension. furthermore, we highlight the viability of utilizing k-means algorithm to handle expansive datasets, thereby facilitating the analysis of larger models and more intricate concepts. moving forward, there remains a compelling avenue for delving deeper into the interpretability of language models beyond the granularity of individual words. this involves unraveling the representation and utilization of complex linguistic constructs for inference purposes. also, our exploration of the leaders algorithm could potentially expand beyond our use with agglomerative clustering. for instance, an intriguing avenue for future research includes enhancing the scalability of k-means, pushing its limits to accommodate even more extensive datasets.