Interpreting Embedding Spaces by Conceptualization (2023)

Abstract: One of the main methods for computational interpretation of a text is mapping it into a vector in some embedding space. Such vectors can then be used for a variety of textual processing tasks. Recently, most embedding spaces are a product of training large language models (LLMs). One major drawback of this type of representation is their incomprehensibility to humans. Understanding the embedding space is crucial for several important needs, including the need to debug the embedding method and compare it to alternatives, and the need to detect biases hidden in the model. In this paper, we present a novel method of understanding embeddings by transforming a latent embedding space into a comprehensible conceptual space. We present an algorithm for deriving a conceptual space with dynamic on-demand granularity. We devise a new evaluation method, using either human rater or LLM-based raters, to show that the conceptualized vectors indeed represent the semantics of the original latent ones. We show the use of our method for various tasks, including comparing the semantics of alternative models and tracing the layers of the LLM. The code is available online https://github.com/adiSimhi/Interpreting-Embedding-Spaces-by-Conceptualization.

## introduction recently, there has been significant progress in natural language processing thanks to the development of large language models (llms). these models are based on deep neural networks and are trained on extensive volumes of textual data @xcite @xcite . while these powerful models show excellent performance on a variety of tasks, they suffer from a significant drawback. their complex structure hinders our ability to understand their reasoning pro-cess. this limitation becomes crucial in several important scenarios, including the need to explain the decisions made by a system that employs the model, the necessity to debug the model and compare it with alternatives, and the requirement to identify any hidden biases within the model @xcite @xcite . current llms process text by projecting it into an internal embedding space. by understanding this space, we can therefore gain an understanding of the model. such understanding, however, is challenging as the dimensions of the embedding space are usually not human-understandable. the importance of interpretability has been recognized by many researchers. several works present methods for explaining the decision of a system that uses the embedding (mainly classifiers) (e.g. @xcite . some works @xcite perform training or retraining for generating a new model that is interpretable, thus detouring the problem of understanding the original one. another line of work assumes the availability of an embedding matrix and uses it to find orthogonal transformations, such that the new dimensions will be more understandable @xcite . probing methods utilize classification techniques to identify the meaning associated with individual dimensions of the original embedding space @xcite . in this work, we present a novel methodology for interpretability of llms by conceptualizing the original embedding space. our algorithm enables the mapping of any vector in the original latent space to a vector in a human-understandable conceptual space. importantly, our approach does not assume that latent dimension corresponds to an explicit and easily-interpretable concept. our method can be used in various ways: our contributions are: our approach only requires a black box that takes a text fragment as input and produces a vector as output. ## the conceptualization algorithm let t be a space of textual objects (sentences, for example). let @xmath0. . . × l k be a latent embedding space of k dimensions. let f : @xmath1. typically, f will be an llm or llm-based. our method requires two components: a set of concepts @xmath2, . . . , c n defining a conceptual space @xmath3. . . × c n , and a mapping function τ : @xmath4. in the pre-processing stage, we map each concept @xmath5, the textual representation of c. we thus define n vectors in l, c 1 , . . . , c n such that given a vector @xmath6, we measure its similarity to each vector c i (that represents the concept c i ) using any given similarity measure sim. the algorithm then outputs a vector in the conceptual space, using the similarities as the dimensions. we have thus defined a meta-algorithm ces (conceptualizing embedding spaces) that, for any given embedding method f , a set of concepts c and a mapping function τ from concepts to text, takes a vector in the latent space l and returns a vector in the conceptual space c: a graphical representation of the process is depicted in figure 1 . if we use cosine similarity as sim, and use a normalised f function, we can implement ces as matrix multiplication, which can accelerate our computation. first, observe that, under these restrictions, cosine similarity is equivalent to the dot product between vectors. let @xmath7, . . . , u k be the standard basis in k dimensions as a base of l. we can look at the projection of u in the c space, by using function ϕ such that we can now create a n × k matrix @xmath8, . . . , ϕ(u k )⟩. using this matrix, we get ces f,c,τ (l) = m • l. ## empirical evaluation it is not easy to evaluate an algorithm whose task is to create an understandable representation that matches the original incomprehensible embedding. we performed a series of experiments, including a human study, that show that our method indeed achieved its desired goal. for all the experiments, we have used roberta sentence embedding model 3 @xcite as our f , unless otherwise specified. all models used in this work were applied with their default parameters. whenever the concept space c * was used, we set @xmath9, but we observed that using much smaller values yielded almost as good results. the default value for removep is false. for τ , the function that maps concepts to text, we have just used the text of the concept name (with a length of 4.25 words on average in g). 4 ## related work the problem of interpretability has received significant attention in recent years. a large body of research @xcite @xcite @xcite @xcite is devoted to generate an explanation for the decision of the model (mostly classification). many methods utilize nearby examples or counterfactuals to provide users with reasoning behind the decision. several works set a goal, like ours, of understanding the model itself, rather than its decisions. most of these works attempt to assign some meaning to the dimensions, either of the original latent space or of a different space that the original one is transformed to. one relatively early approach tries to find orthogonal or close to orthogonal transformations of the original embedding matrix @xcite @xcite such that a set of words with high weight in a given dimension are related and thus hopefully represent some significant concept. the advantage of these orthogonal methods is that they do not lose information due to the orthogonality. several of these works @xcite @xcite @xcite transform the original embedding to a sparse one to improve the interpretability of each dimension. one limitation of these methods is their reliance on an embedding/dictionary matrix. @xcite assigns a specific concept to each dimension. note that our work is different as it does not assume that each latent dimension corresponds to a human-understandable concept. recent methods @xcite assume access to the model's internals, particularly the un-embedding matrix, to map a latent vector to the token space. other works @xcite created new tools to help interpret the model. @xcite uses dictionary learning to view the transformer model as a linear superposition of transformer factors. @xcite introduces a tool for doing simple operations such as pca and t-sne on embedding. probing methods try to interpret the model by studying its internal components. vig et al. (2020) 2 4 6 8 10 12 layers 10 0 10 1 10 2 10 3 rank transport medicine corruption government officials government ministers government documents (a) bert 2 4 6 8 10 12 layers 10 0 10 1 10 2 10 3 rank family weapons quality government officials scientists government documents (b) gpt2 make changes to the input to find out what parts of the model (specific attention heads) a bias comes from. tenney et al. (2019) use probing on bert model to find the role of each layer in the text interpretation process. bau et al. (2019) and dalvi et al. (2019) show how linguistic properties are distributed in the model and in specific neurons. clark et al. (2019) create an attention-based probing classifier to find out what information is captured by each attention head of bert. lastly, sommerauer and fokkens (2018) use supervised classifiers to extract semantic features. some works @xcite @xcite @xcite tackle the problem by training or retraining to create a new interpretable model. unlike those methods, our approach focuses on understanding the original models while preserving their performance, rather than using interpretable models as substitutes. ## conclusion in this work, we introduce a novel approach to llm interpretation that maps the latent embedding space into a space of concepts that are wellunderstood by humans and provide good coverage of the human knowledge. we also present a method for generating such a conceptual space with an ondemand level of granularity. we evaluate our method by an extensive set of experiments including a novel method for evaluating the correspondence of the conceptual embedding to the meaning of the original embedding both by humans and by other models. finally, we showed applications of our method for comparing models, analyzing the layers of the model, and debugging.