Locating and Extracting Relational Concepts in Large Language Models (2024)

Abstract: Relational concepts are indeed foundational to the structure of knowledge representation, as they facilitate the association between various entity concepts, allowing us to express and comprehend complex world knowledge.By expressing relational concepts in natural language prompts, people can effortlessly interact with large language models (LLMs) and recall desired factual knowledge. However, the process of knowledge recall lacks interpretability, and representations of relational concepts within LLMs remain unknown to us. In this paper, we identify hidden states that can express entity and relational concepts through causal mediation analysis in fact recall processes. Our finding reveals that at the last token position of the input prompt, there are hidden states that solely express the causal effects of relational concepts. Based on this finding, we assume that these hidden states can be treated as relational representations and we can successfully extract them from LLMs. The experimental results demonstrate high credibility of the relational representations: they can be flexibly transplanted into other fact recall processes, and can also be used as robust entity connectors. Moreover, we also show that the relational representations exhibit significant potential for controllable fact recall through relation rewriting.

## introduction pretrained gpt-based large language models (llms) can be regarded as knowledge bases, storing a large amount of knowledge about the real world. each factual knowledge can usually be represented in the form of a triple @xmath0, r, o), where s represents the subject entity, o represents the object entity, and r represents the relational concept that connects the subject and the object @xcite . for example, the fact that the capital of france is paris can be expressed as the triple (france, the capital of, paris). llms provide a flexible way to recall internal knowledge: people can use natural language prompts containing subjects and relations to prompt llms, thereby stimulating the model responses containing corresponding objects i.e.(f rance, the capital of ) → p aris. @xcite . in this process, relationships play an important role: they connect paired entities to express various factual knowledge. being able to locate and extract relational representations is of great value in exploring the interpretability of knowledge recall mechanism within llms. recent studies on llms interpretability suggest that it is feasible to identify human-interpretable conceptual representations, including locating knowledge neurons @xcite @xcite , finding semantic directions in activation space @xcite @xcite , extracting task vectors @xcite , and approximating concept mapping @xcite . however, how relational concepts are represented within llms remains unexplored. in this paper, we show that it is possible to locate and extract relational representations from llms, and these extracted representations can be connected with other entity concepts, enabling the expression of valuable knowledge. our research first reports an enlightening observation that provides us inspiration for locating relational representations during the course of casual mediation analysis @xcite . we find that at the last token position, the shallow hidden states solely present the causal effects of relations, and then deeper hidden states simultaneously express the causal effects of both subjects and relations in a more integrated manner, as shown in figure 1 . based on this finding, we hypothesize that at the last position, hidden states that only present relational causal effects can be regarded as relational representations and they can be extracted and reconnected with various other entities. the reason why we focus on the last token is intuitive: due to the casual attention mechanism in gpt-based llms, only the last token captures complete contextual information to predict the object in the next token prediction way. to verify this hypothesis, we conduct various experiments, including hidden states transplantation and zero-shot relational reasoning on data of multiple relation types. the purpose of hidden states transplantation experiments is to verify the location accuracy of relational representations which includes a sliding pointer to specify the layer range for transplantation. the results reveal that when the transplanted layer range only contains hidden states with relational causal effects, they exhibit a remarkable ability to express relational concepts, without absorbing any subject concepts. at this point, these states can be seamlessly transplanted to other fact-recall processes, enabling the recall of other relevant objects. the purpose of zero-shot relational reasoning experiments is to verify the faithfulness and robustness of the relational representations that have been accurately extracted. our research demonstrates that the extracted relational representation acts as an effective entity connector, which is capable of accurately reasoning corresponding objects by taking various subjects as input. we also expand the scope of our research into the more general fields of dialogue and text generation, where we try to use the relational representations to control llms responses. specifically, we propose the relation rewriting method, which can integrate other relational representations into the chatting process so that the model is required to generate content that aligns with the desired relation. the main contributions can be summarized as follows: ## method in this section, we present the procedural details of how we derive our motivating observation and hypothesis for identifying relational representations through causal mediation analysis. ## experiments for hypothesis validation we design experiments from two aspects to verify our hypothesis. the hidden states transplantation experiment aims to prove that the hidden states in the relational emergence stage indeed accurately locate the relational representation, and the zeroshot relational reasoning experiment aims to study the faithfulness and robustness of the relational representations. ## application in controllable fact recall after the above discussion, we have proved that the relational representation can be extracted from llms, with great faithfulness and robustness. in this section, we aim to utilize this property to achieve controllable fact recall by taking advantage of relational representation in more general scenarios. ## limitations although the three-stage observation provides us inspiration for locating relational representation, this observation is uncommon in fact recall processes involving linguistics. for example, when llms predict the past tense or comparative form of a word, the causal effects of subjects emerge before relations, which is inconsistent with our three-stage observation. as shown in figure 9 , we collect the average causal effects of subjects and relations at the last position in a number of linguistic tasks. we find that the causal effect of subjects is very obvious from the first layer, while the causal effects of relations still have a gradual rising process. we speculate that this is because linguistic tasks only entail the application of predetermined grammatical rules, without requiring multi-layer processing to capture the semantics of words. this limits our method to linguistic relation location and extraction. ## conclusion in this paper, we analyze the fact recall process within gpt-based llms. our investigation reveals that within llms, there are specific hidden states that can solely express relation concepts, and we hypothesise that these hidden states can be regarded as relational representations. we design hidden states transplantation experiments and zero-shot relational reasoning experiments to verify our hypothesis. experimental results show that these hidden states indeed accurately locate relational representations without absorbing other entity concepts, and they can be extracted as entity connectors with high faithfulness and robustness. based on the above findings, we explore the application potential of the relational representations in controllable llms response. we leverage the relational representations to intervene with user inquiry requests for llms, and successfully guide the model to output other answers. our work sheds light on the study of interpretability of llms and provides a stepping stone towards understanding the internal knowledge system of llms.