Crawling The Internal Knowledge-Base of Language Models (2023)

Abstract: Language models are trained on large volumes of text, and as a result their parameters might contain a significant body of factual knowledge. Any downstream task performed by these models implicitly builds on these facts, and thus it is highly desirable to have means for representing this body of knowledge in an interpretable way. However, there is currently no mechanism for such a representation. Here, we propose to address this goal by extracting a knowledge-graph of facts from a given language model. We describe a procedure for “crawling” the internal knowledge-base of a language model. Specifically, given a seed entity, we expand a knowledge-graph around it. The crawling procedure is decomposed into sub-tasks, realized through specially designed prompts that control for both precision (i.e., that no wrong facts are generated) and recall (i.e., the number of facts generated). We evaluate our approach on graphs crawled starting from dozens of seed entities, and show it yields high precision graphs (82-92%), while emitting a reasonable number of facts per entity.

## introduction modern language models (lms) @xcite ) are trained on vast amounts of text that captures much of human knowledge, including scientific articles, wikipedia, books, and other sources of information @xcite . consequently, such models encode world knowledge in their parameters, allowing them to generate rich and coherent outputs. past work has illustrated lms can be viewed as knowledge-bases @xcite as well as analyzed the encoded knowledge (e.g., see @xcite and leveraged it for applications such as closed-book qa @xcite and search @xcite , illustrating lms can be viewed as the first question in this endeavour is what is a suitable explicit knowledge representation. a natural candidate structure is a knowledge graph (kg). namely, a graph whose nodes are entities and whose edges represent relations between entities. kgs are appealing since information can be readily "read-off" from the graph, they can be reliably queried, and different kgs can be easily compared. kgs have been extensively used to represent knowledge @xcite , but a key limitation is their low coverage, as they usually require manual curation and depend on a closed schema. conversely, lms might have very high coverage as they are trained on a vast body of knowledge represented as raw text. we thus ask if it is possible to convert an lm to a kg, such that we enjoy its advantages while achieving high coverage. as the full kg encoded in an lm can be large, we reduce the problem to the task of constructing a kg around a given seed entity. for example, fig. 1 shows a kg extracted by our method for the seed entity alan turing. this can be viewed as a crawling procedure which starts from the seed entity and recursively expands it to expose additional facts. this crawling problem introduces several new challenges. first, unlike prior work @xcite @xcite , we are given only an entity, without know- alan turing united kingdom country of citizenship capital official language summer dst time zone member of london english yes gmt (utc+0) european union currency government type internet ltd iso 3166 code calling code pound sterling constitutional monarchy .uk gb +44 british nationality official language of capital country given name family name educated at educated at place of birth field of work alan turing king's college sherborne school paddington, london cryptanalysis suicide manner of death occupation occupation field of work sex or gender field of work mathematician computer science logician male mathematics named after industry information technology notable subfields notable subfields notable subfields notable subfields topology algebra analysis geometry figure 1: an example of a generated depth-2 knowledge graph around the seed entity alan turing, applying lmcrawl (see sec. 3-sec. 4). additional graphs are in sec. e. namely, what does it mean for a model to "believe" a fact, as opposed to the model "knowing" a fact. here we adopt a "dispositional" view of belief, whereby a belief corresponds to a statement by the model, and knowledge is a belief that is true in the world. we test our approach with gpt-3 (text-davinci-002) on 140 seed entities, and show that we can extract accurate kgs (∼82-92% precision) that contain a plausible number of facts per entity. importantly, large lms are not constrained to a predefined schema, and indeed our procedure with gpt-3 generates facts outside the schema of wikidata, e.g., (boston celtics, championships, 17). to conclude, our contributions are: 1) formulating the problem of crawling a kg from an lm, 2) presenting a prompt-based approach that decomposes the problem into multiple sub-tasks, and 3) evaluating the approach with gpt-3, which leads to high-precision graphs. ## problem setup our goal is to uncover the knowledge-base of a given lm. we represent a knowledge-base via a kg, which is a collection of triplets. formally, a kg is a graph @xmath0, r, e), where n is a set of entities, r is a set of relations, and e is a set of subject-relation-object triplets (s, r, o) where s, @xmath1. to simplify the setup, we assume we are given a "seed entity" around which we will expand the graph (for example fig. 1 ). conceptually, we can also let the lm generate seed entities, but we argue seed expansion is a more realistic scenario, where a user is interested in a graph about a certain entity. entities and relations are represented via strings and are not constrained to a given vocabulary (similar to open information extraction. e.g., see @xcite . ## crawling kgs via prompting the core component of our approach is a procedure that takes an entity e, and extracts all relations associated with it, and the corresponding objects. namely, we expand the kg around this entity. we can then recursively apply this procedure to further expand the kg. we refer to this as 'entity expansion', and break it into two high-level steps: • relation generation (sec. 3.1): for an entity e, generate a set of relations r, where e is the subject. to improve recall, we employ an additional paraphrasing procedure (sec. 3.2 and sec. 3.5), which generates alternative strings for a given entity or re- ## experimental setup as mentioned in sec. 3, we use wikidata (publicly available) in constructing the in-context prompts. the number of in-context examples is k @xmath2, k @xmath3, k @xmath4. additionally, we use wikidata to select seed entities for evaluating our approach. for these seeds, we consider the task of constructing kgs around the corresponding entities. we split the seed entities into a validation set (20 entities), which is used to make design choices (e.g., choosing prompt format), and a test set (120 entities), which is used only for the final evaluation. for the development set, we manually chose 20 entities from wikidata. these included women and men with various professions, cities, countries, and various cultural entities such as movies and books. we also aimed to reprsent both head and tail entities in this list. to construct our test set, we defined 25 specific world-entities related categories, which we refer to as the test categories. some of these were more specific, such as ai researchers, and some are more general, such as scientists (see table .6 for the full list). we chose 4 seeds out of each category as follows. we first sorted the set of entities of each group based on the number of wikidata facts associated with them (we view this count as an approximate measure of popularity). then, we randomly sampled two entities out of the full list, and an additional two out of the first 1000. intuitively, the first two represent tail entities, while the other two represents head ones. thus we ended up with 100 seed entities (i.e., 4 different entities out of each of the 25 different subgroups). we refer to these as the main test set (see tab. 6). we created an additional test set of 20 entities that is meant to contain very popular entities. its entities were randomly sampled out of a set of size 1000, which was manually constructed by choosing 40 very well-known entities (i.e., that all people would know) from each of the 25 test categories. all 140 entities were not used in the construction of any of the prompts in sec. 3. tab. 2 shows the full list of validation and head test entities. evaluation metrics given an entity s, our entity expansion process returns a knowledge graph g, that contains the entity s, other entities and relations between them. dev seeds head test seeds abba aristotle alan turing canada angela merkel celine dion augustin-louis cauchy china barack obama emanuel macron bob dylan franz kafka boston celtics grease david bowie hamlet diana, princess of wales jacinda ardern eike von repgow lionel messi inglourious basterds little women marble arch manchester united f.c. marie curie margaret hamilton mikhail bulgakov michelangelo moby-dick mike tyson pablo picasso oprah winfrey paris rosalind franklin philippines steven spielberg rachel carson serena williams shahar pe'er the rolling stones ideally, we want to compare g to a ground truth graph that results from expanding the entity s. given such a graph, we could measure precision and recall over the gold and predicted sets of triplets. however, using large lms to generate graphs leads to several challenges. first, there is no ground-truth graph. while we could presumably use the wikidata graph, we found that it is missing many correct facts predicted by the lm. in fact, improving coverage is a key motivation for our work! second, facts may be reworded in several equivalent ways, rendering comparison between wikidata graphs and predicted graphs difficult. to circumvent these challenges, we use the following notions of precision and recall. in order to check the correctness of a given predicted triplet (e, r, o), we issue a query containing (e, r) to google search, and search whether o appears in the result. we limit the result to first 40 words which are not html labels or url links. if it does, we assume the triplet is correct. foot_1 see sec. 5.3 for an accuracy estimation of the automatic method. manual evaluation was done for all the head test set graphs, as well as all the 1-hop graphs of the main test set. additionally, we performed manual evaluation for 20% randomly sampled triplets from the 2-hop graphs (altogether, the total portion of manually labeled facts from each graph was ∼30%). the rest of the triplets were automatically evaluated. implementation details as the lm in our experiments, we used the openai text-davinci-002 model. we experiment with both greedy decoding and sampling 3 outputs per query (temperature 0.8). we generate graphs with either a single expansion step or two expansion steps, recursively expanding entities found in the first step. after a graph is generated, we remove duplicates by iterating through the facts and removing a fact if the token-wise f 1 between it and another fact is higher than 0.85. ## results we next report results showing that our expansion method is able to generate meaningful knowledge subgraphs, when expanding seed entities. ## related work pretrained lms are at the heart of recent nlp research and applications. as mentioned earlier, @xcite and other works have observed that lms contain rich factual knowledge. we elaborate on other relevant works below. knowledge-base construction. kg construc-tion typically involves both manual and automated aspects. for example, popular kbs such as word-net @xcite , conceptnet @xcite and wikidata @xcite were constructed by heavily relying on manual effort, gathering knowledge from humans. to reduce such manual labor, automated information extraction (ie) methods have been extensively developed @xcite @xcite . knowledge in lms is a fairly recent topic of interest, and has mostly focused on probing for specific facts @xcite . most similar to our work are @xcite , who also extract kgs from lms, however, they require defining the relations of interest through examples before crawling, while our specific goal is to start with a seed entity and allow the lm to determine the relevant relations. another relevant recent work is @xcite who also use in-context learning to extract a kg from gpt3. but they also assume relations are provided, whereas a key aspect of our approach is generating the relations. to the best of our knowledge, ours is the first work to construct a knowledge graph via extracting knowledge directly from lms, using only one seed entity (and no other given relations or entities). ## conclusion understanding large lms is a key part of modern nlp, as they are used across the board in nlp applications. in particular, it is important to under-stand the body of knowledge these models possess, so it can be used and revised as needed, thereby avoiding factual errors and biases. in this work we present an important step towards this goal by extracting a structured kb from an lm. there are many possible exciting extensions for our work. the first is to expand it to a larger graph corresponding to more expansion hops. this would require many more calls to an api, which at present is also costly, and it would be important to develop more cost-effective approaches. second, we have introduced several approaches to controlling the precision and recall of the proposed model, but certainly more can be envisioned. for example, we can introduce various consistency constraints to increase precision (e.g., check that father of and child of are consistent in the generated graph). finally, once a larger kg has been extracted, one can query it to see how well it serves as a question answering mechanism. overall, we find the possibility of seamlessly converting lms to kgs for better interaction and control to be an exciting and fruitful direction for future research.