Defining Knowledge: Bridging Epistemology and Large Language Models (2024)

Abstract: Knowledge claims are abundant in the literature on large language models (LLMs); but can we say that GPT-4 truly “knows” the Earth is round? To address this question, we review standard definitions of knowledge in epistemology and we formalize interpretations applicable to LLMs. In doing so, we identify inconsistencies and gaps in how current NLP research conceptualizes knowledge with respect to epistemological frameworks. Additionally, we conduct a survey of 100 professional philosophers and computer scientists to compare their preferences in knowledge definitions and their views on whether LLMs can really be said to know. Finally, we suggest evaluation protocols for testing knowledge in accordance to the most relevant definitions.

## introduction nlp researchers have used the term knowledge somewhat haphazardly in the context of large language models (llms), e.g., discussing "knowledge contained in language models" (jiang et al., 2020), their "knowledge gaps" (feng et al., 2024b), or how "llms encode knowledge" (farquhar et al., 2023), and "model's internal knowledge" (kassner et al., 2023). petroni et al. (2019) defined an llm to know a fact if it correctly completes a cloze sentence such as "the capital of germany is __", which are typically generated directly from so-called knowledge graphs. many have evaluated knowledge in this way (jiang et al., 2020; paik et al., 2021; dai et al., 2022; kassner et al., 2020, 2021a; keleg and magdy, 2023, inter alia). however, the predictions of semantically equivalent cloze sentences can be inconsistent 1 (elazar et al., 2021; kassner and schütze, 2020; fierro and figure 1 : from our survey ( §4): philosophers and computer scientists prefer different definitions of knowledge. søgaard, 2022), leading to question the meaningfulness of knowledge claims. should we then require an llm to predict correctly all the paraphrases of a given fact to say it knows it? what about related facts? can we really say that an llm knows that 'lionel messi plays for inter miami' if it does not know that 'lionel messi resides in miami'? what, then, are sufficient conditions for saying an llm knows? or more generally, can llms know anything? that is: can llms have bona fide knowledge? whether llms know, or in what sense, depends on how knowing is defined. determining what internal knowledge llms possess could have important implications on their trustworthiness, as knowledge modulates our trust in agents (hardwig, 1991; pederneschi, 2024). we tend to lose trust in others when they do not appear to know what we consider basic facts. furthermore, studying knowledge in llms could potentially have implications for epistemology itself (cappelen and dever, 2021). recent works have approached the question of how to define knowledge, considering additional requirements for determining what an llm knows. some require correct predictions across paraphrases (de cao et al., 2021; zhong et al., p is known if and only if philosopher tb-knowledge p is true, and p is believed + sartwell (1992) j-knowledge p is true, p is believed, and p is justified nozick (2000) g-knowledge p is known sui generis williamson (2005) v-knowledge p is inferred with intellectual virtue zagzebski (1999) p-knowledge p is believed and facilitates correct predictions austin (2000) contributions we survey the most commonly used definitions of knowledge in epistemology, and discuss and formalize how to map these definitions to llms. we compare current research of knowledge in llms to our formal definitions, identifying shortcomings in evaluation practices. we present the results of our survey to philosophers and computer scientists about their views on llms and knowledge, finding disagreements about when llms can be said to know. these disagreements seem to arise from adherence to slightly different definitions of knowledge (figure 1 ). finally, we provide protocols that follow the epistemological definitions for evaluating and testing knowledge in llms. we hope that the connection we provide to epistemology can inform better evaluations and claims regarding knowledge in llms. ## definitions of knowledge while the nlp research community's use of the word knowledge has been somewhat unclear, in philosophy there is a long tradition of trying to pin down exactly what is involved in knowledge claims. knowledge -or propositional knowledge, foot_0 to be precise -is what is at stake when we say that 'x knows that p' where x is an entity whose knowledge is under question, and p is a declara-tive statement. @xcite but what are the necessary and sufficient conditions for knows here? we review 5 definitions of knowledge (see ## knowledge in nlp research now, we discuss perspectives from nlp research on what constitutes knowledge, and how these align with the definitions we extracted from the philosophical literature. some works propose to enhance the llm with an extra component to ensure more consistent beliefs; a so-called belief bank (kassner et al., 2021b) or reflex layer (kassner et al., 2023). this extra component is optimized for consistency via weighted maxsat (park, 2002), and it is used to prompt the model to be consistent to its previous stated beliefs (kassner et al., 2021b), or it is directly used to determine the system's prediction (kassner et al., 2023). both works aim to rely on tb-knowledge, where the extra component approximates belief + . @xcite however, it is only an approximation as the extra component is not necessarily fully consistent and the entailed facts are sampled. this approximation would not be a problem if we consider their approach to be under p-knowledge, although in that case the entailed facts should be selected according to some measure of relevance. furthermore, kassner et al. (2023) are slightly inconsistent in how they use the term knowledge, e.g., using interchangeably "model beliefs" and "models' internal knowledge", if these were to be the same then they would be talking about g-knowledge. a justification for j-knowledge could be provided in different ways, namely, post-hoc attribution to training data using attribution methods (hampel, 1974; koh and liang, 2017; pruthi et al., 2020; akyurek et al., 2022), logical derivation with a chain-of-thought mechanism (wei et al., 2022), generation of factual statements with citations to sources (gao et al., 2023; menick et al., 2022; fierro et al., 2024a), or potentially as jiang et al. (2021) proposed, the probability of a calibrated language model could be use as justification to differentiate between mere beliefs and knowledge. in any case, the jury is still out on which justification procedures are valid and/or superior, but note that all these methods seem to require partial interpretability. in recent works, biran et al. (2024) address the intellectual virtue condition to some extent by only analyzing the model's virtue knowledge. they do this by filtering out facts p that the model can correctly predict without using critical components in the input, thereby merely guessing the fact (acting unvirtuous). this is a step in the right direction, but a more in depth detection of the inner workings of the model is necessary to filter out all the non-virtuous predictions. note that if we interpret the detection of a virtue act can be viewed as a model justification and then it is somewhat unclear what would distinguish jknowledge from v-knowledge. this is unsurprising, however, since v-knowledge can be seen as an attempt to flesh out what justification turns on (greco, 1993). as we insist on concrete methodological interpretations, the two definitions of knowledge may coincide. the logically related facts to evaluate could also be defined in terms of task relevance. for example, in the context of legal knowledge, chalkidis et al. (2023) studied the relevance of the knowledge possessed by an llm for downstream performance in legal classification tasks. ## survey results to determine how researchers think about knowledge, we turn to our survey of how computer scientists and philosophers. we had 105 respondents, out of which 50.4% considered themselves philosophers, 36.2% considered themselves computer scientists, 2.3% both, and 10.5% none of the very comprehensive comprehensive neither limited very limited 0.0 0.1 0.2 0.3 0.4 0.5 percentage how would you describe your understanding of epistemology? philosophers computer scientists figure 3: epistemology understanding of respondents. ## best practices given our discussion of mapping knowledge definitions to llms and the results of our survey, we provide possible protocols for evaluating knowledge of llms in relation to each discussed definition. @xcite we also provide a really simple example to contrast in a more practical manner some of the definitions. we use llama-3-8b-instruct foot_15 with greedy decoding for generating completions. @xcite protocol for tb-knowledge a protocol for evaluating knowledge of p as per definition 2.3 would involve evaluating the three conditions for belief + (definition 2.2), which can be done by evaluating model confidence in the true statement itself, as well as in all that follows logically from the true statement. the model should, of course, have low confidence in statements that could imply ¬p. most current work ( §3) evaluates model confidence in p, but to assert tb-knowledge in llms, we must also evaluate model confidence in all that is implied by p. in our small example ( we first test model confidence in the answer to 'are platypuses mammals?' being yes. we then evaluate the epistemic closure by evaluating model confidence in facts that follow logically from the platypuses being a mammal, e.g., 'do platypuses have hair or fur?' for this question, the model has more confidence in the answer yes, they have fur. we now prompt the model 'do mammals lay eggs?', and the model answers no. its answer to 'do platypus lay eggs?' is yes. therefore, the model believes @xmath1, thus violating condition 3 from the belief + definition; leading us to conclude that llama-3 does not tb-know p. @xcite protocol for j-knowledge if we subscribe to jknowledge -which many computer scientists do ( §4) -then we need to have a two part protocol: (1) same as in tb-knowledge the model's confidence in the true statement should be high; and protocol for g-knowledge if by g-knowing p we simply mean the ability to state p, then gknowledge will not do much work for us. on such an account, knowledge becomes indistinguishable from beliefs. in line with our discussion in §3, we generally recommend to adopt other definitions. protocol for v-knowledge the v-knowledge definition seems to be quite popular among both philosophers and computer scientists. in §3, we cited possible interpretations of intellectual virtue in llms. training data reliability assessments could involve attributing the inference of p to training data that contains p, and showing that the model knows this data is reliable, e.g., by using a linear probe to see whether the model successfully distinguishes reliable from unreliable training data. on the other hand, if the model infers p from in-context data that we know is reliable, we need to show that the model is indeed generating the proposition using the provided in-context knowledge, e.g., via mechanistic interpretability (yu et al., 2023; wu et al., 2024). protocol for p-knowledge if knowledge is something that facilitates correct predictions, we need to be able to sample from the set of relevant situations. this is of course a familiar challenge to llm researchers interested in evaluating performance in the wild. we propose to evaluate p-knowledge as we would evaluate tb-knowledge, albeit in a probabilistic setting, and only over the relevant set of implied propositions. @xcite while computer scientists prefer tb-knowledge over p-knowledge (by some margin; see §4), the definition of p-knowledge seems more in line with current practices in the llm community. following with the example in ## conclusion in this paper, we reviewed epistemological definitions and formalized interpretations in the context of large language models (llms). then, we examined how existing works in nlp research align with these definitions, highlighting gaps in their interpretations of knowledge. furthermore, we presented the results of our survey of philosophers and computer scientists, showcasing the different views in terms of definitions of knowledge and whether llms can be said to know. finally, we outlined protocols of evaluations for each knowledge definition using existing algorithms and methodologies. we hope that the connection to epistemological definitions of knowledge can inform the evaluations of knowledge in llms and can provide a more solid foundation for the necessary tests to determine when an llm truly knows a fact.