Token Knowledge: A New Perspective For Knowledge in Large Language Models (2025)

Abstract: In the era of prosperity of large language models (LLMs), hallucination remains a serious issue hindering LLMs’ expansion and reliability. Predicting the presence (and absence) of certain knowledge in LLMs could aid the hallucination avoidance. However, the token-based generation mode of LLM is different from the knowledge storage structure in the form of triples, which makes it difficult to accurately evaluate the knowledge boundary of LLM. We approach this problem from a novel perspective and, for the first time, introduce the concept of token knowledge in large language models. Consequently, we propose a token knowledge dataset construction method and use the intermediate states during inference to train probes. This allows us to predict if a specific token will appear in the LLM’s generated sequence, without even generating a single token. Our approach unlocks the model’s latent potential, enhancing its accuracy in assessing token knowledge from about 60% to over 90%, with strong out-of-distribution generalization by training on just a few dozen prompts. Finally, we apply KEGT to enhance a state-of-the-art knowledge boundary detection method, achieving improved performance while reducing computational time by over 90%. Furthermore, KEGT enables prevention of hallucinations in certain cases by leveraging its guidance in the token-level knowledge semantic space. Our code is available at https://github.com/CC-2000/KEGT.

## introduction assessing the knowledge stored in large language models (llms) remains a significant challenge @xcite . the common method for verifying llm knowledge involves querying models with fact-based prompts and many studies have employed similar approaches, including comparing responses to gold standard labels @xcite , evaluating semantic consistency in paraphrased responses @xcite , and cross-referencing answers with external evidence. previous research, regardless of assessing an llm's mastery of a particular subject @xcite or its understanding of various relational templates @xcite , or comparing the generated objects to gold labels, all evaluates an llm's higher-level knowledge, which we refer to as abstract knowledge. as shown in figure 1 , given that the text generated by llms is based on tokens as fundamental units @xcite , higher-level entity or relation knowledge is more abstract and harder to obtain compared to token-level knowledge for llms. our main standpoint is that an llm's token-level knowledge is easier-both more accurately and more efficiently-to be obtained as a stepstone for high-level abstract knowledge. thus, unlike prior work, this study aims to focus on token-level knowledge in llms, specifically, predicting whether a model will generate a given token without actually generating a single token. we show that our approach enables a more accessible analysis of abstract knowledge and has potential applications in probing model knowledge boundaries, preventing hallucinations, and mitigating the generation of biased or harmful contents. we define this problem as token knowledge estimation (section 2). while directly generating text and verifying the presence of specific tokens in the output is the most accurate method, the time required for generating long sequences is a significant concern. the time complexity of text generation increases non-linearly with sequence length, due to the attention mechanisms in transformers, which results in o(n 2 ) time complexity @xcite . consequently, time efficiency is a crucial factor for token knowledge estimation, and any approach we consider should at least outperform the direct generation method in terms of speed. to address the token knowledge estimation task, we build on prior work exploring the interpretability of internal states in llms. research has shown that certain internal representations during llm inference exhibit interpretable features @xcite , which can be captured using linear probes @xcite . based on this, we propose kegt (knowledge estimation of generated token), a probing that utilizes these internal representations to predict the likelihood of a token appearing in the generated sequence, even without requiring to generate a single token. as a result, kegt provides a significant efficiency advantage over traditional text generation methods. to implement kegt, we refine the kass dataset @xcite by selecting 2,992 distinct knowledge facts and 48,770 unique prompts for experiments. we evaluate kegt in both indistribution and out-of-distribution settings, where it consistently achieves over 0.9 accuracy. then we test kegt with minimal training data and find that it converges and remains highly accurate with just one knowledge fact (≤30 prompts). this shows kegt captures generalizable features for predicting token appearances in future outputs. evaluations on eight llms and four datasets confirm these features are consistently present and effectively captured by kegt. additionally, we apply kegt to explore the knowledge boundaries of llms. compared to the latest methods for knowledge boundary detection @xcite , our approach is over ten times faster. finally, we conduct intervention experiments to verify that kegt can proactively prevent the hallucination, and perform case studies to demonstrate its effectiveness. overall, our main contributions are as follows: ## token knowledge estimation llms generate knowledge based on tokens as fundamental units, while higher-level entity and relation knowledge is more abstract to the model. aligned with @xcite , which highlights the inadequately defined knowledge boundary of llms, understanding entity knowledge without exploring token-level mechanisms is challenging. thus, this work focuses on measuring token-level knowledge in llms. most methods study abstract knowledge by checking whether model responses consistently include answer entities. similarly, our approach investigates whether a specific token appears in the model's response to a given prompt. the formal definition of this task is as follows: given a prompt p and a specific token t x , the response @xmath0, t 2 , . . . , t m ] generated by llm m consists of m tokens. the goal of token knowledge estimation is to determine whether the token t x will appear in the token sequence r. specifically, the task is to identify whether there exists an i such that 1 ≤ @xmath1, @xmath2, and t @xmath3. the most direct and accurate method is to input p into m , control m to generate sequences of length m, and then traverse each token in the sequences to check whether t x appears. however, this approach is time-consuming, as the generation time increases with sequence length, and this increase is not linear-the time complexity of the attention mechanism is o(m 2 ) @xcite . thus, time effciency is a crucial factor for token knowledge estimation and an optimal solu-tion should take less time than directly generating a response of length m from m for the prompt p. ## methodology previous studies have demonstrated that the hidden states in llms capture rich task-specific information @xcite . however, none has explored whether this information also pertains to token knowledge. to address this gap, we first investigate the potential presence of such information using a carefully curated dataset. we create new prompts (appendix a) by combining a base prompt p with a specific token t x , then analyze the llm's hidden states during inference to determine if they contain task-related features. we select 10 factual prompts as inputs to llama3.1-8b @xcite , generate 20-token responses and manually review these responses to identify key words as t x . to facilitate differentiation, we categorize these tokens as t pos and t neg , where t pos is a token that appears in the response and t neg is one that does not. for example, for the prompt "hepatitis b vaccine is used to immunize against", the response is "hepatitis b virus infection. it is a vaccine that is used to prevent hepatitis b. it is a". we choose t @xmath4, creating a triplet (p, t pos , t neg ). similarly, we replace "may" with "november" and "different" with "same" to create negative tokens, for example. the complete data is provided in appendix b. for carefully prepared data, as shown in figure 2(a), we collect hidden states of m and apply pca for dimensionality reduction @xcite . the resulting two principal components (pcs) are then normalized to a vector with a modulus of 1 @xcite , as shown in figure 2(b) . a clear linear separation is observed between positive and negative samples. upon normalization-focusing on distribution rather than absolute distance-negative samples tend to cluster on the right, while positive samples appear on the left. this suggests that the llm contains general features that effectively capture token knowledge for the token knowledge estimation task. ## experimental setup data we validate token knowledge in llms using kass with diverse paraphrasing, standard qa datasets with varying styles (nq and webqa), and human-written review-style data from amazon. the detailed construction process is provided in appendix c. ## results and analysis in this section, we evaluate the performance of kegt and baselines in both in-distribution and outof-distribution settings, demonstrating that kegt's extracted features are general and dataset-irrelevant. we also show that kegt converges with minimal training data. in layer-based experiments, our findings align with previous knowledge storage studies, with explanations for observed differences. finally, we validate kegt's effectiveness in llm knowledge boundary detection task and illustrate its application through a case study. ## related work to the best of our knowledge, we are the first to investigate token knowledge in llms. we present related work from two perspectives: the knowledge embedded within llms and the probing techniques employed to explore such knowledge. knowledge in llms. llms possess internal knowledge that support complex reasoning and downstream tasks. to identify this knowledge, @xcite introduced lama dataset and probing methods, initially applied to simple masked lms. recognizing the variability in llm responses, @xcite and @xcite assessed knowledge reliability using semantic consistency across multiple generations. @xcite and @xcite highlighted inconsistencies due to prompt variations, prompting greater focus on prompt paraphrasing. @xcite introduced kass dataset, with diverse entity aliases and relation templates. however, @xcite argued that the infinite semantic space of paraphrasing cannot be fully captured, proposing an algorithm for constructing special prompts. @xcite used an auxiliary llm to predict response probabilities with a scoring mechanism and @xcite guides the chat model to recall intrinsic knowledge based on the direction indicated by the sparse autoencoder to accept or reject the answer. while these studies focus on abstract, high-level knowledge, our research targets token-level knowledge within llms, which is more finer-grained and intuitive. notably, @xcite also investigated the relationship between llms and the generated tokens. however, they directly predicted the next four specific tokens, which proved to be a difficult task. in contrast, we shift the perspective: rather than predicting the exact future tokens, we predict whether a certain token will appear in the future-an approach that has never been attempted before. probing in llms. probing is a key tool in llm interpretability, revealing that intermediate representations during inference capture features relevant to specific tasks, which can be detected by lightweight probes @xcite . probes have been used to assess various llm capabilities, such as content accuracy @xcite , truthfulness @xcite , and bias detection @xcite . further, probes have traced the propagation of bias @xcite and identified falsehood generation @xcite . while these studies focus on high-level capabilities, our research specifically investigates token-level knowledge within llms, offering insights into their strong generalizability in probing tasks. ## conclusion in this paper, we present a novel perspective on the study of knowledge in llms: token-level knowledge. we construct a dataset for token knowledge estimation task and develop kegt, a probe that leverages intermediate inference states. kegt reveals llms' latent ability to recognize token knowledge, boosting accuracy from 60% to over 90%. it generalizes well to out-of-distribution data, requires minimal training data, and offers fast training and inference. finally, we apply kegt to enhance the latest knowledge boundary detection method, improving performance while reducing computation time by over 90% and prevent hallucinations in certain situations in advance based on the directions indicated by kegt in the token knowledge semantic space.