Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell (2024)

Abstract: Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. Our study explores LLMs’ long-context reasoning by probing their hidden representations. We find that while LLMs encode the position of target information, they often fail to leverage this in generating accurate responses. This reveals a disconnect between information retrieval and utilization, a “know but don’t tell” phenomenon. We further analyze the relationship between extraction time and final accuracy, offering insights into the underlying mechanics of transformer models.

## introduction the advent of transformer-based large language models (llms) has delivered marked improvement in language processing capabilities @xcite . these models excel at simultaneously processing extended contexts @xcite , significantly benefiting various downstream tasks like long-text question answering, summarization, and inference @xcite @xcite . despite their advanced capabilities, llms often struggle to utilize long inputs fully. this tendency, known as positional bias, leads llms to disproportionately prioritize information at the beginning or end of the input sequence @xcite , while crucial details in the middle are frequently overlooked @xcite . numerous strategies have been proposed to address these biases @xcite @xcite , yet the underlying causes and potential solutions remain unclear. this underscores the need for a deeper investigation into how llms handle long-context integration. to fully assess the capabilities of llms in handling extended contexts, it is not enough to merely evaluate their final performance: some important information is hidden in models' representations. this work presents a probing analysis of llms' long-context generalization. we develop probes based on llms' internal representations for various layers and positions to measure the accuracy of reconstructing the position they correspond to (fig. 1 ). our working hypothesis is that, for llms to effectively process long contexts, they must encode positional information in their intermediate representations. we conduct experiments on two tasks from @xcite and three recent open-source models. our findings reveal a gap between the accuracy of llms' generations and the probes on their repre-sentations. notably, while llm representations identify the position of crucial information within the context (surfaced via probes), they often fail to utilize this information effectively in their responses, leading to what we term the 'know but don't tell' phenomenon. to our knowledge, this is the first work to use probing analysis to highlight this observation. we hope that our work on distinguishing "knowing" and "telling" motivates future work on tackling llms' long-context challenges. in summary, our contributions are as follows: (1) probing analysis: we introduce a novel framework to investigate the long-context reasoning capabilities of llms. this framework allows us to measure how accurately llms encode positional information across various layers and positions within their intermediate representations. (2) empirical evaluation: we conduct comprehensive experiments using tasks from @xcite and three recent open-source models. our empirical evaluation provides new insights into the positional biases of llms and their impact on model performance. (3) 'know but don't tell' phenomenon: our analysis reveals a critical gap between llms' ability to encode and utilize positional information. we identify the "know but don't tell" phenomenon, where llms accurately identify the position of crucial information but fail to leverage this knowledge in generating accurate responses. by distinguishing between the encoding and utilization of positional information, our work lays the foundation for future advancements in llm performance and reliability. probing. probing classifiers are extensively used to elucidate the inner workings of llms @xcite @xcite @xcite @xcite . various works train probes on model representations to assess how well they encode various linguistic features, such as phrase-level, syntactic, and semantic information @xcite @xcite . the efficacy of a classifier in a given task indicates the degree to which that layer successfully captures pertinent information. our study employs probing as a proxy to determine whether the llms accurately identify and represent crucial parts of the context. ## experimental setup we conduct a layer-wise probing analysis to determine if the model accurately identifies target information from a given prompt. we expect that higher probing accuracy shows a stronger connection between the model's hidden representations and its internal understanding of the target information. datasets and prompts. we follow the datasets and prompts used by @xcite . our datasets include: (1) key-value pairs retrieval (kv-pairs) where the context contains a collection of keys and their corresponding values (128-bit randomly generated uuids). the goal of this task is to identify a value given its key. each prompt for this task contains 100 kv-pairs and a target key. (2) multi-document question answering (mdqa) where the context contains multiple sets of evidence paragraphs. the goal of this task is to, given a question, identify the relevant document and produce an answer. each prompt for this task contains 30 documents, and a target question. given a set of key-value pairs/documents, with only one containing target information, the llm is prompted to output the value/answer given the key/question. details (e.g., prompts) are in §a. probing classifiers. for each input prompt, we extract the last token embedding x l from each layer l. we then train a separate linear classifier for each layer l, characterized by weight w l and bias b l . the classifier takes x l as input to predict the gold kv-pair or document id (position among all pairs/documents). the classifier aims to minimize the following objective: models and hyperparameters. we employ two sota models, llama3-8b-instruct (ai@meta, 2024) and mistral-7b-instruct-v0.3 @xcite to do our probing analysis. the results for the third model, gemma @xcite , are provided in the appendix due to space constraints. to minimize uncertainty from random initialization, each classifier is trained ten times. we report the mean and standard deviation (error bars) from these ten independent experiments. as we observe in §4, consistency across the three models confirms that the 'know but don't tell' phenomenon is not specific to a single model's architecture. following liu et al. (2023b) , we use accuracy to evaluate our models' success. we measure two types of accuracy: (a) generation accuracy, which quantifies how well llms generate the correct value in kv-pair retrieval or the correct answer string in mdqa, and (b) probing accuracy, which evaluates how accurately classifiers predict (a) llama3-8b-instruct 5 10 15 20 25 30 layer 0% 20% 40% 60% 80% 100% accuracy 100 kv-pairs probing accuracies across layers gold kv-pair at id 1 gold kv-pair at id 50 gold kv-pair at id 100 5 10 15 20 25 30 layer 0% 20% 40% 60% 80% 100% accuracy 30-documents probing accuracies across layers gold doc at id 1 gold doc at id 15 gold doc at id 30 (b) mistral-7b-instruct-v0.3 5 10 15 20 25 30 layer 0% 20% 40% 60% 80% 100% accuracy 100 kv-pairs probing accuracies across layers gold kv-pair at id 1 gold kv-pair at id 50 gold kv-pair at id 100 5 10 15 20 25 30 layer 0% 20% 40% 60% 80% 100% accuracy 30-documents probing accuracies across layers gold doc at id 1 gold doc at id 15 gold doc at id 30 the gold kv-pair or document id, quantifying the extent to which layers encode information from the input context. ## conclusion our study demonstrates that while llms effectively capture context internally, they often fail to generate the correct answer due to positional bias. through probing experiments, we reveal that information from the middle of the context requires deeper layers for retrieval. additionally, we found that the delayed extraction of this information leads to a noticeable drop in accuracy. we hope these findings shed more light on the underlying cause behind positional bias and point to new areas for improving long-range information processing in llms.