Can Large Language Models Understand Context? (2024)

Abstract: Understanding context is key to understanding human language, an ability which Large Language Models (LLMs) have been increasingly seen to demonstrate to an impressive extent. However, though the evaluation of LLMs encompasses various domains within the realm of Natural Language Processing, limited attention has been paid to probing their linguistic capability of understanding contextual features. This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. This benchmark comprises of four distinct tasks and nine datasets, all featuring prompts designed to assess the models’ ability to understand context. First, we evaluate the performance of LLMs under the in-context learning pretraining scenario. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. Second, as LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings. We find that 3-bit post-training quantization leads to varying degrees of performance reduction on our benchmark. We conduct an extensive analysis of these scenarios to substantiate our experimental results.

## introduction discourse understanding, as one of the fundamental problems in nlp, focuses on modeling linguistic features and structures that go beyond individual sentences @xcite . understanding discourse requires resolving the relations between words/phrases (coreference resolution) and discourse units (discourse parsing and discourse relation classification) in the previous context, iden-tifying carry-over information for the following context (dialogue state tracking), and recognizing discourse-specific phenomena (ellipsis). llms have garnered substantial attention from both academia and the industry due to their remarkable capability in comprehending language and world knowledge. their unparalleled performance across a diverse range of benchmarks and datasets has firmly established their significance in a relatively short period of time. as llms continue to push the boundaries of scale and capability, the evaluation of their multifaceted abilities becomes an equally vital endeavor. consequently, the development of robust evaluation methodologies to assess specific aspects of llms becomes imperative. in addition, these methodologies should focus on helping achieve a comprehensive understanding of their advancement while clearly delineating their limitations. however, recently published llms, such as opt @xcite , llama @xcite and @xcite , are only evaluated on limited benchmarks, and have a significant drawback: they neglect the inclusion of discourse-related datasets for evaluation, thereby limiting the comprehensive assessment of their language understanding capabilities. to provide a comprehensive evaluation, plenty of benchmarks and datasets address various facets of language understanding, including benchmarks that delve into common sense knowledge @xcite , as well as linguistic capabilities like sentiment analysis, natural language inference, summarization, text classification, and more @xcite . these general benchmarks and specific dataset evaluations exhibit certain limitations. despite the requirement for contextual information in these benchmarks to effectively tackle tasks (for example, sentiment analysis requires an understanding of polarities within the given text), none of these benchmarks cater to tasks that demand a nuanced comprehension of linguistic features within a provided context. on the other hand, recent llms, by virtue of possessing billions of parameters, have led to an exponential surge in computational and storage costs @xcite , which hinders the deployment of large models to personal devices and restricts the on-device performance of language understanding tasks. to address this challenge, model compression methods, which can reduce memory and disk requirements of both model training and inference, have gained attention. existing compression techniques, such as 3-bit quantization @xcite , have demonstrated the potential to reduce model sizes with only marginal performance trade-offs. however, the evaluation of quantization methods suffers from two deficiencies. firstly, quantization methods are primarily evaluated on limited benchmarks and datasets, such as lambada @xcite , arc @xcite , piqa @xcite , boolq @xcite , and storycloze @xcite . it is not yet clear whether large, compressed models out-or under-perform their smaller counterparts when understanding context. secondly, previous work has not delved into a linguistic analysis to identify where the model efficacy wanes. given the above shortcomings, this paper evaluates llms on a context understanding benchmark constructed from varied discourse understanding datasets. we conduct an extensive analysis of llm performance on this benchmark, including models of varying sizes and those subjected to compression techniques, aiming to provide a more comprehensive understanding of context understanding capability of the llms. the contributions of this paper can be summarized as follows: ## task selection & design our contextual understanding benchmark includes four tasks with nine datasets, as presented in figure 1 . in the following sections, we provide detailed explanations of each task and the corresponding datasets, along with the designed prompts for icl evaluations. ## case study: query rewriting in this section, we provide in-depth analysis by comparing the two open-sourced model families opt and llama, and the impact of quantization, using query rewriting as the target task. we conduct a careful inspection of the query rewriting task because of three reasons: (1) by the nature of the task, query rewriting is the only one with free-form generation, while the others effectively are either classification-based tasks or heavily constrained in their possible output predictions. the generation task allows us to explore the llms' output in more detail, and to provide more interesting insights; (2) the manual analysis of errors is a time-consuming process, making it challenging to conduct such an in-depth analysis across all four tasks; (3) the query rewriting task covers a diverse range of five datasets, enabling us to compare differences between each dataset and to thereby gain a deeper understanding. ## conclusion this paper introduces a contextual understanding benchmark designed to assess the performance of llms. we collect nine existing datasets spanning four tasks, each carefully tailored to suit generative models. this benchmark encompasses essential elements for assessing linguistic comprehension within context, including both document and dialog based contextual understanding. experimental results reveal that llms under in-context learning struggle with nuanced linguistic features within this challenging benchmark, exhibiting inconsistencies with other benchmarks that emphasize other aspects of language. to the best of our knowledge, we are also the first to compare dense models and post-training quantization models in contextual understanding tasks. this comparison highlights that 3-bit post-training quantization reduces the general understanding capacity of context to different extent across the 4 tasks. the proposed contextual comprehension benchmark thus provides a unique perspective on the contextual dimension of language understanding and offers a valuable addition to existing llm evaluations.