Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding (2024)

Abstract: Large language models (LLMs) tend to inadequately integrate input context during text generation, relying excessively on encoded prior knowledge in model parameters, potentially resulting in generated text with factual inconsistencies or contextually unfaithful content. LLMs utilize two primary knowledge sources: 1) prior (parametric) knowledge from pretraining, and 2) contextual (non-parametric) knowledge from input prompts. The study addresses the open question of how LLMs effectively balance these knowledge sources during the generation process, specifically in the context of open-domain question answering. To address this issue, we introduce a novel approach integrating contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation. Notably, our method operates at inference time without requiring further training. We conduct comprehensive experiments to demonstrate its applicability and effectiveness, providing empirical evidence showcasing its superiority over existing methodologies.

## introduction improving large language models (llms) has been a primary focus in natural language processing research. recent strides have incorporated retrieval mechanisms to enhance llms @xcite @xcite , augmenting their ability to produce contextually relevant and precise responses @xcite . retrievalaugmented llms, which leverage both parametric knowledge acquired during training and nonparametric knowledge retrieved during inference, exhibit potential in addressing challenges such as limited memorization @xcite , knowledge conflicts @xcite , and outdated information @xcite . )) √ z relocated its capital from nanjing to taipei. a republic was formally established on 1 january 1912 ... from its founding until 1949 it was based on mainland china ... relevant context c + irrelevant context c - e sour el ghozlane entente de sour el ghozlane, known as e sour el ghozlane or simply esg for short, is an algerian football club based in sour el-ghozlane in bouira province. the club was ... query x what is the capital of republic of china ? ## experimental setup the present study revolves around open-domain question answering, which involves tasking models to generate responses to factual questions in natural language. specifically, we concentrate on the openbook qa setting @xcite , where we harness non-parametric knowledge by supplying relevant contexts along with the question itself to the model during inference. consistent with prior investigations, we utilize prompting techniques to assess the models' performance. ## results we present the results of models featuring the largest variants in ## conclusion this study introduces a novel decoding strategy, employing contrastive decoding to incorporate relevant and irrelevant context. through diverse experiments and analyses across datasets and model scales, our approach consistently outperforms regular decoding methods. notably, it excels in managing knowledge conflicts, surpassing both regular decoding and cad. moreover, our exploration of retrieval sources underscores the need for refining these modules to enhance efficacy. the demonstration of the method's effectiveness in open-domain question answering also sets the stage for future research. the method's versatility suggests potential applications in various generative tasks, motivating our future exploration in tasks like summarization.