Context-faithful Prompting for Large Language Models (2023)

Abstract: Large language models (LLMs) encode parametric knowledge about world facts and have shown remarkable performance in knowledge-driven NLP tasks. However, their reliance on parametric knowledge may cause them to overlook contextual cues, leading to incorrect predictions in context-sensitive NLP tasks (e.g., knowledge acquisition tasks). In this paper, we seek to assess and enhance LLMs’ contextual faithfulness in two aspects: knowledge conflict and prediction with abstention. We demonstrate that LLMs’ faithfulness can be significantly improved using carefully designed prompting strategies. In particular, we identify opinion-based prompts and counterfactual demonstrations as the most effective methods. Opinion-based prompts reframe the context as a narrator’s statement and inquire about the narrator’s opinions, while counterfactual demonstrations use instances containing false facts to improve faithfulness in knowledge conflict situations. Neither technique requires additional training. We conduct experiments on three datasets of two standard NLP tasks, machine reading comprehension and relation extraction, and the results demonstrate significant improvement in faithfulness to contexts. Code and data are released at https://github.com/wzhouad/context-faithful-llm.

## introduction large language models (llms; @xcite @xcite ) have made remarkable advances in solving various nlp problems, particularly in (context-free) knowledge-driven tasks such as question answering @xcite and commonsense reasoning @xcite . without external context, llms can answer factual questions and achieve comparable results to supervised approaches @xcite , in-1 code and data are released at https://github.com/ wzhouad/context-faithful-llm . context: elon musk is a business magnate and investor. he is the owner and ceo of twitter. question: who is the ceo of twitter? answer: elon musk gpt-3.5: jack dorsey ## related work we discuss two topics of related work that are closely relevant to this work. knowledge conflicts. llms @xcite @xcite have shown promising results in closed-book qa tasks, indicating their ability to memorize facts about the world. however, as the world is constantly evolving, memorized facts may become outdated @xcite @xcite , emphasizing the need to update llms' predictions with new facts. to address this challenge, some studies @xcite @xcite @xcite have explored ways to identify and edit the facts stored in model parameters. however, it remains unclear whether memory editing methods allow sufficient capacity to encompass all new factual knowledge. another promising direction is to augment llm prompting with external context containing relevant knowledge @xcite @xcite . coupled with retrieval systems @xcite @xcite , such methods have the potential to update llms with large amounts of new facts. however, such methods face the challenge that llms may persist with the memorized facts and ignore the provided context @xcite . to tackle this challenge, recent works @xcite fine-tune llms on counterfactual contexts, where the original facts are replaced with counterfactual ones. they find that such finetuning processes can effectively improve the llms' utilization of contexts. in this study, we propose a novel approach using prompting to improve context faithfulness in llms without additional finetuning, which offers a more general and cost-effective method for llms. prediction with abstention. selective prediction with abstention @xcite @xcite is an important problem in trustworthy ai. when models are uncertain about their predictions, it is critical that they should admit the uncertainty instead of returning incorrect predictions. selective prediction may be adopted in different scenarios, such as when instances are close to the decision boundary @xcite @xcite , or when instances are from different domains to training @xcite @xcite . in the scope of context-specific nlp, abstention is preferred when the context is irrelevant to the question. for example, squad 2.0 @xcite introduces unanswerable questions to extractive mrc, while @xcite finds it focused on questions of extreme confusion and thus is less relevant to the focus of our study. coqa @xcite and quac @xcite introduce unanswerable questions to conversational question answering. realtime qa @xcite finds that gpt-3 still generates outdated answers when provided with irrelevant documents. to address the problem, @xcite propose the answerability augmentation where llms should predict unanswerable when presented with an empty or randomly sampled document. several other work employ variants of confidence calibration techniques to encourage the nlp model to avoid giving a high confidence on any decisions when encountering a case to abstain @xcite , which however request white-box accessibility of the incorporated models. we tackle this problem with a part of our prompting method, which we find to significantly enhance the llms' ability to make selective predictions without need re-calibration or white-box accessibility of the model. ## method we focus on context-specific nlp tasks. the input of these tasks is formulated as (c, q) for free-form generation tasks, where c is the context and q is the question, or (c, q, o) for tasks with close decision spaces (e.g., multi-choice tasks), where o is the set of decisions/choices. the desired output can be either a free-form text or a choice. we solve these tasks by prompting llms and study ways of designing prompting templates and demonstrations that are dedicated to improving the faithfulness of llms. specifically, we find two proposed methods, opinion-based prompts and counterfactual demonstrations, to be the most effective ones. our methods only change the prompts without finetuning the llms @xcite @xcite , targeting a more general and affordable solution. ## experiments this section presents our experimental setups ( §4.1) for the evaluation of the proposed methods concerning two aspects of faithfulness: knowledge conflict ( §4.2) and prediction with abstention ( §4.3). we provide additional analysis ( §4.4) on results across different model sizes and results on the original datasets. we also show examples of prompts and llms' outputs in the case study ( §4.5). ## conclusion in this paper, we focus on addressing the faithfulness issue of llms in context-specific nlp tasks, particularly in scenarios with knowledge conflict and prediction with abstention. we propose that two methods, opinion-based prompts and counterfactual demonstrations, are effective in improving llms' faithfulness to contexts. we evaluate our methods on three datasets of two tasks, namely machine reading comprehension and relation extraction, and observed significant improvement in faithfulness to contexts. future work includes evaluating the effectiveness of proposed methods on a broader range of nlp tasks such as open-domain qa and summarization, and studying other techniques to improve faithfulness further.