Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022)

Abstract: While Pre-trained Language Models (PLMs) internalize a great amount of world knowledge, they have been shown incapable of recalling these knowledge to solve tasks requiring complex & multi-step reasoning. Similar to how humans develop a “chain of thought” for these tasks, how can we equip PLMs with such abilities? In this work, we explore an iterative prompting framework, a new prompting paradigm which progressively elicits relevant knowledge from PLMs for multi-step inference. We identify key limitations of existing prompting methods, namely they are either restricted to queries with a single identifiable relation/predicate, or being agnostic to input contexts, which makes it difficult to capture variabilities across different inference steps. We propose an iterative context-aware prompter, which addresses these limitations by learning to dynamically synthesize prompts conditioned on the current step’s contexts. Experiments on three datasets involving multi-step reasoning show the effectiveness of the iterative scheme and the context-aware prompter design.

## introduction humans can develop a "chain of thought" for complex decision making. for example, when asked the question (q) shown in figure 1 , which involves composition, an important type of multi-step reasoning, humans apply two consecutive steps to derive the final answer: 1) find "father" of the topic entity "gwilym lloyd george" (c1); 2) find "birthplace" of the entity returned in the first step (c2). recently, large-scale pre-trained language models (plms) have been shown capable of internalizing a great amount of simple factual knowledge such as c1 and c2, yielding competitive performance on a range of knowledge-intensive tasks without resorting to any external knowledge source in standard probing, a question is directly fed to the plm to output the final answer, which could work for simple factual questions but fails for complex questions that require multi-step reasoning. in contrast, we augment the plm with a prompter, which learns to iteratively prompt the plm to recall a series of knowledge and derive a "chain of thought". @xcite @xcite @xcite . however, work such as @xcite @xcite reveals that plms face difficulties in complex, multi-step reasoning. for example, they struggle with answering complex questions like q without using external sources, no matter whether they are fine-tuned based on qa pairs or simply prompted to produce the answer (where even if they have memorized c1 and c2). in this paper, we study the following question: how to shepherd a plm to recall a series of stored knowledge (e.g., c1 and c2) that is necessary for multi-step inference (e.g., answering q), analogous to how humans develop a "chain of thought" for complex decision making? a direct way would be to fine-tune the plm to generate the series of knowledge all at once (assuming such supervision is available), but soon one realizes the practical issue in this approach: plms which internalize a great amount of knowledge are inevitably large in scale, and fine-tuning all their parameters would become more and more costly as they keep scaling up. there is also the concern that fine-tuning plms may interfere with their implicit knowledge storage, a phenomenon observed in @xcite which is more generally related to the catastrophic forgetting problem of deep learning models @xcite @xcite . therefore, lightweight methods such as prompting @xcite which keep a plm's parameters intact would be preferable for our purpose of eliciting knowledge. however, we find that no matter whether it is fine-tuned or prompted to generate the series of knowledge all at once, the plm tends to lose its "chain of thought" during the process, generating irrelevant facts or suffering from hallucination. motivated by the iterative nature of multistep reasoning problems, we explore an iterative prompting framework in this paper, which elicits knowledge from plms step by step for a given inference task. we have two desiderata in iterative prompting: (1) at different inference steps, the prompts need to focus on different components of the complex query. (2) the prompts should appropriately integrate knowledge gathered in previous steps into the current step; for instance, during the 2nd step in the example in figure 1 , the prompts need to combine the entity "david lloyd george" (from knowledge recalled in the 1st step) with the unresolved part "what is the place of birth of" in the query. a natural thought is to directly apply existing prompting methods in an iterative fashion. unfortunately, their prompts are either restricted to queries with a single, identifiable relation/predicate @xcite @xcite @xcite , or being agnostic and insensitive to step-wise inputs @xcite @xcite , and hence not ideal for our desiderata. we design a novel iterative prompting method towards that end. we augment the plm with an iterative context-aware prompter, a model which learns to dynamically synthesize prompts based on the current step context. at each step, the prompter learns to process the query and previously gathered evidence, and composes a prompt which steers the plm to recall the next piece of knowledge. like other prompting methods, the plm is kept fixed throughout the learning process. in addition, as the plm size increases, the number of trainable parameters in our method scales comparably with or slower than previous prompting methods. we conduct experiments on three datasets involving multi-step reasoning, including two recent multi-hop question answering datasets: 2wiki-multihopqa @xcite and r4c @xcite , and a scientific dataset @xcite for reasoning over taxonomic relations. our experimental results show (1) effectiveness of the iterative scheme ; (2) our proposed context-aware prompter design outperforms existing prompting methods by notable margins; (3) quantitative and qualitative analysis which reveal the faithfulness of our learned prompter. ## methodology in this section, we first formalize our problem setup ( §2.1), and then introduce our iterative prompting framework ( §2.2), followed by our context-aware prompter design ( §2.3) which addresses key limitations of previous prompting methods when applied in this iterative scheme. ## experimental setup our research question is how to shepherd a plm to recall a series of knowledge and derive a "chain of thought" for multi-step reasoning. to this end, we conduct experiments on several datasets that require complex multi-step reasoning and compare different methods to guide the plm via prompt/prefix tuning, fine-tuning, and our prompter design. we use both intrinsic and extrinsic metrics to evaluate the quality of recalled knowledge, considering both end answer accuracy and coverage of intermediate evidence. ## conclusion & future work we explore an iterative prompting framework towards driving a "chain of thought" from plms for multi-step reasoning tasks. we show the superiority of this iterative scheme, and also the effectiveness of our proposed context-aware prompter design, which addresses key limitations of previous prompting methods when applied in this new scheme. in addition, we conduct both quantitative & qualitative analysis on the faithfulness of the learned prompting behaviors. in the future, we aim to further extend and apply our ideas to language model pretraining, with the hope that plms can be inherently equipped with stronger multi-step reasoning capabilities. the iterative framework we explore here also opens the possibility of human intervention and interaction during inference; namely, a human can track along the plm's chain of thought and make edits and corrections at different steps, similarly as in @xcite , which improves the transparency and trustworthiness of inference and also helps reduce error propagation along the reasoning process. we leave these investigations as future work.