Chain of Thought Prompting Elicits Knowledge Augmentation (2023)

Abstract: The knowledge-augmented deep learning paradigm refers to a paradigm in which domain knowledge is identified and integrated into deep models. Conventional methods typically employ task-specific approaches to gather external knowledge from various sources. In contrast, large language models are extensively pre-trained and can serve as a comprehensive source of external knowledge. In this paper, we propose CoT-KA, a Chain-of-Thought-based method that augments knowledge for deep learning. CoT-KA avoids the need for additional knowledge retrieval or knowledge reasoning models, as required in conventional augmentation methods. Our results demonstrate that CoT-KA outperforms both pure CoT-based methods and the non-augmented method across the majority of eleven publicly available benchmarks for various reasoning tasks.

## introduction the knowledge-augmented deep learning (kadl) @xcite paradigm refers to the deep learning paradigm in which domain knowledge is identified and integrated into the deep model. adding domain knowledge makes it possible to develop deep learning that is data-efficient, generalizable, and interpretable @xcite . for example, retrieving external knowledge from an external knowledge pool like wikipedia is typically required for open domain question answering and dialog generation @xcite . logical equivalence laws such as contraposition and transitive laws help extend the implicit logical information @xcite . external knowledge is derived from various sources. for instance, commonsense knowledge can be extracted from commonsense knowledge bases like conceptnet @xcite and atomic @xcite . domain-specific knowledge can be retrieved from knowledge bases such as wikipedia and freebase @xcite . logic knowledge, on the other hand, can be in the form of human-defined propositional or first-order logic, which is then utilized as rules for reasoning. in summary, existing knowledge augmentation methods typically involve either creating a retriever to gather relevant knowledge or developing a reasoner to leverage the logical rules within the external knowledge sources @xcite @xcite . recently, large language models (llms) @xcite have shown their potential as both the source and the retriever or reasoner of external knowledge. llms are pre-trained on a huge scale of datasets. thus, they have already embedded a large amount of knowledge into their parameters, which can be considered a source of external knowledge. the reasoning ability of llms allows them to provide knowledge from their parameters without needing an extra retriever or a reasoner. the latest chain-of-thought (cot) prompting technique @xcite , which elicits llms to generate a series of sentences that mimic the reasoning process for arriving at the answers, improves the reasoning ability of llms. it has proved to be remarkably effective in a variety of complex reasoning tasks such as math word problems and commonsense question answering @xcite . cot prompting shows potential as a general technique to retrieve knowledge from llms. in this paper, we propose cot-ka -a cotbased method to retrieve knowledge from llms for knowledge-augmented deep learning. cot-ka utilizes an llm as a knowledge source, leveraging cot prompting to guide the llm in providing knowledge that can serve as evidence to support downstream reasoning from the input to the answer. unlike conventional kadl approaches, cot-ka eliminates the need for additional knowledge retrieval or a separate knowledge reasoning model. specifically, we begin by extracting cots as knowledge from the llm using either few-shot @xcite or zero-shot @xcite cot prompting. the former involves providing a few demonstrations to guide the llm's reasoning, while the latter employs a template such as "let's think step by step" to inspire the llm. the extracted cots are then appended to the original inputs, marked by a special token, to create augmented text. finally, we fine-tune a small taskrelevant pre-trained language model (plm) on the dataset augmented with cots. we generate cots using the public gpt-3 @xcite ) (175b parameters) api foot_1 . for nlu (natural language understanding) tasks, we employ albert @xcite and deberta @xcite as the task-relevant models. t5 @xcite is utilized as the task-relevant model for nlg (natural language generation) tasks. we evaluate models' performance using eleven benchmarks, including (i) commonsense reasoning (csqa @xcite , strategyqa @xcite , date understanding, sports understanding @xcite ); (ii) arithmetic reasoning (aqua-rat @xcite , gsm8k @xcite , svamp @xcite , multiarith @xcite , singleeq @xcite , addsub (hosseini et al., 2014)); (iii) symbolic reasoning (last letter concatenation @xcite ), where all commonsense reasoning benchmarks and aqua-rat are formulated as nlu tasks, and the other arithmetic reasoning benchmarks and last letter concatenation are formulated as nlg tasks in this paper. particularly, we convert all of the multichoice question answering tasks into nlu tasks. extensive experimental results show that in the majority of tasks, cot-ka outperforms the original fine-tuning results without the use of cots as augmented knowledge. cot-ka also surpasses few-shot-cot and zero-shot-cot on llms, which directly parse answers from the generated cots. fusion-in-decoder and lreasoner inspire our work to extend the external knowledge into the original input. however, the knowledge in these knowledge augmentation methods is sourced from external knowledge bases or pre-defined logical rules, requiring a retriever for knowledge extraction or a reasoner for rule application in the process. in contrast, we utilize llms that eliminate the need for an additional retriever or reasoner to acquire knowledge for augmentation. ## pilot study in this section, we explore the effectiveness of cotaugmented fine-tuning by simply appending one cot to the original input. we assess the validity of this approach on two commonsense reasoning datasets, csqa and strategyqa. ## cot-ka in this section, we propose cot-ka -a cot-based method for knowledge augmentation. our method leverages multiple cots retrieved from llms to provide more auxiliary knowledge for kadl. cot-ka consists of three steps as shown in figure 3: (1) cot generation: generating multiple cots for each sample in the train, dev, and test sets. (2) input augmentation: taking the generated cots as the additional knowledge into the original input text for each sample. (3) task-relevant model training: fine-tuning a task-relevant model using the cotaugmented samples.