C o TEV er: Chain of Thought Prompting Annotation Toolkit for Explanation Verification (2023)

Abstract: Chain-of-thought (CoT) prompting enables large language models (LLMs) to solve complex reasoning tasks by generating an explanation before the final prediction. Despite it’s promising ability, a critical downside of CoT prompting is that the performance is greatly affected by the factuality of the generated explanation. To improve the correctness of the explanations, fine-tuning language models with explanation data is needed. However, there exists only a few datasets that can be used for such approaches, and no data collection tool for building them. Thus, we introduce CoTEVer, a tool-kit for annotating the factual correctness of generated explanations and collecting revision data of wrong explanations. Furthermore, we suggest several use cases where the data collected with CoTEVer can be utilized for enhancing the faithfulness of explanations. Our toolkit is publicly available at https://github.com/SeungoneKim/CoTEVer .

## introduction chain-of-thought prompting @xcite ) generates an explanation before the answer to elicit the reasoning capabilities of large language models. an intuitive way to interpret chain-of-thought prompting is that the process of 'generating an explanation' is analogous to 'decomposing multiple step problems into smaller sub-problems', which enables to solve complex reasoning tasks. therefore, generating a plausible explanation is crucial to derive the correct answer @xcite . to generate a plausible explanation, previous works have attempted to generate multiple explanations and use a task-specific verifier that would access the quality of the explanations and choose one of them @xcite @xcite . a more fundamental solution to this problem is fine-tuning the underlying language model with high-quality annotated explanations @xcite figure 1 : example of explanation verification and answer verification of gpt-3's output. explanation verification requires additional knowledge which makes it hard for annotators to intuitively write a revised explanation and answer. @xcite @xcite . however, fine-tuning would require to gather large amounts of annotated explanation data, which is impractical. collecting large amounts of annotated explanation data is difficult for several reasons. first, while existing works gather explanation data by asking annotators to manually write explanations using existing datasets @xcite , gathering human authored labels is often expensive in terms of time and cost @xcite . second, writing a good quality explanation from scratch is difficult because it requires sufficient background knowledge @xcite . in this paper, we address the question: can we gather explanation data in a more efficient manner? inspired by human-in-the-loop methods, we ask annotators to verify a machine generated explanation instead of manually writing them @xcite @xcite . in other words, annotators get to check whether the underlying language model hallucinate (i.e., generate explanations that are factually incorrect) @xcite . to do this, we provide a set of supporting evidence documents retrieved from the web. annotators access the quality of the given explanation, and provide a feedback score along with a better alternative. as shown in figure 1 , let's consider gathering an explanation and answer for the question, 'can you see harbor seals in washington d.c.?' 1 . in this example, gpt-3 generates an explanation '1) you can see harbor seals in the pacific ocean. 2) washington d.c. is not in the pacific ocean. 3) therefore you cannot see harbor seals in washington d.c.' and predicts 'no' as the answer. in this case, the first sentence of the explanation missed the point that harbor seals not only live in the west coast, but also in the east coast of the us. by providing the background knowledge 'harbor seals live in east and west coasts of united states', annotators could successfully revise the explanation. to this end, we propose cotever (chain of thought prompting annotation toolkit for explanation verification), which is designed to efficiently gather explanation data, by 1) alleviating the role of annotators to verify instead of writing from scratch and 2) supplementing the required background knowledge via evidence documents. with the gathered explanation data, researchers could use them for cot fine-tuning @xcite or transform them into other knowledge intensive datasets. ## system design and workflow in figure 2 , we present an illustration of the overall explanation verification process of cotever with 3 steps and show how the annotated explanations could be obtained effectively. we assume a scenario where a researcher requests a group of annotators to query a large language model and you can see harbor seals along the u.s. west and east coasts. suggest a better alternative for step1! 1 2 3 4 5 q: where can you see harbor seals? a: you can see harbor seals in the pacific ocean. is step 1 reasonable? (1 to 5) is the final answer reasonable? (1 to 5) rate this explanation! can you see harbor seals in washington d.c.? step 1 q: where can you see harbor seals? a: you can see harbor seals in the . pacific ocean step 2 q: is washington d.c. in the pacific ocean? a: washington d.c. is not in the pacific ocean. step 3 q: can you see harbor seals in washington d.c.? a: since you can see harbor seals in the pacific ocean, and washington d.c is not in the pacific ocean, you cannot see harbor seals in washington d.c. user query s1: prompting s3: explanation and answer verification so the answer is no. document 1 harbor seals are among one of the most common marine mammals they are commonly seen resting on rocks and baches alone the coast and on floating ice in glacial ... url: https://www.fisheries.noaa.gov/ species/harbor-seal along the u.s. west and east coasts. document 2 harbor sea' s are brown, silvery white, tan, or gray with distinctive v-shaped nostrils. males may fight over mates under water and on land. females bear a single pup after a nine-month gestation, which they care for alone ... url: https:/ www.marinemammalcenter.org/ animal-care/learn-about-marine-mammal... s2: evidence retrieval 1 2 3 4 5 so the answer is yes. ## analysis of explanation data in this section, we analyze what error cases are abundant within an explanation and show how they can be revised using evidence documents retrieved by cotever. as mentioned in section 3.1, we adopt a self-ask style prompt and use text-davinci-002 @xcite to generate a corresponding explanation and answer for the train set of strategyqa @xcite . then, we sample 300 instances where the prediction is incorrect, ask annotators to classify the error type and revise the explanation using cotever. while we analyze the error types of explanations using human evaluation, automatic evaluation metrics proposed to measure the quality of a given explanation @xcite is another promising direction, and we leave for future work. also, we provide more detail of the human evaluation experiment process in appendix b. ## how to utilize explanation data gathered with cotever in this section, we suggest three promising directions on how the explanation data collected with cotever can be utilized. we define e and a to be the original explanation and answer generated by a language model, respectively. similarly, the revised explanation and answer from the annotator can be defined as e * and a * . explanations consist of pairs of sub-questions sq i and sub-answers sa i which brings the following definition: now for an explanation, sets of documents d i are retrieved for each pair e i , based on sq i . within d i , we define the top-k th document aligned by the re-ranking module as d k i . finally, di is defined as the evidence document chosen by the annotator upon the set d i . ## conclusion in this work, we introduce cotever, an interactive annotation framework designed to verify unfaithful outputs and gather truthful explanation data from annotators. to reduce the cost of manually searching for evidence while verifying an explanation, we provide supporting evidence documents via a search engine. next, we analyze some of the abundant reasons where large language models generated incorrect explanations. also, we suggest three directions on how explanation data gathered with cotever can be utilized. we hope cotever will contribute to gather high quality explanation data used for future research.