A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024)

Abstract: Prompting language models to provide step-by-step answers (e.g., “Chain-of-Thought”) is the prominent approach for complex reasoning tasks, where more accurate reasoning chains typically improve downstream task performance. Recent literature discusses automatic methods to verify reasoning to evaluate and improve their correctness. However, no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods, hindering progress in this direction. We introduce REVEAL: Reasoning Verification Evaluation, a dataset to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings. REVEAL includes comprehensive labels for the relevance, attribution to evidence passages, and logical correctness of each reasoning step in a language model’s answer, across a variety of datasets and state-of-the-art language models. Evaluation on REVEAL shows that verifiers struggle at verifying reasoning chains - in particular, verifying logical correctness and detecting contradictions. Available at https://reveal-dataset.github.io/ .

## introduction complex reasoning tasks involve answering questions that require multiple steps of reasoning @xcite . addressing these questions may require open-domain knowledge @xcite , mathematical reasoning @xcite , logic @xcite , and so on. reasoning chainsbreaking the task into multiple steps explicitly-is useful for improving performance in such tasks, with lms demonstrating better performance when encouraged to generate the reasoning chain behind their answer @xcite @xcite , commonly implemented via chain-of-thought (cot) prompting @xcite , see fig. 1 for an example). evaluation in such settings is traditionally limited to evaluating only whether the final answer is correct @xcite @xcite figure 1 : we collect reveal, an evaluation benchmark for the task of verifying reasoning chains in chainof-thought format, which checks whether a reasoning chain is a correct justification to the final answer (importantly, the answer can be correct even if the reasoning is incorrect, as in the example above). the figure shows four verifiers (middle) verifying the correctness of a cot (left). we use the dataset to benchmark multiple verifiers (right). @xcite . however, correct reasoning chains have been shown to be correlated with better final answers @xcite , with recent literature proposing automatic methods for verifying the quality of the reasoning chains themselves along various axes such as informativeness, relevance, factuality and logical correctness @xcite @xcite . while such verification methods are a promising direction for improving reasoning in llms, it is not clear how to evaluate them due to the lack of high-quality, step-level annotated data, and collecting such data was shown to be difficult (in terms of reaching high inter-annotator agreement) and costly @xcite . we present reveal (reasoning verification evaluation), an evaluation benchmark for complex reasoning verifiers. foot_0 reveal covers a diverse set of reasoning skills, complexity levels, and knowledge domains. it contains 704 unique questions from 4 popular qa datasets, and 1,002 cot answers generated by 3 language models, consisting of 3,360 cot steps in total. each step is first labeled for relevance with respect to the final answer, and then whether the step is an attribution step (introduces factual knowledge which can be attributed to a source), a logical step (introduces logical inference from previous steps) or both. for attribution steps, we collect labels for correctness to retrieved wikipedia paragraphs given as evidence (with full support, partial support, contradiction, or no-support as labels). for logical steps, we label for logical correctness. each label includes free-text justifications written by the annotators. an illustrative instance from the dataset is shown in figures 3 and 4 . we split the dataset into reveal-eval, the main evaluation benchmark containing high inter-annotator-agreement labels, and reveal-open, a smaller set of interesting borderline cases with open labels due to low interannotator agreement. in §5 we describe the dataset, and report fine-grained analyses of non-attributable steps in reveal-eval (i.e., evidence that supports or contradicts them was not found) and of disagreement categories in reveal-open. reveal supports versatile evaluation settings, for example: (1) attribution steps, along with their evidence, can serve as a high-quality natural language inference @xcite benchmark in a setting of factchecking lm outputs @xcite ; (2) cot verifiers can be evaluated question: can an art dealer buy boeing 737-800 with a da vinci painting? ## annotation schema we introduce a labeling schema for reasoning chain verification, according to the formalization in §2. ## data collection process in this section, we describe the full process for constructing reveal, using our annotation schema. the process is divided into two phases: collecting and generating data for annotation (steps 1-2) and the annotation process (step 3). ## reveal we ran the data collection protocol described in the previous section to obtain our dataset, reveal. in terms of inter-annotator agreement, we report a krippendorf's α of 0.49 for attribution steps and 0.46 for logical steps. we split reveal into two subsets, as we observe that some reasoning chains are very challenging to annotate because of ambiguity or other factors (analyzed in §5.2). any cot answer with at least one step that has low inter-annotator agreement is treated as an indecisive case, which applies to 18% of the cot answers. a step is considered to have "low inter-annotator agreement" if less than three annotators agree on any single label for attribution or logical correctness (for example, an attribution step labeled as "partially attributable" by two annotators, "fully attributable" by one annotator, and "contradictory" by two annotators). we release these examples separately in a data subset called reveal-open, as they are valuable instances of difficult or borderline cases, and the rest of the examples in a subset called reveal-eval. since the annotations in reveal-open are indecisive, in §6 we evaluate only on reveal-eval. tab. 1 details statistics on the two data splits, showing that reveal-eval has several hundreds to thousands of examples for each setting to support reliable evaluation. instances are approximately distributed uniformly across source datasets and lm generators. we release all five annotations for each label in the dataset, including anonymized annotator identifiers for each label, to support future annotation methodology research and reproducibility @xcite . considering the label distributions, 98.6% of steps are relevant to answer the question, and 87.5% of logic steps are logically correct. fig. 5 (left) shows label distributions for attribution steps and step-evidence pairs (note that partial attribution is considered as "not enough info" in standard nli formalization). while 43.8% of attribution steps were fully attributable, this is still relatively low, which may point at attribution as a main obstacle in reasoning. however, we do note that unsupported steps are not necessarily incorrect, as lack of support may stem from wrong claims, but also from issues in retrieving relevant supporting evidence. we analyze this further in §5.1. fig. 5 (right) shows the distribution for the required number of evidence paragraphs for finding support or contradiction for each step: in the majority of cases where support was found, it was already present in the first retrieved paragraph. fig. 6 shows distributions for full chain-level attribution and correctness for each subset of the data. in terms of specific types of errors in the cots: 77.3% of cots in reveal-eval have a step which is not fully attributable, while 18.5% have a 6 4620 step which is logically incorrect. ## experiments we use reveal to evaluate existing methods for cot verification, on step-level verification ( §6.1) and cot-level verification ( §6.2). verifiers. we use flan-ul2 (20b), flan-palm (540b), palm-2-l (anil et al., 2023) and gpt-3 (text-davinci-003) for lm baselines in few-shot prompting settings (prompt templates detailed in §b). in addition, we use two specialized baselines: a t5-based model with 11b parameters trained on a mixture of nli datasets @xcite , and factool-a gpt-3-based fact-checking pipeline @xcite . for the prompting-based baselines, we use the label generated by the lm as the predicted class (which always matched to one of the task labels in our experiments). the nli classifier receives as input a premise and hypothesis and predicts a score between 0 and 1 that indicates if the hypothesis is entailed by the premise or not. factool returns a binary factuality classification label, and we use it evaluation. as few-shot evaluation is noisy @xcite , we average the predictions of the lm baselines over 5 different 8-shot prompts, sampled in random order from 13 demonstrations for each sub-task, which we wrote for this purpose (the number of demonstrations is trimmed if it exceeds a given model's context length). all sets of demonstrations were class-balanced and balanced across source datasets and labels. prompt templates are available in §b. for factool, we insert the appropriate evidence from the dataset, instead of allowing factool to retrieve evidence separately, for compatibility with the attribution labels. we release the prompt demonstrations alongside the dataset. ## conclusion we design a methodology for human verification of reasoning chains, and employ it to annotate a dataset of cot-format reasoning chains generated by three lms in open-domain complex-reasoning qa tasks. the dataset, reveal, can be used to benchmark automatic verifiers of lm reasoning. our work advances the research towards lms that can provide correct and attributable reasoning behind their decisions. we find that cots generated by lms are often not fully correct, and that automatic verifiers struggle to verify them appropriately. in particular, between attribution and logic types of correctness, cot-generating lms struggle more with attribution, but in contrast, cot verifiers struggle more with verifying logical correctness.