Challenging BIG -Bench Tasks and Whether Chain-of-Thought Can Solve Them (2023)

Abstract: BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models? In this work, we focus on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the tasks for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average human-rater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.

## introduction scaling language models in terms of model parameters, training compute, and dataset size has led to improved abilities on finetuned tasks @xcite , as well as unseen tasks performed via few-shot prompting @xcite . to measure the capabilities and limitations of large language models, big-bench was introduced as a challenging evaluation suite @xcite . the diversity of this benchmark enables testing the capabilities of language models on a rich set of crowd-sourced tasks. although the benchmark was intended to "focus on tasks beyond the capabilities of current language models" @xcite , @xcite showed that scaling palm to 540b parameters achieves better few-shot performance (with five-shot prompts) than the average score from human-raters over all tasks and outperforms the average human-rater score on 65% of the big-bench tasks. this raises the questions of whether the tasks on which palm performs worse than the average human-rater score are fundamentally not solvable by scaling and/or whether these tasks require a different prompting technique beyond the prompting setup in @xcite . in this paper, we explore both of these questions. we analyze big-bench by identifying tasks for which the highest reported model performances are below the average human-rater score. we select a subset of 23 particularly challenging tasks and group them into an evaluation suite referred to as big-bench hard (bbh), which we make publicly available. as an alternative to the standard prompting setup used in big-bench, we explore whether chain-of-thought prompting (cot; @xcite achieves better performance on bbh. we manually write cot exemplars for bbh and find that combining cot prompting with the strongest openai model (code-davinci-002) surpasses the average human-rater on 17 of 23 evaluation tasks. finally, we study the interaction between scaling and cot prompting across both palm and openai models. this analysis reinforces that performance gains from cot prompting only emerge with sufficient model size, and that cot prompting can enable better than random performance for several bbh tasks that exhibit flat scaling curves. ## big-bench hard big-bench is a collaborative benchmark that aims to quantitatively measure the capabilities and limitations of language models @xcite . the benchmark has over 200 diverse text-based tasks in task categories including traditional nlp, mathematics, commonsense reasoning, and questionanswering. in this paper, we curate big-bench hard (bbh), a subset of 23 particularly challenging big-bench tasks (27 subtasks) for which no prior result from @xcite has outperformed the average human-rater score. in @xcite , the big-bench organizers evaluated performance on each task using several language model families: @xcite , @xcite , palm @xcite , as well as internal dense and sparse models. furthermore, the organizers enlisted a team of raters to manually solve each of the tasks and score them against golden labels to establish human-rater baselines. while the human-rater scores are not representative of the entire population, they express the empirical difficulty of a given task, indicating whether the task is likely to be challenging for language models. given the large number of evaluation tasks in big-bench, the tasks vary in how clean their data and metadata are. for example, some tasks that were added later in the curation process lack human baselines, while other tasks only have the minimum number of examples needed to qualify for submission. we, therefore, choose a set of criteria (listed in these filtering criteria result in 78 clean multiplechoice or exact-match tasks. of these 78 tasks, the best reported model outperforms the average human-rater score on 42 of them, while no previous model surpasses the average human-rater score for the remaining 36. based on manual inspection of 36 these tasks, we divide them into two categories. first, 13 tasks are extremely difficult for authors of this paper; they require domain-specific knowledge or are not practically solvable within twenty minutes. for example, checkmate in one is not doable for non-chess players and requires tracking the state of the board across a long sequence of moves; real or fake text is too challenging due to extremely long inputs; moral permissibility has an ambiguous task formulation (and we did not want to delegate agency to models about moral judgement scenarios @xcite ). we do not think these tasks can be attempted with cot prompting, and we leave them as future work for more powerful models or prompting methods (refer to appendix d for the exact list). we use the remaining 23 tasks as our curated benchmark that we refer to as big-bench hard (bbh). two tasks in this benchmark (namely, logical deduction and tracking shuffled objects) have three subtasks each. for all but three tasks in bbh, we take a random subset of 250 evaluation examples. 2 hence, there are 6,511 evaluation examples in the benchmark in total. 3 # tasks criteria 209 all big-bench tasks 187 -after filtering out tasks with more than three subtasks 130 -after filtering out tasks with fewer than 103 examples (3 for few-shot, 100 for evaluation) 85 -after filtering out tasks without human-rater baselines 78 -after filtering out tasks that do not use multiple-choice or exact match as the evaluation metric 78 clean multiple-choice or exact match tasks 36 -after filtering out tasks in which the best reported model beats average reported human-rater score 23 -after filtering out extremely difficult tasks that are outside the scope of this work 23 remaining @xmath0: if you follow these instructions, do you return to the starting point? turn left. turn right. take 5 steps. take 4 steps. turn around. take 9 steps. options: -yes -no a: let's think step by step. ## experimental setup we evaluate few-shot performance via standard "answer-only" prompting @xcite and chain-of-thought prompting @xcite . we evaluate several model scales for both openai models @xcite @xcite and palm @xcite . few-shot prompting. the baseline approach we consider is few-shot prompting, in which several input-output exemplars are prepended to the prompt before the model must respond to the inference-time example @xcite . we strengthen this baseline by including both a task instruction and answer options, as shown in figure 3 . whereas brown et al. (2020) originally had only brief prefixes describing the prompted task, it has since been shown that including instructions in the prompt can improve performance @xcite current openai api price of $0.02 usd per 1k tokens. 2022; @xcite . moreover, we provide answer options as part of the input prompt, which can improve performance. this follows the recent protocol used in @xcite @xcite and is consistent with empirical work suggesting that language models benefit strongly from knowing the desired output space @xcite . few-shot prompting with chain-of-thought. recent work has suggested that augmenting few-shot exemplars with a chain-of-thought (cot; intermediate reasoning steps) can substantially improve the performance of language models on a range of tasks involving arithmetic reasoning, commonsense reasoning, symbolic manipulation, and program execution @xcite @xcite . we manually compose three chain-ofthought exemplars for each task in bbh; all the prompts used are shown in section c. we prepend ## discussion we first discuss the performance of tasks from bbh presented in ## related work our work is at the intersection of several recent research directions which we briefly recount below. prompting. brown et al. ( 2020 ) demonstrated strong few-shot prompting abilities of gpt-3 on a wide range of nlp tasks. since then, there has been a long line of work proposing new methods for prompting pre-trained language models @xcite @xcite , as well as analysis on why prompting works @xcite @xcite . prompting has also used for many applications both inside nlp and elsewhere, of which a few examples include program synthesis tasks @xcite ; austin et al., ## conclusion in this work, we focused on big-bench hard (bbh), a subset of 23 challenging big-bench tasks on which prior language models fell short of average reported human-rater performance. experiments demonstrate that answer-only prompting underestimates model capabilities and that cot prompting enables the most capable codex model to outperform the average human-rater baseline on 17 out of 23 tasks in bbh. we release the data and prompts used in the work, as well as the codex outputs, to facilitate further research.