The C o T Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning (2023)

Abstract: Language models (LMs) with less than 100B parameters are known to perform poorly on chain-of-thought (CoT) reasoning in contrast to large LMs when solving unseen tasks. In this work, we aim to equip smaller LMs with the step-by-step reasoning capability by instruction tuning with CoT rationales. In order to achieve this goal, we first introduce a new instruction-tuning dataset called the CoT Collection, which augments the existing Flan Collection (including only 9 CoT tasks) with additional 1.84 million rationales across 1,060 tasks. We show that CoT fine-tuning Flan-T5 (3B & 11B) with CoT Collection enables smaller LMs to have better CoT capabilities on unseen tasks. On the BIG-Bench-Hard (BBH) benchmark, we report an average improvement of +4.34% (Flan-T5 3B) and +2.60% (Flan-T5 11B), in terms of zero-shot task accuracy. Furthermore, we show that instruction tuning with CoT Collection allows LMs to possess stronger few-shot learning capabilities on 4 domain-specific tasks, resulting in an improvement of +2.24% (Flan-T5 3B) and +2.37% (Flan-T5 11B), even outperforming ChatGPT utilizing demonstrations until the max length by a +13.98% margin. Our code, the CoT Collection data, and model checkpoints are publicly available.

## introduction language models (lms) pre-trained on massive text corpora can adapt to downstream tasks in both zero-shot and few-shot learning settings by incorporating task instructions and demonstrations @xcite @xcite @xcite @xcite @xcite . one approach that has been particularly effective in enabling lms to excel at a multitude of tasks is chain-of-thought (cot) prompting, making lms generate a rationale to derive its final prediction in a sequential manner @xcite @xcite @xcite . while cot prompting works effectively for large lms with more than 100 billion parameters, it does not necessarily confer the same benefits to smaller lms @xcite @xcite . the requirement of a large number of parameters consequently results in significant computational cost and accessibility issues @xcite @xcite @xcite . recent work has focused on empowering relatively smaller lms to effectively solve novel tasks as well, primarily through fine-tuning with rationales (denoted as cot fine-tuning) and applying cot prompting on a single target task @xcite @xcite . however, solving a single task does not adequately address the issue of generalization to a broad range of unseen tasks. while chung et al. (2022) leverage 9 publicly available cot tasks during instruction tuning to solve multiple unseen tasks, the imbalanced ratio compared to 1,827 tasks used for direct finetuning results in poor cot results across smaller lms @xcite . in general, the community still lacks a comprehensive strategy to fully leverage cot prompting to solve multiple unseen novel tasks in the context of smaller lms. to bridge this gap, we present the cot col-lection, an instruction tuning dataset that augments 1.84 million rationales from the flan collection @xcite across 1,060 tasks. we fine-tune flan-t5 (3b & 11b) using cot col-lection and denote the resulting model as cot-t5. we perform extensive comparisons of cot-t5 and flan-t5 under two main scenarios: (1) zeroshot learning and (2) few-shot learning. in the zero-shot learning setting, cot-t5 (3b & 11b) outperforms flan-t5 (3b & 11b) by +4.34% and +2.60% on average accuracy across 27 datasets from the big bench hard (bbh) benchmark @xcite when evaluated with cot prompting. during ablation experiments, we show that cot fine-tuning t0 (3b) @xcite on a subset of the cot collection, specifically 163 training tasks used in t0, shows a performance increase of +8.65% on average accuracy across 11 datasets from the p3 evaluation benchmark. moreover, we translate 80k instances of cot collection into 5 different languages (french, japanese, korean, russian, chinese) and observe that cot fine-tuning mt0 (3b) @xcite on each language results in 2x ∼ 10x performance improvement on average accuracy across all 5 languages from the mgsm benchmark @xcite . in the few-shot learning setting, where lms must adapt to new tasks with a minimal number of instances, cot-t5 (3b & 11b) exhibits a +2.24% and +2.37% improvement on average compared to using flan-t5 (3b & 11b) as the base model on 4 different domain-specific tasks 2 . moreover, it demonstrates +13.98% and +8.11% improvement over @xcite and claude @xcite ) that leverages icl with demonstrations up to the maximum input length. our contributions are summarized as follows: ## the cot collection despite its effectiveness for cot fine-tuning, rationale data still remains scarce. to the best of our knowledge, recent work mostly rely on 9 publicly available nlp datasets foot_0 for fine-tuning with rationales @xcite @xcite @xcite . this is due to the difficulty in gathering human-authored rationales @xcite . to this end, we create cot collection, an instruction-tuning dataset that includes 1.84 million rationales augmented across 1,060 tasks foot_1 . in this section, we explain the datasets we select to augment into rationales and how we perform the overall augmentation process. broad overview given an input @xmath0, z] composed of an instruction i, and an instance z along with the answer y, we obtain a rationale r by applying in-context learning (icl) with a large lm. note that this differs from previous works which focused on generating new instances z using large lms @xcite @xcite @xcite @xcite while we extend it to generating new rationales r. ## analysis of of cot fine-tuning in this section, we conduct experiments to address the following two research questions: bbh peromance (cot evaluation) 9 cot data (180k instances) cot collection (10k instances) cot collection (100k instances) cot collection (1.84m instances) 40.33 42.04 44.76 47.60 bbh performance (cot eval) based on number of tasks / instances ## conclusion in this work, we show that augmenting rationales from an instruction tuning data using llms (open ai codex), and cot fine-tuning could improve the reasoning capabilities of smaller lms. specifically, we construct cot collection, a largescale instruction-tuning dataset with 1.84m cot rationales extracted across 1,060 nlp tasks. with our dataset, we cot fine-tune flan-t5 and obtain cot-t5, which shows better zero-shot generalization performance and serves as a better base model when training with few number of instances. we hope cot collection could be beneficial in the development of future strategies for advancing the capabilities of lms with cot fine-tuning. this work was partly supported by institute of information & communications technology planning & evaluation (iitp) grant funded by the korea government (msit) (no.2022-0-00264, comprehensive video understanding and generation with knowledge-based deep logic neural network, 50%; no.2021-0-02068, artificial intelligence innovation hub, 20%) and kaist-naver hypercreative ai center.