I t D : Large Language Models Can Teach Themselves Induction through Deduction (2024)

Abstract: Although Large Language Models (LLMs) are showing impressive performance on a wide range of Natural Language Processing tasks, researchers have found that they still have limited ability to conduct induction. Recent works mainly adopt “post processes” paradigms to improve the performance of LLMs on induction (e.g., the hypothesis search & refinement methods), but their performance is still constrained by the inherent inductive capability of the LLMs. In this paper, we propose a novel framework, Induction through Deduction (ItD), to enable the LLMs to teach themselves induction through deduction. The ItD framework is composed of two main components: a Deductive Data Generation module to generate induction data and a Naive Bayesian Induction module to optimize the fine-tuning and decoding of LLMs. Our empirical results showcase the effectiveness of ItD on two induction benchmarks, achieving relative performance improvement of 36% and 10% compared with previous state-of-the-art, respectively. Our ablation study verifies the effectiveness of two key modules of ItD. We also verify the effectiveness of ItD across different LLMs and deductors. The data and code of this paper can be found at https://github.com/forangel2014/ItD.

## introduction induction can take we humans from the observed to the unobserved @xcite . the task of induction aims to discover consistent transformations from a set of input-output pairs, where the transformations map the inputs to the outputs well @xcite . as shown in figure 1 , given the input-output pairs {x i , y i } n @xmath0, the model ## framework in this section, we introduce itd, a framework for empowering the induction capability of llms through their own deduction capability. as shown in figure 3 , itd is composed of two modules: deductive data generation, and naive bayesian induction. for a given induction task, deductive data generation will first leverage the deductive capability of the llms to generate the task data. then we propose naive bayesian induction to allow llms to optimize the use of each observed sample, while taking advantage of the increase in the number of observed samples. ## conclusion in this paper, we propose a novel framework, itd, to enable llms to teach themselves induction through deduction. we conduct a series of experiments on two types of induction datasets and verify that itd is superior to existing methods in empowering the inductive capability of llms. moreover, we verify the effectiveness of deductive data generation and naive bayesian induction. more experiment results support that itd can be effectively applied to llms of different sizes, and a more powerful deductor can further improve the performance of itd.