Summarize-Exemplify-Reflect: Data-driven Insight Distillation Empowers LLM s for Few-shot Tabular Classification (2025)

Abstract: Recent studies show the promise of large language models (LLMs) for few-shot tabular classification but highlight challenges due to the variability in structured data. To address this, we propose distilling data into actionable insights to enable robust and effective classification by LLMs. Drawing inspiration from human learning processes, we introduce InsightTab, an insight distillation framework guided by principles of divide-and-conquer, easy-first, and reflective learning. Our approach integrates rule summarization, strategic exemplification, and insight reflection through deep collaboration between LLMs and data modeling techniques. The obtained insights enable LLMs to better align their general knowledge and capabilities with the particular requirements of specific tabular tasks. We extensively evaluate InsightTab on nine datasets. The results demonstrate consistent improvement over state-of-the-art methods. Ablation studies further validate the principle-guided distillation process, while analyses emphasize InsightTab’s effectiveness in leveraging labeled data and managing bias.

## introduction tabular data is widely used across applications due to the prevalence of relational databases @xcite , inspiring tasks like tabular question answering @xcite , retrieval @xcite , etc. among these tasks, tabular classification plays a key role by categorizing tabular data with numerical and categorical features into predefined classes. traditional tabular classification tasks leverage ensemble techniques with decision trees such as xgboost to handle mixed numerical and categorical data @xcite . the high cost of labeling large datasets and feature engineering has driven interest in few-shot learning, * corresponding author where models make accurate predictions using only a few examples @xcite . these approaches often leverage large language models (llms), which enhance prediction accuracy by transforming structured data into natural language sequences, enabling llms to process and make informed predictions. they show that combining proper serialization with llms achieves strong performance, with later studies refining this through expert and algorithmic prompting @xcite . however, several challenges persist for effective llm-based tabular classification in the few-shot setting. these include: (1) filling knowledge gap. tabular data often includes task-specific details (e.g., task description), failing to incorporate them can hinder the performance of general-purpose llms. (2) unlocking llms' full potential. effective tabular classification requires complex processing. while methods like in-context learning help, they fall short of fully utilizing llms' capabilities, making this an ongoing challenge. (3) balancing performance and cost. tabular classification tasks are critical for real-time services, demanding fast response times and cost efficiency. traditional methods, while effective, are often time-intensive to train. thus, llm-based solutions must be both efficient and economical to meet these demands. to address these challenges, we focus on the task of llm-based tabular classification in the few-shot setting. we pose a fundamental research question: can actionable insights be distilled from the training data to enhance llms' performance in this task? in response, we introduce insight-tab, a novel insight distillation framework that integrates traditional machine learning techniques with llms, enabling them to acquire task-specific knowledge from a data modeling perspective. in this analogy, llms can be seen as students with broad general knowledge but limited task-specific expertise. inspired by human learning processes, insighttab employs three key principles: divideand-conquer, easy-first, and reflective learning. the divide-and-conquer strategy, inspired by curriculum learning @xcite , partitions training data into subsets, enabling llms to derive rules for each, i.e., a cognitive process akin to summarization. this could simultaneously avoid context window overflow and yield more targeted rules than directly feeding all data into an llm. the easy-first principle mimics the human cognitive procedure of exemplifying, prioritizing the selection of representative and simpler samples for initial learning @xcite . finally, the reflective learning strategy allows llms to engage in reflection, i.e., learning from misclassified cases to enhance their distilled insights. we then use three operators for the insight distillation process: group for clustering similar samples, rank for example ordering with prediction difficulty, and summarize for extracting natural language rules. this process fosters collaboration between data modeling methods and llms, leveraging llms' strength in summarization while addressing their limitation in data analysis. we validate insighttab through experiments on nine diverse datasets, demonstrating its superiority over state-of-the-art methods in average performance. additionally, insighttab outperforms other approaches using the same base llms in 19 out of 20 comparisons. this highlights the importance of insight distillation for few-shot tabular classification. ablation and case studies further confirm the effectiveness of each module, and tests on biased data show insighttab's priority in robustness, reducing overfitting and improving generalization. in summary, our contributions are threefold: llms for tabular data. language models have demonstrated effective capabilities in handling structured data @xcite . extensive studies have explored pretraining language models to effectively learn structured table representations, ranging from smaller-scale pretrained models (e.g., bert, gpt-2) @xcite @xcite to state-of-the-art llms @xcite @xcite . for example, zhang et al. (2023) fine-tune llama 2 to develop open-source llms capable of handling a variety of table-based tasks. since llms are designed for natural language, converting tabular data into text enables them to analyze and generate insights effectively @xcite . this process, known as data serialization, typically involves linearizing tables row by row with column separators @xcite . for example, @xcite find that creating incontext learning prompts which include instruction helps improve text-to-sql performance. manikandan et al. (2023) are the first to generate summaries and shows that incorporating task-specific knowledge during serialization can improve tabular classification. building on their findings, our work age education gain 39 bachelor 2174 36 hs-grad 0 64 12th 0 29 doctorate 1086 42 master 594 structured tabular dataset @xmath0. the education is bachelor. the gain is 2174. the age is 36. the education is bachelor. the gain is 2174. the age is 64. the education is bachelor. the gain is 2174 the age is 29. the education is bachelor. the gain is 2174. ## methodology in this section, we describe the structure of insight-tab, which integrates data modeling techniques with llms for better tabular classification. an overview of our approach is presented in figure 1 . ## in-depth analysis we conduct a series of in-depth analysis on the task, which could deepen our understanding for this emerging paradigm. additionally, we put detailed cost analysis in appendix d. ## conclusion we propose insighttab, a novel method for accuarte and robust llm-based few-shot tabular classification. by distilling training data into actionable insights, insighttab adapts llms to tabular tasks via the divide-and-conquer, easy-first, and reflective learning principles, implemented through group, rank, and summarize operators. they enable grouping samples to derive rules, selecting representative instances, and learning from errors to enhance performance. we conduct experiments on nine datasets, comparing our method with stateof-the-art approaches. our method consistently achieves superior average performance across all nine datasets. extensive analysis shows that our model is more robust to positional and class biases.