Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Models (2025)

Abstract: Recent advancements in large language models (LLMs) have shown promise in feature engineering for tabular data, but concerns about their reliability persist, especially due to variability in generated outputs. We introduce a multi-level diagnosis and evaluation framework to assess the robustness of LLMs in feature engineering across diverse domains, focusing on the three main factors: key variables, relationships, and decision boundary values for predicting target classes. We demonstrate that the robustness of LLMs varies significantly over different datasets, and that high-quality LLM-generated features can improve few-shot prediction performance by up to 10.52%. This work opens a new direction for assessing and enhancing the reliability of LLM-driven feature engineering in various domains.

## introduction recent breakthroughs in large language models (llms) have opened new possibilities in tabular learning, such as feature engineering, question answering, and table comprehension @xcite . the extensive pretrained knowledge of llms, when equipped with only a few examples, can automate costly data science workflows manually handled by domain experts. notably, recent studies have shown that llm-driven feature engineering can help outperform traditional tabular prediction methods, especially in a few-or zeroshot settings @xcite @xcite . despite these promising results, the open-ended nature of llm-generated outputs has raised concerns about their robustness @xcite . existing approaches for feature generation focus on either feature-feature relationships with predefined operators @xcite or feature-target relationships with unbounded rule conditions @xcite . they rely on in-context learning by llm with arbitrary domain knowledge and a few samples, which inherently entails risks of inconsistency and unreliability in outputs. accordingly, evaluating the reliability of llm-generated features remains a significant challenge. for example, in figure 1 , given the task of predicting diabetes for a new patient, an llm is asked to generate a set of new features describing featuretarget relationships (i.e., between patient information and diabetes). an ensemble classifier then uses these new features to make a final prediction. the llm-generated features produced by the state-ofthe-art approach @xcite , however, could be ineffective (e.g., '@xmath0. this variability can introduce noise into the resulting prediction probabilities, potentially degrading the overall classifier performance. for more practical and reliable llm-generated features, it is crucial to understand the consistency of their performance on feature engineering under varying contexts. although significant strides have been made in evaluating the robustness of llm @xcite , there remains insufficient exploration of these aspects in feature engineering, especially in the context of feature-target relationships. a recent work elf-gym @xcite has attempted to compare llm-generated features with humancrafted ones, but further investigation is lacking regarding the capabilities and limitations of llms with varying domain knowledge and examples. to address this gap, we propose a framework that systematically diagnoses and evaluates the robustness of llms in feature engineering for tabular data. we focus on how consistently llms maintain reliability in engineering features for tabular prediction, the most prevalent task in tabular learning. specifically, drawing inspiration from real-world practices of domain experts, we identify three core elements considered in feature engineering: golden variable, golden relation, and golden value. our framework incorporates a novel multi-level scheme to analyze llm-generated features, specifically addressing the following research questions. figure 2 shows the overall procedure of the proposed diagnosis and evaluation framework. based on the multi-level scheme, we first conduct a reliability diagnosis to assess the consistency in llm responses across varying contexts at each level. this serves as a fine-grained proxy to measure the trustworthiness of an llm in generating features for a given dataset. the robustness of an llm given a dataset directly influences the quality of the generated features; less robust models may produce features of varying quality, leading to prediction performance degradation. thus, we further conduct an evaluation on the generated features to investigate how high-quality features can enhance the effectiveness of llm-driven feature engineering and ultimately improve the prediction performance. ## conclusion we present a multi-level framework for evaluating the robustness of llms in tabular feature engineering. our analysis reveals that the few-shot prediction performance of llms varies significantly across different datasets, highlighting the need for consistent and reliable methods in real-world applications. by focusing on golden variables, relations, and values, we demonstrate that high-quality features generated by llms can lead to substantial performance improvements. our findings emphasize the importance of robustness in llm-driven feature engineering and provide valuable insights for enhancing its reliability and effectiveness.