LBC : Language-Based-Classifier for Out-Of-Variable Generalization (2025)

Abstract: Large Language Models (LLMs) have great success in natural language processing tasks such as response generation. However, their use in tabular data has been limited due to their inferior performance compared to traditional machine learning models (TMLs) such as XGBoost. We find that the pre-trained knowledge of LLMs enables them to interpret new variables that appear in a test without additional training, a capability central to the concept of Out-of-Variable (OOV). From the findings, we propose a Language-Based-Classifier (LBC), a classifier that maximizes the benefits of LLMs to outperform TMLs on OOV tasks. LBC employs three key methodological strategies: 1) Categorical changes to adjust data to better fit the model’s understanding, 2) Advanced order and indicator to enhance data representation to the model, and 3) Using verbalizer to map logit scores to classes during inference to generate model predictions. These strategies, combined with the pre-trained knowledge of LBC, emphasize the model’s ability to effectively handle OOV tasks. We empirically and theoretically validate the superiority of LBC. LBC is the first study to apply an LLM-based model to OOV tasks. The source code is at https://github.com/MLAI-Yonsei/LBC.git .

## introduction llms have recently been applied to tabular data @xcite @xcite . language-interfaced-fine-tuning (lift) @xcite demonstrated that llms achieve reasonable performance on tabular data tasks while maintaining llm's original structure. however, the pre-trained knowledge of llms holds even more potential: their ability to interpret out-of-variable (oov), which refers to variables that were * corresponding authors figure 1 : (a) illustration of oov task. the variables that were not present in the training data appear in the test data. (b) key components of lbc to increase performance in oov tasks. categorical change refines data to make it easier for lbc to interpret. the advanced order and indicator method enhances the prompts that feed into lbc. the verbalizer aggregates the probabilities for a particular class scattered throughout the logit score and maps them to a specific class. oov tasks are an important problem and are the subject of several ongoing studies @xcite @xcite . however, studies applying llm to tabular data do not handle tabular data in an oov setting. in real-world settings, a variety of constraints often hinder model training, emphasizing the importance of oov tasks. for example, in healthcare, privacy and regulatory barriers prevent data sharing between hospitals. a model trained on hospital a's data may encounter new, unseen variables when applied to hospital b's data, leading to oov situations. we argue that lbc has strengths in handling oov tasks, and our rationale is as follows. converting tabular data to natural language prompts is intuitive, flexible, and easy. this transformation significantly simplifies the handling of oovs, allowing us to seamlessly handle variables that might not have been discovered during training, overcoming a common limitation of tmls. furthermore, lbc leverages the pre-trained knowledge built into llms. unlike tmls, which struggle with data points or scenarios not present in the training set, llms leverage their inherent knowledge. we verified that lbc uses oovs to increase the probability of the correct answer class based on pre-trained knowledge. these advantages are highlighted by the following three methodologies of lbc. first, categorical change involves converting numerical types to categorical types like 'high' and 'low' because these variables align better with the lbc, especially in oov scenarios. second, advanced order and indicators optimize the sequence of variables to generate more effective prompts and introduce indicators to further boost performance. third, the verbalizer focuses on mapping llm's logit scores to the desired class scores rather than relying on inconsistent output text, improving classification performance. we use the low-rank adaptation (lora) @xcite to fine-tune the classifier. we theoretically prove that our model approximates an arbitrary classifier with lora fine-tuning. to the best of our knowledge, lbc is the first study to apply an llm-based classifier to solve the oov tasks, and we validate lbc's superiority empirically and theoretically. ## conclusion in this work, we propose language-based-classifier (lbc) to solve oov tasks. lbc utilizes prompt-based inference, which allows information about oovs to be added to prompts in a straightforward way and enables understanding of the new information through pre-trained knowledge. furthermore, prompt-based tabular data prediction using llms holds even more potential as the reasoning ability of llms continues to improve. lbc's three methodologies maximize the above advantages to achieve high performance on oov tasks. as a result, utilizing llms' pre-trained knowledge is a key strategy for solving the oov task, and we plan to combine it with various statistical methods. lbc is the first approach to apply pre-trained llm to oov tasks.