LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
Abstract
Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks.
However, for non -language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions.
For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively.
A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function?
To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks.
LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling “no-code machine learning with LMs.
” We find that LIFT performs comparably well across a wide range of lowdimensional classification and regression tasks, matching the performances of the best baselines in many cases, especially for the classification tasks.
We also report experimental results on the fundamental properties of LIFT, including the inductive bias, robustness, and sample complexity.
We also analyze the effect of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine-tuning.
Our code is available at https://github. com/UW-Madison-Lee-Lab/LanguageInterfacedFineTuning .
Introduction
Deep neural networks have been highly successful across a multitude of domains, from computer vision [1, 2] and natural language processing [3, 4], to game playing [5, 6].
Most advances in deep learning have come with a variety of domain-specific designs for network architectures, such as convolutional filters [7, 8, 9] for vision tasks, or recurrent modules [10, 11] and attention mechanisms [12, 13] in the context of natural language processing.
A domain-and-modality agnostic model that can be adapted to solve tasks across different modalities and domains has become a desideratum [14], motivating great efforts in transfer learning [15] and multi-modal learning [16].
Recently, transformer-based language models (LMs) [13, 17, 18, 19] exhibited impressive versatility across different domains and modalities.
They have shown great performances for various language-based tasks [20] such as question answering [21, 22], or commonsense reasoning [23].
They have also been applied to non-language modalities [18].
For instance, GPT-2 [17] pretrained on language data can be efficiently fine-tuned to perform image classification and numerical computation [18].
When downstream tasks are language-based tasks, adapting pretrained LMs can be achieved without modifying the models’ architecture.
Typically, this adaptation is enabled via simple fine-tuning [24,
LIFT has a two-phase procedure: (1) converting the dataset into sentences and (2) fine-tuning the pretrained language model ( e.g. , GPT) on the obtained sentences.
This figure visualizes how LIFT can be applied to the Iris classification task.
We first convert the Iris dataset into plain English sentences (left).
Since feature names and the task description are available for this task, one could incorporate them as part of the prompt (as option 1 in the figure).
(In Sec. 4.1, we show that adding such contextual information to prompts helps LIFT achieve higher predictive accuracy.)
One may also choose to use a simpler prompt with a generic naming convention ( x 1 , x 2 , . . .
, x d ) for p features (as option 2 in the figure).
After the sentence conversion step, LIFT fine-tunes a pretrained LM with the sentence set without making any changes to model architecture or loss.
At inference time, we convert the test samples to a sentence form using the same prompt, excluding the label part.
LIFT performs surprisingly well in various non-language regression/classification tasks, and we summarize our main findings in Table 3.
Note that to obtain a model for a given task, all we need here is to design proper sentence templates for LIFT and no changes to architecture or loss functions are needed.
25, 26, 27] or in-context few-shot learning methods [28, 29].
However, not altering the architecture may pose a limitation for transferring to non-language tasks.
As their input and output formats are not in some language form, adapting LMs to these domains may seem to require architectural changes.
Indeed, it has been a common practice to re-design the input/output layers and loss functions to accommodate a different predictive task.
For instance, to adapt GPT-2 [21] to other modalities, the frozen pretrained transformer [18] adds new input/output layers to handle different types of input/output.
To make such changes, one must have a good understanding of the underlying principles of LMs and an ability to make proper modifications at the code level.
A natural question that arises is whether such changes are necessary.
In other words
, Does language model fine-tuning work for non-language tasks without changing the architecture or loss function at all?
To answer this, we consider a simple fine-tuning procedure for LMs, referred to as LanguageInterfaced Fine-Tuning (LIFT) .
This procedure can be used to learn predictors for any classification or regression task.
LIFT runs in two phases: (1) converting labeled samples into sentences, and (2) fine-tuning pretrained LMs on the sentence dataset without altering the architecture or loss function.
Fig.
1 illustrates how we fine-tune GPT with LIFT to solve the Iris classification task [30].
LIFT first converts each labeled sample into a sentence with two options.
The first option is to incorporate feature names and the task description into the sentence template.
In this example, we could convert a training sample r into “An Iris plant with sepal length r.sepal_length , sepal width r.sepal_width , petal length r.petal_length , and petal width r.petal_width is r.class .”
Here, we use the dot notation, i.e., r. ⋆ denotes the string conversion of the corresponding attribute of sample r .
One may also adopt a simpler (and more generic) sentence template, such as “If x1= r.x1 , x2= r.x2 , ..., xp= r.xp , then y= r.y ,” if there are p features.
We then fine-tune LMs without changing either architecture or loss function.
Then, we perform inference as follows.
LIFT first converts test samples into sentences using the same template while leaving the prediction part empty.
It then feeds the converted sentences as prompts to the fine-tuned model.
The output tokens are parsed to provide the final predictions.
Our work empirically shows that LIFT can provide high-accuracy solutions for a variety of nonlanguage tasks.
Fig.
2 shows examples of real functions learned by GPT-J models [31] fine-tuned using LIFT given 1000 samples.
Recall that LIFT does not require any changes in the architecture
[Figure]
R U
E − (
[Figure] [Figure]
(b quadratic
[Figure] [Figure]
(c exponential [Figure]
[Figure]
(d) cosine
[Figure] [Figure]
RAE
[Figure]
[-RAE: 0.35 − 8 0 8 −8 −(c) exponential − 8 0 8 −8 −
 RAE: 0.48 − 8 0 8 −8 −(d) cosine − 8 0 8 −8 −
-][Figure]
[Figure] [Figure] [Figure] (e l1-norm −[-8 0 8 −8 −
-]
[Figure] [Figure] RAE: 0.45 − 8 0 8 −8 −
Motivated by these intriguing properties, we investigate the efficacy and limitations of LIFT on non-language tasks by conducting an extensive empirical study on a suite of classification and regression tasks.
First , we observe that LIFT performs well across a wide range of low-dimensional classification and regression tasks.
In most cases, it nearly matches (or slightly outperforms) the best baselines’ performance.
To further understand LIFT, we conduct experiments testing the fundamental learning properties, e.g., its inductive bias, sample efficiency, ability to extrapolate, worst- and average-case noise robustness, and how the pretraining of LMs affects LIFT.
Third , we study a few unique properties specific to LIFT, e.g., context-aware learning with task-specific prompting, prediction calibration, and the additional use of LIFT for data generation.
Lastly , to improve upon the basic fine-tuning, we employ a few techniques: two-stage fine-tuning with synthetic pretext tasks and data augmentation with mixup.
The two-stage fine-tuning improves LIFT in the low-sample regime.
We finally provide discussions on limitations and future investigations of LIFT.
Scope of the study.
Our work proposes the use of natural language interface for learning with
LMs via LIFT.
We emphasize that our goal is not to achieve the state-of-the-art performance, but to investigate thoroughly: (i) what LIFT can and cannot do, (ii) properties of models fine-tuned via LIFT, and (iii) whether we can improve LIFT with advanced techniques.
Methodology and Experimental Setup
LIFT training.
To fine-tune a pretrained LM with LIFT on a target supervised learning task, we apply two steps: (1) convert each sample into a sentence with a fixed template, and (2) fine-tune
LMs with sentence datasets.
We use the default cross-entropy loss for token prediction in LMs.
Our generic template (without feature names and task description) for sample r is
When we have x1= r.x1 , x2= r.x2 , ..., xp= r.xp , what should be y?
(cid:124) (cid:123)(cid:122) (cid:125) question ### (cid:124)
[Equation]
(cid:124)
[Equation]
answer
[Equation]
end of answer , if r has p attributes.
Here, we use the separator convention recommended by OpenAI [32] – “###” for question/answer separation, and “ @@@ ” for end of generation.
When attributes names and task descriptions are available, one can use a more informative prompt (shown in Fig. 1) with all actual prompts are provided in Sec. 4.1.
We report learning curves of LIFT on several tasks in Appendix E.5.
LIFT inference.
For inference, we use the same prompt template except for the answer and end-of-answer parts.
Once the fine-tuned LM completes the test prompt, we simply parse the output tokens.
For classification, we simply compare the generated text with the string representation of the class names.
For regression, we convert the generated string into a number.
For instance, with the output being “ y =10.35 @@@extratokens ”, we split the output sentence by the stop separator “ @@@ ” into two parts.
Taking the first part “ y =10.35”, we parse the value “10.35” as our final prediction.
The generated output might be invalid .
For classification tasks, the output string may not match any actual class, which we declare as misclassification. Note that one may obtain better accuracy by returning its closest class using string metrics.
For regression tasks, we consider output invalid if the string-to-number parsing fails.
In these cases, we adjust the generation randomness by increasing the decoding temperature [33, 34, 35] from 0 (deterministic mode) to 0 .
75 (random mode).
We repeat the inference up to five times, then return the average value of the training set if all attempts fail.
Note that invalid output occurs very rarely (less than or around 1% in most tested cases).
For evaluation metrics, we use accuracy for classification tasks, and RMSE, RAE errors for regression tasks, where RAE = (cid:80) ni
=
| ˆ y
i − y
i | / (cid:80) ni
=1 | 1 n (cid:80) nj =
y j −
y
i | and RMSE =
(cid:112)(cid:80) ni
=
(ˆ y i − y i ) 2 /n on each dataset D = { ( x i , y i ) } ni =1 with n samples, features x ∈ X ⊂ R p , and outcome y .
Pretrained LMs.
We apply LIFT on two pretrained LMs: GPT-J [31] and GPT-3 [19].
To fine-tune GPT-J, we use LoRA [24], a parameter-efficient method that constrains weight matrix updates to be low-rank.
For experiments on GPT-J, we used p3.8xlarge and p3.2xlarge instances from AWS and RTX3090 GPUs in the local server.
Since GPT-3 is not fully publicly available, we use the API provided by OpenAI to perform black-box GPT-3 fine-tuning.
More details are in Appendix C.2.1.
Datasets.
We design and select a wide range of datasets to better understand the behavior of LIFT.
For classification , we use three types of non-language data: low-dimensional synthetic datasets, real tabular datasets in OpenML [36], and vision datasets (MNIST [37], Fashion-MNIST [38] and their permuted variants [39]).
For regression , we use both synthetic and real datasets.
For synthetic datasets, we defined samples ( x i , y i ) of input-output pair as y ∼ f ( x ) + N (0 , σ 2 ) , where σ 2 ≥ 0 is the noise level.
Unless otherwise stated, we sample the feature x uniformly from a hypercube [ L, U ] p , where L and U are minimum/maximum feature values, and p is the number of features.
Following the suggestion by [40], we consider various functions f for regression tasks: (i) linear function, (ii) quadratic function, (iii) exponential function, (iv) cosine function, (v) ℓ 1-norm function, and (vi) piece-wise linear function.
Their 2D visualizations are provided in the first row of Fig.
We also use four real datasets: Medical Insurance (Insurance) [41], Combined Cycle Power Plant (CCPP) [42],
Servo [43], and Student Performance (Student) [44].
More details are included in Appendix C.1.
Baselines.
We consider standard learning algorithms [45, 46].
For classification, we use logistic regression ( LogReg ), decision tree ( DT ), k-nearest neighbor ( KNN ), support vector machine with
Gaussian kernel ( SVM ), a four-layer ReLU neural network ( MLP ) with 200 neurons per hidden layer, random forest ( RF ), and XGBoost ( XG ).
We also use the majority class classifier ( MCC ) that outputs the most dominant class.
For regression, we use polynomial regression ( PR ), kernel ridge regression ( KR ) with radial basis function kernel, k -nearest neighbors ( KNN ), a three-layer ReLU neural network ( MLP ) with 50 hidden neurons per each layer, Gradient Boosting Trees ( GBT ), random forest ( RF ), and Gaussian process ( GP ).
For hyperparameter selection, we apply the grid search on a set of parameters’ values and use cross-validation on the training set (see details in Appendix C.2).
Basic Findings of LIFT
Table 3 summarizes our main findings.
We also study sample complexity (Sec. 3.2), comparison with in-context learning (Sec. 3.3), models’ decision boundaries (Sec. 3.4), and the effect of LMs’ pretraining on LIFT (Sec. 3.6).
Appendix E provides additional results, including the effect of input and output layers (E.1), model size (E.2), and LIFT for Ridge regression (E.4).
3.1 How
Well Does LIFT Perform on Standard ML Tasks?
Classification.
Table 4 compares classification accuracies between algorithms on a wide range of tasks.
We observe that LIFT achieves comparable performance to most baselines.
In most cases,
LIFT/GPT ranks highly in the top three best-performing methods.
We find that LIFT can learn non-linear relationships between features and the responses: LIFT/GPT-3 achieves 81 .
17% accuracy on the circle dataset, while logistic regression failed to perform better than the MCC ( 50%) .
As the difficulty of tasks varies, which can be estimated by the average performance of baselines, LIFT also suffers from performance degradation.
LIFT can perform comparably well even when the number of features is as large as hundreds, though the limited number of tokens as inputs to LMs restricts the number of features LIFT can input.
However, when the number of classes is large (say 100s), both LIFT/GPT models have lower accuracies than many baselines, though they manage to be better than MCC.
For instance, on the 100-class Margin dataset, the accuracy gap between LIFT/GPT-J and the best algorithm (RBF-SVM) is nearly 30%.
Note that LIFT can directly use raw data while most baselines require feature scaling and normalization for good performance.
More results are provided in Appendix D.1.1, including comparisons with methods leveraging larger models.
Regression.
Tables 19 and 20 present our function approximation comparison.
For the lowdimensional cases, LIFT is comparable to baselines.
Still, it fails to beat the strongest baselines, such
[Table]
LIFT
[Table]
comparable to strong baselines (Table 4). For regression, LIFT well approximates different types of low-dimensional functions (Fig. 2) but does not perform well for high-dimensional cases (Table 19).
[Table]
For
[Table]
, LIFT is robust to outliers in training data (Fig. 28).
For classification, LIFT is comparable to baselines under label corruption on training data (Fig. 29) but more vulnerable to feature corruption on test data (Table.
[Table]
classification [Table] designing
[Table]
specify
[Table]
task. The improvement is significant when the description of the feature names and the target task can
[Table]
interpreted with common knowledge (Table 9)
[Table] [Table]
as GP
[Table]
as GPT models measure the error by comparing tokens instead of measuring how close the prediction values are to true values. We conjecture that we can improve our performance by level encoding,
i
[Table]
e. , representing numerical values as binary values.
We also investigate the interpolation and extrapolation of LIFT and defer the details to Sec.
D.1.1.
All methods fail to extrapolate and interpolate well for all functions
[Table]
and the interpolation performance of LIFT is only good in the linear regression case.
Interestingly, LIFT tends to output seen values (from training data) for extrapolation.
3.2 How Many Samples Does LIFT
Need?
We investigate whether LIFT is sample efficient.
Fig.
25 in Appendix shows the sample complexity evaluation on classification and regression tasks.
We find that GPT models can be quickly fine-tuned to learn new tasks with LIFT.
For classification , as the number of classes increases (left to right columns in Fig. 25a), LIFT does need more samples for adaptation, probably because the data input
[Table]
and output spaces are more complex to learn. For regression tasks, we find that 1000 samples are sufficient for LIFT to have a small RMSE, similar to other baselines.
There exist some functions ( e.g. , cosine and piecewise) where LIFT has lower sample complexity than popular baselines.
3.3 Language-Interfaced Learning: LIFT versus In-Context Learning (ICL)
Beyond fine-tuning (with LIFT), our language-interfaced learning framework can be used for other learning methods for LMs, including in-context learning (ICL) [47, 48, 19] that performs inference on new tasks without fine-tuning by conditioning on a few training examples.
Table 5 compares the classification performances between (a) ICL, (b) LIFT trained on a subset with n samples, and (c) LIFT trained on the full dataset.
Note that the number of training samples ( n ) used for ICL depends on the context length of given LMs.
As we can see, LIFT using the full dataset always achieves the best performances.
However, LIFT/Subset and ICL are more comparable in most cases when they use the same number of training samples, which are sufficiently small for ICL mod to fit in LMs.
Remark.
One can replace fine-tuning with ICL in our language-interfaced procedure when the target tasks require fewer training samples.
3.4 Can We Understand the Inductive Biases of Language Models via LIFT?
We investigate classification decision boundaries of LIFT/GPTs varying boundary complexities for better insight into their inductive biases, as shown in Fig.
We first train a binary-class neural network and use its snapshots at different training epochs to construct datasets having decision boundaries at different complexity levels (first column in Fig. 6).
We observe that LIFT/GPT models adapt well to three boundaries and capture their rough shapes.
Furthermore, their boundary shapes are axis-parallel, similar to the boundary of tree-based classifiers.
They also show a lot of fractals similar to the observations on some convolution neural networks [49].
See
Appendix D.1.3 for results of 3-class and 5class datasets and quantitative measurements.
3.5 How Robust is LIFT?
We investigate the robustness of LIFT against the outlier on training data and the feature corruption on test data.
Appendix D.1.4 provides additional experimental results, including that for the case of label corruption on training data and class-imbalanced data.
Robustness to outliers in training data.
We consider regression tasks where we have outliers whose outcome y is not consistent with the majority of samples in terms of fitting ( x , y ) .
Due to the space limitation, we put the figure in Appendix.
Fig.
28a compares RAE values of methods with and without outliers (2% outliers in the training set).
LIFT/GPT models are among the most
[Table]
robust ones: their performances are almost unaffected, while baselines suffer huge performance drops.
Furthermore, in Fig. 28b, we evaluate models for various percentage of outliers (1%, 2%, 5%, 10%, 20%).
Compared to the robust baselines (median-3NN and median-5NN) [50], LIFT/GPT-3 is comparably robust, while LIFT/GPT-J is more vulnerable when more outliers are present. of MNIST data.
See the full results in Table 33 in Appendix
[Table]
Robustness to feature corruption on test data.
Given a clean test data ( x , y ) having feature x and label y , we explore whether adding small perturbation δ on the feature changes the performance; we measure the accuracy of LIFT on perturbed data ( x + δ , y ) .
We apply transfer attack [51] since we do not have full access to the GPT-3 model, and finding adversarial examples in the discrete input space is complex [52].
Tablereports robustness results on MNIST classification under PGD attacks transferred from LeNet-5.
The perturbation radius is set to ε ∈ [0 , 0 . 01 , 0 . 1 , 0 .
3] where MNIST pixel value is within [0,1].
We compare three networks: LeNet-5, MLP (2 hidden layers with 300 and 100 neurons), and LIFT/GPT-3.
When ϵ ∈ { 0 . 01 , 0 . 1 } , LIFT/GPT-3 tolerates random noise (as in Table 33) but cannot tolerate transferred adversarial attack, implying that the adversarial attack on LeNet-5 is transferred to LIFT/GPT-3.
3.6 Does LIFT Need Large-Scale Models Pretrained on Natural Language Data?
We investigate the requirement of pretrained LMs for which LIFT performs well.
We compare variants of LIFT under different types of LMs: GPTs pretrained on natural language data (our models), a large LM without pretraining (Rand-GPT-J), and LMs pre-trained on non-human language data, including CodeParrot [53] and CodeGen-2B-mono [54] trained mainly on programming language data, and a GPT-J fine-tuned on Gibberish data [55].
See Appendix D.1.5 for the detailed setting.
Does LIFT only need a large pretrained model?
To answer this question, we compare performances of LIFT when GPTs are pretrained (LIFT/GPTs) and when GPT-J have weights being randomly initialized (LIFT/Rand-GPT-J).
More specifically, for LIFT/Rand-GPT-J, we randomly initialized a
GPT-J model and fine-tuned the whole model (instead of LoRA).
As shown in Table 8, accuracies of
LIFT/Rand-GPT-J are much lower than those of our models (LIFT/GPTs), across all datasets.
These results indicate that LIFT benefits from pretraining, not just from the gigantic LMs.
Does LIFT need a model trained on natural language data?
As shown in Table 8, LIFT/GPTs perform much better than LIFT/CodeGen and LIFT/CodeParrot for all datasets.
This implies that
LIFT may perform better with LMs pretrained on natural language data.
When the pretrained GPT-J is fine-tuned on gibberish data [55], the accuracies drop slightly for a few tasks and become lower than LIFT/GPTs overall.
However, LIFT/Gibberish still achieves comparably good performance and its small performance gaps to LIFT/GPT-J can be attributed to the relatively small impact of fine-tuning on large pretrained LMs.
Thus pretraining on natural language data is necessary for LIFT.
Evaluation of LIFT-Specific Learning Properties
In this section, we study the behavior of LIFT in a more fine-grained manner.
4.1 Does LIFT Benefit from Incorporating Feature Names?
Unlike standard machine learning algorithms, LIFT can be provided context information by incorporating the feature names and task descriptions in the prompts.
Intuitively, this incorporation may improve the sample complexity of LIFT as the prior knowledge already learned in the pretraining
[Table]
phase may help LIFT predict better. We design seven prompt templates to assess how incorporating feature names affects the performance of LIFT (see more details in Appendix D.2.1).
We empirically verify this intuition and show our results in Table 9 for several classification tasks using pretrained
GPT-3 models.
We first observe that all LIFT models outperform MCC with significant accuracy gaps, indicating that they are all properly trained.
Second, we observe that correctly incorporating feature names helps boost the performances of LIFT for datasets except for CMC .
Third, if we use similar prompts with shuffled feature names ( Shuffled-Names I, II ), then the performance of
LIFT drops by a significant margin.
These results imply that the aforementioned performance improvements are indeed due to proper prompting with correct feature/value association.
4.2 Is LIFT Well Calibrated?
We investigate whether LIFT is calibrated , i.e. , the prediction reflects the confidence, by exploring how LIFT performs under various noise levels, as shown in Fig.
36 in Appendix.
We conduct experiments on six synthetic regression datasets, each consisting of 1,000 noisy training samples shown as blue markers in the first row.
To be specific, we generate (1) the input x following the guideline in Sec.
C.1 for regression tasks and (2) the noisy outcome y where the standard deviation of noise σ ( x ) =
( x + 10) / 10 increases along the x -axis (from x = − 10 to x = 10 ), and study how different noise level affects the predictive behavior of LIFT.
In the inference phase, we set the decoding temperature T = 1 for LIFT to make random predictions.
For visualization purposes, we generate an additional 103 samples uniformly in [ − 10 , 10] for each task and plot the standard deviation of 20 LIFT/GPT-J predictions on each sample in the bottom row of Fig. 36.
Note that the bottom row of Fig.
36 shows that the standard deviation of LIFT/GPT-J’s prediction nearly matches that of noisy training samples (observations) across different tasks.
These results imply that
LIFT/GPT-J is calibrated.
Similarly, Fig.
37 of Sec.
D.2.2 shows that LIFT/GPT-3 is calibrated.
4.3 Can We Use LIFT for Data Generation?
[Figure]
We observe that LIFT/GPT-J can generate images of comparably high quality. The temperature is set to 1.
Generative models have been widely used in computer vision [56, 57, 58].
Beyond classification and regression tasks, we study whether
LIFT can be used for generative tasks, i.e. , learning the underlying data distribution and generating realistic data samples.
In particular, we consider two image generation tasks on MNIST dataset: (i) generating an image given a digit number, and (ii) completing an image given a digit number and its pixels on the top half of the image.
Fig.
10 shows our generated images for the two tasks.
We observe that the generated images all have correct digit shapes and reasonably high quality, especially for the completion task.
See Appendix D.2.3 for the detailed setting and additional results.
Improving LIFT with Existing Techniques
We improve LIFT with advanced techniques: two-stage fine-tuning and data augmentation.
5.1 Two-Stage Fine-Tuning for LIFT with Synthetic Pretext Tasks
[Figure]
In Sec. 3.2, we observe that LMs need a sufficient number of samples to start adapting.
We suspect that LMs’ adaptation to non-language tasks contains two phases: (1) learn the task description, i.e., input space, label space, and sentence templates [47, 59], and (2) learn the target task.
Thus, we consider utilizing synthetic data to describe the task for LMs in the first phase, thus reducing the sample complexity.
This results in a new two-stage training procedure for
LIFT.
1 In particular, for any given dataset, we first generate two pretext tasks with simple synthetic Gaussian datasets (discussed in C.1) sharing the same number of features and the label space (for classification tasks) or the range of responses’ values (for the regression tasks) to the actual data.
We apply LIFT on pretext tasks for a few (2 or 3) epochs, then continue LIFT with the target (given) dataset.
For GPT-3, it is unclear how to keep the order of samples not shuffled with the current black-box API during the fine-tuning stage.
Hence, we only provide the experimental results of GPT-J. Fig.
11 shows that two-stage fine-tuning improves LIFT over the original fine-tuning when the number of training samples is small on both classification and regression tasks.
5.2 Data Augmentation
[Table]
Data augmentation [61] is a simple tool for improving the generalization performance for various classification problems.
Here, we investigate whether data augmentation benefits LIFT.
Table 12 shows the effect of adding random noise in the training data on the performance of
LIFT/GPT-J for the MNIST classification problem.
Here, we test each model on three settings: (1) clean data, (2) Gaussian noise, and (3) signed constant noise.
We allow each noise can perturb up to the magnitude of ϵ ∈ [0 , 1] at each dimension ( i.e. , each pixel) when the black/white pixel of
MNIST is represented in the [0 , 1] range.
We defer the generation procedure of random Gaussian noise to Sec.
D.1.4 in Appendix.
One can observe that LIFT/GPT-J without any data augmentation (DA) is vulnerable to random noise, unlike existing baselines (LeNet-5 and MLP).
However, when we apply data augmentation, i.e. , train LIFT/GPT-J with noisy training data, the accuracy improves significantly for the perturbed (either adding Gaussian noise or Signed constant noise) test data.
This shows the effectiveness of simple data augmentation in LIFT.
Exploring the effect of other data augmentation schemes, e.g. , mixup [62] and its variants [63, 64, 65], is remained an interesting future work.
Related Works
Fine-tuning for adapting LMs to non-language tasks.
Fine-tuning [66] pretrained LMs is the standard practice for learning downstream tasks, which may involve simple architecture modifi- cations, such as adding linear layers [67, 68] or freezing layers [18, 69, 70].
The recent progress focuses on parameter-efficient techniques for reducing trainable parameters, including adapter-based fine-tuning [25, 26, 27] that trains additional small residual blocks between layers, freezing-based fine-tuning [71, 18, 72] that freezes most of the pretrained parameters, and distillation-based finetuning [73].
Our LIFT/GPT-J is fine-tuned with LoRA [24], a parameter-efficient method approximating the weight updates using low-rank matrices.
To directly adopt existing fine-tuning methods of LMs for non-language tasks, it is common practice to modify the input/output layers and the loss functions, which may cause undesired behaviors like catastrophic forgetting [66, 74].
Our work is highly motivated by Frozen Pretrained Transformer (FPT)
[18] that directly fine-tunes GPT-2 [21] pretrained on language tasks for other modalities by freezing most pretrained parameters and adding only input and output layers for the modality adaptation.
Unlike FPT, our method requires no such changes in the architecture and objective function.
Several works also extend the existing LMs to handle different input data types, such as images [75, 76], audio [77], tabular data [78], and knowledge base [79] by updating the pretraining phase with these data and their corresponding tasks or using general-purpose architecture [80].
Our work is based on GPT language models trained only on textual data.
Analyzing the adaptability of LMs.
Similar to ours, recent works [81, 82, 83, 82] attempt to understand and quantify the adaptability [20] and capacity of large LMs, such as Big-Bench [82] with a new benchmark of more than 200 tasks on a diverse set of topics.
General-purpose models.
A primary goal of our work is to push the limit of the existing generalist language models ( e.g., GPT-3 [19]) to other modalities and domains, supporting the idea of building a domain-and-modality agnostic generalist model [19, 84, 3, 85, 86, 87, 88, 89].
Note that LIFT can be applied to any generalist model with LM-like architectures, such as GATO [89].
Furthermore, our work shares the general goal with automated machine learning (AutoML) [90, 91] in improving the usability of machine learning, though LIFT uses only a single pretrained LMs for all tasks while
AutoML automates the standard machine learning pipeline from a set of existing algorithms.
Discussion and Conclusion
We propose the use of language-interfaced framework, via Language-Interfaced Fine-Tuning (LIFT) , for using LMs to solve non-language downstream tasks without changing the models’ architecture or loss function.
LIFT first converts labeled samples into sentences and then fine-tunes pretrained LMs on the sentence dataset using the standard fine-tuning method and loss function.
Via an extensive empirical study, we show that LIFT/GPT performs relatively well on low-dimensional classification and regression non-language tasks.
Furthermore, LIFT/GPTs are robust in several practical settings, and can properly calibrate the predictions and generate real-like data samples.
LIFT can be improved using in-context feature names, two-stage fine-tuning, and data augmentation.
Moreover, our work is arguably one of the first to thoroughly study the efficacy of language-interfaced learning framework with pretrained language models on standard regression and classification tasks, paving the way for enabling “no-code machine learning with language models.”
Limitations and open questions.
Despite promising performances on various tasks and settings, we observe some limitations of LIFT to basic learning tasks.
LIFT/GPT do not perform well if the features have high dimensions (for regression) or when the number of classes is large (for classification).
In addition, the context length of LIFT is restricted to the context length of LMs and LIFT/GPT is memory-inefficient.
One can combine LIFT with memory-efficient LMs such as
LinTransformer [92] to address this issue.
Besides, our works open some interesting questions for future works.
First, do LMs and LIFT/GPT have behaviors similar to ensemble methods or decision tree since they have similar decision boundaries?
Secondly, are LMs universal models that can adapt well to any modalities and domains?
Lastly, can LIFT/GPTs adapt better for regression tasks using more sophisticated encoding schemes for numeric features?
Social impacts.
Future research should also investigate potential fairness issues of applying LIFT.
Based on large language models, LIFT might have embedded bias targeting certain social groups.
Especially when feature names are included in the training prompts, the models may be more sensitive to social biases and thus might make unfair and harmful predictions.
We leave measuring the embedded bias in LIFT as one of the interesting future directions.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] The contribution and scope of this paper has been summarized in the abstract and the last part of Introduction.
(b) Did you describe the limitations of your work?
[Yes] We wrote the limitations in Sec.7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] This has been discussed in Sec.7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] Our paper conforms to the ethics guideline.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code, data, and instructions are provided in the shared anonymous GitHub repository.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] All the training details are discussed in the paper and they can be found at the anonymous GitHub repository.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All the experiments are run multiple times and at the tables mean and standard deviation of those runs are presented.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] This has been described in Sec.3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Our work uses public datasets/models, and has been cited properly, in the paper and GitHub repo.
(b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
Due to the space limit, many interesting findings, discussions, and further details are included in
Appendix.
We start by discussing the detailed motivation of our work in (A) and introducing related work in (B).
We then describe our datasets in (C.1) and the implementation of all models in (C.2).
For the experimental results, we provide detailed and extended results for all findings presented in the main paper in (D), with more visualizations, score tables, and additional evaluations on more functions, datasets, and properties.
In (E), we present additional interesting findings and visualizations which have not been discussed so far in the main paper.
Lastly, we provide a more profound discussion in (F).
A Further Motivations of LIFT
A.1 Explainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Updatability via Information Retrieval
B Detailed Related Works
C Experimental Setup
C.1 Datasets
C.2 LIFT and Baseline Implementation
C.2.
Pretrained Language Models and Baselines
C.2.
Computing Resources
C.2.
Hyperparameter Selection
D Detailed and Extended Results of Experiments in the Main Paper
D.1 Results for Basic Findings of LIFT
(Section 3) . . . . . . . . . . . . . . . . . . .
D.1.
How Well Does
LIFT
Perform
on
Standard
ML
Tasks? . . . . . . . . . . .
D.1.
How Many Samples Does LIFT Need? .
D.1.3 Can We Understand the Inductive Biases of Language Models via LIFT? .
D.1.
How Robust Is LIFT? .
D.1.5 Does LIFT Need Large-Scale Models Pretrained on Natural Language Data?
D.2 Results for LIFT-Specific Learning Properties (Section 4) . . . . . . . . . . . . .
D.2.1 Does LIFT Benefit from Incorporating Feature Names? . . . . . . . . . .
D.2.2 Is LIFT Calibrated? . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2.3 Can we use LIFT for Generation? . . . . . . . . . . . . . . . . . . . . . .
D.3 Results for improving techniques of LIFT (Section 5) . . . . . . . . . . . . . . . .
D.3.1 Two-Stage Intermediate Fine-Tuning for LIFT . . . . . . . . . . . . . . .
E Additional Experiments and Findings (NOT Discussed in the Main Paper)
E
What Is the Effect of Replacing the Input or Output Layers? . .
E
.2 Does LIFT Benefit from Larger LMs
E
.3 Quantitative Classification Evaluations on Neural-Net-Based Synthetic Datasets
E
.4 Can LIFT Perform Ridge Regression via Data Augmentation
E.5 LIFT’s Training Curve
F Additional Discussion
F.
Limitations and Open Questions . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Broader Impact . . .
A Further Motivations of LIFT
While we mainly emphasized the “no-code-ML” property of LIFT, indeed, it has a lot of potentials to be more useful and powerful than many of the current ML models.
Particularly, LIFT can bring an entirely novel approach to enable (i) explainability and (ii) updatability via information retrieval.
A.1 Explainability
Most ML models cannot interpret their predictions.
While specific algorithms are developed to enable such models’ explainability, their efficacy is still in question.
On the other hand, LMs have been shown to be able to explain their predictions [93, 94].
This ability has been used in many fields such as human moral judgements [95] and mathematical reasoning [96].
Therefore, LIFT, based on a large pretrained LM, can be made to explain its prediction using its reasoning capabilities.
Consider the German-credit dataset [97], which aims to predict whether the bank should approve or decline loan applications.
After asking GPT-3 if one should approve/decline a loan application via LIFT, one can also ask GPT-3 to explain its own prediction result.
This can be implemented by making two consecutive inference calls, as shown in Table 13.
Here, we provide five different responses generated with different random seeds.
Note that these explanations are generated without seeing any reasoning examples, i.e. , these are solely based on the zero-shot reasoning capability of language models.
While these explanations are not even close to perfect, they are still reasonable.
Improving the few-/zero-shot reasoning capabilities of language models is a fast-growing research field, so incorporating new techniques can further improve the explainability of LIFT.
A.2 Updatability via Information Retrieval
Another drawback of current ML models is that it is difficult to “update” models when a distribution shift occurs.
Handling these distribution shifts in a compute-efficient manner has recently become one of the most active research areas in the field.
Recently, augmenting LMs [98, 99, 100, 101] with a retrieval mechanism has shown to be an efficient way of updating LMs.
With such a retrieval mechanism equipped, LMs can be efficiently updated as one can simply update its associated database, or even connect LMs to the Internet [101].
While we only used LMs not equipped with a retrieval mechanism in this paper, it is straightforward to apply our framework to other LMs that can retrieve information from databases or the Internet, i.e. ,
LIFT can support a compute-efficient update mechanism.
B Detailed Related Works
This section provides detailed related work.
Pretraining and
adapting
language
models
LMs).
Our work uses modern large LMs, which promoted significant advances in the field of natural language processing (NLP) [4].
Most popular
LMs use transformer architectures [12, 13] as the backbone, from early models like BERT [102] built on Transformer encoders to GPT variants [103, 21] built on Transformer decoders.
Multiple modern large LMs have been proposed, including RoBERTa [104], ALBERT [105], XLNet [106], and the latest models with billions or trillions of parameters, such as GPT-3 [107], Switch-Transformers [108], and PALM [109].
LMs are trained to encode a large amount of linguistic knowledge from multiple sources in their contextual representations [110], which are helpful and can be easily adapted to various other tasks.
Thus, starting with BERT [102], it has become a standard practice to pretrain and then fine-tunes a large LM for plenty of downstream tasks in lieu of training a model from scratch for a specific task [102, 19, 4, 104, 111].
This technique dramatically impacts a wide range of NLP tasks, such as language modeling [19, 109], question answering systems [112, 113], text summarization [114], neural machine translation [115], and reasoning [116, 117].
However, the excellent performance of fine-tuned LMs without architecture changes has been mainly limited to NLP tasks so far.
This work, instead, investigates whether we can leverage LM fine-tuning
We convert a data sample from German-credit dataset [97] into text and use GPT-3 to predict whether the bank should approve or decline the individual’s loan application.
After receiving the first response of “accept”, we add “The bank will accept this loan application.
Let’s think logically.
This is because” to the original prompt and use it as the new input of GPT-3.
The five different responses generated by
GPT-3 with different random seeds demonstrate the reasoning ability of language models.
Note that these explanations are generated without seeing any reasoning examples, i.e. , these are solely based on the zero-shot reasoning capability of language models.
[Table]
A 32-year-old female is applying for a loan of 18424 credits for 48 months for other purposes.
She has a checking account with less than 200 Deutsche Mark and a savings account with less than 100 Deutsche Mark.
She had no credits or paid back all credits duly.
The installment rate of this applicant in percentage is 1%.
She is employed for more than one year but less than four years and is recognized as a highly qualified employee or is self-employed.
She has lived in a self-owned house.
She owns some building society savings agreements or life insurance.
Given such information, will the bank accept or decline this loan application?
First
[Table]
A 32-year-old female is applying for a loan of 18424 credits for 48 months for other purposes.
She has a checking account with less than 200 Deutsche Mark and a savings account with less than 100 Deutsche Mark.
She had no credits or paid back all credits duly.
The installment rate of this applicant in percentage is 1%.
She is employed for more than one year but less than four years and is recognized as a highly qualified employee or is self-employed.
She has lived in a self-owned house.
She owns some building society savings agreements or life insurance.
Given such information, will the bank accept or decline this loan application?
The bank will accept this loan application.
Let’s think logically.
This is because
Secondresponse
(randomseed1)
the applicant is employed and is qualified in a high level.
Since she is employed, she is able to make all the payments.
Furthermore, she has a self-owned house and owns some building society savings agreements or life insurance.
She has less than
Deutsche Mark in checking and savings account.
She had no credits or paid back all credits duly.
She is able to pay the loans.
All of these are positive factors to this loan.
The bank will accept this loan.
Second response
(randomseed2)
the applicant has a high income.
She has lived in a self-owned house.
She has some savings agreements or life insurance.
She is a highly qualified employee.
She has no negative credits.
All of these facts show that she is not in danger of not being able to pay back the loan.
Therefore, the bank will accept this loan application.
Secondresponse (randomseed3)
the applicant has a checking account with less than 200 Deutsche Mark and savings account with less than 100 Deutsche Mark.
Hence she is a low-risk customer.
Furthermore, she has no credits or paid back all credits duly.
And the installment rate of this applicant is 1%.
She is employed for less than four years and is recognized as a highly qualified employee.
She has lived in a self-owned house.
She owns all the building society savings agreements or life insurance.
Hence she is considered a good customer.
So the bank will accept her loan application
Secondresponse (randomseed4)
the applicant is employed with a high salary and is economically stable.
She is not likely to file for bankruptcy.
She can pay back the loan.
All in all, we can see that the bank will accept this loan application.
Secondresponse (random seed 5) the bank will accept all applications with a very low risk of default.
There is a very low probability that she will be unemployed in the near future.
She can afford to pay off this loan.
Furthermore, this applicant has a high annual income
for non-language tasks across different modalities.
Our work is highly motivated by Frozen Pretrained
Transformer (FPT) [18], which directly adapts GPT-2 [21] pretrained on language tasks and textual data to other modalities.
FPT freezes most pretrained parameters except the layer normalization layers and adds input and output layers for fine-tuning.
The authors empirically show that GPTcan be efficiently fine-tuned for different modalities and domains, including vision and numeric computation.
Nevertheless, FPT requires changes in the architecture and objective function to adapt to different data representations, while our method LIFT does not.
Furthermore, we mainly focus on basic machine learning tasks, such as function approximation or tabular data classification.
Several works have attempted to extend the existing LMs to handle different types of input data, such as images [75, 76], audio [77], tabular data [78], and knowledge bases [79] by updating the pretraining phase with these data and their corresponding tasks.
For instance, XGPT [118] takes images as the input and uses the image captioning task as the primary task in the pretraining stage for
GPT to generate images’ captions.
Similarly, multiple works utilize pretrained LMs for generating text descriptions of images or videos in image captioning (VisualGPT [119]) and video captioning (VideoBERT [120], Unified VLP [121], UniVL [122]).
SpeechBERT [77] also integrates LMs for speech recognition in the weakly-supervised setting to reduce the need for scarce supervised data.
LMs can also adapt to numeric tasks [123] or other domains such as protein folding [124] or symbolic math solvers [125].
Recent works [126, 127, 78] pretrain LMs with large tabular datasets, improving the question answering systems by reasoning over the tables.
Compared with these existing works, our work is unique in that it is based on GPT language models trained only on textual data.
Analyzing the adaptability of LMs.
Similar to our work, recent works [20, 81] have made efforts to understand and quantify the feasibility and limitations of the adaptability of large LMs for upstream performance and downstream tasks.
For instance, the recent work [20] built a benchmark ofsmall language tasks for testing the adaptability of LMs, observing that the LMs [128] can adapt well to an extensive range of complex tasks rather than just memorizing the learned patterns.
BIGbench [82] is recently introduced as a new benchmark for quantifying the capacity of LMs, consisting of more than 200 tasks on a diverse set of topics.
Another relevant work [83] attempts to understand the effect of LM pretraining by studying how the transformer architecture, the backbone of LMs, succeeds at a designed synthetic task.
Similar efforts in this line of work are to analyze the behaviors, representations, and inductive bias of pretrained LMs [129, 130, 131] or investigate different aspects of LMs [132, 133, 134].
For instance, a recent work [132], in the investigation of the difficulty of numeracy in LMs observes that transformer-based language models do not work well on complex numeric tasks and are sensitive to different formats of numeracy.
Note that these existing works focus primarily on downstream language tasks, while we focus on adapting LMs on non-language tasks without any modification of the loss or architecture.
Methods for adapting LMs.
The most common method for adapting LMs is fine-tuning [66] which aims to slightly adjust pretrained parameters for learning the downstream tasks [135, 136].
Finetuning can involve simple architecture modifications, such as adding linear layers [67, 68] or freezing parts of the network [18, 69, 70].
Fine-tuning can be improved with advanced techniques, such as multi-stage methods [137], intermediate fine-tuning [138, 139, 140], or self-supervised training [141].
The recent progress in fine-tuning LMs focuses on the parameter-efficient techniques for minimizing the number of fine-tunable parameters, including adapter-based fine-tuning [25, 26, 27] that adds and trains small residual blocks between transformer layers, freezing-based fine-tuning [71, 18, 72] that freezes most of the pretrained parameters and fine-tunes only tiny parts of the networks, and distillation-based fine-tuning [73].
In this line of work, LoRA [24] further reduces the number of trainable parameters in large LMs by approximating the weight updates using low-rank matrices without changing pretrained parameters.
LoRA is used as the fine-tuning method for GPT-J in our
LIFT/GPT-J framework.
To directly adopt these fine-tuning methods used in LMs for non-language tasks, it is common practice to modify the input/output layers and the loss functions.
However, these modifications might lead to undesired behaviors like catastrophic forgetting [66, 74].
On the other hand, our method LIFT uses the language interface for fine-tuning without making any changes to the architecture or the loss function.
In-context few-shot learning paradigm [142, 143, 28, 29, 144] suggests modifying only the inputs of LMs by adding a few examples of the downstream task.
A critical part of these methods is reformulating the downstream task samples to the language modeling inputs [47, 59], resulting in multiple efforts in generating [145], searching [48], and properly tuning the prompts [146, 147, 148].
While these methods have shown great effectiveness for multiple NLP downstream tasks, it is unclear how to apply them to downstream tasks of other modalities.
On the other hand, our work successfully adapts LMs for non-language tasks, further pushing the application boundaries of large-scale LMs.
Deep learning for tabular datasets.
While deep neural networks have been successfully applied to various data types, such as images or text, they still face difficulties with a few classification and regression tasks on tabular data [149, 150], one of the most popular data types in practice.
This may be due to the heterogeneous nature of tabular data, with their features being sparse, type-mixed, and weaker in correlation than natural image-language data [151, 149].
Multiple deep learning methods and architectures have been proposed for tabular datasets, from making discrete decision trees more differentiable [152, 153], regularizing neural networks’ weights [154, 155], to recent attempts using attention-based modules [156, 157, 158].
Though these transformer-based models are the closest works to us in this line of work, their works focus on designing and improving architecture designs for specifically learning the tabular data rather than adapting the LMs.
To the best of our knowledge, we are the first to thoroughly study large LM adaptation for tabular learning without architecture changes.
Our work shows promising results of LMs in closing the gap to the best-performing methods, including tree-based ensemble algorithms (Random Forest [159] or XGBoost [160]).
General-purpose models (generalist models).
A primary goal of our work is to push the limit of the existing generalist language models ( e.g., GPT-3 [19]) to other modalities and domains, supporting the idea of building a domain-and-modality agnostic generalist model.
Early works [161, 14, 162] explored this idea by developing and training multi-task and multi-modal models on a wide range of diverse tasks to obtain better generalization and adaptation.
The development of large-scale LMs has significantly contributed to the area of generalist models for languages [19, 84, 3], vision [85], visual language [86, 87, 88], and control problems [89].
These generalist models are usually trained with the scale on an extensive range of corpora, probably containing multiple modalities and domains.
In this line of work, a general-purpose architecture [80] has also been studied to handle different input and output data types.
Although LIFT primarily focuses on the LMs, it can be applied to other generalist models with LM-like architectures, such as GATO [89].
Furthermore, it is worth noticing that our work shares the general goal with automated machine learning (AutoML) [90, 91] in improving the usability of machine learning.
AutoML automates the standard machine learning pipeline for model selection and hyperparameter tuning from a set of existing algorithms.
At the same time, LIFT uses only a single pretrained LM for solving all tasks.
C Experimental Setup C.1 Datasets
[Figure]
Classification datasets.
Table 16 summarizes the datasets used for classification tasks.
We use two additional types of synthetic datasets: neural-net-based datasets used for understanding the inductive biases of algorithms (in Sec.
3.4 and Appendix D.1.3) and Gaussian pretext datasets for the two-stage fine-tuning experiments (in Sec. 5.1).
The neural-net-based datasets are generated as follows.
For binary classification, we train a 2-layer neural network with tanh activation functions using the Rolls dataset shown in the leftmost figure in Fig.
14, and take six snapshots of the decision boundary of the trained neural network shown in Fig.
15; we took snapshots at training epochs 10, 40, 80, 210, 320 and 490.
Then, for each snapshot, we define a synthetic dataset (what we call a neural-net-based dataset) having labels as the neural network’s prediction for randomly chosensamples.
For 3-class and 5-class classifications, we also use 2000 samples to train a 2-layer neural network using the Rolls dataset shown as the second and the third figure in Fig.
The decision boundaries of networks trained on more epochs are visually more complex.
Hence, the corresponding classification tasks are becoming more complicated, from the left column to the right column in
Fig.
In the manuscript, we tested on three out of six datasets, obtained by snapshots at 10, 80, and 490 epochs, respectively.
Given a target dataset of n classes and d features, Gaussian pretext datasets are generated as follows: using scikit-learn 2 , we randomly generate datasets of n clusters, where each cluster has 100 normally distributed samples in the d -dimensional space.
[Figure]
( Image datasets )
For MNIST-based datasets, as the context length of LMs is limited, we (center) scale and crop the size of images from 28 × 28 into 18 × 18 and use the integral format of pixel values [0 , 255] .
We convert each image into a sequence of pixels in the order of left to right and top to bottom.
Each pixel sequence is then converted into the sentence input as of our language-interfaced framework.
Regression datasets.
We test LIFT/GPT on regression problems for both synthetic/real datasets.
( Synthetic datasets )
To assess the regression performance of LIFT in different datasets, we generate synthetic datasets based on six different functions types, including smooth functions, non-smooth functions, and non-continuous functions:
Linear functions y =
f ( x ) =
x ⊤ 1 /p 2.
Quadratic functions y = f
( x ) = x T Ix /p
, where I is the identity matrix
Continuous exponential function y = f
( x ) = (cid:80) pi
=1 e 0 .
2 x i /p 4.
Cosine functions y
= f ( x ) = (cid:80) pi
=1 cos(0 .
5 πx i ) /p
(Non-smooth) ℓ 1 -norm function y = f
( x ) =
∥ x ∥ 1 /p 6.
(Non-continuous)
Piecewise linear function
[Equation]
We let x i ∼ Unif( − 10 , 10) for each coordinate i , and the noise level σ = 0 .
1 by default.
We normalize all the functions above for fair comparison among different functions so that y ∈ [ − 9 , 9] when x ∈ [ − 10 , 10] p .
In particular, to assess whether LIFT is better at dealing with positive numbers or integers, we generate additional datasets by further manipulating the ( x , y ) distribution of linear and piecewise functions.
For datasets with real numbers, we generate the 1D dataset x ∼ Unif( − 150 , − 150) .
For datasets with only positive numbers, we generate the dataset x i ∼ Unif(0 , 300) .
To generate the datasets with all integer prompts, we round down all the features to integers.
For visualization, in addition to the training, validation, and test datasets, we generated grid datasets.
Unless otherwise stated, we generate uniformly spaced 200 samples for 1D visualizations and 2,500 samples for 2D visualizations, with each coordinate x i ∈ [ − 10 , 10] for i = 1 , . .
p .
To visualize the extrapolation performance, we let the x i ∈
[ − 15 , 15] .
( Real datasets )
We consider four real datasets: Medical Insurance dataset [41] with 1,338 samples and 6 features, Combined Cycle Power Plant (CCPP) dataset [42] with 9,568 samples and 4 features,
Servo [43] dataset with 167 samples and 4 features, and Student [44] dataset with 649 samples and
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
Pizza
[Figure] [Figure]
[Figure] [Figure]
[Figure]
- PermutedMNISTP-MNIST  70k - FashionMNISTFMNIST  70k - PermutedFashionMNISTP-FMNIST  70kfeatures.
The Medical Insurance and Student datasets contain feature names that can be interpreted using common knowledge (see feature lists in Table 35), while CCPP and Servo do not.
C. LIFT and Baseline Implementation
This section provides details of our models and implementation.
We describe our pretrained language models and the baseline implementations (C.2.1), the computing resources used for running experiments (C.2.2), and how to fine-tune and select the hyperparameters (C.2.3).
C.2.1 Pretrained Language Models and Baselines
Pretrained language models.
Our main results are with two pretrained language models: GPTJ [31] and GPT-3 [19].
We mainly focus on GPT-J for the reproducibility purpose and provide additional results on GPT-3 as reference.
For GPT-J, we use a quantized version 3 of 6 billion parameters with 8-bit weights.
For GPT-3, we use the GPT-3 OpenAI API 4 and employ the Ada version by default.
In Sec.
E.2, we compare two previously mentioned models with three bigger versions of GPT-3 ( Baggage, Curie, Davinci ).
The largest one is Davinci -GPT-3 containing approximately 175 billions of parameters.
Since GPTs can output any language token, the output might not be appropriate for the desired task.
For example, GPT might output non-numerical words for regression task approximating a function f : R n → R .
In such a case, we categorize this as invalid output .
Baselines.
For XGBoost, we use the open-source XGBoost module 5 .
For other baselines, we implement them using scikit-learn module 6 .
C.2.2 Computing Resources
For experiments on GPT-J, we use GPU computing from two 24GB-RTX3090 GPUs and AWS EC instances 7 ( p3.8xlarge , p3.2xlarge ).
For other models, we run experiments on CPU instances.
C.2.3 Hyperparameter Selection
In fine-tuning GPT-J, we use Adam-8bit optimizer implementation 8 with weight decay of 0 .and
6 warm-up steps.
The learning rate is chosen from 1e-4 and 2e-4 for the synthetic/OpenML datasets, and 1e-5 for the vision datasets.
We use a linear learning scheduler for the optimizer.
For classification, the batch size depends on the number of features of the datasets.
We set the batch size to be 128, 32, 16, and 2 for datasets with the number of features being no greater than 2, between 2 and 6, between 6 and 20, and greater than 20, respectively.
For regression, we set batch size asby default and reduce it to 1 to avoid the memory issue when the number of features increases.
For
GPT-3, we use the API provided by OpenAI to perform black-box GPT-3 fine-tuning with the default setting.
Our implementation is with PyTorch framework 9 .
We perform hyperparameter selection based on validation results for all methods for a fair comparison.
The hyperparameter tuning scheme for all methods is detailed as follows:
Classification methods.
• LIFT/GPT-J: number of epochs ∈ { 5 , 10 , 15 } •
LIFT/GPT-3: learning rate multiplier ∈ { 0 . 05 , 0 . 1 , 0 . 2 } • Random Forest ( RF ): maximum depth ∈ { 3 , 5 , 10 } , minimum number of samples required to split an internal node ∈ { 2 , 5 , 10 } • Decision Tree ( DT ): maximum depth of the tree ∈ { 3 , 5 , 20 } and criterion ∈ { Gini impurity, Shannon information gain } • Support Vector Machine ( SVM ): kernel ∈ { linear kernel, radial basis function } , regulariza- tion parameter ∈ { 1 , 10 , 100 } • Multilayer Perceptron ( MLP ): initial learning rate ∈ { 0 . 001 , 0 . 01 , 0 . 1 } • Logistic regression ( LogReg ): inverse of regularization strength ∈ { 1 , 10 , 100 } • K-Nearest Neighbor ( KNN ): number of neighbors to use ∈ { 1 , 3 , 5 } , power parameter for the Minkowski metric ∈ { 1 , 2 } • XGBoost ( XG ): maximum depth ∈ { 3 , 5 , 10 }
Regression methods
• LIFT/GPT-J: number of epochs ∈ { 2 , 6 , 10 } •
LIFT/GPT-3: learning rate multiplier ∈ { 0 . 05 , 0 . 1 , 0 . 2 } • Polynomial Regression ( PR ): no hyperparameter selection but fixed the degree at 3 since higher-order polynomial regression introduces out-of-memory error, especially for highdimensional datasets • K-Nearest Neighbor ( KNN ): number of neighbors ∈ { 2 , 5 , 8 } • Kernel Regression ( KR ): Gamma parameter of Radial Basis Kernel ∈ { 0 . 01 , 0 .
1 , 1 } • Multilayer Perceptron
( MLP ): initial learning rate ∈ { 0 . 0001 , 0 .
001 , 0 . 01 } • Gradient Boosting Tree ( GBT ): learning rate ∈ { 0 .
001 , 0 . 01 , 0 . 1 } • Random Forest ( RF ): maximum depth ∈ { 4 , 6 } • Gaussian Process ( GP ): the number of optimizer restarts used to find parameters of the kernel that maximize the log marginal likelihood ∈ { 5 , 10 } .
Note that we perform model selection based on validation RAE, instead of validation loss.
D Detailed and Extended Results of Experiments in the Main Paper
In this section, we provide the extended version of experimental results presented in the main paper, including the primary findings of LIFT (D.1), the specific learning properties of LIFT (D.2), and improving techniques for LIFT (D.3).
D.1 Results for Basic Findings of LIFT (Section 3) D.1.1 How Well Does LIFT Perform on Standard ML Tasks?
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
We now provide full results of classification and regression performances with all baselines, including the investigation of the interpolation and extrapolation performance of LIFT for regression tasks.
Classification.
Table 17 presents the classification performance with other baselines, including
KNN, MLP, and Random Forest.
We further consider two additional baselines here, which has larger model sizes compared to the baselines we discussed in Sec. 2.
TabNet [157] and TabTransformer [156] are deep neural network models based on architectures specifically designed for tabular data.
The results are presented in Table 18.
We observe that LIFT achieves comparable performance to TabNet and TabTransformer.
This further highlight the good performance of LIFT.
Regression.
Table 19 provides the regression evaluation with all regression baselines on synthetic datasets, and Table 20 provides the results for real datasets.
Since experiments with LIFT/GPT-J are conduced on AWS and local server and due to this limitation of memory resources, we fail to run experiments of LIFT/GPT-J on high-dimensional datasets.
Therefore, for 50D and 120D synthetic datasets, only results of LIFT/GPT-3 are reported.
We further provide the visualization of regression models.
Fig.
21 and 22 visualize the 2D predictions for various functions with 200 and 1000 samples training datasets, respectively.
Each coordinate of the training sample is drawn uniformly from [ − 10 , 10] .
Specifically, the prediction is performed on the interval [ − 12 , 12] .
Fig. 23 and Fig.
24 visualize the interpolation and extrapolation of various methods.
All methods fail to extrapolate and interpolate well for all functions.
It turns out that LIFT is not having good interpolation performance except in the linear regression case.
An interesting observation is that
LIFT tends to output seen values (from training data) for extrapolation.
For example, in Fig.
24b, the outputs of LIFTs for x / ∈ [ − 10 , 10] (extrapolation) lie in the range of outputs for x ∈ [ − 10 , 10] (trained data), and similar behaviors are observed for other functions as well.
[Table] [Table]
−
[Table]
P
[Table]
−
[Table]
−
[Table]
8 −8 K
R −
RAE: 0.
− 8 0 8 −8 K
NN −
RAE: 0.
− 8 0 8 −
NN −
RAE: 0.
− 8 0 8 −8 X
G −
RAE: 0.
− 8 0 8 −
R F −
RAE: 0.
− 8 0 8 −8 G
P −
RAE: 0.
[Table]
−
[Table]
−
[Table]
−
[Table]
RAE:
[Table]
−
[Table]
8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
[Table] [Table]
−
[Table] [Table]
[Table] [Table]
−
[Table]
−
[Table]
−
[Table]
8 −8 −
RAE:
[Table]
−
[Table]
8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
[Table]
−
[Table]
−
[Table]
−
[Table]
−8 −
[Table]
RAE
[Table]
−
[Table]
−8 −
[Table] [Table]
[Table] [Table]
8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −  (e) l1-norm − 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −  (f) piecewise
[Table]
[Table] [Table]
−
[Table]
−
K
R −
RAE: 0.
−
[Table]
−
K
NN −
RAE: 0.
− 8 0 8 −
NN −
RAE: 0.
− 8 0 8 −8 X
G −
RAE: 0.
− 8 0 8 −
R F −
RAE: 0.
− 8 0 8 −8 G
P −
RAE: 0.
[Table]
−
[Table]
RAE
[Table]
−
[Table]
−8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
[Table] [Table]
−
[Table] [Table]
−
[Table] [Table]
[Table]
[Table] [Table]
−
[Table]
−8 −
RAE
[Table]
−
[Table]
8 −8 −
RAE:
[Table]
− 8 0 8 −8 −
RAE: 0.
− 8 0 8 −8 −
RAE: 0.
[Table]
−
[Table]
−
[Table]
−
[Table]
−
[Table]
RAE
[Table]
−
[Table]
8 −8 −
RAE
[Table]
− 8 0 8 −8 −
RAE
[Table] [Table]
−
[Table]
−
[Table] [Table]
−
[Table]
8 −
[Table]
−
[Table]
−8 −
[Figure]
−
[Figure]
− 8 0 8 −8 −  (e) l1-norm − 8 0 8 −8 −
RAE: 0.
− 8
[Figure]
8 −
RAE: 0.
[Figure] [Figure]
RAE: 0.
[Figure] [Figure]
− 8 0 8 −8 −  (f) piecewise
−
G P T J
[Figure]
−
G P T−
P R −
K R −
K NN −
M L P −
X G −
R F −  −
G P −  −  −  −  − 7 x
We consider two baselines here TabNet [157] TabTransformer [156] We observe that LIFT achieves comparable performance to TabNet and TabTransformer, which is more evidence of the good performance of LIFT
[Table] [Table] [Table]
[Figure] [Figure]
L I F T /
G P T
[Figure] [Figure]
P R −
K R
[Figure]
K N N
[Figure] [Figure]
L P
[Figure] [Figure]
T
[Figure] [Figure]
D.1.2 How Many Samples Does LIFT Need?
Fig. 25a and Fig.
25b provide the sample complexity comparisons between evaluated methods in the classification and regression settings.
D.1.
Can We
Understand
the Inductive
Biases
of
Language
Models via
LIFT?
Continuing from Sec. 3.4, here we provide more experiments with detailed measurements that quantify the similarity between decision boundaries.
Visualizing decision boundary.
We construct datasets with various classification complexities to investigate the adaptability of LIFT.
In particular, we construct three datasets: a binary classification dataset, a 3-class and a 5-class dataset (shown in the first column of Fig. 26a, Fig. 26b, and Fig. 26c).
We call these datasets neural-net-based synthetic datasets since we generate them using a 2-layer
[Figure] [Figure]
.g.
, piecewise function.
neural network.
See Fig.
14 and Appendix C.1 for detailed explanations of how we generated these datasets.
Note that Fig.
26a is the same as Fig.
6 in Sec.
3.4, which we put for completeness here.
Fig.
26 visualizes the decision boundaries of models trained on the neural-net-based synthetic data.
In addition, we also visualize the decision boundaries of models trained on the label-corrupted versions of three binary classification datasets, with the corruption probabilities being 5% and 20% (see details in Sec.
D.1.4), shown in Fig. 26d and Fig.
26e.
Specifically, we consider the binary classification tasks and flip the training data labels with the provided probabilities.
Overall, the same observation of
Sec. 3.4 also holds for the 3-class and 5-class datasets — both GPT-J and GPT-3 models fine-tuned with LIFT can adapt well to different boundaries.
They can capture the rough shapes of the decision boundaries in all three settings.
Besides, when the level of corruption increases, decision trees and XGboost are the most affected baselines.
While roughly capturing the boundary, LIFT/GPT-J also shows more noisy predictions.
In contrast, LIFT/GPT-3 displays great robustness against the corrupted labels while capturing the correct boundary shapes.
Nevertheless, this experiment indicates the different behaviors of LIFTs from the baseline algorithms and their adaptability to different types of decision boundary.
One interesting observation here is that LIFT’s decision boundaries are axis-parallel and show a lot of fractals.
The axis-parallel boundary looks similar to the boundary of tree-based classifiers, and the fractal shapes of LIFT’s boundaries are similar to the observations on the decision boundaries of
True Function
LogReg
KNN
DT
MLP
RBF-SVM
RF
XG
LIFT/GPT-J LIFT/GPT- (a) Binary classification
True Function
LogReg
KNN
DT
MLP
RBF-SVM
RF
XG
LIFT/GPT-J LIFT/GPT- (b) 3-way classification
True Function
LogReg
KNN
DT
MLP
RBF-SVM
RF
XG
LIFT/GPT-J LIFT/GPT- (c) 5-way classification
True Function
LogReg
KNN
DT
MLP
RBF-SVM
RF
XG
LIFT/GPT-J LIFT/GPT-
[Figure]
True Function
LogReg
KNN
DT
MLP
RBF-SVM
RF
XG
LIFT/GPT-J LIFT/GPT-
[Figure]
The first of each shows the decision boundary the training datasets we visualize the decision boundaries of models trained on datasets with two, three and five classes.
In (d) and (
), we consider the label-corrupted version of the binary-class datasets, with corruption probabilities of 5% and 20%.
We find that LIFT/GPTs adapt well and roughly estimate the true decision boundaries.
The shapes of LIFT’s decision boundary are likely to be axis- and multiple
[Table]
some convolution neural networks [49].
However, the main reason why LIFT’s decision boundary has such patterns seems to be due to the way it interprets numbers.
Since we rely solely on the language interface, there are some artifacts due to the decimal numeral system.
For instance, 0 .and 0 .
99 are only one-character different, but 0 .
99 and 1 . 00 are three-characters different.
We believe that such an artifact is the reason behind axis-parallel decision boundaries and fractal-like patterns.
Quantifying the similarity of decision boundaries.
To further verify whether LIFTs behave similarly to any standard algorithm
[Table]
we quantify the similarity between the decision boundaries of
LIFT/GPT-3 and those of the baselines.
Specifically, the similarity score is the percentage of the exact classification matches between LIFT/GPT-3 and the compared method.
We randomly sample two sets of 200 data points from the original dataset for training and evaluation, respectively, and the results of all methods are reported in Table 27.
Based on the similarity score, while we observe no similar discernible pattern between LIFT/GPT-3 and the baselines, we find that LIFT/GPT-3 appear to share the most similar behavior pattern to RBF-SVM, random forest (E=100), and MLP (W=100).
D.1.4 How Robust Is LIFT?
Robustness to outliers in training data.
Fig.
28 visualizes the outlier robustness results discussed in Sec. 3.5.
Robustness to label corruption.
We choose a subset of samples and corrupt the label of the chosen samples, using two corruption schemes: (1) random errors (randomly select another label with an equal probability for all labels) and (2) systematic errors [163] (replace a label with its next label in the target label list, e.g. , 0 → 1 , 1 → 2 , 2 → 0 for a 3-way classification).
As shown in
Fig. 29 , LIFT/GPT-3 can perform well under label corruption; it follows the general trend of other baselines, not outperforming or underperforming.
Note that LIFT/GPT-3 almost always displays greater robustness than KNN.
Table 30 and Table 31 extend the results reported in Fig. 29.
These additional datasets follow a similar trend to what was discussed in Sec. 3.5.
Robustness to class-imbalance of training data.
We evaluate LIFT on class-imbalanced classification tasks (OpenML datasets Pizza , Climate , and Customers having IDs 1444, 1467, and 1511), shown in Table 32.
We use additional metrics: F1, precision, and recall (higher scores indicate better performance), which are considered as better measurements for the imbalanced data than the accuracy.
The higher values of the MCC’s accuracy imply the higher levels of imbalance in the data (50% shows the perfect balance).
For the reference, we report the performances of the deterministic classifiers that always return the label of class 0 (DC-0) and class 1 (DC-1).
[Figure]
Though evaluated datasets all have high class-imbalance ratios, we find that LIFT can perform well, achieving high F1, precision, and recall scores across the tasks.
For instance, on the Customers dataset (the class-imbalance ratio is nearly 8 ), MCC gets 0 for both precision and recall as all predicted labels of MCC are 0 ( that is , the major class), while LIFT/GPT-J achieves the best recall ( 82 . 61 ± 7 . 10 ) and F1 scores ( 84 .
43 ± 1 . 43 ).
Here, the 0 value of precision and recall in MCC means that MCC classifies all samples as negative, which is the major class in the training dataset.
Robustness to feature corruption on test data.
Here we provide detailed experiment setting and experiment results on random noise perturbation.
Given a perturbation budget ϵ ≥ 0 , we consider two types of perturbation δ with ∥ δ ∥ ∞ ≤ ϵ : random noise and adversarial perturbation [164].
For random noise δ , we test on two types: (1) random Gaussian noise δ ∼ N ( 0 , I p ) scaled to satisfy ∥ δ ∥ ∞ =
ϵ , and (2) signed constant noise δ where each element δ i has magnitude ϵ and random sign.
For adversarial perturbation δ , we test on the transfer attack [51], i.e. , we generate an adversarial example ( x + δ , y ) for a source neural network (that we can access) with constraint ∥ δ ∥
≤ ε , and test whether the target network correctly classifies the adversarial examples.
Table 33 shows the results of LIFT and baselines for the MNIST classification problem.
We test on random noise (Gaussian and signed constant) and PGD attacks transferred from LeNet-5 and MLP.
We compare the results for three networks:
LeNet-5, MLP (having 2 hidden layers, each withneurons and 100 neurons), and LIFT/GPT-3.
LIFT/GPT-3 is observed to tolerate random noise (both
Gaussian and signed constant) for small perturbation radius ϵ = 0 .
We do not include the result for LIFT/GPT-J since it is not even robust against simple noise.
Please refer to Section 5.2 to check the vulnerability of LIFT/GPT-J against test-time noise and how data augmentation improves the robustness of LIFT/GPT-J.
[Figure]
D.1.5 Does LIFT Need Large-Scale Models Pretrained on Natural Language Data?
Continuing from Sec.
3.6, we provide the detailed setup of this experiment.
We obtain the Gibberish model by fine-tuning the entire GPT-J model (rather than LoRA [24]) on the Gibberish dataset [55] for 10 epochs at learning rate 0.1 .
For LIFT/Code-Gen, LIFT/CodeParrot, and LIFT/Gibberish, we follow the same e as LIFT/GPT-J, which we have discussed in Appendix C. To fine-tune LIFT/RandGPT-J for specific tasks, we set the temperature as 1 instead, as we observed the 0 temperature consistently gives us poor performances.
Note that only 10%–15% of LIFT/Rand-GPT-J’s outputs of are valid.
The accuracies listed in the table are computed among valid outputs.
Other settings of
LIFT/RAnd-GPT-J are the same as LIFT/GPT-J.
D.2 Results for LIFT-Specific Learning Properties (Section 4) D.2.1 Does LIFT Benefit from Incorporating Feature Names?
Continuing from Sec.
4.1, we provide more details of experiment settings, further evaluations with
GPT-J models (Table 34), and results on regression tasks (Table 35) here.
Prompt templates.
We design five prompts templates to assess how incorporating feature names affects the performance of LIFT.
For instance, consider a data sample “ x = ( English speaker , 23 , 3 , summer , 19) , y = 3 ” from TAE dataset where the feature names are “ native speaker, instructor, course, semester, class size ”, and the target attribute is teaching performance .
We can incorporate the contextual information by either simply replacing the “ x i ” in the prompts with the corresponding feature names or converting this sample into a coherent sentence.
Meanwhile, we also investigate how shuffled feature names affect the performance of LIFT by designing the prompts accordingly.
For illustration purposes, we provide the example of the five prompt templates as below.
• (W/O Names)
“ When we have x 1 = 1 , x 2 = 23 , x 3 = 3 , x 4 = 1 , x 5 = 19 , what should be y value? ”
• (Correct-Names I)
“ When we have native speaker=
English speaker, course instructor= 23 , course= 3 , semester=summer, class size=19, how is the teaching performance? ”
[Table]
In this regime
[Table]
we corrupt a label by assigning all corrupted labels of one class to a single label.
[Table]
imbalanced datasets in OpenML ( Pizza , Climate , Customers ).
All datasets are for binary classification and are highly imbalanced.
The class-imbalance ratio (Imb.
Ratio) is defined as the ratio of the number of samples in the majority class and that in the minority class.
Here, DC-0 and DC-1 refer to deterministic classifiers that constantly predict all samples as class 0 and 1 respectively.
MCC refers to the majority class classifier that returns the major class learned from the training dataset.
LIFT/GPTs achieve comparably high scores across the three tasks.
For instance, LIFT/GPT-J achieves the best F1 on datasets Pizza and Customers .
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] • (Correct-Names II) “ In the course 3 offered in the summer semester, there was a native English-speaking teaching assistant and an instructor whose ID is 23.
How is the teaching performance? ”
[Table]
• (Shuffled-Names I)
“ When we have semester=
English speaker, class size= 23 , semester= 3 , course instructor=summer, native speaker=19, how is the teaching performance? ”
• (Shuffled-Names II) “ In the course summer offered in the 3 semester, there was a 19 teaching assistant and an instructor whose ID is summer.
How is the teaching performance? ”
We note that the sentence generated using the (Shuffled-Names II) template can be incoherent.
Settings.
(Datasets)
Among the OpenML datasets evaluated in Table 4, we select three datasets:
CMC , TAE , and Vehicle (with IDs being 23, 48, and 54) whose all provided feature names are meaningful and relevant to the prediction task and the response values.
(Baselines)
We compare our target model LIFT when feature names are correctly incorporated ( Correct-Names I, II ) with the versions of LIFT when feature names are incorrectly incorporated with randomly shuffled orders ( Shuffled-Names I, II ) and when feature names are not included ( W/o Names ).
Also, we compare all models with the simple baseline MCC and the strong baseline XGBoost.
Classification.
We provide additional evaluation of LIFT/GPT-J on three datasets used in the main paper.
Table 34 presents our result with the same settings in the main paper.
We can see that correctly using feature names helps improve the performance of LIFT from the models without feature names or the models with randomly shuffled feature names.
This finding is consistent with the finding in the main papers on the usefulness of incorporating the feature names.
[Table]
Regression.
To investigate whether incorporating feature names in prompts improves the regression performance of LIFT, similar to the datasets selection process of classification tasks, we evaluate the effect of feature names on the datasets Insurance and Student , whose tasks can be helped by common knowledge.
To be more specific, while the task of Insurance dataset is to predict the insurance costs, the key features of Insurance dataset are age , body mass index , and smoke or not , which are intuitively closely related to the task.
For the Student dataset, the task is to predict students’ grades based on their weekly study time, previous grades, etc.
Therefore, the features and task of Student are also highly correlated.
Table 35 presents our evaluation of regression tasks.
We find that fine-tuning with feature names does not necessarily help with the regression tasks.
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
r d D e v i a t
i
[Figure]
n
[Figure] [Figure]
[Figure] [Figure]
[Figure]
D.2.2 Is LIFT Calibrated?
[Figure] [Figure]
[Figure]
S t a nd a r d D e v i a t i o n
ObservationPrediction
[Figure]
−
[Figure] [Figure]
[Figure] [Figure]
deviation of LIFT/GPT-J predictions are computed based 20 repeated random predictions.
We observe that the standard deviations of predictions from LIFT/GPT-3 aligns well with that of noisy training samples (observations), implying that LIFT/GPT-3 can reflect the confidence, i.e. , LIFT/GPT-3 is calibrated.
Continuing the discussion in Sec.
4.2, Fig.
37 indicates that LIFT/GPT-3 is calibrated.
D.2.3 Can we use LIFT for Generation?
Here we provide the detailed experiment setting of Sec. 4.3.
Data pre-processing.
We preprocess the MNIST dataset as below.
First, we crop each 28 ×image at the center to make an 18 × 18 image.
Then, we represent the cropped image as a sequence of 324 pixel values, where each pixel is an integer in { 0 , 1 , · · · , 255 } .
The training process for both tasks (i) and (ii) are identical, detailed as below.
Given an image in the training set, we put the digit of the image into a prompt and put the sequence of 324 pixel values into the completion, as in the example shown below.
• (Train Prompt) “ Generate an image of digit 9.
### ” • (Train Completion) “ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[Table]
[Table] [Table]
[Table]
The testing process is different for tasks (i) and (ii).
For task (i), we put the prompt as above and check the image drawn by pixel values generated in the completion.
For task (ii), the test prompt is set as follows.
First, we randomly sample an image with the target digit from the MNIST set.
We then include its digit number and the top half pixels (first 162 pixels) in the prompt.
Once this test prompt is given, we observe the remaining 162 pixel values generated in the completion, and check how the entire 324 pixels look like.
Examples of the test prompt are given below.
• (Test Prompt for task (i))
“ Generate an image of digit 9.
### ” • (Test Prompt for task (ii))
“ Generate an image of digit 9.
[Table]
[Table] [Table]
Experiment setup.
For GPT-3, we fine-tune with the pre-processed MNIST training set forepochs.
For GPT-J, we fine-tune for three epochs.
We use the learning rate 10 − 5 .
In the inference phase, as described above, we use different prompts for the two tasks and collect the first 324 and 162 white space-separated tokens in the output for tasks (i) and (ii), respectively.
In addition, for task (ii), we concatenate the provided 162 tokens in the prompt and the 162 tokens from the model output into a sequence of 324 tokens.
The collected 324 tokens are converted to an 18 × 18 image if all of them are valid numbers in { 0 , 1 , · · · , 255 } .
We assess the performance of LIFT as a generator under different temperatures ∈
Under a fixed temperature, we generate images of each digit 5 times.
We use the standard perplexity score to evaluate the performance of the obtained generative models.
Here perplexity is a standard metric for assessing the performance of a generative model, which reflects the inverse of the probability of the given data samples being produced by the model.
In this experiment, we assess the generalization performance of models by comparing the perplexity scores between training and test sets.
If the gap between the training perplexity and the test perplexity is small, it indicates that the model generalizes well.
Results.
Fig.
38 visualizes the output of LIFT in generating the images of different digits under different temperatures.
We observe that LIFT is able to generate reasonable images when both the digit number and top half pixels are given, as shown in Fig.
38c and Fig. 38d.
In contrast, Fig. 38a and Fig.
38b show that LIFT is able to work only under high temperatures ( e.g. , Temp = 0.9, 1 ).
This might be because, under low temperatures, the model is likely to generate output associated with the highest probability.
In contrast, a higher temperature gives chances to generate outputs associated with slightly lower probability, which introduces more variety and creativity.
It has been observed in compositional text generation tasks such as completing stories, high temperature leads to better and more creative performance [168].
[Table]
D.3 Results for improving techniques of LIFT (Section 5) D.3.1 Two-Stage Intermediate Fine-Tuning for LIFT
We provide an additional evaluation of the two-stage intermediate fine-tuning with LIFTGPT-J on four more datasets.
For any given dataset, we first generate two pretext tasks with simple synthetic
Gaussian samples (discussed in C.1).
We fine-tune the GPT with pretext tasks for a few (2–3) epochs, then fine-tune the newly fine-tuned GPT with the target (given) dataset.
Here, due to the black-box
API of GPT-3, we currently can neither keep the order of samples unchanged (pretext, target) during the fine-tuning stage nor fine-tune the model twice.
Hence, we only use GPT-J in this experiment.
Fig.
40 presents results from eight datasets, including a regression task, three OpenML tasks, and other synthetic tasks.
We see that two-stage intermediate LIFT helps to improve the original finetuning, especially when the number of training samples is small.
Its effect is more clearly shown in synthetic classification datasets ( Blobs , Circles , Moon , and Two Circles ).
We also observe that besides the number of features and number of classes, the pretexts do not need to represent any other characteristics of the target dataset, such as the linear/non-linear correlation or the relevance of features.
This makes it simpler to generate the pretexts.
[Figure] [Figure]
y
[Figure]
E Additional Experiments and Findings (NOT Discussed in the Main Paper)
Here, we provide results of additional experiments that have not been discussed in the main manuscript.
To be more specific, we study the effect of replacing the input or output layers (E.1), effect of large
LMs (E.2), quantitative classification evaluation on neural-net-based synthetic datasets (E.3)
[Figure]
and the ability of LIFT performing the ridge regression (E.4).
We also provide the visualization of LIFT’s training curve in Section E.5.
E.1 What Is the Effect of Replacing the Input or Output Layers?
In this experiment, we assess the performance of transformer fine-tuned with replaced input / output layer, following the methods used in Frozen Pretrained Transformers (FPT) [18].
We consider vanilla
FPT and its two variants.
Specifically, for vanilla FPT, we reinitialized a trainable input layer and a trainable output layer, with frozen pretrained GPT-J transformer architectures in the middle.
As in [18], the input dimension equals to the number of features and the output dimension equals to the number of classes, which varies depending on the tasks.
Table 41 compares the result of our method
LIFT/GPT-J, FPT and the two variants of FPT: (i) FPT (Output Only) which only replaces the output layer, and (ii) FPT (Input Only), which only replaces the input layer.
We observe that both FPT and
FPT (Output Only) perform slightly better than LIFT on almost all tested cases, while FPT (Input
Only) performs the worst.
Our justification for this observation is that training an output layer is similar to training a linear classifier, which might be easier than training an input layer as an encoder
[Table] [Table]
E.2 Does LIFT Benefit from Larger LMs?
[Table] [Table]
In this experiment
[Table]
we apply LIFT to different pretrained LMs to verify whether LIFT benefits more from larger LMs.
Together with previously used GPT-J and GPT-3 (the version named Ada ), we consider three bigger versions of GPT-3, namely Baggage , Curie
[Table]
and Davinci (in the ascending order of the number of parameters).
We compare all models on several classification tasks in Tableand regression tasks in Table 43.
Overall, we find that the performance gain of using larger LMs is not consistently significant for LIFT.
Although larger LMs outperform smaller LMs in many cases, the improvements are relatively small.
Verifying the capability of large LMs, when LIFT is not used.
We first verify if larger LMs are more helpful for the evaluated downstream tasks.
We evaluate LMs in the in-context classification when no fine-tuning (LIFT) is involved.
Table 42 shows consistent improvements in classification performance when the size of LMs increases across all the tasks.
Thus, larger LMs, with larger embedded knowledge, are more useful for these downstream tasks.
When LIFT is used.
Both Table 42 and Table 43 show that using larger LMs may positively affect
LIFT in several tasks and settings compared to the smaller LMs.
However, the performance gains from replacing the smaller LMs with larger LMs are not consistent across the settings.
For instance, in the classification settings without feature names, Davinci performs better than GPT-J on four datasets and worse than on one dataset.
For the setting with feature names, GPT-J performs better than Davinci on two out of three tasks.
Furthermore, the performance gains of large LMs over the smaller models are not relatively significant.
We note that LIFT always outperforms the in-context learning using the same pretrained LMs in most cases.
The regression results shown in Tablefurther confirm that the improvement from utilizing larger LMs is relatively small.
E.3 Quantitative Classification Evaluations on Neural-Net-Based Synthetic Datasets
[Figure]
Clean Data
[Figure]
In Sec. 3.4 and Appendix D.1.3, we assess how well LIFT/GPTs adapt to different shapes of decision boundaries on neural-net-based synthetic datasets.
We now provide the test accuracies for all models.
For binary-class datasets, Fig.
44 shows the accuracies on three binary-class datasets when the difficulty of classification tasks varies by using different network checkpoints at different epochs.
As the difficulty increases or the level of corruption increases, all methods tend to decrease classification accuracy.
We provide three settings of training data: clean data, 5% label corruption, and 20% label corruption.
We observe that LIFT/GPT-3 outperforms logistic regression and decision tree, especially when the label corruption is up to 20%.
However, LIFT/GPT-J is performing worse than other baselines in the data corruption scenarios.
For 3-class and 5-class datasets, both LIFT/GPT-J and LIFT/GPT-3 achieve approximately 90% , while the best baselines (MLP and XGBoost) obtains approximately 92% and 91% for the 3-class and 5-class data, respectively.
E.4 Can LIFT Perform Ridge Regression via Data Augmentation?
[Table]
As shown in Fig.
2, LIFT can perform linear regression.
We take one step further and study whether
LIFT can perform Ridge regression.
Note that this is a non-trivial task as the LIFT framework does not allow any changes to the loss function.
Consider a standard ridge regression problem solving the optimal w with p parameters so that ∥ y − Xw ∥ 22 + λ ∥ w ∥ 22 is minimized.
Note that this problem is equivalent to minimizing ∥ [ y T , 0] T − [ X T , √ λ
I ] T w ∥ 22 .
Therefore, if we add p additional training samples √ λI , one can perform ridge regression via data augmentation.
Inspired by this, we study whether one can perform ridge regression via data augmentation within the framework of the LIFT framework.
The results of LIFT on Ridge regression are reported in Table 45.
We observe that LIFT fails to perform Ridge regression.
This is expected, as LIFT is shown to be robust to outliers (in Sec.
3.5 and Appendix D.1.4).
E. LIFT’s Training Curve
We report the learning curves of LIFT in terms of LM-loss and accuracies/RAE for several classification and regression tasks.
We observe a decrease in training loss over the tasks and datasets.
We select the best models based on the validation criteria (accuracy for classification and RAE for regression) on the validation sets.
Fig.
46 visualize the accuracy and loss of LIFT/GPT-J in the training and validation process for classification tasks.
For the regression task, Fig. 47 shows that the decrease in
RAE does not necessarily imply a decrease in loss.
Furthermore, we observe that LIFT only requires a few epochs to achieve good performance.
[Figure]
plot the accuracy (top row) and the loss (bottom row) of LIFT varying the number of training epochs.
[Figure]
We observe that LIFT/GPT-J only requires a few epochs to achieve good performance.
F Additional Discussion
Continuing from Sec. 7, here we elaborate on the difficulty of regression tasks and the broader impact of LIFT, and discuss the limitation on classification tasks and other open questions.
F.1 Limitations and Open Questions
The difficulty of regression tasks.
For regression tasks, in addition to poor performance on high-dimensional functions, some interesting phenomena observed in the classification tasks are not consistently observed in the regression tasks.
For example, incorporating feature names in the prompts does not consistently improve LIFT in the regression tasks (see Sec. D.2.1).
As previously discussed in Sec. 3.1, the difficulty of regression tasks on LIFT may come from the classification loss function used in LMs.
Due to the adoption of the classification loss function, two different predictions will lead to the same loss, even if one of the predictions is closer to the true y value.
As a result, we also observe that a reduction in RAE does not necessarily imply a reduction in LM loss (see Fig. 47).
Therefore, we use RAE as the criterion for model selection.
Moreover, how LIFT understands numerical values may also limit the regression performance of
LIFT.
Recent works [169, 134, 170, 171] have illustrated the difficulty and failures of the existing
LMs in understanding the numbers because two numbers with close values can have very different tokenizations [134].
Recent attempts [132, 172, 173, 174, 175] propose new encoding schemes of numbers to improve the LMs’ numerical capabilities, probably helping LIFT in the regression tasks.
A promising method for improving LIFT on regression is level encoding .
The idea of level encoding is to discretize the continuous values of the output y to better utilize the classification loss of LMs.
Assuming that the range of y is known, we can partition this range into a finite number of bins and represent all values in the same bin by a unique canonical representation in a way that the number of mismatched bits between the representations of two values is proportional to their absolute difference.
For instance, for all real-value y ∈ [0 , 3] , we can define three bins as { [0 , 1) , [1 , 2) , [2 , 3] } with the canonical representations being 00 , 01 , 11 .
With these bins, 0.3 and 0.7 are represented as 00 , and 1 . 5 and 1 . 1 are represented as 01 .
The distance between representations of 0 . 3 and 1 . 1 is onlybit, which is proportional to their absolute distance of 0 . 8 .
For the training of LIFT, we convert all output values in the original training dataset into the level-encoding canonical representation and use them as the target values.
By using the level encoding technique, the loss function of LMs can better capture the distance between the prediction and the true values, thus potentially improving the generalization of LIFT on regression tasks.
We leave this as one of the interesting directions for our future investigation.
The limitation of LIFT on classification tasks.
We observe that LIFT does not perform comparably well on classification tasks when the number of classes is large.
For instance, Table 4 shows that the accuracies of LIFT/GPT-3 are lower than RBF-SVM and XGboost on the datasets withclasses.
Another limitation is that the dimension of features LIFT can handle is upper bounded due to the limited context length of LMs.
This limitation may be mitigated by using LMs with a more memory-efficient variant or implementation of transformer models, e.g. , see [176].
Other open questions.
In addition to previously discussed questions of improving LIFT for regression and classification tasks, our pioneering work on LIFT is also expected to open up interesting research questions on generalist models.
First, can generalist LMs ( e.g. , GPTs) play a leading role in developing universal models that can adapt well to any data?
Second, can we apply
LIFT to different generalist models, such as GATO [89]?
F.2 Broader Impact
LIFT greatly simplifies the machine learning pipeline that requires only the reformatting of training datasets of the target task.
This simplicity helps enable no-code ML for the masses, where general users without prior knowledge of the ML frameworks can use LIFT for their target non-language tasks by properly designing the input/output prompt format.
Therefore, LIFT can apply to a wide range of applications and areas, such as credit loaning, disease diagnosis, and criminal sentencing.
This is closely related to the line of automated machine learning research [90, 91], which aims to automate the standard machine learning methods pipeline.
Employing LIFT without careful justification or understanding will lead to undesired outcomes, such as discrimination.
Since most existing language models (LMs) are pretrained on a large amount of human-annotated data, LIFT could exhibit discrimination against different demographic groups ( e.g. , gender, race, ethnicity) due to the bias existing in the training datasets.
In other words, LIFT may prefer certain groups while making decisions in downstream tasks, especially when feature names and different demographic contexts are fed at training and inference time.
This effect is exacerbated by the use of large pretrained LMs ( i.e. , GPT-J and GPT-3), which have been known to inherently contain bias [177].
The bias in the pretraining data for these large language models adds an opaque layer to regression and classification tasks beyond bias within the downstream data.
Therefore, adopting LIFT in tasks that consider demographic information requires more consideration to avoid discrimination.
To further remove the bias, users can combine LIFT with the existing fairness-aware reweighting mechanisms [178, 179] or data augmentation and parameter-efficient fine-tuning techniques [180].
Finally, we emphasize that more model evaluation steps are required when applying LIFT instead of using it as a panacea for all applications.
We believe our work can significantly benefit society by providing a simple tool for handling various tasks with proper justification.