On Embeddings for Numerical Features in Tabular Deep Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Recently, Transformer-like deep architectures have shown strong performance ontabular data problems.
Unlike traditional models, e.g., MLP, these architecturesmap scalar values of numerical features to high-dimensional embeddings beforemixing them in the main backbone.
In this work, we argue that embeddings fornumerical features are an underexplored degree of freedom in tabular DL, whichallows constructing more powerful DL models and competing with GBDT on some GBDT-friendly benchmarks.
We start by describing two conceptually differentapproaches to building embedding modules: the first one is based on a piecewiselinear encoding of scalar values, and the second one utilizes periodic activations.
Then, we empirically demonstrate that these two approaches can lead to significantperformance boosts compared to the embeddings based on conventional blocks suchas linear layers and ReLU activations.
Importantly, we also show that embeddingnumerical features is beneficial for many backbones, not only for Transformers.
Specifically, after proper embeddings, simple MLP-like models can perform onpar with the attention-based architectures.
Overall, we highlight embeddings fornumerical features as an important design aspect with good potential for furtherimprovements in tabular DL.
Introduction
Tabular data problems are currently a final frontier for deep learning research.
While the most recentbreakthroughs in NLP, vision and speech are achieved by deep models [11], their success in thetabular domain is not convincing yet.
Despite a large number of proposed architectures for tabular DL[2, 3, 12, 16, 20, 23, 30, 39, 40], the performance gap between them and the “shallow” ensembles ofdecision trees, like GBDT, often remains significant [12, 36].
The recent line of works [12, 23, 39] reduce this performance gap by successfully adapting the
Transformer architecture [44] for the tabular domain.
Compared to traditional models, like MLPor ResNet, the proposed Transformer-like architectures have a specific way to handle numericalfeatures of the data.
Namely, they map scalar values of numerical features to high-dimensionalembedding vectors, which are then mixed by the self-attention modules.
Beyond transformers,mapping numerical features to vectors was also employed in different forms in the CTR predictionproblems [7, 13, 40].
Nevertheless, the literature is mostly focused on developing more powerfulbackbones while keeping the design of embedding modules relatively simple.
In particular, theexisting architectures [12, 13, 23, 39, 40] construct embeddings for numerical features using quiterestrictive parametric mappings, e.g., linear functions, which can lead to suboptimal performance.
Inthis work, we demonstrate that the embedding step has a substantial impact on the model effectiveness,and its proper design can be a game-changer in tabular DL
Specifically, we describe two different building blocks suitable for constructing embeddings fornumerical features.
The first one is a piecewise linear encoding that produces alternative initial representations for the original scalar values and is based on feature binning, a long-existing preprocessingtechnique [10].
The second one relies on periodic activation functions, which is inspired by theirusage in implicit neural representations [27, 38, 41] and positional encodings in NLP and CV tasks[24, 44].
We observe that DL models equipped with our embedding schemes successfully competewith GBDT on GBDT-friendly benchmarks and achieve the new state-of-the-art of tabular DL.
As another important finding, we demonstrate that the step of embedding the numerical features isuniversally beneficial for different deep architectures, not only for Transformer-like ones.
In particular,we show, that after proper embeddings, simple MLP-like architectures often provide the performancecomparable to the state-of-the-art attention-based models.
Overall, our work demonstrates the largeimpact of the embeddings of numerical features on the tabular DL performance and shows thepotential of investigating more advanced embedding schemes in future research.
To sum up, our contributions are as follows:
We demonstrate that embedding schemes for numerical features are an underexplored
research question in tabular DL.
Namely, we show that more expressive embedding schemes
can provide substantial performance improvements over prior models.
We show that the profit from embedding numerical features is not specific for Transformer-
like architectures, and proper embedding schemes benefit traditional models as well.
On a number of public benchmarks, we achieve the new state-of-the-art of tabular DL.
Related work
Tabular deep learning.
During several recent years, the community has proposed a large number ofdeep models for tabular data [2, 3, 12, 14, 16, 20, 23, 30, 39, 40, 45].
However, when systematicallyevaluated, these models do not consistently outperform the ensembles of decision trees, such as GBDT(Gradient Boosting Decision Tree) [6, 18, 31], which are typically the top-choice in various MLcompetitions [12, 36].
Moreover, several recent works have shown that the proposed sophisticatedarchitectures are not superior to properly tuned simple models, like MLP and ResNet [12, 17].
In thiswork, unlike the prior literature, we do not aim to propose a new backbone architecture.
Instead, wefocus on more accurate ways to handle numerical features, and our developments can be potentiallycombined with any model, including traditional MLPs and more recent Transformer-like ones.
Transformers in tabular DL.
Due to the tremendous success of Transformers for different domains[9, 44], several recent works adapt their self-attention design for tabular DL as well [12, 16, 23, 39].
Compared to existing alternatives, applying self-attention modules to the numerical features of tabulardata requires mapping the scalar values of these features to high-dimensional embedding vectors.
So far, the existing architectures perform this “scalar” → “vector” mapping by relatively simplecomputational blocks, which, in practice, can limit the model expressiveness.
For instance, therecent FT-Transformer architecture [12] employs only a single linear layer.
In our experiments, wedemonstrate that such embedding schemes can provide suboptimal performance, and more advancedschemes often lead to substantial profit.
CTR Prediction.
In CTR prediction problems, objects are represented by numerical and categoricalfeatures, which makes this field highly relevant to tabular data problems.
In several works, numericalfeatures are handled in some non-trivial way while not being the central part of the research [7, 40].
Recently, however, a more advanced scheme has been proposed in Guo et al.
Nevertheless, it isstill based on linear layers and conventional activation functions, which we found to be suboptimal inour evaluation.
Feature binning.
Binning is a discretization technique that converts numerical features to categoricalfeatures.
Namely, for a given feature, its value range is split into bins (intervals), after which theoriginal feature values are replaced with discrete descriptors (e.g. bin indices or one-hot vectors) ofthe corresponding bins.
We point to the work by Dougherty et al.
[10] which performs an overview of some classic approaches to binning and can serve as an entry point to the relevant literature on the topic.
In our work, however, we utilize bins in a different way.
Specifically, we use their edges to construct lossless piecewise linear representations of the original scalar values.
It turns out that suchrepresentations can provide substantial benefit to deep models on several tabular problems
Periodic activations.
Recently, periodic activation functions have become a key component inprocessing coordinates-like inputs, which is required in many applications.
Examples include NLP[44], vision [24], implicit neural representations [27, 38, 41].
In our work, we show that periodicactivations can be used to construct powerful embedding modules for numerical features in tabulardata problems.
Importantly, contrary to some of the aforementioned papers, where components ofthe multidimensional coordinates are mixed (e.g. with linear layers) before passing them to periodicfunctions [38, 41], we find it crucial to embed each feature separately before mixing them in the mainbackbone.
Embeddings for numerical features
In this section, we describe the general framework for what we call "embeddings for numericalfeatures" and the main building blocks used in the experimental comparison in section 4.
Notation.
For a given supervised learning problem on tabular data, we denote the dataset as 100 (cid:8)(cid:0) x
j , y j (cid:1)(cid:9) n j =1 where y j ∈ Y represents the object’s label and x j = (cid:0) x
j ( num ) , x j ( cat ) (cid:1) ∈
X represents the object’s features (numerical and categorical).
x j ( num ) i , in turn, denotes the i -th numericalfeature of the j -th object.
Depending on the context, the j index can be omitted.
The dataset is splitinto three disjoint parts: 1 , n = J train ∪ J val ∪ J test , where the “train” part is used for training, the“validation” part is used for early stopping and hyperparameter tuning, and the “test” part is used forthe final evaluation.
3.1 General framework
We formalize the notion of "embeddings for numerical features" as z i = f i (( x ( num ) i ) ∈
R d i ,where f i ( x ) is the embedding function for the i -th numerical feature, z i is the embedding of thei -th numerical feature and d i is the dimensionality of the embedding.
Importantly, the proposedframework implies that embeddings for all features are computed independently of each other.
Notethat the function f
i can depend on parameters that are trained as a part of the whole model or insome other fashion (e.g. before the main optimization).
In this work, we consider only embeddingschemes where the embedding functions for all features are of the same functional form.
We nevershare parameters of embedding functions of different features.
The subsequent use of the embeddings depends on the model backbone.
For MLP-like architectures,they are concatenated into one flat vector (see Appendix A for illustrations).
For Transformer-basedarchitectures, no extra step is performed and the embeddings are passed as is, so the usage is definedby the original architectures.
3.2 Piecewise linear encoding
While vanilla MLP is known to be a universal approximator [8, 15], in practice, due to optimizationpeculiarities, it has limitations in its learning capabilities [33].
However, the recent work by Tanciket al.
[41] uncovers the case where changing the input space alleviates the above issue.
Thisobservation motivates us to check if changing the representations of the original scalar values ofnumerical features can improve the learning capabilities of tabular DL models.
At this point, we take inspiration from the one-hot encoding algorithm that is widely and successfullyused for representing discrete entities such as categorical features in tabular data problems or tokensin NLP.
We note that the one-hot representation can be seen as an opposite solution to the scalarrepresentation in terms of the trade-off between parameter efficiency and expressivity.
To checkwhether the one-hot-like approach can be beneficial for tabular DL models, we design a continuousalternative to the one-hot encoding (since the vanilla one-hot encoding is barely applicable tonumerical features).
Formally, for the i -th numerical feature, we split its value range into the disjoint set of T i intervals
B i 1 , . . .
, B iT , which we call bins : B it =
[ b it − 1 , b it ) .
The splitting algorithm is an important implementation detail that we discuss later.
From now on, we omit the feature index i for simplicity.
Oncethe bins are determined, we define the encoding scheme as in Equation 1:
PLE ( x ) =
[ e 1 , . .
, e T ] ∈ R T
[Equation]
b t − b t
[Equation]
, otherwise
[Equation]
where PLE stands for “ p eicewise l inear e ncoding”.
We provide the visualization in
Figure 1.
Note that:
• PLE produces alternative initial representations for the numerical features and can be viewed as
a preprocessing strategy.
These representations are computed once and then used instead of the
original scalar values
the main optimization.
• For T = 1 , the PLE -representation is effectively equivalent to the scalar representation.
• Contrary to categorical features, numerical features are ordered; we express that by setting to
the components corresponding to bins with the right boundaries lower than the given feature
value (this approach resembles how labels are encoded in ordinal regression problems).
• The cases ( x < b 0 ) and ( x ≥ b T ) are also covered by Equation 1 (which leads to ( e 1 ≤ 0) and( e T ≥ 1) respectively).
• The choice to make the representation piecewise linear is itself a subject for discussion.
We
analyze some alternatives in subsection 5.2.
• PLE -representations are invariant to shifting and scaling.
In subsection D.1, we show that models
using PLE are less sensitive to data preprocessing than vanilla models.
A note on attention-based models.
While the described PLE -representations can be passed to MLPlike models as is, attention-based models are inherently invariant to the order of input embeddings, soone additional step is required to add the information about feature indices to the obtained encodings.
Technically, we observe that it is enough to place one linear layer after PLE (without sharing weightsbetween features).
Conceptually, however, this solution has a clear semantic interpretation.
Namely,it is equivalent to allocating one trainable embedding v t ∈ R d for each bin B t and obtaining
the
final feature
embedding
by aggregating
the
embeddings
of its bins
with
e t as weights, plus
bias
v 0 .
Formally: f i ( x ) = v 0 + (cid:80) Tt =1 e t · v t =
Linear ( PLE ( x )) .
In the following two sections, we describe two simple algorithms for building bins suitable for PLE .
Namely, we rely on the classic binning algorithms [10] and one of the two algorithms is unsupervised,while another one utilizes labels for constructing bins.
3.2.1 Obtaining bins from quantiles
A natural baseline way to construct the bins for PLE is by splitting value ranges according to theuniformly chosen empirical quantiles of the corresponding individual feature distributions.
Formally,for the i -th feature: b t = Q tT (cid:16) { x j ( num ) i } j ∈ J train (cid:17) , where Q is the empirical quantile function.
Trivialbins of zero size are removed.
In subsection D.2, we demonstrate the usefulness of the proposedscheme on the synthetic GBDT-friendly dataset described in section 5.1 in Gorishniy et al.
3.2.2 Building target-aware bins
In fact, there are also supervised approaches that employ training labels for constructing bins [10].
Intuitively, such target-aware algorithms aim to produce bins that correspond to relatively narrowranges of possible target values.
The supervised approach used in our work is identical in its spirit tothe "C4.5 Discretization" algorithm from Kohavi and Sahami [22].
In a nutshell, for each feature, werecursively split its value range in a greedy manner using target as guidance, which is equivalent tobuilding a decision tree (which uses for growing only this one feature and the target) and treating theregions corresponding to its leaves as the bins for PLE (see the illustration in Figure 4).
Additionally,we define b i 0 = min j ∈ J train x ji and b iT = max j ∈ J train x ji .
3.3 Periodic activation functions
Recall that in subsection 3.2 the work by Tancik et al.
[41] was used as a starting point of ourmotivation for developing PLE .
Thus, we also try to adapt the original work itself for tabular dataproblems.
Our variation differs in two aspects.
First, we take into account the fact the embeddingframework described in subsection 3.1 forbids mixing features during the embedding process (seesubsection D.3 for additional discussion).
Second, we train the pre-activation coefficients insteadof keeping them fixed.
As a result, our approach is rather close to Li et al.
[24] with the number of“groups” equal to the number of numerical features.
We formalize the described scheme in Equation 2,
[Equation]
186 where c i are trainable parameters initialized from N (0 , σ ) .
We observe that σ is an importanthyperparameter.
Both σ and k are tuned using validation sets.
3.4 Simple differentiable layers
In the context of Deep Learning, embedding numerical features with conventional differentiablelayers (e.g. linear layers, ReLU activation, etc.) is a natural approach.
In fact, this technique isalready used on its own in the recently proposed attention-based architectures [12, 23, 39] and insome models for CTR prediction problems [13, 40].
However, we also note that such conventionalmodules can be used on top of the components described in subsection 3.2 and subsection 3.3.
Insection 4, we find that such combinations often lead to better results.
Experiments
In this section, we empirically evaluate the techniques discussed in section 3 and compare them with
Gradient Boosted Decision Trees to check the status quo of the “DL vs GBDT” competition.
4.1 Datasets
[Table]
We use twelve public datasets mostly from the previous works on tabular DL and Kaggle competitions.
Importantly, we focus on the middle and large scale tasks, and our benchmark is biased towards
GBDT-friendly problems, since, as of now, closing the gap with GBDT models on such tasks is oneof the main challenges for tabular DL.
The main dataset properties are summarized in Table 1 and theused sources and additional details are provided in Appendix C.
4.2 Implementation details
We mostly follow Gorishniy et al.
[12] in terms of the hyperparameter tuning, training and evaluationprotocols.
Nevertheless, for completeness, we list all the details in Appendix E.
In the next paragraph,we describe the implementation details specific to embeddings for numerical features.
Embeddings for numerical features.
If linear layers are used, we tune their output dimensions.
The
PLE hyperparameters are the same for all features.
For quantile-based PLE , we tune the number ofquantiles.
For target-aware PLE , we tune the following parameters for decision trees: the maximumnumber of leaves, the minimum number of items per leaf, and the minimum information gain requiredfor making a split when growing the tree.
For the Periodic module (see Equation 2), we tune σ andk (these hyperparameters are the same for all features).
4.3 Model names
In the experiments, we consider different combinations of backbones and embeddings.
For convenience, we use the “Backbone-Embedding” pattern to name the models, where “Backbone” denotes the backbone (e
[Table]
g. MLP, ResNet, Transformer) and “Embedding” denotes the embedding type.
See Table 2 for all considered embedding modules.
Note that:
• Periodic is defined in Equation 2.
• PLE q denotes the quantile-based PLE .
PLE t denotes the target-aware PLE .
• Linear − denotes bias-free linear layer.
LReLU denotes leaky ReLU.
AutoDis was proposed in Guo et al.
[13] • “Transformer-L” is equivalent to FTTransformer [12].
[Table]
4.4 Simple differentiable embedding modules
[Table]
We start by evaluating embedding modules consisting of “conventional” differentiable layers (linearlayers, ReLU activations, etc.).
The results are summarized in Table 3.
The main takeaways :
• first and foremost, the results indicate that MLP can benefit from embedding modules.
Thus, we
conclude that this backbone is worth attention when it comes to evaluating embedding modules.
• the simple LR module leads to modest, but consistent improvements when applied to MLP.
Interestingly, the “redundant” MLP-L configuration also tends to outperform the vanilla MLP.
Although the improvements are not dramatic, the special property of this architecture is that the linearembedding module can be fused together with the first linear layer of MLP after training, whichcompletely removes the overhead.
As for LRLR and AutoDis, we observe that these heavy modulesdo not justify the extra costs (see the results in Appendix F).
4.5 Piecewise linear encoding
In this section, we evaluate the encoding scheme described in subsection 3.2.
The results aresummarized in Table 4.
The main takeaways :
• The piecewise linear encoding is often beneficial for both types of architectures (MLP andTransformer) and the profit can be significant (for example, see the CA and AD datasets).
• Adding differentiable components on top of the PLE can improve the performance.
Though, the most expensive modifications such as Q-LRLR and T-LRLR
are not worth it (see Appendix F).
Note that the benchmark is biased towards GBDT-friendly problems, so the typical superiority oftree-based bins over quantile-based bins, which can be observed in Table 4, may not generalize tomore DL-friendly datasets.
Thus, we do not make any general claims about the relative advantages ofthe two schemes here.
In practice, we recommend considering the type of embedding scheme as anadditional hyperparameter that can be tuned on the validation set.
[Table]
4.6 Periodic activation functions
[Table]
In this section, we evaluate embedding modules based on periodic activation functions as describedin subsection 3.3.
The results are reported in Table 5.
The main takeaway : on average, MLP-P is superior to the vanilla MLP.
However, adding adifferentiable component on top of the Periodic module should be the default strategy (which isin line with Li et al.
Indeed, MLP-PLR and MLP-PL provide meaningful improvements over
MLP-P (e.g. see GE, EY, CA, HO) and even “fix” MLP-P where it is inferior to MLP (OT, FB).
Although MLP-PLR is usually superior to MLP-PL, we note that in the latter case the last linear layerof the embedding module is “redundant” in terms of expressivity and can be fused with the first linearlayer of the backbone after training, which, in theory, can lead to a more lightweight model.
Finally,we observe that MLP-PLRLR and MLP-PLR do not differ significantly enough to justify the extracost of the PLRLR module (see Appendix F).
4.7 Comparing DL models and GBDT
In this section, we perform a big comparison of different approaches to identify the best embeddingmodules and backbones, as well as to check if embeddings for numerical features allow DL models tocompete with GBDT on more tasks than before.
Importantly, we compare ensembles of DL modelsagainst ensembles of GBDT, since Gradient Boosting is essentially an ensembling technique, so suchcomparison will be fairer.
Note that we focus only on the best metric values without taking efficiencyinto account, so we only check if DL models are conceptually ready to compete with GBDT.
We consider three backbones: MLP, ResNet, and Transformer, since they are reported to be representative of what baseline DL backbones are currently capable of [12, 17, 23, 39].
Note that we donot include the attention-based models that also apply attention on the level of objects [23, 34, 39],since this non-parametric component is orthogonal to the central topic of our work.
The results aresummarized in Table 6.
[Table]
The main takeaways for DL models :
• For most datasets, embeddings for numerical features can provide noticeable improvements forthree different backbones.
Although the average rank is not a good metric for making subtleconclusions, we can’t help noting the epic difference in average ranks between the MLP andMLP-PLR models.•
On the considered datasets, the P-LR module provides the best average performance. However,given the limited size of the benchmark, we cannot conclude that P-LR is the only solutionthat is worth attention.
Thus, we also highlight the PLE -based embeddings, which outperformembeddings-free models in many cases and additionally are more hardware-friendly, since theydo not rely on computationally expensive periodic functions.• We highlight LR as a simple and stable embedding module for the MLP and ResNet backbones:although the performance gains are not dramatic, its main advantage is consistency, which makesMLP-LR and ResNet-LR attractive as simple DL solutions.• Importantly, after the MLP-like architectures are coupled with embeddings for numericalfeatures, they perform on par with the Transformer-based models.
The main takeaway for the “DL vs GBDT” competition : embeddings for numerical features is asignificant design aspect that has a great potential for improving DL models and closing the gap with
GBDT on GBDT-friendly tasks.
Let us illustrate this claim with several observations:
• The benchmark is initially biased to GBDT-friendly problems, which can be observed by
comparing GBDT solutions with the vanilla DL models (MLP, ResNet, Transformer-L).
• However, after coupling DL models with embeddings for numerical features, the situationchanges and the MI dataset remains the only one where all DL models lag behind GBDT.
• Additionally, to the best of our knowledge, it is the first time when DL models perform on par with GBDT on the well-known California Housing and Adult datasets.
That said, compared to GBDT models, efficiency can still be an issue for the considered DL
architec-
tures.
In any case, the trade-off completely depends on the specific use case and requirements
Analysis
5.1 Comparing model sizes
To quantify the effect of embeddings for numerical features on model sizes
[Table]
we report the parametercounts in Table 7.
Overall, introducing embeddings for numerical features can cause non-negligibleoverhead in terms of model size.
Importantly, the overhead in terms of size does not translate to thesame overhead in terms of training times and throughput.
For example, the almost 2000 -fold increasein the parameter count for MLP-LR on the CH dataset results in only 1 . 5 -fold increase in trainingtimes.
Finally, in practice, we observe that coupling MLP and ResNet with embedding modules leadsto architectures that are still faster than Transformer-based models.
[Table]
5.2 Ablation study
[Table]
In this section [Table] we compare two alternative binning-based encoding schemes with PLE (see subsection 3.2).
The first one ("binary") assigns the value 1 instead of the ratio in the representationcomponent corresponding to the bin to which the feature value belongs.
The second alternative is ageneralized version of the one-blob encoding [28] (see subsection E.1 for details).
We follow thetuning and evaluation protocols described in subsection 4.2 and report results in Table 8.
The tableindicates that making the binning-based encoding piecewise linear is a good default strategy.
Conclusion & Future work
In this work [Table] we have demonstrated that embeddings for numerical features are an important designaspect of tabular DL architectures.
Namely, it allows existing DL backbones to achieve noticeablybetter results and significantly reduce the gap with Gradient Boosted Decision Trees.
We havedescribed two approaches illustrating this phenomenon, one using the piecewise linear encoding oforiginal scalar values, and another using periodic functions.
We have also shown that traditional
MLP-like models coupled with embeddings can perform on par with attention-based models.
Nevertheless, we have only scratched the surface of the new direction.
For example, it is still tobe explained how exactly the discussed embedding modules help optimization on the fundamentallevel.
Additionally, we have considered only schemes where the same functional transformation wasapplied to all features, which may be a suboptimal choice.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the analysis in subsection 5.1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The work
focuses on a generic aspect of deep learning models.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] We do not include theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] The supplementary material includes the script used to createdata splits.
The hyperparameters are either explicitly described in subsection 4.2 and supplementary material, or tuned as described in subsection 4.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We provide standard deviations in the supplementary material, see Table 17 and see Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The experiment reports included in the supplementary material provide the information about the used hardware and execution times.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix C.
(b) Did you mention the license of the assets?
[Yes]
In the README.md file in the supple-
mentary material, we refer to the original licenses of the used datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]We do not provide new datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We use publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] We use publicly available datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
We did not use crowdsourcing.
We did not conduct research with
[Table]
human subjects.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
We did not use crowdsourcing.
We did
[Table]
not conduct research with human subjects.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
We did not use crowdsourcing.
We did not
[Table]
conduct research with human subjects.
[Table]
Supplementary material
A MLP with embeddings for numerical features
We provide visual explanation of how embeddings are passed to MLP in Figure 2 and Figure 3.
Also,we provide the formal explanation in Equation 3 (categorical features are omitted for simplicity).
[Equation]
B Target-aware piecewise linear encoding
We provide visualisation of target-aware PLE (subsubsection 3.2.2) in Figure 4.
[Figure]
C Additional details on datasets
[Table]
We used the following datasets:
• Gesture Phase Prediction (Madeo et al.
[26])• Churn Modeling 1• Eye Movements (Salojarvi et al.
[35])• California Housing (real estate data, Kelley Pace and Barry [19])• House 16H
2• Adult (income estimation, Kohavi [21])• Otto Group Product Classification 3• Higgs (simulated physical particles, Baldi et al.
[4]; we use the version with 98K samplesavailable in the OpenML repository [43])• Santander Customer Transaction Prediction 4• Facebook Comments (Singh et al.
[37])• Covertype (forest characteristics, Blackard and Dean.
• Microsoft (search queries, Qin and Liu [32]).
We follow the pointwise approach to learning-to-
[Table]
rank and treat this ranking problem as a regression problem
D Additional analysis
D.1 Is piecewise linear encoding sensitive to data preprocessing?
[Table]
It is known that data preprocessing, such as standardization or quantile transformation, is oftencrucial for DL models for achieving competitive performance.
Moreover, the performance cansignificantly vary between different types of preprocessing.
At the same time, PLE -representationscontain only values from [0 , 1] and they are invariant to shifting and scaling.
These observationsmotivate us to check if models using PLE are sensitive to preprocessing to the same extent as vanillamodels.
For datasets where the quantile transformation was used in section 4, we reevaluate thetuned configurations of MLP, MLP-Q, and MLP-T with different preprocessing policies and reportthe results in Table 10 (note that standardization is equivalent to no preprocessing for models with
PLE ).
First, the vanilla MLP often becomes unusable without preprocessing.
Second, for the vanilla
MLP, it can be important to choose one specific type of preprocessing (CA, HO, FB, MI), which isless pronounced for MLP-Q and not the case for MLP-T (though, this specific observation can bethe property of the benchmarks, not of MLP-T).
Overall, the results indicate that models using PLEare less sensitive to data preprocessing compared to the vanilla MLP.
This is an additional benefit of
PLE -representations for practitioners since the aspect of preprocessing becomes less critical with PLE .
D.2 Testing quantile-based PLE on the synthetic GBDT-friendly dataset
[Figure]
that in practice, increasing the number of bins does not always lead to better results.
In this section, we apply the quantile-based piecewise linear encoding (described in subsubsection 3.2.1 to MLP and Transformer on the synthetic GBDT-friendly dataset described in section 5.1in Gorishniy et al.
In a nutshell, features of this dataset are sampled randomly from N (0 , 1) ,and the target is produced by an ensemble of randomly constructed decision trees applied to thesampled features.
This task turns out to be easy for GBDT, but hard for traditional DL models [12].
The technical details are provided in subsection D.2.
The results are visualized in Figure 5.
As theplot shows, PLE -representations can be helpful for both MLP and Transformer backbones.
In theconsidered synthetic setup, increasing the number of bins leads to better results, however, in practice,using too many bins can lead to overfitting; therefore, we recommend tuning the number of binsbased on a validation set.
Technical details.
Our dataset has 10 , 000 objects, 8 features and the target was produced by 16decision trees of depth 6 .
CatBoost is trained with the default hyperparameters.
Transformer istrained with the default hyperparameters of FT-Transformer.
The MLP backbone has four layers ofsize 256 each.
Importantly, the task GBDT-friendly, which can be illustrated by the performance ofthe tuned MLP: 0 . 2229 ± 0 .
0055 (it is still worse than the performance of CatBoost).
The remainingdetails can be found in the source code.
D.3 Fourier features
In this section, we test Fourier features implemented exactly as in Tancik et al.
[41], i.e. pre-activationcoefficients are not trained and features are mixed right from the start.
Importantly, the latter meansthat this approach is not covered by the embedding framework described in subsection 3.1.
Asreported in Table 11, MLP equipped with the original Fourier features does not perform well evencompared to the vanilla MLP.
So, it seems to be important to embed each feature separately asdescribed in subsection 3.1.
[Table]
E Implementation details
[Table]
We mostly follow Gorishniy et al.
[12] in terms of the tuning, training and evaluation protocols.
Data preprocessing .
Preliminary data preprocessing is known to be crucial for the optimization oftabular DL models.
For each dataset, the same preprocessing was used for all deep models for a faircomparison.
By default, we use the quantile transformation from the Scikit-learn library [29].
Weapply standardization (mean subtraction and scaling) to Eye Movements and no preprocessing to
Otto Group Product Classification since these preprocessing strategies lead to much better results formost DL models compared to the quantile-based one.
We also apply standardization to regressiontargets for all algorithms.
Tuning .
For every dataset, we carefully tune each model’s hyperparameters.
The best hyperparametersare the ones that perform best on the validation set, so the test set is never used for tuning.
For mostalgorithms, we use the Optuna library [1] to run Bayesian optimization (the Tree-Structured Parzen
Estimator algorithm), which is reported to be superior to random search [42].
The search spaces forall hyperparameters are reported in the appendix.
Evaluation .
For each tuned configuration, we run 15 experiments with different random seeds andreport the average performance on the test set.
Ensembles .
For each model-dataset pair, we obtain three ensembles by splitting the 15 single modelsinto three disjoint groups of equal size and averaging predictions of single models within each group.
Neural networks .
The implementations of the MLP, ResNet, and Transformer backbones are takenfrom Gorishniy et al.
We minimize cross-entropy for classification problems and mean squarederror for regression problems.
We use the AdamW optimizer [25].
We do not apply learning rateschedules.
For each dataset, we use a predefined batch size (see Appendix C for the specific values).
We continue training until there are patience + 1 consecutive epochs without improvements on thevalidation set; we set patience = 16 for all models.
Categorical features.
For CatBoost, we employ the built-in support for categorical features.
For allother algorithms, we use the one-hot encoding.
E.1 One-blob encoding
In subsection 5.2, we used a slightly generalized version of the original one-blob encoding [28].
Namely, while the original sets the width of the kernel to T − 1 ( T is the number of bins), we set it to
T − γ and tune γ .
E.2 Hyperparameter tuning configurationsE.3 CatBoost
We fix and do not tune the following hyperparameters:
• early-stopping-rounds =
• od-pval = 0 .
[Table]
• iterations =
For tuning on the MI and CO datasets
[Table]
we set the task_type parameter to “GPU”.
In all other cases(including the evaluation on these two datasets), we set this parameter to “CPU”.
[Table]
E.4 XGBoost
We fix and do not tune the following hyperparameters:
• booster = "gbtree"
• early-stopping-rounds =
• n-estimators =
[Table] [Table]
ResNet
[Table]
[Table]
PLE .
We share the same hyperparameter space for PLE across all datasets and models.
For thequantile-based PLE , the distribution for the number of quantiles is UniformInt[2 , 256] .
For thetarget-aware (tree-based) PLE , the distribution for the number of leaves is UniformInt[2 , 256] , thedistribution for the minimum number of items per leaf is UniformInt[1 , 128] and the distribution forthe minimum information gain required for making a split is LogUniform[1 e - 9 , 0 . 01] .
Periodic .
The distribution for k (see Equation 2) is UniformInt[1 , 128] .
F Extended tables with experimental results
The scores with standard deviations for single models and ensembles are provided in Table 17 and
Table 18 respectively.
[Table] [Table]