On Embeddings for Numerical Features in Tabular Deep Learning
Abstract
Recently, Transformer-like deep architectures have shown strong performance on tabular data problems.
Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone.
In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with gradient boosted decision trees (GBDT) on some GBDT-friendly benchmarks (that is, where GBDT outperforms conventional DL models).
We start by describing two conceptually different approaches to building embedding modules: the ﬁrst one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations.
Then, we empirically demonstrate that these two approaches can lead to signiﬁcant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations.
Importantly, we also show that embedding numerical features is beneﬁcial for many backbones, not only for Transformers.
Speciﬁcally, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures.
Overall, we highlight embeddings for numerical features as an important design aspect with good potential for further improvements in tabular DL.
The source code is available at https://github.com/Yura52/tabular-dl-num-embeddings .
Introduction
Tabular data problems are currently a ﬁnal frontier for deep learning (DL) research.
While the most recent breakthroughs in NLP, vision, and speech are achieved by deep models [12], their success in the tabular domain is not convincing yet.
Despite a large number of proposed architectures for tabular DL [2, 3, 13, 17, 21, 24, 31, 39, 40], the performance gap between them and the “shallow” ensembles of decision trees, like GBDT, often remains signiﬁcant [13, 36].
The recent line of works [13, 24, 39] reduce this performance gap by successfully adapting the
Transformer architecture [45] for the tabular domain.
Compared to traditional models, like MLP or ResNet, the proposed Transformer-like architectures have a speciﬁc way to handle numerical features of the data.
Namely, they map scalar values of numerical features to high-dimensional embedding vectors, which are then mixed by the self-attention modules.
Beyond transformers, mapping numerical features to vectors was also employed in different forms in the click-through rate (CTR) prediction problems [8, 14, 40].
Nevertheless, the literature is mostly focused on developing more powerful backbones while keeping the design of embedding modules relatively simple.
In particular, the existing architectures [13, 14, 24, 39, 40] construct embeddings for numerical features using quite restrictive parametric mappings, e.g., linear functions, which can lead to suboptimal performance.
In this work, we demonstrate that the embedding step has a substantial impact on the model effectiveness, and its proper design can signiﬁcantly improve tabular DL models.
Speciﬁcally, we describe two different building blocks suitable for constructing embeddings for numerical features.
The ﬁrst one is a piecewise linear encoding that produces alternative initial representations for the original scalar values and is based on feature binning, a long-existing preprocessing technique [11].
The second one relies on periodic activation functions, which is inspired by their usage in implicit neural representations [28, 38, 42], NLP [41, 45] and CV tasks [25].
The ﬁrst approach is simple, interpretable and non-differentiable, while the second demonstrates better results on average.
We observe that DL models equipped with our embedding schemes successfully compete with GBDT on GBDT-friendly benchmarks and achieve the new state-of-the-art on tabular DL.
As another important ﬁnding, we demonstrate that the step of embedding the numerical features is universally beneﬁcial for different deep architectures, not only for Transformer-like ones.
In particular, we show, that after proper embeddings, simple MLP-like architectures often provide the performance comparable to the state-of-the-art attention-based models.
Overall, our work demonstrates the large impact of the embeddings of numerical features on the tabular DL performance and shows the potential of investigating more advanced embedding schemes in future research.
To sum up, our contributions are as follows:
We demonstrate that embedding schemes for numerical features are an underexplored research question in tabular DL.
Namely, we show that more expressive embedding schemes can provide substantial performance improvements over prior models.
We show that the proﬁt from embedding numerical features is not speciﬁc for Transformerlike architectures, and proper embedding schemes beneﬁt traditional models as well.
On a number of public benchmarks, we achieve the new state-of-the-art on tabular DL.
Related work
Tabular deep learning.
During several recent years, the community has proposed a large number of deep models for tabular data [2, 3, 13, 15, 17, 21, 24, 31, 39, 40, 46].
However, when systematically evaluated, these models do not consistently outperform the ensembles of decision trees, such as GBDT (Gradient Boosting Decision Tree) [7, 19, 32], which are typically the top-choice in various ML competitions [13, 36].
Moreover, several recent works have shown that the proposed sophisticated architectures are not superior to properly tuned simple models, like MLP and ResNet [13, 18].
In this work, unlike the prior literature, we do not aim to propose a new backbone architecture.
Instead, we focus on more accurate ways to handle numerical features, and our developments can be potentially combined with any model, including traditional MLPs and more recent Transformer-like ones.
Transformers in tabular DL.
Due to the tremendous success of Transformers for different domains [10, 45], several recent works adapt their self-attention design for tabular DL as well [13, 17, 24, 39].
Compared to existing alternatives, applying self-attention modules to the numerical features of tabular data requires mapping the scalar values of these features to high-dimensional embedding vectors.
So far, the existing architectures perform this “scalar” ! “vector” mapping by relatively simple computational blocks, which, in practice, can limit the model expressiveness.
For instance, the recent FT-Transformer architecture [13] employs only a single linear layer.
In our experiments, we demonstrate that such embedding schemes can provide suboptimal performance, and more advanced schemes often lead to substantial proﬁt.
CTR Prediction.
In CTR prediction problems, objects are represented by numerical and categorical features, which makes this ﬁeld highly relevant to tabular data problems.
In several works, numerical features are handled in some non-trivial way while not being the central part of the research [8, 40].
Recently, however, a more advanced scheme has been proposed in Guo et al.
Nevertheless, it is still based on linear layers and conventional activation functions, which we found to be suboptimal in our evaluation.
Feature binning.
Binning is a discretization technique that converts numerical features to categorical features.
Namely, for a given feature, its value range is split into bins (intervals), after which the original feature values are replaced with discrete descriptors (e.g. bin indices or one-hot vectors) of the corresponding bins.
We point to the work by Dougherty et al.
[11], which performs an overview of some classic approaches to binning and can serve as an entry point to the relevant literature on the topic.
In our work, however, we utilize bins in a different way.
Speciﬁcally, we use their edges to construct lossless piecewise linear representations of the original scalar values.
It turns out that this simple and interpretable representations can provide substantial beneﬁt to deep models on several tabular problems.
Periodic activations.
Recently, periodic activation functions have become a key component in processing coordinates-like inputs, which is required in many applications.
Examples include
NLP [45], CV [25], implicit neural representations [28, 38, 42].
In our work, we show that periodic activations can be used to construct powerful embedding modules for numerical features in tabular data problems.
Contrary to some of the aforementioned papers, where components of the multidimensional coordinates are mixed (e.g. with linear layers) before passing them to periodic functions [38, 42], we ﬁnd it crucial to embed each feature separately before mixing them in the main backbone.
Embeddings for numerical features
In this section, we describe the general framework for what we call "embeddings for numerical features" and the main building blocks used in the experimental comparison in section 4.
Notation.
For a given supervised learning problem on tabular data, we denote the dataset as  x j , y j
[Equation] j =1 where y j 2 Y represents the object’s label and x j =
[Equation]
sents the object’s features (numerical and categorical).
x j ( num ) i , in turn, denotes the i -th numerical feature of the j -th object.
Depending on the context, the j index can be omitted.
The dataset is split into three disjoint parts: 1 , n = J train [ J val [ J test , where the “train” part is used for training, the “validation” part is used for early stopping and hyperparameter tuning, and the “test” part is used for the ﬁnal evaluation.
3.1 General framework
We formalize the notion of "embeddings for numerical features" as z i = f i (( x ( num ) i ) 2 R d i , where f i ( x ) is the embedding function for the i -th numerical feature, z i is the embedding of the i -th numerical feature and d i is the dimensionality of the embedding.
Importantly, the proposed framework implies that embeddings for all features are computed independently of each other.
Note that the function f i can depend on parameters that are trained as a part of the whole model or in some other fashion (e.g. before the main optimization).
In this work, we consider only embedding schemes where the embedding functions for all features are of the same functional form.
We never share parameters of embedding functions of different features.
The subsequent use of the embeddings depends on the model backbone.
For MLP-like architectures, they are concatenated into one ﬂat vector (see Appendix A for illustrations).
For Transformer-based architectures, no extra step is performed and the embeddings are passed as is, so the usage is deﬁned by the original architectures.
3.2 Piecewise linear encoding
While vanilla MLP is known to be a universal approximator [9, 16], in practice, due to optimization peculiarities, it has limitations in its learning capabilities [34].
However, the recent work by Tancik et al.
[42] uncovers the case where changing the input space alleviates the above issue.
This observation motivates us to check if changing the representations of the original scalar values of numerical features can improve the learning capabilities of tabular DL models.
At this point, we try to start simple and turn to "classical" machine learning techniques.
Namely, we take inspiration from the one-hot encoding algorithm that is widely and successfully used for representing discrete entities such as categorical features in tabular data problems or tokens in NLP.
We note that the one-hot representation can be seen as an opposite solution to the scalar representation in terms of the trade-off between parameter efﬁciency and expressivity.
To check whether the onehot-like approach can be beneﬁcial for tabular DL models, we design a continuous alternative to the one-hot encoding (since the vanilla one-hot encoding is barely applicable to numerical features).
Formally, for the i -th numerical feature, we split its value range into the disjoint set of T i intervals B i 1 , . . .
, B iT , which we call bins : B it =
[ b it  1 , b it ) .
The splitting algorithm is an important imple- mentation detail that we discuss later.
From now on, we omit the feature index i for simplicity.
Once the bins are determined, we deﬁne the encoding scheme as in Equation 1:
PLE ( x ) =
[ e 1 , . .
, e T ] 2 R T e t =>< >: 0 , x < b t  1 AND t >1 , x  b t AND t < T x  b t  1 b t  b t  1 , otherwise (1) where PLE stands for “ p eicewise l inear e ncoding”.
We provide the visualization in
Figure 1.
Note that:
• PLE produces alternative initial representations for the numerical features and can be viewed as a preprocessing strategy.
These representations are computed once and then used instead of the original scalar values during the main optimization.
• For T = 1 , the PLE -representation is effectively equivalent to the scalar representation.
• Contrary to categorical features, numerical features are ordered; we express that by setting tothe components corresponding to bins with the right boundaries lower than the given feature value (this approach resembles how labels are encoded in ordinal regression problems).
• The cases ( x < b 0 ) and ( x  b T ) are also covered by Equation 1 (which leads to ( e 1  0) and ( e T  1) respectively).
• The choice to make the representation piecewise linear is itself a subject for discussion.
We analyze some alternatives in subsection 5.2.
• PLE can be viewed as feature preprocessing, which is additionally discussed in subsection 5.3.
A note on attention-based models.
While the described PLE -representations can be passed to MLP- like models as is, attention-based models are inherently invariant to the order of input embeddings, so one additional step is required to add the information about feature indices to the obtained encodings.
Technically, we observe that it is enough to place one linear layer after PLE (without sharing weights between features).
Conceptually, however, this solution has a clear semantic interpretation.
Namely, it is equivalent to allocating one trainable embedding v t 2 R d for each bin B t and obtaining the ﬁnal feature embedding by aggregating the embeddings of its bins with e t as weights, plus bias v 0 .
Formally: f i ( x [Equation] = [Equation]
In the following two sections, we describe two simple algorithms for building bins suitable for PLE .
Namely, we rely on the classic binning algorithms [11] and one of the two algorithms is unsupervised, while another one utilizes labels for constructing bins.
3.2.1 Obtaining bins from quantiles
A natural baseline way to construct the bins for PLE is by splitting value ranges according to the uniformly chosen empirical quantiles of the corresponding individual feature distributions.
Formally, for the i -th feature: b t = Q tT ⇣ {
[Equation] num ) i } [Equation] 2 J train ⌘ , where Q is the empirical quantile function.
Trivial bins of zero size are removed.
In subsection D.1, we demonstrate the usefulness of the proposed scheme on the synthetic GBDT-friendly dataset described in section 5.1 in Gorishniy et al.
3.2.2 Building target-aware bins
In fact, there are also supervised approaches that employ training labels for constructing bins [11].
Intuitively, such target-aware algorithms aim to produce bins that correspond to relatively narrow ranges of possible target values.
The supervised approach used in our work is identical in its spirit to the "C4.5 Discretization" algorithm from Kohavi and Sahami [23].
In a nutshell, for each feature, we recursively split its value range in a greedy manner using target as guidance, which is equivalent to building a decision tree (which uses for growing only this one feature and the target) and treating the regions corresponding to its leaves as the bins for PLE (see the illustration in Figure 4).
Additionally, we deﬁne b i 0 = min j 2 J train x ji and b iT = max j 2 J train x ji .
3.3 Periodic activation functions
Recall that in subsection 3.2 the work by Tancik et al.
[42] was used as a starting point of our motivation for developing PLE .
Thus, we also try to adapt the original work itself for tabular data problems.
Our variation differs in two aspects.
First, we take into account the fact the embedding framework described in subsection 3.1 forbids mixing features during the embedding process (see subsection D.2 for additional discussion).
Second, we train the pre-activation coefﬁcients instead of keeping them ﬁxed.
As a result, our approach is rather close to Li et al.
[25] with the number of “groups” equal to the number of numerical features.
We formalize the described scheme in Equation 2,
[Equation]
i
[Equation]
where c i are trainable parameters initialized from N (0 ,  ) .
We observe that  is an important hyperparameter.
Both  and k are tuned using validation sets.
3.4 Simple differentiable layers
In the context of Deep Learning, embedding numerical features with conventional differentiable layers (e.g. linear layers, ReLU activation, etc.) is a natural approach.
In fact, this technique is already used on its own in the recently proposed attention-based architectures [13, 24, 39] and in some models for CTR prediction problems [14, 40].
However, we also note that such conventional modules can be used on top of the components described in subsection 3.2 and subsection 3.3.
In section 4, we ﬁnd that such combinations often lead to better results.
Experiments
In this section, we empirically evaluate the techniques discussed in section 3 and compare them with
Gradient Boosted Decision Trees to check the status quo of the “DL vs GBDT” competition.
4.1 Datasets
[Table]
We use eleven public datasets mostly from the previous works on tabular DL and Kaggle competitions.
Importantly, we focus on the middle and large scale tasks, and our benchmark is biased towards
GBDT-friendly problems, since, as of now, closing the gap with GBDT models on such tasks is one of the main challenges for tabular DL.
The main dataset properties are summarized in Table 1 and the used sources and additional details are provided in Appendix C.
4.2 Implementation details
We mostly follow Gorishniy et al.
[13] in terms of the hyperparameter tuning, training and evaluation protocols.
Nevertheless, for completeness, we list all the details in Appendix E.
In the next paragraph, we describe the implementation details speciﬁc to embeddings for numerical features.
Embeddings for numerical features.
If linear layers are used, we tune their output dimensions.
The
PLE hyperparameters are the same for all features.
For quantile-based PLE , we tune the number of quantiles.
For target-aware PLE , we tune the following parameters for decision trees: the maximum number of leaves, the minimum number of items per leaf, and the minimum information gain required for making a split when growing the tree.
For the Periodic module (see Equation 2), we tune  and k (these hyperparameters are the same for all features).
4.3 Model names
In the experiments, we consider different combinations of backbones and embeddings.
For convenience, we use the “Backbone-Embedding” pattern to name the models, where “Backbone” denotes the backbone (e.g. MLP, ResNet, Transformer) and “Embedding” denotes the embedding type.
See Table 2 for all considered embedding modules.
Note that:
• Periodic is deﬁned in Equation
• PLE q denotes the quantile-based PLE .
PLE t denotes the target-aware PLE .
• Linear  denotes bias-free linear layer.
LReLU denotes leaky ReLU.
AutoDis was proposed in Guo et al.
[14] • “Transformer-L” is equivalent to FTTransformer [13].
[Table]
4.4 Simple differentiable embedding modules
[Table]
We start by evaluating embedding modules consisting of “conventional” differentiable layers (linear layers, ReLU activations, etc.).
The results are summarized in Table 3.
The main takeaways :
• ﬁrst and foremost, the results indicate that MLP can beneﬁt from embedding modules.
Thus, we conclude that this backbone is worth attention when it comes to evaluating embedding modules.
• the simple LR module leads to modest, but consistent improvements when applied to MLP.
Interestingly,
the “redundant” MLP-L conﬁguration also tends to outperform the vanilla
MLP Although the improvements are not dramatic, the special property of this architecture is that the linear embedding module can be fused together with the ﬁrst linear layer of MLP after training, which completely removes the overhead.
As for LRLR and AutoDis, we observe that these heavy modules do not justify the extra costs (see the results in Appendix F).
4.5 Piecewise linear encoding
In this section, we evaluate the encoding scheme described in subsection 3.2.
The results are summarized in Table 4.
The main takeaways :
• The piecewise linear encoding is often beneﬁcial for both types of architectures (MLP and Transformer) and the proﬁt can be signiﬁcant (for example, see the CA and AD datasets).
• Adding differentiable components on top of the PLE can improve the performance.
Though, the most expensive modiﬁcations such as Q-LRLR and T-LRLR are not worth it (see Appendix F).
Note that the benchmark is biased towards GBDT-friendly problems, so the typical superiority of tree-based bins over quantile-based bins, which can be observed in Table 4, may not generalize to more DL-friendly datasets.
Thus, we do not make any general claims about the relative advantages of the two schemes here.
Notation follows Table 3 and Table 2.
The best are deﬁned separately
[Table]
4.6 Periodic activation functions
[Table]
In this section
[Table]
we evaluate embedding modules based on periodic activation functions as described in subsection 3.3.
The results are reported in Table 5.
The main takeaway : on average, MLP-P is superior to the vanilla MLP.
However, adding a differentiable component on top of the Periodic module should be the default strategy (which is in line with Li et al.
Indeed, MLP-PLR and MLP-PL provide meaningful improvements over
MLP-P (e.g. see GE, CA, HO) and even “ﬁx” MLP-P where it is inferior to MLP (OT, FB).
Although MLP-PLR is usually superior to MLP-PL, we note that in the latter case the last linear layer of the embedding module is “redundant” in terms of expressivity and can be fused with the ﬁrst linear layer of the backbone after training, which, in theory, can lead to a more lightweight model.
Finally, we observe that MLP-PLRLR and MLP-PLR do not differ signiﬁcantly enough to justify the extra cost of the PLRLR module (see Appendix F).
4.7 Comparing DL models and GBDT
In this section, we perform a big comparison of different approaches to identify the best embedding modules and backbones, as well as to check if embeddings for numerical features allow DL models to compete with GBDT on more tasks than before.
Importantly, we compare ensembles of DL models against ensembles of GBDT, since Gradient Boosting is essentially an ensembling technique, so such comparison will be fairer.
Note that we focus only on the best metric values without taking efﬁciency into account, so we only check if DL models are conceptually ready to compete with GBDT.
We consider three backbones: MLP, ResNet [Table] and Transformer, since they are reported to be representative of what baseline DL backbones are currently capable of [13, 18, 24, 39].
Note that we do not include the attention-based models that also apply attention on the level of objects [24, 35, 39], since this non-parametric component is orthogonal to the central topic of our work.
The results are summarized in Table 6.
[Table]
The main takeaways for DL models :
• For most datasets, embeddings for numerical features can provide noticeable improvements for three different backbones.
Although the average rank is not a good metric for making subtle conclusions, we highlight the impressive difference in average ranks between the MLP and MLP-PLR models.
• The simplest LR embedding is a good baseline solution: although the performance gains are not dramatic, its main advantage is consistency (e.g. see MLP vs MLP-LR).
• The PLR module provides the best average performance.
Empirically, we observe  (see Equation 2) to be an important hyperparameter that should be tuned.
• Piecewise linear encoding ( PLE ) allows building well performing embeddings (e.g. T-LR, Q-LR). In
addition to that, PLE itself is worth attention because of its simplicity, interpretability and efﬁciency (no computationally expensive periodic functions).
• Importantly, after the MLP-like architectures are coupled with embeddings for numerical features, they perform on par with the Transformer-based models.
The main takeaway for the “DL vs GBDT” competition : embeddings for numerical features is a signiﬁcant design aspect that has a great potential for improving DL models and closing the gap with
GBDT on GBDT-friendly tasks.
Let us illustrate this claim with several observations:
• The benchmark is initially biased to GBDT-friendly problems, which can be observed by comparing GBDT solutions with the vanilla DL models (MLP, ResNet, Transformer-L).
• However, for the vast majority of the “backbone & dataset” pairs, proper embeddings are the only thing needed to close the gap with GBDT.
Exceptions (rather formal) include the MI dataset and the following pairs: “ResNet & GE”, “Transformer & FB”, “Transformer & GE”, “Transformer & OT”.
• Additionally, to the best of our knowledge, it is the ﬁrst time when DL models perform on par with GBDT on the well-known California Housing and Adult datasets.
That said, compared to GBDT models, efﬁciency can still be an issue for the considered DL architectures.
In any case, the trade-off completely depends on the speciﬁc use case and requirements.
Analysis 5.1 Comparing model sizes
To quantify the effect of embeddings for numerical features on model sizes
[Table]
we report the parameter counts in Table 7.
Overall, introducing embeddings for numerical features can cause non-negligible overhead in terms of model size.
Importantly, the overhead in terms of size does not translate to the same overhead in terms of training times and throughput.
For example, the almost 2000 -fold increase in the parameter count for MLP-LR on the CH dataset results in only 1 . 5 -fold increase in training times.
Finally, in practice
[Table]
we observe that coupling MLP and ResNet with embedding modules leads to architectures that are still faster than Transformer-based models.
[Table]
5.2 Ablation study
linear encoding ( PLE ) with the two variations described in subsection 5.2.
Notation follows Table 3 and Table 2.
[Table]
In this section
[Table]
we compare two alternative binning-based encoding schemes with PLE (see subsection 3.2).
The ﬁrst one ("thermometer" [6]) sets the value 1 instead of the piecewise linear term (see
Equation 1).
The second one is a generalized version of the one-blob encoding [29] (see subsection E.1 for details).
The tuning and evaluation protocols are the same as in subsection 4.2.
The results in table Table 8 indicate that making the binning-based encoding piecewise linear is a good default strategy.
5.3 Piecewise linear encoding as a feature preprocessing technique
It is known that data preprocessing, such as standardization or quantile transformation, is often crucial for DL models for achieving competitive performance.
Moreover, the performance can signiﬁcantly vary between different types of preprocessing.
At the same time, PLE -representations contain only values from [0 , 1] and they are invariant to shifting and scaling, which makes PLE itself a general feature preprocessing technique potentially suitable for DL models without the need to use traditional preprocessing ﬁrst.
To illustrate that, for datasets where the quantile transformation was used in section 4, we reevaluate the tuned conﬁgurations of MLP, MLP-Q, and MLP-T with different preprocessing policies and report the results in Table 9 (note that standardization is equivalent to no preprocessing for models with PLE ).
[Table]
First, the vanilla MLP often becomes unusable without preprocessing.
Second, for the vanilla MLP, it can be important to choose one speciﬁc type of preprocessing (CA, HO, FB, MI), which is less pronounced for MLP-Q and not the case for MLP-T (though, this speciﬁc observation can be the property of the benchmarks, not of MLP-T).
Overall, the results indicate that models using PLE are less sensitive to the initial preprocessing compared to the vanilla MLP.
This is an additional beneﬁt of
PLE -representations for practitioners since the aspect of preprocessing becomes less critical with PLE .
5.4 The “feature engineering” perspective
[Table]
At ﬁrst sight, feature embeddings may resemble feature engineering and should be suitable for all kinds of models.
However, the proposed embedding schemes are motivated by DL-speciﬁc aspects of training (see the motivational parts of subsection 3.2 and subsection 3.3).
While our methods are likely to transfer well to models with similar training properties (e.g. to linear models since those are a special case of deep models), it is not the case in general.
To illustrate that, we try adopting the
Periodic module for XGBoost by ﬁxing the random coefﬁcients from Equation 2.
We also keep the original features instead of dropping them.
The tuning and evaluation protocols are the same as in subsection 4.2.
The results in Table 10 show that this technique, while being useful for DL models, does not provide any beneﬁts for XGBoost.
Conclusion & Future work
In this work, we have demonstrated that embeddings for numerical features are an important design aspect of tabular DL architectures.
Namely, it allows existing DL backbones to achieve noticeably better results and signiﬁcantly reduce the gap with Gradient Boosted Decision Trees.
We have described two approaches illustrating this phenomenon, one using the piecewise linear encoding of original scalar values, and another using periodic functions.
We have also shown that traditional
MLP-like models coupled with embeddings can perform on par with attention-based models.
Nevertheless, we have only scratched the surface of the new direction.
For example, it is still to be explained how exactly the discussed embedding modules help optimization on the fundamental level.
Additionally, we have considered only schemes where the same functional transformation was applied to all features, which may be a suboptimal choice.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the analysis in subsection 5.1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The work focuses on a generic aspect of deep learning models.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] We do not include theoretical results.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The supplementary material includes the script used to create data splits.
The hyperparameters are either explicitly described in subsection 4.2 and supplementary material, or tuned as described in subsection 4.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We provide standard deviations in the supplementary material, see Table 18 and see Table(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The experiment reports included in the supplementary material provide the information about the used hardware and execution times.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix C. (b) Did you mention the license of the assets?
[Yes] In the README.md ﬁle in the supplementary material, we refer to the original licenses of the used datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We do not provide new datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] We use publicly available datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use crowdsourcing.
We did not conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
We did not use crowdsourcing.
We did not conduct research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We did not use crowdsourcing.
We did not conduct research with human subjects.