Embeddings for Numerical Features Using tanh Activation (2025)

Abstract: Recent advances in tabular deep learning have demonstrated the importance of embeddings for numerical features, where scalar values are mapped to high-dimensional spaces before being processed by the main model. Here, we propose an embedding method using the hyperbolic tangent (tanh) activation function that allows neural networks to achieve better accuracy on tabular data via an inductive bias similar to that of decision trees. To make training with the new embedding method reliable and efficient, we additionally propose a principled initialization method. Experiments demonstrate that the new approach improves upon or matches accuracy results from previously proposed embedding methods across multiple tabular datasets and model architectures.

## introduction deep learning has achieved success in various domains, from computer vision to natural language processing. however, its application to tabular data has been challenging, with gradient-boosted decision trees (gbdts) typically outperforming neural networks. this has led researchers to investigate how neural networks can better capture the inductive bias that makes tree-based models effective on tabular data. work by @xcite has demonstrated that proper embedding of numerical features is beneficial for achieving performance competitive with that of gbdts. recent developments have introduced additional approaches to tabular embeddings: @xcite use tree ensembles to transform numerical variables into binarized embeddings, while @xcite suggest a two-step feature expansion and deep transformation technique. we propose here an approach to numerical feature embeddings based on properties of the hyperbolic tangent (tanh) function. the tanh function exhibits a dual nature that aligns well with the structure of tabular data: with large weight w, tanh(wx + b) captures a tree-like inductive bias by creating a sharp decision boundary, while with small w it approximates a linear function; see figure 1 . however, with poor initialization, neural network training using tanh can lead to vanishing gradients and unstable learning. to overcome this, we introduce an initialization method based on a simple probability argument. the new method ensures that the embedding parameters w and b start in a region that facilitates both tree-like and linear representations. empirically, the new initialization method does achieve the desired benefits of more stable training and better accuracy. experiments demonstrate the effectiveness of our approach in two scenarios. in the first scenario, we compare embeddings using fixed dimensions, where the model hyperparameters are tuned without considering the embedding layer. in this case, the new tanh-based approach consistently outperforms previous embedding methods across various datasets and model architectures. in the second scenario, we compare against previous relu embeddings, where both the model parameters and the embedding dimensions were tuned for the use of relu. even in this challenging comparison, tanhbased embeddings lead to accuracy improvements. overall, the new approach can achieve competitive or superior performance with minimal tuning overhead, making it particularly practical for scenarios where extensive hyperparameter search is not feasible. ## related work the application of deep learning to tabular data has historically been challenging, with gbdts often achieving superior performance @xcite . recent studies provide insights into this performance gap: @xcite demonstrated that tree-based models' success stems from their inherent ability to learn effective decision boundaries and handle heterogeneous features, while @xcite identified specific data characteristics where neural networks can potentially outperform gbdts. traditional neural networks treat numerical features as direct inputs without specialized processing. this approach has limitations in capturing complex feature interactions and non-linear relationships. recent work by @xcite has studied simple differentiable embeddings, which apply a linear transformation followed by an activation function, and piecewise linear embeddings, which creates disjoint learnable bins for feature values. their experiments demonstrated that these embeddings can significantly improve neural network performance on tabular data. more recently, @xcite proposed a treeregularized method that uses tree ensembles to transform numerical variables into binarized embeddings, and @xcite introduced a unified framework employing lightweight neural networks for both numerical and categorical features, utilizing two-step feature expansion and transformation. importantly, neither of these methods is a standard single neural network that can be trained by backpropagation in a standard way, whereas the method that we suggest below can be. recent research has also made progress in closing the performance gap with gbdts through other innovations in feature processing and model architecture. transformer-based models such as tabtransformer @xcite and ft-transformer @xcite tokenize the features, using attention mechanisms to capture complex feature interactions. hybrid approaches such as node @xcite incorporate tree-like structures into neural architectures, while dcn v2 @xcite uses cross networks to model feature interactions. however, these methods are also not a simple single neural network that is trainable in a standard way. ## the tanh-based embedding method given a tabular dataset with numerical features, our goal is to develop an embedding method that can capture both linear and non-linear relationships in the data. let @xmath0, where p is the number of features. the new embedding method maps each feature x i to r d , where d is the embedding dimension. previous approaches using relu activation functions in the embedding layer can be expressed as e @xmath1. we propose replacing the relu activation with tanh, so e @xmath2. the advantage is that with large embedding weights w i,j ≫ 1, each component of the embedding captures a tree-like inductive bias, by creating a sharp decision boundary. conversely, with a small weight w i,j ≪ 1, a component approximates a linear transformation, because tanh(x) ≈ x for small x. see figure 1 . we also propose an enhanced embedding variant with a second transformation layer: ## design of experiments the embedding dimension is a hyperparameter that has to be chosen to balance model expressiveness with computational efficiency. we conduct experiments under two scenarios that differ in how base model parameters (e.g., hidden dimensions of mlp) and embedding dimensions are selected. scenario 1 -preassigned dimensions: in this scenario, we adopt the optimized base model parameters (hidden dimensions and dropout rates for mlp, number of blocks etc. for resnet and transformer) obtained from hyperparameter search without considering embeddings, and use preassigned embedding dimensions. this allows us to evaluate the impact of replacing relu with tanh activations while keeping all architectural choices fixed. scenario 2 -relu-optimized dimensions: here, we adopt both the base model parameters and embedding dimensions that are obtained from hyperparameter search for relu embeddings variants. this scenario is split into two cases. scenario 2a uses tuned hyperparameter for linear embedding, and scenario 2b uses tuned hyperparameter for piecewise linear embedding, as reported in @xcite . this creates a challenging comparison where we replace relu with tanh in settings optimized for relu, demonstrating the robustness of our approach. importantly, we do not perform additional hyperparameter search for model parameters or embedding dimensions for the tanh-based approach. this evaluation strategy demonstrates that the benefits of our method are inherent rather than the result of hyperparameter search, making it a drop-in replacement for relu-based embeddings in applications where extensive tuning may be infeasible. as baselines for comparison, we consider two control groups that, before feeding the input data to the base model, process it through an extra single fully-connected layer for all features, with either relu or tanh activation functions. thus @xmath3, @xmath4. these control groups separate the impact of our feature-wise embedding approach from the increase in dimensionality by comparing against a baseline fully-connected layer. we evaluate three base model architectures, following the implementations in @xcite . ## results and analysis in tables 3 to 7 mse (↓: lower is better) denotes mean-square error and auc (↑: higher is better) denotes area under the roc curve. the best results for each dataset are shown in bold. multiple bold entries in the same column indicate results that are statistically equivalent.