Learning Enhanced Representations for Tabular Data via Neighborhood Propagation
Abstract
Prediction over tabular data is an essential and fundamental problem in manyimportant downstream tasks.
However, existing methods either take a data instanceof the table independently as input or do not fully utilize the multi-rows featuresand labels to directly change and enhance the target data representations.
Inthis paper, we propose to 1) construct a hypergraph from relevant data instanceretrieval to model the cross-row and cross-column patterns of those instances, and 2)perform message P ropagation to E nhance the target data instance representation for
T abular prediction tasks.
Specifically, our specially-designed message propagationstep benefits from 1) fusion of label and features during propagation, and 2)locality-aware high-order feature interactions.
Experiments on two importanttabular data prediction tasks validate the superiority of the proposed PET modelagainst other baselines.
Additionally, we demonstrate the effectiveness of themodel components and the feature enhancement ability of PET via various ablationstudies and visualizations.
Introduction
Prediction over tabular data is a fundamental and essential problem in many data science applicationsincluding recommender systems (Bobadilla et al., 2013;
Ying et al., 2018), online advertising(Richardson et al., 2007;
Zhou et al., 2018), fraud detection (Bolton and Hand, 2002), questionanswering (Chen et al., 2020) etc.
Most existing methods seek to capture the patterns of featureinteractions within an instance independently using tree models (Chen and Guestrin, 2016) or deepnetworks (Guo et al., 2017).
Recently, as shown by Papernot and McDaniel (2018) feeding in neighbors of the target as inputcan improve the robustness of data representations and help the model generalize better to out-ofdistribution samples.
Some retrieval based methods (Pi et al., 2020; Qin et al., 2020, 2021) then seekto utilize multiple neighboring data instances of the target for label prediction.
These methods take theauxiliary instances as extra inputs but do not consider that such auxiliary information could enhancethe target representation.
Other existing graph-based methods on tabular data (You et al., 2020;
Wu et al., 2021a; Guo et al., 2021b) aim to learn robust representations with locality structure, as
Verma and Zhang (2019) proved the stabilization and generalization ability of graph neural networks.
However, they either omit the high-order product feature interactions, or else serve as a pluginwhile resorting to other architectures like factorization machines (Rendle, 2010) to explore featureinteractions.
Moreover, they ignore the mutual enhancement of the representations between labelsand features.
In this paper, with the aim of learning enhanced data representations of tabular data,we propose to construct a hypergraph among relevant data instances to model the set relationships(Srinivasan et al., 2021).
Additionally, we design an end-to-end graph neural network prediction
[Figure]
model that generates high-order feature interactions with the assistance of the locality structure andlabel adjustment.
Specifically, we design PET, a novel architecture that P ropagates and E nhances the T abular datarepresentations based on the hypergraph for target label prediction.
We first retrieve from theobserved data instances pool to get the neighboring data instances for each target.
The resultinginstance set can be seen as a hypergraph, where sets of feature columns of data instances formhyperedges and each distinct feature value of these data instances form a node.
This hypergraphmodels the cross-row and cross-column relations of the resulting instance set.
Then we conductmessage propagation on the graph.
The propagation serves three purposes.
First, auxiliary labelinformation from the retrieved data instances propagates through the common feature value nodes tohelp the target prediction.
Second, feature representations get enhanced through the locality structure.
Our interactive message generation, attention-based aggregation, and update generate locality-awarehigh-order feature interactions.
Third, the labels are incorporated into the propagating messages todirectly adjust the feature spaces and generate label-enhanced feature representations.
The main contributions are summarized as follows:
• We propose a retrieval-based hypergraph to capture the feature and label correlations amongtabular data instances.•
We design an end-to-end graph neural network prediction model that unifies the productfeature interaction, locality mining, and label enhancement.•
We utilize the observed labels in the resulting set to guide the feature learning process anduse the propagated labels to enhance predictions.
We evaluate the proposed PET model on
two prediction tasks, i.e., binary classification and top-nranking, over five tabular datasets, where substantial performance improvement against other strongbaselines validates the superiority of PET.
Preliminaries and Related Work
Tabular data prediction.
Tabular data prediction treats every row of the table as a data instance andevery column/field as a feature attribute.
1 We consider tabular data prediction under a single tablescenario in this paper, and prediction over multiple tables can be seen as the prediction over a joinedtable.
We also focus on tabular discrete data in this paper, while the continuous feature values can bediscretized in various ways (Guo et al., 2021a).
Currently some of the most widely used models fortabular data prediction include Gradient Boosting Decision Trees (GBDTs)
(Friedman, 2001) and
Factorization Machines (FM) (Rendle, 2010).
Variants of FM such as DeepFM (Guo et al., 2017) and
Wide & Deep (Cheng et al., 2016) are especially popular in industrial applications.
For a given row,such models simply take in the row and make predictions, i.e., these models assume that the rowsare IID.
Such an assumption, however, is reasonable only if the embedding representations of the
[Figure]
tabular data can sufficiently host the high-order interaction patterns within the data instance, which ispractically impossible.
Graph neural networks on tabular data.
As Graph Neural Networks (GNNs) become popular,multiple attempts to apply GNNs on tabular data prediction have emerged.
Since the model takesin a graph that connects the rows and columns together, the prediction on a single row no longerdepends only on itself, but also other rows, and is therefore suitable for exploiting non-IID propertiesin tabular data.
Examples of using GNNs on tabular data include Wu et al.
(2021a) that constructsa mini-batch data-feature bipartite graph, You et al.
(2020) that treats the (incomplete) table as anadjacency matrix of a bipartite graph, and Guo et al.
(2021b) that treats rows as nodes and build edgesaccording to predefined rules.
These methods omit the product feature interactions or else resort toother architecture like factorization machines to explore them.
Moreover, they ignore the mutualenhancement between labels and features.
Hypergraphs and hyperedge classification/regression.
If we treat each individual value in a tableas a single node, then a row describes an n -ary relationship among the nodes.
Normal graphs havetrouble describing this relation since edges can only connect two nodes at a time.
A hypergraphgeneralizes graphs such that an edge can connect more than two nodes.
It is defined as a pair
G =
( V, E ) with its node set V and its hyperedge set E ⊆ P ( V ) , where P ( V ) is the power set of
V , meaning that each "edge" becomes simply a subset of V , regardless of the number of nodes inthe "edge".
We can construct a hypergraph from a set of rows, where each hyperedge correspondto a row and each node correspond to the distinct feature values among all the rows.
Tabular dataprediction problem can be cast into a hyperedge classification/regression problem, where every rownow corresponds to a hyperedge.
Hypergraph neural networks.
Hypergraph neural networks are an adaptation of GNNs with a message passing paradigm whereby node representations are used to update hyperedge representations,which in turn update the node representations again (Srinivasan et al., 2021; Bai et al., 2021; Fenget al., 2019).
Using a hypergraph neural network on tabular data allows us to obtain a representationfor each row from its corresponding hyperedge, as well as a representation for each individual valueof each column from its corresponding node.
Methodology
When making prediction for a sample, feeding in similar or relevant samples together as inputis known to contribute to robustness and generalization abilities for out-of-distribution samples(Papernot and McDaniel, 2018).
In light of this, for a given data instance, our model retrieves a set ofrelevant data instances according to a relevance metric, aiming to take auxiliary information fromrelevant data instances.
The resulting instances set can be seen as a hypergraph, where each distinct feature value forms anode and a collection of them, i.e., a data instance, forms a hyperedge.
After the star expansion(Agarwal et al., 2006; Srinivasan et al., 2021), we get a bipartite graph with feature value nodes onone side and data instance nodes on the other, on which we propagate and enhance representations.
An advantage of message passing is that it allows us to capture higher-order interactions amongnodes and hyperedges (Srinivasan et al., 2021), i.e. the interactions among the individual columnvalues as well as the rows.
We argue that these high-order interactions play an essential role inthe real-world scenarios.
{+Assume that there are F fields in the given tabular, then there exists C 
+}
Fsecond order interactions and C ·· denotes the combination operation (i.e., C NF N -th order interactions).
Notice that (2-way) factorization machine (Rendle, 2010) solely modelling the second-order featureinteractions is widely employed in real-world scenarios and inspires a series of tabular predictors (Wuet al., 2021b; Qin et al., 2021). Our method leverages the hypergraph structure of the given tabulardata to capture high-order interactions beyond the second-order ones, and uses GNNs to representthese high-order features. The other advantage of message passing is that we can utilize the labels ofretrieved data instances to interact with features and take advantage of label propagation at the sametime.
The enhanced data instance representations are then used for prediction.
Figure 2 illustrates the framework of PET.
Detailed descriptions of each individual component areprovided in the following subsections.
3.1 Graph Construction
For each target data instance, we retrieve K relevant data instances from the observed data instancesset using ElasticSearch 2 and construct a hypergraph from the resulting instance set.
Let X t = { x tf | f = 1 , . .
. , F } be the feature of a target data instance t , where F is the number offeature fields and x tf is the feature value of the f -th field of data instance t .
We first conduct a booleanquery to obtain instances that have at least one common feature value with the target data instance.
Then we retrieve the top- K relevant instances r 1 , · · · , r K from the filtered instances set according tothe relevance value (Qin et al., 2021) defined as:
[Equation]
where 1 ( · ) is the indicator function, N is the number of data instances in the table, and N ( x tf ) is thenumber of data instances that have feature value x tf in the f -th field.
This metric is equivalent to the
BM25 (Robertson et al., 1995) metric if we treat each data instance as a document and their featurevalues as the terms.
After the retrieval, the resulting instances set { X t , X r 1 , · · · , X r K } can be seen as a hypergraph,where each distinct feature value in each individual field forms a node and each data instanceconstructs a hyperedge.
We then perform a star expansion (Agarwal et al., 2006) on the hypergraph,i.e. construct a bipartite graph G =
( V D , V F , E ) where V D = { t, r 1 , . . .
, r K } , V F = { x jf | f =1 , · · · , F ; j = t, r 1 , . . .
, r K } , and an edge exists between two nodes i ∈ V F and j ∈ V D if i ∈
X j .
3.2 Message Passing and Interaction
After the graph is constructed, we propagate and enhance the representations on it.
The propagationserves three purposes.
First, the label information propagates through common feature value nodesto help the label prediction of the target data instance node.
Second, the features get enhancedthrough taking in high-order information.
Locality-aware high-order product feature interactionsare generated through the interactive message generation and attention based aggregation.
Third,the label embeddings directly interact with the features to adjust the feature spaces and generatelabel-enhanced features and high-order interactions.
Then we detail the components as follows.
Initialization.
Before message passing, we initialize the node representations and edge representations with trainable embedding vectors.
Let Φ x and Φ y be different trainable embedding layers forfeature value nodes and data instances nodes, respectively.
And let Φ in and Φ out be the embeddinglayers for the edges.
We first initialize the feature value node representations as
[Equation]
We also initialize the data instance node representations of the retrieved rows with a trainable embedding vector associated with their labels.
As for the target data node, we initialize its representationswith a constant zero vector.
[Equation]
where y j denotes the label of the data instance j .
In addition, we initialize each edge representation with the embedding of its incident data instancenode label to guide the feature learning process and generate high-order interactions. [Equation]
[Equation]
Message Generation.
Then we use the interactions between edge and node representations alongwith the original node representations to generate the messages:
[Equation]
where ⊙ denotes the Hadamard product, superscript l means the l -th layer, and ∥ denotes theconcatenation operation.
As edge representations carry the label information initially, the Hadamard product term generateslabel-enhanced features.
In addition, the edges will contain node features after updating edgerepresentations with incident node representations, then the Hadamard product term produces highorder product feature interactions.
The high-order product feature interactions prove to be importantin many interaction-based tabular data prediction methods (Qu et al., 2018), which are often missed inother graph- networks captured an extra
Message Aggregation.
The incoming neighboring messages are then aggregated based on anattention mechanism:
[Equation]
Node Embedding Update.
After receiving the aggregated neighboring messgages, the node embeddings are updated based on the aggregated messages and their own node embeddings.
[Equation]
where σ ( · ) denotes the ReLU activation function.
Edge Embedding Update.
Then we update edge embeddings using their incident node embeddingsas Equation (9).
As edge embeddings are updated using the node embeddings, the features and labelson nodes are propagated to the edges as well.
After rounds of propagation, the messages generatedcontain high-order feature interactions and high-order feature-label interactions.
[Equation]
[Figure]
20s
Teacher
Female
Doctor ...
Initialization
One round of propagation and update
Received messages
Node information from last layer
Two rounds of propagation and update
Representation of target:
First-order features:
Propagated label:
Label-enhanced features:
Second-order features: ...
MLP
Higher-order interactions
3.3 Label Prediction
Then we use the target data node embedding of the last layer for prediction:
[Equation]
For binary classification, the training objective is set as cross entropy:
[Equation]
where y t is the label of the target data instance t .
3.4 Analysis
We illustrate the representation of the target node after propagation in Figure 3.
Following the proposed propagation method, the target node will receive the first order features, second-order features,label-enhanced features, and the propagated labels after two rounds of propagation.
Propagating morethan two layers will further produce higher-order feature interactions, cross-rows feature interactions,and feature-label interactions.
Furthermore, the feature messages are adjusted by correspondinglabels and attentively aggregated based on the locality structure.
Experiments
In this section, we show the experimental results and the corresponding settings.
Generally, weexperiment on two kinds of tabular data prediction tasks: click-through rate (CTR) prediction andtop-n recommendation.
The results showcase the value of our model on tabular data prediction tasks.
Additionally, we conduct several ablation studies to validate the components of our model.
4.1 Setup
[Table]
[Table]
We evaluate the performance of PET on five datasets.
Forthe CTR prediction task, we conduct experiments on threelarge-scale datasets, i.e., Tmall 3 , Taobao 4 , and Alipay 5 .
For the top-n recommendation task, we experiment ontwo widely-used public recommendation datasets, i.e.,
Movielens-1M 6 and LastFM 7 .
The statistics of the useddatasets are summarized in Table 1. [Table]
[Table] [Table] [Table] The evaluation metrics include area under ROC curve (AUC) and negative log-likehood (LogLoss)for CTR prediction tasks and hit rate (HR), normalized discounted cumulative gain (NDCG) andmean reciprocal rank (MRR) for top-n recommendation tasks.
Following Qin et al.
(2021), we spilt the datasets according to the global timestamps.
The earliestdata instances are grouped into the retrieval pool.
The latest data instances form the test pool.
Thenthe remaining data instances are grouped into the train pool.
To avoid unfair comparisons, thenon-retrieval model takes the retrieval pool as additional train pool.
On CTR prediction tasks, we compare our model against eight widely-used and strong baselines.
GBDT (Chen and Guestrin, 2016) is the widely-used tree model.
DeepFM (Guo et al., 2017) is theinner-product interaction-based model.
DIN (Zhou et al., 2018) and DIEN (Zhou et al., 2019)
arethe attention-based sequential models.
SIM (Pi et al., 2020), UBR (Qin et al., 2020), and RIM (Qinet al., 2021) are the retrieval-based models.
On top-n recommendation tasks, we compare PET withsix strong recommendation models, including factorization-based FPMC (Rendle et al., 2010) and TransRec (He et al., 2017), and recently proposed DNN models NARM (Li et al., 2017), GRU4Rec(Hidasi et al., 2016), SASRec (Kang and McAuley, 2018), and RIM (Qin et al., 2021).
As for the hyperparameters, we test the number of GNN layers in { 2 , 3 } .
The embedding sizes of allthe models are consistent to ensure the fair comparison.
More detailed hyperparameters are providedin the Appendix.
4.2 Overall performance comparison
We first validate the effectiveness of the proposed PET model.
The main results are summarized in
Tables 2 and 3, where we can see the proposed PET performs consistently better on all datasets.
The results demonstrate the superiority of PET against the baselines on both tasks.
On the CTRprediction task, PET achieves relatively 2 .
54% higher AUC over the best performedbaseline on Tmall, Taobao, Alipay, respectively.
The results show that PET can learn effective tabulardata representation for better prediction performance.
Furthermore, compared with RIM that takesexactly the same inputs with PET, the improvements of PET are statistically significant under 95%confidence level.
Given that RIM also utilizes the labels of relevant rows, this empirically justifies the capability of our propagation method.
On the top-n recommendation task, PET shows significantimprovements on the recommendation task against other baselines, too.
The results show that PETperforms substantially better than the strong baselines in all comparisons.
4.3 Ablation study
4.3.1 Impact of the label usages
We further study the usage of labels of PET.
The results are summarized in Table 4.
For fastexploration, we randomly sample 10% data from training pool and test pool on the three largest CTRdatasets, respectively.
The best performed baseline RIM is tested on the sampled data for comparison.
Both RIM and PET utilize the same set of retrieved data instances and their label information.
Wethen examine the impact of different label usages on the sampled data.
In the PET model, we initializethe embeddings of data instances nodes and edges with the corresponding label embeddings.
We theninspect the impact of such initialization and the operations on edges.
[Table]
We first remove edge embedding initialization with labels and eliminate all the operations involvedwith edges, thus the message becomes
[Equation]
In addition, the edge embedding update step in Equation (9) is also omitted.
As such, the labelsonly serve from propagation among nodes.
The high-order product interaction between featuresand label-feature interaction are omitted.
As shown in Table 4, PET without edge labels performsweaker than the original PET model, which demonstrates the power of high-order product interactionsand label-enhanced features.
Moreover, PET without edge labels still performs better than RIM,which validates the superiority of label propagation through common feature nodes against simpleattention-based aggregation.
By removing the node labels, the embeddings of all the data instance nodes are initialized as constantvectors.
In this way, no pure first-order label embeddings are used for prediction.
Since anyinformation related with labels will be propagated to nodes after a product interaction with features.
One can see that PET without node labels performs better than RIM, which further justifies theeffectiveness of high-order product interactions and label-enhanced features.
Additionally, PETwithout node labels performs weaker than the original PET, which implies the power of initializingnode embeddings with pure label embeddings for propagation.
4.3.2 Visualization of the feature distribution
In order to further explore the representation enhancement of PET, we visualize the features of PET and RIM. Figure 4 illustrates the distributions of data instance embeddings and the featureembeddings of PET and RIM with t-SNE (Van der Maaten and Hinton, 2008).
For both models, werandomly choose 10,000 samples from the train data and 10,000 samples from the test data.
For PET,the data instance embeddings are taken from the data instance nodes (inputs of the MLP predictor)and the feature embeddings are the concatenated feature node embeddings for each data instance.
For RIM, the data instance embeddings are the inputs of the final MLP predictor and the featureembeddings are the concatenated feature embeddings of each data instance.
From Figure 4, we can see that the distributions of train positive data points and train negative datapoints are more dissimilar than those of RIM.
The same phenomenon happens in the distributions oftest positive data points and test negative data points.
This illustrates that PET gives more informativerepresentations.
[Table]
4.3.3 Impacts of different retrieval schemes
In this section, we compare the performance of PET w.r.t.
different retrieval sizes and retrievalschemes.
First, we evaluate the effectiveness of current retrieval by comparing the current retrievalscheme with the random retrieval scheme.
The results are summarized in Table 5.
One can see thatthe relevance-based retrieval performs much better the random retrieval.
Since the relevance-basedretrieval helps the resulting graph to achieve the homophily assumption, which is important in GNNs.
We also study the impact of using different retrieval sizes K .
The results are put in the Appendixdue to the page limit.
Generally, too few retrieval samples may fail to carry enough information toenhance the representation, while too many retrieved samples will introduce more noise.
Conclusion
In this paper, we focus on improving the prediction of tabular data, which is essential in manyimportant downstream tasks.
Existing methods either take each data instance independently ordirectly take multiple data instances as input without enhancing the target data instance representation.
We propose to construct a retrieval-based hypergraph to model the cross-row and cross-columnrelations of tabular data, utilizing the propagation on the resulting graph to directly change andenhance the target data instance representations.
Concretely, we utilize a relevance retrieval toconstruct the hyperedges set of the hypergraph, aiming to resort to relevant patterns and reach thehomophily assumption of GNNs.
Then we design PET, a novel architecture that propagates andenhances the tabular data representations based on the hypergraph for target label prediction.
Thepropagation serves from three aspects: 1) label propagates through common feature values; 2) featuresget enhanced through locality and high-order product feature interactions are generated through theinteractive message passing framework; and 3) labels are used to generate the label-enhanced features.
Experiments on two important tabular data prediction tasks validate the superiority of the proposed PET model over strong baselines.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets? [No] The code and the data areproprietary.• Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes]