Learning Enhanced Representations for Tabular Data via Neighborhood Propagation
Abstract
Prediction over tabular data is an essential and fundamental problem in manyimportant downstream tasks.
However, existing methods either take a data instanceof the table independently as input or do not fully utilize the multi-rows featuresand labels to directly change and enhance the target data representations.
Inthis paper, we propose to 1) construct a hypergraph from relevant data instanceretrieval to model the cross-row and cross-column patterns of those instances, and 2)perform message P ropagation to E nhance the target data instance representation for
T abular prediction tasks.
Specifically, our specially-designed message propagationstep benefits from 1) fusion of label and features during propagation, and 2)locality-aware high-order feature interactions.
Experiments on two importanttabular data prediction tasks validate the superiority of the proposed PET modelagainst other baselines.
Additionally, we demonstrate the effectiveness of themodel components and the feature enhancement ability of PET via various ablationstudies and visualizations.
Introduction
Prediction over tabular data is a fundamental and essential problem in many data science applicationsincluding recommender systems (Bobadilla et al., 2013;
Ying et al., 2018), online advertising(Richardson et al., 2007;
Zhou et al., 2018), fraud detection (Bolton and Hand, 2002), questionanswering (Chen et al., 2020) etc.
Most existing methods seek to capture the patterns of featureinteractions within an instance independently using tree models (Chen and Guestrin, 2016) or deepnetworks (Guo et al., 2017).
Recently, as shown by Papernot and McDaniel (2018) feeding in neighbors of the target as inputcan improve the robustness of data representations and help the model generalize better to out-ofdistribution samples.
Some retrieval based methods (Pi et al., 2020; Qin et al., 2020, 2021) thenseek to utilize multiple neighboring data instances of the target for label prediction.
These methodstake the auxiliary instances as extra inputs but do not consider that such auxiliary information couldenhance the target representation.
Other existing graph-based methods on tabular data (You et al.
,2020; Wu et al., 2021; Guo et al., 2021b) aim to learn robust representations with locality structure, as
Verma and Zhang (2019) proved the stabilization and generalization ability of graph neural networks.
However, they either omit the high-order product feature interactions, or else serve as a pluginwhile resorting to other architectures like factorization machines (Rendle, 2010) to explore featureinteractions.
Moreover, they ignore the mutual enhancement of the representations between labelsand features.
In this paper, with the aim of learning enhanced data representations of tabular data,we propose to construct a hypergraph among relevant data instances to model the set relationships(Srinivasan et al., 2021).
Additionally, we design an end-to-end graph neural network prediction
[Figure]
model that generates high-order feature interactions with the assistance of the locality structure andlabel adjustment.
Specifically, we design PET, a novel architecture that P ropagates and E nhances the T abular datarepresentations based on the hypergraph for target label prediction.
We first retrieve from theobserved data instances pool to get the neighboring data instances for each target.
The resultinginstance set can be seen as a hypergraph, where sets of feature columns of data instances formhyperedges and each distinct feature value of these data instances form a node.
This hypergraphmodels the cross-row and cross-column relations of the resulting instance set.
Then we conductmessage propagation on the graph.
The propagation serves three purposes.
First, auxiliary labelinformation from the retrieved data instances propagates through the common feature value nodes tohelp the target prediction.
Second, feature representations get enhanced through the locality structure.
Our interactive message generation, attention-based aggregation, and update generate locality-awarehigh-order feature interactions.
Third, the labels are incorporated into the propagating messages todirectly adjust the feature spaces and generate label-enhanced feature representations.
The main contributions are summarized as follows:
• We propose a retrieval-based hypergraph to capture the feature and label correlations amongtabular data instances.•
We design an end-to-end graph neural network prediction model that unifies the productfeature interaction, locality mining, and label enhancement.•
We utilize the observed labels in the resulting set to guide the feature learning process anduse the propagated labels to enhance predictions.
We evaluate the proposed PET model on
two prediction tasks, i.e., binary classification and top-nranking, over five tabular datasets, where substantial performance improvement against other strongbaselines validates the superiority of PET.
Preliminaries and Related Work
Tabular data prediction.
Tabular data prediction treats every row of the table as a data instance andevery column/field as a feature attribute.
1 We consider tabular data prediction under a single tablescenario in this paper, and prediction over multiple tables can be seen as the prediction over a joinedtable.
We also focus on tabular discrete data in this paper, while the continuous feature values can bediscretized in various ways (Guo et al., 2021a).
Currently some of the most widely used models fortabular data prediction include Gradient Boosting Decision Trees (GBDTs)
(Friedman, 2001) and
Factorization Machines (FM) (Rendle, 2010).
Variants of FM such as DeepFM (Guo et al., 2017) and
Wide & Deep (Cheng et al., 2016) are especially popular in industrial applications.
For a given row,such models simply take in the row and make predictions, i.e., these models assume that the rowsare IID.
Such an assumption, however, is reasonable only if the embedding representations of the
[Figure]
tabular data can sufficiently host the high-order interaction patterns within the data instance, which ispractically impossible.
Graph neural networks on tabular data.
As Graph Neural Networks (GNNs) become popular,multiple attempts to apply GNNs on tabular data prediction have emerged.
Since the model takesin a graph that connects the rows and columns together, the prediction on a single row no longerdepends only on itself, but also other rows, and is therefore suitable for exploiting non-IID propertiesin tabular data.
Examples of using GNNs on tabular data include Wu et al.
(2021) that constructsa mini-batch data-feature bipartite graph, You et al.
(2020) that treats the (incomplete) table as anadjacency matrix of a bipartite graph, and Guo et al.
(2021b) that treats rows as nodes and build edgesaccording to predefined rules.
These methods omit the product feature interactions or else resort toother architecture like factorization machines to explore them.
Moreover, they ignore the mutualenhancement between labels and features.
Hypergraphs and hyperedge classification/regression.
If we treat each individual value in a tableas a single node, then a row describes an n -ary relationship among the nodes.
Normal graphs havetrouble describing this relation since edges can only connect two nodes at a time.
A hypergraphgeneralizes graphs such that an edge can connect more than two nodes.
It is defined as a pair
G =
( V, E ) with its node set V and its hyperedge set E ⊆ P ( V ) , where P ( V ) is the power set of
V , meaning that each "edge" becomes simply a subset of V , regardless of the number of nodes inthe "edge".
We can construct a hypergraph from a set of rows, where each hyperedge correspondto a row and each node correspond to the distinct feature values among all the rows.
Tabular dataprediction problem can be cast into a hyperedge classification/regression problem, where every rownow corresponds to a hyperedge.
Hypergraph neural networks.
Hypergraph neural networks are an adaptation of GNNs with a message passing paradigm whereby node representations are used to update hyperedge representations,which in turn update the node representations again (Srinivasan et al., 2021; Bai et al., 2021; Fenget al., 2019).
Using a hypergraph neural network on tabular data allows us to obtain a representationfor each row from its corresponding hyperedge, as well as a representation for each individual valueof each column from its corresponding node.
Methodology
When making prediction for a sample, feeding in similar or relevant samples together as inputis known to contribute to robustness and generalization abilities for out-of-distribution samples(Papernot and McDaniel, 2018).
In light of this, for a given data instance, our model retrieves a set ofrelevant data instances according to a relevance metric, aiming to take auxiliary information fromrelevant data instances.
The resulting instances set can be seen as a hypergraph, where each distinct feature value forms anode and a collection of them, i.e., a data instance, forms a hyperedge.
After the star expansion(Agarwal et al., 2006; Srinivasan et al., 2021), we get a bipartite graph with feature value nodes onone side and data instance nodes on the other, on which we propagate and enhance representations.
An advantage of message passing is that it allows us to capture higher-order interactions among nodesand hyperedges (Srinivasan et al., 2021), i.e. the interactions among the individual column values aswell as the rows.
[-Assume that there are F fields in the given tabular, then there exists C 
-]
The other advantage of message passing is that we can utilize the labels of retrieveddata instances to interact with features to generate label-enhanced messages, guide the messageaggregation process, and take advantage of label propagation at the same time.
After message passing,the enhanced data instance representations are then used for prediction.
Figure 2 illustrates the framework of PET.
Detailed descriptions of each individual component areprovided in the following subsections.
3.1 Graph Construction
For each target data instance, we retrieve K relevant data instances from the observed data instancesset using ElasticSearch 2 and construct a hypergraph from the resulting instance set.
Let X t = { x tf | f = 1 , . .
. , F } be the feature of a target data instance t , where F is the number offeature fields and x tf is the feature value of the f -th field of data instance t .
We first conduct a booleanquery to obtain instances that have at least one common feature value with the target data instance.
Then we retrieve the top- K relevant instances r 1 , · · · , r K from the filtered instances set according tothe relevance value (Qin et al., 2021) defined as:
[Equation]
where 1 ( · ) is the indicator function, N is the number of data instances in the table, and N ( x tf ) is thenumber of data instances that have feature value x tf in the f -th field.
This metric is equivalent to the
BM25 (Robertson et al., 1995) metric if we treat each data instance as a document and their featurevalues as the terms.
After the retrieval, the resulting instances set { X t , X r 1 , · · · , X r K } can be seen as a hypergraph,where each distinct feature value in each individual field forms a node and each data instanceconstructs a hyperedge.
We then perform a star expansion (Agarwal et al., 2006) on the hypergraph,i.e. construct a bipartite graph G =
( V D , V F , E ) where V D = { t, r 1 , . . .
, r K } , V F = { x jf | f =1 , · · · , F ; j = t, r 1 , . . .
, r K } , and an undirected edge exists between two nodes i ∈ V F and j ∈ V Dif i ∈
X j .
3.2 Message Passing and Interaction
After the graph is constructed, we propagate and enhance the representations on it.
The propagationserves three purposes.
First, the label information propagates through common feature value nodesto help the label prediction of the target data instance node.
Second, the features get enhancedthrough taking in high-order information.
Locality-aware high-order product feature interactionsare generated through the interactive message generation and attention based aggregation.
Third,the label embeddings directly interact with the features to adjust the feature spaces and generatelabel-enhanced features and high-order interactions.
Then we detail the components as follows.
Initialization.
Before message passing, we initialize the node representations and edge representations with trainable embedding vectors.
Let Φ x and Φ y be different trainable embedding layers forfeature value nodes and data instances nodes, respectively.
And let Φ in and Φ out be the embeddinglayers for the edges.
We first initialize the feature value node representations ash (
[Equation]
We also initialize the data instance node representations of the retrieved rows with a trainable embedding vector associated with their labels.
As for the target data node, we initialize its representationswith a constant zero vector.
[Equation]
where y j denotes the label of the data instance j .
In addition, we initialize each edge representation with the embedding of its incident data instancenode label to guide the feature learning process and generate high-order interactions.
[Equation]
Message Generation.
Then we use the interactions between edge and node representations alongwith the original node representations to generate the messages:
[Equation]
where ⊙ denotes the Hadamard product, superscript l means the l -th layer, and ∥ denotes theconcatenation operation.
As edge representations carry the label information initially, the Hadamard product term generateslabel-enhanced features.
In addition, the edges will contain node features after updating edgerepresentations with incident node representations, then the Hadamard product term produces highorder product feature interactions.
The high-order product feature interactions prove to be importantin many interaction-based tabular data prediction methods (Qu et al., 2018), which are often missedin other graph-based tabular networks or captured by an extra factorization machine (Wu et al., 2021).
Message Aggregation.
The incoming neighboring messages are then aggregated based on anattention mechanism:
[Equation]
Node Embedding Update.
After receiving the aggregated neighboring messgages, the node embeddings are updated based on the aggregated messages and their own node embeddings.
[Equation]
where σ ( · ) denotes the ReLU activation function.
Edge Embedding Update.
Then we update edge embeddings using their incident node embeddingsas Equation (9).
As edge embeddings are updated using the node embeddings, the features and labelson nodes are propagated to the edges as well.
After rounds of propagation, the messages generatedcontain high-order feature interactions and high-order feature-label interactions. [Equation] [Equation]
3.3 Label Prediction
Then we use the target data node embedding of the last layer for prediction:
[Equation]
[Figure]
Relevant [Figure] (retrieved) 20s
Teacher
Female
Doctor ...
Initialization
One round of propagation and update
Received messages
Node information from last layer
Two rounds of propagation and update
Representation of target:
First-order features:
Propagated label:
Label-enhanced features:
Second-order features: ...
MLP
Higher-order interactions
For binary classification, the training objective is set as cross entropy:
[Equation]
where y t is the label of the target data instance t .
3.4 Analysis
We illustrate the representation of the target node after propagation in Figure 3.
Following the proposed propagation method, the target node will receive the first order features, second-order features,label-enhanced features, and the propagated labels after two rounds of propagation.
Propagating morethan two layers will further produce higher-order feature interactions, cross-rows feature interactions,and feature-label interactions.
Furthermore, the feature messages are adjusted by correspondinglabels and attentively aggregated based on the locality structure.
Experiments
In this section, we show the experimental results and the corresponding settings.
Generally, weexperiment on two kinds of tabular data prediction tasks: click-through rate (CTR) prediction andtop-n recommendation.
The results showcase the value of our model on tabular data prediction tasks.
Additionally, we conduct several ablation studies to validate the components of our model.
4.1 Setup
[Table]
We evaluate the performance of PET on five datasets.
Forthe CTR prediction task, we conduct experiments on threelarge-scale datasets, i.e., Tmall 3 , Taobao 4 , and Alipay 5 .
For the top-n recommendation task, we experiment ontwo widely-used public recommendation datasets, i.e.,
Movielens-1M 6 and LastFM 7 .
The statistics of the useddatasets are summarized in Table 1.
The evaluation metrics include area under ROC curve (AUC) and negative log-likehood (LogLoss)for CTR prediction tasks and hit rate (HR), normalized discounted cumulative gain (NDCG) andmean reciprocal rank (MRR) for top-n recommendation tasks.
Following Qin et al.
(2021), we spilt the datasets according to the global timestamps.
The earliestdata instances are grouped into the retrieval pool.
The latest data instances form the test pool.
Then [Table] [Table] the remaining data instances are grouped into the train pool.
To avoid unfair comparisons, thenon-retrieval model takes the retrieval pool as an additional train pool.
On CTR prediction tasks, we compare our model against nine widely-used and strong baselines.
GBDT (Chen and Guestrin, 2016) is the widely-used tree model.
DeepFM (Guo et al., 2017) is theinner-product interaction-based model.
FATE (Wu et al., 2021) and TabGNN (Guo et al., 2021b)
arethe recent graph models.
DIN (Zhou et al., 2018) and DIEN (Zhou et al., 2019) are the attentionbased sequential models.
SIM (Pi et al., 2020), UBR (Qin et al., 2020), and RIM (Qin et al., 2021)are the retrieval-based models.
On top-n recommendation tasks, we compare PET with six strongrecommendation models, including factorization-based FPMC (Rendle et al., 2010) and TransRec(He et al., 2017), and recently proposed DNN models NARM (Li et al., 2017), GRU4Rec (Hidasiet al., 2016), SASRec (Kang and McAuley, 2018), and RIM (Qin et al., 2021).
As for the hyperparameters, we test the number of GNN layers in { 2 , 3 } .
The embedding sizes of allthe models are consistent to ensure the fair comparison.
More detailed hyperparameters are providedin the Appendix.
4.2 Overall performance comparison
We first validate the effectiveness of the proposed PET model.
The main results are summarized in
Tables 2 and 3, where we can see the proposed PET performs consistently better on all datasets.
The results demonstrate the superiority of PET against the baselines on both tasks.
On the CTRprediction task, PET achieves relatively 2 .
54% higher AUC over the best performed baseline on Tmall, Taobao, Alipay respectively.
The results show that PET can learn effective tabular data representation for better prediction performance.
Furthermore, compared with RIM that takes exactly the same inputs with PET, the improvements of PET are statistically significant under 95% confidence level.
Given that RIM also utilizes the labels of relevant rows, this empirically justifies the capability of our propagation method.
On the top-n recommendation task, PET shows significantimprovements on the recommendation task against other baselines, too.
The results show that PET performs substantially better than the strong baselines in all comparisons
4.3 Ablation study
4.3.1 Impact of the label usages
We further study the usage of labels of PET.
The results are summarized in Table 4.
For fastexploration, we randomly sample 10% data from training pool and test pool on the three largest CTRdatasets, respectively.
The best performed baseline RIM is tested on the sampled data for comparison.
Both RIM and PET utilize the same set of retrieved data instances and their label information.
Wethen examine the impact of different label usages on the sampled data.
In the PET model, we initializethe embeddings of data instances nodes and edges with the corresponding label embeddings.
We theninspect the impact of such initialization and the operations on edges.
[Table]
We first remove edge embedding initialization with labels and eliminate all the operations involvedwith edges, thus the message becomes
[Equation]
In addition, the edge embedding update step in Equation (9) is also omitted.
As such, the labelsonly serve from propagation among nodes.
The high-order product interaction between featuresand label-feature interaction are omitted.
As shown in Table 4, PET without edge labels performsweaker than the original PET model, which demonstrates the power of high-order product interactionsand label-enhanced features.
Moreover, PET without edge labels still performs better than RIM,which validates the superiority of label propagation through common feature nodes against simpleattention-based aggregation.
By removing the node labels, the embeddings of all the data instance nodes are initialized as constantvectors.
In this way, no pure first-order label embeddings are used for prediction.
Since anyinformation related with labels will be propagated to nodes after a product interaction with features.
One can see that PET without node labels performs better than RIM, which further justifies theeffectiveness of high-order product interactions and label-enhanced features.
Additionally, PETwithout node labels performs weaker than the original PET, which implies the power of initializingnode embeddings with pure label embeddings for propagation. We also provide PET (w/o all labels) to see the feature interactions under the framework. Concretely,we randomly initialize the label values. The results show the power of feature interactions and theadvantages of feeding in labels.
4.3.2 Visualization of the feature distribution
In order to further explore the representation enhancement of PET, we visualize the features of PETand RIM with t-SNE (Van der Maaten and Hinton, 2008). Figure 4 illustrates the distributions of datainstance embeddings and the feature embeddings of PET and RIM on Tmall.
For both models, werandomly choose 10,000 samples from the train data and 10,000 samples from the test data.
For PET,the data instance embeddings are taken from the data instance nodes (inputs of the MLP predictor)and the feature embeddings are the concatenated feature node embeddings for each data instance.
For RIM, the data instance embeddings are the inputs of the final MLP predictor and the featureembeddings are the concatenated feature embeddings of each data instance.
From Figure 4, we can see that the distributions of train positive data points and train negative datapoints are more dissimilar than those of RIM.
The same phenomenon happens in the distributions oftest positive data points and test negative data points.
This illustrates that PET gives more informativerepresentations.
(Visualizations on more datasets can be found in Appendix A.3.)
[Table]
4.3.3 Impacts of different retrieval schemes
First, we evaluate the effectiveness of current retrieval by comparing the current retrieval schemewith the random retrieval scheme.
The results are summarized in Table 5.
One can see that therelevance-based retrieval performs much better than random retrieval.
This validates the effectivenessof the relevance-based retrieval and feeding in relevant rows.
We also study the impact of using different retrieval sizes K .
The results are put in Appendix A.4due to the page limit.
Generally, too few retrieval samples may fail to carry enough information toenhance the representation, while too many retrieved samples will introduce more noise.
Conclusion
In this paper, we focus on improving the prediction of tabular data, which is essential in manyimportant downstream tasks.
We propose to construct a retrieval-based hypergraph to model thecross-row and cross-column relations of tabular data, utilizing the propagation on the resulting graphto directly change and enhance the target data instance representations.
Concretely, we utilize arelevance retrieval to construct the hyperedges set of the hypergraph, aiming to resort to relevantpatterns.
Then we design PET, a novel architecture that propagates and enhances the tabular datarepresentations based on the hypergraph for target label prediction.
The propagation serves fromthree aspects: 1) label propagates through common feature values; 2) features get enhanced throughlocality and high-order product feature interactions are generated through the interactive messagepassing framework; and 3) labels are used to generate the label-enhanced features.
Experiments ontwo important tabular data prediction tasks validate the superiority of the proposed PET model overstrong baselines.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We summarize the main contributions in the introduction
and abstract.
(b) Did you describe the limitations of your work?
[Yes] Please see Appendix A.2.2.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The code is
included in the supplemental material. (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please see Appendix A.6.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] Please see Appendix A.5.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix A.6.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We give the assetssources in the footnotes.
(b) Did you mention the license of the assets?
[Yes] The assets are public.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include our code in the supplemental material.(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[Yes] We use public data that contains no identifiableinformation or offensive content.5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
The pseudo code of the proposed PET is displayed in Algorithm 1. l − ⊙ h l − 1) i )
∥ h l − 1) i |∀ i ∈ N j N j
end for do
end 13 end 14 ˆ y t ←
A.2 Discussion
A.2.1 Complexity Analysis
Let K indicate the number of retrieved data instances, F indicate the number of feature fields, N bethe number of features, and d be the embedding size.
For the retrieval stage, as discussed in RIM, the average length of the posting lists in the inverted indexis | SearchPool | N .
The total time complexity of retrieval is O ( F | SearchPool | N ) .
As for constructing theinverted index, it could be done offline and only once.
The complexity of it is O
( F | SearchPool | ) .
The main computation of PET lies in the message passing part.
The number of nodes in the graph fora data instance is no more than ( K + 1)( F + 1) , and the number of edges is 2( K + 1) F .
Both ofthem are O ( KF ) .
The time complexity of single-layer message passing of PET is O ( KFd 2 ) .
Thismakes PET scalable as the complexity is linear with respect to the number of feature fields (numberof table columns).
In addition, the graph construction and message passing can be highly parallel: 1)
The graph construction for each data instance can be done in parallel for a batch data.
We batch allthe resulting graphs for the data instances in the batch to form one single batched graph.
Then we canapply GNN on the resulting single batched graph to make predictions parallelly.
For the GNN, theoperation of the attention computation can be parallelized across all edges, and the computation ofrepresentations can be parallelized across all nodes and edges.
For the MLP computation, PET inputs the target node embedding for prediction.
The complexity ofit is O ( d 2 ) , which is independent of the number of feature fields.
In addition, the number of modelparameters of PET is light.
Both other methods (e.g., DeepFM, RIM, etc.) and PET have embeddingtables of the same size Nd .
The main differences lie in the linear layers of models.
And for PET, thenumber of parameters of linear layers are O ( d 2 ) , which is independent of the number of fields.
A.2.2 Future Work
When dealing with datasets with a large number of fields, the constructed graph may be dense.
Inthis way, we could potentially meet the oversmoothing problem, which may degrade the modelperformance.
However, this problem can still be overcome by some methods.
For example, we couldretrieve and construct the graph over selected fields (e.g., top 10 fields that have the largest mutualinformation with the labels).
A.3 Visualization Results on More Datasets
In Section 4.3.2, we have presented the t-SNE visualization of the embeddings for Tmall dataset.
Here we provide the visualization results for Taobao and Alipay in Figures 5 and 6.
Similarly, we randomly sample 10,000 data instances from the train data and 10,000 data instancesfrom the test data.
Then we visualize the data instance node embeddings (output embeddings) and the feature embeddings with t-SNE.
The embeddings of positive data samples are visualized in green,while those of negative data samples are visualized in purple.
From the visualization results, we can see that PET yields more informative embeddings, i.e., therepresentation of tabular data, that separate positive data and negative data better.
A.4 Impact of Different Retrieval Sizes
We further study the impact of different retrieval sizes.
Results are displayed in Figure 7.
[Figure]
From the results, we can see that the optimal retrieval sizes are similar for different datasets.
Generally,more retrieved instances can contain more auxiliary information and give better results, but too manyretrieved instances may introduce noises.
A.5 Error Bars
Due to the page limit, we report the standard errors for PET and the second best model here.
Theresults are summarized in Tables 6 and 7.
[Table]
[Table]
From the results
[Table]
we can see that the performance of PET is stable.
In addition, PET consistentlyoutperforms other baselines.
A.6 Experiment Settings
In this section, we offer the detailed hyperparameters settings to reproduce the results.
The hyperparameters of PET for each dataset are summarized in Table 8.
[Table]
The hyperparameters of PET are inherited from RIM.
For the baseline tuning, we follow previouswork.
Concretely, we use the optimal hyperparameters reported by RIM (Qin et al., 2021) for thenon-graph methods.
For the graph methods, the learning rate is selected from {1e-4, 3e-4, 5e-4, 1e-3},and l2 regularization is selected from {1e-4, 1e-5, 5e-5}.
The embedding sizes of all the models areconsistent to ensure the fair comparison.
We use ElasticSearch 8 to retrieve the relevant data instances.
The model is implemented based on
DGL 9 .
All the experiments are run on Tesla T4 instances.
The code is included in the supplementarymaterial.
https://www.
.ai/