Tabular Deep Learning when d ≫ n by Using an Auxiliary Knowledge Graph
Anonymous Author(s)
AffiliationAddress email
Abstract
Machine learning models exhibit strong performance on datasets with abundantlabeled samples.
However, for tabular datasets with extremely high d -dimensionalfeatures but limited n samples (i.e. d ≫ n ), machine learning models struggle toachieve strong performance.
Here, our key insight is that even in tabular datasetswith limited labeled data, input features often represent real-world entities aboutwhich there is abundant prior information which can be structured as an auxiliaryknowledge graph (KG).
For example, in a tabular medical dataset where every inputfeature is the amount of a gene in a patient’s tumor and the label is the patient’ssurvival, there is an auxiliary knowledge graph connecting gene names with drug,disease, and human anatomy nodes.
We therefore propose P LATO , a machinelearning model for tabular data with d ≫ n and an auxiliary KG with input featuresas nodes.
P LATO uses a modified multilayer perceptron (MLP) to predict the outputlabels from the tabular data and the auxiliary KG with two components.
First, P LATO predicts the parameters in the first layer of the MLP from the auxiliary
KG.
P LATO thereby reduces the number of trainable parameters in the MLP andintegrates auxiliary information about the input features.
Second, P LATO predictsdifferent parameters in the first layer of the MLP for every input sample, therebyincreasing the MLP’s representational capacity by allowing it to use different priorinformation for every input sample.
Across 10 state-of-the-art baselines and 6d ≫ n datasets, P LATO exceeds or matches the prior state-of-the-art, achievingperformance improvements of up to 10.19%.
Overall, P LATO uses an auxiliary KGabout input features to enable tabular deep learning prediction when d ≫ n .
Introduction
Machine learning models have reached state-of-the-art performance in domains with abundant labeleddata like computer vision [65, 9] and natural language processing [60, 11, 44].
However, for tabulardatasets in which the number d of features vastly exceeds the number n of samples, machine learningmodels struggle to achieve strong performance [23, 34].
Unfortunately, many high impact domainslike chemistry [21], biology [28, 68, 18, 17], and physics [29] produce datasets with high-dimensionalfeatures but limited labeled samples due to the high time and labor costs associated with experiments.
In chemistry, for example, mass spectrometry datasets can have tens of thousands of features but onlytens or hundreds of samples [21].
For these and other tabular datasets with d ≫ n , the performanceof machine learning systems is currently limited.
To date, deep learning approaches for tabular data have focused on data regimes with far moresamples than features ( n ≫ d ) [20, 19, 51].
In the low-data regime with far more features thansamples ( d ≫ n ), the dominant approaches are statistical methods [23] that reduce the dimensionality of the input space[1, 34, 57, 58], select features [55, 7, 15, 38], impose regularization penalties onparameter magnitudes [37], or use ensembles of weak tree-based models [16, 6, 30, 35, 43].
Here, we present a novel problem setting and framework for tabular deep learning when d ≫ n(Figure 1).
Our key insight is that even in tabular settings with limited labeled data, input featuresoften represent real-world entities about which there is abundant prior information which can bestructured as an auxiliary knowledge graph (KG).
We propose a novel problem setting in whichevery input feature of a tabular dataset corresponds to a node in an auxiliary KG (Figure 1a).
Forexample, consider a tabular medical dataset in which every row is a cancer patient, every column isa gene, every value is the amount of that gene in the patient’s tumor, and the task is to predict thepatient’s survival.
For this tabular dataset, there exists an auxiliary KG which consists of each gene’sfunction, the relationships between genes, how a gene affects a part of human anatomy, and howhuman anatomy itself is structured.
Note that the KG does not capture the relationships betweeninput data samples but instead captures the relationships between input features .
Within our novel problem setting, we propose P LATO , a deep learning method for tabular data withd ≫ n and an auxiliary KG with input features as nodes (Figure 1(b)-(e)).
P LATO uses a modifiedmultilayer perceptron (MLP) to predict the output labels from the input samples and the auxiliary KGwith two methodological components.
First, the parameters in the first layer of the MLP are predictedfrom the auxiliary KG and the input sample rather than learned from just the tabular data.
P LATOthereby integrates prior information about the input features from the auxiliary KG and drasticallyreduces the number of trainable parameters in the MLP.
Second, the parameters in the first layer ofthe MLP are predicted differently for every sample by using the auxiliary KG and the sample values.
P LATO thereby increases the representational capacity of the MLP and enables effective predictions.
We exhibit P LATO ’s performance on 6 datasets.
We choose biomedicine as it is a rich domain ford ≫ n in which we can construct a single knowledge graph to serve as a unified backbone formany distinct tabular datasets with distinct input features.
We compare P LATO to 10 state-of-the-artbaselines spanning dimensionality reduction, feature selection, classic statistical models, deep tabularlearning methods, and parameter-prediction methods.
Following a rigorous evaluation protocol fromthe tabular deep learning literature [20, 19], P LATO achieves or matches the prior state-of-the-arton all 6 datasets, achieving performance improvements of up to 10.19%.
Ablation studies furtherdemonstrate the necessity of each methodological component of P LATO .
Ultimately, P LATO uses anauxiliary KG about input features to enable tabular deep learning prediction when d ≫ n .
Related Work
Tabular deep learning methods.
In contrast to P LATO ’s setting, tabular deep learning methods havebeen developed for settings with far more samples than features ( i.e. n ≫ d ).
Recent tabular learningbenchmarks ignore datasets with high numbers of features and low numbers of samples [20, 19, 51].
In the n ≫ d setting, various categories of deep tabular models have been benchmarked.
We selectthe state-of-the-art models to compare against P LATO .
First, decision tree models like NODE [42]make decision trees differentiable to enable gradient-based optimization [24, 31, 69].
Second, tabulartransformer architectures use an attention mechanism to select and learn interactions among features.
These include TabNet [2], TabTransformer [27], FT-Transformer [19], and others [53, 52, 32].d ≫ n methods.
For P LATO ’s setting in which d ≫ n , various tabular machine learning approachesexist [23].
First, dimensionality reduction techniques like PCA [1] aim to reduce the dimensionalityof the input data while preserving as much of the variance in the data as possible [34, 57, 58].
Second, feature selection approaches select a parsimonious set of features, leading to a smaller featurespace.
Classical feature selection approaches include LASSO [55] and its variants [7, 15, 38].
Forfeature selection with deep learning, Stochastic Gates [66] are among the best performing of manyvariants [3, 36].
Finally, classical tree-based models like XGBoost learn ensembles of weak decisiontrees models to make an overall prediction [16, 6, 30, 43].
Knowledge graph approaches.
Existing knowledge graph approaches are designed for tasks directlyon the graph such as link prediction [61, 56, 62, 67, 12].
By contrast, P LATO does not make anypredictions on the knowledge graph.
Instead, P LATO makes predictions on a separate, tabular datasetby using the knowledge graph as prior information about the features and domain.
Graph classificationapproaches are similarly not relevant to P LATO ’s problem setting (Appendix C).
Parameter prediction.
Using one network to predict the parameters of another is extensivelystudied [10, 50, 5].
For example, Ha et al.
predict the weights in all layers of a sequential model ( i e.
RNN, LSTM) by using information about the structure of the weights[22].
Diet Networks [46] predictparameters by hand-crafting prior information about the input features or using random projections.
By contrast, P LATO predicts parameters in a network by leveraging prior information about the inputfeatures in an auxiliary KG.
P LATO systematically constructs a feature embedding which containsthe prior information about the feature that is most relevant to a given sample.
P LATO
P LATO is a deep learning method for tabular datasets with d ≫ n and an auxiliary KG with inputfeatures as nodes (Section 3.1).
The key insight of P LATO is that even in tabular datasets with limitedlabeled samples, input features often represent real world entities about which there is abundantprior information which can be structured as an auxiliary knowledge graph (KG) G (Figure 1(a)).
P LATO uses a modified MLP to predict the output labels from the input samples and the auxiliary KG.
P LATO ’s modified MLP has two key components.
First, the parameters in the first layer of the MLPare predicted from the auxiliary KG and the input sample rather than learned from just the tabulardata (Figure 1(b-e)).
P LATO thereby integrates prior information about the input features from theauxiliary KG and drastically reduces the number of trainable parameters in the MLP.
Second, theparameters in the first layer of the MLP are predicted differently for every sample.
P LATO therebyincreases the representational capacity of the MLP and enables effective predictions.
The P LATOmethod is described below and summarized as pseudocode in Appendix A.
3.1 Problem Setting
Consider a tabular dataset X ∈ R n × d with labels y ∈ R n and far more features than samples such thatd ≫ n .
The goal is to train a machine learning model F to predict labels ˆy from the input X .
P LATOassumes the existence of an auxiliary knowledge graph G =
( V, E ) with | V | nodes and | E | edges suchthat every input feature j corresponds to a node in G .
Formally, ∀ j ∈ { 1 , . . .
, d } , ∃ v ∈ V s.t.
j (cid:55)→ v , as shown in Figure 1(a).
G also contains additional nodes which represent broader knowledge aboutthe domain.
The edges in G are (head node, relation type, tail node) triplets.
3.2 The P LATO MLP F
Consider a standard MLP ˆy =
T ( X ; Θ ) with L layers, h hidden units in the first layer, and trainableparameters Θ =
{ Θ [1] , Θ [2] , . . .
, Θ [ L ] } .
The P LATO MLP F differs from T in two key ways.
First, the parameters in the first layer of P LATO ’s MLP F are predicted from prior informationrather than learned from the tabular data.
We observe that every parameter in the first layer of Tis associated with a input feature j .
In particular, Θ [1] ∈ R
d × h such that Θ [1]
j ∈ R
h is a vector ofparameters connecting input feature j to every hidden unit in the first layer of the MLP (Figure 1e).
We observe that if two input features j and k represent real-world entities that are related, then theircorresponding parameters Θ [1] j and Θ [1] k are likely to be related too.
Typically this intuition is missedsince T learns the parameters Θ [1] j and Θ [1] k associated with two features j and k independently bygradient backpropagation.
To capture the intuition that related input features j and k should haverelated parameters, P LATO ’s MLP F instead predicts ˆΘ [1] j and ˆΘ [1] k from prior information knownabout j and k .
How such parameter prediction occurs is left for Section 3.3.
For now, note that theparameters in the first layer of P LATO ’s MLP F are predicted such that ˆΘ [1] ∈ R d × h .
The parametersΘ [2] , . . .
, Θ [ L ] in the remaining layers of P LATO ’s
MLP F are learned normally.
Second, the parameters in the first layer of P LATO ’s MLP are allowed to vary for every inputsample.
In the standard MLP T , all parameters Θ [1] , . . .
, Θ [ L ] are the same for every input sample
X i .
In the first layer of P LATO ’s MLP F , however, the parameters ˆΘ [1] are being predicted fromprior information about the input features.
We observe that for each input sample X i , the mostrelevant prior information about each input feature j might differ.
Therefore for each sample X i ,
P LATO uses different prior information about each input feature j to predict the parameters ˆΘ [1] j .
Asa result, the parameters ˆΘ [1] in the first layer of F vary with each input sample X i , increasing therepresentational capacity of F .
How P LATO uses different prior information for parameter predictionis left to Section 3.3.
For now, note that P LATO predicts ˆΘ [1] j from prior information about feature j .
The prior information about feature j that is used depends on the input sample
X i .
As a result, ˆΘ [1]is conditional on X i according to ˆΘ [1] | X i .
Formal Notation.
Overall, P LATO ’s
MLP F takes the form
[Equation]
F has parameters ˆΘ = { ˆΘ [1] | X i } ∪ { Θ [2] , . . .
, Θ [ L ] } where L is the number of layers in F .
Theparameters ˆΘ [1] | X i in the first layer of F are predicted from the input sample X i via message-passingon the KG according to Section 3.3.
For every sample i , a new ˆ Θ [1] is predicted such that ˆ Θ [1]is conditional on X i at both training and inference time.
The dimensionality of ˆΘ [1] ∈ R d × h isthe same as in a normal MLP where h is the number of hidden units in the first layer of F .
Theparameters in the remaining layers of P LATO ’s MLP Θ [2] , . .
, Θ [ L ] are the same as in a standard
MLP: they are learned normally, are the same for every sample at both training and inference time,and are thus not conditional on X i .
3.3 Predicting the parameters in the first layer of P LATO ’s MLP F
P LATO uses prior information about the input features to predict the parameters in the first layer of
P LATO ’s MLP F .
P LATO predicts these parameters in three steps.
First, P LATO uses self-supervisionon the auxiliary KG to pretrain an embedding for every input feature (Section 3.3.1, Figure 1(b)).
Second, since different input samples might rely on different prior information about each inputfeature, P LATO updates each feature embedding to contain the most relevant prior information aboutthe input feature for the given input sample (Section 3.3.2, Figure 1(c)).
Finally, P LATO predicts theparameters in the first layer of F from the updated feature embeddings with a small neural networkthat is shared across input features (Section 3.3.3, Figure 1(d)(e)).
3.3.1 Pretraining feature embeddings with self-supervision
First, P LATO learns general prior information about each input feature j from the auxiliary KG
G (Figure 1b).
P LATO represents the general prior information about each input feature j as alow-dimensional embedding M j ∈ R c .
Since every input feature j corresponds to a specific node in
G , P LATO can learn M j by learning an embedding for the corresponding feature node in G .
Anyself-supervised node embedding method on G can be used within P LATO ’s framework.
Formal notation.
Formally, P LATO uses self-supervision on G to pretrain an embedding for everyinput feature according to
M = H ( G ) .
M ∈ R d × c is the matrix of all feature embeddings.
H is a self-supervised node embedding method.
We refer to Eq.
(2) as pretraining since only the auxiliary KG G is used but the tabular data X , y isignored.
After pretraining, the feature embeddings M are fixed.
Implementation.
For H , we choose ComplEx, a KG node embedding method [56].
ComplEx uses aself-supervised objective which learn an embedding for every node in G by classifying whether aproposed edge exists in G .
ComplEx’s proposed edges include both feature nodes and other nodes in
G , thereby integrating prior information about the input features and the broader domain.
3.3.2 Updating feature embeddings to contain the most relevant information for an input
sample
Since different input samples might rely on different prior information about each input feature,
P LATO next updates each feature embedding M j ∈ R d × c to Q j ∈ R d × c , a feature embedding whichcontain the most relevant prior information about feature j for a given input sample X
i (Figure 1(c)).
P LATO uses a message-passing network Q on the KG to update the feature embeddings in a way thatminimizes the number of additional trainable parameters.
[Equation]
Q uses an attention mechanism which considers the sample values X i to update the feature embeddings.
The attention mechanism has a small number of trainable parameters Π .
The message passing network Q .
Let Q [ r ] j be the embedding of input feature j after roundr ∈ { 1 , ..., R } of message passing.
For every input feature j , Q first initializes the updated featureembedding to the pretrained feature embedding.
[Equation]
Q then conducts R rounds of message passing.
In each round of message passing, the featureembedding Q [ r ] j is updated from the feature embedding of each neighbor k in the prior round Q [ r − 1] kand its own feature embedding in the prior round Q [ r − 1] j .
The “message” being passed is theembedding of each feature from the prior round.
[Equation]
σ is an optional nonlinearity.
N j are the neighbors of feature node j in G .
During message-passing, Q uses two scalar values β ∈ R and A ijk ∈ R to control the weights ofmessages.
First, Q uses hyperparameter β to control the weight of the messages aggregated from allneighbors vs. the message from the feature node j itself.
Second, Q calculates an attention score
A ijk to control the weight of the specific message between feature j and neighbor k .
The attentionscore is different for every sample i and is calculated by a shallow neural network A with a smallnumber of trainable parameters Π .
The attention score A ijk thus enables Q to update the informationin the feature embedding in a way that is most relevant for the input sample i .
Formally:
[Equation]
[Table]
The number of trainable parameters in Π is small since the input of A is R 2 and the output of A is ascalar R .
A and its parameters Π are shared for all samples and features.
Finally, the updated feature embeddings Q j are set after R rounds of message-passing.
[Equation]
3.3.3 Predicting the first layer of parameters in F from the updated feature embeddings
Finally, P LATO predicts the parameters in the first layer of F from each updated feature embedding(Figure 1(d)(e)).
Every parameter in the first layer of F is associated with a feature j .
P LATO thuspredicts ˆΘ j , the parameters associated with the feature j , from Q j , the prior information about j .
Formal notation.
P LATO predicts the parameters associated with every input feature j in the firstlayer of F according to
[Equation]
P is a shallow neural network parameterized by Φ .
Q j is the updated feature embedding of j whichis conditional on the specific input sample X i .
P and its parameters Φ are shared for every featurej ∈ { 1 , . . .
, d } .
P LATO drastically reduces the number of trainable parameters compared to a standard MLP.
The sharing of P and Φ across all input features enables a drastic reduction in the number of trainableparameters compared to a standard MLP.
For a high-dimensional tabular dataset ( i.e. d ≫ n ),a standard MLP T with h hidden units has a large number of trainable parameters in the firstlayer since Θ [1] ∈ R d × h .
A standard MLP T must learn all dh of these trainable parametersindependently.
By contrast, P uses a shared set of trainable parameters Φ to predict ˆΘ j from Q jfor every j ∈ { 1 , . . .
, d } .
The number of trainable parameters in Φ is small compared to dh since Pneed only transform every Q j ∈ R c to ˆΘ [ 1 ] ∈ R h .
Thus, | Φ | = ch (assuming that P is a single layerneural network).
c , the dimensionality of the feature embedding, is much less than d the number ofinput features.
As a result, | Φ | = ch ≪ dh and P LATO drastically reduces the number of trainableparameters in the first layer of a MLP.
Experiments
Datasets and Auxiliary Knowledge Graph.
We evaluate P LATO on 6 tabular datasets in biomedicinewith d ≫ n and 4 with d ∼ n [17, 18, 28, 68].
We construct a general biomedical KG from priorstudies [13, 14, 33, 48, 54, 63, 64].
Our KG contains 108,447 nodes, 3,066,156 edges, and 99 relationtypes.
The same KG is used across all datasets even though all datasets have distinct feature sets withdistinct cardinalities.
P LATO thus allows a single KG to serve as a unified knowledge backbone fordifferent datasets in a domain.
Additional details are in Appendix F, G.
[Table]
[Table] [Table]
Baselines.
We compare P LATO to 10 state-of-the art statistical and deep baselines.
We considerregularization with Ridge Regression [37], dimensionality reduction with PCA [1], feature selectionwith LASSO [55] deep feature selection with Stochastic Gates [66], and gradient boosted decisiontrees with XGBoost [6].
We also consider deep tabular methods including a standard MLP, selfattention-based methods with TabTransformer [27] and TabNet [2], differentiable decision trees with
NODE [42], and parameter-prediction with Diet Networks [46].
We also tried FT-Transformer [19],but it experienced out of memory issues on all datasets due to the large number of features.
Fair Comparison of P LATO with Baselines.
To ensure a fair comparison with baselines, wefollow evaluation protocols in tabular benchmarks [20, 19].
We conduct a random search with500 configurations of every model (including P LATO ) on every dataset across a broad range ofhyperparameters (Appendix B).
We split data with a 60/20/20 training, validation, test split.
Allresults are computed across 3 data splits and 3 runs of each model in each data split.
We report themean and standard deviation of the Pearson correlation (PearsonR) between y and ˆy across runs andsplits on the test set.
Each model is run on a GeForce RTX 2080 TI GPU.
4.1 Results
P LATO outperforms statistical and deep baselines when d ≫ n .
P LATO outperforms all baselinesacross all 6 datasets with d ≫ n (Table 1).
P LATO achieves the largest improvement on the PDACdataset, improving by 10.19% vs. XGBoost, the best baseline for PDAC (0.400 vs. 0.363).
While
P LATO achieves the strongest performance across all 6 datasets, the best performing baseline varieswith different datasets.
Ridge Regression is the strongest baseline for BRCA, LASSO for CM and
CRC, XGBoost for PDAC and CH, and TabTransformer for MNSCLC.
The remaining baselines(PCA, STG, Diet Networks, MLP, NODE, and TabNet) are not the strongest baseline for any dataset.
We also find that the performance of a specific baseline varies with the dataset.
TabTransformer, forexample, is the best baseline for MNSCLC but the worst for CH (model ranking in Appendix D).
P LATO ’s performance depends on updating feature embeddings to contain information relevantto a sample.
P LATO predicts the parameters ˆΘ [ 1 ] in the first layer of a modified MLP F by usingfeature embeddings which contain prior information about the input features.
P LATO first pretrains general feature embeddings M ∈ R d × c .
P LATO then updates the feature embeddings to Q ∈ R d × cwhich contains information about the input features that is most relevant to a given sample X i .
Wetest whether updating the feature embeddings based on a given X i is necessary by evaluating P LATOon the BRCA dataset in three configurations (Table 2).
The default configuration uses the updatedfeature embeddings Q to predict ˆΘ [1] according to ˆΘ [ 1 ]
= P ( Q | X i ) .
The second configuration usesthe general feature embeddings M instead of Q to predict ˆΘ [1] according to ˆΘ [ 1 ] = P ( M ) .
Thethird configuration does not use feature embeddings and thus reduces to a standard MLP.
We compare P LATO ’s performance when it uses feature embeddings which contain the relevantinformation for a given sample X i ( i.e. ˆΘ [1] =
P ( Q | X i ) )
vs. the pretrained feature embeddings Mwhich contain general information about the input features that is not specific to a given sample ( i.e.ˆΘ [1] =
P ( M ) ).
Using general feature embeddings M improves over not using feature embeddingsat all (0.522 vs. 0.240).
Using feature embeddings Q that are specific to a given input sample furtherimproves performance (0.583 vs. 0.522).
Therefore, updating the feature embeddings to Q such thatthey contain the information specific to a given sample is critical to P LATO ’s performance.
P LATO ’s performance depends on both feature nodes and broader knowledge nodes in theauxiliary KG.
P LATO relies on an auxiliary KG G which contains information about input featuresand information about the broader domain.
Information about input features is represented as featurenodes while information about the broader domain is represented as other nodes in G (Methods 3.1).
To test the relative importance of the feature information in G vs. the broader domain information, wemeasured the performance of P LATO on the BRCA dataset in two KG configurations: P LATO with thefull KG ( i.e. both the feature nodes and the broader domain nodes) and P LATO with a “feature-only
KG” ( i.e. an induced subgraph on only the feature nodes) (Table 3).
We also compare to a “No KG”configuration in which P LATO does not have access to the KG.
Without auxiliary information aboutthe input features or the broader domain, P LATO is ablated to become a standard MLP.
We find that both the feature nodes and the broader knowledge nodes are important for P LATO ’sperformance.
Using the “feature-only KG” configuration of P LATO improves performance vs the “no
KG” configuration of P LATO (0.539 vs 0.240).
Using the “full KG” configuration further improvesperformance vs the “feature-only KG” configuration (0.583 vs 0.539).
P LATO ’s performance thusrelies on both the feature information and the broader domain information in the KG.
For datasets with d ∼ n , P LATO is competitive with baselines.
Finally, we test P LATO ’s performance for datasets with d ∼ n .
We test 4 datasets with d ∼ n ranging from dn = 1 . 1 to dn = 2 .
0(Table 4).
We find that on 4 datasets with d ∼ n , P LATO is competitive with the best performingbaseline, XGBoost, but does not improve performance substantially.
P LATO ’s stronger performancefor datasets with d ≫ n than for datasets with d ∼ n is justified.
P LATO ’s key idea is to includeauxiliary information about the input features.
Auxiliary information is likely to help performancethe most in settings with the least labeled data ( i.e. d ≫ n ).
When d ∼ n , auxiliary information isless helpful since the tabular dataset may already have enough information to train a strong predictivemodel.
We further find that XGBoost is consistently the strongest baseline for datasets with d ∼ n , incontrast to the varied performance of XGBoost on the datasets with d ≫ n (Table 1).
Discussion
P LATO is a deep learning model for tabular data with d ≫ n and an auxiliary KG with input featuresas nodes.
Across 6 datasets and 10 baselines, P LATO achieves state-of-the-art performance, includingperformance improvements of up to 10 .
Ablation studies confirm the importance of eachcomponent of P LATO .
P LATO has several limitations.
First, P LATO matches but does not improvethe performance of baselines for high-dimensional datasets with more samples ( i.e. d ∼ n ).
Second,
P LATO relies on the coverage of prior information.
Datasets with input features that have less priorinformation in the KG are less likely to benefit from P LATO .
Overall, P LATO uses an auxiliary KGabout input features to enable tabular deep learning when d ≫ n .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
Code and data will bereleased with the final version of the manuscript.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 under “Fair Comparison of P LATO with Baselines”and Appendix B.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See figures and tables.(d)
Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4 under “FairComparison of P LATO with Baselines” header.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] All datasets andbaseline models are cited.(b)
Did you mention the license of the assets?
[Yes] All licenses are present in the citedwork.(c)
Did you include any new assets either in the supplemental material or
as a URL?
Codeand data will be released with the final version of the manuscript.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you
used
crowdsourcing or conducted research with human
subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A The PLATO Algorithm
Algorithm 1: The P LATO Algorithm.
t ∈ Nj exp( where A is a NN parameterized by Π
[Equation]
(cid:80)
[Equation]
k
[Equation]
B Evaluation Protocol and Hyperparameter Ranges
To ensure a fair comparison with baselines, we follow evaluation protocols outlined in tabularbenchmarks [20, 19].
We conduct a random search with 500 configurations of every model (including
P LATO ) on every dataset across a broad range of hyperparameters.
We base the hyperparameterranges on the ranges used in prior tabular learning benchmarks [20, 19] and the ranges mentioned inthe original papers of the methods.
Hyperparameter ranges for P LATO are given in Supplementary
Table 1.
Hyperparameter ranges for baseline methods are given in Supplementary Table 2.
[Table]
[Table]
C Graph classification approaches
Graph classification models are not relevant for P LATO ’s setting.
In graph classification models,every input sample is a graph with node attributes, and a model must make a prediction for thatgraph.
The P LATO problem setting breaks fundamental assumptions made by graph classificationmodels, rendering them not applicable.
First, graph classification models assume that differentsamples correspond to different graphs [70, 26, 25].
However, in P LATO every sample correspondsto the exact same graph.
There is a single background knowledge graph for all samples, and everysample has input features that correspond to the exact same nodes within the knowledge graph.
Second, graph classification approaches typically assume that every node in an input graph has anode attribute [70, 26, 25].
However, in P LATO only a small subset of the nodes in the knowledgegraph have measured feature values.
Finally, graph classification approaches typically assume smallgraphs: the largest graph classification task in the Open Graph Benchmark has only 244 nodes [25].
However in P LATO , the knowledge graph contains 108,447 and the smallest dataset has 12,932features corresponding to nodes.
D Rank ordering of methods for datasets with d ≫ n
In Supplementary Table 3, we show the rank order performance of all models on all d ≫ n datasets.
We find that P LATO exhibits consistent and strong performance while the performance of the baselinesdepends on the specific d ≫ n dataset.
For example, TabTransformer is the second best performingof all models on the MNSCLC dataset but the worst performing of all models on the PDAC and CHdatasets.
Similarly, XGBoost is the second best performing of all models on PDAC but only theseventh best performing of all models on BRCA.
The baselines with the most stable performanceare LASSO and Ridge Regression which rank consistently between the second and fifth best of allmodels.
[Table] [Table]
E Code Details
Code to run PLATO will be included as a supplementary file in the final version of the manuscript.
F Dataset Details
We compiled 6 datasets with d ≫ n and 4 datasets with d ∼ n .
In all datasets, a machine learningmodel must predict the response of a cell or mouse to a drug.
In the tabular data, every rowcorresponds to a specific cell or mouse.
Every column corresponds to a gene name.
Every valuecorresponds to the amount of that gene in the tumor of the specific cell or mouse.
The label is theresponse of the cell or mouse.
All genes are nodes in the knowledge graph.
In practice, the numberof genes is large for all tasks and the number of samples is comparatively small making the drugresponse setting appropriate for d ≫ n .
Dataset statistics, names, and sources are given below.
Dataset pre-processing followed a standardprocess described in [40].
Briefly, gene expression values underwent TMM normalization and logtransformation ( i.e. log( x +1) ).
Values were made to have zero mean and unit standard deviation.
Aslabels, we used ln-ic50 for datasets from [28, 68, 18] and the minimum average percent tumor growth( i.e. “min-avg-pct-tumor-growth”) for datasets from [17].
For all datasets, we use a 200-dimensional
ComplEx [56] embedding of the drug as the input feature vector.
All datasets will be released in thefinal version of the manuscript.
[Table]
G Knowledge Graph Details
As a unified knowledge backbone for the datasets, we compile a general biomedical knowledge graphfrom prior studies [13, 14, 33, 48, 54, 63, 64].
A schematic of the KG is in Supplementary Figure 2.
A detailed breakdown of relation types is in Supplementary Table 5.
Our knowledge graph contains 108,447 total nodes, including 7,975 drugs, 18,370 diseases, 11,447phenotypes, 22,319 genes, 11,153 molecular functions, 28,748 biological processes, and 4,184cellular components.
Our knowledge graph contains 3,066,156 edges with 99 distinct relation types.
All datasets include features which map to a subset of nodes in the knowledge graph, primarily genesand drugs.
The remaining node types and their relationships serve as broader domain knowledge.
Edges between drug nodes and gene/protein nodes were derived from Drugbank [64], Gao [17], andthe Genomics of Drug Sensitivity in Cancer [68, 28, 18].
Edges between diseases and genes/proteinswere derived from DisGeNet [4].
Edges between diseases and phenotypes were derived fromthe Human Phenotype Ontology [14].
Edges between drugs and diseases were derived from the
Multiscale Interactome [48].
Edges between drugs and side effects were derived from SIDER [33].
Edges between genes/proteins and other genes/proteins were derived from BioGRID [41], [47], the
Database of Interacting Proteins [49], [13], [39], [45], [71], [59], and STRING [54].
Finally, edgesfrom genes/proteins to molecular functions, biological processes, and cellular components as well asedges between molecular functions, biological processes, and cellular components were derived fromthe Gene Ontology [8].
The full knowledge graph will be included as a supplementary file in the finalversion of the manuscript.
[Table]
type
Relation
Tail type # edges
BiologicalProcess
EndsDuring
BiologicalProcess BiologicalProcess
HappensDuring
BiologicalProcess BiologicalProcess
HasPart
BiologicalProcess BiologicalProcess
IsA
BiologicalProcess BiologicalProcess
NegativelyRegulates
BiologicalProcess BiologicalProcess
PartOf
BiologicalProcess BiologicalProcess
PositivelyRegulates
BiologicalProcess BiologicalProcess
Regulates
BiologicalProcess BiologicalProcess
OccursIn
CellularComponent
BiologicalProcess
HasPart
MolecularFunction
BiologicalProcess
NegativelyRegulates
MolecularFunction
BiologicalProcess
PositivelyRegulates
MolecularFunction
BiologicalProcess
Regulates
MolecularFunction
CellularComponent
HasPart
CellularComponent CellularComponent
IsA
CellularComponent CellularComponent
PartOf
CellularComponent
Disease
AlteredExpression
Gene
Disease
Biomarker
Gene
Disease
ChromosomalRearrangement
Gene
Disease
FusionGene
Gene
Disease
GeneticVariation
Gene
Disease
GermlineCausalMutation
Gene
Disease
ModifyingMutation
Gene
Disease
SomaticCausalMutation
Gene
Disease
SusceptibilityMutation
Gene
Disease
Therapeutic
Gene
Disease
Has
Phenotype
Drug
Treats
Disease
Drug
Carries
Gene
Drug
Enzymes
Gene
Drug
Targets
Gene
Drug
Transports
Gene
Drug
Has
Phenotype
Gene
Associates
BiologicalProcess
Gene
NotAssociates
BiologicalProcess
Gene
Associates
CellularComponent
Gene
Colocalizes
CellularComponent
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Gene
DirectInteraction
Gene Gene
DisulfideBond
Gene Gene
DosageGrowthDefect
Gene Gene
DosageLethality
Gene Gene
DosageRescue
Gene Gene
Enzymatic
Gene Gene
Expression
Gene Gene
GeneticInterference
Gene Gene
Hydroxylation
Gene Gene
Inhibition
Gene Gene
Kinase
Gene Gene
Literature
Gene Gene
Metabolic
Gene Gene
Methylation
Gene Gene
NegativeGenetic
Gene Gene
OxidoreductaseActivityElectronTransferAssay Gene
Gene
PhenotypicEnhancement
Gene Gene
PhenotypicSuppression
Gene Gene
Phosphorylation
Gene Gene
Phosphotransfer
Gene Gene
PhysicalAssociation
Gene Gene
PositiveGenetic
Gene Gene
PostTranslationalModification
Gene Gene
ProteinCleavage
Gene Gene
PutativeSelfInteraction
Gene Gene
Reaction
Gene Gene
Regulation
Gene Gene
Signaling
Gene Gene
SyntheticGrowthDefect
Gene Gene
SyntheticLethality
Gene Gene
SyntheticRescue
Gene Gene
Associates
MolecularFunction
Gene
Contributes
MolecularFunction
Gene
NotAssociates
MolecularFunction
Gene
NotContributes
MolecularFunction MolecularFunction
PartOf
BiologicalProcess
MolecularFunction
Regulates
BiologicalProcess
MolecularFunction
OccursIn
CellularComponent
MolecularFunction
HasPart
MolecularFunction MolecularFunction
IsA
MolecularFunction MolecularFunction
NegativelyRegulates
MolecularFunction MolecularFunction
PartOf
MolecularFunction MolecularFunction
PositivelyRegulates
[Table]