Tabular Deep Learning when d ≫ n by Using an Auxiliary Knowledge Graph
Abstract
Machine learning models exhibit strong performance on datasets with abundant labeled samples.
However, for tabular datasets with extremely high d -dimensional features but limited n samples ( i.e. d ≫ n ), machine learning models struggle to achieve strong performance.
Here, our key insight is that even in tabular datasets with limited labeled data, input features often represent real-world entities about which there is abundant prior information which can be structured as an auxiliary knowledge graph (KG).
For example, in a tabular medical dataset where every input feature is the amount of a gene in a patient’s tumor and the label is the patient’s survival, there is an auxiliary knowledge graph connecting gene names with drug, disease, and human anatomy nodes.
We therefore propose P LATO , a machine learning model for tabular data with d ≫ n and an auxiliary KG with input features as nodes.
P LATO uses a modified multilayer perceptron (MLP) to predict the output labels from the tabular data and the auxiliary KG with two components.
First,
P LATO predicts the parameters in the first layer of the MLP from the auxiliary
KG.
P LATO thereby reduces the number of trainable parameters in the MLP and integrates auxiliary information about the input features.
Second, P LATO predicts different parameters in the first layer of the MLP for every input sample, thereby increasing the MLP’s representational capacity by allowing it to use different prior information for every input sample.
Across 10 state-of-the-art baselines andd ≫ n datasets, P LATO exceeds or matches the prior state-of-the-art, achieving performance improvements of up to 10.19%.
Overall, P LATO uses an auxiliary KG about input features to enable tabular deep learning prediction when d ≫ n .
Introduction
Machine learning models have reached state-of-the-art performance in domains with abundant labeled data like computer vision [1, 2] and language [3, 4, 5].
However, for tabular datasets in which the number d of features vastly exceeds the number n of samples, machine learning models struggle to achieve strong performance [6, 7].
Unfortunately, many high impact domains like chemistry [8], biology [9, 10, 11, 12], and physics [13] produce datasets with high-dimensional features but limited labeled samples due to the high time and labor costs of experiments.
In chemistry, for example, mass spectrometry datasets can have tens of thousands of features but only hundreds of samples [8].
For these tabular datasets with d ≫ n , the performance of machine learning systems is currently limited.
To date, deep learning approaches for tabular data have focused on data regimes with far more samples than features ( n ≫ d ) [14, 15, 16].
In the low-data regime with far more features than samples ( d ≫ n ), the dominant approaches are statistical methods [6] that reduce the dimensionality of the input space [17, 7, 18, 19], select features [20, 21, 22, 23], impose regularization penalties on parameter magnitudes [24], or use ensembles of weak tree-based models [25, 26, 27, 28, 29].
Here, we present a novel problem setting and framework for tabular deep learning when d ≫ n (Figure 1).
Our key insight is that even in tabular settings with limited labeled data, input features often represent real-world entities about which there is abundant prior information which can be structured as an auxiliary knowledge graph (KG).
We propose a novel problem setting in which every input feature of a tabular dataset corresponds to a node in an auxiliary KG (Figure 1a).
For example, consider a tabular medical dataset in which every row is a cancer patient, every column is a gene, every value is the amount of that gene in the patient’s tumor, and the task is to predict the patient’s survival.
For this tabular dataset, there exists an auxiliary KG which consists of each gene’s function, the relationships between genes, how a gene affects a part of human anatomy, and how human anatomy itself is structured.
Note that the KG does not capture the relationships between input data samples but instead captures the relationships between input features .
Within our novel problem setting, we propose P LATO , a deep learning method for tabular datasets with d ≫ n and an auxiliary KG with input features as nodes (Figure 1(b)-(e)).
P LATO uses a modified multilayer perceptron (MLP) to predict the output labels from the input samples and the auxiliary KG with two components.
First, the parameters in the first layer of the MLP are predicted from the auxiliary KG and the input sample rather than learned from just the tabular data.
P LATO thereby integrates prior information about the input features from the auxiliary KG and drastically reduces the number of trainable parameters in the MLP.
Second, the parameters in the first layer of the MLP are predicted differently for every sample by using the auxiliary KG and the sample values.
P LATO thereby increases the representational capacity of the MLP and enables effective predictions.
We exhibit P LATO ’s performance on 6 datasets.
We choose biomedicine as it is a rich domain for d ≫ n in which we can construct a single knowledge graph to serve as a unified backbone for many distinct tabular datasets with distinct input features.
We compare P LATO to 10 state-of-the-art baselines spanning dimensionality reduction, feature selection, classic statistical models, deep tabular learning methods, and parameter-prediction methods.
Following a rigorous evaluation protocol from the tabular deep learning literature [14, 15], P LATO achieves or matches the prior state-of-the-art on all 6 datasets, achieving performance improvements of up to 10.19%.
Ablation studies further demonstrate the necessity of each methodological component of P LATO .
Ultimately, P LATO uses an auxiliary KG about input features to enable tabular deep learning prediction when d ≫ n .
Related Work
Tabular deep learning methods.
In contrast to P LATO ’s setting, tabular deep learning methods have been developed for settings with far more samples than features ( i.e. n ≫ d ).
Indeed, recent tabular benchmarks ignore datasets with high numbers of features and low numbers of samples [14, 15, 16].
In the n ≫ d setting, various categories of deep tabular models have been benchmarked.
We select the state-of-the-art models to compare against P LATO .
First, decision tree models like NODE [30] make decision trees differentiable to enable gradient-based optimization [31, 32, 33].
Second, tabular transformer architectures use an attention mechanism to select and learn interactions among features.
These include TabNet [34], TabTransformer [35], FT-Transformer [15], and others [36, 37, 38].
d ≫ n methods.
For P LATO ’s setting of d ≫ n , various tabular learning approaches exist [6].
First, dimensionality reduction methods like PCA [17] reduce the dimensionality of the input data while preserving as much of the data variance as possible [7, 18, 19].
Second, feature selection approaches select a parsimonious set of features, leading to a smaller feature space.
Classical feature selection approaches include LASSO [20] and its variants [21, 22, 23].
For feature selection with deep learning,
Stochastic Gates [39] are among the best performing of variants [40, 41].
Finally, tree-based models like XGBoost learn ensembles of weak decision trees to make an overall prediction [25, 26, 27, 29].
Knowledge graph approaches.
Existing knowledge graph approaches are designed for tasks directly on the graph such as link prediction [42, 43, 44, 45, 46].
By contrast, P LATO does not make any predictions on the knowledge graph.
Instead, P LATO makes predictions on a separate, tabular dataset by using the knowledge graph as prior information about the features and domain.
Graph classification approaches are similarly not applicable to P LATO ’s problem setting (Appendix C).
Parameter prediction.
Using one network to predict the parameters of another is well studied [47, 48, 49].
For example, hypernetworks predict the weights in all layers of a sequential model ( i.e. RNN,
LSTM) by using information about the structure of the weights [50].
Diet Networks [51] predict parameters by hand-crafting prior information about the features or using random projections.
By contrast, P LATO predicts parameters in a network by leveraging prior information about the input features in an auxiliary KG.
P LATO systematically constructs a feature embedding which contains the prior information about the feature that is most relevant to a given sample.
P LATO
P LATO is a deep learning method for tabular datasets with d ≫ n and an auxiliary KG with input features as nodes (Section 3.1).
Our key insight is that even in tabular datasets with limited labeled samples, input features often represent real world entities with abundant prior information that can be structured as an auxiliary knowledge graph (KG) (Figure 1(a)).
P LATO uses a modified MLP to predict the output labels from the input samples and the KG.
P LATO ’s modified MLP has two parts.
First, the parameters in the MLP’s first layer are predicted from the KG and the input sample rather than learned from just the tabular data (Figure 1(b-e)).
P LATO thus integrates prior information about the input features from the KG and drastically reduces the number of trainable parameters in the MLP.
Second, the parameters in the MLP’s first layer are predicted differently for every sample.
P LATO thus increases the MLP’s representational capacity.
3.1 Problem Setting
Consider a tabular dataset X ∈ R n × d with labels y ∈ R n and far more features than samples such that d ≫ n .
The goal is to train a machine learning model F to predict labels ˆy from the input X .
P LATO assumes the existence of an auxiliary knowledge graph G =
( V, E ) with | V | nodes and | E | edges such that every input feature j corresponds to a node in G .
Formally, ∀ j ∈ { 1 , . . .
, d } , ∃ v ∈ V s.t.
j (cid:55)→ v , as shown in Figure 1(a).
G also contains additional nodes which represent broader knowledge about the domain.
The edges in G are (head node, relation type, tail node) triplets.
3.2 The P LATO MLP F
Consider a standard MLP ˆy =
T ( X ; Θ ) with L layers, h hidden units in the first layer, and trainable parameters
Θ = { Θ [1] , Θ [2] , . .
, Θ [ L ] } .
The P LATO MLP F differs from T in two key ways.
First, the parameters in the first layer of P LATO ’s MLP F are predicted from prior information rather than learned from the tabular data.
We observe that every parameter in the first layer of T is associated with a input feature j .
In particular, Θ [1] ∈ R
d × h such that Θ [1]
j ∈ R
h is a vector of parameters connecting input feature j to every hidden unit in the first layer of the MLP (Figure 1e).
We observe that if two input features j and k represent real-world entities that are related, then their corresponding parameters Θ [1] j and Θ [1] k are likely to be related too.
Typically this intuition is missed since T learns the parameters Θ [1] j and Θ [1] k associated with two features j and k independently by gradient backpropagation.
To capture the intuition that related input features j and k should have related parameters, P LATO ’s MLP F instead predicts ˆΘ [1] j and ˆΘ [1] k from prior information known about j and k .
How such parameter prediction occurs is left for Section 3.3.
For now, note that the parameters in the first layer of P LATO ’s MLP F are predicted such that ˆΘ [1] ∈ R d × h .
The parameters Θ [2] , . . .
, Θ [ L ] in the remaining layers of P LATO ’s
MLP F are learned normally.
Second, the parameters in the first layer of P LATO ’s MLP are allowed to vary for every input sample.
In the standard MLP T , all parameters Θ [1] , . . .
, Θ [ L ] are the same for every input sample
X i .
In the first layer of P LATO ’s MLP F , however, the parameters ˆΘ [1] are being predicted from prior information about the input features.
We observe that for each input sample X i , the most relevant prior information about each input feature j might differ.
Therefore for each sample X i ,
P LATO uses different prior information about each input feature j to predict the parameters ˆΘ [1] j .
As a result, the parameters ˆΘ [1] in the first layer of F vary with each input sample X i , increasing the representational capacity of F .
How P LATO uses different prior information for parameter prediction is left to Section 3.3.
For now, note that P LATO predicts ˆΘ [1] j from prior information about feature j .
The prior information about feature j that is used depends on the input sample
X i .
As a result, ˆΘ [1] is conditional on X i according to ˆΘ [1] | X i .
Formal Notation.
Overall, P LATO ’s
MLP F takes the form
[Equation]
F has parameters ˆΘ = { ˆΘ [1] | X i } ∪ { Θ [2] , . . .
, Θ [ L ] } where L is the number of layers in F .
The parameters ˆΘ [1] | X i in the first layer of F are predicted from the input sample X i via message-passing on the KG according to Section 3.3.
For every sample i , a new ˆΘ [1] is predicted such that ˆΘ [1] is conditional on X i at both training and inference time.
The dimensionality of ˆΘ [1] ∈ R d × h is the same as in a normal MLP where h is the number of hidden units in the first layer of F .
The parameters in the remaining layers of P LATO ’s MLP Θ [2] , . .
, Θ [ L ] are the same as in a standard
MLP: they are learned normally, are the same for every sample, and are thus not conditional on X
i
. 3.3 Predicting the parameters in the first layer of P LATO ’s MLP F
P LATO uses prior information about the input features to predict the parameters in the first layer of
P LATO ’s MLP F .
P LATO predicts these parameters in three steps.
First, P LATO uses self-supervision on the auxiliary KG to pretrain an embedding for every input feature (Section 3.3.1, Figure 1(b)).
Second, since different input samples might rely on different prior information about each input feature, P LATO updates each feature embedding to contain the most relevant prior information about the input feature for the given input sample (Section 3.3.2, Figure 1(c)).
Finally, P LATO predicts the parameters in the first layer of F from the updated feature embeddings with a small neural network that is shared across input features (Section 3.3.3, Figure 1(d)(e)).
3.3.1 Pretraining feature embeddings with self-supervision
First, P LATO learns general prior information about each input feature j from the auxiliary KG
G (Figure 1b).
P LATO represents the general prior information about each input feature j as a low-dimensional embedding M j ∈ R c .
Since every input feature j corresponds to a specific node in
G , P LATO can learn M j by learning an embedding for the corresponding feature node in G .
Any self-supervised node embedding method on G can be used within P LATO ’s framework.
Formal notation.
Formally, P LATO uses self-supervision on G to pretrain an embedding for every input feature according to
[Equation]
M ∈ R d × c is the matrix of all feature embeddings.
H is a self-supervised node embedding method.
We refer to Eq.
(2) as pretraining since only the auxiliary KG G is used but the tabular data X , y is ignored.
After pretraining, the feature embeddings M are fixed.
Implementation.
For H , we choose ComplEx, a KG node embedding method [43].
ComplEx uses a self-supervised objective which learn an embedding for every node in G by classifying whether a proposed edge exists in G .
ComplEx’s proposed edges include both feature nodes and other nodes in
G , thereby integrating prior information about the input features and the broader domain.
3.3.2 Updating feature embeddings to contain the most relevant information for an input sample
Since different input samples might rely on different prior information about each input feature,
P LATO next updates each feature embedding M j ∈ R c to Q j ∈ R c , a feature embedding which contain the most relevant prior information about feature j for a given input sample X
i (Figure 1(c)).
P LATO uses a message-passing network Q on the KG to update the feature embeddings in a way that minimizes the number of additional trainable parameters.
[Equation]
Q uses an attention mechanism which considers the sample values X i to update the feature embeddings.
The attention mechanism has a small number of trainable parameters Π .
The message passing network Q .
Let Q [ r ] j be the embedding of input feature j after round r ∈ { 1 , ..., R } of message passing.
For every input feature j , Q first initializes the updated feature embedding to the pretrained feature embedding.
[Equation]
Q then conducts R rounds of message passing.
In each round of message passing, the feature embedding Q [ r ] j is updated from the feature embedding of each neighbor k in the prior round Q [ r − 1]
k and its own feature embedding in the prior round Q [ r − 1] j .
The “message” being passed is the embedding of each feature from the prior round.
[Equation]
σ is an optional nonlinearity.
N j are the neighbors of feature node j in G .
During message-passing, Q uses two scalar values β ∈ R and A ijk ∈ R to control the weights of messages.
First, Q uses hyperparameter β to control the weight of the messages aggregated from all neighbors vs. the message from the feature node j itself.
Second, Q calculates an attention score
A ijk to control the weight of the specific message between feature j and neighbor k .
The attention score is different for every sample i and is calculated by a shallow neural network A with a small number of trainable parameters Π .
The attention score A ijk thus enables Q to update the information in the feature embedding in a way that is most relevant for the input sample i .
Formally:
[Equation]
The number of trainable parameters in Π is small since the input of A is R 2 and the output of A is a scalar R .
A and its parameters Π are shared for all samples and features.
Finally, the updated feature embeddings Q j are set after R rounds of message-passing.
[Equation]
3.3.3 Predicting the first layer of parameters in F from the updated feature embeddings
Finally, P LATO predicts the parameters in the first layer of F from each updated feature embedding (Figure 1(d)(e)).
Every parameter in the first layer of F is associated with a feature j .
P LATO thus predicts ˆΘ j , the parameters associated with the feature j , from Q j , the prior information about j .
Formal notation.
P LATO predicts the parameters associated with every input feature j in the first layer of F according to
[Equation]
B is a shallow neural network parameterized by Φ .
Q j is the updated feature embedding of j which is conditional on the input sample X i .
B and Φ are shared for every feature j ∈ { 1 , . . .
, d } .
P LATO drastically reduces the number of trainable parameters vs. a standard MLP.
Sharing
B and Φ across all input features drastically reduces the number of trainable parameters compared to a standard MLP.
For a high-dimensional dataset ( i.e. d ≫ n ), a standard MLP T with h hidden units has a large number of trainable parameters in the first layer since Θ [1] ∈ R d × h .
A standard
MLP T must learn these dh trainable parameters independently.
By contrast, B uses a shared set of trainable parameters Φ to predict ˆΘ j from Q j for every j ∈ { 1 , . . .
, d } .
The number of trainable parameters in Φ is small compared to dh since B need only transform every Q j ∈ R c to ˆΘ [ 1 ] ∈ R h .
Thus, | Φ | = ch (assuming that B is a single layer neural network).
c , the dimensionality of the feature embedding, is much less than d , the number of input features.
As a result, | Φ | = ch ≪ dh and
P LATO reduces the number of trainable parameters in the MLP’s first layer.
3.4 The P LATO Algorithm
Pseudocode for the P LATO algorithm is in Appendix A.
Experiments
Datasets and KG.
We evaluate P LATO on 6 tabular datasets with d ≫ n and 4 with d ∼ n [12, 11, 9, 10].
We construct a general biomedical KG [52, 53, 54, 55, 56, 57, 58] with 108,447 nodes, 3,066,156 edges, and 99 relation types.
The same KG is used for all datasets even though the datasets have distinct feature sets with distinct cardinalities.
P LATO thus allows a single KG to serve as a unified knowledge backbone for different datasets in a domain.
Details are in Appendix F, G.
Fair Comparison of P LATO with Baselines.
We compare P LATO to 10 state-of-the art baselines.
We consider Ridge Regression [24], dimensionality reduction with PCA [17], feature selection with LASSO [20], deep feature selection with Stochastic Gates [39], and gradient boosted decision trees with XGBoost [26].
We also consider deep tabular methods including a standard MLP, selfattention-based methods with TabTransformer [35] and TabNet [34], differentiable decision trees with
NODE [30], and parameter-prediction with Diet Networks [51].
We also tried FT-Transformer [15], but it experienced out of memory
[Table] [Table]
[Table]
To ensure a fair comparison
[Table]
we follow evaluation protocols in tabular benchmarks [14, 15].
We conduct a random search with 500 configurations of every model (including P LATO ) on every dataset across a broad range of hyperparameters (Appendix B).
We split data with a 60/20/20 training, validation, test split.
All results are computed across 3 data splits and 3 runs of each model in each data split.
We report the mean and standard deviation of the Pearson correlation (PearsonR) between y and ˆy across runs and splits on the test set.
Each model is run on a GeForce RTX 2080 TI GPU.
4.1 Results
P LATO outperforms statistical and deep baselines when d ≫ n .
P LATO outperforms all baselines across all 6 datasets with d ≫ n (Table 1).
P LATO achieves the largest improvement on the PDAC dataset, improving by 10.19% vs. XGBoost, the best baseline for PDAC (0.400 vs. 0.363).
While
P LATO achieves the strongest performance across all 6 datasets, the best performing baseline varies with different datasets.
Ridge Regression is the strongest baseline for BRCA, LASSO for CM and
CRC, XGBoost for PDAC and CH, and TabTransformer for MNSCLC.
The remaining baselines (PCA, STG, Diet Networks, MLP, NODE, and TabNet) are not the strongest baseline for any dataset.
We also find that the performance of a specific baseline varies with the dataset.
TabTransformer, for example, is the best baseline for MNSCLC but the worst for CH (model ranking in Appendix D).
P LATO ’s performance depends on updating feature embeddings to contain information relevant to a sample.
P LATO predicts the parameters ˆΘ [ 1 ] in the first layer of a modified MLP F by using feature embeddings which contain prior information about the input features.
We conduct an ablation study to compare P LATO ’s performance when it uses feature embeddings which contain the relevant information for a given sample X i ( i.e. ˆΘ [1] =
B ( Q | X i ) )
vs. the pretrained feature embeddings M which contain general information about the input features that is not specific to a given sample ( i.e.
ˆΘ [1] =
B ( M ) ).
We also compare to a P LATO ablation that does not use any feature embeddings and is thus a standard MLP.
Using general feature embeddings M improves over not using feature embeddings (0.522 vs. 0.240).
Using feature embeddings Q that are specific to a given input sample further improves performance (0.583 vs. 0.522).
Therefore, updating the feature embeddings to Q so they contain the information specific to a given sample is critical to P LATO ’s performance.
[Table]
P LATO ’s performance depends on both feature nodes and broader knowledge nodes in the auxiliary KG.
P LATO relies on an auxiliary KG G which contains information about input features and information about the broader domain.
Information about input features is represented as feature nodes while information about the broader domain is represented as other nodes in G (Methods 3.1).
To test the relative importance of the feature information in G vs. the broader domain information, we measured the performance of P LATO on the BRCA dataset in two KG configurations: P LATO with the full KG ( i.e. both the feature nodes and the broader domain nodes) and P LATO with a “feature-only
KG” ( i.e. an induced subgraph on only the feature nodes) (Table 3).
We also compare to a “No KG” configuration in which P LATO does not have access to the KG and is thus ablated to a standard MLP.
We find that both the feature nodes and the broader knowledge nodes are important for P LATO ’s performance.
Using the “feature-only KG” configuration of P LATO improves performance vs the “no
KG” configuration of P LATO (0.539 vs 0.240).
Using the “full KG” configuration further improves performance vs the “feature-only KG” configuration (0.583 vs 0.539).
P LATO ’s performance thus relies on both the feature information and the broader domain information in the KG.
For datasets with d ∼ n , P LATO is competitive with baselines.
Finally, we test P LATO ’s performance for datasets with d ∼ n .
We test 4 datasets with d ∼ n ranging from dn = 1 . 1 to dn = 2 .(Table 4).
We find that on 4 datasets with d ∼ n , P LATO is competitive with the best performing baseline, XGBoost, but does not improve performance substantially.
P LATO ’s stronger performance for datasets with d ≫ n than for datasets with d ∼ n is justified.
P LATO ’s key idea is to include auxiliary information about the input features.
Auxiliary information is likely to help performance the most in settings with the least labeled data ( i.e. d ≫ n ).
When d ∼ n , auxiliary information is less helpful since the tabular dataset may already have enough information to train a strong predictive model.
We further find that XGBoost is consistently the strongest baseline for datasets with d ∼ n , in contrast to the varied performance of XGBoost on the datasets with d ≫ n (Table 1).
Discussion
P LATO is a deep learning model for tabular data with d ≫ n and an auxiliary KG with input features as nodes.
Across 6 datasets, P LATO achieves state-of-the-art, including improvements up to 10 .
Ablations validate each component of P LATO .
P LATO has several limitations.
First, P LATO matches but does not improve the performance of baselines for high-dimensional datasets with more samples ( i.e. d ∼ n ).
Second, P LATO relies on the coverage of prior information.
Datasets with input features that have less prior information in the KG are less likely to benefit from P LATO .
Overall, P LATO uses an auxiliary KG about input features to enable tabular deep learning when d ≫ n .
[Table]
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Code and data will be released with the archival version of the manuscript.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4 under “Fair Comparison of P LATO with Baselines” and Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See figures and tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4 under “Fair Comparison of P LATO with Baselines” header.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] All datasets and baseline models are cited.
(b) Did you mention the license of the assets?
[Yes] All licenses are present in the cited work.
(c) Did you include any new assets either in the supplemental material or as a URL?
Code and data will be released with the archival version of the manuscript.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Acknowledgments.
We thank Maria Brbic, Michael Moor, Kaidi Cao, Weihua Hu and Rajas
Bansal for discussions and for providing feedback on our manuscript.
C.R. is supported by a Siebel
Fellowship, a National Science Foundation Graduate Research Fellowship under Grant No. DGE- and a Stanford Enhancing Diversity in Graduate Education (EDGE) Fellowship.
H.R. is supported by a Masason Foundation, an Apple Fellowship, and a Baidu Scholarship.
We also gratefully acknowledge the support of DARPA under Nos.
HR00112190039 (TAMI), N (MCS); ARO under Nos.
W911NF-16-1-0342 (MURI), W911NF-16-1-0171 (DURIP); NSF under
Nos.
OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), NIH under
No. 3U54HG010426-04S1 (HuBMAP), Stanford Data Science Initiative, Wu Tsai Neurosciences
Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
B Evaluation Protocol and Hyperparameter Ranges
To ensure a fair comparison with baselines, we follow evaluation protocols outlined in tabular benchmarks [14, 15].
We conduct a random search with 500 configurations of every model (including
P LATO ) on every dataset across a broad range of hyperparameters.
We base the hyperparameter ranges on the ranges used in prior tabular learning benchmarks [14, 15] and the ranges mentioned in the original papers of the methods.
Hyperparameter ranges for P LATO are given in Supplementary
Table 1.
Hyperparameter ranges for baseline methods are given in Supplementary Table 2.
[Table]
[Table]
C Graph classification approaches
Graph classification models are not applicable for P LATO ’s setting.
In graph classification models, every input sample is a graph with node attributes, and a model must make a prediction for that graph.
The P LATO problem setting breaks fundamental assumptions made by graph classification models, rendering them not applicable.
First, graph classification models assume that different samples correspond to different graphs [59, 60, 61].
However, in P LATO every sample corresponds to the exact same graph.
There is a single background knowledge graph for all samples, and every sample has input features that correspond to the exact same nodes within the knowledge graph.
Second, graph classification approaches typically assume that every node in an input graph has a node attribute [59, 60, 61].
However, in P LATO only a small subset of the nodes in the knowledge graph have measured feature values.
Finally, graph classification approaches typically assume small graphs: the largest graph classification task in the Open Graph Benchmark has only 244 nodes [61].
However in P LATO , the knowledge graph contains 108,447 nodes and the smallest dataset has 12, features corresponding to nodes.
D Rank ordering of methods for datasets with d ≫ n
In Supplementary Table we show the rank order performance of all models on all d ≫ n datasets.
We find that P LATO exhibits consistent and strong performance while the performance of the baselines depends on the specific d ≫ n dataset.
For example, TabTransformer is the second best performing of all models on the MNSCLC dataset but the worst performing of all models on the PDAC and CH datasets.
Similarly, XGBoost is the second best performing of all models on PDAC but only the seventh best performing of all models on BRCA.
The baselines with the most stable performance are LASSO and Ridge Regression which rank consistently between the second and fifth best of all models.
[Table] [Table]
E Code Details
Code to run PLATO will be included as a supplementary file in the archival version of the manuscript.
F Dataset Details
We compiled 6 datasets with d ≫ n and 4 datasets with d ∼ n .
In all datasets, a machine learning model must predict the response of a cell or mouse to a drug.
In the tabular data, every row corresponds to a specific cell or mouse.
Every column corresponds to a gene name.
Every value corresponds to the amount of that gene in the tumor of the specific cell or mouse.
The label is the response of the cell or mouse.
All genes are nodes in the knowledge graph.
In practice, the number of genes is large for all tasks and the number of samples is comparatively small making the drug response setting appropriate for d ≫ n .
Dataset statistics, names, and sources are given below.
Dataset pre-processing followed a standard process described in [62].
Briefly, gene expression values underwent TMM normalization and log transformation ( i.e. log( x +1) ).
Values were made to have zero mean and unit standard deviation.
As labels, we used ln-ic50 for datasets from [9, 10, 11] and the minimum average percent tumor growth ( i.e. “min-avg-pct-tumor-growth”) for datasets from [12].
For all datasets, we use a 200-dimensional
ComplEx [43] embedding of the drug as the input feature vector.
All datasets will be released in the archival version of the manuscript.
[Table]
G Knowledge Graph Details
As a unified knowledge backbone for the datasets, we compile a general biomedical knowledge graph from prior studies [52, 53, 54, 55, 56, 57, 58].
A schematic of the KG is in Supplementary Figure 2.
A detailed breakdown of relation types is in Supplementary Table 5.
Our knowledge graph contains 108,447 total nodes, including 7,975 drugs, 18,370 diseases, 11, phenotypes, 22,319 genes, 11,153 molecular functions, 28,748 biological processes, and 4, cellular components.
Our knowledge graph contains 3,066,156 edges with 99 distinct relation types.
All datasets include features which map to a subset of nodes in the knowledge graph, primarily genes and drugs.
The remaining node types and their relationships serve as broader domain knowledge.
Edges between drug nodes and gene/protein nodes were derived from Drugbank [58], Gao [12], and the Genomics of Drug Sensitivity in Cancer [10, 9, 11].
Edges between diseases and genes/proteins were derived from DisGeNet [63].
Edges between diseases and phenotypes were derived from the Human Phenotype Ontology [53].
Edges between drugs and diseases were derived from the
Multiscale Interactome [55].
Edges between drugs and side effects were derived from SIDER [54].
Edges between genes/proteins and other genes/proteins were derived from BioGRID [64], [65], the
Database of Interacting Proteins [66], [52], [67], [68], [69], [70], and STRING [56].
Finally, edges from genes/proteins to molecular functions, biological processes, and cellular components as well as edges between molecular functions, biological processes, and cellular components were derived from the Gene Ontology [71].
The full knowledge graph will be included as a supplementary file in the archival version of the manuscript.
[Table]
type
Relation
Tail type # edges
BiologicalProcess
EndsDuring
BiologicalProcess BiologicalProcess
HappensDuring
BiologicalProcess BiologicalProcess
HasPart
BiologicalProcess BiologicalProcess
IsA
BiologicalProcess BiologicalProcess
NegativelyRegulates
BiologicalProcess BiologicalProcess
PartOf
BiologicalProcess BiologicalProcess
PositivelyRegulates
BiologicalProcess BiologicalProcess
Regulates
BiologicalProcess BiologicalProcess
OccursIn
CellularComponent
BiologicalProcess
HasPart
MolecularFunction
BiologicalProcess
NegativelyRegulates
MolecularFunction
BiologicalProcess
PositivelyRegulates
MolecularFunction
BiologicalProcess
Regulates
MolecularFunction
CellularComponent
HasPart
CellularComponent CellularComponent
IsA
CellularComponent CellularComponent
PartOf
CellularComponent
Disease
AlteredExpression
Gene
Disease
Biomarker
Gene
Disease
ChromosomalRearrangement
Gene
Disease
FusionGene
Gene
Disease
GeneticVariation
Gene
Disease
GermlineCausalMutation
Gene
Disease
ModifyingMutation
Gene
Disease
SomaticCausalMutation
Gene
Disease
SusceptibilityMutation
Gene
Disease
Therapeutic
Gene
Disease
Has
Phenotype
Drug
Treats
Disease
Drug
Carries
Gene
Drug
Enzymes
Gene
Drug
Targets
Gene
Drug
Transports
Gene
Drug
Has
Phenotype
Gene
Associates
BiologicalProcess
Gene
NotAssociates
BiologicalProcess
Gene
Associates
CellularComponent
Gene
Colocalizes
CellularComponent
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Gene
DirectInteraction
Gene Gene
DisulfideBond
Gene Gene
DosageGrowthDefect
Gene Gene
DosageLethality
Gene Gene
DosageRescue
Gene Gene
Enzymatic
Gene Gene
Expression
Gene Gene
GeneticInterference
Gene Gene
Hydroxylation
Gene Gene
Inhibition
Gene Gene
Kinase
Gene Gene
Literature
Gene Gene
Metabolic
Gene Gene
Methylation
Gene Gene
NegativeGenetic
Gene Gene
OxidoreductaseActivityElectronTransferAssay Gene
Gene
PhenotypicEnhancement
Gene Gene
PhenotypicSuppression
Gene Gene
Phosphorylation
Gene Gene
Phosphotransfer
Gene Gene
PhysicalAssociation
Gene Gene
PositiveGenetic
Gene Gene
PostTranslationalModification
Gene Gene
ProteinCleavage
Gene Gene
PutativeSelfInteraction
Gene Gene
Reaction
Gene Gene
Regulation
Gene Gene
Signaling
Gene Gene
SyntheticGrowthDefect
Gene Gene
SyntheticLethality
Gene Gene
SyntheticRescue
Gene Gene
Associates
MolecularFunction
Gene
Contributes
MolecularFunction
Gene
NotAssociates
MolecularFunction
Gene
NotContributes
MolecularFunction MolecularFunction
PartOf
BiologicalProcess
MolecularFunction
Regulates
BiologicalProcess
MolecularFunction
OccursIn
CellularComponent
MolecularFunction
HasPart
MolecularFunction MolecularFunction
IsA
MolecularFunction MolecularFunction
NegativelyRegulates
MolecularFunction MolecularFunction
PartOf
MolecularFunction MolecularFunction
PositivelyRegulates
[Table]