Ultra-marginal Feature Importance
Anonymous Author(s)
AffiliationAddress email
Abstract
Scientists frequently prioritize learning from data rather than training the bestpossible model; however, research in machine learning often prioritizes the latter.
Marginal contribution feature importance (MCI) was developed to break thistrend by providing a useful framework for quantifying the relationships in datain an interpretable fashion.
In this work, we aim to improve upon the theoreticalproperties, performance, and runtime of MCI by introducing ultra-marginal feature importance (UMFI), which uses preprocessing methods from the AI fairnessliterature to remove dependencies in the feature set prior to measuring predictivepower.
We show on real and simulated data that UMFI performs better than MCI,especially in the presence of correlated interactions and unrelated features, whilepartially learning the structure of the causal graph and reducing the exponentialruntime of MCI to super-linear.
Introduction
Scientists often seek to determine the true relationships between a set of characteristics and someoutcome of interest.
These relationships are ideally determined by performing carefully controlledexperiments so that causality can be established.
However, experiments can be difficult and costlyto pursue, unethical to perform, or impossible to control [51, 44], leaving only observational dataavailable.
The relationships that are hidden within vast quantities of observational data are oftendifficult to determine, so statistical tools, such as feature importance, have been explored.
Recently, feature importance methods such as Shapely-values [40, 13, 33], SAGE [14], accumulatedlocal effects (ALE) [3], permutation importance (PI) [8], and conditional permutation importance(CPI)
[16] have been used in high-impact journal papers by scientists who want to explain themechanisms within data [2, 5, 42, 29, 38, 19, 26].
However, these methods may not adequatelyexplain data in certain circumstances [12, 11].
ALE can only easily show first order effects [36], andalthough CPI improves upon some limitations of PI, CPI has the property that two perfectly correlatedfeatures with significant predictive power would both be deemed unimportant [14].
Further, only onemodel is trained in ALE, CPI, and PI.
Thus, correlated features, which can alter the model assemblyprocess, could be given artificially low importance if the goal is to explain the data [24].
Insteadof exploring a single model, the developers of SAGE, SPVIM, and marginal contribution featureimportance (MCI) evaluate the difference in accuracy between a model trained with the feature ofinterest and a model trained without it, across all feature subsets [11, 14, 49], though these methodsare prevented from being accepted by a wider scientific audience because of their high computationalcost.
In particular, we note that MCI is the current state-of-the-art method for explaining data asit was shown in extensive experiments to have better quality and robustness when compared to
Shapely-values, SAGE, ablation, and bivariate methods [11].
Though MCI can be seen as the current state-of-the-art method for explaining the data, it has threekey shortcomings.
First, the exact computation of MCI requires an exponential number of modeltrainings, which makes MCI ineffective at interpreting large datasets (e.g., gene expression studies).
Second, although it can handle complex feature interactions and data with correlated features, MCIunderestimates the importance of correlated features that form interaction effects because MCIusually ignores features that share information with the feature of interest x i .
Even if x i and x j forman interaction effect, the additional predictive power offered by x i on top of a subset S would bediminished by the presence of x j ∈ S , provided that the correlation between x j and x i is strongenough.
Third, MCI can give non-zero importance to features that are completely unrelated tothe response variable, as experimentally shown in Catav et al.
[11, Figure S3] and theoreticallyshown in Harel et al.
We hypothesize that constructing independent and information-preservingrepresentations of the data could resolve these three issues.
With this in mind, we introduce ultramarginal feature importance (UMFI), a new variable importance method that can better explain thedata while drastically reducing runtime.
The rest of this paper is organized as follows.
Axioms for explaining the data are proposed in
Section 2.
The framework for UMFI is then formally presented in Section 3 along with its theoreticalproperties and its simple algorithm.
In Section 4, we conduct experiments on simulated and realdata to assess the quality, robustness, and time complexity of UMFI compared to MCI.
Finally, anoverview of the work, its limitations, and ideas for future work are discussed in Section 5.
Related work
This paper is greatly inspired by the development of marginal contribution feature importance (MCI)by Catav et al.
Although other methods, such as SAGE [14], have been retooled to betterexplain data [12], up until this point, MCI had been the only feature importance method developedspecifically to explain data.
Let F = { x 1 , ..., x p } be the set of features used to predict the responsevariable, Y .
Recall that the universal predictive power of a set of features S ⊆ F is given by
[Equation]
where l is a specified loss function and G ( S ) is the set of all predictive models restricted to usingfeatures in S ⊆ F .
ν is closely related to mutual information, with equality under ideal conditions[14], and in practice, ν is often approximated by machine learning evaluation functions.
Using this,
Catav et al.
[11] defined the marginal contribution feature importance (MCI) of a feature x i ∈ F by
[Equation]
To achieve our goal of improving upon the shortcomings of MCI, we evaluate the importance of afeature of interest x
i after preprocessing the data to remove dependencies on x i .
Finding independentrepresentations of predictors for creating improved feature importance methods is a novel objective,though similar ideas have been suggested as future work in König et al.
[30] and Chen et al.
The weaker concept of finding orthogonal representations of data has been discussed previously[18], though the discussion has been limited to relative importances measures for multiple linearregression, mostly in the domain of psychology [6, 52].
While orthogonalizing predictors can be doneeasily with simple techniques, methods which can not only remove correlations between features,but also remove more general dependencies, have seen great progress within the domains of AIfairness and privacy.
Some examples of these techniques include regression [7], optimal transport[28], neural networks [10, 41], convex optimization [10], and principal inertial components [45].
Linear regression and optimal transport were implemented for UMFI in this paper.
Axioms for explaining data
Any attempt to build a method that explains the data should begin by rigorously defining whatexplaining the data truly means.
Different definitions and goals have been formulated by Chen et al.[12] and Catav et al.
Inspired by these definitions, we provide three intuitive, justified, andrigorous axioms for true-to-data feature importance methods.
Given a feature set F , a response Y ,and a feature of interest x i ∈ F , the feature importance of x i is defined as Imp F,Y ( x i )
∈ R ≥ 0 .
Wedefine the following three axioms as vital for any method that claims to explain the data:
Elimination axiom: Eliminating a feature x j from the feature set F can only decrease the importance of the feature of interest: ∀
[Equation]
F \ { x j } , Imp F \{ x j } ,Y ( x i ) ≤ Imp F,Y ( x i ) .
Duplication invariance and symmetry axiom:
Adding a duplicate copy of a feature ˆ x = x j already in the feature set F will not change the importance of the other features in F , and the duplicated feature will have importance equal to the original feature: ∀ x i ∈ F, Imp F,Y ( x i ) =
Imp F ∪{ ˆ x } ,Y ( x i ) and Imp F ∪{ ˆ x } ,Y (ˆ x ) = Imp F ∪{ ˆ x } ,Y ( x j ) .
Blood relation axiom: If data is generated from a causal graph, feature x i will be given non-zero and positive importance if and only if it is blood related to the response Y in the causal graph.
Two vertices in a causal graph are said to be blood related if there is a directed path between them or if there is a backdoor path between them via a common ancestor.
[Equation]
The elimination axiom comes directly from Catav et al.
Once a feature is observed to besignificantly related to the response, the relationship strength between the feature and response shouldnot drop, regardless of the additional features added.
In fact, often times the importance shouldincrease since adding features could reveal further synergistic information about the response Y .
The duplication invariance and symmetry axiom separates feature importance methods that are fordata explanation from methods intended for model optimization [11].
A model may use the twoidentical features equally often and therefore spread the importance equally between them (randomforests), or only one of the features may be given importance (lasso) [12].
However, from the data’sperspective, both features should be equally related to the response and the original importance foundbefore duplication should still be true.
Further, after duplication, no additional interaction capabilityis available [22], so the importance of all other features should remain the same.
The blood relation axiom asserts that feature importance scores intended for data explanation shouldextract reliable knowledge about the underlying causal graph and data generating process.
A statisticalassociation between a feature and the response, which is a quality of interest for many applications(e.g. , genome-wide association studies), exists precisely when the two features are blood related, orequivalently, when there an open path them ( [48] for a more in-depth explanation of this definition as well as other relevant concepts about causalgraphs).
Thus, a feature importance metric satisfying this axiom would give non-zero importanceto a feature if and only if there is a statistical association between that feature and the response.
Additionally, if the goal is to construct a causal graph to represent the relationships in the data, thena feature importance metric satisfying this axiom can partition the feature set into features that areblood related to the response and features that are not blood related to the response.
Although itdoes not enable us to immediately recover the full causal graph, this partitioning may be a helpfulsupplemental tool for other causal discovery methods.
See Supplement B for further discussion.
Ultra-marginal feature importance
Let F = { x 1 , ..., x p } be a set of p features of arbitrary type used to predict the response Y .
We notethat features may be viewed as random variables, or as realizations of random variables according totheir joint distribution, in the form of a dataset.
In order to define ultra-marginal feature importance, we require that the evaluation functionν , which measures the predictive power of a group of features [11], and which approximates
Equation (1), is also defined for transformations of the feature set following the removal ofdependencies.
We therefore define the space of information subsets of a feature set F as
I ( F ) = { g ( F ) : g is any function defined on F } .
We call these information subsets of F because
I ( Y ; g ( F )) ≤
I ( Y ; F ) holds for any function g by Theorem A.3.
Definition 1.
We denote S
Fx i as a preprocessed feature set after dependencies on the feature ofinterest x i have been removed from F .
An optimally preprocessed feature set is denoted by ˆ S Fx i , andwe say that a preprocessing S Fx i is optimal if it obeys the following properties:
[Equation]
The first property ensures that S Fx i ∈
I ( F ) , and hence, no information from outside of F is gainedduring the transformation.
The second property upholds that the random vector S Fx i is independentof x i , and the last property affirms the optimality of S Fx i in the sense that there is no unnecessaryinformation loss incurred during preprocessing.
Given that it exists, an optimal preprocessing ˆ S Fx iis not unique, since scaling g ( F ) by a constant does not affect the last two properties.
In practice,the last two properties can be difficult to guarantee, but we see later in Section 4 that non-optimalpreprocessings are good enough in many circumstances. Definition 2.
Given an evaluation function ν : I ( F ) → R ≥ 0 and a feature set F , we define theultra-marginal feature importance (UMFI) of a feature x i ∈ F as
[Equation]
UMFI obeys the three axioms given in Section 2 under certain assumptions as proven in Appendix
C. Mainly, we assume that ν ( · ) ≈
I
( Y ; · ) .
Under ideal conditions, this relationship holds when νsatisfies Equation (1) [14], but in practice, the accuracy of the approximation depends on the qualityof the method, the specified loss function, and the response variable’s distribution [15].
See Covertet al.
[15] and Appendix A.3 for a more thorough overview.
Since UMFI is model-agnostic, we provide a general algorithm for computing the ultra-marginalfeature importance of a feature x i ∈ F , which can be applied using any pair of preprocessing andmodeling techniques.
We note that ν f is not restricted to the domain of machine learning models oreven models in general.
For example, one could also implement UMFI with measures of dependencesuch as the Hilbert–Schmidt independence criterion [21] or non-ML estimates of mutual information[31].
Furthermore, if machine learning modeling techniques are used for UMFI, we advise that themedian score over multiple iterations of the algorithm is used to account for the variance of ν f .
1: Let Y be the response variable of the set of predictors F .
Choose a feature x i ∈ F . 2: Obtain S Fx i by using a technique that optimally removes dependencies on x i from F . 3: Specify a method f and a corresponding evaluation function ν f . 4: Estimate the predictive power, ν f
( S Fx i ) ,
that S Fx i has about Y .
5: Estimate the predictive power, ν f ( S Fx i ∪ { x i } ) , that S Fx i ∪ { x i } has about Y . : return
F,Y
f x
=
Experiments
We perform experiments to compare UMFI and MCI with respect to quality, robustness, and timecomplexity.
To implement UMFI, we consider optimal transport [28] (UMFI_OT) and linear regression [7] (UMFI_LR) as methods to remove dependencies from the data.
A detailed overview ofthese implementations is shown in Appendix E and experiments comparing these methods appear in
Appendix F. For all experiments, we use random forests’ out-of-bag accuracy ( R 2 OOB-accuracy forregression tasks and OOB classification accuracy for classification tasks) as the evaluation metricν f
We use the ranger R package to implement random forests with default hyperparametersand 100 for the number of trees [50].
All experiments were run in Microsoft R Open Version 4.0.2[35].
Appendix G contains additional experiments comparing UMFI and MCI with other featureimportance metrics including ablation, permutation importance, and conditional permutation importance.
In the same section, we rerun the experiments comparing MCI and UMFI using extremelyrandomized trees instead of random forests and do an additional comparison on a real dataset fromhydrology [1]. Code for all experiments can be found in the Supplement.
4.1 Experiments on simulated data
We run UMFI on simulated data to verify that it performs well compared to MCI.
The data in allsimulation studies contains
4.1.1 Nonlinear interactions
Interaction effects are common in many scientific disciplines where assessing feature importanceis prevalent, including hydrology [27, 2, 32], genomics [11, 47, 37], and glaciology [17, 4, 9, 39].
So, as was done in Catav et al.
[11], we assess the ability of MCI and UMFI to detect nonlinearinteraction effects in the data [34].
We consider:
[Equation]
Feature importance metrics should ideally conclude that x 1 and x 2 have higher importance comparedto x 3 and x 4 because of the extra interaction term, sign ( x 1 ∗ x 2 ) .
Figure 1a shows consistently goodperformance across all methods.
Each method gave high relative importance scores to x 1 and x 2 ,while x 3 and x 4 received less, but still substantial importance.
All methods show similar variability.
4.1.2 Correlated interactions
Interacting features are often correlated [25, 27].
So, this simulation study aims to repeat the nonlinearinteractions study, except now x 1 and x 2 are highly correlated with eachother.
In the same way, x 3and x 4 are highly correlated with eachother.
Let A, B, C, D, E, G ∼ N (0 , 1) .
We consider:
[Equation]
Just as with the interaction experiment with independent features, we would expect x 1 and x 2 to bemore important than x 3 and x 4 because of the extra interaction term, sign ( x 1 ∗ x 2 ) .
The results in
Figure 1b clearly show that UMFI provides better estimations of feature importance compared to MCIwhen correlated interactions are present.
MCI estimates that all features have approximately the samefeature importance scores, while both UMFI methods show significantly greater importance for x 1and x 2 compared to x 3 and x 4 .
MCI fails in this experiment because it penalizes feature subsets thatshare information with the feature of interest
x i when evaluating the importance of x i via Equation(2) .
For example, if we are assessing the MCI score for x 1 , since x 2 is strongly correlated with x 1 ,then the predictive power offered by x 1 on top of a subset S would be diminished by the presenceof x 2 ∈ S .
Therefore, x 2 is not utilized in the MCI score for x 1 , which prevents the detection ofthe interaction term sign ( x 1 ∗ x 2 ) .
UMFI is able to detect this interaction because it can extract theinformation from x 2 that interacts with x 1 while keeping this extracted feature independent of x 1 .
Although not yet tested, we suspect that similar results would be demonstrated in the presence ofdependent, but uncorrelated interactions.
4.1.3 Correlation
Feature importance methods that seek to explain data, such as MCI and UMFI, should not changethe measured importance of features in the presence of highly correlated or duplicated variablesaccording to the duplication invariance and symmetry axiom.
To test this, we implement a simulationstudy similar to the ones found in Catav et al.
Let ϵ ∼ N (0 , 0 . 01) .
We consider:
[Equation]
The addition of x 3 , which is approximately a duplicate of x 1 , should not alter the importance of x 1 ,and x 1 should remain equally as important as x 2 , since they have the same influence on the response
Y .
The results shown in Figure 1c show that both MCI and UMFI work reasonably well.
As with theprevious simulation experiment, the variability is consistent across methods.
As was desired, UMFIwith linear regression shows equal relative importance scores for x 1 and x 2 .
The importance given tox 2 was slightly greater than x 1 according to MCI and UMFI with optimal transport.
Interestingly,
MCI assigns some importance to x 4 , which was independent of the response, while both UMFImethods assign importance scores close to zero.
Because of this, we conclude that UMFI with linearregression performs the best in this simulated scenario.
4.1.4 Blood relation
To ensure that UMFI is true to the data and could be used to learn part of the structure of the causalgraph in theory as well as in practice, we implement the blood relation simulation experiment.
In this [Figure] study, data is generated from the causal graph in Figure 7 from the Supplement, which was inspiredby the collider causal graph found in Harel et al.
The feature S is unobserved, thus only x 3 andx 4 are blood related to the response Y .
Because of this, according to the blood relation axiom, x 3and x 4 should be given high and positive importance while x 1 and x 2 should receive zero importance.
In Section 3, we proved that in ideal scenarios, UMFI will only give non-zero importance to bloodrelated features.
We hypothesize that we can extend this to real-world scenarios where non-Gaussianfeatures and interaction information appear.
To test this, we consider:
[Equation]
The results shown in Figure 1d indicate that MCI fails to distinguish the blood related features, sincemost of the importance is given to x 1 , x 2 ̸∈ BR ( Y ) .
In contrast, UMFI_LR and UMFI_OT detectthat x 1 and x 2 should have zero importance while giving most of the importance to x 4 and the rest ofthe relative importance to x 3 .
4.2 BRCA experiments
We use the same breast cancer (BRCA) classification dataset [43] used in previous feature importancestudies including Catav et al.
[11] and Covert et al.
[14] to test the quality and robustness of UMFI on real data.
The original data contains over 17 , 000 genes and 571 anonymous patients that havebeen diagnosed with one of 4 breast cancer sub-types.
We consider the same subset of 50 genesas in Catav et al.
[11] and Covert et al.
[14] for easier computation and result visualization.
Ofthe 50 selected genes, 10 are known to be associated with breast cancer, while the other 40 genesare randomly sampled.
This data was downloaded from https://github.com/TAU-MLwell/
Marginal-Contribution-Feature-Importance/tree/main/BRCA_dataset (MIT License).
In Catav et al.
[11] and Covert et al.
[14], these 40 randomly sampled genes are assumed to beunassociated with breast cancer.
However, to ensure a more definitive ground truth, we also randomlypermute the values of these 40 genes across their respective 571 observations to further reduce thechance that these genes have any association with breast cancer.
Quality is then measured with thetrue positive and true negative rates: the 10 BRCA associated genes should have some non-zeroimportance (positive), and the other 40 genes should have exactly zero importance (negative).
Theseexperiments were run 200 times on different seeds and with a different random sample of 500 patientsfor each iteration.
Robustness is measured using the standardized interquartile range (SIQR) from therepeated experiments, which is calculated by dividing the average IQR across the 50 features by theaverage median.
This experiment is too computationally intensive for MCI to be calculated exactly,so we implement MCI assuming soft 2-size submodularity.
We found that MCI and UMFI (UMFI_LR and UMFI_OT) correctly gave significant importanceto the 10 genes that are known to be associated with breast cancer (Figure 2).
Interestingly, theordering of important features was similar across methods, with BCL11A and SLC22A5 always being the most important and TEX14 always being the least important of the 10 BRCA-associatedgenes.
However, MCI consistently gives non-zero importance to all features, while UMFI correctlygives zero importance to the majority of the randomized genes.
Furthermore, UMFI’s performance inthis experiment improves with increased iterations.
After running the experiment 5000 times, both
UMFI methods have a perfect overall accuracy when distinguishing between important and permutedfeatures (Appendix G.2.1).
Although UMFI scores have higher variability than MCI (Table 1), it isclear from Figure 2 that UMFI separates the 10 associated genes from the 40 unassociated genesbetter than MCI does.
[Table]
4.3 Computational complexity
MCI must train and evaluate a model for each element of the power set of the feature set, whichimplies O (2 p ) model trainings if there are p features.
If the evaluation function ν obeys soft k -sizesubmodularity, then the maximizing subset has no more than k elements, which reduces the numberof model trainings to O ( p k +1 )
UMFI circumvents the exponential training time since it canbe evaluated immediately after removing the dependencies of x i from the feature set F .
To confirmthe above statements, and to show that the extra model trainings required for MCI dominate thecomputation time for removing dependencies in UMFI, we ran a simple experiment.
For a rangeof dataset sizes from the BRCA data, we evaluate the computation time for calculating the featureimportance scores of all features using MCI and UMFI.
We ran this experiment for a dataset with 5features, and then slowly added features until our given time budget of 1 hour ran out.
Once all
BRCA features were used, more features were randomly generated.
All datasets had 571 observations.
These experiments were run using an Intel Core i9-9980HK CPU 2.40GHz with 32GB of RAM.
Code was parallelized in R, and 12 of the 16 available threads were used.
From Figure 3, we can observe that UMFI is approximately superlinear, with UMFI_OT incuringmore computational cost compared to UMFI_LR.
Giving each method 1 hour to run, MCI processed 19 features, MCI with the soft 2-size submodularity assumption processed 130 features, UMFI_OTprocessed about 3000 features, and UMFI_LR processed about 4000 features (Table 1).
Conclusion
In this study, we introduced ultra-marginal feature importance (UMFI), a new method that usespreprocessing techniques, originally developed in the domain of AI fairness, to provide fast andaccurate feature importance scores for the purposes of explaining data.
We introduced three idealaxioms that feature importance measures should satisfy if they claim to explain the data, which areall satisfied by UMFI under some basic assumptions (Appendix C).
Optimal transport and linearregression were explored as preprocessing techniques to remove dependencies from data.
Whencompared with MCI, the previous state-of-the-art method for explaining data, experimental resultsshowed that UMFI was able to provide faster and more accurate estimates of feature importance onreal and simulated data, particularly in the presence of correlated interactions and unrelated features.
UMFI’s superior time complexity could be leveraged to run feature importance on larger datasets orto achieve more accurate results by utilizing its median scores after many iterations.
Throughout the work on this paper, several shortcomings appeared.
First, we only considered twosimple methods for removing dependencies, linear regression and pairwise optimal transport.
Othermethods certainly exist in the literature, including optimal transport with chaining [28], neuralnetworks [10, 41], or principal inertial components [46].
Though our two methods performed fairlywell on the real and simulated datasets in Section 4, optimal transport and linear regression failed tofind representations of the data that were independent of the protected attribute when we tested themethods on a hydrology dataset with more shared information compared to BRCA [1] (Appendix
G.4).
However, neural nets or principal inertial components certainly could have given better results.
Also, despite requiring significantly more computational cost, better methods for estimating theconditional CDF, or using optimal transport with chaining, should give better estimates for S Fx i whenimplementing UMFI_OT.
Even though dependencies were not removed optimally for the hydrologydataset, the estimates of feature importance were still reasonably accurate.
Second, UMFI scores areless robust than MCI since they have higher variability, however, because of the significantly lowercomputational cost, UMFI can be run multiple times and averaged to increase robustness.
Third, it isnot clear how closely ν f approximates mutual information in practice.
Finally, though UMFI canwork for any arbitrary feature type, in this paper, we have only considered datasets with continuousexplanatory variables.
In future work, we would like to test how well other methods, such as neural networks, pair with
UMFI while further testing on a wider variety of random variable types such as binary, categorical,and ordinal features.
Further, we would like to explore how well dependence can be removed and
UMFI can be estimated on real data as the number of features increases to sizes much larger than 50 .
To reiterate, UMFI is a powerful tool for detecting and explaining the relationships hidden within data.
We emphasise that UMFI is just a framework.
A variety of other methods can be used to estimate theuniversal predictive power ν including, but not limited to, XGBoost, neural networks, or Gaussianprocesses.
Even non-model-based methods such as Hilbert-Schmidt independence criterion could beexplored in future applications.
Furthermore, new preprocessing techniques for dependence removalare still being developed in the AI fairness community, so these, in addition to other existing methods,can be used in future applications of UMFI for additional improvements.
Broader Impact
We hope that UMFI will be a useful tool in a variety of disciplines including bioinfomatics, ecology,earth sciences, and health science for discovering scientific processes and relationships hidden withindata.
Though we think that our contributions can only lead to positive social and environmentalimpacts by aiding scientific discoveries in domains like earth science and bioinformatics, statisticalmethods, especially those that are aimed at genetics research, have historically been used to justifyharmful and misleading claims.
If such claims arise using our methods, then they should be dismissedsince direct causal effects cannot be concluded after using our methods alone.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] After reading the abstract and introduction please readSection 4 to see quantitative support for our claims.(b) Did you describe the limitations of your work?
[Yes] Please see Section 5 for a detaileddescription of limitations.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please see Section (b) Did you include complete proofs of all theoretical results?
[Yes] Please see Section 3and the Supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Please seeSection 4 and the Supplementary material.(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please see Section 4 and the Supplementary material.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] In most cases we did, see Section 4 and the Supplementarymaterial, but for the computational complexity experiment, we did not because of the
higher computational cost and the fact that repeated experiments would not significantly
change the results (this was tested but not shown).
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Section 4 and the
Supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Please see Section4.2.(b)
Did you mention the license of the assets?
[Yes] Please see Section 4.2.
The BRCAdataset is available on Github via the MIT License.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Please see the supplemental material for code.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Please see Section 4.2.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] Please see Section 4.2.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]