Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Consider making a prediction over new test data without any opportunity to learnfrom a training set of labelled data - instead given access to a set of expert modelsand their predictions alongside some limited information about the dataset used totrain them.
In scenarios from finance to the medical sciences, and even consumerpractice, stakeholders have developed models on private data they either cannot,or do not want to, share.
Given the value and legislation surrounding personalinformation, it is not surprising that only the models, and not the data, will bereleased - the pertinent question becoming: how best to use these models?
Previouswork has focused on global model selection or ensembling, with the result of asingle final model across the feature space.
Machine learning models performnotoriously poorly on data outside their training domain however, and so we arguethat when ensembling models the weightings for individual instances
must reflecttheir respective domains - in other words models that are more likely to have seeninformation on that instance should have more attention paid to them.
We introducea method for such an instance-wise ensembling of models, including a novelrepresentation learning step for handling sparse high-dimensional domains.
Finally,we demonstrate the need and generalisability of our method on classical machinelearning tasks as well as highlighting a real world use case in the pharmacologicalsetting of vancomycin precision dosing.
Introduction
Sharing data is often a very problematic affair - before we even arrive at whether a stakeholderwill want to, given the modern day value - it may not even be allowed.
When the data containsidentifiable and personal information it may be inappropriate, and illegal, to do so.
A commonsolution is to provide some proxy of the true data in the form of a generated fully synthetic datasetor one that has undergone some privatisation or anonymisation process (Elliot et al., 2018)
butthis can often lead to low-quality or extremely noisy data (Alaa et al., 2021) that is hard to gaininsight on.
Alternatively, groups may release models that they have trained on their private data.
If for a given task there are multiple of such models, we are left with the task of how to best usethese models in combination when they have potentially conflicting predictions.
This problem isknown as unsupervised ensemble learning since we have no explicit signal on the task, and weaim to construct a combination of the models for making future predictions (Jaffe et al., 2016).
Without having trained the models ourselves, it is a very challenging task for us to know how wellthe individual models should perform, making the task of choosing the most appropriate model (orensemble) difficult.
This is compounded by the problem that the provided models could performpoorly for not one, but two main reasons: Firstly, the model itself may not have been flexibleenough to properly capture the underlying true function present in the data; and secondly, in the area that they are making a prediction there may not have been sufficient training data used forthe model to have been able to learn appropriately - i.e. the model is extrapolating (potentiallyunreasonably) to cover a new feature point - the main issue in covariate shifted problems (Bickelet al., 2009).
Current practice is usually to try and select the globally optimal model (Shahamet al., 2016; Dror et al., 2017), that is to say of those made available, which model should beused to make predictions for any given test point in the feature space.
This approach potentiallyaddresses the first point, it completely overlooks the second - and this is what we will focus on.
In order to consider this problem of extrapolation amongst the individual models, the importantquestion is: what might a solution look like?
We must consider the desiderata that given noaugmentation of the individual models, the ensemble weights must vary depending on the testfeatures and additionally, these weights should reflect the confidence that a model will be able tomake an appropriate prediction, and we should be able to tell generally when our confidence is low.⨯ A ⨯ B
[Figure]
Given new test points A, B, and C, we need to construct predictions from these models.
A is well represented by both M 2 and M 3 while
B only has significant density under M 3 .
C looks like none of the models will be able to make confident predictions.
Contributions In this work we make a three-foldcontribution.
First, we establish and document theneed for instance-wise predictions in the setting ofunsupervised ensemble learning - introducing the concept of Synthetic Model Combination (SMC).
Second, we introduce a novel unsupervised representation learning procedure that can be incorporated into SMC - allowing for more appropriate separation ofmodels and estimation of ensemble weights in sparsehigh-dimensional settings.
Finally, we provide practical demonstrations of the success and failure casesof traditional methods and SMC, in synthetic examples as well as a real example of precision dosing.
Problem
Formulation and Relationships
Formulation Consider access to N provided tuples { ( M j , I j ) } Nj =1 of models M j and associatedinformation I j .
With each M j : X (cid:55)→ Y mappingfrom some covariate space
X to target space Y andhaving been trained on some dataset D j which isnot observed although is in some way summarisedby I j .
We are presented with some test dataset
D T = { x i } Mi =1 and tasked with making predictions.
Goal Our aim is to construct a convex combination of models in a way so as to produceoptimal model predictions ˆ y =
M ∗ ( x ) = (cid:80) N j =1 w ( x ) j M j ( x ) with (cid:80) N j =1 w ( x ) j = 1 andw ( x ) j taking the j th index of the output of the function w : X (cid:55)→ ∆ N that maps featuresto a set of weights on the probability N -simplex.
We can summarise our task process as:
Given: { ( M j , I j ) } Nj =1 and { x i } Mi =
Obtain: w to predict { ˆ y i } Mi =
Having established the setting we are focused on, we can explore contemporary methods andcompare how they relate to our work.
We consider methods that are interested ultimately in thetest-time distribution of labels conditional on feature covariates p test ( Y | X ) .
We discuss the differences in methods focuses on the distributional information available to them .
For example, instandard supervised learning, samples of the feature-label joint distribution p train ( X,Y ) are given - thetraining-time conditional distribution is then estimated and assumed to be equivalent at test-time.
In our case, we assume information from considerably less informative distributions, the training time feature distributions { p M j ( X ) } Nj =1 , were the information can take a variety of forms,most practically though through samples of the features or details of the first and second mo-
[Table]
ments.
Practically, this appears to be the minimal set of information for which we can do something useful since if only given a set of models and no accompanying information then thereis no way to determine which models may be best in general let alone for specific features.
How are models normally ensembled?
The literature on ensemble methods is vast, and we do notintend to provide a survey, a number of which already exist (Sagi and Rokach, 2018;
Dong et al.
The focus is often on training ones own set of models that can then be ensembled for epistemicuncertainty insight (Rahaman et al., 2021) or boosted for performance (Chen and Guestrin, 2016).
In terms of methods of ensembling models that are provided to a practitioner (instead of onestrained by them as well) then closest is the setting of unsupervised ensemble regression - whichlike us does not consider any joint feature label distribution.
To make progress though someinformation needs to be provided, with Dror et al.
(2017) considering the marginal label distributionp test ( Y ) , making the strong assumption the first two moments of the response are known.
Insteadof being directly provided the mean and variance, another strand of work assumes conditionalindependence given the label (Dawid and Skene, 1979), meaning that any predictors that agreeconsistently will be more likely to be accurate.
Jaffe et al.
(2016) relaxes this assumption, aimingto detect strong dependence and construct a meta-learner for better predictions.
Recent work of Shaham et al.
(2016) attempts to learn the dependence structure using restricted Boltzmann machines.
Bayesian model averaging when validation data is available.
Moving from the unsupervisedmethods mentioned above, which at most considered information from only marginal distributions, wecome to the case of when we may have some validation data from the joint distribution p test ( X,Y ) (or oneassumed to be the same).
This can effectively be used to evaluate a ranking on the given models formore informed ensembles.
This ensemble can be created in a number of ways, a practical and commonapproach being Bayesian Model Averaging (BMA) (Raftery et al., 1997).
Given an appropriateprior, we calculate the posterior probability that a given model is the optimal one - and once thisis obtained the models can be marginalised out during test time predictions, creating an ensembleweighted by each model’s posterior probability.
The posterior being intractable, the probabilityis approximated using the Bayesian Information Criterion (BIC) (Neath and Cavanaugh, 2012)
-which requires a likelihood estimate over some validation set and is estimated as: p ( M i |D )
=
exp ( − 12 BIC ( M i )) / (cid:80) Ni =1 exp ( − 12 BIC ( M i ))
With this, along with all the ensemble methodspreviously mentioned, it is important to note the subtle difference in setup to the problem we aretrying to work with.
In all cases, it is assumed that there is some ordering for the models thatholds across the feature space and so a global ensemble is produced with a fixed weighting ˆ wsuch that w ( x ) =
ˆ w ∀ x ∈ X .
This causes failure cases when there is variation in the modelsacross the feature space, since it is a key point that BMA is not model combination (Minka, 2000).
This being an important distinction and one of the main reasons BMA has been shown to performbadly under covariate shifted tasks (Izmailov et al., 2021).
That being said, it can be extended byconsidering the set of models being averaged to be every possible combination of the providedmodels (Kim and Ghahramani, 2012), although this becomes even more computationally infeasible.
Is this some form of unsupervised domain adaptation or transfer learning then?
Giventhe focus on models performing on some region of the feature space outside their training domain this may seem like a natural question.
Unsupervised domain adaptation represents thistask at an individual model level but usually considers, access to unlabelled data in the tar- get domain as well as labelled data from a (different) source domain p train ( X,Y ) , p test ( X ) .
We referthe interested reader to Kouw and Loog (2019) for a detailed review given space constraints.
In a very similar vein, the transfer learning (Torrey and Shavlik, 2010) task also involves a changein feature distribution but instead of being completely unsupervised tends to include some labelson the target set, thus requiring some information on the joint distribution at both test and trainingtime p train ( X,Y ) , p test ( X,Y ) .
A great deal of work then involves learning a prior on the first domain thatcan be updated appropriately on the target domain (Raina et al., 2006; Karbalayghareh et al., 2018).
In contrast to both of these areas, we do not aim to improve the performance of the individualmodels, but rather combine them based on how well we expect them to perform in the new domain.
Introducing Synthetic Model Combination
Our success hinges on the assumption that the quality of a model’s prediction will depend onthe context features .
That is to say that the performance ordering of the models will not stayconstant across the feature space.
With this being the case, it should seem obvious that theweightings of individual models should depend on the features presented.
As such, we introduce our notion of Synthetic Model Combination 1 - a method that constructs a representation space within which we can practically reason, enabling it to select weights for the models based on a given feature’s location within the representation space.
Recalling our startingpoint of { ( M j , I j ) } Nj =1 and { x i } Mi =1 , we proceed in three main steps which are outlined below 2 :
Estimate densities for models:
From each information I j , we generate a density p X j ( x ) .
Learn low-dimensional representation space:
Using { x i } Mi =1 and { p X j ( x ) } Nj =1 , learn a mapping to a low dimensional representation space f θ : X (cid:55)→ Z . 3.
Calculate ensemble weights for predictions: Evaluate weights w ( x ) so that predictions can then be made as ˆ y = (cid:80) Nj =1 w ( x ) j M j ( x ) .
3.1 From Information to Probability Densities
The first step in SMC is to use the information I j to produce a density estimate such that we can sample from each model’s effective support.
Given the flexibility in the form of what we allow I j to take,
SMC must remain relatively agnostic to this step.
A common example of the type of information weexpect will simply be example feature samples, and in this case a simple kernel density estimate (Terrell and Scott, 1992) or other density estimation method could be employed.
On the other hand, in themedical setting for example, when models are published authors will often also provide demographicsinformation on the patients that were involved in the study, such as the mean and variance of eachcovariate recorded.
In this case we may simply want to approximate the density using a Gaussian andmoment-matching for example.
When the information is provided in the form of samples it is possibleto skip this step, as they can be used directly in the subsequent representation learning step’s losses.
3.2 Learning a Separable and Informative Space
The point of learning a new representation space - and not simply using the original feature space- is twofold.
Firstly, we would like to reduce the dimensionality , leading to a more compact representation and is important because in higher dimensions the densities we model will often endup effectively non-zero in only very small regions, making the last step of SMC difficult.
Secondly, we would like to induce structure , so that distances and regions in the space better reflecthow capable different models will be over the space.
This should aid in better selection of modelweights when making predictions by allowing SMC to better understand the relationship betweenthe models and the representation space.
We should note that if the feature dimensions are low These steps are Algorithm 1 as well as pictorially in Figure 2. then this step is not strictly necessary - the model densities may have appropriate coverage ofthe space, but this does not then allow you to obtain none of the benefits of the second point.
[Figure]
To proceed we define a space Z on which wewill work and aim to learn a parameterised mapping f θ : X (cid:55)→ Z such that the representationsof features on this space is useful and aids us inour goal of constructing instance-wise ensembles.
In each optimisation step of our algorithmwe will sample a model dataset D M =
ˆ x j } Nj =
1- sampling one feature example 3 from each density associated with a model.
This model datasetwill play an important part in the learning process as these examples serve as proxies to theregions in the feature space that a model is confident on.
Now, given both D T and D M , we aimto construct an optimisation target for a representation learning step that can be learnt end-to-endin an autoencoder fashion.
As such, we introduce a total loss that can be broken down into three parts:
L total =
L reconstruction + L connection + L seperation .
The first, L reconstruction , is the simplest as a simple regularised autoencoder loss and isused so that the latent space can accurately represent and reconstruct the feature space:
[Equation]
This introduces a second function g ϕ : Z
(cid:55)→ X that can be jointly optimised in order tolearn the space using standard autoencoder techniques.
Alternatively, to incorporate a probabilistic element you can substitute in a ( β -)VAE loss (Kingma and Welling, 2013; Higginset al., 2016) by considering samples from an approximate posterior, the probabilistic naturehas been shown to improve interpretability in the representation space (Burgess et al., 2018).
We next introduce the first of our more specialised components with the L connection loss:
[Equation]
[Table]
[Equation]
which is designed so that models that make similar predictions have domains that map to similarareas in the space.
Here, D Y ( · || · ) is a normalised distance metric over Y and is used to calculatethe Predictive Similarity Score between two models evaluated on the same feature.
This loss aimsto minimise the distance
|| f θ (ˆ x i ) − f θ (ˆ x j ) || 2 - which is a proxy for the distance between thedomains of models i and j - when the predictions between the two models are more similar.
Thisis based on the intuition that if models are making randomly incorrect predictions for a featurethey are unlikely to agree with other models (Dror et al., 2017).
Thus, these points are morelikely to be correctly classified - suggesting some overlap in training domain of the two models.
Given the non-negativity of distances we will need to balance the previous loss, which willalways aim to reduce the distance between all the pairs in the loss, albeit up-weightingthose that have more similar predictions between models.
Thus, we employ L separation :
[Equation]
which is designed so that models are naturally moved away from each other - essentially encodinga prior that data distributions for the different models will be distinct.
With 1 denoting the indicator function and x ( m ) denoting the model density from which the point was sampled, this losspushes apart points that were not sampled from the same model.
These losses can be balancedwith weighting hyperparameters, the potential optimisation of which is discussed in the appendix.
3.3 Weights Estimation
Once a space is learnt, we can use it to makepredictions.
Given model densities in the feature space p X j ( x ) , we construct a corresponding density in the representation space p Z j ( z )- this can be achieved simply by samplingfrom p X j ( x ) , passing through f θ and modelling the new density with a kernel densityestimate.
From here, we calculate weightsas the relative density a feature representation has under the densities in the new space:
[Equation]
with γ a regularisation hyperparameter chosen to be very small such that an outlier’s weights are notdominated by the closest model.
The quantity (cid:80) Nj =1 p Z j ( f θ ( x )) can be used to inform the confidenceof any prediction made by SMC.
Particularly low values will indicate that the feature had low densityunder all the domains and as such it may be likely that none of the models were accurate.
We noteas well that assuming a hierarchical generative model for the test data where one of the modelstraining data distributions is selected and then sampled from - this can be interpreted as the posteriorprobability that a test instance was sampled from a model’s domain and is thus well represented by it.
Experimental Demonstration
In this section we will use a series of experiments to make the following concrete points aboutour method: 1) In common scenarios, global ensembles do not work , and we must make instancewise predictions (Section 4.1); 2) Doing so in even slightly high dimensions requires a representation learning step, and our proposed losses improves the quality of the learnt representation(Section 4.2); 3) We can make good predictions with surprisingly little information (Section 4.2);4)
This is useful beyond synthetic example setups in real-world case scenarios (Section 4.3); 5)
Naturally, there are setups where we will underperform , which we should understand (Section4.4); and 6) We are completely agnostic to the type of models used (Section 4 - we demonstrateacross a variety of models from simple regressions to convolutional nets and differential equations).
4.1 A Simple Regression Example
In this first example, we aim to demonstrate why this approach is a clear necessity when makingfuture predictions, showing that we need to construct ensembles with an instance-wise approach togenerate predictions that are accurate and appropriately calibrated.
In our examples, we simulatefeature data from experiment-dependant distributions and targets from a simple noisy sin wave curve.
Instance-wise ensembles are necessary under our assumptions.
In our first example we consider two models: both simple neural networks, one trained on features sampled from a Gaussiancentred at 5, with the other centred at 15, and standard deviation 3.5 - as can be seen from Figure
a There is very little overlap in the support of the training data for the two models, andit can be seen that Model 1 clearly makes appropriate predictions when x < 10 , while it is
Model 2 that is accurate when x > 10 .
This relationship cannot be captured by a global ensemble of the two methods, as seen in Figure 3 b - whereas SMC is perfectly able to do so.
Our uncertainty can show when we are not confident.
In our previous example, SMC wasable to match the target function across the feature space given the distribution of the trainingfeatures of the two models.
What happens though if none of the models make accurate predictions on an instance because it is outside all of their domains?
We explore this by moving thetwo Gaussians used to generate features for the two models further apart, centring them at
and20 respectively - resulting in a region around 10 where none of the models make correct predictions, and consequently SMC’s predictions also suffer as seen in Figure 3 c .
However, using theuncertainty explained in Section 3.3 (which is visualised by the background colour of the plot)
[Figure]
[Figure] [Figure] [Figure] [Figure]
D i m
[Figure]
we can see that the accuracy of SMC is calibrated well against its confidence.
SMC can provide useful information about which parts of the feature space it is relatively less confident on.
Higher Dimensions with MNIST
[Figure]
In order to develop further points of our method, wemove to a more complicated example with the mosticonic ML problem of handwritten digit classification.
Using MNIST ( d = 784 ), we construct a problemwhere ten different classifiers are trained to each individually identify a single digit effectively while theirperformances on other digits are significantly lowerthis is achieved by providing mostly only data of therespective single digit.
The information provided permodel involves a number of feature samples from thetraining set (number depending on exact experiment).
High dimensions need to be reduced.
As discussedearlier, density estimation becomes ineffective in highdimensions.
As the space gets bigger, the relativeproportion covered by the data reduces.
We find thateven in the case where the full set of features (but of course not targets) used to train the predictive models are made available to SMC, none of the test set features have non-zero density tonumerical precision - and hence a meaningful prediction cannot be made.
On the other hand, oncewe incorporate a representation learning step, the coverage of the training examples significantlyincreases.
This can be seen in Figure 4, which shows two separate 2-dimensional learnt represen- tation spaces: on the left using only the L rec loss; and on the right including the L sep and L conlosses.
Here, test features are plotted as points, while the training features used per model arerepresented as density estimates.
It can be very evidently seen that the majority of test features havesignificant density under at least one of the models in both of the example representation spaces learnt.
Standard representation learning is good - but we are better.
We have established that some lowdimensional representation learning is necessary for problems like these - the fact is that by simplyincorporating a standard representation learning step, using a standard autoencoder for example,will allow us to make reasonable predictions, indeed we can achieve a OneVsRest Area Under the
Receiver Operating Characteristic curve (AUROC) of 0.969 on the test set.
By including the extraregularisation though we can improve the ability of our algorithm.
We can immediately see witha visual inspection of Figure 4 and the two learnt representation spaces that their inclusion resultsin more spread out and differentiated densities (both are plotted on the same scale).
Additionally,we see and increase in the AUROC to 0.972.
This appears to allow more appropriate evaluation ofpoints like the example ‘4’ in Figure 4, which could reasonably be mistaken for an incomplete ‘9’.
Many examples are unnecessary for significant performance gain.
In Figure 5 we plot thepredictive AUROC of SMC against the number of feature samples from the training data passed tothe method.
In this setup the information provided to SMC are a number of feature examples permodel that can be sub-sampled from to evaluate the regularisation losses and then used to constructdensity estimates in the representation space.
Naturally, as the number of samples increases so doesthe quantity of information and so too does the performance of SMC.
In the extremes, of course withno samples no meaningful prediction can be made, and when all the training features are provided an
AUROC of about 0.97 is achieved.
Interestingly, we can see that actually not a very large number ofsamples are required to obtain strong performance here.
Notably, after only 3 samples are provided,
SMC sees significant performance improvement to 0.83, compared to the baseline of 0.50.
Thisis not a significant amount of information, with just three features and none of the target details.
4.3 Case Study: Vancomycin Precision Dosing
Population pharmacodynamic (PopPK) models aredifferential equations that model the concentrationof a drug in the bloodstream of a patient.
Vancomycin one of the most common antibiotics - withmultiple PopPK models (Broeker et al., 2019) hav- been developed on different patient populations.
For many drugs, including vancomycin, the areaunder the curve (AUC) is an important marker inestimating the effectiveness of a drug interventionand can be used to support dose individualisationby predicting drug response in the future.
An example of the models are found in Figure 6, showing the predicted concentration levels in an example patient for a number of different PopPK models.
SMC can be applied to real challenges.
Thisis a setting where SMC would be a particularlyappropriate method of choice.
First, the privatemedical nature of the problem means that data on drug response in humans is not widelyavailable, and so new models cannot be trained on the data of the previous models.
Second, the datasets that the models are built on are often relatively small and focus on aspecific subpopulation, such as those at a particular hospital, or suffering a specific comorbidity.
Third, researchers usually publish the model alongside demographic information onthe patients used to produce them, fulfilling SMC’s need for both a model and information.
Accurate Drug Response Estimation.
We base our experiment around those of Uster et al.
(2021)who themselves consider a model averaging approach through the application of BMA.
We usesimulated patients provided by the authors to evaluate the effectiveness of SMC in the accuracyof predicting the AUC across a number of settings when a number ∈ { 0 (A priori) , 1 , 2 , 3 , 4 } ofconcentration measurements are taken in a 36-hour period.
Ultimately, we have five models, eachfrom a separate subpopulation {extremely obese, critically ill post heart surgery, trauma patients,intensive care patients, hospitalised patients}, as well as a variety of demographic information
[Table]
for each 4 .
In our experiments we focus on the age, height, weight, sex, and creatinine clearancelevels as have been shown to be strongly associated with drug response (Uster et al., 2021) andare provided for each model.
In Table 2 we report the relative root-mean-square error RMSEof the predictions - the lower, the better.
We can see that SMC consistently performs welland indeed performs the best when at least two observations are made available to the methods.
4.4 Understanding Challenging Scenarios for SMC
There’s no such thing as a free lunch, and SMC does not have the answer for everything- there are scenarios where it may underperform for some reason other than obvious caseswhen model information is not available or where individual models are themselves bad.
Performance discrepancy between models leads to worse predictions.
Table 2 highlights asituation where SMC may underperform.
We see that SMC performs worse when there is highvariability in the performance of individual models.
For example, in the A priori setting, thereis a very large range in RMSE, from 39.6 all the way up to 81.9.
Since SMC does not attemptto evaluate the relative performances of the models, when there are models that just perform verybadly they can severely detract from SMC’s performance.
This highlights that SMC performs bestwhen all the models perform well in their respective domain , but that those domains are relativelydisjoint .
Could you do anything if you know (through perhaps a validation set) that some modelsperform very badly?
A simple solution would involve also calculating the weights given by BMA aswell, before averaging both sets of weights.
This would weight models globally by some level ofhow confident we are that the model is good as well as locally by how well we believe the modelwill be able to perform on a specific feature, balancing the potential causes of poor performance.
Domain overlaps result in marginal improvement.
To examine this point we shall revisit theregression example of earlier.
SMC assumes that the domains of the different models are at leastpartially different so that the feature dependant weights allow for selecting the model(s) that are mostappropriate.
This benefit is lost when all the model domains are the same, as we show in Figure 3 d- all the models are equally good across the feature space and so an ensemble will perform just aswell as SMC.
It is important to note SMC does not perform worse, it just does not perform better.
Discussion
Society and Ethics.
The setting of our method is primarily focused on situations where the sharingof data is in some way tricky or limited - a key example being the medical regime.
As such,our method is designed to be able to perform well in this area and so would hopefully have anoverwhelmingly positive impact.
Of course as with any method there is potential for it to be misused inan application which has damaging effects, but it seems unlikely that SMC poses any intrinsic danger.
Conclusions.
In this paper we have introduced the framework of Synthetic Model Combination - an instance-wise approach to unsupervised ensemble learning, having established thatthere are many cases when global ensembles are simply inadequate for meaningful predictions.
We additionally introduced a novel unsupervised representation learning method for the sparsehigh-dimensional setting, and shown the use of our method in both example synthetic problems and the real case study of estimating the effectiveness of vancomycin precision dosing.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See main text.
(b) Did you describe the limitations of your work?
[Yes] See Sections 4.4 and 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Our policy is to
release code upon paper acceptance.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix(c)
Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]