Autoinverse: Uncertainty Aware Inversion of Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural networks are powerful surrogates for numerous forward processes.
The inversion of such surrogates is extremely valuable in science and engineering.
The most important property of a successful neural inverse methodis the performance of its solutions when deployed in the real world, i.e., onthe native forward process (and not only the learned surrogate).
We propose Autoinverse, a highly automated approach for inverting neural networksurrogates.
Our main insight is to seek inverse solutions in the vicinity ofreliable data which have been sampled form the forward process and usedfor training the surrogate model.
Autoinverse finds such solutions by takinginto account the predictive uncertainty of the surrogate and minimizing itduring the inversion.
Apart from high accuracy, Autoinverse enforces thefeasibility solutions, comes with embedded regularization, and is initializationfree.
We verify our proposed method through addressing a set of real-worldproblems in control, fabrication, and design.
Our code and data are availableat: https://anonymous.4open.science/r/auto-inverse-BE88/README.md
Introduction
“... optimizing for the wrong thing — offering prayers to the bronze snake of data rather the largerforce behind it.”
With the deep learning breakthrough during the last decade, data-driven modeling using neuralnetwork based surrogates is now a standard practice in science and engineering.
These surrogatesstrive to imitate the behavior of a native forward process (NFP), such as a physics simulation, thatmaps a design into its performance 1 .
While forward processes are essential for troubleshootingand analysis, oftentimes their ultimate application lies in their inversion, i.e., the reverse process ofmapping performances into designs.
Despite the recent progress in inversion of neural networks within multiple disciplines [32, 15, 12, 33],a fundamental unaddressed question is still standing out.
Inversion of a surrogate model, fitted to thedata sampled from the NFP, is ultimately different than the inversion of the NFP itself.
The source of this gap could be explicit, such as the noise in measurements, or implicit, such as the poor samplingof the NFP.
Although the obtained solutions from inverting the surrogate can be evaluated on the NFP,none of the current neural inversion methods offers a tailored solution for addressing this important gap
Our main insight in this work is to expect and account for any potential mismatch between thedata, and consequently the surrogate, on the one hand and the NFP on the other.
Our proposedmethod, Autoinverse , realizes this vision by taking into account the predictive uncertainty of thesurrogate and minimizing it during the inversion.
Therefore, the obtained inversion solutions avoidthe unreliable regions within the training data.
We show that our Autoinverse strategy can augment existing neural inversion methods (bothoptimization-based and architecture-based approaches) with uncertainty compensation in a simpleand practical manner.
Autoinverse closes the gap between the surrogate and the NFP not throughattempting a perfect fit of the surrogate to the NFP, an onerous task, but by finding inverse solutionsin the vicinity of the reliable training data where the surrogate and the NFP are most similar.
Neuralinverse methods equipped with Autoinverse outperform their counterparts significantly on bothstandard data sampled from NFP and imperfect data, e.g., those corrupted by noise.
Apart fromhigh accuracy, Autoinverse enforces the feasibility solutions, comes with embedded regularization(freeing the inversion approaches from hand-crafted regularizations based on domain knowledge),and is initialization free.
It achieves all these properties in a highly automated manner and only witha light, intuitive tuning.
Related work
Neural Network Inversion We can divide neural network inversion approaches into two categories:inverse architectures , where we compute a network architecture that takes a given performance andmaps it into a (distribution of) design(s), and optimization , where we optimize for a design such thatit produces the desired performance.
Although the simplest inverse architecture can be attempted bytraining a neural network in the reverse direction, it fails because of the one-to-many nature of themapping.
Tandem networks use an inverse architecture by employing a pre-trained forward networkin order to compute a consistent loss.
The tandem approach has been developed independentlyacross different disciplines [21, 29, 30, 35] dating back (at least) to Tominaga [31].
Many inversearchitectures try to model the conditional posterior, p ( x | y ) , using variational methods [22, 17] basedon (conditional) variational auto-encoders [18].
These networks condition the design on the targetperformance and yield a distribution of solutions from which multiple samples could be drawn.
Kruseet al.
[19] show that the invertible neural network (INN), and its conditional variant, give the highestaccuracy in terms of both resimulation error and design posterior compared to a wide range of inversearchitectures.
In this method, a specialized architecture built upon normalizing flows [9] is trainedsimultaneously in both forward and inverse directions leading to a bijective mapping between designand performance spaces.
When using direct [30] optimization, gradient-based optimizers can be readily used as the neuralsurrogate is differentiable.
Ren et al.
[28] use stochastic gradient descent via backpropagationwith respect to the design variables, and present a highly accurate and practical method.
Theybenchmark their method, dubbed as neural adjoint ( NA ), against a set of inverse architectures andobtain significantly more accurate solutions.
Sun et al.
[30] showed a similar approach exceptusing a quasi-Newton method for optimization.
Ansari et al.
[2] push forward in this direction bydemonstrating that, for piecewise linear neural networks, e.g., those with ReLU activation, the directoptimization can be formulated as a mixed-integer linear program (MILP) and thus obtain globallyoptimal solutions.
While optimization methods are very accurate, their main disadvantage is theirperformance as they can be orders of magnitude slower than inverse architectures.
Neural Networks and Predictive Uncertainty While nerual networks are ubiquitous in almost allbranches of natural sciences, their weakness at quantifying predictive uncertainty impedes their use incrucial applications.
Using a Bayesian formalism [4], Bayesian neural networks (BNNs) [26, 8, 24],given the training data and a prior over network’s parameters, compute the posterior distribution of theparameters.
Having computed the posterior, the predictive uncertainty can be computed.
The inferencestep in computing BNNs is known to be computationally hard [16].
This explains the popularity ofsimpler methods for estimating predictive uncertainty, such as Monte Carlo dropout [10] and Deep Ensembles [20].
Deep Ensembles strikes a good balance between simplicity and practicality on theone hand and predictive performance on the other hand (Section 3.1).
Neural networks capable ofpredictive uncertainty are increasingly adopted in many different applications, such as reinforcement learning and active learning [11].
As we shall see, neural inversion is yet another domain that takesadvantage of this trend.  Method
Autoinverse is an easy-to-implement technique for augmenting neural inverse methods with uncertainty awareness.
Autoinverse achieves this goal by: 1) Replacing the surrogate with a newsurrogate capable of predictive uncertainty [23], and 2) Modifying the inverse cost function in orderto not only find accurate designs but also minimize the uncertainty.
After reviewing Deep Ensemblesas our surrogate (Section 3.1), we introduce two unertainty-aware methods each belonging to the twomain categories of inverse methods (Section 3.2 and 3.3). 3.1 Predictive uncertainty using Deep Ensemble m Deep Ensembles comprises of an ensemble of M neural networks ˆ f m eachcapable of a prediction µ m ( x ) and its associated uncertainty σ m ( x ) (see inset)in form of a Gaussian distribution N ( µ m ( x ) , σ m ( x )) .
The cost function fortraining each network in the ensemble is the negative log likelihood [27].
[Equation]
Intuitively, in case of the aleatoric uncertainty , i.e., noisy data, µ ( x ) fails to reliably predict y ∗ .
Hence, σ 2 ( · ) must increase in order to reduce the loss.
The first term ensures σ 2 ( · ) does not divergeto infinity.
Next step is to ensemble the results of all the networks into a single prediction and a single uncertainty.
Deep Ensembles [20] models the ensemble as a single Gaussian distribution N ( F µ ( x ) , F σ ( x ))approximating the mixture of M previously computed Gaussian distributions
[Equation]
The uncertainty of the ensemble can be decomposed into two uncertainties and be predicted for eachinput x , aleatoric ( F σA ( x ) ) and epistemic ( F σE ( x ) ).
Slight modification to Equation 2b enables usto control the behavior of the neural inversion by tuning the importance of these two uncertainties.
[Equation]
We introduce α and β as hyperparameters to adjust the relative significance of these two uncertainties.
Aleatoric uncertainty increases as the noise level in the training data increases.
Epistemic uncertaintymeasures the consensus between the M networks.
3.2 Uncertainty aware neural adjoint ( UANA )
Neural adjoint ( NA ) [28] uses the cost function L NA ( · ) to pushed designs x to have a performanceˆ f ( x ) (using neural surrogate ˆ f ( · ) ) as close as possible to a desired performance y ∗ :
[Equation]
[Equation]
D es i gn
E n se m b l e ...
Forward Network m m (a)
M i n i m i z e
D es i gn
E n se m b l e ...
P r e d i c t e d p e r f o r m a n ce
Forward Network
Inverse network
T a r g e t p
e r f o r m n ce fX m (b)
NA uses gradient descent to iteratively reduce the cost function, a scheme much like training neuralnetworks but with the input as the optimization variable instead of network’s weights and biases.
Equation 5 shows a single NA iteration with as the step size.
[Equation]
We propose to modify NA in order to take into account the uncertainty:
[Equation]
Equation 7 shows how one iteration of UANA requires the back-propagation using the ensemble of allgradients of M networks of Deep Ensembles:
[Equation]
where ˆ f m represents one of the networks in the ensemble.
Figure 1(a) depicts this collective procedurewhere each individual network in the ensemble votes for the direction where updating the design willlead to the minimum loss and uncertainty.
3.3 Uncertainty aware tandem ( UA-tandem )
Tandem is a representative of architecture-based methods in which we train an inverse network ˆ f − 1 ( · )in a manner resembling the encoder-decoder ([29–31]).
Unlike the encoder-decoder approach, westart with training the forward model ˆ f ( · ) .
Then we freeze the trainable parameters of ˆ f ( · ) and addthe inverse model
( ˆ f − 1 ( · ) ) in the position of the encoder and train it to decrease the cost function:
[Equation]
Once ˆ f − 1 ( · ) is trained we can simply query it to find the designs with our desired performance:
[Equation]
[Equation] Autoinverse modifies tandem by replacing ˆ f ( · ) with F µ ( · ) , and considering the uncertainty F σ ( · ) :
[Equation]
Figure 1(b) depicts the architecture of UA-tandem .
3.4 Implementation
Following [27], we take a 3-step procedure for implementing deep ensemble predictive uncertainty.
In practice, training the ensemble networks directly with negative log likelihood loss (Equation 1)is challenging. Instead, we start with training an ensemble of conventional networks with diverseactivation functions and MSE norm as its loss.
These networks are in fact the initialization of F µ ( · ) .
The next step is training F σ ( · ) and fine tuning F µ ( · ) jointly with the negative log likelihood loss(Equation 1).
Finally, we replace ˆ f ( · ) with F µ ( · ) and incorporate F σ ( · ) in the Autoinverse loss.
Evaluation
We evaluate the performance of Autoinverse via evaluating the existing neural inverse methodsand their uncertainty-aware counterparts.
In the paper, we focus on NA and UANA while tandem and
UA-tandem are evaluated mainly in Supplementary Materials (SM).
We perform the evaluation on aset of applications in robotics, and fabrication-oriented computational design.
We report the error byrunning the experiments 3 times to capture the variations.
More details are provided in each casestudy and in SM.
Equations 6 and 10 have two hyperparameters ( α and β ) that keep the balance between the MSE,the aleatoric uncertainty and the epistemic uncertainty.
We observe that with a larger epistemicuncertainty we obtain better results.
Exploiting this intuition, we tune these parameters for 3 differentsets of values for { α , β }: {{ 0 . 1 , 1 } , { 1 , 10 } , { 10 , 100 }} .
We then make two finer step depending onthe outcome of the former evaluation and choose the best set of weights.
We keep this budget of5 experiment runs for rival methods as well.
Typically, we use 10% of the target performance fortuning our inverse methods.
4.1 Experiments
Multi-joint robot is a simple inverse kinematics problem which is being used as a standard testfor neural inverse problems [19, 28].
In this problem, the design is the 1D position of the base ofthe multi-joint robot along with the 1D rotation of its three joints.
The inverse problem concernsfinding a combination of positions and angles for the base and the joints such that the tip of the armlands the target position, i.e., the performance ).
We follow [28] for setting up this experiment andits corresponding analytical equation used as the NFP.
The training data consists of 10,000 pair ofsamples generated by randomly sampling the NFP.
Spectral printer Spectral printing enables digital fabrication of the object’s appearance faithfully([13, 5]).
Unlike reproduction of the color (e.g., RGB), reproducing the spectrum ensures that theoriginal and the duplicate remain visually similar independent of the color of the light source.
Spectralprinting has various important applications specially in fine art reproduction using both 2D and 3Dprinting [25, 1, 29].
Deep neural networks are becoming the main computational tool for modelingthe spectral printing process specially when dealing with a large number of inks.
The final objectiveis to find the correct ink densities at each pixel that can best estimate the target 31D spectrum.
In this experiment, we create the NFP by simulating a printer using an ensemble of 20 neuralnetworks.
The design space comprises the ink densities and our spectral printer NFP predictsthe spectrum of the resulting color.
We use real, measured data from an 8-channel printer with
EPSON inks including Cyan (C), Magenta (M), Yellow (Y), Black (K), Light black (LK), Light lightblack (LLK), Light Cyan (LC), and Light Magenta (LM).
The light inks, added to the standard CMYKto improve the print quality, introduce significant multi-modality.
All networks in the ensemble NFPare trained on 40,000 printed patches [1] consisting of different ink-density combinations and theircorresponding spectra.
The ensembling is intended for an accurate NFP and is independent of ouruse of Deep Ensembles for computing the uncertainty.
The visual nature of spectral printermakes it attractive for analyzing different methods.
We release this NFP to the public to add anotherexample to the neural inversion testbed.
Soft robots are made of soft, flexible materials.
This unique property has made them suitablecandidates for interaction with humans in applications like minimally invasive surgery and advancedprosthetics [7].
Unlike multi-joint robot with a limited number of rotating joints, each segment [Table] of the soft robot is a potential actuator that through their contraction and expansion can determine therobot’s final shape.
This inverse kinematics problem is typically solved through partial differentialequations ([34]).
In order to accelerate the solve time of this inverse kinematics problem, Sun et al.[30] proposed a neural surrogate modeling of the problem and its inversion via tandem .
The designspace in this problem consists of the contraction or expansion of 40 controllable soft edge segments.
The superposition of all the actuations determines the final deformation position of the soft robot viathe position of its 206 vertices.
We use an FEM-based simulation ([34, 14] as our NFP.
We model the relationship between theactuations and the final shape of the soft robot with a neural-network surrogate.
Our goal is to solvethe neural inversion to find a suitable set of actuations (design) that brings the tip of the soft robot tothe target position (performance).
The training data consists of 50,000 samples queried by randomsampling the actuation with an expansion ratios between -0.2 and 0.2 [30].
The designs are thenevaluated on the FEM-based NFP to calculate their deformations.
4.2 Quantitative comparison of different neural inversion methods
We evaluate the accuracy of a set of neural inversion methods on multi-joint robot in terms ofboth the NFP and the surrogate errors.
The surrogate error is the difference between the predictionof the neural network and the target performance.
The NFP error is the difference between thetarget and the performance of the generated design evaluated on the NFP.
In addition to the coremethods described so far, we evaluate mixed-integer neural inversion ( MINI ) [2], and invertible neuralnetworks ( INN ) [3].
Table 1 summarizes the inversion results on 1000 randomly sampled target locations for the multi-jointrobotic arm.
We keep our evaluation fair by setting the capacity of the neural surrogates comparablewherever possible.
For instance, all methods except MINI have around 3 million parameters (see SMfor more details).
We also kept the fairness in tuning stage.
Methods with hyper parameters like UANA, UA-tandem are tuned in 5 stages by hand.
Alternatively, the methods without hyper parameters ( NA,tandem ) are given 5 × extra budget for inversion. We run NA and tandem 5 × and choose the modelthat generates the best NFP error.
We repeat this process 3 × and report the standard error.
MILPand INN are fundamentally different in nature.
MILP finds the global optimum and as a result doesnot need tuning.
INN has a latent space which we can sample to generate diverse designs that cangenerate our target.
We sample the latent space 1024 times for all 1000 targets then we evaluate themon the NFP and report the best NFP error.
We repeat this process three times to generate the standarderror.
Table 1 shows the outstanding accuracy of the inverse methods that adopt Autoinverse , i.e., UANAand UA-tandem , in terms of the NFP error.
It also demonstrate how even a perfect surrogate error(e.g., MINI ) does not guarantee accurate solutions when tested on the NFP.
A second look at Table 1reveals further interesting insights.
Although NA obtains notably lower surrogate error than itsuncertainty-aware counterpart ( UANA ), it performs significantly worse in terms of the NFP error.
The main reason for this trend is that, when optimizing the surrogate, UANA is not only concernedwith finding accurate designs leading to a small accuracy gap between the target and candidateperformances, i.e., the surrogate error, but also with those designs featuring low uncertainty (throughthe uncertainty term in the cost function in Equations 3).
Therefore, UANA achieves high accuracyin terms of the essential NFP error at the cost of increasing the inconsequential surrogate error.
Furthermore, UA-tandem , for example, achieves better performance than NA .
This is a remarkableresult for an architecture-based inverse method that is tremendously faster than the optimization-based NA .
In terms of computation time, UANA takes roughly 3 × longer during inversion compared to NA .
Thelonger training time for the ensemble is a less important concern as it is trivially parallelizable.
UA-tandem on the other hand takes roughly 10 times longer than tandem to train. However, the [Table] N o i s y d a t a
S p a r s e d a t a
[Table]
training time is amortized during the inversion of many samples using the inverse architecture.
Wereport extensively the time performance and the details of the training in SM.
4.3 Neural inversion in the presence of imperfect data
One of the main advantages of Autoinverse appears in scenarios when the training data suffers fromnoise, e.g., due to noise in measurement or poor sampling, e.g., due to sparsity in some regions.
Weevaluate the performance of Autoinverse on imperfect training data using NA and UANA ( tandemand UA-tandem are evaluated under the same configuration in the SM).
Locally sparse data We sample both soft robot and spectral printer NFPs (Section 4.1)in a way that the data does not contain any samples from one of the inputs in a specific interval.
Forspectral printer , we would like to find ink densities to reproduce the spectra of the colors in thepainting in Figure 4 (made of 3568 color spectra).
We sample the printer channels at 0 (no ink), 0.05,0.1, 0.5, and 1 (full ink) densities to form the standard training data (with no known uncertainty).
Wecreate a partially sparse dataset similar to the standard one except for the Light Cyan (LC) channelfor which we only have samples at 0, 0.05 and 0.1.
Table 2 shows that while for the standard dataset
UANA finds inverse solutions that include the LC channel frequently (437 times), for the sparse datasetit avoids this channel completely and compensate for it using the Cyan channel.
UANA is able toavoid this channel as the epistemic uncertainty increase in sparse regions of dataset (see SM for moredetails).
In soft robot we sample its 16th (among 40) controllable edge only in negative range (contractiononly).
We then use the trained network to invert 1000 test samples.
Figure 2 shows the distribution ofeach edge for 1000 inversion tasks.
We have plotted the distribution for the 16th edge as well as for 7other randomly chosen edges.
As evident from Figure 2 bottom row, the inversion was reluctant tochoose designs with positive actuation for this edge.
Locally noisy data set With the same problem configuration as before, we would like to test therobustness of Autoinverse on a data set locally corrupted with noise.
We start with a standarddata set and inject Gaussian noise N (0 , 0 . 1) to the spectrum of the samples with more than 0 .
4 LCdensity.
Table 2 shows how after introducing noise to light cyan, the network avoids that channel andcompensates it by using more cyan instead.
In soft robot we corrupt the final shape of the soft [Figure] [Figure] ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) (b) robot with Gaussian noise N (0 , 0 . 5) for all the shapes where the 16th edge has positive actuation.
As we can see in Figure 2 (top row), the inversion has completely avoided positive actuation for thisedge.
4.4 Autoinverse brings AutoML to neural inversion
Autoinverse incorporates feasibility.
Deep Ensembles produces high epistemic uncertainty outsidethe distribution of the training data.
This includes the regions where the NFP is not defined andthus not sampled.
For example, ink densities outside [0 , 1] are not printable.
If such cases ariseduring inversion, they are clipped to [0 , 1] [1].
In such regions, networks in the ensemble do not agreeand thus the epistemic uncertainty increases.
Autoinverse automatically avoids these unfeasibleregions.
Revisiting the painting reproduction experiment using spectral printer , we observe how NA , after optimizing for the ink densities, their clipping and feeding them to spectral printer
NFP, results in a poor reproduction of the original painting (Figure 4 left).
UANA , on the other hand,achieves spectacular reproduction accuracy.
This is explained by the plot in Figure 4 right, showingthe distribution of ink densities obtained from both methods.
Autoinverse has built-in regularization.
We incorporate regularization into inversion methods inorder to obtain solutions that, among other things, agree with the observations and follow a certainstatistical distribution.
Oftentimes, regularization is case specific, requires human knowledge, andcomes with unexpected side effects.
Here we show that Autoinverse follows the distributionof the training data naturally, by taking into account the epistemic uncertainty.
We validate thispoint using both spectral printer and soft robot experiments.
We show that Autoinversewithout any explicit regularization performs better or on par with its counterpart inversion methodsthat incorporate regularization.
In the spectral printer experiment, we evaluate the effect of the boundary loss , originallyproposed as a generic regularization for NA to limit the designs in a box constraint (see [28] and
SM).
The boundary loss is added to Equation 4 and weighted using a hyperparameter.
We tunethis parameter with the same tuning budget we allocate for tuning the uncertainty weights (5 set ofinversions on evaluation data).
In Figure 4, we observe that although NA with boundary loss improvesthe distribution of ink densities within the valid region ( [0 , 1] ), it still trails the regularization-free
UANA significantly.
Regularization is less intuitive for soft robot as the superposition of all actuations determines thefinal shape and whether it is physically plausible.
In [30], the regularization is a smoothness term thatkeeps actuation values near each other (see SM).
Figure 3(b)(1) demonstrates how NA fails withoutregularization to control the robot with a reasonable deformation.
Once the regularization is addedto NA , the designs become physically meaningful (Figure 3(b)(2)).
Figure 3(b)(3) shows how UANAperforms comparably without any regularization. [Figure] NA UANA
Valid design
NA w i t h bound a r y l o ss
Autoinverse is initialization free.
Sensitivity to the initialization is a widely known issue in nonconvex optimizations.
Despite equipping NA with the smoothness term in soft robot , Figure 3(a)demonstrates how an incorrect initialization can result in solutions with seemingly good surrogate andregularization loss (see SM) but in the infeasible region of design space (Figure 3(b)(4)).
UANA withthe same incorrect initialization and without any regularization, leads to soft robot designs that reachthe target location accurately and produce plausible deformations (Figure 3(b)(5)).
For UANA , thesolver first reduces the main contributor to the optimization objective, i.e., the epistemic uncertainty.
Once a region with a reasonably small uncertainty is reached, the accuracy term starts to take effectand a desirable solution within the valid range of the design space is found.
4.5 Ablation studies
Separating the ensembling effect Although in both NA and UANA we use surrogates with similarcapacity (number of network parameters), one could argue that the higher performance of the UANAcome from the ensemble architecture of its surrogate.
We detach the impact of ensembling anduncertainty awareness on the inversion performance by implementing NA ensemble where, insteadof a single, high-capacity forward neural network, it uses an ensemble of networks.
The inversionprocedure is kept identical to NA (see SM).
We employ NA ensemble in multi-joint robotwith the same configurations as in Section 4.2.
The surrogate and NFP error for NA ensemble are(3 .
30 ± 0 . 59) × 10 − 9 and (1 . 17 ± 0 .
32) × 10 − 4 , respectively.
Comparing these values with thosein Table 1, NA ensemble shows a slight improvement over NA but is significantly outperformed by
UANA .
Diversity of activation functions in Deep Ensembles Originally Deep Ensembles uses a seimialrset of networks with identical activation function [20].
In practice, we observe that a diverse setof activation layers leads to a better performance of Autoinverse .
Different activation functionsgenerate different behaviours, show higher disagreement where the training data is under-representedand, thus, result in a clear landscape of epistemic uncertainty.
For all experiments in this work, weuse a diverse range of activation functions (see SM).
As an ablation, we run UANA on spectralprinter using sparse data (same configuration as Section 4.3) but with ReLU as the only activationlayer.
In contrast to Table 2, UANA (with ReLU only activation) does not completely avoid the sparsedomain and delivers 19 ± 4 solutions that contain LC with densities ≥ 0 . 4 .
Discussion
We look forward to see Autoinverse adopted for more inverse architectures, beyond tandem andmore optimizations (beyond first-order NA ).
The Autoinverse cost function is multi-objective.
It ishighly interesting to capture the trade-off between accuracy and uncertainty using the Pareto front. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes] If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)? [Yes] (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets? [Yes] (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]