Autoinverse: Uncertainty Aware Inversion of Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural networks are powerful surrogates for numerous forward processes.
The inversion of such surrogates is extremely valuable in science and engineering.
The most important property of a successful neural inverse methodis the performance of its solutions when deployed in the real world, i e., onthe native forward process (and not only the learned surrogate).
We propose Autoinverse, a highly automated approach for inverting neural networksurrogates.
Our main insight is to seek inverse solutions in the vicinity ofreliable data which have been sampled form the forward process and used fortraining the surrogate model.
Autoinverse finds such solutions by taking intoaccount the predictive uncertainty of the surrogate and minimizing it duringthe inversion.
Apart from high accuracy, Autoinverse enforces the feasibilityof solutions, comes with embedded regularization, and is initialization free.
We verify our proposed method through addressing a set of real-world problems in control, fabrication, and design.
Our code and data are available at:https://gitlab.mpi-klsb.mpg.de/nansari/autoinverse
Introduction
“... optimizing for the wrong thing — offering prayers to the bronze snake of data rather the largerforce behind it.”
With the deep learning breakthrough during the last decade, data-driven modeling using neuralnetwork based surrogates is now a standard practice in science and engineering.
These surrogatesstrive to imitate the behavior of a native forward process (NFP), such as a physics simulation, thatmaps a design into its performance 1 .
While forward processes are essential for troubleshootingand analysis, oftentimes their ultimate application lies in their inversion, i.e., the reverse processof mapping performances into designs.
Despite the recent progress in inversion of neural networkswithin multiple disciplines [33, 15, 12, 34], a fundamental unaddressed question is still standingout.
Inversion of a surrogate model, fitted to the data sampled from the NFP, is ultimately differentthan the inversion of the NFP itself.
The source of this gap could be explicit, such as the noise inmeasurements, or implicit, such as the poor sampling of the NFP.
Although the obtained solutionsfrom inverting the surrogate can be evaluated on the NFP, none of the current neural inversion methodsoffers a tailored solution for addressing this important gap.
Our main insight in this work is to expect and account for any potential mismatch between the data, and consequently the surrogate, on the one hand and the NFP on the other.
Our proposed method, Autoinverse , realizes this vision by taking into account the predictive uncertainty of the surrogateand minimizing it during the inversion.
Therefore, the inverted solutions avoid the unreliable regionswithin the training data.
We show that our Autoinverse strategy can augment existing neural inversion methods (bothoptimization-based and architecture-based approaches) with uncertainty compensation in a simpleand practical manner.
Autoinverse closes the gap between the surrogate and the NFP not throughattempting a perfect fit of the surrogate to the NFP, an onerous task, but by finding inverse solutionsin the vicinity of the reliable training data where the surrogate and the NFP are most similar.
Neuralinverse methods equipped with Autoinverse outperform their counterparts significantly on bothstandard data sampled from the NFP and imperfect data, e.g., those corrupted by noise.
Apartfrom high accuracy, Autoinverse enforces the feasibility of solutions, comes with embeddedregularization (freeing the inversion approaches from hand-crafted regularizations based on domainknowledge), and is initialization free.
It achieves all these properties in a highly automated mannerand only with a light, intuitive tuning.
Related work
Neural Network Inversion We can divide neural network inversion approaches into two maincategories. First, inverse architectures , where we compute a network architecture that takes a givenperformance and maps it into a (distribution of) design(s). Second, direct optimization [31], wherewe optimize for a design such that it produces the desired performance.
Although the simplestinverse architecture can be attempted by training a neural network in the reverse direction, it failsbecause of the one-to-many nature of the mapping.
Tandem networks use an inverse architecture byemploying a pre-trained forward network in order to compute a consistent loss.
The tandem approachhas been developed independently across different disciplines [22, 30, 31, 36] dating back (at least)to Tominaga [32].
Many inverse architectures try to model the conditional posterior, p ( x | y ) , usingvariational methods [23, 18] based on (conditional) variational auto-encoders [19].
These networkscondition the design on the target performance and yield a distribution of solutions from whichmultiple samples could be drawn.
Kruse et al.
[20] show that the invertible neural networks (INNs),built upon normalizing flows [9], give the highest accuracy in terms of both surrogate error and designposterior compared to a wide range of inverse architectures.
When using direct optimization methods, gradient-based optimizers can be readily used as the neuralsurrogate is differentiable.
Ren et al.
[29] use stochastic gradient descent via backpropagationwith respect to the design variables, and present a highly accurate and practical method.
Theybenchmark their method, dubbed as neural adjoint ( NA ), against a set of inverse architectures andobtain significantly more accurate solutions.
Sun et al.
[31] showed a similar approach exceptusing a quasi-Newton method for optimization.
Ansari et al.
[2] push forward in this direction bydemonstrating that, for piecewise linear neural networks, e.g., those with ReLU activation, the directoptimization can be formulated as a mixed-integer linear program (MILP) and thus obtain globallyoptimal solutions.
While optimization methods are very accurate, their main disadvantage is theirperformance as they can be orders of magnitude slower than inverse architectures.
Neural Networks and Predictive Uncertainty While neural networks are ubiquitous in almost allbranches of natural sciences, their weakness at quantifying predictive uncertainty impedes their use incrucial applications.
Using a Bayesian formalism [4], Bayesian neural networks (BNNs) [27, 8, 25],given the training data and a prior over network’s parameters, compute the posterior distribution of theparameters.
Having computed the posterior, the predictive uncertainty can be computed.
The inferencestep in computing BNNs is known to be computationally hard [16].
This explains the popularity ofsimpler methods for estimating predictive uncertainty, such as Monte Carlo dropout [10] and Deep
Ensembles [21].
Deep Ensembles strikes a good balance between simplicity and practicality on theone hand and predictive performance on the other hand (Section 3.3).
Neural networks capable ofpredictive uncertainty are increasingly adopted in many different applications, such as reinforcementlearning and active learning [11].
As we shall see, neural inversion is yet another domain that takes advantage of this
D es i gn
E n se m b l e ...
Forward Network m m m m m m m (a)
M i n i m i z e
D es i gn
E n se m b l e ...
P r e d i c t e d p e r f o r m a n ce
Forward Network
Inverse network
T a r g e t p
e r f o r m n ce fX m m m m m (b)
Method
Autoinverse is an easy-to-implement technique for augmenting neural inverse methods with uncertainty awareness.
Autoinverse achieves this goal by, first, training a surrogate capable of predictiveuncertainty [24].
Second, using this trained surrogate during the inversion, it modifies the inversecost function in order to not only find accurate designs but also minimize the uncertainty. Weapply Autoinverse on two inverse methods belonging to the two main neural inversion categories,i.e., optimization- and architecture-based in Sections 3.1 and 3.2, respectively. As we will see in Section 3.3, we rely on established methods [21, 17] to train networks equipped with predictiveuncertainty.
3.1 Uncertainty aware neural adjoint ( UANA )
Given a pretrained neural surrogate ˆ f ( · ) , neural adjoint ( NA ) [29] is an inverse method that uses thecost function L NA ( · ) to push designs x to have a performance ˆ f ( x ) as close as possible to a desiredperformance y ∗ :
[Equation]
NA uses gradient descent to iteratively reduce the cost function, a scheme much like training neuralnetworks but with the input as the optimization variable instead of network’s weights and biases.
Equation 2 shows a single NA iteration with δ as the step size.
[Equation]
Autoinverse proposes to perform the inversion using a pretrained BNN. We use Deep Ensembles [21] made of M neural networks capable of a prediction F µ ( x ) , as well as its aleatoric F σA ( x )and epistemic F σE ( x ) uncertainties (Section 3.3). Aleatoric uncertainty increases as the noiselevel in the training data increases. Epistemic uncertainty measures the uncertainty in the model.
Autoinverse modifies NA such that we obtain solutions x that have performances close to the targetperformance y ∗ while resulting in small aleatoric and epistemic uncertainties:
[Equation]
We introduce α and β as hyperparameters to adjust the relative significance of aleatoric and epistemicuncertainties, respectively.
Equation 4 shows how one iteration of UANA requires the back-propagation using the ensemble of allgradients of M networks of Deep Ensembles:
[Equation]
where ˆ f m represents one of the networks in the ensemble.
Figure 1(a) depicts this collective procedurewhere each individual network in the ensemble votes for the direction where updating the design willlead to the maximal accuracy and minimal uncertainty.
3.2 Uncertainty aware tandem ( UA-tandem )
Tandem is a representative of architecture-based methods in which we train an inverse network ˆ f − 1 ( · )in a manner resembling the encoder-decoder architecture ([30–32]).
Unlike the encoder-decoderapproach, we start with training the forward model ˆ f ( · ) .
Then we freeze the trainable parameters ofˆ f ( · ) and train the inverse model
ˆ f − 1 ( · ) in the position of the encoder in order to decrease the costfunction: [Equation]
[Equation]
Once ˆ f − 1 ( · ) is trained we can simply query it to find designs with our desired performances:
[Equation]
The uncertainty-aware tandem ( UA-tandem ) follows the same procedure except that it replaces ˆ f ( · )with F µ ( · ) .
Additionally, it includes the uncertainties in the loss:
[Equation]
Figure 1(b) depicts the architecture of UA-tandem .
3.3 Predictive uncertainty using Deep Ensembles
m m m Deep Ensembles comprises of an ensemble of M neural networks
ˆ f m eachcapable of a prediction µ m ( x ) and its associated uncertainty σ m ( x ) (see inset)in form of a Gaussian distribution N ( µ m ( x ) , σ m ( x )) . The cost function fortraining each network in the ensemble is the negative log likelihood [28].
[Equation] Intuitively, in case of the aleatoric uncertainty µ m ( x ) fails to reliably predict y ∗ . Hence, σ 2 m ( · ) mustincrease to reduce the loss while the first term ensures σ 2 m ( · ) does not diverge to infinity. The next step is to ensemble the results of all the networks into a single prediction and a single uncertainty.
Deep Ensembles [21] models the ensemble as a single Gaussian distribution N ( F µ ( x ) , F σ ( x ))approximating the mixture of M previously computed Gaussian distributions
[Equation] The uncertainty of the ensemble can be decomposed into two input-dependent uncertainties, i.e.,aleatoric F σA ( x ) and epistemic F σE ( x ) through slight modification to Equation 9b [17]. This enablesus to control the behavior of the neural inversion by tuning the relative importance of these twouncertainties. [Equation]
Evaluation
We evaluate the performance of Autoinverse via evaluating the existing neural inverse methodsand their uncertainty-aware counterparts.
In the paper, we focus on NA and UANA while tandem and
UA-tandem are evaluated mainly in Supplementary Materials (SM).
We perform the evaluation on aset of applications in robotics, and fabrication-oriented computational design.
We report the error byrunning the experiments 3 times to capture the variations.
More details are provided in each casestudy and in SM.
Equations 3 and 7 have two hyperparameters ( α and β ) that keep the balance between the MSE,the aleatoric uncertainty and the epistemic uncertainty.
We observe that with a larger epistemicuncertainty we obtain better results.
Exploiting this intuition, we tune these parameters for 3 differentsets of values for { α , β }: {{ 0 . 1 , 1 } , { 1 , 10 } , { 10 , 100 }} .
We then make two finer step depending onthe outcome of the former evaluation and choose the best set of weights.
We keep this budget of 5experiment runs for the rival methods as well.
Typically, we use 10% of the target performance fortuning our inverse methods.
4.1 Experiments
Multi-joint robot is a simple inverse kinematics problem which is being used as a standard testfor neural inverse problems [20, 29].
In this problem, the design is the 1D position of the base ofthe multi-joint robot along with the 1D rotation of its three joints.
The inverse problem concernsfinding a combination of positions and angles for the base and the joints such that the tip of the armlands the target position, i.e., the performance .
We follow [29] for setting up this experiment andits corresponding analytical equation used as the NFP.
The training data consists of 10,000 pair ofsamples generated by randomly sampling the NFP.
Spectral printer Spectral printing enables digital fabrication of the object’s appearance faithfully([13, 5]).
Unlike reproduction of the color (e.g., RGB), reproducing the spectrum ensures that theoriginal and the duplicate remain visually similar independent of the color of the light source.
Spectralprinting has various important applications specially in fine art reproduction using both 2D and 3Dprinting [26, 1, 30].
Deep neural networks are becoming the main computational tool for modelingthe spectral printing process specially when dealing with a large number of inks.
The final objectiveis to find the correct ink densities at each pixel that can best estimate the target 31D spectrum.
In this experiment, we create the NFP by simulating a printer using an ensemble of 20 neuralnetworks.
The design space comprises the ink densities and our spectral printer NFP predictsthe spectrum of the resulting color.
We use real, measured data from an 8-channel printer with
EPSON inks including Cyan (C), Magenta (M), Yellow (Y), Black (K), Light black (LK), Light lightblack (LLK), Light Cyan (LC), and Light Magenta (LM).
The light inks, added to the standard CMYKto improve the print quality, introduce significant multi-modality.
All networks in the ensemble NFPare trained on 40,000 printed patches [1] consisting of different ink-density combinations and theircorresponding spectra.
The ensembling is intended for an accurate NFP and is independent of ouruse of Deep Ensembles for computing the uncertainty.
The visual nature of spectral printermakes it attractive for analyzing different methods.
We release this NFP to the public to add anotherexample to the neural inversion testbed.
Soft robots are made of soft, flexible materials.
This unique property has made them suitablecandidates for interaction with humans in applications like minimally invasive surgery and advancedprosthetics [7].
Unlike multi-joint robot with a limited number of rotating joints, each segmentof the soft robot is a potential actuator that through their contraction and expansion can determine therobot’s final shape.
This inverse kinematics problem is typically solved through partial differentialequations ([35]).
In order to accelerate the solve time of this inverse kinematics problem, Sun et al.[31] proposed a neural surrogate modeling of the problem and its inversion via tandem .
The designspace in this problem consists of the contraction or expansion of 40 controllable soft edge segments.
The superposition of all the actuations determines the final deformation position of the soft robot viathe position of its 206 vertices.
We use an FEM-based simulation ([35, 14]) as our NFP.
We model the relationship between theactuations and the final shape of the soft robot with a neural-network surrogate.
Our goal is to solve [Table] the neural inversion to find a suitable set of actuations (design) that brings the tip of the soft robot tothe target position (performance).
The training data consists of 50,000 samples queried by randomsampling the actuation with an expansion ratios between -0.2 and 0.2 [31].
The designs are thenevaluated on the FEM-based NFP to calculate their deformations.
4.2 Quantitative comparison of different neural inversion methods
We evaluate the accuracy of a set of neural inversion methods on multi-joint robot in terms ofboth the NFP and the surrogate errors.
The surrogate error is the difference between the ‘re-prediction’of the obtained inverse solution by the surrogate neural network and the target performance.
The NFPerror is the difference between the target and the performance of the generated design evaluated onthe NFP.
In addition to the core methods described so far, we evaluate mixed-integer neural inversion( MINI ) [2], and the invertible neural network ( INN ) [3].
Table 1 summarizes the inversion results on 1000 randomly sampled target locations for the multi-jointrobotic arm.
We keep our evaluation fair by setting the capacity of the neural surrogates comparablewherever possible.
For instance, all methods except MINI have around 3 million parameters (see SMfor more details).
We also kept the fairness in tuning stage.
Methods with hyperparameters, like UANAand UA-tandem , are tuned in 5 stages by hand.
Alternatively, the methods without hyperparameters( NA, tandem ) are given 5 × extra budget for inversion: We run NA and tandem 5 × and choose themodel that generates the best NFP error.
We repeat this process 3 × and report the standard error.
MILP and INN are fundamentally different methods.
MILP finds the global optimum and thus doesnot need tuning.
INN has a latent space which we can sample to generate diverse designs.
We sample INN’s latent space 1024 times for all 1000 targets, evaluate them on the NFP, and report the best NFPerror.
We repeat this process three times to generate the standard error.
Table 1 shows the outstanding accuracy of the inverse methods that adopt Autoinverse , i.e., UANAand UA-tandem , in terms of the NFP error.
It also demonstrate how even a perfect surrogate error (e.g., MINI ) does not guarantee accurate solutions when tested on the NFP.
A second look at Table 1 revealsfurther interesting insights.
Although NA obtains notably lower surrogate error than its uncertaintyaware counterpart ( UANA ), it performs significantly worse in terms of the NFP error.
The main reasonfor this trend is that, when optimizing the surrogate, UANA is not only concerned with finding accuratedesigns leading to a small accuracy gap between the target and candidate performances, i.e., thesurrogate error, but also with those designs featuring low uncertainty (through the uncertainty term in Equations 10).
Therefore, UANA achieves high accuracy in terms of the essential NFP error at the costof worsening the inconsequential surrogate error.
Furthermore, UA-tandem , for example, achievesbetter performance than NA .
This is a remarkable result for an architecture-based method given it isextremely faster than the optimization-based NA .
We report extensively the time performance and thedetails of the training for this and next experiments in SM.
4.3 Neural inversion in the presence of imperfect data
One of the main advantages of Autoinverse appears in scenarios where the training data suffersfrom noise, e.g., measurement nois or poor sampling, e.g., sparsity in some regions.
We evaluatethe performance of Autoinverse on imperfect training data using NA and UANA ( tandem and UA-tandem are evaluated under the same configuration in the SM).
data We sample both soft robot and spectral printer NFPs (Section 4.1)in a way that the data does not contain any samples from one of the inputs in a specific interval.
Forspectral printer , we would like to find ink densities to reproduce the spectra of the colors in the [Table] N o i s y d a t a S p a r s e d a t a
[Table] painting in Figure 4 (made of 3568 color spectra). We sample the printer channels at 0 (no ink), 0.05,0.1, 0.5, and 1 (full ink) densities to form the standard training data (with no known uncertainty).
Wecreate a partially sparse dataset similar to the standard one except for the Light Cyan (LC) channelfor which we only have samples at 0, 0.05 and 0.1.
Table 2 shows that while for the standard dataset
UANA finds inverse solutions that include the LC channel frequently (437 times), for the sparse datasetit avoids this channel completely and compensate for it using the Cyan channel.
UANA is able toavoid this channel as the epistemic uncertainty increase in sparse regions of dataset (see SM for moredetails).
In soft robot we sample its 16th (among 40) controllable edge only in negative range (contractiononly).
We then use the trained network to invert 1000 test samples.
Figure 2 shows the distribution ofeach edge for 1000 inversion tasks.
We have plotted the distribution for the 16th edge as well as for 7other randomly chosen edges.
As evident from Figure 2 bottom row, the inversion was reluctant tochoose designs with positive actuation for this edge.
Locally noisy data set With the same problem configuration as before, we would like to test therobustness of Autoinverse on a data set locally corrupted with noise.
We start with a standarddata set and inject Gaussian noise N (0 , 0 . 1) to the spectrum of the samples with more than 0 .
4 LCdensity.
Table 2 shows how after introducing noise to light cyan, the network avoids that channel andcompensates it by using more cyan instead.
In soft robot we corrupt the final shape of the softrobot with Gaussian noise N (0 , 0 . 5) for all the shapes where the 16th edge has positive actuation.
As we can see in Figure 2 (top row), the inversion has completely avoided positive actuation for thisedge.
4.4 Autoinverse brings AutoML to neural inversion
Autoinverse incorporates feasibility.
Deep Ensembles produces high epistemic uncertainty outsidethe distribution of the training data.
This includes the regions where the NFP is not defined andthus not sampled.
For example, ink densities outside [0 , 1] are not printable.
If such cases ariseduring inversion, they are clipped to [0 , 1] [1].
In such regions, networks in the ensemble do not agreeand thus the epistemic uncertainty increases.
Autoinverse automatically avoids these unfeasibleregions.
In order to simulate the result of the inversion, after clipping the out of range densities wefeed them into the NFP.
As evident from Figure 4 left, inversion via NA results in a poor reproduction [Figure] [Figure] of the original painting.
UANA , on the other hand, achieves spectacular reproduction accuracy.
This isexplained by the plot in Figure 4 right, showing the distribution of ink densities obtained from bothmethods.
Autoinverse has built-in regularization.
We incorporate regularization into inversion methods inorder to obtain solutions that, among other things, agree with the observations and follow a certainstatistical distribution.
Oftentimes, regularization is case specific, requires human knowledge, andcomes with unexpected side effects.
Here we show that Autoinverse follows the distribution of thetraining data naturally, by taking into account the epistemic uncertainty.
We validate this point usingboth spectral printer and soft robot experiments.
We show that Autoinverse without anyexplicit regularization performs better or on par with its counterpart inversion methods equipped withregularization.
In the spectral printer experiment, we evaluate the effect of the boundary loss , originallyproposed as a generic regularization for NA to limit the designs in a box constraint (see [29] and
SM).
The boundary loss is added to Equation 1 and weighted using a hyperparameter.
We tunethis parameter with the same tuning budget we allocate for tuning the uncertainty weights (5 set ofinversions on evaluation data).
In Figure 4, we observe that although NA with boundary loss improvesthe distribution of ink densities within the valid region ( [0 , 1] ), it still trails the regularization-free
UANA significantly.
Regularization is less intuitive for soft robot as the superposition of all actuations determines thefinal shape and whether it is physically plausible.
In [31], the regularization is a smoothness term thatkeeps actuation values near each other (see SM).
Figure 3(b)(1) demonstrates how NA fails withoutregularization to control the robot with a reasonable deformation.
Once the regularization is addedto NA , the designs become physically meaningful (Figure 3(b)(2)).
Figure 3(b)(3) shows how UANAperforms comparably without any regularization.
Autoinverse is initialization free.
Sensitivity to the initialization is a widely known issue in nonconvex optimizations.
Despite equipping NA with the smoothness term in soft robot , Figure 3(a)demonstrates how an incorrect initialization can result in solutions with seemingly good surrogateand regularization loss (see SM) but in the infeasible region of design space (Figure 3(b)(4)).
UANAwith the same incorrect initialization and without any regularization, leads to soft robot designs thatreach the target location accurately and produce plausible deformations (Figure 3(b)(5)).
For UANA ,the solver starts with reducing the main contributor to the optimization objective, i.e., the epistemicuncertainty.
Once a region with a reasonably small uncertainty is reached, the accuracy term starts totake effect and a desirable solution within the valid range of the design space is found.
4.5 Ablation studies
Separating the ensembling effect Although in both NA and UANA we use surrogates with similarcapacity (network size), one could argue that the higher performance of the UANA comes from [Figure] NA UANA Valid design
NA w i t h bound a r y l o ss the ensemble architecture of its surrogate. We detach the impact of ensembling and uncertaintyawareness on the inversion performance by implementing NA ensemble where, instead of a single,large forward neural network, it uses an ensemble of networks.
The inversion procedure is identicalto NA (see SM).
We employ NA ensemble in multi-joint robot with the same configurationsas in Section 4.2.
The surrogate and NFP error for NA ensemble are (3 . 30 ± 0 . 59) × 10 − 9 and(1 . 17 ± 0 .
32) × 10 − 4 , respectively.
Comparing these values with those in Table 1, NA ensembleshows a slight improvement over NA but is significantly outperformed by UANA .
Diversity of activation functions in Deep Ensembles Originally Deep Ensembles uses a similarset of networks with identical activation function [21].
In practice, we observe that a diverse set ofactivation layers leads to a better performance of Autoinverse .
Different activation functions generate different behaviours, show higher disagreement where the training data is under-represented and,thus, result in an accurate estimation of the landscape of epistemic uncertainty.
For all experimentsin this work, we use a diverse range of activation functions (see SM).
As an ablation, we run UANAon spectral printer using sparse data (same configuration as Section 4.3) but with ReLU as theonly activation layer.
In contrast to Table 2, UANA (with ReLU only activation) does not completelyavoid the sparse domain and delivers 19 ± 4 solutions that contain LC with densities ≥ 0 . 4 .
Discussion
The Autoinverse cost function is multi-objective.
In addition to finding a single solution througha weighted combination of these objectives (what we have seen so far), we capture the trade-offbetween accuracy and uncertainty using the Pareto front (see SM).
Autoinverse is an inversionstrategy that could be applied to various inverse methods, especially those that concern imitation ofan original process (dubbed as NFP in our work).
We look forward to see Autoinverse adopted formore inverse architectures beyond tandem , and more direct optimizations beyond first-order NA .
Predicting the uncertainty using Deep Ensembles [21] is not the most efficient solution.
This isbecause multiple forward networks are trained in two stages (once for the mean, and once jointlyfor mean and variance).
We are working on integrating the more efficient Monte Carlo dropout into
Autoinverse .
It is interesting to see how forgoing some accuracy in predictive uncertainty couldaffect the overall inversion performance.