Autoinverse: Uncertainty Aware Inversion of Neural Networks
Abstract
Neural networks are powerful surrogates for numerous forward processes.
The inversion of such surrogates is extremely valuable in science and engineering.
The most important property of a successful neural inverse method is the performance of its solutions when deployed in the real world, i.e., on the native forward process (and not only the learned surrogate).
We propose Autoinverse , a highly automated approach for inverting neural network surrogates.
Our main insight is to seek inverse solutions in the vicinity of reliable data which have been sampled form the forward process and used for training the surrogate model.
Autoinverse finds such solutions by taking into account the predictive uncertainty of the surrogate and minimizing it during the inversion.
Apart from high accuracy, Autoinverse enforces the feasibility of solutions, comes with embedded regularization, and is initialization free.
We verify our proposed method through addressing a set of real-world problems in control, fabrication, and design.
Our code and data are available at: https://gitlab.mpi-klsb.mpg.de/nansari/autoinverse
Introduction
“... optimizing for the wrong thing — offering prayers to the bronze snake of data rather the larger force behind it.”
With the deep learning breakthrough during the last decade data-driven modeling using neural network based surrogates is now a standard practice in science and engineering.
These surrogates strive to imitate the behavior of a native forward process (NFP) such as a physics simulation which maps a design into its performance 1 .
While forward processes are essential for troubleshooting and analysis, oftentimes their ultimate application lies in their inversion, i.e., the reverse process of mapping performances into designs.
Despite the recent progress in inversion of neural networks within multiple disciplines [35, 16, 13, 37], a fundamental unaddressed question is still standing out.
Inversion of a surrogate model, fitted to the data sampled from the NFP, is ultimately different than the inversion of the NFP itself.
The source of this gap could be explicit, such as the noise in term measurements, or implicit, such as the poor sampling of the NFP.
Although the obtained solutions from inverting the surrogate can be evaluated on the NFP, none of the current neural inversion methods offers a tailored solution for addressing this important gap.
Our main insight in this work is to expect and account for any potential mismatch between the data, and consequently the surrogate, on the one hand and the NFP on the other.
Our proposed method,
Autoinverse , realizes this vision by taking into account the predictive uncertainty of the surrogate and minimizing it during the inversion.
Therefore, the inverted solutions avoid the unreliable regions within the training data.
We show that our Autoinverse strategy can augment existing neural inversion methods (both optimization-based and architecture-based approaches) with uncertainty compensation in a simple and practical manner.
Autoinverse closes the gap between the surrogate and the NFP not through attempting a perfect fit of the surrogate to the NFP, an onerous task, but by finding inverse solutions in the vicinity of the reliable training data where the surrogate and the NFP are most similar.
Neural inverse methods equipped with Autoinverse outperform their counterparts significantly on both standard data sampled from the NFP and imperfect data, e.g., those corrupted by noise.
Apart from high accuracy, Autoinverse enforces the feasibility of solutions, comes with embedded regularization (freeing the inversion approaches from hand-crafted regularizations based on domain knowledge), and is initialization-free.
It achieves all these properties in a highly automated manner and only with a light, intuitive tuning.
Related work
Neural Network Inversion We can divide neural network inversion approaches into two main categories.
First, inverse architectures , where we compute a network architecture that takes a given performance and maps it into a (distribution of) design(s).
Second, direct optimization [33], where we optimize for a design such that it produces the desired performance.
Although the simplest inverse architecture can be attempted by training a neural network in the reverse direction, it fails because of the one-to-many nature of the mapping.
Tandem networks use an inverse architecture by employing a pre-trained forward network in order to compute a consistent loss.
The tandem approach has been developed independently across different disciplines [23, 32, 33, 39] dating back (at least) to Tominaga [34].
Many inverse architectures try to model the conditional posterior, p ( x | y ) , using variational methods [24, 19] based on (conditional) variational auto-encoders [20].
These networks condition the design on the target performance and yield a distribution of solutions from which multiple samples could be drawn.
Kruse et al.
[21] show that the invertible neural networks (INNs), built upon normalizing flows [10], give the highest accuracy in terms of both surrogate error and design posterior compared to a wide range of inverse architectures.
When using direct optimization methods, gradient-based optimizers can be readily used as the neural surrogate is differentiable.
Ren et al.
[30] use stochastic gradient descent via backpropagation with respect to the design variables, and present a highly accurate and practical method.
They benchmark their method, dubbed as neural adjoint ( NA ), against a set of inverse architectures and obtain significantly more accurate solutions.
Sun et al.
[33] showed a similar approach except using a quasi-Newton method for optimization.
Ansari et al.
[2] push forward in this direction by demonstrating that, for piecewise linear neural networks, e.g., those with ReLU activation, the direct optimization can be formulated as a mixed-integer linear program (MILP) and thus obtain globally optimal solutions.
While optimization methods are very accurate, their main disadvantage is their performance as they can be orders of magnitude slower than inverse architectures.
Neural Networks and Predictive Uncertainty While neural networks are ubiquitous in almost all branches of natural sciences, their weakness at quantifying predictive uncertainty impedes their use in crucial applications.
Using a Bayesian formalism [4], Bayesian neural networks (BNNs) [28, 9, 26], given the training data and a prior over network’s parameters, compute the posterior distribution of the parameters.
Having computed the posterior, the predictive uncertainty can be computed.
The inference step in computing BNNs is known to be computationally hard [17].
This explains the popularity of simpler methods for estimating predictive uncertainty, such as Monte Carlo dropout [11] and Deep
Ensembles [22].
Deep Ensembles strikes a good balance between simplicity and practicality on the one hand and predictive performance on the other hand (Section 3.3).
One of the main advantages of the Deep Ensembles is its capability to predict aleatoric and epistemic uncertainty separately.
D es i gn
E n se m b l e ...
Forward Network
M
M M
M M (a)
M i n i m i z e
D es i gn
E n se m b l e ...
P r e d i c t e d p e r f o r m a n ce
Forward Network
Inverse network
T a r g e t p
e r f o r m n ce fX
M M
M (b)
Aleatoric and epistemic uncertainty carry different information regarding the surrogate [18] and considering both of them improves the quality of the neural inversion (Section 4.3).
Neural networks capable of predictive uncertainty are increasingly adopted in many different applications, such as reinforcement learning and active learning [12].
As we shall see, neural inversion is yet another domain that takes advantage of this trend.
Method
Autoinverse is an easy-to-implement technique for augmenting neural inverse methods with uncertainty awareness.
Autoinverse achieves this goal by, first, training a surrogate capable of predictive uncertainty [25].
Second, relying on this trained surrogate and using a novel inversion cost function, Autoinverse finds accurate designs with minimal uncertainty.
We apply Autoinverse on two inverse methods belonging to the two main neural inversion categories, i.e., optimization- and architecture-based in Sections 3.1 and 3.2, respectively.
As we will see in Section 3.3, we rely on established methods [22] to train networks equipped with predictive uncertainty.
3.1 Uncertainty aware neural adjoint ( UANA )
Given a pretrained neural surrogate ˆ f ( · ) , neural adjoint ( NA ) [30] is an inverse method that uses the cost function L NA ( · ) to push designs x to have a performance ˆ f ( x ) as close as possible to a desired performance y ∗ :
[Equation]
NA uses gradient descent to iteratively reduce the cost function, a scheme much like training neural networks but with the input as the optimization variable instead of network’s weights and biases.
Equation 2 shows a single NA iteration with δ as the step size:
[Equation]
Autoinverse proposes to perform the inversion using a pretrained BNN.
We use Deep Ensembles [22] made of M neural networks capable of a prediction F µ ( x ) , as well as its aleatoric F σA ( x ) and epistemic F σE ( x ) uncertainties (Section 3.3).
Aleatoric uncertainty increases as the noise level in the training data increases.
Epistemic uncertainty measures the uncertainty in the model.
Autoinverse modifies NA such that we obtain solutions x that have performances close to the target performance y ∗ while resulting in small aleatoric and epistemic uncertainties:
[Equation]
We introduce α and β as hyperparameters to adjust the relative significance of aleatoric and epistemic uncertainties, respectively.
Equation 4 shows how one iteration of UANA requires the back-propagation using the ensemble of all gradients of M networks of Deep Ensembles:
[Equation]
where ˆ f m represents one of the networks in the ensemble.
Figure 1(a) depicts this collective procedure where each individual network in the ensemble votes for the direction where updating the design will lead to the maximal accuracy and minimal uncertainty.
3.2 Uncertainty aware tandem ( UA-tandem )
Tandem is a representative of architecture-based methods in which we train an inverse network ˆ f − 1 ( · ) in a manner resembling the encoder-decoder architecture ([32–34]).
Unlike the encoder-decoder approach, we start with training the forward model
ˆ f ( · ) .
Then we freeze the trainable parameters of ˆ f ( · ) and train the inverse model
ˆ f − 1 ( · ) in the position of the encoder in order to decrease the cost function:
[Equation]
Once ˆ f − 1 ( · ) is trained we can simply query it to find designs with our desired performances:
[Equation]
The uncertainty-aware tandem ( UA-tandem ) follows the same procedure except that it replaces ˆ f ( · ) with F µ ( · ) .
Additionally, it includes the uncertainties in the loss:
[Equation]
Figure 1(b) depicts the architecture of UA-tandem .
3.3 Predictive uncertainty using Deep Ensembles
m m m
Deep Ensembles comprises of an ensemble of M neural networks ˆ f m each capable of a prediction µ m ( x ) and its associated uncertainty σ m ( x ) in form of a Gaussian distribution N ( µ m ( x ) , σ m ( x )) .
The cost function for training each network in the ensemble is the negative log likelihood [29]:
[Equation]
Intuitively, in case of the aleatoric uncertainty µ m ( x ) fails to reliably predict y ∗ .
Hence, σ 2 m ( · ) must increase to reduce the loss while the first term ensures σ 2 m ( · ) does not diverge to infinity.
The next step is to ensemble the results of all the networks into a single prediction and a single uncertainty.
Deep Ensembles [22] models the ensemble as a single Gaussian distribution N ( F µ ( x ) , F σ ( x )) approximating the mixture of M previously computed Gaussian distributions
[Equation]
The uncertainty of the ensemble can be decomposed into two input-dependent uncertainties, i.e., aleatoric F σA ( x ) and epistemic F σE ( x ) through slight modification to Equation 9b [18].
[Equation]
This enables us to control the behavior of the neural inversion by tuning the relative importance of these two uncertainties.
Evaluation
We evaluate the performance of Autoinverse through experimenting with the existing neural inverse methods and their uncertainty-aware counterparts.
In the paper, we focus on NA and UANA while tandem and UA-tandem are evaluated mainly in the Appendix.
We perform the evaluation on a set of applications in robotics, and fabrication-oriented computational design.
We report the error by running the experiments 3 times to capture the variations.
More details are provided in each case study and in Appendix (Section A).
Equations 3 and 7 have two hyperparameters ( α and β ) that keep the balance between the MSE, the aleatoric uncertainty and the epistemic uncertainty.
We observe that with a relatively larger epistemic weight ( β ) we obtain better results.
Exploiting this intuition, we tune these parameters for 3 different sets of values for { α , β }: {{ 0 . 1 , 1 } , { 1 , 10 } , { 10 , 100 }} .
We then make two finer step depending on the outcome of the former evaluation and choose the best set of weights.
We keep this budget ofexperiment runs for the rival methods as well.
Typically, we use 10% of the target performance for tuning our inverse methods.
4.1 Experiments
Multi-joint robot is a simple inverse kinematics problem which is being used as a standard test for neural inverse problems [21, 30].
In this problem, the design is the 1D position of the base of the multi-joint robot along with the 1D rotation of its three joints.
The inverse problem concerns finding a combination of positions and angles for the base and the joints such that the tip of the arm lands the target position, i.e., the performance .
We follow [30] for setting up this experiment and its corresponding analytical equation used as the NFP.
The training data consists of 10,000 pairs of samples generated by randomly sampling the NFP.
Spectral printer Spectral printing enables digital fabrication of the object’s appearance faithfully ([14, 5]).
Unlike reproduction of the color (e.g., RGB), reproducing the spectrum ensures that the original and the duplicate remain visually similar independent of the color of the light source.
Spectral printing has various important applications specially in fine art reproduction using both 2D and 3D printing [27, 1, 32].
Deep neural networks are becoming the main computational tool for modeling the spectral printing process specially when dealing with a large number of inks.
The final objective is to find the correct ink densities at each pixel that can best estimate the target 31D spectrum.
In this experiment, we create the NFP by simulating a printer using an ensemble of 20 neural networks.
The design space comprises the ink densities and our spectral printer NFP predicts the spectrum of the resulting color.
We use real, measured data from an 8-channel printer with
EPSON inks including Cyan (C), Magenta (M), Yellow (Y), Black (K), Light black (LK), Light light black (LLK), Light Cyan (LC), and Light Magenta (LM).
The light inks, added to the standard CMYK to improve the print quality, introduce significant multi-modality.
All networks in the ensemble NFP are trained on 40,000 printed patches [1] consisting of different ink-density combinations and their corresponding spectra.
The ensembling is intended for an accurate NFP and is independent of our use of Deep Ensembles for computing the uncertainty.
The visual nature of spectral printer makes it attractive for analyzing different methods.
We release this NFP to the public to add another example to the neural inversion testbed.
[Table]
Soft robot are made of soft, flexible materials.
This unique property has made them suitable candidates for interaction with humans in applications like minimally invasive surgery and advanced prosthetics [7].
Unlike multi-joint robot with a limited number of rotating joints, each segment of the soft robot is a potential actuator that through their contraction and expansion can determine the robot’s final shape.
This inverse kinematics problem is typically solved through partial differential equations ([38]).
In order to accelerate the solve time of this inverse kinematics problem, Sun et al.
[33] proposed a neural surrogate modeling of the problem and its inversion via tandem .
The design space in this problem consists of the contraction or expansion of 40 controllable soft edge segments.
The superposition of all the actuations determines the final deformation position of the soft robot via the position of its 206 vertices.
We use an FEM-based simulation ([38, 15]) as our NFP.
We model the relationship between the actuations and the final shape of the soft robot with a neural-network surrogate.
Our goal is to solve the neural inversion to find a suitable set of actuations (design) that brings the tip of the soft robot to the target position (performance).
The training data consists of 50,000 samples queried by random sampling the actuation with an expansion ratios between -0.2 and 0.2 [33].
The designs are then evaluated on the FEM-based NFP to calculate their deformations.
4.2 Quantitative comparison of different neural inversion methods
We evaluate the accuracy of a set of neural inversion methods on multi-joint robot in terms of both the NFP and the surrogate errors.
The surrogate error is the difference between the ‘re-prediction’ of the obtained inverse solution by the surrogate neural network and the target performance.
The NFP error is the difference between the target and the performance of the generated design evaluated on the NFP.
In addition to the core methods described so far, we evaluate mixed-integer neural inversion ( MINI ) [2], and the invertible neural network ( INN ) [3].
Table 1 summarizes the inversion results on 1000 randomly sampled target locations for the multi-joint robotic arm.
We keep our evaluation fair by setting the capacity of the neural surrogates comparable wherever possible.
For instance, all methods except MINI have around 3 million parameters (see
Appendix, Table 3 for more details).
We also used equal computational resources for the tuning.
Methods with hyperparameters, like UANA and UA-tandem , are tuned in 5 stages by hand.
Alternatively, the methods without hyperparameters ( NA, tandem ) are given 5 × extra budget for inversion:
We run NA and tandem 5 × and choose the model that generates the best NFP error.
We repeat this process 3 × and report the standard error.
MILP and INN are fundamentally different methods.
MILP finds the global optimum and thus does not need tuning.
INN has a latent space which we can sample to generate diverse designs.
We sample INN’s latent space 1024 times for all 1000 targets, evaluate them on the NFP, and report the best NFP error.
We repeat this process three times to generate the standard error.
Table 1 shows the outstanding accuracy of the inverse methods that adopt Autoinverse , i.e., UANA and UA-tandem , in terms of the NFP error.
It also demonstrate how even a perfect surrogate error (e.g., MINI ) does not guarantee accurate solutions when tested on the NFP.
A second look at Tablereveals further interesting insights.
Although NA obtains notably lower surrogate error than its uncertainty-aware counterpart ( UANA ), it performs significantly worse in terms of the NFP error.
The main reason for this trend is that, when optimizing the surrogate, UANA is not only concerned with finding accurate designs leading to a small accuracy gap between the target and candidate performances, i.e., the surrogate error, but also with those designs featuring low uncertainty (through the uncertainty term in Equation 10).
Therefore, UANA achieves high accuracy in terms of the essential
NFP error at the cost of worsening the inconsequential surrogate error.
Furthermore, UA-tandem , for example, achieves better performance than NA .
This is a remarkable result for an architecture-based
[Table]
N o i s y d a t a
S p a r s e d a t a
[Table]
method given it is significantly faster than the optimization-based NA .
We report extensively the time performance and the details of the training for this and the next experiments in the Appendix.
4.3 Neural inversion in the presence of imperfect data
One of the main advantages of Autoinverse appears in scenarios where the training data suffers from noise, e.g., measurement noise or poor sampling, e.g., sparsity in some regions.
We evaluate the performance of Autoinverse on imperfect training data using NA and UANA ( tandem and
UA-tandem are evaluated under the same configuration in the Appendix Section C).
Locally sparse data We sample both soft robot and spectral printer NFPs (Section 4.1) in a way that the data does not contain any samples from one of the inputs in a specific interval.
For spectral printer , we would like to find ink densities to reproduce the spectra of the colors in the painting in Figure 3 (made of 3568 distinct color spectra).
We sample the printer channels at(no ink), 0.05, 0.1, 0.5, and 1 (full ink) densities to form the standard training data (with no known uncertainty).
We create a partially sparse dataset similar to the standard one except for the Light
Cyan (LC) channel for which we only have samples at 0, 0.05 and 0.1.
Table 2 shows that while for the standard dataset UANA finds inverse solutions that include the LC channel frequently (437 times), for the sparse dataset it avoids this channel completely and compensate for it using the Cyan channel.
UANA is able to avoid this channel as the epistemic uncertainty increases in sparse regions of dataset (see Appendix Section C.4 for more details).
In soft robot we sample the 16th (among 40) controllable edge only in the negative range (contraction only).
We then use the trained network to invert 1000 test samples.
Figure 2 shows the distribution of each edge for 1000 inversion tasks.
We have plotted the distribution for the 16th edge as well as for 7 other randomly chosen edges.
As evident from Figure 2 bottom row, UANA is highly reluctant to choose designs with positive actuations for this edge.
Locally noisy dataset With the same problem configuration as before, we would like to test the robustness of Autoinverse on a dataset locally corrupted with noise.
We start with a standard dataset and inject Gaussian noise N (0 , 0 . 1) to the spectrum of the samples with more than 0 .
4 LC density.
Table 2 shows how after introducing noise to the LC, the network avoids that channel and compensates it by using more Cyan instead.
In soft robot we corrupt the final shape of the soft
[Figure]
NA UANA
Valid design
NA w i t h bound a r y l o ss robot with Gaussian noise N (0 , 0 . 5) for all the shapes where the 16th edge has positive actuation.
As we can see in Figure 2 (top row), the inversion has completely avoided positive actuations for this edge.
4.4 Autoinverse brings AutoML to neural inversion
Autoinverse incorporates feasibility.
Deep Ensembles produces high epistemic uncertainty outside the distribution of the training data.
This includes the regions where the NFP is not defined and thus not sampled.
For example, ink densities outside [0 , 1] are not printable.
If such cases arise during the inversion, they are clipped to [0 , 1] [1].
In such regions, networks in the ensemble do not agree and thus the epistemic uncertainty increases.
Autoinverse automatically avoids these unfeasible regions.
In order to simulate the result of the inversion, after clipping the out-of-range densities we feed them into the NFP.
As evident from Figure 3 left, inversion via NA results in a poor reproduction of the original painting.
UANA , on the other hand, achieves spectacular reproduction accuracy.
This is explained by the plot in Figure 3 right, showing the distribution of ink densities obtained from both methods.
Autoinverse has built-in regularization.
We incorporate regularization into inversion methods in order to obtain solutions that, among other purposes, agree with the observations and follow a certain statistical distribution.
Oftentimes, regularization is case-specific, requires human knowledge, and comes with unexpected side effects.
Here we show that Autoinverse follows the distribution of the training data naturally by taking into account the epistemic uncertainty.
We validate this point using both spectral printer and soft robot experiments.
We show that Autoinverse without any explicit regularization performs better or on par with its counterpart inversion methods equipped with regularization.
In the spectral printer experiment, we evaluate the effect of the boundary loss , originally proposed as a generic regularization for NA to limit the designs within a box constraint (see [30] and
Appendix Section D).
The boundary loss is added to Equation 1 and weighted using a hyperparameter.
We tune this parameter with the same tuning budget we allocate for tuning the uncertainty weights ( set of inversions on evaluation data).
In Figure 3, we observe that although NA with boundary loss improves the distribution of ink densities within the valid region ( [0 , 1] ), it still trails the regularizationfree UANA significantly.
Regularizing the soft robot is less intuitive as the superposition of all actuations determines the final shape and whether it is physically plausible.
In [33], the regularization is a smoothness term that keeps actuation values near each other (see Appendix Section D).
Figure 4(a)(1) demonstrates how NA fails without regularization to control the robot with a reasonable deformation.
Once the regularization is added to NA , the designs become physically meaningful (Figure 4(a)(2)).
Figure 4(a)(3) shows how UANA performs comparably without any regularization.
[Figure]
Autoinverse is initialization-free.
Sensitivity to the initialization is a widely known issue in nonconvex optimizations.
Despite equipping NA with the smoothness term in soft robot , Figure 4(b) demonstrates how an incorrect initialization can result in solutions with seemingly good surrogate and regularization loss (see Appendix Section D) but in the infeasible region of the design space (Figure 4(a)(4)).
UANA with the same incorrect initialization and without any regularization, leads to soft robot designs that reach the target location accurately and produce plausible deformations (Figure 4(a)(5)).
For UANA , the solver starts with reducing the main contributor to the optimization objective, i.e., the epistemic uncertainty.
Once a region with a reasonably small uncertainty is reached, the accuracy term starts to take effect and a desirable solution within the valid range of the design space is found.
4.5 Ablation studies
Separating the ensembling effect Although in both NA and UANA we use surrogates with similar capacity (network size), one could argue that the higher performance of UANA comes from the ensemble architecture of its surrogate.
We detach the impact of ensembling and uncertainty awareness on the inversion performance by implementing NA ensemble where, instead of a single, large forward neural network, it uses an ensemble of networks.
The inversion procedure is identical to
NA (see Appendix Section E and Figure 5).
We employ NA ensemble in multi-joint robot with the same configurations as in Section 4.2.
The surrogate and NFP error for NA ensemble are (3 . 30 ± 0 . 59) × 10 − 9 and (1 . 17 ± 0 .
32) × 10 − 4 , respectively.
Comparing these values with those in Table 1, NA ensemble shows a slight improvement over NA but is significantly outperformed by
UANA .
Diversity of activation functions in Deep Ensembles Deep Ensembles uses a similar set of networks with identical activation functions [22].
In practice, we observe that a diverse set of activation layers leads to a better performance of Autoinverse .
Different activation functions generate different behaviours, show higher disagreement where the training data is under-represented and, thus, result in an accurate estimation of the landscape of epistemic uncertainty [36].
For all experiments in this work, we use a diverse range of activation functions (see Appendix Section
A).
As an ablation, we run UANA on spectral printer using sparse data (same configuration as
Section 4.3) but with ReLU as the only activation layer.
In contrast to Table 2, UANA (with ReLU only activation) does not completely avoid the sparse domain and delivers 19 ± 4 solutions that contain
LC with densities ≥ 0 . 4 .
Discussion
The Autoinverse cost function is multi-objective.
Instead of finding a single solution through a weighted combination of these objectives (what we have seen so far), we can capture the trade-off between the accuracy and uncertainty through computing the Pareto front (see Appendix Section F for more details).
Autoinverse is an inversion strategy that could be applied to various inverse methods, especially those that concern imitation of an original process (dubbed as NFP in our work).
We look forward to see Autoinverse adopted for more inverse architectures beyond tandem , and more direct optimizations beyond first-order NA .
Predicting the uncertainty using Deep Ensembles [22] is not the most efficient solution.
This is because multiple forward networks are trained in two stages (once for the mean, and once jointly for the mean and variance).
It is highly interesting to integrate more efficient uncertainty estimation methods, such as Monte Carlo dropout [11, 18], into Autoinverse .
Acknowledgments and Disclosure of Funding
We are grateful to Xingyuan Sun and Szymon Rusinkiewicz who provided the repository of the soft robot simulation [33], Sebastian Cucerca for the revision of the figures, and anonymous reviewers for valuable feedback.
Special thanks to Jalil Maani and Soraya Iranpak for their kind support
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Standard error calculation, tuning, and implementation details
In this section we provide more details on different inverse methods we use in the paper.
As many of the inverse methods we use in the paper have a stochastic component, we perform all following experiments 3 times and report the standard error.
Since we have a budget of 5 runs for tuning the hyperparameters of Autoinverse methods, we allocate similar or higher resources for other methods in order to make the comparisons fair.
NA To avoid local minima, we run 50 solves of inversion with random initialization (each having up to 2000 iterations).
As an alternative to tuning, we run NA 5 times and report the best NFP error.
To accelerate the inversion in this paper we perform batch optimization.
Thus, we can increase the number of target samples (up to GPU memory limit) without impacting the inversion time noticeably.
NA ensemble The configurations for NA ensemble is similar to NA except for the forward model.
Unlike NA , we have an ensemble of networks ( F µ ) that comprises the forward model of NA ensemble :
[Equation]
Unlike UANA , single networks in the ensemble are incapable of predicting uncertainty.
The cost function for NA ensemble is therefore defined as:
[Equation]
Similar to UANA , the back propagation is based on an ensemble of gradients coming from all the single networks in the ensemble (Figure 5).
[Equation]
Similar to NA , we run NA ensemble 5 times and pick the best results with respect to the NFP error.
tandem Since tandem does not explicitly possess any hyperparameter, instead of tuning, we evaluate 5 pre-trained inverse models (each initialized differently) and select the one that has the best
NFP error on 10% of the target data.
We then perform the inversion on the remaining target data using the best-performing inverse model.
UANA and UA-tandem
In Section 4.5 of paper we showed how using a diverse range of activation functions in the ensemble network improves the quality of epistemic uncertainty and consequently the results of the inversion.
For training the surrogates used in UANA , UA tandem and NA ensemble we use 10 networks in the ensemble with the following activations F µ :
[Figure]
[Equation]
[Figure]
[Table]
• Tanh ×• ReLU ×• CELU ×• LeakyReLU
×• ELU • Hardswish
We use ReLU activation functions for all other methods.
INN For each one of the 1000 target performances we randomly sample the latent space of the INN architecture [30] 1024 times.
Thus, we end up with 1024 designs (for a single target) and evaluate all designs on the NFP and report the best error as the NFP error.
For the surrogate error, we report the average forward loss.
MINI MINI is based on a mixed-integer optimization which is capable of finding the globally optimum solution [2].
This method is deterministic and every run returns the same solution, as a result we do not report the standard error for MINI .
Hardware To have a fair comparison we run all the methods on the same GPU machine for evaluating time performances.
We used an NVIDIA TITAN X GPU for time evaluation.
For other evaluations we used a GPU cluster.
Training the forward models is trivially parallelizable.
Moreover, we can parallelize 50 iterations of NA and UANA and aggregate the data in a post-processing step and choose the best results based on the surrogate error .
Nevertheless, we are reporting our computation time assuming both training and inversion are performed serially on a single GPU.
B Training details for neural surrogate models in multi-joint robot (
Table 1 in paper)
In Table 3 and Table 4, we can compare the capacity, training time, and accuracy of the neural network surrogates used for the inversion of multi-joint robot .
We keep similar training capacity for all methods except MINI .
MINI uses a combinatorial optimization and is not scalable to large networks [2].
We trained a smaller surrogate for MINI but at the same time we monitored its training loss to lie within a reasonable range.
C Details for ‘Neural inversion in the presence of imperfect data’ C.1 Counterpart results of tandem and UA-tandem for spectral printer
Table 5 presents the results of spectral printer , similar to Section 4.3 (Table 2) of the paper but comparing tandem and UA-tandem .
As evident from the table, we obtain similar performance gain
[Table] [Table]
when augmenting tandem with uncertainty awareness.
Similar to UANA , UA-tandem has completely avoided the LC channel.
C.2 Soft robot actuation distribution
Figure 2 in the paper showed the actuation distribution of only 8 soft robot edges (Section 4.
in paper).
In Figure 6 we show the distribution of actuation for all edges computed by UANA on surrogates learned using partially noisy and partially sparse data.
We repeat this experiment using
UA-tandem to emphasize on the generality of Autoinverse (Figure 7)
[Table]
C.3 Training details of surrogates used for spectral printer and soft robot
Tables 6 and 7 show the layer configuration, number of trainable parameters, the training and inversion time
[Table]
and the training loss of different surrogate models used in spectral printer .
Tables 8 andshow the layer configuration, number of trainable parameters, the training and inversion time
[Table]
and the training loss of different surrogate models used in soft robot .
From the tables we can observe that
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Table] [Table]
3 . 38 ×
[Table] [Table]
networks 1 . 15 × 103 .
44 × 10 −
[Table]
networks
Tandem
Forward network 1 .
[Table]
[Table] [Table]
Standard data
Sparse data
Noisy data
LC
E p i s t
e
m i c un c e r t a i n t y (a) 1010 - 10101010 - 10Standard data
Sparse data
Noisy data
LC
A l ea t o r i c un c e r t a i n t y
[Figure]
the training capacity of different surrogates models is comparable.
Furthermore, the training accuracy of all models is similar.
C.4 Epistemic and aleatoric loss behaviour for spectral printing experiment
Epistemic uncertainty The key to handling design feasibility is the epistemic uncertainty (Equation 3(c) paper).
We know that the scarcity of data results in higher epistemic uncertainty [18].
On the other hand, by definition, we do not have any infeasible or out-of-range data points in our dataset.
Hence, if we query a network with an infeasible or out-of-range input, we will get high uncertainty for the prediction.
We use this trend to avoid such samples in the inversion.
Figure 8(a) shows the trend of the epistemic uncertainty values for spectral printer (Section 4.3, Table 2 in paper).
For that experiment, we ran the inversion using UANA on 3 different datasets: standard, noisy, and sparse and observed how the problematic ink channel (LC) is avoided.
Figure 8(a) demonstrates why that ink channel is avoided.
In Figure 8(a), we set all ink channels except LC to 0 while increasing the values of LC ink density from 0 (the x-axis of the plot).
As expected, for all three datasets moving away from the feasible region (between 0 and 1) increases the epistemic uncertainty.
When trained for the sparse data (red curve), where the LC channel has not been sampled after 0.4, the epistemic uncertainty starts to increase earlier.
Outside the feasible region, each network in the ensemble has to extrapolate as it has not been trained in those regions.
Consequently, the predictions of ensemble networks diverge.
The divergence of the networks increases the epistemic uncertainty and, during the inversion, the uncertainty aware methods can reject solutions in these regions.
Aleatoric uncertainty Figure 8(b) demonstrates the behavior of the aleatoric uncertainty of the surrogate used for the same experiment ( UANA on spectral printer ).
Similarly, to generate the plots, we set all the ink channels to 0 and change the values of the Light Cyan ink densities.
As evident from Figure 8(b), the level of uncertainty increases significantly for the noisy dataset, while for sparse and standard datasets aleatoric uncertainty is at least two orders of magnitude smaller.
The increase of aleatoric uncertainty for the noisy data helps UANA avoid any samples from those regions (Tables 2 in the paper and Table 5).
D Details for ‘Autoinverse brings AutoML to neural inversion’
Spectral printer Complementary to Figure 4 of the paper, in Figure 9 we compare the inversion performance using a diverse range of inverse methods.
Here we clearly see that basic methods, such as NA and tandem fail spectacularly in computing feasible designs.
[Figure]
Boundary loss is a semi-generic regularization, suitable for handling box constraints [30]:
[Equation]
where R x is the value range of design samples in the dataset and µ x is their average.
Incorporating the boundary loss in NA in Figure 9 results in an improvement in the distribution of the ink intensities.
However, the results are still far from acceptable.
Note how applying hand-crafted regularization (ink intensity regularization) [32] improves the quality of tandem significantly ( tandem with regularization in Figure 9).
UANA and UA-tandem however perform comparably without any regularization.
Soft robot objective for the Neural Adjoint method The objective function for soft robot inversion comprises of two terms, one is responsible for bringing the tip of the robot to the target and the other one ( R ( x in ) )
guarantees the deformations to remain physical [33].
[Equation]
[Equation]
where t represents the target location and x outi represents the position of all 206 vertices of soft robot, among which i ∈ [123 , 124] represent the position of its tip.
Also, λ adjusts the importance of the smoothness term and R ( x in ) regulates the actuation of the flexible edges ( x in ) to insure that the deformation of the robot is physical.
Sensitivity to initialization ( soft robot )
In Section 4.4 of the paper, we learned how handcrafted regularization improves the quality of the designs.
Despite having regularization, soft robot inversion using NA fails when initialized with values far from feasible region.
This is evident from both the irregular robot shapes (Figure 3(b) in the paper) and the distribution of the actuation (Figure 10) computed using regularized NA but with wrong initialization.
At the same time, UANA without any regularization and with a wrong initialization produces plausible robot shapes (Figure 3(b) in the paper) and actuation distribution (centered around 0).
Interestingly, the smoothness score for
Correct initialization
Wrong initialization
[Figure]
smoothness score = 0. smoothness score = 0.
smoothness score = 0.
[Figure]
clearly failed shapes obtained from NA with regularization, are for both initializations very small (Figure 10).
This shows how even regularization can be multi-modal and fall into a wrong local minima and generate designs with nonphysical shapes (Figure 3(b))
[Figure]
E Details for ‘Ablation studies’
As discussed in Section 4.5 of the paper, the power of UANA lies in its uncertainty awareness and not the ensembling process.
We include the result of NA ensemble for paitning reproduction in Figure 9.
While we observe a marginal improvement of NA ensemble over NA , it is clearly outperformed by
UANA .
Sensitivity to uncertainty weights Autoinverse is extremely stable when tuning its hyperparameters, i.e., uncertainty weights ( α and β in Equation 3 in paper).
This ensures that a light hyperparameter tuning is enough for obtaining reasonable results.
We evaluate this behavior by using a wide range of α and β values spanning over 5 orders of magnitude.
This ablation is performed using UANA .
In the ablation of aleatoric ( α ) and epistemic ( β ) weight, we have used the noisy and epistemic data of spectral printer , respectively.
When evaluating α we keep β fixed at the tuned value.
Alternatively for the evaluation of β , α is constant at the tuned value.
[Figure]
U n c e r t a i n t y sc
o r e
[Figure]
The most interesting fact about Figures 11(a) and 11(b), is the correlation of the α and β with the
NFP error.
This correlation means that adjusting the importance of these weights on the surrogate model directly improves the quality of the inversion in reality.
Interestingly, we can observe the robustness of Autoinverse against the variation of α and β , such that for a range of aroundorders of magnitude the NFP loss remains stable around a desirable value.
The larger the weights,
Autoinverse chooses less and less samples in the problematic regions (LC channel density larger than 0.4).
This trend continues with very large uncertainty weights.
However, these weights cannot be indefinitely increased as the MSE term of the objective (Equations 6 and 10 in the paper) will be undermined and inversion’s NFP error increases.
F Pareto front of accuracy versus uncertainty
We calculate the Pareto front for 12 randomly chosen targets from the spectral printer experiment with the standard dataset (Figure 12).
We use the NSGA II [8], an evolutionary algorithm that samples our forward BNN to discover the Pareto front iteratively.
The uncertainty score in this experiment is the weighted sum of aleatoric and epistemic uncertainty.
We set the values of the weights on the tuned values on the inversion task.
The population size and the number of generations in this experiment are 1000 and 100, respectively.
Figure 12 suggests that the losses of uncertainty aware inversion are conflicting such that for example reducing the MSE loss will lead to the increase of the uncertainty score.
G Ablation of the number of networks in the ensemble
In this experiment we investigate the importance of the accuracy of the calculated uncertainties on the final NFP error.
We have trained deep ensemble networks with a varying number of networks in the ensemble.
The networks are trained on the spectral printer experiment for both noisy and sparse datasets.
As evident in Figure 13, by increasing the number of networks in the ensemble the
NFP error improves significantly.
This trend indicates the importance of accurate prediction of the uncertainties.
[Figure] [Figure]
r
[Figure]
# of networks
[Figure] [Figure]
H Implementation
In practice, training the ensemble networks directly with negative log likelihood loss (Equation 8) is challenging [31].
Instead, following [29], we take a 3-step procedure for implementing deep ensemble predictive uncertainty.
We start with training an ensemble of conventional networks with diverse activation functions and MSE as its loss.
These networks are in fact the initialization of F µ ( · ) .
The next step is training F σ ( · ) and fine tuning F µ ( · ) jointly with the negative log likelihood loss (Equation 8).
Finally, we replace ˆ f ( · ) with F µ ( · ) and incorporate F σ ( · ) in the Autoinverse loss.