Batch Multi-Fidelity Active Learning with Budget Constraints
Abstract
Learning functions with high-dimensional outputs is critical in many applications, such as physical simulation and engineering design.
However, collecting training examples for these applications is often costly, e.g. , by running numerical solvers.
The recent work (Li et al., 2022) proposes the ﬁrst multi-ﬁdelity active learning approach for high-dimensional outputs, which can acquire examples at different ﬁdelities to reduce the cost while improving the learning performance.
However, this method only queries at one pair of ﬁdelity and input at a time, and hence has a risk to bring in strongly correlated examples to reduce the learning efﬁciency.
In this paper, we propose Batch Multi-Fidelity Active Learning with Budget Constraints (BMFAL-BC), which can promote the diversity of training examples to improve the beneﬁt-cost ratio, while respecting a given budget constraint for batch queries.
Hence, our method can be more practically useful.
Speciﬁcally, we propose a novel batch acquisition function that measures the mutual information between a batch of multi-ﬁdelity queries and the target function, so as to penalize highly correlated queries and encourages diversity.
The optimization of the batch acquisition function is challenging in that it involves a combinatorial search over many ﬁdelities while subject to the budget constraint.
To address this challenge, we develop a weighted greedy algorithm that can sequentially identify each (ﬁdelity, input) pair, while achieving a near (1 − 1 /e )
-approximation of the optimum.
We show the advantage of our method in several computational physics and engineering applications.
Introduction
Applications, such as in computational physics and engineering design, often demand we calculate a complex mapping from low-dimensional inputs to high-dimensional outputs, such as ﬁnding the optimal material layout (output) given the design parameters (input), and solving the solution ﬁeld on a mesh (output) given the PDE parameters (input).
Computing these mappings is often very expensive, e.g. , iteratively running numerical solvers.
Hence, learning a surrogate model to outright predict the mapping, which is much faster and cheaper, is of great practical interest and importance (Kennedy and O’Hagan, 2000; Conti and O’Hagan, 2010)
However, collecting training examples for the surrogate model becomes another bottleneck, since each example still requires a costly computation.
To alleviate this issue, Li et al.
(2022) developed
DMFAL, a ﬁrst deep multi-ﬁdelity active learning algorithm, which can acquire examples at different ﬁdelities to reduce the cost of data collection.
Low-ﬁdelity examples are cheap to compute ( e.g. , with coarse meshes) yet inaccurate; high-ﬁdelity examples are accurate but much more expensive ( e.g. , calculated with dense grids).
See Fig.
1 for an illustration.
DMFAL uses an optimization-based acquisition method to dynamically identify the input and ﬁdelity at which to query new examples, so as to improve the learning performance, lower the sample complexity, and reduce the cost.
Despite its effectiveness, DMFAL can only optimize and query at one pair of input and ﬁdelity each time and hence ignores the correlation between consecutive queries.
As a result, it has a risk of bringing in strongly correlated examples, which can restrict the learning efﬁciency and lead to a suboptimal beneﬁt-cost ratio.
In addition, the sequential querying and training strategy is difﬁcult to utilize parallel computing resources that are common nowadays ( e.g. , multi-core CPUs/GPUs and computer clusters) to query concurrently and to further speed up.
In this paper, we propose BMFAL-BC, a batch multi-ﬁdelity active learning method with budget constraints.
Our method can acquire a batch of multi-ﬁdelity examples at a time to inhibit the example correlations, promote diversity so as to improve the learning efﬁciency and beneﬁt-cost ratio.
Our method can respect a given budget in issuing batch queries, hence are more widely applicable and practically useful.
Speciﬁcally, we ﬁrst propose a novel acquisition function, which measures the mutual information between a batch of multi-ﬁdelity queries and the target function.
The acquisition function not only can penalize highly correlated queries to encourage diversity, but also can be efﬁciently computed by an Monte-Carlo approximation.
However, optimizing the acquisition function is challenging because it incurs a combinatorial search over ﬁdelities and meanwhile needs to obey the constraint.
To address this challenge, we develop a weighted greedy algorithm.
We sequentially ﬁnd one pair of ﬁdelity and input each step, by maximizing the increment of the mutual information weighted by the cost.
In this way, we avoid enumerating the ﬁdelity combinations and greatly improve the efﬁciency.
We prove that our greedy algorithm nearly achieves a (1 − 1 e ) -approximation of the optimum, with a few minor caveats.
For evaluation, we examined BMFAL-BC in ﬁve real-world applications, including three benchmark tasks in physical simulation (solving Poisson’s, Heat and viscous Burger’s equations), a topology structure design problem, and a computational ﬂuid dynamics (CFD) task to predict the velocity ﬁeld of boundary-driven ﬂows.
We compared with the budget-aware version of DMFAL, single multi-ﬁdeity querying with our acquisition function, and several random querying strategies.
Under the same budget constraint, our method consistently outperforms the competing methods throughout the learning process, often by a large margin.
Background 2.1 Problem Setting
Suppose we aim to learn a mapping f : Ω ⊆ R r
→ R d where r is small but d is large, e.g. , hundreds of thousands.
To economically learn this mapping, we collect training examples at M ﬁdelities.
Each ﬁdelity m corresponds to mapping f m : Ω → R d m .
The target mapping is computed at the highest ﬁdelity, i.e. , f ( x ) =
f M ( x ) .
The other f m can be viewed as a (rough) approximation of f .
Note that d m is unnecessarily the same as d for m < M .
For example, solving PDEs on a coarse mesh will give a lower-dimensional output (on the mesh points).
However, we can interpolate it to the d -dimensional space to match f ( · ) (this is standard in physical simulation (Zienkiewicz et al., 1977)).
Denote by λ m the cost of computing f m ( · ) at ﬁdelity m .
We have λ 1 ≤ . . .
≤ λ M .
2.2 Deep Multi-Fidelity Active Learning (DMFAL)
To effectively estimate f while reducing the cost, Li et al.
(2022) proposed DMFAL, a multi-ﬁdelity deep active learning approach.
Speciﬁcally, a neural network (NN) is introduced for each ﬁdelity m , where a low-dimensional hidden output
h m ( x ) is ﬁrst generated, and then projected to the high-dimensional observation space.
Each NN is parameterized by ( A m , W m , θ m ) , where A m is the projection matrix, W m is the weight matrix of the last layer, and θ m consists of the remaining
NN parameters.
The model is deﬁned as follows,
[Equation]
where x m is the input to the NN at ﬁdelity m , y m ( x ) is the observed d m dimensional output, ξ m ∼
N ( ·| 0 , τ m I ) is a random noise, φ θ m ( x m ) is the output of the second last layer and can be viewed as a nonlinear feature transformation of x m .
Since x m includes not only the original input x , but also the hidden output from the previous ﬁdelity, i.e. , h m − 1 ( x ) , the model can propagate information throughout ﬁdelities and capture the complex relationships ( e.g. , nonlinear and nonstationary) between different ﬁdelities.
The whole model is visualized in Fig.
5 of Appendix.
To estimate the posterior of the model, DMFAL uses a structural variational inference algorithm.
A multi-variate Gaussian
[Table]
[Table] [Table]
[Equation]
[Table]
+hQ8pAzaqz00JNpv1xxq+4cZJV4OalAjka//NUbxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bH7qlJxZZUDCWNmShszV3xMZjbSeRIHtjKgZ6WVvJv7ndVMTXvsZl0lqULLFojAVxMRk9jcZcIXMiIkllClubyVsRBVlxqZTsiF4yy+vktZF1bus1u5rlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwBh0Y3g</latexit> ⌫
[Figure]
solvers posterior is introduced for each weight matrix, q ( W m )
=
N (cid:0) vec( W m ) | µ m , Σ m (cid:1) .
A variational evidence lower bound (ELBO) is maximized via stochastic optimization and the reparameterization trick (Kingma and Welling, 2013).
To conduct active learning, DMFAL views the most valuable example at each ﬁdelity m as the one that can best help its prediction at the highest ﬁdelity M ( i.e. , the target function).
Accordingly, the acquisition function is deﬁned as
[Equation]
where I ( · , · ) is the mutual information, H ( · ) is the entropy, and D is the current training dataset.
The computation of the acquisition function is quite challenging because y m and y M are both high dimensional.
To address this issue, DMFAL takes advantage of the fact that each low-dimensional output
h m ( x ) is a nonlinear function of the random weight matrices { W 1 , . .
, W m } .
Based on the variational posterior { q ( W j ) } , DMFAL uses the multi-variate delta method (Oehlert, 1992; Bickel and Doksum, 2015) to estimate the mean and covariance of (cid:98) h
=
[ h m ; h M ] , with which to construct a joint Gaussian posterior approximation for (cid:98) h by moment matching.
Then, from the projection in (1), we can derive a Gaussian posterior for [ y m ; y M ] .
By further applying Weinstein-Aronszajn identify (Kato, 2013), we can compute the entropy terms in (2) conveniently and efﬁciently.
Each time, DMFAL maximizes the acquisition function (2) to identify a pair of input and ﬁdelity at which to query.
The new example is added into D , and the deep multi-ﬁdelity model is retrained and updated.
The process repeats until the maximum number of training examples have been queried or other stopping criteria are met.
Batch Multi-Fidelity Active Learning with Budget Constraints
While effective, DMFAL can only identify and query at one input and ﬁdelity each time, hence ignoring the correlations between the successive queries.
As a result, highly correlated examples are easier to be acquired and incorporated into the training set.
Consider we have found ( x ∗ , m ∗ ) that maximizes the acquisition function (2).
It is often the case that a single example will not cause a signiﬁcant change of the model posterior { q ( W m ) } (especially when the current dataset D is not very small).
When we optimize the acquisition function again, we are likely to obtain a new pair (ˆ x ∗ , ˆ m ∗ ) that is close to ( x ∗ , m ∗ )
( e.g. , ˆ m ∗ = m ∗ and ˆ x ∗ close to x ∗ ).
Accordingly, the queried example will be highly correlated to the previous one.
The redundant information within these examples can restrict the learning efﬁciency, and demand for more queries (and cost) to achieve the satisfactory performance.
Note that the similar issue have been raised in other single-ﬁdelity, pool-based active learning works, e.g. , (Geifman and El-Yaniv, 2017; Kirsch et al., 2019); see Sec 4.
To overcome this problem, we propose BMFAL-BC, a batch multi-ﬁdelity active learning approach to reduce the correlations and to promote the diversity of training examples, so that we can improve the learning performance, lower the sample complexity, and better save the cost.
In addition, we take into account the budget constraint in querying the batch examples, which is common in practice (like cloud or high-performance computing)
(Mendes et al., 2020).
Under the given budget, we aim to ﬁnd a batch of multi-ﬁdelity queries that improves the beneﬁt-cost ratio as much as possible.
3.1 Batch Acquisition Function
Speciﬁcally, we ﬁrst consider an acquisition function that allows us to jointly optimize a set of inputs and ﬁdelities.
While it seems natural to consider how to extend (2) to the batch case, the acquisition function in (2) is about the mutual information between y m ( x ) and y M ( x ) .
That means, it only measures the utility of the query ( m, x ) in improving the estimate of the target function at x itself ( i.e. , y M ( x ) ), rather than at any other input.
To take into account the utility in improving the function estimation at all the inputs, we therefore propose a new single acquisition function,
[Equation]
where x (cid:48) ∈ Ω and p ( x (cid:48) ) is the distribution of the input in Ω .
We can see that, by varying the input x (cid:48) in the second argument of the mutual information, we are able to evaluate the utility of the query in improving the estimation of the entire body of the target function.
Hence, it is more reasonable and comprehensive.
Now, consider querying a batch of examples under the budget B , we extend (3) to
[Equation]
where M = { m 1 , . . .
, m n } ,
X = { x 1 , . .
, x n } , and D is the current training dataset.
We can see that the more correlated { y m j ( x j ) } nj =1 , the smaller the mutual information, and hence the smaller the expectation in (4).
Therefore, the batch acquisition function implicitly penalizes strongly correlated queries and favors diversity.
The expected mutual information in (4) is usually analytically intractable.
However, we can efﬁciently compute it with an Monte-Carlo approximation.
That is, we draw A independent samples from the input distribution, x (cid:48) 1 , . . .
, x (cid:48)
A ∼ p ( x (cid:48) ) , and compute
[Equation]
It is straightforward to extend the multi-variate delta method used in DMFAL to calculate the mutual information in (5).
We can then maximize (5) subject to the budget constraint, (cid:80)
Bj =
λ m j ≤ B , to acquire more diverse and informative training examples.
In addition, when parallel computing resources ( e.g. , multi-core CPUs/GPUs and computer clusters) are available, we can acquire these queries in parallel to further speed up the active learning.
However, directly maximizing (5) will incur a combinatorial search over multiple ﬁdelities in M , and hence is very expensive.
Note that the number of examples n is not ﬁxed, and can vary as long as the cost does not exceed the budget B , which makes the optimization even more challenging.
To address these issues, we propose a weighted greedy algorithm to sequentially determine each ( m j , x j ) pair.
3.2 Weighted Greedy Optimization
Speciﬁcally, at each step, we maximize the mutual information increment on one pair of input and ﬁdelity, weighted by the corresponding cost.
Suppose at step k , we have identiﬁed a set of k inputs and ﬁdelities at which to query, Q k
= { ( x j , m j ) } kj =1 .
To identify a new pair of input and ﬁdelity at step k + 1 , we maximize a weighted incremental version of (5),
[Equation]
[Equation]
where Y k = { y m j ( x j ) | ( x j , m j ) ∈ Q k } , and M k are all the ﬁdelities in Q k .
At the beginning ( k = 0 ), we have
Q k = ∅ , Y k = ∅ and M k = ∅ .
Each step, we look at each valid ﬁdelity, i.e. ,
[Equation]
m , and ﬁnd the optimal input.
We then add the pair ( m, x ) that gives the largest ˆ a k +1 into Q k , and proceed to the next step.
Our greedy approach is summarized in Algorithm 1.
1: k ← 0 , Q k ← ∅ , C k ← 0 2: while C k ≤ B do 3:
Optimize the weighted incremental acquisition function in (6): ( x ∗ , m ∗ ) = argmax x ∈ Ω , 1 ≤ m ≤ B − C k ˆ a k
+
( x , m ) 4: if Infeasible then 5: break 6: end if 7: k ← k +8:
Q k ← Q k − 1 ∪ { ( x ∗ , m ∗ ) } 9:
C k ← C k − 1 + λ m ∗ 10: end while 11:
Return Q k
The intuition of our approach is as follows.
Mutual information is a classic submodular function (Krause and Guestrin, 2005), and hence if there were no budget constraints or weights, greedily choosing the input which increases the mutual information most achieves a solution within (1 − 1 /e ) of the optimal (Krause and Golovin, 2014).
However, Leskovec et al.
(2007) observed that when items have a weight (corresponding to the cost for different ﬁdelities in our case) and there is a budget constraint, then the approximation factor can be arbitrarily bad.
We observe, and prove , that this only occurs as the budget is about to be ﬁlled, and up until that point, the weighted greedy optimization achieves the best possible (1 − 1 /e )
-approximation of the optimal.
We can formalize this near (1 − 1 /e ) -approximation in two ways, proven in the Appendix.
Let OPT( B ) be the maximum amount of mutual information that can be achieved with a budget B .
Theorem 3.1.
At any step of Weighted-Greedy (Algorithm 1) before any choice of ﬁdelity would exceed the budget, and the total budget used to that point is B (cid:48) <
B , then the mutual information of the current solution is within (1 − 1 /e ) of OPT( B (cid:48) ).
Corollary 3.1.
If Weighted-Greedy (Algorithm 1) is run until input-ﬁdelity pair ( x , m ) that corresponds with the maximal acquisition function ˆ a k +1 ( x , m ) would exceed the budget, it selects that input-ﬁdelity pair anyways (the solution exceeds the budget B ) and then terminates, the solution obtained is within (1 − 1 /e ) of OPT( B ).
We next sketch the main idea of how to prove the main result.
Adding a new ﬁdelity-input pair ( m, x ) gives an increment of learning beneﬁt ∆ j
= I
( Y k ∪ { y m ( x ) } , y M ( x (cid:48) l ) |D ) − I ( Y k , y M ( x (cid:48) l ) |D ) .
Since we need to trade off to the cost λ m , we can view there are λ m independent copies of x (for the particular ﬁdelity m ), and adding each copy gives ∆ j λ m beneﬁt.
We choose the optimal x and m that maximizes the beneﬁt ∆ j λ m .
Since all the λ m copies of x have the equal, biggest beneﬁt (among all possible choices of inputs in Ω and ﬁdelities in M ), we choose to add them ﬁrst (greedy strategy), which in total gives ∆ j beneﬁt – their beneﬁt does not diminish as each copy is added.
Via dividing by λ m and considering the copies, which each have unit weight, we can apply the standard submodular optimization analysis obtaining (1 − 1 /e )
OPT, at least until we encounter the budget constraint.
3.3 Algorithm Complexity
The time complexity of our weighted greedy optimization is O
( Bλ 1 MG ) where λ 1 is the cost for the lowest ﬁdelity, G is the cost of the underlying numerical optimization ( e.g. , L-BFGS) and acquisition function computation (detailed in (Li et al., 2022)).
The space complexity is O
( Bλ 1 ( r + d + 1)) , which is to store at most B/λ 1 pairs of input locations and ﬁdelities, and their corresponding outputs ( i.e. , the querying results).
Therefore, both the time and space complexities are linear to the budget B .
Related Work
As an important branch of machine learning, active learning has been studied for a long time (Balcan et al., 2007; Settles, 2009; Balcan et al., 2009; Dasgupta, 2011; Hanneke et al., 2014).
While many prior works develop active learning algorithms for kernel based models, e.g. , (Schohn and Cohn, 2000;
Tong and Koller, 2001; Joshi et al., 2009; Krause et al., 2008; Li and Guo, 2013; Huang et al., 2010), recent research focus has transited to deep neural networks.
Gal et al.
(2017) used Monte-Carlo (MC)
Dropout (Gal and Ghahramani, 2016) to generate posterior samples of the neural network output, based on which a variety of acquisition functions, such as predictive entropy and Bayesian Active
Learning by Disagreement (BALD) (Houlsby et al., 2011), can be calculated and optimized to query new examples in active learning.
This approach has been proven successful in image classiﬁcation.
Kirsch et al.
(2019) developed BatchBALD, a greedy approach that incrementally selects a set of unlabeled images under the BALD principle and issues batch queries to improve the active learning efﬁciency.
They show that the batch acquisition function based on BALD is submodular, and hence the greedy approach achieves a 1 − 1 /e approximation.
Other works along this line includes (Geifman and El-Yaniv, 2017; Sener and Savarese, 2018) based on core-set search, (Gissin and Shalev-Shwartz, 2019; Ducoffe and Precioso, 2018) based on adversarial networks or samples, (Ash et al., 2019) based on the uncertainty evaluated in the gradient magnitude, etc .
Different from most methods, DMFAL (Li et al., 2022) conducts optimization-based, rather than pool-based active learning.
That is, they optimize the acquisition function in the entire domain rather than rank a set of pre-collected examples to label.
It is also related to Bayesian experimental design (Kleinegesse and Gutmann, 2020).
In addition, DMFAL for the ﬁrst time automatically queries examples at different ﬁdelities, and integrates these examples in a deep multi-ﬁdelity model to improve the active learning efﬁciency while reducing the cost — which is critical in physical simulation and engineering design.
The pioneer work of Settles et al.
(2008) empirically studies how the human labeling cost can vary in practical active learning, but does not provide a scheme to identify multi-ﬁdelity queries.
Our work is an extension of (Li et al., 2022) to generate a batch of multi-ﬁdelity queries so as to reduce the query correlations, improve the diversity and quality of the training examples, while respecting a given budget constraint.
A counterpart in the Bayesian optimization domain is the recent work BMBO-DARN (Li et al., 2021), which considers batch multi-ﬁdelity queries for optimizing a black-box function.
From the high-level view, BMBO-DARN and our work employ a similar interleaving procedure, i.e. , determining a new batch of queries by optimizing an acquisition function, issuing the queries to collect new examples, and updating the surrogate model.
However, both the learning setting and acquisition function are different.
More important, we consider the budget constraint while BMBO-DARN does not.
Thereby, the computation and optimization techniques of the two works are totally different.
The BMBO-DARN uses Hamiltonian Monte-Carlo samples of the single function output prediction and constructs empirical covariance matrices to compute the acquisition function, while our method and (Li et al., 2022) use the multi-variate delta method and matrix identities to overcome the challenge of the high-dimensional outputs.
BMBODARN uses alternating optimization to search over multiple ﬁdelities, while our work develops a weighted greedy algorithm with additional theoretical guarantees to respect the budget constraint.
Experiment 5.1 Solving Partial Differential Equations
We ﬁrst evaluated BMFAL-BC in several benchmark tasks of computational physics, i.e. , predicting the solution ﬁelds of partial differential equations (PDEs), including Heat , Poisson ’s, and Burgers ’ equations (Olsen-Kettle, 2011).
A numerical solver was used to collect the training examples.
High-ﬁdelity examples were generated by running the solver with dense meshes, while low-ﬁdelity examples by coarse meshes.
The dimension of the output is the number of the grid points.
For example, a 50 × 50 mesh means the output dimension is 2 , 500 .
We provided two-ﬁdelity queries for each PDE, corresponding to 16 × 16 and 32 × 32 meshes.
We also tested three-ﬁdelity queries for
Poisson’s equation, with a 64 × 64 mesh to generate examples at the third ﬁdelity.
We denote the three-ﬁdelity setting by Poisson-3.
The input comprises of the PDE parameters and/or boundary/initial condition parameters.
The details are provided in (Li et al., 2022).
We ran the solver at each ﬁdelity for many times, and computed the average running time.
We normalized the average running time to obtain the querying cost at each ﬁdelity, λ 1 = 1 , λ 2 = 3 and λ 3 = 10 .
To collect the initial training dataset for active learning, we generated 10 ﬁdelity-1 examples and 2 ﬁdelity-2 examples for in the two-ﬁdelity setting, and 10 , 5 , and 2 examples of ﬁdelity-1, 2, 3, respectively, for the three-ﬁdelity setting.
The training inputs of the initial dataset were uniformly sampled from the domain.
To evaluate the prediction accuracy, for each PDE, we randomly sampled 500 inputs, calculated the ground-truth outputs from a much denser mesh: 128 × 128 for Burger’s and Poisson’s and 100 × for Heat equation.
The predictions at the highest ﬁdelity were then interpolated to these denser meshes (Zienkiewicz et al., 1977) to evaluate the error.
Competing Methods.
Since currently there is not any budget-aware, batch multi-ﬁdelity active learning approach (to our knowledge), for comparison, we ﬁrst made a simple extension of the state-of-the-art multi-ﬁdelity active learning method, DMFAL (Li et al., 2022).
Speciﬁcally, to obtain a batch of queries, we ran DMFAL as it is, namely, each step acquiring one example by maximizing (2) and then retraining the model, until the budget is exhausted or no new queries can be issued (otherwise the budget will be exceeded).
We denote this method by (i) DMFAL-BC .
Note that it is still sequentially querying and training inside each batch, but respects the budget.
Next, to conﬁrm the effectiveness of the proposed new acquisition function (3) (based on which, we propose our batch acquisition function (4)), we ran active learning in the same away as DMFAL-BC , except the acquisition function is replaced by a s ( m, x ) λ m where a s is deﬁned by (3).
To compute a s , we used an Monte-Carlo approximation similar to (5), where the number of samples A is 20 .
We denote this method by (ii) MFAL-BC .
In addition, we compared with (iii) DMFAL-BC-RF , which follows the running of DMFAL-BC , but each time, it randomly selects a ﬁdelity m , then maximize the mutual information I (cid:0)
y m ( x ) , y M ( x ) |D (cid:1) — the numerator of (2) — to identify the corresponding input.
Similarly, we tested (iv) MFAL-BC-RF , which follows the execution of MFAL-BC , but each time, it randomly selects a ﬁdelity m and maximizes a s ( m, x ) .
Again a s is computed by the Monte-Carlo approximation.
For all these methods, we used L-BFGS for the input optimization.
Finally, we tested (v) Batch-FR-BC , which randomly samples both the ﬁdelity and input at each step ( i.e. , fully random), until the budget is used up or no more ﬁdelities are available.
Then the batch of the acquired examples were added into the dataset altogether to retrain the model.
Note that there are other possible acquisition functions, e.g. , the popular predictive variances and BALD (Houlsby et al., 2011).
Their multi-ﬁdelity versions have been tested and compared against DMFAL in (Li et al., 2022), and turns out to be inferior.
Hence, we did not test their possible variants/extensions in our paper.
Settings and Results.
All the methods were implemented by Pytorch (Paszke et al., 2019).
We followed the same setting as in Li et al.
(2022) to train the deep multi-ﬁdelity model (see Sec. 2.2), which employed a two-layer NN at each ﬁdelity, tanh activation,
and the layer width was selected from { 20 , 40 , 60 , 80 , 100 } from the initial training data.
The dimension of the latent output was 20.
The learning rate was tuned from { 10 − 4 , 5 × 10 − 4 , 10 − 3 , 5 × 10 − 3 , 10 − 2 } .
We set the budget for acquiring each batch to 20 (normalized seconds), and ran each method to acquire 25 batches of training examples.
We tracked the running performance of each method in terms of the normalized root-mean-square-error (nRMSE).
We repeated the experiment for ﬁve times, and report the average nRMSE vs. the accumulated cost ( i.e. , the summation of the corresponding λ ’s in each acquired example) in Fig.
The shaded region exhibits the standard deviation.
As we can see, BMFAL-BC consistently outperforms all the competing methods by a large margin.
At very early stages, all the methods exhibit similar prediction accuracy.
This is reasonable, because they started with the same training set.
However, along with more batches of queries, BMFAL-BC shows better performance, and its discrepancy with the baselines is in general growing.
In addition, MFAL-BC constantly outperforms DMFAL-BC.
Since they conduct the same one-by-one querying and training strategy, the improvement MFAL-BC reﬂects the advantage of our new single acquisition function (3) over the one used in DMFAL.
Together these results have demonstrated the advantage of our method.
5.2 Topology Structure Optimization
Next, we applied our approach in topology structure optimization, which is critical in many manufacturing and engineering design problems, such as 3D printing, bridge construction, and aircraft engine production.
A topology structure is used to describe how to allocate the material, e.g. , metal and concrete, in a designated spatial domain.
Given the environmental input, e.g. , a pulling force or pressure, we want to identify the structure with the optimal property of interest, e.g. , maximum stiffness.
Traditionally, we convert it to a constraint optimization problem, for which we minimize a compliance objective with a total volume constraint (Sigmund, 1997).
Since the optimization often needs to repeatedly run numerical solvers to solve relevant PDEs, the computation is very expensive.
We aim to learn a surrogate model with active learning, which can predict the optimal structure outright given the environmental input.
Speciﬁcally, our task is to design a linear elastic structure in an L-shape domain discretized in [0 , 1] ×
The environmental input is a load at the bottom half right and described by two
[Figure]
Accumulated cost 0 .0 .0 .1 .n
R
M
SE
[Figure] [Figure]
[Figure]
Accumulated cost
[Figure]
1 .n R
M
SE
[Figure] [Figure]
Accumulated cost 0 .0 .0 .1 .n
R
M
SE
[Figure]
Accumulated cost
[Figure]
parameters: angle (in [0 , π 2 ] ) and location (in [0 . 5 , 1] ).
Given a particular load, we aim to ﬁnd the structure that has the maximum stiffness.
To calculate the compliance in the structure optimization, we need to repeatedly apply the Finite Element Method (FEM) (Keshavarzzadeh et al., 2018), where the ﬁdelity is determined by the mesh.
In the active learning, the training examples can be acquired at two ﬁdelities.
One corresponds to a 50 × 50 mesh used in the FEM, and the other 75 × 75 .
The output dimension of the two ﬁdelities is therefore 2 , 500 and 5 , 625 , respectively.
The querying cost is the normalized average time to ﬁnd the optimal structure (with the standard approach), λ 1 = 1 and λ 2 = 3 .
To evaluate the performance, 500 test structures were randomly generated with a 100 ×mesh.
We interpolated the high-ﬁdelity prediction of each method to the 100 × 100 mesh and then evaluated the prediction error.
At the beginning, we uniformly sampled the input ( i.e. , load angle and location) to collect 10 structures at the ﬁrst ﬁdelity and 2 at the second ﬁdelity, as the initial training set.
We then ran all the active learning methods, with budget 20 per batch, to acquire 25 batches of examples.
We examined the average nRMSE along with the accumulated cost of acquiring the examples.
The results are reported in Fig. 3a.
We can see that the prediction accuracy of BMFAL-BC is consistently better than all the competing methods during the active learning.
The improvement becomes larger when more examples are acquired.
The results conﬁrm the advantage of BMFAL-BC.
5.3 Predicting Spatial-Temporal Fields of Flows
Third, we evaluated BMFAL-BC in computational ﬂuid dynamics (CFD).
We considered a classical application where the ﬂow is inside a rectangular domain (represented by [0 , 1] × [0 , 1] ), and driven
[Figure] [Figure]
by rotating boundaries with a constant velocity (Bozeman and Dalton, 1973).
The velocities of different parts inside the ﬂow will vary differently
[Figure]
and eventually lead to turbulent ﬂuids.
To compute the velocity ﬁeld in the time spatial domain
[Figure]
we need to solve the incompressible Navier-Stokes equations (Chorin, 1968)
[Figure]
which is known to be computationally challenging, especially under large
Reynolds numbers.
We considered the active learning task of predicting the ﬁrst component of the velocity ﬁeld at 20 evenly spaced points in the temporal domain [0 , 10] .
The training examples can be queried at two ﬁdelities.
The ﬁrst ﬁdelity uses a 50 × 50 mesh in the [0 , 1] × [0 , 1] spatial domain, and the second ﬁdelity 75 × 75 .
Accordingly, the output dimensions are 50 , 000 and 112 , 500 ; the cost per query is λ 1 = 1 and λ 2 = 3 .
The input is ﬁve dimensional, including the velocities of the four boundaries and the Reynold number.
The details are given in (Li et al., 2022).
For testing, we computed the solution ﬁelds of 256 random inputs, using a 128 × 128 mesh.
We used the cubic-spline interpolation to align the prediction made by each method to the 128 × 128 grid, and then calculated normalized RMSE.
To conduct the active learning, we randomly generated 10 and 2 examples in each ﬁdelity as the initial training set.
We set the budget to 10 , and ran each method to acquire 25 batches.
We repeated the experiment for ﬁve times, and examined how the average nRMSE varied along with the accumulated cost.
As shown in Fig.
3b, BMFAL-BC keeps exhibiting superior predictive performance during the course of active learning.
Again, the more examples acquired, the more improvement of BMFAL-BC upon the competing methods.
The results are consistent with the previous experiments.
Note that throughout these comparisons, we focus on the accumulated cost of querying (or generating) new examples, because it dominates the total cost, and in practice is the key bottleneck in learning the surrogate model.
For example, in topology optimization (Sec 5.2) and the ﬂuid dynamic experiment, running a high-ﬁdelity solver once takes 300-500 seconds on our hardware, while the surrogate training takes less than 2 seconds, and our weighted greedy optimization of the batch acquisition function (Algorithm 1) takes a few seconds.
One can imagine for practical larger-scale problems, the simulation cost, e.g. , taking hours or even days to generate one example, can be even more dominant and decisive.
5.4 Inﬂuence of Different Budgets
Finally, we examined how the budget choice will inﬂuence the performance of active learning.
To this end, we varied the budget B in { 20 , 35 , 50 } and tested all the methods for Poisson’s, Burger’s and heat equations.
We used the same two-ﬁdelity settings as in Sec. 5.1.
For each budget, we ran the experiment for ﬁve times.
We show the average nRMSE vs. the accumulated cost in Fig.
As we can see, the larger the budget per batch, the better the running performance of BMFAL-BC.
This is reasonable, because a larger budget allows our method to generate more queries in each batch and in the meantime to account for their correlations or information redundancy.
Accordingly, the acquired training examples are overall more diverse and informative.
Again, BMFAL-BC outperforms all the competing methods under every budget.
The improvement of BMFAL-BC is bigger under larger budget choices.
This together demonstrates the advantage of batch active learning that takes into account the correlations between queries.
Conclusion
We have presented BMFAL-BC, a budget-aware, batch multi-ﬁdelity active learning algorithm for high-dimensional outputs.
Our weighted greedy algorithm can efﬁciently generate a batch of inputﬁdelity pairs for querying under the budget constraint, without the need for combinatorially searching over the ﬁdelities, while achieving good approximation guarantees.
The results on several typical computational physical applications are encouraging.
Acknowledgments
This work has been supported by MURI AFOSR grant FA9550-20-1-0358 and NSF CAREER Award
IIS-2046295.
JP thanks NSF CDS&E-1953350, IIS-1816149, CCF-2115677, and Visa Research.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]