Para-CFlows : C k -universal Diffeomorphism Approximators as Superior Neural Surrogates
Abstract
Invertible neural networks based on Coupling Flows ( CFlows ) have various applications such as image synthesis and data compression.
The approximation universality for CFlows is of paramount importance to ensure the model expressiveness.
In this paper, we prove that CFlows can approximate any diffeomorphism in C k -norm if its layers can approximate certain single-coordinate transforms.
Specifically, we derive that a composition of affine coupling layers and invertible linear transforms achieves this universality.
Furthermore, in parametric cases where the diffeomorphism depends on some extra parameters, we prove the corresponding approximation theorems for parametric coupling flows named
Para-CFlows .
In practice, we apply Para-CFlows as a neural surrogate model in contextual Bayesian optimization tasks, to demonstrate its superiority over other neural surrogate models in terms of optimization performance and gradient approximations.
Code will be avaliable at https://gitee.com/mindspore/models/ tree/master/research/bo/paracflow .
Introduction
Invertible neural networks (INNs) such as coupling flows are firstly introduced as a class of generative models with a tractable likelihood [14, 28, 44] and have shown their usefulness and powerfulness in various machine learning tasks such as inverse problems [2], probabilistic inference [32] and feature extraction [25] in recent years.
With plenty of successful applications of INNs, one would wonder if such a type of models have the universal expressiveness.
As most generative models mainly concern about the transform between distributions, existing works such as [22, 26] focused on the expressiveness from the distribution perspective.
However, the expressiveness from the distribution perspective does not result in the expressiveness from the mapping perspective, as there are a large (or even infinite) number of functions mapping the given source µ to the given target ν .
In many applications, knowing the distributional universality is not yet enough.
One may be interested in knowing if the optimal transport [45], which finds emerging applications in many fields, e.g., machine learning [35], wireless communication [33] and economics [18], can be approximated by invertible neural networks.
Therefore, beyond the distributional universality, it is also important to investigate the universality from the mapping perspective.
As INNs are always invertible, it is natural to consider their approximation ability to diffeomorphisms.
Diffeomorphism plays an important role in mathematics, physics and engineering domains with applications in fluid dynamics [16], wave propagation [20], robot controls [37] etc.
A natural question comes to the surface: can all diffeomorphisms be approximated by INNs?
Besides, whether the INNs can approximate the derivatives in the meanwhile is also interesting and important, e.g., it provides theoretical guarantee for black box optimization tasks based on surrogate models when gradients are utilized.
More importantly, in many real-world problems, e.g., 3D Euclidean groups, ODE systems and invertible PDE systems with time t as its parameter, diffeomorphisms are usually described as a parametric type: a parametric diffeomorphism is a function f
( y , x ) : R m + d → R d such that for any fixed parameter y 0 ∈ R m , f ( y 0 , · ) is a diffeomorphism between R d and R d .
It is important to know if they can also be approximated by INNs with additional inputs of parameters.
Existing works including [42] have been proposed to investigate the universality of INNs over diffeomorphisms, however it is not able to handle C k -diffeomorphisms when k
=
d + 1 with d as the dimension of the space.
In this paper, we address this limitation and more importantly, inspired by the contextualization way proposed in [3], we generalize this structural theorem to the parametric diffeomorphism situation, by concatenating parameters with the variable to be transformed, forming a new subgroup over a higher dimensional space, and provide a complete proof.
This structural theorem shows that any compact-supported diffeomorphisms can be decomposed into finitely many compositions of singlecoordinate transforms, which we can easily prove to be C k -approximated by, e.g., affine coupling flows together with dimension augmentation ([15], [46]).
Lastly, we suggest using Para-CFlow , an affine coupling flow model with dimension augmentation and contextualization, as a superior neural surrogate for black box optimization.
The superiority lies in: 1) it uses INN architecture to preserve full rank of the action; 2) it has C k -universality when the target system satisfies certain conditions.
Specifically, 2) facilitates much more efficient surrogate and gradient-descent based black box optimization compared to other neural surrogates.
Overall, our contribution is three-fold: 1) We improve the results in [42] to higher-order derivatives with a simpler proof, and give an upper bound on number of layers needed for approximating a certain diffeomorphism; 2) We generalize coupling flows to parametric coupling flows and prove their C k -universal approximation to parametric diffeomorphisms using a novel proving technique; 3)
We propose a practical neural network structure of the parametric affine coupling flows and verify the advantage of using this for contextual Bayesian Optimization (BO) tasks.
Preliminary
Here we introduce some prior knowledge on (parametric) diffeomorphism, INNs and universality, as well as existing works on universality of INNs.
As our theoretical results are motivated by and improve upon [42], some of our notations and preliminaries are adopted from them.
In what follows, we always assume k, m, n, d ∈ N + , where k represents the derivative order, and m, n, d represent dimensionalities for some Euclidean spaces.
All vectors are supposed to be row vectors.
2.1 C k -diffeomorphism groups on R d
C k -diffeomorphisms (group).
Consider an invertible map f from R d to R d .
f is said to be a
C k -diffeomorphism, if f has up to k -th continuous derivatives, and det Df ( x ) ̸ = 0 for all x .
One can easily verify that, given f, g : C k diffeomorphisms from R d to R d , f ◦ g and f − 1 is still a C kdiffeomorphism from R d to R d .
If we denote Diff k ( R d ) ≜ { f : f is a C k diffeomorphism over R d } , we see that Diff k
( R d ) has a natural group structure with composition as its group operator.
Compactly supported C k -diffeomorphisms.
In real applications, finite data cannot cover the whole space R d .
Besides, existing approximation theories only guarantee the capability over some bounded compact set K .
Therefore we consider compactly supported functions or diffeomorphisms.
A function f : R d → R is said to be compactly supported, if there exists a compact set K such that f ( x ) = 0 , for all x / ∈ K .
Similarly, a diffeomorphism f is said to be compactly supported, if there exists a compact set K such that f ( x ) = x , for all x / ∈ K , resulting in that all the nondiagonal components of Df are compactly supported.
We define Diff kc ( R d ) ≜ { f ∈
Diff k
( R d ) : f compactly supported } .
Obviously Diff kc ( R d ) is a subgroup of Diff k
( R d ) .
C k -parametric diffeomorphisms.
A parametric diffeomorphism is a family of diffeomorphisms with some parameter α : { f α } α ∈ A , where for any given α ,
f α is a R d → R d diffeomorphism.
Usually the parameter is described by some vector y ∈ R m , and f y varies continuously or smoothly w.r.t.
y .
We denote f ( y , x ) =
f y ( x ) .
We can embed it into a higher dimensional diffeomorphism F ( y , x ) = (cid:0) y , f ( y , x ) (cid:1) .
One can verify directly that F ∈ Diff k
( R m + d ) given that (1) f
( y 0 , x ) ∈
Diff k
( R d ) for any fixed y 0 ; (2) f is k -th differentiable w.r.t.
( y , x ) .
Obviously Diff k,m,dc ≜ { F ∈ Diff kc
( R m + d ) :
F ( y , x ) =
( y , f
( y , x )) with y ∈ R m , x ∈ R d } , whose elements keep the first m coordinates and change the last d ones, is a subgroup of Diff kc ( R m + d ) .
2.2 INNs based on parametric coupling flows
Here we introduce the classical INNs and investigate the space generated by them.
Invertible linear transforms.
First, let us define the invertible linear transforms (ILT):
[Equation]
and the parametric case where y are parameters,
here GL d ( R ) includes all d × d regular matrices.
[Equation]
Coupling flows.
We now define invertible coupling flows [34], a specific type of nonlinear transforms:
[Equation]
, where ϕ ( x ≤ i , · ) : R d − i → R d − i is a diffeomorphism for each fixed x ≤ i .
Specifically, when ϕ ( x ≤ i , x >i ) = x >i ⊙ exp ( σ ( x ≤ i ))
+ t ( x ≤ i ) , it is the so-called affine coupling flow [13] and we denote Φ d,i,σ,t = Φ d,i,ϕ for such ϕ .
σ, t are some functions with d − i output units, typically modeled with deep neural networks.
⊙ represents the point-wise product.
x ≤ i =
( x 1 , · · · , x i ) , x >i =
( x i +1 , · · · , x d ) for x = ( x 1 , · · · , x d ) .
Similarly, for parametric cases, we have:
[Equation]
and specifically, ϕ ( y , x ≤ i , x >i ) = x >
[Equation]
⊙ exp
[Equation]
≤
[Equation]
+ t ( y , x ≤
[Equation]
) for affine-type coupling flows
and we denote Φ d,i,m,σ,t =
Φ d,i,m,ϕ for such ϕ .
Single-coordinate affine coupling flows (SACFs) denotes the flows with only the last coordinate changed.
Let H d be a set of functions from R d to R .
We define
H -SACF d ≜ { Φ d,d − 1 ,σ,t : σ, t ∈ H
d − 1 } , and H -SACF m,d ≜ { Φ d,d − 1 ,m,σ,t : σ, t ∈ H m + d − 1 } .
Note that any multi-coordinates affine coupling flows can be represented by finite composition of
SACFs and invertible linear transforms, it suffices to just consider the universality of SACFs.
Invertible neural networks.
Now let us combine linear invertible transform layers and some coupling flow layers to construct our INNs.
Let G be a set consisting of invertible coupling flows.
We define the set of INNs based on G and the parametric case:
[Equation]
When G contains H -SACF d (or H -SACF m,d in parametric cases), it is equivalent to replacing ILT d (or ILT m,d ) by the symmetric group S d containing all the permutation over d coordinates operating on x .
This type of networks are well known as Real-NVPs [13].
One can also define other types of coupling flows.
Nevertheless, the theoretical guarantee of any coupling flow can be verified by simply checking their universality to single-coordinate transforms, as our main result stated in Thm. 3.5.
2.3 Different types of universality and their relations
Here we define the functional universality.
See Appx.
A for distributional universality and more.
Definition 2.1.
( L p /L ∞ /C
k -universality [9]).
Let M be a set of measurable mappings from R n to
R d .
Let p ∈
[1 , ∞ ) , k ∈ N
+ and let F be a set of measurable mappings f : U f → R d , where U f is a measurable subset of R n which may depend on f .
We say that M has L p ( or L ∞ , C k ) -universality for F , if for any f ∈ F any ϵ > 0 , and any compact subset K ⊆
U f , there exists a g ∈ M such that, ∥ f − g ∥
L p ( K ) <
ϵ ( or ∥
f − g ∥
L ∞ ( K ) , ∥ f − g ∥
C k ( K ) <
ϵ , definition of these norms in Appx.
A).
If M has the C k -universality, M has the L ∞ -universality because C k ( K ) is dense in L ∞ ( K ) for any compact K .
Also, if M has the L ∞ -universality, M has the L p -universality for any 1 ≤ p < ∞ .
If an L p -universality is satisfied for some 1 < p < ∞ , then for any 1 ≤ q < p , L q -universality is ensured.
Finally, L 1 -universality implies distributional universality but the reverse is not true.
2.4 Related works
Coupling (or triangular) flows.
Authors of [24, 6] proved that the distributional universality of a flow family h can be deduced, if h is dense in the set of all monotone functions by pointwise convergence topology.
In particular, [22] proved the distributional universality for neural autoregressive flows, and [26] proved that for sum-of-square flows.
[42] generalized it to L p universality with 1 ≤ p < ∞ for affine coupling flows, and L ∞ universality for autoregressive flows and sum-of-square flows.
Non-coupling flows.
The expressiveness of general non-coupling flows is not well studied, as most existing works restrict the form of nonlinearity to certain types, e.g., planar and radial flows [36],
Sylvester flows [43], in order to easily compute the determinant of the Jacobian matirx and the inverse maps.
[29] gave a first study over some specific distributions, while the universal distributional results still remains unknown.
Another type, iResNet [5], was proposed based on residual network (ResNet) [21] to improve nonlinearity as well as the log determinant computation efficiency.
[46] proved that iResNet, capped by linear layers or with extra dimensions, has C 0 -universality.
Continuous time flows.
The ODE-based method is also a major class of flow models as introduced in [10, 19, 11, 38].
[11] gave counterexamples for the C 0 -universality of neural ODEs, however its distributional universality is not yet addressed.
An “augmented” neural ODEs was proposed by [15] and then analyzed by [46].
They showed that embedding the original d dimensional neural ODE into a d + 1 dimensional space can bypass the original counterexamples, achieving a C 0 -universality.
Main results
In this section, we present our main results, following the architecture similar to [42].
Section 3. provides a proof for non-parametric cases, showing that: 1) the C k -universality over the Diff kc
( R d ) can be achieved when the C k -universality over a simple subgroup of Diff kc ( R d ) is ensured; 2) affine coupling layers with one zero-padding has C k -universality over such a simple subgroup.
In addition, we estimate the number of layers needed for approximate certain diffeomorphism in Thm 3.3.
In
Section 3.2, we generalize these results to parametric cases with a novel proof technique.
3.1 Non-parametric cases
Here we outline the main steps of our proof, and the complete proof is available in Appx.
B.
Recall the space Diff kc ( R d ) :
All the C k , R d → R d diffeomorphisms which are compactly supported.
There are two advantages of choosing Diff kc ( R d ) instead of
Diff k
( R d ) : firstly the compactly supported property greatly simplifies the structure of this group; secondly for any diffeomorphism F ∈
Diff k
( R d ) and any compact set K , there exists a compactly supported diffeomorphism f ∈ Diff kc ( R d ) such that F | K = f
| K , i.e., F ( x ) = f
( x ) , for all x ∈ K (See Lemma 6 in [42]).
Directly proving the universality to a general diffeomorphism is difficult.
It is beneficial to decompose the original hard problem into a series of simpler ones as follows.
Definition 3.1.
A diffeomorphism τ ∈ Diff kc ( R d ) is in single-coordinate transforms S k,
dc , if τ ( x ) =
( x 1 , x 2 , · · · , x i − 1 , τ i ( x ) , x i +1 , · · · , x d ) , i.e., only one coordinate is altered, and τ i ( x )
=
τ i ( x 1 , · · · , x d ) is a function from R d to R which is monotonic for x i .
The theorem as follows shows that, all diffeomorphisms in Diff kc ( R d ) can be constructed from S k,dc .
Theorem 3.2.
For any subgroup H ⊆
Diff kc ( R d ) s.t.
S k,
dc ⊆ H , H
=
Diff kc ( R d ) .
That is, for any f ∈ Diff kc
( R d ) , there exists τ 1 , τ 2 , · · · , τ s ∈ S k,dc , s ∈ N s.t.
f
= τ s ◦ τ s − 1 ◦ · · · ◦ τ 1 .
Proof.
The following two steps are sketched: 1) If H ⊆
Diff kc ( R d ) and S k,
dc ⊆ H , H contains all near-identity diffeomorphisms (Def. B.3).
The detail is stated in Cor.
B.6.
If H ⊆ Diff kc ( R d ) contains all near-identity diffeomorphisms, H
=
Diff kc ( R d ) .
The detail is stated in Lem.
B.4.
We further investigate the number of single-coordinate transforms needed to represent an arbitrary diffeomorphism in Thm. 3.3 as follows.
The definition of dist Diff 1 c ( R d ) and the proof is provided in
Appx.
B.
Theorem 3.3.
Given f ∈ Diff kc ( R d ) , we denote dist Diff 1 c ( R d ) ( f, I ) =
ℓ for some ℓ > 0 , the minimal length of paths between f and the identity map I lying in Diff 1 c ( R d ) with the C 1 -norm induced metric.
Then f can be decomposed into d ( d − 1) ℓ single-coordinate transforms.
Thm.
3.2 decomposes the original general complex diffeomorphism into finitely-many simple diffeomorphisms.
It is natural to ask: if we can approximate all the S k,dc well, can we also approximate
Diff kc ( R d ) well?
Thm.
3.4 gives a positive answer to the question.
Theorem 3.4.
(Approximation for composition.)
Suppose G is a group of diffeomorphisms over R d .
Given a set of diffeomorphisms F over R d , denote F as the semigroup generated by F .
If G has
C k -universality for F , G has C k -universality for F .
Proof.
We only need to prove the following statement: If G has C k -universality for f 1 , f 2 ∈ F , then G has C k -universality for f 1 ◦ f 2 ∈ F .
By the property of semigroups, we know that G has
C k -universality for the whole F .
Details can be found in Appx.
B.3.
Combining Thm. 3.2 and 3.4, noticing that S k,dc is inverse-invariant (for any τ ∈ S k,dc , τ − 1 ∈ S k,dc ), we can immediately obtain following corollary.
Corollary 3.5.
Suppose G is a group of diffeomorphisms over R d , if G has C k -universality for S k,dc , then G has C k -universality for Diff kc ( R d ) .
As stated in Cor. 3.5, any INNs that approximate S k,dc will eventually approximate Diff kc ( R d ) in
C k -norm, which is an extension from the L p and L ∞ universality stated in [42].
Furthermore, we generalize this result to parametric cases in the next section.
Affine-Coupling Flows.
Now we only need to prove the universality of G -INN d over S k,d c , when G contains H -SACF d .
However, even for such a simple class of diffeomorphisms, G -INN d is not able to approximate it very well: existing works [42] shows that G -INN d has L p universality over S k,dc .
Whereas they gave a negative conjecture for L ∞ universality, let alone the C k -universality.
However, despite the lack of easily computed log-determinant for density evaluation, dimension augmentation has been proven to reach better approximation ability for Neural ODEs [15], [46].
Similarly, when the single-transform is embedded into a higher dimension Euclidean space by padding zeros, the approximation ability becomes much stronger, achieving C k -universality, far beyond L p universality.
We first define the canonical immersion ι d , and submersion π d :
[Equation]
Thm.
3.6 says by raising the dimension by 1, we can achieve C k -universality.
Proof is in Appx.
B.4.
Theorem 3.6.
For any compact set K ⊆ R d , and any τ ∈ S k,dc , τ : ( x 1 , x 2 , · · · , x d − 1 , x d ) −→ ( x 1 , x 2 , · · · , x d − 1 , τ d ( x )) , there exists ˜ τ in G -INN d +1 with G ⊇ H -SACF
d +1 , such that ∥ τ − π d ◦ ˜ τ ◦ ι d ∥ C k ( K ) <
ϵ , given that H d has C k -universality over functions K → R .
3.2 Parametric cases
Following Section 3.1, we generalize the results to parametric cases.
We use a novel proof technique, by concatenating parameters with the primary input to be transformed, forming a new subgroup over a higher dimensional space, and exploiting the structure of this subgroup.
We denote S k,m,dc ≜ S k,m + d c ∩ Diff k,m,dc , which is the set of transforms acting on a single coordinate of x .
Recall the definition of compactly supported parametric diffeomorphisms group in Section 2.1, where y acts as parameters.
One might raise a question: why not directly use the result in non-parametric case, as all the elements in Diff k,m,dc are also in Diff kc
( R m + d ) ?
Note that the decomposition theorem only ensures that S k,m + d c can generate Diff kc ( R m + d ) and then Diff k,m,dc , it does not guarantee that S k,m,dc can generate Diff k,m,dc .
Without such a guarantee, we have to use operators that may alter coordinates of both y and x , and the coordinates of y in the output layer should be exactly the same as the coordinates of y in the input layer, which restricts the flexibility and increases the approximation difficulty.
Thus we derive new dedicated theorems for parametric cases.
Theorem 3.7.
For any subgroup H ⊆
Diff k,m,dc s.t.
S k,m,
dc ⊆ H , H =
Diff k,m,dc .
That is, for any f ∈ Diff k,m,dc , there exists τ 1 , τ 2 , · · · , τ s ∈ S k,m,dc , s ∈ N s.t.
f
= τ s ◦ τ s − 1 ◦ · · · ◦ τ 1 .
Proof.
The following two steps are sketched: 1) If H ⊆
Diff k,m,dc and S k,m,
dc ⊆ H , H contains all near-identity diffeomorphisms (Def. B.3).
The detail is stated in Cor.
B.6.
If H ⊆ Diff k,m,dc contains all near-identity diffeomorphisms, H
=
Diff k,m,dc
The detail is stated in Lem.
B.10.
Given the above theorem, we easily obtain the C k -universality for Diff k,m,dc following Section 3.1.
Theorem 3.8.
Suppose G is a group of diffeomorphisms over R m + d .
If G has C k -universality for
S k,m,dc , then G has C k -universality for Diff k,m,dc .
A proof of G -INN m,d universality over S k,m,dc , when G contains H -SACF m,d , follows Thm.
3.6 in
Section 3.1, and thus the detail of the proof is skipped here.
Para-CFlows as superior neural surrogates for black box optimization
In real applications such as control and optimization of telecommunication networks, power grid systems, and electronic systems, we aim to optimize a complicated system under various contexts.
Specifically, we are interested in an unknown function b ( a , c ) :
A × C → R , where A ⊆ R d a , C ⊆
R d c are connected and positive measurable.
a is a set of actions one can tune and c is a context vector representing exogenous stimuli.
In most cases, we are interested in optimizing the function b , i.e. finding a ∗ =
arg max a b ( c , a ) given a context c .
Our task is to quickly find a ∗ at each c using as few queries as possible.
In this paper, we advocate to use Para-CFlows , affine coupling flow models with dimension augmentation and parameterization (context information inputted as parameters), to model the relationship between b and a , c .
Specifically, Para-CFlows are constructed as follows:
Step 1 : We first map a into a higher dimensional space R d ( d > d a ): ϕ 0 : a (cid:55)→ ( a , aW ) ≜ ˜ a (0) , where
W ∈ R
d
a × ( d − d a ) .
By construction, rank ( ϕ 0 ) =
d a .
Step 2 : We further compose ϕ
0 with a series of invertible affine coupling layers ϕ
i , 1 ≤ i ≤ N operating in R d (Eq.
For any x ∈ R d , let d ′ =
max { d a , [ d 2 ] } , we define
[Equation]
σ i , t i are functions with d − d ′ outputs, implemented by any nonlinear functions like DNNs.
Step 3 : We write ˜ a ( i ) =
ϕ i P i
(˜ a ( i − 1) ) , 1 ≤ i ≤ N , where P i are any fixed random permutation, and
[Equation]
which is followed by a simple function ˆ b : R d → R represented by a simple neural network that shall be trained.
Then the whole model is trained by minimizing the Mean Square Error (MSE).
[Equation]
=
[Equation]
c j
[Equation]
trainning samples.
Although it is a straight-forward extension of coupling flows and a similar one was found in [3], our key contribution mainly lie in the theoretical part: for the first time, we prove its C k -universality to
[Figure]
MLP
Regressor
AT: Affine Transform ST: Straight-Through
DA: Dimension Ascending SF: Shuffling
AT
ST
DA
SF SF
ST ST
ST parametric diffeomorphisms and with this property, we advocate to apply it to address an vital action sensitivity issue in black box optimization other than computer vision tasks where flow models are most widely applied.
We now analyze the superiority of Para-CFlows as the surrogate for black box optimization.
Property A: Para-CFlows preserves full information of actions.
Existing methods [30, 12, 1, 17, 23] for BO do not handle well the weak action sensitivity issue of many real system optimization tasks [40], where the actions/configuration parameters’ effect is overwhelmed by the highly volatile system state and thus the surrogate model
ˆ f
( c , a ) can easily degenerate to ˆ f ( c ) due to a small sample size, especially when d c ≫ d a .
On the contrary, Para-CFlow , by construction, preserves all the information (the rank) of a up to the last feature extraction layer before the final regression layer.
Property B: Para-CFlows has C k universality to all rank-preserving systems.
Based on the universality of parametric coupling flows in Section 3, we further investigate what kind of objective function b can be approximated by Para-CFlows .
We give the necessary condition to b as follows.
Assumption 4.1.
b is an observation of the inner system state Φ : b ( c , a ) =
l (Φ( c , a )) and the system is rank-preserving , i.e. the inner state Φ( c , a ) :
C × A → R d ϕ is a differentiable function and preserves full information of actions, i.e., rank (cid:0) ∂
Φ ∂ a (cid:1)
=
d a .
Under Assumption 4.1, we can get the following result, which ensures the expressiveness of our architecture,
Theorem 4.2.
For any system Φ : C × A → R d ϕ satisfying Asmp.
4.1, if d ≥ 2 d ϕ and Φ have up to k -th order derivatives, then for any ϵ > 0 , we can always find a ϕ in the form (2) ,
such that ∥ Φ( c , a ) − πϕ ( c , a ) ∥ C k ( C×A ) ≤ ϵ , given that σ i , t i , ˆ b are C k -universal function approximator.
Here π is the canonical submersion: R d → R d ϕ , ( x 1 , x 2 , · · · , x d ϕ , · · · , x d ) (cid:55)→ ( x 1 , x 2 , · · · , x d ϕ ) .
Readers can check Appx.
B.6 for the proof.
Thm 4.2 shows that the inner system Φ can be uniformly approximated by the feature extraction layer ϕ of Para-CFlow .
The observation function l ( · ) can then be approximated by ˆ b as a result of the universality of Multi-Layer Perceptons (MLPs).
The C k -approximability property of Para-CFlow are of vital importance for black box optimization.
Let us just focus on the exploitation phase, i.e., a ∗ = arg maxˆ b ( ϕ ( c , a )) .
When the action space is continuous and high dimensional (though in this paper, we focus on the context dominating tasks where d c ≫ d a ), one is not able to find the optima easily by brute-force search.
Many existing works utilize meta-heuristic such as Evolutionary Algorithm (EA) of which the performance heavily depends on the population size and the number of iterations, and thus it is computation demanding and time consuming.
Instead, one can apply Gradient Descent (GD) to the surrogate if it is differentiable, i.e. a t +1 ← a t + η ∇ a ˆ b ( ϕ ( c , a ))
| a = a t .
However, for the GD method to work, it requires that the surrogate approximates not only the function value but also its first-order derivative.
However function value approximation does not necessarily guarantee derivative approximation.
A simple illustrative example is f n ( x ) =
g ( x ) + sin ( nx ) /n .
One can easily observe that f n ( x ) → g ( x ) , but f ′ n ( x ) =
g ′ ( x ) + cos ( nx ) will never converge to g ′
( x ) when n → ∞ .
With this argument, we advocate Para-CFlow as an adequate choice for surrogate modeling for continuous action space
BO.
We also find it suitable to approximate Q-function in Reinforcement Learning (RL), especially for DPG [39], DDPG[31] and references therein where one need to estimate the gradient of the
Q-function w.r.t.
the action to train the actor network.
Extension to DRL will be left for future work.
With the aforementioned properties, our proposed Para-CFlow model can serve as a superior neural surrogates and in the setting of BO, we can easily adopt existing SOTA methods of Bayesian deep learning and deep ensembles to facilitate the uncertainty modeling with Para-CFlow .
Experiments
In this section, we conduct experiments to verify the expressiveness of our proposed Para-CFlow and demonstrate its advantage as neural surrogate models in application to black-box optimizations.
5.1 Learning diffeomorphism “Taiji”
We demonstrate the expressiveness of our model to learn a parametric diffeomorphism with a synthetic task called “Taiji”.
We define our target parametric diffeomorphism:
[Equation]
where ˜ θ
= θ + y arccos(min( ρ, 1)) .
[Equation]
On the unit circle ρ = 1 , it is not differentiable, thus we expect the trained model to have irregular derivatives over the unit circle, while maintain accurate derivatives elsewhere.
First, we design a task with sufficient amount of data to verify the C 1 approximation ability of
Para-CFlows .
We generate samples { ( x i , y i ) } 1 ≤ i ≤ 30000 with x i ∼ U [ − 1 , 1]
and y i ∼ U [0 , 1] independently.
Under the polar coordinate representation, i.e., x i = ( ρ i cos θ i , ρ i sin θ i ) , we calculate the target f y i ( x i )
=
( ρ i cos ˜
θ i , ρ i sin ˜ θ i ) according to Eq.
We use a 6-layers affine coupling flows, each layer composed with a random permute.
The coupling functions σ, t are implemented by 1-hidden-layer Multi-Layer Perceptron ( MLP ) with hidden-unit number comparable to input dimension.
We pad zeros to the input x , and use the first two output dimensions to compute the loss.
[Figure]
the N -fold composition ˆ f 1 ◦ · · · ◦ ˆ f 1 . 2(b):
Comparing Para-CFlow with Resnet and MLP with small sample size and parameter y
( i ) =
y, 1 ≤ i ≤ 100 .
Grey color area has no point resides.
On the left panel of Fig. 2(a), we observe good consistency between the trained model ˆ f y ( x ) and f y ( x ) , when y ∈ [0 , 1] .
Note that we only use samples with y i ∈ [0 , 1] for training, and thus the model generalizability to y > 1 is not guaranteed.
However, as the true diffeomorphism satisfies f y 1 ◦ f y 2 = f
y 1 + y 2 , we iterate the trained model recurrently and expect the final output to achieve the effect for y > 1 .
The results on the right panel of Fig. 2(a) justify that the cumulative error from composition does not grow fast, and our model can approximate complex diffeomorphisms with sufficient layers.
In Fig. 3(a) and Fig.
3(b) we plot the derivatives of Para-CFlow , compared to the true one.
The non-differentiability of the groundtruth on the unit circle results in irregular behaviors of trained model near the unit circle; elsewhere it shows good consistency.
To further verify our theorem stated in Section 3, specifically the effect of padding zero(s), we test the same model under different hidden dimensions d hid and number of stacking layers in Fig. 3(c).
[Figure]
Here hidden dimension equals to the input dimension (2 in this experiment) plus the number of padding zero(s).
As we expected, according to our theorems, with padding zero(s) Para-CFlow enjoys universality.
We also investigate the expressiveness when hidden dimensions = 2.
When we increase the number of stacking layers, MSE also decrease, which justifies the L p universality result in [42].
However, increasing the hidden dimension is much more efficient in terms of improving expressiveness than increasing the number of stacking layers.
Such results support our theory of zero-padding.
More results of different architecture settings are available in Appx.
D.1.
Next we design a more challenging task to compare Para-CFlow against a simple MLP with a hidden-layer-setting as (128 , 64 , 32) , and Residual Nets ( Resnet ) [21], that concatenates y and x as the network’s input and then appends x to the output of each layer forward for prediction.
Unlike the previous experiment where a scalar y is used, we generate 3000 samples of a 100-dimensional parameter vector y i
=
( y (1) i , · · · , y (100) i ) where y ( j ) i ∼ N ( y i , 0 . 16) and y i ∼ U [0 , 1] .
Then ( x i , y i ) is fed to each model to learn the target f avg ( y i ) ( x i ) .
Fig. 2(b) shows that Para-CFlow learn a much more continuous mapping than MLP and Resnet .
While Resnet learns relatively better the outer part than MLP , similar to MLP it suffers from bad mapping in the inner part, i.e. the grey area inside the unit circle is much larger than Para-Flow , which represents bad continuity of the learnt mapping.
5.2 Application to contextual BO
In this section,we ensemble Para-CFlow s as a novel surrogate model for BO with three well-known benchmark functions: Rastrigin , Ackley and Trid [41].
Note that the original benchmark tasks are contextless , here we simply set the first d c dimensions as the context vector sampled from a multivariate uniform distribution and leave the last dimension as the action to optimize.
To construct context-dominating tasks we set d c ≫ 1 .
In all benchmarks above, the contexts c are uniformly sampled at random from [ − 3 , 3] d c sequentially while the action a is optimized by gradient descent in [ − 3 , 3] over surrogate models.
The used ensemble-like surrogate models for contextual BO are listed as follows: (1) Para-CFlow : Ensemble of Para-CFlow s for prediction with uncertainty; (2)
MLP : Ensemble of MLP s for uncertainty modelling by concatenating context c and a to obtain input ( c , a ) ∈ R d c +1 ; (3) MLP-Ascend : Similar to MLP , the difference is that it firstly raises both c and a to the same hidden dimension max(5 , d c ) and then concatenates them as inputs to a MLP ; (4) Resnet :
Ensemble of multiple Resnet that concatenates c and a as the network’s input
and then appends a to the output of each layer forward for prediction.
More experimental settings are available in Appx.
D.
Firstly, we investigate the action sensitivity of each model trained with 100 random samples.
The dimensionality of context is set to 5, 10, 20, which is significantly greater than that of action.
The action sensitivity is calculated as the ratio of Permutation Feature Importance ( PFI ) [8] of the action w.r.t.
that of the context, i.e. PFI ( a ) / max i ( PFI ( c i )) and reported in Tab.1.
We observe that
Para-CFlow and Resnet have much higher action sensitivities thanks to the model architecture, while MLP and MLP-Ascend are dominated by contexts as expected.
We further investigate whether different surrogates and their gradients manage to guide the real objective function optimization.
Denote each trajectory as { ( b t , ˆ b t , a t )
Tt =1 | c } , where ( a t )
Tt =1 is a T -step gradient trajectory over the
[Figure]
surrogate ˆ b ( c , a ) and b ( c , a ) is the corresponding real objective value .
We visualize the quiver plot of (ˆ b t )
Tt =1 ( x -axis) v.s. ( b t )
Tt =
( y -axis) along each gradient trajectory for all compared surrogates in Fig.4(a).
We can easily observe that Para-CFlow outperforms other surrogate models as the red arrows move towards the bottom left corner which exactly means the gradient trajectory on Para-CFlow indeed minimizes the real objective function, and thus the result supports our main results of C k approximability, whereas the compared trajectories move towards the top left corner after a few steps probably because those surrogates suffer from more severe local oscillation w.r.t.
a , comparing to Para-CFlow , resulting in unexpected gradient explosion.
We then compare different surrogate models’ performances on BO.
The mean and standard deviation of the cumulative regrets for different surrogate models are reported in Fig.
4(b) where we observe improvement using
Para-CFlow as the neural surrogates, especially for the Rastrigin benchmark.
[Table]
Conclusion
[-=
-]
[-Note that there exists an isotopy h t from I to f such that h 
-]
[-H ( t, x ) for all t ∈ [0 , 1] , h 
-]
In this paper, we firstly prove the equivalence between the C k -universality over compactly-supported diffeomorphisms and that over single-coordinate transforms, resulting in the C k universal approximation ability of affine coupling flows.
Furthermore, we generalize the main theorems to parametric cases and propose a practical model called Para-CFlow s which could serve as a good neural surrogate for gradient-based search and optimization.
With good capabilities both in universal approximation and robust sensing of critical features in parametric diffeomorphisms, we empirically exhibit the advantages of Para-CFlow using various benchmarks.
However, due to the dimension augmentation (for expressiveness purpose), Para-CFlow is not able to benefit from fast log-determinant estimation if it is applied to generative model tasks which will be studied in our future work.