Poisson Flow Generative Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a new “Poisson ﬂow” generative model (PFGM) that maps a uniformdistribution on a high-dimensional hemisphere into any data distribution.
Weinterpret the data points as electrical charges on the z = 0 hyperplane in a spaceaugmented with an additional dimension z , generating a high-dimensional electricﬁeld (the gradient of the solution to Poisson equation).
We prove that if thesecharges ﬂow upward along electric ﬁeld lines, their initial distribution in the z = 0plane transforms into a distribution on the hemisphere of radius r that becomesuniform in the r ! 1 limit.
To learn the bijective transformation, we estimate thenormalized ﬁeld in the augmented space.
For sampling, we devise a backward ODEthat is anchored by the physically meaningful additional dimension: the sampleshit the (unaugmented) data manifold when the z reaches zero. Experimentally,
PFGM achieves a record-breaking Inception score of 9.68 and a FID score of 2.48on CIFAR-10 among the normalizing ﬂow models. It also performs on par with thestate-of-the-art SDE approaches while offering 10 ⇥ to 20 ⇥ acceleration on imagegeneration tasks.
Additionally, PFGM appears more tolerant of estimation errorson a weaker network architecture and robust to the step size in the Euler method.
Introduction
Deep generative models are a prominent approach for data generation and have been used to producehigh quality samples in image [1], text [2] and audio [33], as well as improve semi-supervisedlearning [18], domain generalization [23] and imitation learning [14].
However, current deepgenerative models also have limitations, such as unstable training objectives (GANs [1, 11, 16]) andlow sample quality (VAEs [19], normalizing ﬂows [5]).
New techniques [11, 22] are introducedto stablize the training of CNN-based or ViT-based GAN models.
Although recent advances ondiffusion [15] and scored-based models [31] achieve comparable sample quality to GAN’s withoutadversarial training, these models have a slow stochastic sampling process. [31] proposes backward ODE samplers (normalizing ﬂow) that speed up the sampling process but these methods have not yetperformed on par with the SDE counterparts.
We present a new “Poisson ﬂow” generative model ( PFGM ), exploiting a remarkable physics factthat generalizes to N dimensions.
As illustrated in Fig. 1(a), motion in a viscous ﬂuid transformsany planar charge distribution into a uniform angular distribution.
Speciﬁcally, we interpret Ndimensional data points x (images, say) as positive electric charges in the z = 0 plane of an
N + 1 -dimensional space
(see Fig. 1(a)) ﬁlled with a viscous liquid (say honey).
A positive chargewith z > 0 will be repelled by the other charges and move in the direction of their repulsive force,eventually crossing an imaginary hemisphere of radius r .
We show that, remarkably, if the the originalcharge distribution is let loose just above z = 0 , this law of motion will cause a uniform distributionfor their hemisphere crossings in the r ! 1 limit.
Our Poisson ﬂow generative process reverses the forward process: we generate a uniform distributionof negative charges on the hemisphere, then track their motion back to the z = 0 plane, where theywill be distributed as the data distribution.
A Poisson ﬂow can be viewed as a type of continuousnormalizing ﬂows [4, 9, 31] in the sense that it continuously maps between an arbitrary distributionand an easily sampled one: in the previous works an N -dimensional Gaussian and in PFGM auniform distribution on an N -dimensional hemisphere.
In practice, we implement the Poisson ﬂow
[Figure]
by solving a pair of forward/backward ordinary differential equations (ODEs) induced by the electricﬁeld (Fig. 1(b)) given by the N -dimensional version of Coulombs law (the gradient of the solution tothe Poisson’s equation with the data as sources).
We will interchangeably refer to this gradient as the Poisson ﬁeld , since electric ﬁelds normally refer to the special case N = 3 .
The proposed generative model PFGM has a stable training objective and empirically outperformspreviously state-of-the-art continuous ﬂow methods [28, 31].
As a different iterative method, PFGMoffers two advantages compared to score-based methods [30, 31].
First, the ODE process of PFGMachieves faster sampling speeds than the SDE samplers in [31]. while retaining comparable performance.
Second, our backward ODE exhibits better generation performance than the reverse-time ODEs of VE/VP/sub-VP SDEs [31], as well as greater stability on a weaker architecture NSCNv2 [30].
The rationale for robustness is that the time variables in these ODE baselines are strongly correlatedwith the sample norms during training time, resulting in a less error-tolerant inference.
In contrast,the tie between the anchored variable and the sample norm in PFGM is much weaker.
Experimentally, we show that PFGM achieves state-of-the-art performance on CIFAR-10 datasetin the normalizing ﬂow family, with FID/Inception scores of 2 . 54 / 9 . 62 (w/ DDPM++ [31]) and2 . 48 / 9 . 68 (w/ DDPM++ deep [31]).
It performs competitively with state-of-the-art SDE samplers [31] and provides 10 ⇥ to 20 ⇥ speed up across datasets.
Notably, the backward ODE in PFGM 59 is the only ODE-based sampler that can produce decent samples on its own on NCSNv2 [30], whileother ODE baselines fail without corrections.
In addition, PFGM demonstrates the robustness to thestep size in the Euler method, with a varying number of function evaluations (NFE) ranging from 10to 100 .
We further showcase the utility of the invertible forward/backward ODEs of the Poisson ﬁeldon likelihood evaluation and image manipulations.
Background and Related works
Poisson equation Let x 2 R N and ⇢ ( x ) : R N !
R be a source function.
{+We assume that the sourcefunction has a compact support, ⇢ 2 C 
+}
and N  3 .
The Poisson equation is
[Equation]
[Equation]
where ' ( x ) : R N !
R is called the potential function , and r 2 ⌘
P
Ni =1 @ 2 @ x 2 i is the Laplacianoperator.
It is usually helpful to deﬁne the gradient ﬁeld E ( x ) =
r ' ( x ) and rewrite the Poissonequation as r · E = ⇢ , known in physics as Gauss’s law [10].
The Poisson equation is widely used 70 in physics, giving rise to Newton’s gravitational theory [8] and the electrostatic theory [10], when⇢ ( x ) is interpreted as mass density or electric charge density, respectively.
E is the N -dimensionalanalog of the electric ﬁeld.
The Poisson equation Eq.
(1) (with zero boundary condition at inﬁnity)admits a unique simple integral solution 1 :
[Equation]
1 Eq.
(2) is valid for N  3 . When N = 2 , the Green’s function is G ( x , y ) =  log( || x  y || ) / 2 ⇡ . We where S N  1 (1) is a geometric constant representing the surface area of the unit ( N  1) -sphere 2 , 75 and G ( x , y ) is the extension of Green’s function in N -dimensional space (details in Appendix C).
The negative gradient ﬁeld of ' ( x ) , referred as Poisson ﬁeld of the source ⇢ , is
[Equation]
Qualitatively, the Poisson ﬁeld E ( x ) points away from sources, or equivalently  E ( x ) points towardssources, as illustrated in Fig.
It is straightforward to check that
when ⇢ ( x ) !
( x  y ) , we get 79 ' ( x ) !
G ( x , y ) and E ( x ) !
r x G ( x , y ) .
This implies that G ( x , y ) and r x G ( x , y ) can be 80 interpreted as the potential function and the gradient ﬁeld generated by a unit point source, e.g . , a pointcharge, located at y .
When ⇢ ( x ) takes general forms but has bounded support, simple asymptoticsexist for ||
x ||
||
y || .
To the lowest order, E ( x ) =
r x G ( x , y ) | y
=
⇠ x / || x
|| N behaves as if it 83 were generated by a unit point source at y = 0 .
In physics, the power law decay is considered to belong-range (compared to exponential decay) [10].
Particle dynamics in a Poisson ﬁeld The Poisson ﬁeld immediately deﬁnes a ﬂow model, where theprobability distribution evolves according to the gradient ﬂow @
p t ( x ) / @
t = r · ( p t ( x ) E ( x )) . The 87 gradient ﬂow is a special case of the Fokker-Planck equation [26], where the diffusion coefﬁcient iszero. Intuitively we can think of p t ( x ) as represented by a population of particles. The corresponding(non-diffusion) case of the It
ˆ o process is the forward ODE d x dt = E ( x ) .
We can interpret thetrajectories of the ODE as particles moving according to the Poisson ﬁeld E ( x ) , with initial statesdrawn from p 0 . The physical picture of the forward ODE is a charged particle under the inﬂuence ofelectric ﬁelds in the overdamped limit (details in Appendix J).
The dynamics is also rescalable in the sense that the particle trajectory remains the same for d x dt =± f ( x ) E ( x ) for f ( x ) > 0 , f ( x ) 2 C 1 , because the time rescaling dt !
f ( x ( t )) dt recovers d x dt
=± E ( x ) .
Note that the dynamics is stiff due to the power law factor in the denominator in Eq.
(3),posing computational challenges.
Luckily the rescalablility allows us to rescale E ( x ) properly to getnew ODEs (formally deﬁned later in Section 3.3) that are more amenable for sampling.
Generative Modeling via ODE Generative modeling can be done by transforming a base distributionto a data distribution via mappings deﬁned by ODEs.
The ODE-based samplers allow for adaptivesampling, exact likelihood evaluation and modeling of continuous-time dynamics [4, 31].
Previousworks broadly fall into two lines.
[4, 3] introduce a continuous-time normalizing ﬂow model thatcan be trained with maximum likelihood by the instantaneous change-of-variables formula [4].
Forsampling, they directly integrate the learned invertiable mapping over time.
Another work [31] uniﬁesthe scored-based model [29, 30] and diffusion model [15] into a general diffusion process, and usesthe reverse-time ODE of the diffusion process for sampling.
They show that the reverse-time ODEproduces high quality samples with improved architecture.
Poisson Flow Generative Models
In this section, we start with the properties of the Poisson ﬂow in the augmented space and showhow to draw samples from the data distribution by following the backward ODE of the Poissonﬂow (Section 3.1).
We then discuss how to actually learn a normalized Poisson ﬁeld from datasamples through simulations of the forward ODE (Section 3.2) and present an equivalent backward
ODE that allows for exponentially decay on z (Section 3.3).
3.1 Augment the data with additional dimension
We wish to generate samples x 2 R N from a distribution p ( x ) supported on a bounded region.
Wemay set the source ⇢ ( x ) = p ( x ) 2 C 0 3 and compute the resulting gradient ﬁeld E ( x ) from Eq.
Since  E ( x ) points towards sources, the backward ODE d x /dt =
E ( x ) will take samples close tothe sources.
One may naively hope that the backward ODE is a generative model that recovers p ( x ) .
Unfortunately, the backward ODE has the problem of mode collapse.
We illustrate this phenomenonwith a 2D uniform disk.
The reverse Poisson ﬁeld  E ( x ) on the 2D ( x, y ) -plane points towards thecenter of the disk O (Fig. 2(a) left), so all particle trajectories (blue lines) will eventually hit O . If we
No
2D)
Augmentation (3D) (a) d  in = p ( x ) dA / d  d  out = d  / S N (1)
Gauss  s Law
N dim
O r   d  in d  out =
z
S 5 S S S 1 S(b) instead add an additional dimension z (Fig. 2(a) right), particles can hit different points on the diskand faithfully recover the data distribution.
Consequently, instead of solving the Poisson equation r 2 ' ( x )
=
p ( x ) in the original data space,we solve the Poisson equation in an augmented space ˜ x
= ( x , z ) 2 R N +1 with an additional variablez 2 R .
We augment the training data ˜ x in the new space by setting z = 0 such that ˜ x
= ( x , 0) .
As aconsequence, the data distribution in the augmented space is ˜ p (˜ x )
=
p ( x )  ( z ) , where  is the Diracdelta function.
By Eq.
(3), the Poisson ﬁeld by solving the new Poisson equation r 2 ' (˜ x ) =
˜ p
(˜ x )has an analytical form:
[Equation]
The associated forward/backward ODEs of the Poisson ﬁeld are d ˜ x
/dt
=
E (˜ x ) , d ˜ x /dt
=
E (˜ x ) .
130 Intuitively, theses ODEs uniquely deﬁne trajectories of particles between the z = 0 hyperplaneand an enclosing hemisphere (cf.
Fig.
1(a)).
In the following theorem, we show that the backward
ODE deﬁnes a transformation between the uniform distribution on an inﬁnite hemisphere and thedata distribution ˜ p (˜ x ) in the z = 0 plane.
We present the formal proof to Appendix A, illustratedby Fig. 2(b).
The proof is based on the idea that when the radius of hemisphere r ! 1 , the data 135 distribution ˜ p (˜ x ) can be effectively viewed as a delta distribution at origin.
Consequently, the Poissonﬁeld points in the radial direction at r ! 1 , perpendicular to S + N ( r ) (Green arrows in Fig. 2(b)).
Theorem 1.
Suppose particles are sampled from a uniform distribution on the upper ( z > 0 ) half ofthe sphere of radius r and evolved by the backward ODE d ˜x dt =  E ( ˜x ) until they reach the z = 0hyperplane, where the Poisson ﬁeld E ( ˜x ) is generated by the source ˜ p (˜ x ) .
In the r ! 1 limit, under 140 some mild conditions detailed in Appendix A, this process generates a particle distribution ˜ p (˜ x ) , i.e.,a distribution p ( x ) in the z = 0 hyperplane.
Proof sketch.
Suppose the ﬂux of the backward ODE connects a solid angle d ⌦ (on S + N ( r ) ) withan area dA (on supp(˜ p (˜ x ) ).
According to Gauss’s law, the outﬂow ﬂux d  out = d ⌦ /S N (1) on thehemisphere (Green arrows in Fig. 2(b)) equals the inﬂow ﬂux d  in = p ( x ) dA/ 2 on supp(˜ p (˜ x ))
(Redarrows in Fig. 2(b)).
d  in = d  out gives d ⌦ /dA
= p ( x ) S N (1) / 2 / p ( x ) .
Together, by change-ofvariable, we conclude that the ﬁnal distribution in the z = 0 hyperplane is p ( x ) .
The theorem states that starting from an inﬁnite hemisphere, one can recover the data distribution ˜ pby following the inverse Poisson ﬁeld  E ( ˜x ) .
We defer the formal proof and technical assumptionsof the theorem to Appendix A.
The property allows generative modeling by following the Poissonﬂow of r 2 ' (˜ x ) =
˜ p
(˜ x ) .
3.2 Learning the normalized Poisson Field
Given a set of training data D = { x i } ni =1 i.i.d sampled from the data distribution p ( x ) , we deﬁne the empirical version of the Poisson ﬁeld (Eq.
(4)) as follows:
[Equation]
where the gradient ﬁeld is calculated on n augmented datapoints { ˜ x i
=
( x i , 0) } ni =1 , and c (˜ x ) =1 /
P ni =  || ˜ x  ˜ x
i || N +1 is the multiplier for numerical stability.
We further normalize the ﬁeld toresolve the variations in the magnitude of the norm k ˆ E (˜ x ) k 2 , and ﬁt the neural network to themore amenable normalized ﬁeld v (˜ x ) =
p N ˆ E (˜ x ) /
k ˆ E (˜ x ) k 2 .
The Poisson ﬁeld is rescalable (cf.
Section 2) and thus trajectories of its forward/backward ODEs are invariant under normalization.
Wedenote the empirical ﬁeld calculated on batch data B by ˆ E B and the normalized ﬁeld as v B (˜ x ) =
p N ˆ E B (˜ x ) / k ˆ E B (˜ x ) k 2 .
Similar to the scored-based models, we sample points inside the hemisphere by perturbing theaugmented training data.
Given a training point x 2 D , we add noise to its augmented version{ ˜ [-u 
-]x i = ( x i , 0) } ni =1 to construct the perturbed point ( y , z ) :
[Equation]
(5) where ✏ = ( ✏ x , ✏ z ) ⇠ N (0 ,  2 I N +1 ⇥ N +1 ) , u ⇠ U ( S N (1)) and m ⇠ U [0 , M ] .
The upper limit M , 163 standard deviation  and ⌧ are hyper-parameters.
With ﬁxed ✏ and u , the added noise increasesexponentially with m .
The rationale behind the design is that points farther away from the datasupport play a less important role in generative modeling, sharing a similar spirit with the choice ofnoisy scales in score-based models [30, 31].
In practice, we sample the points by perturbing a mini-batch data B = { x i } |B| i =1 in each iteration.
We 168 uniformly sample the power m in [0 , M ] for each datapoint.
We select a large M (typically around300 ) to ensure the perturbed points can reach a large enough hemisphere.
We use a larger batch B L for the estimation of normalized ﬁeld since the empirical normalized ﬁeld is biased, , whichempirically gives better results.
Denoting the set of perturbed points as { ˜ y i } |B| i =1 , we train the neural 172 network f ✓ on these points to estimate the normalized ﬁeld by minimizing the following loss:
[Equation]
We summarize the training process in Algorithm 1.
In practice, we add a small constant  to thedenominator of the normalized ﬁeld to overcome the numerical issue
when 9 i, || ˜ x  ˜ x i || ⇡ 0 .
Algorithm
Input: Training iteration T , Initial model f ✓ , dataset D , constant  , learning rate ⌘ . for t = 1 . . .
T do
Sample a large batch B L from D and subsample a batch of datapoints B = { x i } |B| i =1 from B L
Simulate the ODE: { ˜ y i
= perturb ( x i ) } |B| i =
Calculate the normalized ﬁeld by B L : v B L (˜ y i ) =
p N ˆ E B L (˜ y i ) /
( k ˆ E B L (˜ y i )
k 2 +  ) ,
i
Calculate the loss: L ( ✓ ) = 1 |B|
P |B| i =
k f ✓
(˜ y i )  v B L (˜ y i ) k
Update the model parameter: ✓ = ✓  ⌘ r L ( ✓ ) end for return f ✓
Algorithm 2 perturb ( x )
Sample the power m ⇠ U [0 , M ] Sample the initial noise ( ✏ x , ✏ z ) ⇠ N (0 ,  2 I ( N +1) ⇥ ( N +1) )
Uniformly sample the vector from the unit ball u ⇠ U ( S N (1))
Construct training
[Equation]
return ˜ y = ( y , z ) 3.3 Backward ODE anchored by the additional dimension
After estimating the normalized ﬁeld v , we can sample from the data distribution by the backward
ODE d ˜ x =  v ( ˜ x ) dt .
Nevertheless, the boundary condition of the above ODE is unclear: the startingand terminal time t of the ODE are both unknown.
To remedy the issue, we propose an equivalent backward ODE in which x evolves with the augmented variable z :
[Equation]
where v (˜ x ) x , v (˜ x ) z are the corresponding components of x , z in vector v (˜ x ) .
In the new ODE, wereplace the time variable t with the physically meaningful variable z , permitting explicit startingand terminal conditions: when z = 0 , we arrive at the data distribution and we can freely choosea large z max as the starting point in the backward ODE.
The backward ODE is compatible withgeneral-purpose ODE solvers, e.g . , RK45 method [21] and forward Euler method.
The popularblack-box ODE solvers, such as the one in Scipy library [35], typically use a common starting timefor the same batch of samples.
Since the distribution on the z =
z max hyperplane is no longer uniform,we derive the prior distribution by radially projecting uniform distribution on the hemisphere withradius r = z max to the z
= z max hyperplane:
[Equation]
where S N ( r ) is the surface area of N -sphere with radius r .
The reason behind the radial projection isthat the Poisson ﬁeld points in the radial direction at r ! 1 .
The new backward ODE also deﬁnes 191 a bijective transformation between p prior ( x ) on the inﬁnite hyperplane ( z max ! 1 ) and the data 192 distribution ˜ p (˜ x ) , analogous to Theorem 1.
In order to sample from p prior ( x ) , it is sufﬁce to samplethe norm from the distribution: p prior ( k x k 2 ) / k x k N  1/ ( k x k 22 + z 2 max ) N +12 and then uniformlysample its angle.
Detailed derivations are deferred to Appendix D. We further achieve exponentiallydecay on the z dimension by introducing a new variable t 0 : [Backward ODE
[Equation]
The z component in the backward ODE, i.e . , dz =  zdt 0 , can be solved by z = e  t 0 .
Since zreaches zero as t 0 !
1 , we instead choose a tiny positive number z min as the terminal condition.
The corresponding starting/terminal time of the variable t 0 are  log z max /  log z min respectively.
199 Empirically, this simple change of variable leads to⇥ faster sampling with almost no harm to the 200 sample quality.
We defer more details to Appendix E.2.
Generative Modeling via the Backward ODE
In this section, we demonstrate the effectiveness of the backward ODE associated with PFGM onimage generation tasks.
In Section 4.1, we show that PFGM achieves best in class performance in thenormalizing ﬂow family. In comparison to the existing state-of-the-art SDE or MCMC approaches, PFGM exhibits 10 ⇥ or 20 ⇥ acceleration while maintaining competitive or higher generation quality.
Meanwhile, unlike existing ODE baselines that heavily rely on corrector to generate decent sampleson weaker architectures, PFGM exhibits greater stability against error (Section 4.2).
Finally, we showthat PFGM is robust to the step size in the Euler method (Section 4.3), and its associated ODE allowsfor likelihood evaluation and image manipulation by editing the latent space (Section 4.4).
4.1 Efﬁcient image generation by PFGM
Setup For image generation tasks, we consider the CIFAR-10 [20], CelebA 64 ⇥ 64 [36] and
LSUN bedroom 256 ⇥ 256 [37]. Following [30], we ﬁrst center-crop the CelebA images and thenresize them to 64 ⇥ 64 .
We choose M = 291 ( CIFAR-10 and CelebA ) / 356 ( LSUN bedroom ) , 214  = 0 . 01 and ⌧ = 0 .
03 for the perturbation Algorithm 2, and z min = 1 e  3 , z max = 215 40 ( CIFAR-10 and CelebA ) / 50 ( LSUN bedroom ) for the backward ODE.
We adopt the DDPM++and DDPM++ deep architectures [31] as our backbones.
We add the scalar z (resp.
predicted directionon z ) as input (resp.
output) to accommodate the additional dimension.
We take the same set ofhyper-parameters, such as batch size, learning rate and training iterations from [31].
We providemore training details in Appendix E.1.
Baselines We compare PFGM to modern autoregressive model [34], GAN [16, 22], normalizingﬂow [17] and EBM [7].
We also compare with variants of score-based models such as DDIM [28] andstate-of-the-art SDE/ODE methods [31].
We denote the methods that use forward-time SDEs in [31]such as Variance Exploding ( VE ) SDE/Variance Preserving ( VP ) SDE/ sub-Variance Preserving ( subVP ), and the corresponding backward SDE/ODE, as A-B , where A 2 { VE, VP, sub-VP } and B 2{ SDE, ODE } . We follow the model selection protocol in [31], which selects the checkpoint with the 226 smallest FID score over the course of training every 50k iterations. [Table]
Numerical Solvers
The backward ODE (Eq.
(6)) is compatible with any general purpose ODEsolver.
In our experiments, the default solver of ODEs is the black box solver in the Scipy library [35]with the RK45 [6] method ( RK45 ), unless otherwise speciﬁed.
For VE/VP/subVP-SDEs, weuse the predictor-corrector ( PC ) sampler introduced in [31].
For VP/sub-VP-SDEs, we apply thepredictor-only sampler, because its performance is on par with the PC sampler while requiring halfcomputation.
Results For quantitative evaluation on CIFAR-10, we report the Inception [27] (higher is better)and FID [12] scores (lower is better) in Table 1.
We also include our preliminary experimental resultson a weaker architecture NCSNv2 [30] in Appendix H.1.
We measure the inference speed by theaverage NFE (number of function evaluation).
We also explicitly indicate which methods belong tothe invertible ﬂow family.
Our main ﬁndings are: (1) PFGM achieves the best Inception scores and FID scores among thenormalizing ﬂow models. Speciﬁcally, PFGM obtains an Inception score of 9 . 68 and a FID score of 240 2 . 48 using the DDPM++ deep architecture.
To our best knowledge, these are the highest FID and
Inception scores by ﬂow models on CIFAR-10.
(2) PFGM achieves a 10 ⇥ ⇠ 20 ⇥ faster inference 242 speed than the SDE methods using the similar architectures, while retaining comparable samplequality.
As shown in Table 1, PFGM requires NFEs of 110 whereas the SDE methods typically use 244⇠ 2000 inference steps. PFGM outperforms all the baselines on DDPM++ in all metrics.
In 245 addition, PFGM generally samplers faster than other ODE baselines with the same RK45 solver. The backward ODE in PFGM is compatible with architectures with varying capacities.
PFGM 247 consistently outperforms other ODE baselines on DDPM++ (Table 1) or NCSNv2 (Appendix H.1)backbones.
In Fig.
3, we visualize the uncurated samples from PFGM on CIFAR-10, CelebA 64 ⇥ 64 and LSUNbedroom 256 ⇥ 256 .
We provides more samples in Appendix I. 4.2 Failure of VE/VP-ODEs on NCSNv2 architecture
[Figure] In our preliminary experiments on NCSNv2 architectures,we empirically observe that the VE/VP-ODEs have FIDscores greater than 90 on CIFAR-10. In particular, VE/VPODEs can only generate decent samples when applyingthe Langevin dynamics corrector, and even then, their performances are still inferior to PFGM (Table 8, Table 9).
The poor performance on NCSNv2 stands in striking contrast to their high sample quality on NCSN++/DDPM++in [31]. It indicates that the VE/VP-ODEs are moresusceptible to estimation errors than PFGM.
We hy- 262 pothesize that the strong norm-  correlation seen duringthe training of score-based models causes the problem. For score-based models, the l 2 norms of perturbed training samples and the standard deviations  ( t )of Gaussian noises have strong correlation, e.g . , l 2 norm ⇡  ( t ) p N for large  ( t ) in VE [31]. Incontrast, as shown in Fig.
4, PFGM allocates high mass across a wide spectrum of the training samplenorms.
During sampling, VE/VP-ODEs could break down when the trajectories of backward ODEsdeviate from the norm-  ( t ) relation to which most training samples pertain.
The weaker NCSNv2backbone incurs larger errors and thus leads to their failure.
The PFGM is more resistant to estimateerrors because of the greater range of training sample norms.
To further verify the hypothesis above, we split a batch of VE-ODE samples into cleaner and noisiersamples according to visual quality (Fig. 8(a)).
In Fig. 5(a), we investigate the relation for cleaner andnoisier samples during the forward Euler simulation of VE-ODE when  ( t ) < 15 .
We can see thatthe trajectory of cleaner samples stays close to the norm-  ( t ) relation (the red dash line), whereasthat of the noisier samples diverges from the relation.
The Langevin dynamics corrector changesthe trajectory of noisier samples to align with the relation.
Fig.
5(b) further shows that the anchoredvariable z ( t 0 ) and the norms in the backward ODE of PFGM are not strongly correlated, giving riseto the robustness against the imprecise estimation on NCSNv2.
We defer more details to Appendix G.
In order to accelerate the inference speed of ODEs, we can increase the step size (decrease the NFEs)in numerical solvers such as the forward Euler method. It also enables the trade-off between samplequality and computational efﬁciency in real-world deployment.
We study the effects of increasingstep size on PFGM, VP-ODE and DDIM [28] using the forward Euler method, with a varying NFEranging from 10 to 100 . In Fig. 5(c), we report the sample quality measured by FID scores on CIFAR-10. As expected, all themethods have higher FID scores when decreasing the NFE. We observe that the sample quality of
PFGM degrades gracefully as we decrease the NFE. Our method shows signiﬁcantly better robustnessto step sizes than the VP-ODE, especially when only taking a few Euler steps. In addition, PFGMobtains better FID scores than DDIM on most NFEs except for 10 where PFGM is marginally worse. This suggests that the PFGM is a promising method for accommodating instantaneous resourceavailability, as high-quality samples can be generated in limited steps.
4.4 Utilities of ODE: likelihood evaluation and latent representation
Similar to the family of discrete normalizing ﬂows [5, 17, 13] and continuous probability ﬂow [31],the forward ODE in PFGM deﬁnes an invertible mapping between the data space and latent spacewith a known prior.
Formally, we deﬁne the invertible forward M mapping by integrating the 296 corresponding forward ODE d ( x , z ) =
( v ( ˜ x ) x v ( ˜ x )  1 z z, z ) dt 0 of Eq.
[Equation]
where log z min / log z max are the starting/terminal time in the forward ODE.
The forward mappingtransfers the data distribution to the prior distribution p prior on the z =
z max hyperplane (cf.
Section 3.3): p prior ( x (log z max ))
=
M ( p ( x (log z min ))) .
The invertibility enables likelihood evaluationand creates a meaningful latent space on the z = z max hyperplane.
In addition, we can adapt to thecomputational constraints by adjusting the step size or the precision in numerical ODE solvers.
[Table] Likelihood evaluation We evaluate the data likelihoodby the instantaneous change-of-variable formula [4, 31].
In Table 2, we report the bits/dim on the uniformly dequantized CIFAR-10 test set and compare with existing baselines that use the same setup.
We observe that PFGM achieves better likelihoods than discrete normalizing ﬂow models, even without maximum likelihood training.
Among the continuous ﬂow models, sub-VP-ODEshows the lowest bits/dim, although its sample quality isworse than VP-ODE and PFGM (Table 1).
The exploration of the seeming trade-off between likelihood andsample quality is left for future works. Latent representation Since the samples are uniquely identiﬁable by their latents via the invertiblemapping M , PFGM further supports image manipulation using its latent representation on the z = 316 z max hyperplane.
We include the results of image interpolation and the temperature scaling [5, 17, 31]to Appendix H.3 and Appendix H.4.
For interpolation, it shows that we can travel along the latentspace to obtain perceptually consistent interpolations between CelebA images.
Conclusion
We present a new deep generative model by solving the Poisson equation whose source term is thedata distribution.
We estimate the normalized gradient ﬁeld of the solution in an augmented spacewith an additional dimension.
For sampling, we devise a backward ODE that exponential decays onthe physically meaningful additional dimension.
Empirically, our approach has best performanceover other normalizing ﬂow baselines, and achieving 10 ⇥ to 20 ⇥ acceleration over the stochastic 325 methods.
Our backward ODE shows greater stability against errors than popular ODE-based methods,and enables efﬁcient adaptive sampling.
We further demonstrate the utilities of the forward ODE onlikelihood evaluation and image interpolation.
Future directions include improving the normalizationof Poisson ﬁelds.
More principled approaches can be used to get around the divergent near-ﬁeldbehavior.
For example, we may exploit renormalization, a useful tool in physics, to make the Poissonﬁeld well-behaved in near ﬁelds.