Poisson Flow Generative Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a new “Poisson ﬂow” generative model that maps a uniform distribution on a high-dimensional hemisphere into any data distribution.
We interpret thedata points as electrical charges on the z = 0 hyperplane in a space augmented withan additional dimension z , generating a high-dimensional electric ﬁeld (the gradientof the solution to Poisson equation).
We prove that if these charges ﬂow upwardalong electric ﬁeld lines, their initial distribution in the z = 0 plane transforms intoa distribution on the hemisphere of radius r that becomes uniform in the r ! 1limit.
To learn the bijective transformation, we estimate the normalized ﬁeld alongthe trajectories of its associated forward ODE.
For sampling, we devise a backward ODE that is anchored by the physically meaningful additional dimension: thesamples hit the (unaugmented) data manifold when the z reaches zero.
Experimentally, the backward ODE of Poisson ﬂow is on par with the state-of-the-artscore-based MCMC and SDE approaches while achieving 10 ⇥ to 20 ⇥ accelerationon image generation tasks. The backward ODE also outperforms popular ODEmethods using the same NCSNv2 architecture [26] and demonstrates superiorstability.
Finally, we showcase the utility of the invertibility of Poisson ﬂow onlikelihood evaluation, meaningful latent representations and adaptive sampling.
Introduction
Deep generative models are a prominent approach for data generation, and have been used to producehigh quality samples in image [1], text [2] and audio [28], as well as improve semi-supervisedlearning [17], domain generalization [21] and imitation learning [14].
However, current deepgenerative models also have limitations, such as unstable training objectives (GANs [1, 11]) and lowsample quality (VAEs [18], normalizing ﬂows [5]).
Although recent advances on diffusion [15] andscored-based models [27] achieve comparable sample quality to GAN’s without adversarial training,these models have a slow stochastic sampling process, or an unstable deterministic reverse-time ODE.
We present a new “Poisson ﬂow” generative model ( PFGM ), exploiting a remarkable physics factthat generalizes to N dimensions.
As illustrated in Fig. 1(a), motion in a viscous ﬂuid transformsany planar charge distribution into a uniform angular distribution.
Speciﬁcally, we interpret Ndimensional data points x (images, say) as positive electric charges in the z = 0 plane of an
N + 1 -dimensional space
(see Fig. 1(a)) ﬁlled with a viscous liquid (say honey).
A positive chargewith z > 0 will be repelled by the other charges and move in the direction of their repulsive force,eventually crossing an imaginary hemisphere of radius r .
We show that, remarkably, if the the originalcharge distribution is let loose just above z = 0 , this law of motion will cause a uniform distributionfor their hemisphere crossings in the r ! 1 limit.
Our Poisson ﬂow generative process reverses the forward process: we generate a uniform distributionof negative charges on the hemisphere, then track their motion back to the z = 0 plane, where theywill be distributed as the training data.
A Poisson ﬂow is similar to a normalizing ﬂow [5, 16] in thesense that it maps between an arbitrary distribution and an easily sampled one: in the former case an N -dimensional normal distribution and in the latter case a uniform distribution on an N -dimensionalhemisphere.
In practice, we implement the Poisson ﬂow by solving a pair of forward/backward
[Figure]
ordinary differential equations (ODEs) induced by the electric ﬁeld (Fig. 1(b)) given by the Ndimensional version of Coulombs law (the gradient of the solution to the Poisson’s equation with thedata as sources).
We will interchangeably refer to this gradient as the Poisson ﬁeld , since electricﬁelds normally refer to the special case N = 3 .
The proposed generative model PFGM has a stable training objective and produces high-qualitysamples on par with the the score-based MCMC or SDE methods [26, 27] on the same architecture.
PFGM has two advantanges compared to score-based methods [26, 27].
First, PFGM’s deterministicbackward ODE sampler achieves faster sampling speeds than the score-based stochastic sampler [26,27].
Second, our backward ODE exhibits greater stability than other ODE baselines, e.g . , thebackward ODEs of VE/VP SDEs [27].
The rationale is that the time variables in these ODE baselinesare strongly correlated with the sample norms because of the Gaussian perturbation kernels duringtraining, resulting in a less error-tolerant inference-time ODE simulation.
In contrast, the tie betweenthe anchored variable and the sample norm in the backward ODE of PFGM is much weaker.
Experimentally, we show that PFGM has competitive performance with state-of-the-art method onthe NCSNv2 architecture on CIFAR-10, and outperforms all the baselines on CelebA 64 ⇥ 64 datasetin terms of FID score [12].
Meanwhile, the backward ODE gives 10 ⇥ to 20 ⇥ speed up comparedto stochastic score-based samplers across datasets.
Notably, the backward ODE in PFGM is theonly ODE-based sampler that can produce decent samples on its own on the NCSNv2 architecture,while other ODE baselines fail without correction.
PFGM still performs better than other ODE-basedsamplers with correctors.
We further showcase the utility of the invertible forward/backward ODE ofthe Poisson ﬁeld on likelihood evaluation, meaningful latent representations, and adaptive sampling.
Background and Related works
Poisson equation Let x 2 R N and ⇢ ( x ) : R N !
R be a source function.
[-We assume that the sourcefunction has a compact support, ⇢ 2 C 
-]
The Poisson equation is
[Equation]
where ' ( x ) : R N !
R is called the potential function , and r 2 ⌘
P
[Equation] 2 @ x 2 i is the Laplacianoperator.
It is usually helpful to deﬁne the gradient ﬁeld E ( x ) =
r ' ( x ) and rewrite the Poissonequation as r · E = ⇢ , known in physics as Gauss’s law [10].
The Poisson equation is widely usedin physics, giving rise to Newton’s gravitational theory [8] and the electrostatic theory [10], when⇢ ( x ) is interpreted as mass density or electric charge density, respectively.
E is the N -dimensionalanalog of the electric ﬁeld.
The Poisson equation Eq.
(1) (with zero boundary condition at inﬁnity)admits a simple integral solution:
[Equation]
where G ( x , y ) is the Green’s function, and S N  1 (1) is a geometric constant representing the surfacearea of the unit ( N  1) -sphere 1 .
The negative gradient ﬁeld of ' ( x ) , referred as Poisson ﬁeld of thesource ⇢ , is
[Equation]
Qualitatively, the Poisson ﬁeld E ( x ) points away from sources, or equivalently  E ( x ) points towardssources, as illustrated in Fig. 1(a).
It is straightforward to check that
when ⇢ ( x ) !
( x  y ) , we get' ( x ) !
G ( x , y ) and E ( x ) !
r x G ( x , y ) .
This implies that G ( x , y ) and r x G ( x , y ) can beinterpreted as the potential function and the gradient ﬁeld generated by a unit point source, e.g . , a pointcharge, located at y .
When ⇢ ( x ) takes general forms but has bounded support, simple asymptoticsexist for ||
x ||
||
y || .
To the lowest order, E ( x ) =
r x G ( x , y ) | y
=
⇠ x / || x
|| N behaves as if it 79 were generated by a unit point source at y = 0 .
In physics, the power law decay is considered to belong-range (compared to exponential decay) [10].
Particle dynamics in a Poisson ﬁeld The Poisson ﬁeld has two associated forward and backward
ODEs.
To obtain physical intuition, we imagine a particle with mass m = 1 and charge q = 1 pushedby an electric ﬁeld E through an an extremely viscous ﬂuid (e.g., honey), causing its velocity (asopposed to its acceleration as would be the case in vacuum) to be proportional to E .
[ q = 1 , forward ODE] [Equation] [Equation]
which trace the electric lines in the forward and backward way, respectively.
The dynamics isalso rescalable in the sense that the particle trajectory remains the same for d x dt = ± f ( x ) E ( x )when f ( x ) > 0 , because the time rescaling dt !
f ( x ( t )) dt recovers d x dt
= ± E ( x ) .
Note that thedynamics deﬁned in Eq. (4) is stiff due to the power law factor in the denominator in Eq.
(3), posingcomputational challenges.
Luckily the rescalablility allows us to rescale E ( x ) properly to get new ODEs (formally deﬁned later in Section 3.3) that are more amenable for sampling.
Generative Modeling via ODE Generative modeling can be done by transforming a base distributionto a data distribution via mappings deﬁned by ODEs.
The ODE-based samplers allow for adaptivesampling, exact likelihood evaluation and modeling of continuous-time dynamics [4, 27].
Previousworks broadly fall into two lines.
[4, 3] introduce a continues-time normalizing ﬂow model thatcan be trained with maximum likelihood by the instantaneous change-of-variables formula [4].
Forsampling, they directly integrate the learned invertiable mapping over time.
Another work [27] uniﬁesthe scored-based model [25, 26] and diffusion model [15] into a general diffusion process, and usesthe reverse-time ODE of the diffusion process for sampling.
They show that the reverse-time ODEproduces high quality samples with improved architecture.
Poisson Flow Generative Models
In this section, we start with the properties of the Poisson ﬂow in the augmented space and showhow to draw samples from the data distribution by following the backward ODE of the Poissonﬂow (Section 3.1).
We then discuss how to actually learn a normalized Poisson ﬁeld from datasamples through simulations of the forward ODE (Section 3.2) and present an equivalent backward
ODE that allows for exponentially decay on z (Section 3.3).
3.1 Augment the data with additional dimension
We wish to generate samples x 2 R N from a distribution p ( x ) supported on a bounded region.
Wemay set the source ⇢ ( x ) = p ( x ) and compute the resulting gradient ﬁeld E ( x ) from Eq.
Since E ( x ) points towards sources, the backward ODE d x /dt =
E ( x ) will take samples close to thesources.
One may naively hope that the backward ODE is a generative model that recovers p ( x ) .
Unfortunately, the backward ODE has the problem of mode collapse.
We illustrate this phenomenonwith a 2D uniform disk.
The reverse Poisson ﬁeld  E ( x ) on the 2D ( x, y ) -plane points towards thecenter of the disk O (Fig. 2(a) left), so all particle trajectories (blue lines) will eventually hit O .
If we instead add an additional dimension z (Fig. 2(a) right), particles can hit different points on the diskand faithfully recover the data distribution.
[Figure]
d  in = p ( x ) dA / d  d  out = d  / S N (1)
Gauss  s Law
N dim
O r   d  in d  out =
z
[Figure]
Consequently, instead of solving the Poisson equation r 2 ' ( x )
=
p ( x ) in the original data space,we solve the Poisson equation in an augmented space ˜ x
= ( x , z ) 2 R N +1 with an additional variablez 2 R .
We augment the training data ˜ x in the new space by setting z = 0 such that ˜ x
= ( x , 0) .
As aconsequence, the data distribution in the augmented space is ˜ p (˜ x )
=
p ( x )  ( z ) , where  is the Diracdelta function.
By Eq.
(3), the Poisson ﬁeld by solving the new Poisson equation r 2 ' (˜ x ) =
˜ p
(˜ x )has an analytical form:
[Equation]
The associated forward/backward ODEs of the Poisson ﬁeld are d ˜ x
/dt
=
E (˜ x ) , d ˜ x /dt
=
E (˜ x ) .
Intuitively, theses ODEs uniquely deﬁne trajectories of particles between the z = 0 hyperplaneand an enclosing hemisphere (cf.
Fig.
1(a)).
In the following theorem, we show that the backward
ODE deﬁnes a transformation between the uniform distribution on an inﬁnite hemisphere and thedata distribution ˜ p (˜ x ) in the z = 0 plane.
We present the formal proof to Appendix A, illustratedby Fig. 2(b).
The proof is based on the idea that when the radius of hemisphere r ! 1 , the datadistribution ˜ p (˜ x ) can be effectively viewed as a delta distribution at origin.
Consequently, the Poissonﬁeld points in the radial direction at r ! 1 , perpendicular to S + N ( r ) (Green arrows in Fig. 2(b)).
Theorem 1.
Suppose particles are sampled from a uniform distribution on the upper ( z > 0 ) half ofthe sphere of radius r and evolved by the backward ODE d ˜x dt =  E ( ˜x ) until they reach the z = 0hyperplane, where the Poisson ﬁeld E ( ˜x ) is generated by the source ˜ p (˜ x ) .
in the r ! 1 limit, thisprocess generates a particle distribution ˜ p (˜ x ) , i.e., a distribution p ( x ) in the z = 0 hyperplane.
Proof sketch.
Suppose the ﬂux of the backward ODE connects a solid angle d ⌦ (on S + N ( r ) ) withan area dA (on supp( p ( x ) ), and the stationary distribution on supp( p ( x )) by q ( x ) .
According to Gauss’s law, the outﬂow ﬂux d  out = d ⌦ /S N (1) on the hemisphere (Green arrows in Fig. 2(b))equals the inﬂow ﬂux d  in = p ( x ) dA/ 2 on supp( p ( x ))
(Red arrows in Fig. 2(b)).
Furthermore,particle conservation gives q ( x ) dA = U ( S + N ( r ))
r N d ⌦ = 2 d ⌦ /S N (1) .
Together, we conclude thatthe ﬁnal distribution in the z = 0 plane is q ( x ) = p ( x ) .
The theorem states that starting from an inﬁnite hemisphere, one can recover the data distribution ˜ pby following the inverse Poisson ﬁeld  E ( ˜x ) .
The property allows generative modeling by followingthe Poisson ﬂow of r 2 ' (˜ x ) =
˜ p
(˜ x ) .
3.2 Learning the normalized Poisson Field
Given a set of training data D = { x i } ni =1 i.i.d sampled from the data distribution p ( x ) , we deﬁne the 145 empirical version of the Poisson ﬁeld (Eq.
as follows:
[Equation]
where the gradient ﬁeld is calculated on n augmented datapoints { ˜ x i
=
( x i , 0) } ni =1 , and c (˜ x ) =
N  1 nS N (1) min j || ˜ x  ˜ x
i || N +1 is the multiplier for numerical stability.
We further normalize the ﬁeldto resolve the variations in the magnitude of the norm k ˆ E (˜ x ) k 2 , and ﬁt the neural network tothe more amenable normalized ﬁeld v (˜ x ) =
ˆ E (˜ x ) /
k ˆ E (˜ x ) k 2 .
The Poisson ﬁeld is rescalable (cf.
Section 2) and thus trajectories of its forward/backward ODEs are invariant under normalization.
Wedenote the empirical ﬁeld calculated on batch data B by ˆ E B and the normalized ﬁeld as v B (˜ x ) =
ˆ E B (˜ x ) / k ˆ E B (˜ x ) k 2 .
We sample training points on the trajectories of the forward ODE d ˜ x /dt = v (˜ x ) , so that the neuralapproximations are tailored to capture the normalized ﬁeld well in areas likely traversed alongbackward ODE trajectories. Given a training point x 2 D , we discretize the forward ODE by Euler 156 method with the step sizes as a geometric sequence. Because the empirical Poisson ﬁelds are zeroon { ˜ x i =
( x i , 0) } ni =1 , we perturb the {+augmented data ˜ u 
 =+} ( x , 0) by a small Gaussian noise toconstruct the initial value ˜ u 1 :
[Equation]
where ✏ = ( ✏ x , ✏ z ) ⇠ N (0 ,  2 I N +1 ⇥ N +1 ) .
The terminal time M , standard deviation  and ⌧ are 160 hyper-parameters.
By the nature of the step size
[Equation]
k 2 ⌧ (1 + ⌧ ) t  1 , one can show that thesequence of arc lengths in the discretization above, i.e . , { P t i =1 k ˜ u i  ˜ u i  1 k 2 } Mt =1 , also form a 162 geometric sequence (Appendix E.1).
The rationale behind the design is that training points fartheraway from the data support play a less important role in generative modeling, sharing a similar spiritwith the choice of noisy scales in score-based models [26, 27].
In practice, we simulate the forward ODE for a mini-batch data B = { x i } |B| i =1 in each iteration.
We 166 uniformly sample the terminal time M in { 0 , 1 , . . . , MAX STEP } for each datapoint.
We select alarge constant MAX STEP as maximum number of steps in the forward ODE (Eq.
(typicallyaround 300 ) to ensure the terminal points can reach a large enough hemisphere.
Since the empiricalnormalized ﬁeld is biased, we use a larger batch B L for the estimation, which empirically gives betterresults.
Denoting the set of terminal points as { ˜ y i } |B| i =1 , we train the neural network f ✓ on the terminal 171 points to estimate the normalized ﬁeld by minimizing the following loss:
[Equation]
We summarize the training process in Algorithm 1.
In practice, we add a small constant  to thedenominator of the normalized ﬁeld to overcome the numerical issue
when 9 i, || ˜ x  ˜ x i || ⇡ 0 .
The 174 calculation of normalized ﬁelds in the forward ODE simulation (Algorithm 2) are based on a batch ofsamples, and can be easily parallelized on deep learning libraries like Pytorch [23].
Input: Training iteration T , Initial model f ✓ , dataset D , constant  , learning rate ⌘ . for t = 1 . . .
T do
Sample a large batch B L from D and subsample a batch of datapoints B = { x i } |B| i =1 from B L
Simulate the ODE: { ˜ y i =
forwardODE ( x i , B L ) } |B| i =
Calculate the normalized ﬁeld by B L : v B L (˜ y i )
= ˆ E B L (˜ y i ) /
( k ˆ E B L (˜ y i )
k {+2 +  ) , 
+}
i
Calculate the loss: L ( ✓ ) = 1 |B|
P {+|B| i =
+}
k f ✓
(˜ y i )  v B L (˜ y i ) k
Update the model parameter: ✓ = ✓  ⌘ r L ( ✓ ) end for return f ✓
3.3 Backward ODE anchored by the additional dimension
After estimating the normalized ﬁeld v , we can sample from the data distribution by the backward
ODE d ˜ x =  v ( ˜ x ) dt .
Nevertheless, the boundary condition of the above ODE is unclear: the startingand terminal time t of the ODE are both unknown.
To remedy the issue, we propose an equivalent Algorithm Sample # of forward step M ⇠ U{ 0 , 1 , . .
. , MAX STEP } Initialize ˜ u
[Equation]
+
[Equation]
for t = 2 . . .
M do ˜
[Equation]
) end for return ˜ u M backward ODE in which x evolves with the augmented variable z :
[Equation]
where v (˜ x ) x , v (˜ x ) z are the corresponding components of x , z in vector v (˜ x ) .
In the new ODE, wereplace the time variable t with the physically meaningful variable z , permitting explicit startingand terminal conditions: when z = 0 , we arrive at the data distribution and we can freely choosea large z max as the starting point in the backward ODE.
The backward ODE is compatible withgeneral-purpose ODE solvers, e.g . , RK45 method [20] and forward Euler method.
The popularblack-box ODE solvers, such as the one in Scipy library [30], typically use a common starting timefor the same batch of samples.
Since the distribution on the z =
z max hyperplane is no longer uniform,we derive the prior distribution by radially projecting uniform distribution on the hemisphere withradius r = z max to the z
= z max hyperplane:
[Equation]
where S N ( r ) is the surface area of N -sphere with radius r .
The reason behind the radical projectionis that the Poisson ﬁeld points in the radial direction at r ! 1 .
The new backward ODE deﬁnesa bijection between p prior ( x ) on the inﬁnite hyperplane ( z max ! 1 ) and the data distribution ˜ p (˜ x ) ,analogous to Theorem 1.
In order to sample from p prior ( x ) , it is sufﬁce to sample the norm fromthe distribution: p prior ( k x k 2 ) / k x k N  2/ ( k x k 22 + z 2 max ) N 2 and then uniformly sample its angle.
Detailed derivations are deferred to Appendix D. We further achieve exponentially decay on the zdimension by introducing a new variable t 0 :
[Equation]
The z component in the backward ODE, i.e . , dz =  zdt 0 , can be solved by z = e  t 0 .
Since zreaches zero as t 0 !
1 , we instead choose a tiny positive number z min as the terminal condition.
The corresponding starting/terminal time of the variable t 0 are  [Equation] z [Equation]
respectively.
200 Empirically, this simple change of variable leads to⇥ faster sampling with almost no harm to the 201 sample quality.
We defer more details to Appendix E.2.
Generative Modeling via the Backward ODE
In this section, we demonstrate the effectiveness of the backward ODE associated with PFGM onimage generation tasks.
We show that PFGM has competitive or even better generation quality thancurrent state-of-the-art methods under the same architecture, and achieving 10 ⇥ ⇠ 20 ⇥ acceleration.
Meanwhile, unlike existing ODE baselines that heavily rely on corrector to generate decent samples,the backward ODE of PFGM exhibits greater stability against error.
In addition, we show that thegeneration process of PFGM inherits the merits of ODE samplers, such as likelihood evaluation,latent representations and adaptive sampling.
4.1 Efﬁcient and stable image generation by Poisson Flow
Setup We consider the CIFAR-10 [19] and CelebA [31] dataset for image generation task.
Following [26], we ﬁrst center-crop the CelebA images and then resize them to 64 ⇥ 64 . We choose
MAX STEP = 300 ,  = 0 . 01 and ⌧ = 0 .
03 for the forward ODE during training, and z min = 2 e  3 ,z max = 40 ( CIFAR-10 ) / 60 ( CelebA 64 ⇥ 64) for the backward ODE.
Due to computational constraints, we use the NCSNv2 architecture in [26] as our backbone, instead of the SOTA architecture
NCSN++ in [27]. The architecture of NCSN++ is larger and takes signiﬁcantly longer training time. [Table] Note that PFGM can be readily paired with the NCSN++ architecture for a better performance.
Weadd the scalar z (resp.
predicted direction on z ) as input (resp.
output) to accommodate the additionaldimension.
We use the same set of hyper-parameters, such as batch size, learning rate and trainingiterations in [26].
We provide more training details in Appendix E.1.
Baselines We compare PFGM to modern autoregressive models [29], GANs [11, 22] and EBMs [7].
We also compare with variants of score-based models, from the annealed Langevin dynamicssampler [25, 26] to state-of-the-art SDE/ODE sampler [27].
For fair comparison, PFGM and scorebased models use the NCSNv2 [26] architecture as the backbone. We denote the methods that useforward-time SDEs such as Variance Exploding ( VE ) SDE and Variance Preserving ( VP ) SDE [27]and the corresponding backward SDE/ODE, as A-B , where A 2 { VE, VP } and B 2 { SDE, ODE } .
Numerical Solvers
The backward ODE (Eq.
(7)) is compatible with any general purpose ODEsolver.
In our experiments, we choose the forward Euler method ( Euler ) with ﬁxed number of steps,and the black box ODE solver in the Scipy library [30] with the RK45 [6] method ( RK45 ).
For VE-SDE and VP-SDE, we use the predictor-corrector ( PC ) sampler introduced in [27].
We explicitlyname the sampler, with forward Euler method as predictor and Langevin dynamics as corrector, as Euler w/ corrector .
[Table] Results For quantitative evaluation on CIFAR-10 and CelebA, we report the Inception [24] (higher is better) and FID [12]scores (lower is better) in Table 1 and Table 2.
We omit the Inception score on CelebA as itis not widely reported. We do not report theresults of VE/VP-ODE without corrector, suchas VE-ODE (RK45), since they have worse-offsample quality and get FIDs greater than 90 on CIFAR-10 and 60 on CelebA.
We measure theinference speed by the average NFE (numberof function evaluation).
Our main ﬁndings are: (1) Table 1 shows that PFGM with RK45 method has competitive FID/Inception scores with the Langevin dynamics, which was the best model on the NCSNv2architecture before, and requires 10 ⇥ less NFE.
In addition, PFGM performs better than all the other
ODE samplers.
(2) As shown in Table 2, PFGM achieves best FID scores than all the baselines on CelebA dataset, while accelerating the inference speed around 20 ⇥ .
Remarkably, PFGM outperformsthe Langevin dynamics and reverse-time SDE samplers, which are usually considered better than theirdeterministic counterparts. (3) Our method is more tolerant of sampling error.
Among the compared
ODEs, our backward ODE (Eq. (7)) is the only one that successfully generates high quality sampleswhile the VE/VP-ODE fail w/o the Langevin dynamics corrector.
The backward ODE still beats the
[Figure]
baselines w/ corrector. In Fig. 3, we visualize the uncurated samples from PFGM for CIFAR-10 and
CelebA 64 ⇥ 64 .
Failure of VE/VP ODEs Although [27] demonstratesthat both VE/VP-ODE have high sample quality on NCSN++, we empirically observe that these backward ODEs perform poorly on the NCSNv2 backbone.
For thescore-based model, the l 2 norm of perturbed training dataand the standard variance of Gaussian perturbation kernel ( t ) have strong correlation, e.g . , l 2 norm ⇡  ( t ) p N withlarge  ( t ) in VE [27]. We hypothesize that the VE/VPODE are more sensitive to estimation error because ofsuch correlation.
VE/VP-ODE can break down when thetrajectories of backward ODEs deviate from the norm-  ( t ) relation.
The smaller NCSNv2 backboneincurs larger errors and thus leads to their failure.
Fig. 4 shows the ideal sample norm distributions with varying time variables of different ODEs. We observe that the PFGM has much weakernorm-
z ( t 0 ) correlation while VE puts most mass around k x k 2 =  ( t ) p N . In our experiments, we observe that many samples in VE-ODE have large noise. To verify thehypothesis above, we split a batch of VE-ODE samples into cleaner/noisier samples by their visualquality.
In Fig. 5(a), we investigate the relation for cleaner/noisier samples during the simulation of VE-ODE (Euler) when  ( t ) < 15 .
We can see that the trajectory of cleaner samples stays close tothe norm-  ( t ) relation, while the noisier samples diverge from the relation.
The Langevin dynamicscorrector changes the trajectory of noisier samples to align with the relation.
We also examine
thenorm- z ( t 0 ) relation during the backward ODE of PFGM.
Fig.
5(b) shows that the anchored variablez ( t 0 ) and the norms in the backward ODE are not strongly correlated, giving rise to the robustnessagainst the imprecise estimation on smaller backbone.
We defer more details to Appendix F.
[Figure]
[Figure]
[Figure]
4.2 Utilities of ODE: likelihood evaluation, latent representation and adapative sampling
Similar to the family of normalizing ﬂows [5, 16, 13] and reverse-time ODE samplers [27], theforward ODE in PFGM deﬁnes an invertible mapping between the data space and latent space with aknown prior.
Formally, we deﬁne the invertible forward M mapping by integrating the correspondingforward ODE d ( x , z ) =
( v (˜ x ) x v (˜ x )  1 z z, z ) dt 0 of Eq.
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
where log z min / log z max are the starting/terminal time in the forward ODE.
The forward mappingtransfers the data distribution to the prior distribution p prior on the z =
z max hyperplane (cf.
Section 3.3): p prior ( x (log z max ))
=
M ( p ( x (log z min ))) .
The invertibility enables likelihood evaluationand creates a meaningful latent space on the z = z max hyperplane.
In addition, we can adapt to thecomputational constraints by adjusting the precision in numerical ODE solvers.
Likelihood evaluation We evaluate the data likelihood by the instantaneous change-of-variableformula [4, 27].
In Table 3, we report the bits/dim on the uniformly dequantized CIFAR-10 testset. We compare with existing baselines that use the same setup.
We observe that PFGM achieves abits/dim on par with SOTA normalizing ﬂow models, even without maximum likelihood training.
Wealso compare with baselines using the backbone in DDPM [15], which is similar to NCSN++ and hasmuch larger capacity than our NCSNv2 backbone. The PFGM still gets competitive likelihood withthe VP-ODE with the DDPM backbone.
Latent representation We demonstrate the utility of the latent representation by conducting imageinterpolation. As shown in Fig. 6, we can travel along the latent space, i.e . , the z = z max hyperplane,to obtain perceptually consistent interpolations between CelebA images.
We defer the temperaturescaling on the latent space [5, 16, 27] to Appendix G.2.
Adaptive sampling We can trade off sample quality for computational efﬁciency by adjusting therelative and absolute tolerances rtol , atol in the RK45 method, or the number of steps in the forward Euler method. As shown in Fig.
7, the sample quality of PFGM degrades gracefully as we reduce thenumber of steps in the forward Euler method. In addition, our method shows better adaptivity thanthe VE-ODE w/ corrector, especially when only taking few Euler steps. In Appendix G.1 we showsimilar results by enlarging the error tolerance in the RK45 method.  Conclusion
We present a new deep generative model by solving the Poisson equation whose source term is thedata distribution.
We estimate the normalized gradient ﬁeld of the solution in an augmented spacewith an additional dimension.
For sampling, we devise a backward ODE that exponential decays onthe physically meaningful additional dimension.
Empirically, our approach has competitive or betterperformance than baselines on image generation, while achieving 10 ⇥ to 20 ⇥ acceleration.
Our 312 backward ODE shows greater stability against error than popular ODE-based methods.
We furtherdemonstrate the utility of the forward ODE on likelihood evaluation, image interpolation and adaptivesampling.
Future directions include improving the normalization of Poisson ﬁelds and speeding upthe training process.
More principled approaches can be used to get around the divergent near-ﬁeldbehavior.
For example, we may exploit renormalization, a useful tool in physics, to make the Poissonﬁeld well-behaved in near ﬁelds.
Another direction is to explore faster sampling algorithms to avoidsequential simulation in the forward ODE.