Original: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models
Revised: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models

Original: Abstract
Revised: Abstract

Original: Generative models (e.g., GANs and diffusion models) learn the underlying data distribution in an unsupervised manner.
Revised: Generative models (e.g., GANs, diffusion models) learn the underlying data distribution in an unsupervised manner.

Original: However, many applications of interest require sampling from a speciﬁc region of thegenerative model’s output space or  evenly over a range of characteristics.
Revised: However, many applications of interest require sampling from a particular region of the output space or sampling evenly over a range of characteristics.

Original: To allow efﬁcient sampling in these scenarios, we propose Generative Visual Prompt (PromptGen), a framework for distributional control over pre-trained generative models by incorporating knowledge of arbitrary off-the-shelf models.
Revised: For efﬁcient sampling in these scenarios, we propose Generative Visual Prompt (PromptGen), a framework for distributional control over pre-trained generative models by incorporating knowledge of other off-the-shelf models.

Original: PromptGen deﬁnes control asan energy-based model (EBM) and samples images in a feed-forward manner by approximating the EBM with invertible neural networks, avoiding optimization at inference.
Revised: PromptGen deﬁnes control as energy-based models (EBMs) and samples images in a feed-forward manner by approximating the EBM with invertible neural networks, avoiding optimization at inference.

Original: We demonstrate how
Revised: Our experiments demonstrate how

Original: PromptGen can control several  generative models (e.g., StyleGAN2, StyleNeRF, diffusion autoencoder,and NVAE)  using various off-the-shelf models: (1) with the CLIP model, PromptGen can sample images guided by text, (2) with image classiﬁers, PromptGen can de-bias generative models across a set of attributes, and (3) with inverse graphics models, PromptGen can sample images of the same identity in different poses.
Revised: PromptGen can efﬁciently sample from several unconditional generative models (e.g., StyleGAN2, StyleNeRF, diffusion autoencoder, NVAE) in a controlled or/and de-biased manner using various off-the-shelf models: (1) with the CLIP model as control, PromptGen can sample images guided by text, (2) with image classiﬁers as control, PromptGen can de-bias generative models across a set of attributes or attribute combinations, and (3) with inverse graphics models as control, PromptGen can sample images of the same identity in different poses.

Original: (4) Finally, PromptGen reveals that the CLIP model shows  “reporting bias” when used as control, and PromptGen can further de-bias this controlled distribution in an iterative manner.
Revised: (4) Finally, PromptGen reveals that the CLIP model shows a “reporting bias” when used as control, and PromptGen can further de-bias this controlled distribution in an iterative manner.

Original:  Introduction
Revised:  Introduction

Original: Generative models learn the underlying high-dimensional data distribution and have achieved promising performance on image synthesis [4, 73, 35, 21].
Revised: Generative models learn the underlying high-dimensional data distribution and have achieved promising performance on image synthesis [4, 74, 35, 21].

Original: Though being well praised, they still face two main criticisms.
Revised: Though being well praised, they still face two main limitations.

Original: First, since generative models are typically trained in an unsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁc region of the space.
Revised: First, since generative models are typically trained in an unsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁc region of the space.

Original: Second, generative models are prone to inherit the imbalance and bias of training data [58, 30].
Revised: Second, generative models are prone to inherit the imbalance and bias of training data [59, 30].

Original: For instance, StyleGAN2 is more likely to produce images of white individuals, see Figure 1(e).
Revised: For instance, StyleGAN2 is more likely to produce images of white individuals, see Figure 1(e).

Original: Previous works have studied these challenges separately, and typical methods include editing of “style” codes [22, 66, 30] and explicit conditions [41, 67].
Revised: Previous works have studied these challenges separately, and typical methods include editing of “style” codes [22, 67, 30] and explicit conditions [42, 68].

Original: However, these methods are either model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiring all training samples to be labeled for explicit conditions), limiting their generality and practical use.
Revised: However, these methods are either model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiring all training samples to be labeled for explicit conditions), limiting their generality and practical use.

Original: To address the above challenges, this paper advocates a uniﬁed formulation, distributional control of generative models , which enables controllability by incorporating the knowledge of off-the-shelf models (e.g., CLIP [56], classiﬁers, or inverse graphics models [16]).
Revised: To address the above challenges, this paper advocates a uniﬁed formulation, distributional control of generative models , which enables controllability by incorporating the knowledge of off-the-shelf models (e.g., CLIP [57], classiﬁers, or inverse graphics models [16]).

Original: Based on this uniﬁed view, we propose to learn distributions in the latent space of a pre-trained generative model while keeping the pre-trained weights ﬁxed.
Revised: Based on this uniﬁed view, we propose to learn distributions in the latent space of a pre-trained generative model while keeping the pre-trained weights ﬁxed.

Original: Given its conceptual similarity to prompt learning [40, 83, 84,  29], we term our framework as Generative Visual Prompt (PromptGen).
Revised: Given its conceptual similarity to prompt learning [41, 84, 85, 29], we term our framework as Generative Visual Prompt (PromptGen).

Original: PromptGen requires no training data , and the only supervision comes from off-the-shelf models that help deﬁne the control.
Revised: PromptGen requires no training data , and the only supervision comes from off-the-shelf models that help deﬁne the control.

Original: Speciﬁcally, 
Revised: Speciﬁcally, [Figure]

Original: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated by an invertible neural network (INN).
Revised: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated by an invertible neural network (INN).

Original: Unlike methods that require optimization at inference in
Revised: Unlike methods that require optimization at inference in

Original: EBM sampling [47, 14, 43, 50], PromptGen samples images in a feed-forward manner, which is highly efﬁcient.
Revised: EBM sampling [48, 14, 44, 51], PromptGen samples images in a feed-forward manner, which is highly efﬁcient.

Original: Moreover, PromptGen stands alone at inference, meaning that we can discard the off-the-shelf models, which deﬁne the control, after training.
Revised: Moreover, PromptGen stands alone at inference, meaning that we can discard the off-the-shelf models, which deﬁne the control, after training.

Original: We illustrate the beneﬁts of PromptGen with several experiments.
Revised: We illustrate the beneﬁts of PromptGen with several experiments.

Original: Figure 1 demonstrates our main ﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for
Revised: Figure 1 demonstrates our main ﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for

Original: StyleNeRF [21], diffusion autoencoder [55], and NVAE [73] in Section 4.
Revised: StyleNeRF [21], diffusion autoencoder [56], and NVAE [74] in Section 4.

Original: Figure 1(a) illustrates how we can sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.
Revised: Figure 1(a) illustrates how we can sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.

Original: Figure 1(b) shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) to sample uniformly across all races, de-biasing the pre-trained StyleGAN2.
Revised: Figure 1(b) shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) to sample uniformly across all races, de-biasing the pre-trained StyleGAN2.

Original: Figure 1(d) illustrates that
Revised: Figure 1(d) illustrates that

Original: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.
Revised: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.

Original: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic design and modularity for control composition, but also enables iterative controls when some controls are contingent on others.
Revised: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic design and modularity for control composition, but also enables iterative controls when some controls are contingent on others.

Original: For instance, one may train a text-controlled distribution and then de-bias this distribution.
Revised: For instance, one may train a text-controlled distribution and then de-bias this distribution.

Original: To achieve this, we view the composition of the INN and the generative model as a new “generative model” to be controlled.
Revised: To achieve this, we view the composition of the INN and the generative model as a new “generative model” to be controlled.

Original: Figure 1(f) illustrates that PromptGen reveals “reporting bias” of the CLIP model [56], where “without makeup” is – perhaps surprisingly – positively correlated with female, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control. between methods. StyleFlow [1] PPGM [47] /
Revised: Figure 1(f) illustrates that PromptGen reveals “reporting bias” of the CLIP model [57], where “without makeup” is – perhaps surprisingly – positively correlated with female, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control

Original: [Table]

Original: LACE

Original: [Table]

Original: DDPM [10]

Original: [Table]

Original: [Table]

Original: X

Original: X

Original: [Table]X X [Table]
Revised: [Table]

Original: X

Original: [Table]X X [Table]
Revised: [Table]

Original: X

Original:  Related Work
Revised:  Related Work

Original: Over the past few years, generative models have gained the ability to generate images with high visual quality.
Revised: Over the past few years, generative models have gained the ability to generate images with high visual quality.

Original: A few of the most widely used methods include generative adversarial networks (GANs) [18],
Revised: A few of the most widely used methods include generative adversarial networks (GANs) [18],

Original: VAE [38], invertible neural networks [11, 12], and diffusion models [24, 69].
Revised: VAE [39], invertible neural networks [11, 12] [Table] and diffusion models [24, 70].

Original: In particular, generative models trained on large amounts of unlabeled data, e.g., BigGAN [4], StyleGANs [34, 35, 33, 62],
Revised: In particular, generative models trained on large amounts of unlabeled data, e [Table] g., BigGAN [4], StyleGANs [34, 35, 33, 63],

Original: Glow [37], and diffusion models [49, 10], achieve promising image synthesis results.
Revised: Glow [38] [Table] and diffusion models [50, 10], achieve promising image synthesis results.

Original: Despite their success, controllability and de-biasing are still two fundamental challenges generative models face.
Revised: Despite their success, controllability and de-biasing are still two fundamental challenges generative models face.

Original: For controllability, existing methods include explicit conditioning at training [41, 67] or local editing of the learned representation, e.g, “style” codes [1, 54, 44, 66, 78].
Revised: For controllability, existing methods include explicit conditioning at training [42, 68] or local editing of the learned representation, e.g, “style” codes [1, 55, 45, 67, 79].

Original: For de-biasing, existing methods include local editing of “style” codes [58, 30] and importance sampling for either training [20] or inference [28].
Revised: For de-biasing, existing methods include local editing of “style” codes [59, 30] and importance sampling for either training [20] or inference [28].

Original: Existing works study these problems separately, each requiring a speciﬁc design for the task studied.
Revised: Existing works study these problems separately, each requiring a speciﬁc design for the task studied.

Original: On the contrary, PromptGen is a uniﬁed framework for arbitrary controls deﬁned by off-the-shelf models.
Revised: On the contrary, PromptGen is a uniﬁed framework for arbitrary controls deﬁned by off-the-shelf models.

Original: The beneﬁts of unifying tasks have been shown by a recent trend of works in multiple research areas across vision and language [57, 46, 61, 79, 59].
Revised: The beneﬁts of unifying tasks have been shown by a recent trend of works in multiple research areas across vision and language [58, 47, 62, 80, 60].

Original: Moreover, one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen: since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuning can be applied before or after PromptGen training.
Revised: Moreover, one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen: since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuning can be applied before or after PromptGen training.

Original: We leave this exploration to future studies.
Revised: We leave this exploration to future studies.

Original: Previous methods usually sample from EBM [39] with Markov Chain Monte Carlo (MCMC) [71, 74, 14, 19, 13, 43].
Revised: Previous methods usually sample from EBM [40] with Markov Chain Monte Carlo (MCMC) [72, 75, 14, 19, 13, 44].

Original: Among them, plug-and-play generative models (PPGMs) [47] and LACE [50] deﬁne latent-space EBMs.
Revised: Among them, plug-and-play generative models (PPGMs) [48] and LACE [51] deﬁne latent-space EBMs.

Original: However, MCMC requires inference-time optimization, which is inefﬁcient and requires the off-the-shelf models to be available at inference; this is also the criticism for diffusion models, regardless of being gradient-guided [49, 65] or not [24, 25, 48].
Revised: However, MCMC requires inference-time optimization, which is inefﬁcient and requires the off-the-shelf models to be available at inference; this is also the criticism for diffusion models, regardless of being gradient-guided [50, 66] or not [24, 25, 49].

Original: In contrast, PromptGen achieves efﬁcient, feed-forward sampling.
Revised: In contrast, PromptGen achieves efﬁcient, feed-forward sampling.

Original: Table 1 shows a comparison with previous methods.
Revised: Table 1 shows a comparison with previous methods.

Original: Our
Revised: Our

Original: INN training is similar to that proposed by [52] to sample from physical systems, which is later used by [76] to solve inverse problems with two composed INNs.
Revised: INN training is similar to that proposed by [53] to sample from physical systems, which is later used by [77] to solve inverse problems with two composed INNs.

Original: Notably, [52] and [76] do not leverage a low-dimensional latent space, and [76] requires training a separate model for each image sample.
Revised: Notably, [53] and [77] do not leverage a low-dimensional latent space, and [77] requires training a separate model for each image sample.

Original: In this paper, we use INN to model arbitrary EBMs for various generative models. 
Revised: In this paper, we use INN to model arbitrary EBMs for various generative models. [Figure] views the composition G

Original:  Method
Revised:  Method

Original: Figure 2 illustrates our PromptGen framework.
Revised: Figure 2 illustrates our PromptGen framework.

Original: To begin, the user selects a pre-trained generative model G .
Revised: To begin, the user selects a pre-trained generative model G .

Original: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .
Revised: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .

Original: We train an INN f ✓ to approximate p ( z |C ) .
Revised: We train an INN f θ to approximate p ( z |C ) .

Original: PromptGen views the functional composition G  f ✓ as a new generative model and can perform iterative control.
Revised: PromptGen views the functional composition G ◦ f θ as a new generative model and can perform iterative control.

Original: Algorithm 1 describes the overall procedure. 
Revised: Algorithm 1 describes the overall procedure. Algorithm 1:

Original: [Table] [Table] [Table]
Revised: p

Original: [Table]
Revised: z | f θ R d

Original: Input: control C of the current iteration 2. Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3. Train an INN f ✓ : Rd !
Revised: R

Original: R d to approximate p ( z |C) (Section 3.2) 4. G G  f ✓
Revised: approximate p ( z |C

Original:  3.1 Latent-Space EBM for Distributional Control
Revised: Section 3.1 Latent-Space EBM for Distributional Control

Original: The plug-and-play generative model [47] was ﬁrst proposed to use a latent-space EBM for controllable image synthesis.
Revised: The plug-and-play generative model [48] was ﬁrst proposed to use a latent-space EBM for controllable image synthesis.

Original: If a ﬁxed generative model is used, then theoretically, any image-space EBM can be viewed as a latent-space EBM, as shown by [19] and [50].
Revised: If a ﬁxed generative model is used, then theoretically, any image-space EBM can be viewed as a latent-space EBM, as shown by [19] and [51].

Original: We deﬁne the latent-space EBM following similar formulation as [47, 19, 50], with some new energy functions.
Revised: We deﬁne the latent-space EBM following similar formulation as [48, 19, 51], with some new energy functions.

Original: We then extend the formulation to incorporate the moment constraint [7], which was adopted for language modeling [36], but unlike [36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.
Revised: We then extend the formulation to incorporate the moment constraint [7], which was adopted for language modeling [36], but unlike [36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.

Original: We deﬁne a control C as M independent properties { y 1 , . . .
Revised: We deﬁne a control C as M independent properties { y 1 , . . .

Original: , y M } , e.g., y 1 can be a text description and y 2 can be an attribute.
Revised: , y M } , e.g., y 1 can be a text description and y 2 can be an attribute.

Original: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)
Revised: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)

Original: [Equation]
Revised: [Equation]

Original: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy are preferred.
Revised: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy are preferred.

Original: Using a pre-trained generative model G : R d ! X that maps a latent code z to an image x , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping it to x = G ( z ) .
Revised: Using a pre-trained generative model G : R d → X that maps a latent code z to an image x , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping it to x = G ( z ) .

Original: Appendix B.2 shows that this EBM is equivalent to the latent-space EBMp ( z |C ) = p z (
Revised: Appendix B.2 shows that this EBM is equivalent to the latent-space EBM

Original: [Equation]
Revised: [Equation]

Original: Latent-space EBM allows us to use any off-the-shelf model to specify the control.
Revised: Latent-space EBM allows us to use any off-the-shelf model to specify the control.

Original: The following are some examples that are discussed in this paper (explained in Appendix B.1):
Revised: The following are some examples that are discussed in this paper (explained in Appendix B.1):

Original: Classiﬁer energy:
Revised: Classiﬁer energy:

Original: Given a classiﬁer P ( ·| x) and the target class a that we want to
Revised: Given a classiﬁer P ( ·| x

Original:  sample images from, we deﬁne the classiﬁer energy as E classiﬁer ( x , a ) =
Revised: ) and the target class a that we want to sample images from, we deﬁne the classiﬁer energy as E classiﬁer ( x , a ) =

Original:  log P ( a | x ) .
Revised: − log P ( a | x ) .

Original: CLIP energy: Using the CLIP model [56], we deﬁne the CLIP energy as the cosine 
Revised: CLIP energy: Using the CLIP model [57], we deﬁne the CLIP energy as the cosine distance between the embeddings of the image and the text t , averaged over L differentiable augmentations [83, 45]:

Original: distance

Original: between

Original: the embeddings of the image and the text t

Original: averaged over

Original: L

Original: differentiable augmentations [

Original: [Equation]
Revised: [Equation]

Original: Inverse graphics energy:
Revised: Inverse graphics energy:

Original: Given an inverse graphics model, f X ! P , which infers image parameters (e.g., pose and expression), and the target parameters ⇢ , we deﬁne the inverse graphics energy as
Revised: Given an inverse graphics model, f X→P , which infers image parameters (e.g., pose and expression), and the target parameters ρ , we deﬁne the inverse graphics energy as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where d h · , · i is the geodesic distance between the inferred parameters and the target parameters.
Revised: where d (cid:104)· , ·(cid:105) is the geodesic distance between the inferred parameters and the target parameters.

Original: Moment constraint:
Revised: Moment constraint:

Original: Some controls cannot be directly deﬁned by off-the-shelf models, and the moment constraint [7, 36] is one of them.
Revised: Some controls cannot be directly deﬁned by off-the-shelf models, and the moment constraint [7, 36] is one of them.

Original: Given a mapping  : X !
Revised: Given a mapping γ : X → R K

Original: R K (e.g.,  can be a classiﬁer that outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as
Revised: (e.g., γ can be a classiﬁer that outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation] [Table] from [Table]
Revised: Deviation

Original: [Table]
Revised: [Table]

Original: distribution
Revised: distribution

Original: [Equation]
Revised: [Equation]

Original: Moment constraint
Revised: (cid:125)

Original: [Equation]
Revised: [Equation]

Original: where µ is the user-speciﬁed vector.
Revised: where µ is the user-speciﬁed vector.

Original: For example, if we want to generate images that are uniformly distributed across races, we may use a race classiﬁer as  and deﬁne µ =  |A|  1 , . .
Revised: For example, if we want to generate images that are uniformly distributed across races [Equation] we may use a race classiﬁer as γ and deﬁne µ = (cid:0) |A| − 1 , . .

Original:  , |A|  where
Revised: . , |A| − 1 (cid:1) where

Original: A is the set of races.
Revised: A is the set of races.

Revised: It means that we would like to ﬁnd a distribution that 1) stays close to the original, pre-trained generative model G and 2) mitigates G ’s bias.

Original: In this paper, we generalize the moment constraint to the latent space, and approximate the above objective as (detailed in Appendix B.6): 
Revised: In this paper, we generalize the moment constraint to the latent space [Equation] and approximate the above objective as (detailed in Appendix B.6): [Equation]

Revised: Notably, to optimize ˆ β in Eq.

Original: [Equation]
Revised: [Equation]

Original: 3.2 Approximating EBM with Invertible Neural Network
Revised: we leverage existing gradient-based optimization tools such as stochastic gradient descent (SGD) and Adam [37]. [Figure] while not converged do Sample (cid:15) ∼ N ( 0 , I ) 2.

Revised: Map (cid:15) to latent code z = f θ ( (cid:15) ) 3.

Original:  Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcient sampling network.
Revised: Map z to an image x = G ( z ) 4. Optimize θ with gradient ∇ θ (cid:16) − log | det( ∂f θ ∂ (cid:15) ) | − log p z ( z ) + E C ( x ) (cid:17) 3.2 Approximating EBM with Invertible Neural Network Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcient sampling network.

Original: To achieve this, we train a distribution p ✓ ( z ) that minimizes the KL divergence 
Revised: To achieve this, we train a distribution p θ ( z ) that minimizes the KL divergence D

Original: D KL ( p ✓ ( z ) k p ( z |C )) .
Revised: KL ( p θ ( z ) (cid:107) p ( z |C )) .

Original: Estimating D KL ( p ✓ ( z ) k p ( z |C )) requires easily sampling z ⇠ p ✓ and tractably computing p ✓ ( z ) .
Revised: Estimating D KL ( p θ ( z ) (cid:107) p ( z |C )) requires easily sampling z ∼ p θ and tractably computing p θ ( z ) .

Original: Inspired by [52], we model p ✓ with an INN f ✓ that deﬁnes a bijection z
Revised: Inspired by [53], we model p θ with an INN f θ that deﬁnes a bijection z

Original: = f ✓ ( ✏ ) , which has two merits besides the invertibility: (1) one can easily sample z by sampling ✏ ⇠ N ( 0 , I ) and mapping it to z= f ✓ ( ✏ ) , and (2) p ✓ ( z ) has a closed-form solution: [Equation] [Figure] not
Revised: = f θ ( (cid:15) ) , which has two merits besides the invertibility: (1) one can easily sample z by sampling (cid:15) ∼ N ( 0 , I ) and mapping it to z

Original: Sample ✏ ⇠ N ( 0 , I ) 2. Map ✏ to latent code z = f ✓ ( ✏ ) 3. Map z to an image x = G ( z ) 4. Optimize ✓ with gradient r ✓
Revised: = f θ ( (cid:15) ) , and (2) p θ ( z ) has a closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: ⌘

Original: Based on these properties of INN
Revised: Based on these properties of INN

Original: [Equation]
Revised: [Equation]

Original: we can rewrite our KL divergence objective D KL ( p ✓ ( z ) k p ( z |C )) as (full derivations in Appendix B.3)
Revised: we can rewrite our KL divergence objective D KL ( p θ ( z ) (cid:107) p ( z |C )) as (full derivations in Appendix B.3)

Original: the following form:
Revised: the following form:

Original: [Equation]
Revised: [Equation]

Original: Since H N ( 0 , I ) and log
Revised: Since H N ( 0 , I ) and log

Original: Z are independent of ✓ , our training objective becomes 
Revised: Z are independent of θ [Equation] our training objective becomes arg min θ

Original: [Equation]
Revised: [Equation]

Original: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.
Revised: Figure 3 gives an illustration of our process [Equation] and Algorithm 2 describes the algorithmic details.

Original: PromptGen in a class-embedding space (Figure 5(e))
Revised: Figure 5(e)

Original: Previous works [4, 62] have shown that class conditioning boosts generative models’ performances on ImageNet [60].
Revised: Previous works [4, 63] have shown that class conditioning boosts generative models’ performances on ImageNet [61].

Original: Speciﬁcally, classconditioned generative models map a latent code z and a class embedding y to x = G ( z , y ) .
Revised: Speciﬁcally, classconditioned generative models map a latent code z and a class embedding y to x = G ( z , y ) .

Original: To extend PromptGen to these models, we train an INN h ✓ to map ⇠ ⇠ N ( µ ,  2 I ) to y =
Revised: To extend PromptGen to these models, we train an INN h θ to map ξ ∼ N ( µ , σ 2 I ) to y =

Original: h ✓ ( ⇠ ) , where µ and  are the mean and standard deviation of G ’s class embeddings.
Revised: h θ ( ξ ) , where µ and σ are the mean and standard deviation of G ’s class embeddings.

Original: The motivation for deﬁning the distribution of ⇠ as N ( µ ,  2 I ) but not N ( 0 , I ) is that we want the learned INN h ✓ to be volumepreserving, which is easier to train.
Revised: The motivation for deﬁning the distribution of ξ as N ( µ , σ 2 I ) but not N ( 0 , I ) is that we want the learned INN h θ to be volumepreserving, which is easier to train.

Original: The training objective is to minimize D KL ( p ✓ ( z , y ) k p ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4) arg min ✓
Revised: The training objective is to minimize D KL ( p θ ( z , y ) (cid:107) p ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4):

Original: [Equation]
Revised: [Equation]

Original: where p y ( y ) =
Revised: where p y ( y ) =

Original: N ( y | µ ,  2 I ) is the estimated class-embedding distribution.
Revised: N ( y | µ , σ 2 I ) is the estimated class-embedding distribution.

Original: PromptGen with conditional INN To generalize the control to continuous values, e.g., the scene parameters ⇢ in Eq.
Revised: PromptGen with conditional INN To generalize the control to continuous values, e.g., the scene parameters ρ in Eq.

Original: (4), we condition the INN on ⇢ .
Revised: (4), we condition the INN on ρ .

Original: The condition is modeled by replacing z = f ✓ ( ✏ ) with z = f ✓ ( ✏ , ⇢ ) .
Revised: The condition is modeled by replacing z = f θ ( (cid:15) ) with z = f θ ( (cid:15) , ρ ) .

Original: Space limited, we provide details and derivations in Appendix B.5.
Revised: Space limited, we provide details and derivations in Appendix B.5.

Original: This extension results in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label pairs and is only applicable to explicit condition, which is not capable of modeling EBMs.
Revised: This extension results in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label pairs and is only applicable to explicit condition, which is not capable of modeling EBMs.

Original: When the condition is modeled by the equivariant operations proposed by [77], PromptGen also satisﬁes the homomorphism property [77] in the latent space.
Revised: When the condition is modeled by the equivariant operations proposed by [78], PromptGen also satisﬁes the homomorphism property [78] in the latent space.

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] visualization

Revised: C for results on more datasets and descriptions

Original:  Experiments
Revised:  Experiments

Original: This section describes the experimental validation of PromptGen.
Revised: This section describes the experimental validation of PromptGen.

Original: See Appendix A for experiments on synthetic data and Appendix C, D, E and F for additional experiments on images and 3D meshes.
Revised: See Appendix A for experiments on synthetic data and Appendix C, D, E, and F for additional experiments on images and 3D meshes.

Original: Additional experimental details are provided in Appendix B.8.
Revised: Additional experimental details are provided in Appendix B.8.

Original: 4.1 Image Synthesis based on Text Description
Revised: 4.1 Image Synthesis based on Text Description

Original: This experiment illustrates the capability of PromptGen to sample images from a generative model driven by a text description t using the pre-trained CLIP model [56] (the ViT-B/32 version).
Revised: This experiment illustrates the capability of PromptGen to sample images from a generative model driven by a text description t using the pre-trained CLIP model [57] (the ViT-B/32 version).

Original: We used the CLIP energy from Eq.
Revised: We used the CLIP energy from Eq.

Original: (3), with text descriptions such as a photo of a baby.
Revised: (3), with text descriptions such as a photo of a baby.

Original: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generation methods, StyleCLIP [54] and StyleGAN2-NADA [17].
Revised: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generation methods, StyleCLIP [55] and StyleGAN2-NADA [17].

Original: We observe that PromptGen generates diverse and high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and
Revised: We observe that PromptGen generates diverse and high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and

Original: StyleGAN2-NADA generates baby-like adults.
Revised: StyleGAN2-NADA generates baby-like adults.

Original: These results show that (1) locally editing the latent code (i.e., StyleCLIP) is not always an effective method for controlling generative models (e.g., not all images’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA) is not effective in seeking modes in a generative model.
Revised: These results show that (1) locally editing the latent code (i.e., StyleCLIP) is not always an effective method for controlling generative models (e.g., not all images’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA) is not effective in seeking modes in a generative model.

Original: Figure 5(a), Figure 5(b), and Figure 5(d) show how PromptGen applies to StyleNeRF [21], NVAE [73], and diffusion autoencoder (DiffAE; [55]).
Revised: Figure 5(a), Figure 5(b), and Figure 5(c) show how PromptGen applies to StyleNeRF [21], NVAE [74], and diffusion autoencoder (DiffAE; [56]).

Original: Although these generative models are all trained on FFHQ, baby images sampled from them have distinct characteristics.
Revised: Although these generative models are all trained on FFHQ, baby images sampled from them have distinct characteristics.

Original: Figure 5(c) shows the ability of PromptGen to generate cats of a particular species.
Revised: Figure 5(d) shows the ability of PromptGen to generate cats of a particular species.

Original: Figure 5(e) shows the extension (Section 3.2) of our PromptGen to the class-embedding space of BigGAN [4], a class-conditional GAN trained on ImageNet [60].
Revised: Figure 5(e) shows the extension (Section 3.2) of our PromptGen to the class-embedding space of BigGAN [4], a class-conditional GAN trained on ImageNet [61].

Original: We observe that PromptGen helps BigGAN generate images with complex text descriptions, which are out of BigGAN’s training image distribution; however, the diversity seems to be limited in this extension.[Figure] [Figure] [Figure]
Revised: We observe that PromptGen helps BigGAN generate images with complex text descriptions, which are out of BigGAN’s training image distribution; however, the diversity seems to be limited in this extension.

Original: r o p o r t i o n

Original: Race

Original: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern

Original: [Figure]
Revised: [Figure]

Original: t

Original: [Figure]

Original: n

Original: Race

Original: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] 4.2 De-Biasing Pre-Trained Generative Models
Revised: 4.2 De-Biasing Pre-Trained Generative Models

Original: An important problem in generative models is to generate fair distributions w.r.t a set of attributes of interest.
Revised: An important problem in generative models is to generate fair distributions w.r.t a set of attributes of interest.

Original: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races and ages.
Revised: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races and ages.

Original: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] in terms of categorical attributes, using the moment constraint deﬁned in Eq.
Revised: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] in terms of categorical attributes, using the moment constraint deﬁned in Eq.

Original: (5) and Eq.
Revised: (5) and Eq.

Original: As control, we used a classiﬁer trained on FairFace 224 2 [31] as  .
Revised: As control, we used a classiﬁer trained on FairFace 224 2 [31] as γ .

Original: We deﬁned µ =  |A|  1 , . ..
Revised: We deﬁned µ = (cid:0) |A| − 1 , . .

Original:  , |A|  , where
Revised: . , |A| − 1 (cid:1) , where

Original: A is the set of races.
Revised: A is the set of races.

Original: Similar to the energy weights  i deﬁned in Eq.
Revised: Similar to the energy weights λ i deﬁned in Eq.

Original: (2), we propose to rescale the trained ˆ  as  ˆ  .
Revised: (2), we propose to rescale the trained ˆ β as λ ˆ β .

Original: Figure 6 shows that PromptGen de-biases the race and age effectively.
Revised: Figure 6 shows that PromptGen de-biases the race and age effectively.

Original: Existing de-biasing baselines consider binary attributes [30].
Revised: Existing de-biasing baselines consider binary attributes [30].

Original: For a fair comparison with them, we adopted their setting to use binary classiﬁers trained on CelebA [45] for de-biasing and evaluation.
Revised: For a fair comparison with them, we adopted their setting to use binary classiﬁers trained on CelebA [46] for de-biasing and evaluation.

Original: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, we did not use the moment constraint for this experiment.
Revised: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, we did not use the moment constraint for this experiment.

Original: Instead, since PromptGen allows conditional image generation with the classiﬁer energy, we generated the same number of samples conditioned on each attribute or attribute combination.
Revised: Instead, since PromptGen allows conditional image generation with the classiﬁer energy, we generated the same number of samples conditioned on each attribute or attribute combination.

Original: Table 2 shows that PromptGen has competitive performance on de-biasing for attributes and attribute combinations. 
Revised: Table 2 shows that PromptGen has competitive performance on de-biasing for attributes and attribute combinations. [Table] [Table]

Original: 4.3 Pose-Guided Face Synthesis
Revised: 4.3 Pose-Guided Face Synthesis

Original: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.
Revised: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.

Original: We used the DECA model [16], which infers the parameters of FLAME [42], a parametric facial graphics model.
Revised: We used the DECA model [16], which infers the parameters of FLAME [43], a parametric facial graphics model.

Original: We set ⇢ 2 SO (3) as FLAME’s three neck poses and used the conditional INN extension introduced in Section 3.2.
Revised: We set ρ ∈ SO (3) as FLAME’s three neck poses and used the conditional INN extension introduced in Section 3.2.

Original: To enable generating different poses of the same identity (ID), we propose ID energy using the IR-SE50 model [8].
Revised: To enable generating different poses of the same identity (ID), we propose ID energy using the IR-SE50 model [8].

Original: Speciﬁcally, given a canonical pose ⇢ 0 , we deﬁne z [-
-] =
Revised: Speciﬁcally, given a canonical pose ρ 0 , we deﬁne z [-
-]0 =

Original: f ✓ ( ✏ , ⇢ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)
Revised: f θ ( (cid:15) , ρ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)

Original: [Equation]
Revised: [Equation]

Original: where R is the IR-SE50 model [8] that computes face embeddings.
Revised: where R is the IR-SE50 model [8] that computes face embeddings.

Original: Figure 7 shows that PromptGen generates faces of the same ID in different poses, even without being explicitly trained with poses as conditions.
Revised: Figure 7 shows that PromptGen generates faces of the same ID in different poses, even without being explicitly trained with poses as conditions.

Original: We computed the FID score [23] of each model using Clean-FID [53]; following [35],[Table] we did not use the truncation trick when computing the FID score.
Revised: We computed the FID score [23] of each model using Clean-FID [54]; following [35], we did not use the truncation trick when computing the FID score.

Original: Table 3 shows that PromptGen outperforms existing models in terms of the FID score.
Revised: Table 3 shows that PromptGen outperforms existing models in terms of the FID score.

Original: Following [67], we then reported the average
Revised: Following [68], we then reported the average

Original: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.
Revised: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.

Original: Results show that PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.
Revised: Results show that PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.

Original: 4.4 Decomposing Complex Control via Energy Composition
Revised: 4.4 Decomposing Complex Control via Energy Composition

Original: Energy composition E [Equation] Mi
Revised: Energy composition E C ( x ) = (cid:80) Mi

Original: [Equation] E i ( x , y i )  allows us to decompose  controls into simple ones. {+For instance, Figure 8(a) shows that y 
+}
Revised: =1 λ i E i ( x , y i ) in Eq. 1 allows us to decompose a complex controls into simple ones. {+For instance, Figure 8(a) shows that y 
+}

Original: We show that y 
 {+= photo of a bald man ( λ 1 = 1500 ), y 2 = photo of a black man ( λ 1 = 3000 ), and y 
 = photo of a man with beard ( λ 1 = 1500 ), we have better control, as shown in Figure 8(b).+}
Revised: = photo of a bald black man with beard ( λ 1 = 6000 ) generates good samples of men with beard but not all of them have a dark-skin tone; however, by decomposing it as y 
 {+= photo of a bald man ( λ 1 = 1500 ), y 2 = photo of a black man ( λ 1 = 3000 ), and y 
 = photo of a man with beard ( λ 1 = 1500 ), we have better control, as shown in Figure 8(b).+}

Original: = photo of a bald black man with beard (  1 = 6000 ) is not successful (Figure 8(a))
Revised: Note that the random seed for Figure 8(a) and Figure 8(b) is the same, and hence the identities are preserved and mostly the skin tone has changed. [Figure] 4.5 Iterative Distributional Control via Functional Composition This section discusses an interesting bias that PromptGen reveals about the CLIP model.

Original: ; by decomposing it as y 1 = photo of abald man (  1 = 1500 ), y 2 = photo of a black man (  1 = 3000 )
Revised: Figure 9(a) shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males. This bias should not be attributed to image pre-training data since images contain a bias in the opposite direction, i.e., men are less likely to have makeup. We argue that this bias should be explained by CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likely to say “a person without makeup” when the person is a female (detailed analysis in Appendix F).

Revised: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterative control, enabled by the functional composition in Algorithm 1.

Revised: Speciﬁcally, in the second iteration, we de-biased the gender distribution of G ◦ f θ instead of G , where f θ is the INN learned for the text control.

Revised: For de-biasing, we used the moment constraint with ˆ β trained for G ◦ f θ .

Revised: Figure 9(b) shows that females and males are uniformly distributed after the moment constraint in the second iteration.

Revised: [Figure]

Revised: “reporting bias”.

Revised: Besides revealing this “reporting bias”, PromptGen can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution.

Revised: All images are resized for visualization.

Revised: 4.6 Inference Latency

Revised: We compared our PromptGen and the plug-and-play generative model (PPGM) [48] in terms of the inference latency.

Revised: PPGM uses Langevin dynamics [75] to optimize over the latent space at inference, while PromptGen samples images in a feed-forward manner.

Revised: In this experiment, we used PPGM and our PromptGen to approximate the same EBM with CLIP energy.

Revised: Inference times were estimated on an NVIDIA RTX A4000 GPU.

Revised: Table 4 shows that our PromptGen has the highest performance and efﬁciency.

Revised: Compared to our PromptGen, PPGM requires 100 × inference time to achieve comparable results.

Revised: Moreover, learning f θ in PromptGen requires training-time optimization, and this amortized optimization is useful when one wants to reuse a controlled distribution many times.

Original: [Table] 
Revised: [Table] [Table]

Revised:  Conclusions and Future Work

Revised: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributional control of pre-trained generative models, by leveraging the knowledge of other off-the-shelf models.

Revised: Unlike previous approaches, which require optimization for generating each sample at inference

Revised: [Table]

Revised: PromptGen trains an invertible neural network to approximate the latent distribution.

Original: and y 3 = photo of  a man with beard (  1 = 1500 ), we have better control (Figure 8(b)).
Revised: Thus, PromptGen can be viewed as an amortization of the inference-time optimization into a feed-forward neural network, allowing for efﬁcient, feed-forward sampling at inference. Further, this amortization also makes PromptGen not require the availability of those off-the-shelf models at inference. In practice [Table]

Revised: PromptGen offers generality for algorithmic design and modularity for control composition

Revised: [Table]

Revised: and it also enables iterative controls via functional composition.

Revised: Experiments validate that PromptGen applies to various generative models (StyleGAN2, StyleNeRF, diffusion autoencoder

Revised: [Table]

Revised: and NVAE), control types (continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inverse graphics models), and data domains (faces, churches, animals, ImageNet

Revised: [Table]

Revised: and landscapes).

Revised: Limitation and future work:

Revised: We provide an error analysis in Appendix F and a discussion on societal impact in Section 6.

Revised: PromptGen is restricted by the pre-trained generative model’s coverage [3, 27].

Revised: Notably, since PromptGen focuses on mode-seeking and reweighting instead of domain adaptation, the support set of the output distribution will not change after PromptGen learning.

Revised: We ﬁnd it interesting to further explore combining PromptGen and domain adaptation [17] to achieve better generalization and adaptability.

Revised: Also, PromptGen depends on the off-the-shelf models that provide knowledge about the control (Appendix F).

Revised: In subsequent work, it can be beneﬁcial to explore how to learn energy functions [15, 26] (besides our moment constraint used for de-biasing), which may provide ﬁne-grained control with less bias.

Revised:  Societal Impact

Revised: Like any technology, generative models have both beneﬁts and drawbacks for society; therefore, it is essential to investigate and understand these issues.

Revised: On the positive side, we have demonstrated that

Revised: PromptGen may be used to uncover biases learnt by text-image models (like CLIP) and to de-bias text-image models and pre-trained generative models, suggesting that PromptGen may be a helpful tool for fair AI if used properly.

Revised: The efﬁcient inference provided by PromptGen also helps reduce the computational expense, which has a positive impact on the environment.

Revised: However, improved controllability inevitably makes it easier to synthesize targeted images, which might have negative societal implications such as the creation of deceptive media (like DeepFakes [76, 73]) or privacy leakage (like identity-conditioned human face synthesis).

Revised: To battle these cases, we expect researchers and practitioners to use current technologies that can detect fake media and mitigate privacy leakage.

Revised: We also encourage practitioners to consider these risks when using PromptGen to develop systems for machine learning, visual understanding, and media synthesis.

Revised: Acknowledgments and Disclosure of Funding

Revised: The authors would like to thank the anonymous reviewers, Zoltán Ádám Milacski, Jianchun Chen, and Shubhra Aich for their valuable feedback on drafts of this paper.

Revised: A PromptGen on Synthetic Data

Revised: We demonstrate the behavior of our PromptGen with two-dimensional synthetic data, using GAN as the generative model.

Revised: This synthetic experiment is illustrated in Figure 10.

Revised: Speciﬁcally, we created a “real” distribution that is a Gaussian mixture distribution biased towards one mode.

Revised: We derive a closed-form fair classiﬁer based on this distribution, detailed in the purple block in Figure 10.

Revised: When a GAN is trained to approximate this distribution, its outputs are biased towards the over-represented mode, as shown by Figure 10(b).

Revised: Figure 10(c) illustrates the controllability experiment: using the fair classiﬁer as control, PromptGen learns a distribution concentrated in a speciﬁc region of the output space.

Revised: Figure 10(d) illustrates the de-biasing experiment: using the moment constraint (Section 3.

Revised: and Section 4.2), PromptGen upweights the under-represented regions of the output space.

Revised: B Method Details and Derivations B.1 Controllability as EBM

Revised: In Section 3, we deﬁned a control C as M independent properties { y 1 , . . .

Revised: , y M } .

Revised: For example, y 1 can be a text description, and y 2 can be an attribute.

Revised: In this part, we ﬁrst elaborate on why controllability can be formed as EBMs in Eq.

Revised: We then provide concrete examples of the distributions derived from different energy functions deﬁned in Section 3.1.

Revised: We denote the image prior as p x ( x ) , the only distribution that can be estimated from data when labels for the control are not provided during generative model pre-training.

Revised: Given the control

Revised: C = { y 1 , . . .

Revised: , y M } , we resort to Bayes’ theorem to rewrite the conditional distribution p ( x |C ) as

Revised: [Equation]

Revised: For Eq.

Revised: (15) to be well-deﬁned, we need to deﬁne p ( y i | x ) for each y i .

Revised: In order to incorporate the knowledge of arbitrary off-the-self models (besides image classiﬁers), we deﬁne each p ( y i | x ) as

Revised: [Equation]

Revised: [Equation]

Revised: Eq.

Revised: (16) says that p ( y i | x ) is proportional to the exponential of an energy function

Revised: E i ( x , y i ) , where y i with lower energy has higher density or mass.

Revised: Note that Eq.

Revised: (16) deﬁnes a distribution over all possible values of y i instead of all possible values of x .

Revised: Combining Eq.

Revised: (15) and Eq.

Revised: (16), we have p

Revised: [Equation]

Revised: which is the same equation as Eq.

Revised: In the following, we use Eq.

Revised: (16) to derive the distributions from the classiﬁer energy, CLIP energy, and inverse graphics energy deﬁned in Section 3.1.

Revised: Classiﬁer energy:

Revised: Given a classiﬁer P ( ·| x ) and the target class a , we deﬁne the classiﬁer energy as

Revised: E classiﬁer ( x , a ) = − log P ( a | x ) .

Revised: Using Eq.

Revised: (16), we arrive at:

Revised: [Equation]

Revised: which is equivalent to a temperature-adjusted distribution of the original classiﬁer.

Revised: CLIP energy: Using Eq.

Revised: (16), the CLIP energy in Eq.

Revised: (3) is equivalent to

Revised: [Equation]

Revised: Inverse graphics energy: Using Eq.

Revised: (16), the inverse graphics energy in Eq.

Revised: (4) is equivalent to

Revised: [Equation]

Revised: If the geodesic distance d (cid:10) · , · (cid:11) is the Euclidean distance, then p inv-graphics ( ρ | x ) is a Gaussian distribution whose mean is ρ and whose variance depends on the hyperparameter λ inv-graphics ; if the geodesic distance d (cid:10) · , · (cid:11) is the spherical distance (e.g., the distance between pose parameters deﬁned on a unit sphere), then p inv-graphics ( ρ | x ) is a vMF distribution.

Revised: B.2 Equivalence between Image-Space EBM and Latent-Space EBM

Revised: Proposition 1.

Revised: Deﬁne x ∼ p x ( x ) as z ∼ p

Revised: z

Revised: ( z ) , x = G ( z ) and p ( x |C ) as z ∼ p ( z |C ) , x = G ( z ) , where p ( z |C ) is deﬁned as the following EBM:

Revised: [Equation]

Revised: We have

Revised: [Equation]

Revised: In spirit, our proof follows the proof in [51], which follows [19].

Revised: The difference between our proof and that in [51] is that we derive p ( x |C ) from p ( z |C ) while they derived p ( z |C ) from p ( x |C ) .

Revised: Proof.

Revised: Based on Lemma 1 in [19] and Lemma 1 in [51], p ( z |C ) is equivalent to rejection sampling with proposal distribution p z ( z ) and acceptance probability

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: [Equation]

Revised: Since p ( x |C ) is deﬁned as z ∼ p

Revised: z ( z |C ) , x = G ( z ) , p ( x |C ) is equivalent to rejection sampling with proposal distribution p z ( z ) , x = G ( z ) and acceptance probability

Revised: [Equation]

Revised: [Equation]

Revised: Note that the above proposal distribution

Revised: z ∼ p

Revised: z

Revised: ( z ) , x = G ( z ) is the same as p x ( x ) by deﬁnition.

Revised: Based on Lemma 1 in [19] and Lemma 1 in [51], we arrive at:

Revised: [Equation]

Revised: B.3 Derivation of Eq. (9): Approximating EBM with INN

Revised: The full derivation of Eq.

Revised: (9) is given by:

Revised: [Equation]

Revised: B.4 Derivation of Eq. (11): Extension to Generative Models with a Class-Embedding Space

Revised: The full derivation of Eq.

Revised: (11) is given by:

Revised: [Equation]

Revised: y are independent

Revised: [Equation]

Revised: , z

Revised: [Equation]

Revised: , y

Revised: [Equation]

Revised: h

Revised: [Equation]

Revised: , x

Revised: [Equation]

Revised: y

Revised: [Equation]

Revised: − log |

Revised: [Equation]

Revised: ∂f θ

Revised: [Equation]

Revised: ) | − log | det

Revised: [Equation]

Revised: ∂h θ

Revised: [Equation]

Revised: ) |

Revised: [Equation]

Revised: while not converged do

Revised: Sample (cid:15) ∼ N ( 0 , I ) , ρ ∼ p ρ

Revised: [Equation]

Revised: ρ

Revised: [Equation]

Revised: Map (cid:15) to latent code z = f θ ( (cid:15) , ρ ) 3.

Revised: Map z to an image x = G ( z ) 4.

Revised: Optimize θ with gradient

Revised: [Equation]

Revised: B.5 Extension to Conditional INN

Revised: Algorithm 3 illustrates how we extend the training algorithm

Revised: (Algorithm 2) to conditional INN, which helps generalize to controls speciﬁed by continuous values ρ .

Revised: The training objective becomes

Revised: [Equation]

Revised: where C ρ means that the control is speciﬁed by value ρ .

Revised: The derivation of Algorithm 3 is given by:

Revised: [Equation]

Revised: B.6 Derivation for Latent-Space Moment Constraint

Revised: Given a mapping γ : X → R K , moment constraint [7, 36] deﬁnes the target distribution p ( x |C ) as

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: where µ is the user-speciﬁed constraint.

Revised: Examples are provided in the main text

Revised: [Equation]

Revised: and we omit them here for brevity.

Revised: [7] showed that Eq.

Revised: (33) can be approximated to an arbitrary precision by

Revised: [Equation]

Revised: (34) where ˆ β needs to be computed.

Revised: [36] estimates ˆ β by solving the following regression problem:

Revised: [Equation]

Revised: In [36], sampling x ∼ p x ( x ) is straightforward since they focus on autoregressive language models.

Revised: In the context of latent-variable vision generative models, where x ∼ p x ( x ) is implicitly deﬁned as z ∼ p

Revised: z ( z ) , x = G ( z ) , Eq.

Revised: (35) is equivalent to

Revised: [Equation]

Revised: arg min

Revised: [Equation]

Revised: [Equation]

Revised: Finally, based on Proposition 1, we can generalize the moment constraint to the latent space as

Revised: [Equation]

Revised: Note that Eq.

Revised: (36) and Eq.

Revised: (37) are verbatim copies of Eq.

Revised: (7) and Eq.

Revised: (6), respectively.

Revised: Algorithm 4:

Revised: 4.3) while not converged do

Revised: Sample (cid:15) ∼ N ( 0 , I ) , ρ ∼ p ρ ( ρ ) 2.

Revised: Map (cid:15) to latent codes z = f θ ( (cid:15) , ρ ) and z 0 = f

Revised: θ ( (cid:15) , ρ 0 ) 3.

Revised: Map z to an image x = G ( z ) and z 0 to an image x 0 =

Revised: G

Revised: ( z 0 ) 4.

Revised: Optimize θ with gradient

Revised: [Equation]

Revised: B.7 Inverse Graphics Control with Identity Energy

Revised: This subsection provides more detail on how we use an inverse graphics model, DECA [16], to control the pose of faces generated by StyleGAN2 trained on FFHQ 1024 2 , as introduced in Section 4.3.

Revised: Recall that we use the conditional INN extension, detailed in Algorithm 3, for this experiment.

Revised: To enable generating different poses of the same identity, we add additional identity energy using the

Revised: IR-SE50 model [8].

Revised: The training algorithm is detailed in Algorithm 4.

Original: {+Speciﬁcally, we generate a canonical latent code z 
+}
Revised: {+Speciﬁcally, we generate a canonical latent code z 
+}

Revised: =

Revised: f

Revised: θ ( (cid:15) , ρ 0 ) for each (cid:15) , where ρ 0 is the canonical pose.

Revised: The latent code z and the canonical latent code z 0 are mapped to the image x

Revised: and the canonical image x 0 .

Revised: The identity energy in Eq.

Revised: (12), copied below, encourages the embeddings of the two images to be similar:

Revised: [Equation]

Revised: B.8 Experimental Details

Revised: Our INN architecture contains 8 blocks.

Revised: Each block consists of a soft permutation of channels [2], an afﬁne coupling layer [12], and an ActNorm layer [38].

Revised: In the afﬁne coupling layer, we model the sub-network as an MLP with one hidden layer, where the hidden dimension is 256 and the non-linear activation is LeakyReLU 0 .

Revised: For the prior distribution p ρ ( ρ ) of the pose parameter ρ in the inverse graphics experiments (Section 4.3), we sample the x -axis and y -axis rotations relative to a canonical pose ρ 0 , whose x -axis and y -axis rotations are 0 .

Revised: 3 and 0 , respectively.

Revised: The relative pose’s x

Revised: -axis and y -axis rotations are uniformly sampled from [ − (1 / 9) π, (1 / 9) π ] .

Revised: Each experiment was run on an NVIDIA RTX A4000 GPU (with 16G memory).

Revised: Our code implementation is based on the PyTorch framework.

Revised: Our code can be found at https://github.com/

Revised: ChenWu98/Generative-Visual-Prompt .

Original: [Figure] 
Revised: [Figure] [Table]

Revised: C Additional Results for Image Synthesis based on Text Description

Revised: This section presents additional results of PromptGen for image synthesis from text, using the CLIP model as control.

Revised: As the pre-trained generative model, we explore StyleGAN2 trained on AFHQ-Cats [6], LSUN-Churches [82], FFHQ [34], and Landscape-HQ [69].

Revised: Figure 11 and Figure 12 present more results of PromptGen with diverse text descriptions.

Revised: D Additional Results for De-Biasing Generative Models

Revised: Table 5 provides quantitative results of de-biasing StyleGAN2 across categorical attributes (Figure 6).

Revised: Speciﬁcally, we used the pre-trained classiﬁer provided by FairFace [31] to classify the attributes of the generated images.

Revised: We then report the KL divergence between the attribute distribution of the generated images and the uniform distribution.

Revised: Figure 13 and Figure 14 visualize the de-biasing results.

Revised: We report the KL divergence between the generated distribution and the uniform distribution.

Revised: Interestingly, on MetFaces, de-biasing the race results in more sculptures, and we postulate that the reason is that

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: r t i

Revised: [Figure]

Revised: V e r ti ca l m ou t h s t r e t

Revised: c h

Revised: Horizontal mouth stretch

Revised: [Figure]

Revised: E y e c l o s e

Revised: [Figure]

Revised: N o s eti p s t r e t c h

Revised: [Figure]

Revised: E Experiments on Generative 3D Face Models

Revised: 3D face modeling has been an active area of research that has recently gained major interest due to applications in virtual humans, deep faces, and digital actors.

Revised: Existing 3D deep learning generative models build 3D compact representations of shape and appearance capable of modeling non-linearity (e.g., scatter effects, specularities) that is necessary for generating photo-realistic faces.

Revised: Providing a generative model with the capability of generating 3D faces with a particular geometry (e.g., a speciﬁc distance between eyes or nose length) is particularly useful for technical artists when creating new characters.

Original: 4.5 Iterative Distributional Control via Functional Composition
Revised: However, this problem remains unaddressed, partially due to the lack of publicly available 3D databases labeled with such constraints. We refer to this capability as the intra-sample constraint in 3D generative models, and this section shows how our PromptGen framework is able to achieve this by deﬁning a new energy function based on signed distances. We conducted the experiments on the FaceScape dataset [81], a large-scale 3D human face dataset consisting of 16 , 940 topologically uniformly registered 3D face meshes along with high-quality textures. The dataset contains 847 identities with 20 expressions per identity, and each mesh consists of 26 , 317 vertices of which 68 are 3D landmark vertices covering eyes, nose, mouth, jaw, and eyebrows.

Revised: With some abuse of notation, we denote a 3D face mesh as x ∈ R 3 V (recall that x is used to represent an image in other parts of this paper), where V = 26 , 317 is the number of vertices.

Revised: Given an index l of a vertex (e.g., one of the 68 landmark vertices), the 3D coordinate of this vertex is deﬁned as x [ l ] ∈ R 3 .

Original: This section discusses an interesting bias that  PromptGen reveals about the CLIP model.
Revised: We pre-train a generative mesh model G that maps each latent code z ∼ N ( 0 , I ) to a 3D mesh x . PromptGen allows us to obtain geometric control on the 3D meshes. Speciﬁcally, given two landmark vertex indices l 1 and l 2 , we deﬁne an intra-sample constraint that enforces the signed distance (along a given direction) between the two vertices to be s . To this effect, we deﬁne the signed-distance energy as

Original: Figure 9(a) shows images generated by PromptGen (with StyleGAN2) with the CLIP model and  the text description a photo of a person without makeup, where more females are  generated than males.
Revised: [Equation] where d ( · , · ) is the signed-distance (along a given direction), and | · | is the absolute value. Figure 15 presents several examples of 3D mesh control, in which random textures are applied to the generated meshes. It shows that PromptGen successfully controls the generative 3D mesh model in terms of the mouth stretch, eye openness, nose-tip stretch, and lip forward.

Original: This bias should not be attributed to image pre-training data since images contain a bias in the opposite direction, i.e., men are less likely to have makeup.
Revised: As we discussed in Section 5, the controlled distribution depends on (1) the pre-trained generative model’s coverage and (2) the off-the-shelf models used for the control. In this section, we provide some examples of PromptGen failing to generate satisfactory images using the CLIP energy. For the pre-trained generative model, we use StyleGAN2 trained on FFHQ, AFHQ-Wild, or LSUN-Churches.

Original: We argue that this bias should be explained by
Revised: [Figure] sometimes fails or shows the “reporting bias” that we discuss in Section 4.5.

Original: CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likely to say “a person without makeup” when the person is  a female (detailed analysis in Appendix F).
Revised: For a deeper understanding of these failures and biases, in Figure 17, we provide some image samples from the LAION-400M dataset [64] with CLIP retrieval, using the same text descriptions as this ﬁgure. Our ﬁrst observation is that the CLIP model sometimes fails in modeling linguistic negation. For instance, the text description photo of a man without beard results in a distribution of photos of men with beard (Figure 16(a)).

Original: Besides revealing theabove “reporting bias”, PromptGen can also mitigate thisbias via an iterative control, enabled by  the functional composition in Algorithm 1.
Revised: Meanwhile, Figure 16(b) shows that CLIP is capable of modeling the negation of having makeup, but with the “reporting bias” discussed in Section 4.5. Moreover, CLIP seems to have difﬁculty in numerical reasoning, and gaining control over the count of speciﬁc objects in a scene tends to be unsuccessful. We showed this by specifying photo of a church with three windows, which did not result in the desired speciﬁcation (Figure 16(c)). To gain a deeper understanding of the above failures and biases, in Figure 17, we provide some image samples from the LAION-400M dataset [64] with CLIP retrieval, using the same text descriptions as

Original: Speciﬁcally, in the second iteration, we de-biased the gender distribution of G  f ✓ instead of G , where f ✓ is  the INN learned for the  text control.
Revised: Figure 16. CLIP retrieval is based on the CLIP embedding similarity between the web images and text descriptions, while the original text below each individual image is not used.

Original: For de-biasing, we used the moment constraint with ˆ  trained for G  f ✓ . Figure 9(b) shows that females and males are uniformly distributed after the moment constraint in  the second iteration. [Figure]
Revised: We observe an impressive consistency between CLIP retrieval and PromptGen: in both Figure 16(a) and Figure 17(a), most images have beard; in both Figure 16(b) and Figure 17(b), all images are female; in both Figure 16(c) and Figure 17(c), some images do not have exactly three windows. This consistency suggests that the failures and biases in Figure 16 should be mostly attributed to the CLIP model rather than to our

Original: PromptGen 
Revised: PromptGen algorithm.

Original: Besides revealing this “reporting bias”, PromptGen can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution. All images are resized for visualization.
Revised: We believe that our observation sheds light on the intricacy of contrastive multi-modal (vision-language) pre-training, which is worthy of being further investigated. [Figure] illustrated in web images and text descriptions, while the original text below each individual image is not used.

Original: We compared our PromptGen and the plug-and-play generative model (PPGM) [47] in terms of the inference latency.
Revised: In

Original: PPGM uses Langevin dynamics [74] to optimize over the latent space at inference, while PromptGen samples images in a feed-forward manner. In this experiment, we used PPGM and our PromptGen to approximate the same EBM with CLIP energy.
Revised: Figure 17(a), most images have a beard; in Figure 17(b), all images are female; in Figure 17(c), some images do not have exactly three windows. These observations are consistent with those in Figure 16. Another observation is that the control tends to fail when the text description requires sampling from low-density regions of the pre-trained generative model’s output space.

Original: Inference times were estimated on an NVIDIA RTX A4000 GPU. Table 4 shows that our PromptGen has the highest performance and efﬁciency. Compared to our PromptGen, PPGM requires 100 ⇥ inference time to achieve comparable results.
Revised: In other words, the control usually fails if the pre-trained generative model does not cover the mode we are trying to gain control over. For example, images faithful to photo of a person yawning and photo of a baby with long hair are not commonly observed in the FFHQ dataset and, hence, these two text descriptions result in degeneration (Figure 18(a)) or weird images (Figure 18(b)).

Original: Moreover, learning f ✓ in PromptGen requires training-time optimization, and this amortized optimization is useful when one wants to reuse a controlled distribution many times. [Table] n !
Revised: Another example is photo of an animal from the side, which is not commonly observed in the AFHQ-Wild dataset, and Figure 18(c) shows that the generated images fail to follow this description.

Original: [Equation] [Equation] [Table]
Revised: Even when the control is successful (e.g., when a complex description is decomposed in Figure 8(b)), sampling from low-density regions results in limited diversity (e.g., the backgrounds in Figure 8(b) look similar to each other).

Original: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributional control of pre-trained generative models.
Revised: Finally, we also ran some failure controls using PPGM, showing that PromptGen and PPGM reveal similar failure cases of pre-trained generative models and CLIP.

Original: PromptGen leverages the knowledge of various off-the-shelf models and, unlike previous methods, it does not require the availability of these models at inference. PromptGen can sample images in a feed-forward manner, which is more efﬁcient than methods that require optimization at inference. PromptGen offers the generality for algorithmic design and modularity for control composition, and it also enables iterative controls.
Revised: Results are shown in Figure 19.

Original: Experiments validate that PromptGen applies to various generative models (StyleGAN2, StyleNeRF, NVAE), control types (continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inverse graphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).
Revised: [Figure]

Original: Limitation and future work: We provide an error analysis in Appendix F and a discussion on societal impact in Appendix G. PromptGen is restricted by the pre-trained generative model’s coverage [3, 27]. PromptGen focuses on mode-seeking and -reweighting instead of domain adaptation, and it is possible to combine PromptGen and domain adaptation of generative models [17]. Also, PromptGen depends on the off-the-shelf models that provide knowledge about the control (Appendix F).
Revised: [Figure]

Original: In subsequent work, it can be beneﬁcial to explore how to learn energy functions [15, 26] (besides our moment constraint used for de-biasing), which may provide ﬁne-grained control with less bias. Acknowledgments and Disclosure of Funding The authors would like to thank the anonymous reviewers, Zoltán Ádám Milacski, Jianchun Chen, and Shubhra Aich for their valuable feedback on drafts of this paper.
Revised: [Figure]