Original: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models
Revised: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Generative models (e.g., GANs, VAEs) learn the underlying data distribution inan unsupervised manner.
Revised: Generative models (e.g., GANs and diffusion models) learn the underlying data distribution in an unsupervised manner.

Original: However, many applications of interest require samplingfrom a speciﬁc region of the generative model’s output space or evenly over arange of characteristics.
Revised: However, many applications of interest require sampling from a speciﬁc region of the generative model’s output space or evenly over a range of characteristics.

Original: To allow efﬁcient sampling in these scenarios, we propose  PromptGeneration (PromptGen), a framework to achieve distributional controlover pre-trained generative models by incorporating knowledge of arbitrary offthe-shelf models.
Revised: To allow efﬁcient sampling in these scenarios, we propose Generative Visual Prompt (PromptGen), a framework for distributional control over pre-trained generative models by incorporating knowledge of arbitrary off-the-shelf models.

Original: PromptGen deﬁnes control as an energy-based model (EBM)and samples images in a feed-forward manner by approximating the EBM withinvertible neural networks, avoiding optimization at inference.
Revised: PromptGen deﬁnes control as an energy-based model (EBM) and samples images in a feed-forward manner by approximating the EBM with invertible neural networks, avoiding optimization at inference.

Original: We demonstratehow PromptGen can control several generative models  using various off-the-shelfmodels: (1) with the CLIP model, PromptGen can sample images guided by text,(2) with image classiﬁers, PromptGen can de-bias generative models across a set ofattributes, and (3) with inverse graphics models, PromptGen can sample images ofthe same identity in different poses.
Revised: We demonstrate how PromptGen can control several generative models (e.g., StyleGAN2, StyleNeRF, diffusion autoencoder, and NVAE) using various off-the-shelf models: (1) with the CLIP model, PromptGen can sample images guided by text, (2) with image classiﬁers, PromptGen can de-bias generative models across a set of attributes, and (3) with inverse graphics models, PromptGen can sample images of the same identity in different poses.

Original: (4) Finally, PromptGen reveals that the CLIPmodel shows “reporting bias” when used as control, and PromptGen can furtherde-bias this controlled distribution in an iterative manner.
Revised: (4) Finally, PromptGen reveals that the CLIP model shows “reporting bias” when used as control, and PromptGen can further de-bias this controlled distribution in an iterative manner.

Original:  Introduction
Revised:  Introduction

Original: Generative models(e.g., GANs, VAEs) learn the underlying  data distributionin high-dimensionalspaces and achieve promising performance on image synthesis [4, 74, 35, 21].
Revised: Generative models learn the underlying high-dimensional data distribution and have achieved promising performance on image synthesis [4, 73, 35, 21].

Original: Though being wellpraised, they still face two main criticisms.
Revised: Though being well praised, they still face two main criticisms.

Original: First, since generative models are typically trained in anunsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁcregion of the space.
Revised: First, since generative models are typically trained in an unsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁc region of the space.

Original: Second, generative models are prone to inherit the imbalance and bias of trainingdata [58, 30].
Revised: Second, generative models are prone to inherit the imbalance and bias of training data [58, 30].

Original: For instance, StyleGAN2 is more likely to produce images of white individuals, see Figure 1(e).
Revised: For instance, StyleGAN2 is more likely to produce images of white individuals, see Figure 1(e).

Original: Previous works have studied these challenges separately, and typical methods includeediting of “style” codes [22, 66, 30] and explicit conditions [41, 67].
Revised: Previous works have studied these challenges separately, and typical methods include editing of “style” codes [22, 66, 30] and explicit conditions [41, 67].

Original: However, these methods areeither model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiringall training samples to be labeled for explicit conditions), limiting their generality and practical use.
Revised: However, these methods are either model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiring all training samples to be labeled for explicit conditions), limiting their generality and practical use.

Original: To address the above challenges, this paper advocates a uniﬁed formulation, distributional controlof generative models , which enables controllability by incorporating the knowledge of off-the-shelfmodels (e.g., CLIP [56], classiﬁers, or inverse graphics models [16]).
Revised: To address the above challenges, this paper advocates a uniﬁed formulation, distributional control of generative models , which enables controllability by incorporating the knowledge of off-the-shelf models (e.g., CLIP [56], classiﬁers, or inverse graphics models [16]).

Original: Based on this uniﬁed view, wepropose to learn distributions in the latent space of a pre-trained generative model while keeping thepre-trained parameters ﬁxed.
Revised: Based on this uniﬁed view, we propose to learn distributions in the latent space of a pre-trained generative model while keeping the pre-trained weights ﬁxed.

Original: Given its conceptual similarity to prompt learning [40,  84, 72, 85, 29],we term our framework as  PromptGeneration (PromptGen).
Revised: Given its conceptual similarity to prompt learning [40, 83, 84, 29], we term our framework as Generative Visual Prompt (PromptGen).

Original: PromptGen requires no training data ,and the only supervision comes from off-the-shelf models that help deﬁne the control.
Revised: PromptGen requires no training data , and the only supervision comes from off-the-shelf models that help deﬁne the control.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated  by an invertible neural network (INN).
Revised: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated [Figure] by an invertible neural network (INN).

Original: Unlike methods that require optimization at inference in
Revised: Unlike methods that require optimization at inference in

Original: EBM sampling [47, 14, 43, 50], PromptGen samples images in a feed-forward manner, which ishighly efﬁcient.
Revised: EBM sampling [47, 14, 43, 50], PromptGen samples images in a feed-forward manner, which is highly efﬁcient.

Original: Moreover, PromptGen stands alone at inference, meaning that we can discard theoff-the-shelf models, which deﬁne the control, after training.
Revised: Moreover, PromptGen stands alone at inference, meaning that we can discard the off-the-shelf models, which deﬁne the control, after training.

Original: We illustrate the beneﬁts of PromptGen with several experiments.
Revised: We illustrate the beneﬁts of PromptGen with several experiments.

Original: Figure 1 demonstrates our mainﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for
Revised: Figure 1 demonstrates our main ﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for

Original: StyleNeRF [21] and NVAE [74] on non-facial data in Section 4.
Revised: StyleNeRF [21], diffusion autoencoder [55], and NVAE [73] in Section 4.

Original: Figure 1(a) illustrates how wecan sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.
Revised: Figure 1(a) illustrates how we can sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.

Original: Figure 1(b)shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) tosample uniformly across all races, de-biasing the pre-trained StyleGAN2.
Revised: Figure 1(b) shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) to sample uniformly across all races, de-biasing the pre-trained StyleGAN2.

Original: Figure 1(d) illustrates that
Revised: Figure 1(d) illustrates that

Original: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.
Revised: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.

Original: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic designand modularity for control composition, but also enables iterative controls when some controls arecontingent on others.
Revised: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic design and modularity for control composition, but also enables iterative controls when some controls are contingent on others.

Original: For instance, one may train a text-controlled distribution and then de-bias thisdistribution.
Revised: For instance, one may train a text-controlled distribution and then de-bias this distribution.

Original: To achieve this, we view the composition of the INN and the generative model as a new“generative model” to be controlled.
Revised: To achieve this, we view the composition of the INN and the generative model as a new “generative model” to be controlled.

Original: Figure 1(f) illustrates that PromptGen reveals “reporting bias” ofthe CLIP model [56], where “without makeup” is – perhaps surprisingly – positively correlated withfemale, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control.
Revised: Figure 1(f) illustrates that PromptGen reveals “reporting bias” of the CLIP model [56], where “without makeup” is – perhaps surprisingly – positively correlated with female, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control

Original: [Table] 
Revised: [Table] [Table]

Original:  Related Work
Revised:  Related Work

Original: Over the past few years, generative models have gained the ability to generate images with high visualquality.
Revised: Over the past few years, generative models have gained the ability to generate images with high visual quality.

Original: A few of the most widely used methods include generative adversarial networks (GANs) [18],
Revised: A few of the most widely used methods include generative adversarial networks (GANs) [18],

Original: VAE [38], invertible neural networks [11, 12], and diffusion models [24, 69].
Revised: VAE [38], invertible neural networks [11, 12] [Table] and diffusion models [24, 69].

Original: In particular, generativemodels trained on large amounts of unlabeled data, e.g., BigGAN [4], StyleGANs [34, 35, 33, 62],
Revised: In particular, generative models trained on large amounts of unlabeled data, e [Table] g., BigGAN [4], StyleGANs [34, 35, 33, 62],

Original: Glow [37], and diffusion models [49, 10], achieve promising image synthesis results.
Revised: Glow [37] [Table] and diffusion models [49, 10], achieve promising image synthesis results.

Original: Despite their success, controllability and de-biasing are still two fundamental challenges generativemodels face.
Revised: Despite their success, controllability and de-biasing are still two fundamental challenges generative models face.

Original: For controllability, existing methods include explicit conditioning at training [41, 67]  or local editing of the learned representation, e.g, “style” codes [1, 54, 44, 66, 79].
Revised: For controllability, existing methods include explicit conditioning at training [41, 67] [Figure] PromptGen views the composition G ◦ f θ as a new “generative model” for iterative control. or local editing of the learned representation, e.g, “style” codes [1, 54, 44, 66, 78].

Original: For de-biasing,existing methods include local editing of “style” codes [58, 30] and importance sampling for eithertraining [20] or inference [28].
Revised: For de-biasing, existing methods include local editing of “style” codes [58, 30] and importance sampling for either training [20] or inference [28].

Original: Existing works study these problems separately, each requiring aspeciﬁc design for the task studied.
Revised: Existing works study these problems separately, each requiring a speciﬁc design for the task studied.

Original: On the contrary, PromptGen is a uniﬁed framework for arbitrarycontrols deﬁned by off-the-shelf models.
Revised: On the contrary, PromptGen is a uniﬁed framework for arbitrary controls deﬁned by off-the-shelf models.

Original: The beneﬁts of unifying tasks have been shown by a recenttrend of works in multiple research areas across vision and language [57, 46, 61, 80, 59].
Revised: The beneﬁts of unifying tasks have been shown by a recent trend of works in multiple research areas across vision and language [57, 46, 61, 79, 59].

Original: Moreover,one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen:since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuningcan be applied before or after PromptGen training.
Revised: Moreover, one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen: since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuning can be applied before or after PromptGen training.

Original: We leave this exploration to future studies.
Revised: We leave this exploration to future studies.

Original: Previous methods usually sample from EBM [39] with Markov Chain Monte Carlo (MCMC) [71, 75,14, 19, 13, 43].
Revised: Previous methods usually sample from EBM [39] with Markov Chain Monte Carlo (MCMC) [71, 74, 14, 19, 13, 43].

Original: Among them, plug-and-play generative models (PPGMs) [47] and LACE [50] deﬁnelatent-space EBMs.
Revised: Among them, plug-and-play generative models (PPGMs) [47] and LACE [50] deﬁne latent-space EBMs.

Original: However, MCMC requires inference-time optimization, which is inefﬁcient andrequires the off-the-shelf models to be available at inference; this is also the criticism for diffusionmodels, regardless of being gradient-guided [49, 65] or not [24, 25, 48].
Revised: However, MCMC requires inference-time optimization, which is inefﬁcient and requires the off-the-shelf models to be available at inference; this is also the criticism for diffusion models, regardless of being gradient-guided [49, 65] or not [24, 25, 48].

Original: In contrast, PromptGenachieves efﬁcient, feed-forward sampling.
Revised: In contrast, PromptGen achieves efﬁcient, feed-forward sampling.

Original: Table 1 shows a comparison with previous methods.
Revised: Table 1 shows a comparison with previous methods.

Original: Our
Revised: Our

Original: INN training is similar to that proposed by [52] to sample from physical systems, which is later usedby [77] to solve inverse problems with two composed INNs.
Revised: INN training is similar to that proposed by [52] to sample from physical systems, which is later used by [76] to solve inverse problems with two composed INNs.

Original: Notably, [52] and [77] do not leverage alow-dimensional latent space, and [77] requires training a separate model for each image sample.
Revised: Notably, [52] and [76] do not leverage a low-dimensional latent space, and [76] requires training a separate model for each image sample.

Original: Inthis paper, we use INN to model arbitrary EBMs for various generative models.
Revised: In this paper, we use INN to model arbitrary EBMs for various generative models.

Original:  Method
Revised:  Method

Original: Figure 2 illustrates our PromptGen framework.
Revised: Figure 2 illustrates our PromptGen framework.

Original: To begin, the user selects a pre-trained generativemodel G .
Revised: To begin, the user selects a pre-trained generative model G .

Original: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .
Revised: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .

Original: We train an INN f ✓ to approximate p ( z |C ) .
Revised: We train an INN f θ to approximate p ( z |C ) .

Original: PromptGen views the functional composition G  f ✓ as anew generative model and can perform iterative control.
Revised: PromptGen views the functional composition G ◦ f θ as a new generative model and can perform iterative control.

Original: Algorithm 1 describes the overall procedure.Algorithm
Revised: Algorithm 1 describes the overall procedure.

Original: Input: control C of the current iteration 2.
Revised: Input: control C of the current iteration 2.

Original: Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3.
Revised: Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3.

Original: Train an INN f ✓ : R d ! R d to approximate p ( z |C ) (Section 3.2) 4.
Revised: Train an INN f θ : R d → R d to approximate p ( z |C ) (Section 3.2) 4.

Original: G  G  f ✓
Revised: G ← G ◦ f θ

Original: The plug-and-play generative model [47] was ﬁrst proposed to use a latent-space EBM for controllableimage synthesis.
Revised: The plug-and-play generative model [47] was ﬁrst proposed to use a latent-space EBM for controllable image synthesis.

Original: If a ﬁxed generative model is used, then theoretically, any image-space EBM can beviewed as a latent-space EBM, as shown by [19] and [50].
Revised: If a ﬁxed generative model is used, then theoretically, any image-space EBM can be viewed as a latent-space EBM, as shown by [19] and [50].

Original: We deﬁne the latent-space EBM followingsimilar formulation as [47, 19, 50], with some new energy functions.
Revised: We deﬁne the latent-space EBM following similar formulation as [47, 19, 50], with some new energy functions.

Original: We then extend the formulationto incorporate the moment constraint [7], which was adopted for language modeling [36], but unlike[36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.
Revised: We then extend the formulation to incorporate the moment constraint [7], which was adopted for language modeling [36], but unlike [36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.

Original: We deﬁne a control C as M independent properties { y 1 , . . .
Revised: We deﬁne a control C as M independent properties { y 1 , . . .

Original: , y M } , e.g., y 1 can be a text descriptionand y 2 can be an attribute.
Revised: , y M } , e.g., y 1 can be a text description and y 2 can be an attribute.

Original: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)
Revised: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)

Original: [Equation]
Revised: [Equation]

Original: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy arepreferred.
Revised: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy are preferred.

Original: Using a pre-trained generative model G : R d ! X that maps a latent code z to an imagex , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping itto x = G ( z ) .
Revised: Using a pre-trained generative model G : R d → X that maps a latent code z to an image x , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping it to x = G ( z ) .

Original: Appendix B.2 shows that this EBM is equivalent to the latent-space EBMp
Revised: Appendix B.2 shows that this EBM is equivalent to the latent-space EBM p ( z |

Original: [Equation]
Revised: [Equation]

Original: |
Revised: = p

Original: [Equation]
Revised: [Equation]

Original: Latent-space EBM allows us to use any off-the-shelf model to specify the control.
Revised: Latent-space EBM allows us to use any off-the-shelf model to specify the control.

Original: The following aresome examples that are discussed in this paper (explained in Appendix B.1):
Revised: The following are some examples that are discussed in this paper (explained in Appendix B.1):

Original: Classiﬁer energy:
Revised: Classiﬁer energy:

Original: Given a

Original: classiﬁer

Original:  P ( ·| x ) and the target class a that we want to sample imagesfrom, we deﬁne the classiﬁer energy as E classiﬁer 
Revised: Given a classiﬁer P ( ·| x ) and the target class a that we want to sample images from, we deﬁne the classiﬁer energy as E classiﬁer ( x , a ) =

Original: x , a ) =  log P ( a | x ) .
Revised: − log P ( a | x ) .

Original: CLIP energy: Using the CLIP model [56], we deﬁne the CLIP energy as the cosine distance betweenthe embeddings of the image and the text t , averaged over L differentiable augmentations [83, 44]:
Revised: CLIP energy: Using the CLIP model [56], we deﬁne the CLIP energy as the cosine distance between the embeddings of the image and the text t , averaged over L differentiable augmentations [82, 44]:

Original: [Equation]
Revised: [Equation]

Original: Inverse graphics energy:
Revised: Inverse graphics energy:

Original: Given an inverse graphics model, f X ! P , which infers image parameters105 (e.g., pose and expression), and the target parameters ⇢ , we deﬁne the inverse graphics energy as
Revised: Given an inverse graphics model, f X→P , which infers image parameters (e.g., pose and expression), and the target parameters ρ , we deﬁne the inverse graphics energy as

Original: [Equation]
Revised: [Equation]

Original: where d h · , · i is the geodesic distance between the inferred parameters and the target parameters.
Revised: where d (cid:104)· , ·(cid:105) is the geodesic distance between the inferred parameters and the target parameters.

Original: Moment constraint: Some controls cannot be directly deﬁned by off-the-shelf models, and themoment constraint [7, 36] is one of them.
Revised: Some controls cannot be directly deﬁned by off-the-shelf models, and the moment constraint [7, 36] is one of them.

Original: Given a mapping  : X !
Revised: Given a mapping γ : X → R K

Original: R K (e.g.,  can be a classiﬁerthat outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as
Revised: (e.g., γ can be a classiﬁer that outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as

Original: [Equation]
Revised: [Equation]

Original: where µ is the user-speciﬁed vector.
Revised: where µ is the user-speciﬁed vector.

Original: For example, if we want to generate images that are uniformlydistributed across races, we may use a race classiﬁer as  and deﬁne µ =  |A|  1 , . .
Revised: For example, if we want to generate images that are uniformly distributed across races, we may use a race classiﬁer as γ and deﬁne µ = (cid:0) |A| − 1 , . .

Original:  , |A|  where
Revised: . , |A| − 1 (cid:1) where

Original: A is the set of races.
Revised: A is the set of races.

Original: In this paper, we generalize the moment constraint to the latent space, andapproximate the above objective as (detailed in Appendix B.6):
Revised: In this paper, we generalize the moment constraint to the latent space, and approximate the above objective as (detailed in Appendix B.6):

Original: [Equation]
Revised: [Equation]

Original: 3.2 Approximating EBM with Invertible Neural Network
Revised: 3.2 Approximating EBM with Invertible Neural Network

Original: Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcientsampling network.
Revised: Given the functional form of the EBM p ( z |C ) [Equation] our next step is to approximate it with an efﬁcient sampling network.

Original: To achieve this, we train a distribution p ✓ ( z ) that minimizes the KL divergence 
Revised: To achieve this [Equation] we train a distribution p θ ( z ) that minimizes the KL divergence D

Original: D KL ( p ✓ ( z ) k p ( z |C )) .
Revised: KL ( p θ ( z ) (cid:107) p ( z |C )) .

Original: Estimating D KL ( p ✓ ( z ) k p ( z |C )) requires easily sampling z ⇠ p ✓ and tractably118 computing p ✓ ( z ) .
Revised: Estimating D KL ( p θ ( z ) (cid:107) p ( z |C )) requires easily sampling z ∼ p θ and tractably computing p θ ( z ) .

Original: Inspired by [52], we model p ✓ with an INN f ✓ that deﬁnes a bijection z
Revised: Inspired by [52], we model p θ with an INN f θ that deﬁnes a bijection z

Original: = f ✓ ( ✏ ) ,which has two merits besides the invertibility: (1) one can easily sample z by sampling ✏ ⇠ N ( 0 , I )and mapping it to z = f ✓ ( ✏ ) , and (2) p ✓ ( z ) has a closed-form solution:
Revised: = f θ ( (cid:15) ) , which has two merits besides the invertibility: (1) one can easily sample z by sampling (cid:15) ∼ N ( 0 , I ) and mapping it to z = f θ ( (cid:15) ) , and (2) p θ ( z ) has a closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: while not converged 
Revised: while not converged do

Original: Sample ✏ ⇠ N ( 0 , I ) 2.
Revised: Sample (cid:15) ∼ N ( 0 , I ) 2.

Original: Map ✏ to latent code z = f ✓ ( ✏ ) 3.
Revised: Map (cid:15) to latent code z = f θ ( (cid:15) ) 3.

Original: Map z to an image x = G ( z ) 4.
Revised: Map z to an image x = G ( z ) 4.

Original: Optimize ✓ with gradient r ✓
Revised: Optimize θ with gradient ∇ θ (cid:16) − log | det( ∂f θ ∂ (cid:15) ) | − log p z ( z ) + E C ( x ) (cid:17) Based on these properties of INN

Original: [Equation]
Revised: [Equation]

Original: ⌘ Based on these properties of INN, we can rewrite our KL divergence objective D KL ( p ✓ ( z ) k p ( z |C ))as (full derivations in Appendix B.3)
Revised: we can rewrite our KL divergence objective D KL ( p θ ( z ) (cid:107) p ( z |C )) as (full derivations in Appendix B.3)

Original: the following form:
Revised: the following form:

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Since H N ( 0 , I ) and log
Revised: Since H N ( 0 , I ) and log

Original: Z are independent of ✓ , our training objective becomes
Revised: Z are independent of θ , our training objective becomes

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.
Revised: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.

Original: PromptGen for generative models with a class-embedding space (Figure 5(e))
Revised: PromptGen in a class-embedding space (Figure 5(e))

Original: Previous works[4, 62] have shown that class conditioning boosts generative models’ performances ondiverse datasuch as ImageNet [60].
Revised: Previous works [4, 62] have shown that class conditioning boosts generative models’ performances on ImageNet [60].

Original: Speciﬁcally, class-conditioned generative models map a latent code z and aclass embedding y to x = G ( z , y ) .
Revised: Speciﬁcally, classconditioned generative models map a latent code z and a class embedding y to x = G ( z , y ) .

Original: To extend PromptGen to these models, we train an INN h ✓ tomap ⇠ ⇠ N ( µ ,  2 I ) to y =
Revised: To extend PromptGen to these models, we train an INN h θ to map ξ ∼ N ( µ , σ 2 I ) to y =

Original: h ✓ ( ⇠ ) , where µ and  are the mean and standard deviation of G ’s classembeddings.
Revised: h θ ( ξ ) , where µ and σ are the mean and standard deviation of G ’s class embeddings.

Original: The motivation for deﬁning the distribution of ⇠ as N ( µ ,  2 I ) but not N ( 0 , I ) is thatwe want the learned INN h ✓ to be volume-preserving, which is easier to train.
Revised: The motivation for deﬁning the distribution of ξ as N ( µ , σ 2 I ) but not N ( 0 , I ) is that we want the learned INN h θ to be volumepreserving, which is easier to train.

Original: The training objectiveis to minimize D KL ( p ✓ ( z , y ) k p ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4) 
Revised: The training objective is to minimize D KL ( p θ ( z , y ) (cid:107) p ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4) arg

Original: [Equation]
Revised: [Equation]

Original:  where p y ( y ) =
Revised: (11) where p y ( y ) =

Original: N ( y | µ ,  2 I ) is the estimated class-embedding distribution.
Revised: N ( y | µ , σ 2 I ) is the estimated class-embedding distribution.

Original: PromptGen with conditional INN To generalize the control to continuous values, e.g., the sceneparameters ⇢ in Eq.
Revised: PromptGen with conditional INN To generalize the control to continuous values, e.g., the scene parameters ρ in Eq.

Original: (4), we condition the INN on ⇢ .
Revised: (4), we condition the INN on ρ .

Original: The condition is modeled by replacing z = f ✓ ( ✏ )with z = f ✓ ( ✏ , ⇢ ) .
Revised: The condition is modeled by replacing z = f θ ( (cid:15) ) with z = f θ ( (cid:15) , ρ ) .

Original: Space limited, we provide details and derivations in Appendix B.5.
Revised: Space limited, we provide details and derivations in Appendix B.5.

Original: This extensionresults in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label 
Revised: This extension results in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label pairs and is only applicable to explicit condition, which is not capable of modeling EBMs.

Original: [Figure] [Figure]
Revised: When the condition is modeled by the equivariant operations proposed by [77], PromptGen also satisﬁes the homomorphism property [77] in the latent space.

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: Figure 5(e) is the extension to the embedding space of BigGAN [4] on

Original:  Experiments
Revised:  Experiments

Original: This section describes the experimental validation of PromptGen.
Revised: This section describes the experimental validation of PromptGen.

Original: See Appendix A for experimentson synthetic data and Appendix C, D, E and F for additional experiments on images and 3D meshes.
Revised: See Appendix A for experiments on synthetic data and Appendix C, D, E and F for additional experiments on images and 3D meshes.

Original: Additional experimental details are provided in Appendix B.8.
Revised: Additional experimental details are provided in Appendix B.8.

Original: 4.1 Image Synthesis based on Text Description
Revised: 4.1 Image Synthesis based on Text Description

Original: This experiment illustrates the capability of PromptGen to sample images from a generative modeldriven by a text description t using the pre-trained CLIP model [56] (the ViT-B/32 version).
Revised: This experiment illustrates the capability of PromptGen to sample images from a generative model driven by a text description t using the pre-trained CLIP model [56] (the ViT-B/32 version).

Original: We usedthe CLIP energy from Eq.
Revised: We used the CLIP energy from Eq.

Original: (3), with text descriptions such as a photo of a baby.
Revised: (3), with text descriptions such as a photo of a baby.

Original: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generationmethods, StyleCLIP [54] and StyleGAN2-NADA [17].
Revised: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generation methods, StyleCLIP [54] and StyleGAN2-NADA [17].

Original: We observe that PromptGen generates diverseand high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and
Revised: We observe that PromptGen generates diverse and high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and

Original: StyleGAN2-NADA generates baby-like adults.
Revised: StyleGAN2-NADA generates baby-like adults.

Original: These results show that (1) locally editing the latentcode (i.e., StyleCLIP) is not always as effective method for controlling generative models (e.g., not allimages’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA)is not effective in seeking modes in a generative model.
Revised: These results show that (1) locally editing the latent code (i.e., StyleCLIP) is not always an effective method for controlling generative models (e.g., not all images’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA) is not effective in seeking modes in a generative model.

Original: Figure 5(a), Figure 5(b), and Figure 5(d) showhow PromptGen applies to StyleNeRF [21], NVAE [74], and diffusion autoencoder (DiffAE; [55]).
Revised: Figure 5(a), Figure 5(b), and Figure 5(d) show how PromptGen applies to StyleNeRF [21], NVAE [73], and diffusion autoencoder (DiffAE; [55]).

Original: Although these generative models are all trained on FFHQ, baby images sampled from them havedistinct characteristics.
Revised: Although these generative models are all trained on FFHQ, baby images sampled from them have distinct characteristics.

Original: Figure 5(c) shows the ability of PromptGen to generate cats of a particularspecies.
Revised: Figure 5(c) shows the ability of PromptGen to generate cats of a particular species.

Original: Figure 5(e) shows the extension (Section 3.2) of our PromptGen to the class-embedding spaceof BigGAN [4], a class-conditional GAN trained on ImageNet [60].
Revised: Figure 5(e) shows the extension (Section 3.2) of our PromptGen to the class-embedding space of BigGAN [4], a class-conditional GAN trained on ImageNet [60].

Original: We observe that PromptGenhelps BigGAN generate images with complex text descriptions, which are out of BigGAN’s trainingimage distribution; however, the diversity seem to be limited in this extension.
Revised: We observe that PromptGen helps BigGAN generate images with complex text descriptions, which are out of BigGAN’s training image distribution; however, the diversity seems to be limited in this extension.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  r o p o r t i o n
Revised: P r o p o r t i o n

Original: Race
Revised: Race

Original: WhiteBlackLatinoHispanicEastAsian
Revised: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern
Revised: SoutheastAsianIndianMiddleEastern

Original: [Figure]
Revised: [Figure]

Original: t
Revised: o n

Original: [Figure]
Revised: [Figure]

Original:  n
Revised: P r o p o r t i o n

Original: Race
Revised: Race

Original: WhiteBlackLatinoHispanicEastAsian
Revised: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern
Revised: SoutheastAsianIndianMiddleEastern

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Table] [Table] StyleGAN2 [35] [Table]
Revised: FFHQ

Original: [Table]
Revised: [Figure]

Original: StyleFlow [1] [Table] [Table] FairGen [70]
Revised: binary

Original: [Table] [Table]
Revised: [Figure] [Table]

Original: FairStyle [30] [Table] [Table]
Revised: ↓

Original: PromptGen [Table] ours)
Revised: FFHQ (real data) 0.

Original: [Table]
Revised: [Table]

Original: 4.2 De-Biasing Pre-Trained Generative Models
Revised: 4.2 De-Biasing Pre-Trained Generative Models

Original: An important problem in generative models is to generate fair distributions w.r.t a set of attributes ofinterest.
Revised: An important problem in generative models is to generate fair distributions w.r.t a set of attributes of interest.

Original: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races andages.
Revised: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races and ages.

Original: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] interms of categorical attributes, using the moment constraint deﬁned in Eq.
Revised: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] in terms of categorical attributes, using the moment constraint deﬁned in Eq.

Original: (5) and Eq.
Revised: (5) and Eq.

Original: As control,we used a classiﬁer trained on FairFace 224 2 [31] as  .
Revised: As control, we used a classiﬁer trained on FairFace 224 2 [31] as γ .

Original: We deﬁned µ =  |A|  1 , . ..
Revised: We deﬁned µ = (cid:0) |A| − 1 , . .

Original:  , |A|  , where
Revised: . , |A| − 1 (cid:1) , where

Original: A is the set of races.
Revised: A is the set of races.

Original: Similar to the energy weights  i deﬁned in Eq.
Revised: Similar to the energy weights λ i deﬁned in Eq.

Original: (2), we propose to rescale thetrained ˆ  as  ˆ  .
Revised: (2), we propose to rescale the trained ˆ β as λ ˆ β .

Original: Figure 6 shows that PromptGen de-biases the race and age effectively.
Revised: Figure 6 shows that PromptGen de-biases the race and age effectively.

Original: Existing de-biasing baselines consider binary attributes [30].
Revised: Existing de-biasing baselines consider binary attributes [30].

Original: For a fair comparison with them, weadopted their setting to use binary classiﬁers trained on CelebA [45] for de-biasing and evaluation.
Revised: For a fair comparison with them, we adopted their setting to use binary classiﬁers trained on CelebA [45] for de-biasing and evaluation.

Original: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, wedid not use the moment constraint for this experiment.
Revised: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, we did not use the moment constraint for this experiment.

Original: Instead, since PromptGen allows conditionalimage generation with the classiﬁer energy, we generated the same number of samples conditioned oneach attribute or attribute combination.
Revised: Instead, since PromptGen allows conditional image generation with the classiﬁer energy, we generated the same number of samples conditioned on each attribute or attribute combination.

Original: Table 2 shows that PromptGen has competitive performanceon de-biasing for attributes and attribute combinations.
Revised: Table 2 shows that PromptGen has competitive performance on de-biasing for attributes and attribute combinations.

Original: 4.3 Pose-Guided Face Synthesis
Revised: 4.3 Pose-Guided Face Synthesis

Original: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.
Revised: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.

Original: We used the DECA model [16], which infers the parameters of FLAME [42], a parametric facialgraphics model.
Revised: We used the DECA model [16], which infers the parameters of FLAME [42], a parametric facial graphics model.

Original: We set ⇢ 2 SO (3) as FLAME’s three neck poses and used the conditional INNextension introduced in Section 3.2.
Revised: We set ρ ∈ SO (3) as FLAME’s three neck poses and used the conditional INN extension introduced in Section 3.2.

Original: To enable generating different poses of the same identity (ID),we proposean additional ID energy using the IR-SE50 model [8].
Revised: To enable generating different poses of the same identity (ID), we propose ID energy using the IR-SE50 model [8].

Original: Speciﬁcally, given a canonicalpose ⇢ 0 , we deﬁne z [-
-] =
Revised: Speciﬁcally, given a canonical pose ρ 0 , we deﬁne z [-
-]0 =

Original: f ✓ ( ✏ , ⇢ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)
Revised: f θ ( (cid:15) , ρ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)

Original: [Equation]
Revised: [Equation]

Original: where R is the IR-SE50 model [8] that computes face embeddings.
Revised: where R is the IR-SE50 model [8] that computes face embeddings.

Original: Figure 7 shows that PromptGengenerates faces of the same ID in different poses, even without being explicitly trained with poses ascondition.
Revised: Figure 7 shows that PromptGen generates faces of the same ID in different poses, even without being explicitly trained with poses as conditions.

Original: We reported the FID score [23] of each model, computed with Clean-FID [53]; following controls the pose of StyleGAN preserving the identity All images are resized for visualization
Revised: We computed the FID score [23] of each model using Clean-FID [53]; following [35],

Original: [Table]
Revised: [Table]

Original: [35], we did not use the truncation trick when computing the FID score.
Revised: we did not use the truncation trick when computing the FID score.

Original: Table 3 shows that PromptGenoutperforms existing models in terms of the FID score.
Revised: Table 3 shows that PromptGen outperforms existing models in terms of the FID score.

Original: Following [67], we then reported the average
Revised: Following [67] [Table] we then reported the average

Original: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.
Revised: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.

Original: Results showthat PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.
Revised: Results show that PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.

Original: 4.4 Decomposing Complex Control via Energy Composition
Revised: 4.4 Decomposing Complex Control via Energy Composition

Original: Energy composition E [Equation] = P
Revised: Energy composition E C ( x ) = (cid:80) Mi

Original: Mi

Original: [Equation]  [Equation] E i ( x , y i ) allows us to decompose controls into simple ones.
Revised: =1 λ i E i ( x , y i ) allows us to decompose controls into simple ones.

Original: We show that y 

Revised: We show that y

Original: = photo of a bald black man with beard (  1 = 6000 ) is not successful (Figure [-8(a));by decomposing it as y 1 = photo of a bald man (  1 = 1500 ), y 
-]
Revised: = photo of a bald black man with beard ( λ 1 = 6000 ) is not successful (Figure [-8(a));by decomposing it as y 1 = photo of a bald man (  1 = 1500 ), y 
-]8(a))

Original:  = photo of a black man (  1 =3000 ), and y [-
-] = photo of a man with beard (  1 = 1500 ), we have better control (Figure 8(b)).
Revised: ; by decomposing it as y 1 = photo of a bald man ( λ 1 = 1500 ), y 2 = photo of a black man ( λ 1 = 3000 ), and y [-
-]3 = photo of a man with beard ( λ 1 = 1500 ), we have better control (Figure 8(b)).

Original: [Figure]
Revised: [Figure]

Original: 4.5 Iterative Distributional Control via Functional Composition
Revised: 4.5 Iterative Distributional Control via Functional Composition

Original: This section discusses an interesting bias that PromptGen reveals about the CLIP model.
Revised: This section discusses an interesting bias that PromptGen reveals about the CLIP model.

Original: Figure 9(a)shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males.
Revised: Figure 9(a) shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males.

Original: Thisbias should not be attributed to image pre-training data since images contain a bias in the oppositedirection, i.e., men are less likely to have makeup.
Revised: This bias should not be attributed to image pre-training data since images contain a bias in the opposite direction, i.e., men are less likely to have makeup.

Original: We argue that this bias should be explained by
Revised: We argue that this bias should be explained by

Original: CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likelyto say “a person without makeup” when the person is a female (detailed analysis in Appendix F).
Revised: CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likely to say “a person without makeup” when the person is a female (detailed analysis in Appendix F).

Original: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterativecontrol, enabled by the functional composition in Algorithm 1.
Revised: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterative control, enabled by the functional composition in Algorithm 1.

Original: Speciﬁcally, in the second iteration,we de-biased the gender distribution of G  f ✓ instead of G , where f ✓ is the INN learned for the textcontrol.
Revised: Speciﬁcally, in the second iteration, we de-biased the gender distribution of G ◦ f θ instead of G , where f θ is the INN learned for the text control. For de-biasing, we used the moment constraint with ˆ β trained for G ◦ f θ . Figure 9(b) shows that females and males are uniformly distributed after the moment constraint in the second iteration. [Figure]

Revised: PromptGen

Revised: Besides revealing this “reporting bias”, PromptGen can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution.

Revised: All images are resized for visualization.

Original: For de-biasing, we used  the moment constraint with ˆ  trained for G  f ✓ .
Revised: We compared our PromptGen and the plug-and-play generative model (PPGM) [47] in terms of the inference latency. PPGM uses Langevin dynamics [74] to optimize over the latent space at inference, while PromptGen samples images in a feed-forward manner. In this experiment, we used PPGM and our PromptGen to approximate the same EBM with CLIP energy.

Original: Figure 9(b) showsthat females and males are uniformly distributed after the moment constraint in the second iteration.
Revised: Inference times were estimated on an NVIDIA RTX A4000 GPU. Table 4 shows that our PromptGen has the highest performance and efﬁciency. Compared to our PromptGen, PPGM requires 100 × inference time to achieve comparable results. Moreover, learning f θ in PromptGen requires training-time optimization, and this amortized optimization is useful when one wants to reuse a controlled distribution many times.

Original: [Figure]
Revised: [Table]

Revised:  Conclusions and Future Work

Original:  PromptGen generates more images male, showing
Revised: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributional control of pre-trained generative models. PromptGen leverages the knowledge of various off-the-shelf models and, unlike previous methods, it does not require the availability of these models at inference. PromptGen can sample images in a feed-forward manner, which is more efﬁcient than methods that require optimization at inference. PromptGen offers the generality for algorithmic design and modularity for control composition, and it also enables iterative controls.

Revised: Experiments validate that

Revised: PromptGen applies to various generative models (StyleGAN2, StyleNeRF, NVAE), control types (continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inverse graphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).

Revised: Limitation and future work:

Revised: We provide an error analysis in Appendix F and a discussion on societal impact in Appendix G. PromptGen is restricted by the pre-trained generative model’s coverage [3, 27].

Original: Besides revealing this “reporting bias”, PromptGen  can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution.
Revised: PromptGen focuses on mode-seeking and -reweighting instead of domain adaptation, and it is possible to combine PromptGen and domain adaptation of generative models [17]. Also, PromptGen depends on the off-the-shelf models that provide knowledge about the control (Appendix F). In subsequent work, it can be beneﬁcial to explore how to learn energy functions [15, 26] (besides our moment constraint used for de-biasing), which may provide ﬁne-grained control with less bias. Acknowledgments and Disclosure of Funding

Revised: The authors would like to thank the anonymous reviewers, Zoltán Ádám Milacski, Jianchun Chen, and Shubhra Aich for their valuable feedback on drafts of this paper.

Revised: A PromptGen on Synthetic Data

Revised: We demonstrate the behavior of our PromptGen with two-dimensional synthetic data, using GAN as the generative model.

Revised: This synthetic experiment is illustrated in Figure 10.

Revised: Speciﬁcally, we created a “real” distribution that is a Gaussian mixture distribution biased towards one mode.

Revised: We derive a closed-form fair classiﬁer based on this distribution, detailed in the purple block in Figure 10.

Revised: When a GAN is trained to approximate this distribution, its outputs are biased towards the over-represented mode, as shown by Figure 10(b).

Revised: Figure 10(c) illustrates the controllability experiment: using the fair classiﬁer as control, PromptGen learns a distribution concentrated in a speciﬁc region of the output space.

Revised: Figure 10(d) illustrates the de-biasing experiment: using the moment constraint (Section 3.

Revised: and Section 4.2), PromptGen upweights the under-represented regions of the output space.

Revised: B Method Details and Derivations B.1 Controllability as EBM

Revised: In Section 3, we deﬁned a control C as M independent properties { y 1 , . . .

Revised: , y M } .

Revised: For example, y 1 can be a text description, and y 2 can be an attribute.

Revised: In this part, we ﬁrst elaborate on why controllability can be formed as EBMs in Eq.

Revised: We then provide concrete examples of the distributions derived from different energy functions deﬁned in Section 3.1.

Revised: We denote the image prior as p x ( x ) , the only distribution that can be estimated from data when labels for the control are not provided during generative model pre-training.

Revised: Given the control

Revised: C = { y 1 , . . .

Revised: , y M } , we resort to Bayes’ theorem to rewrite the conditional distribution p ( x |C ) as

Revised: [Equation]

Revised: For Eq.

Revised: (15) to be well-deﬁned, we need to deﬁne p ( y i | x ) for each y i .

Revised: In order to incorporate the knowledge of arbitrary off-the-self models (besides image classiﬁers), we deﬁne each p ( y i | x ) as

Revised: [Equation]

Revised: [Equation]

Revised: Eq.

Revised: (16) says that p ( y i | x ) is proportional to the exponential of an energy function

Revised: E i ( x , y i ) , where y i with lower energy has higher density or mass.

Revised: Note that Eq.

Revised: (16) deﬁnes a distribution over all possible values of y i instead of all possible values of x .

Revised: Combining Eq.

Revised: (15) and Eq.

Revised: (16), we have p

Revised: [Equation]

Revised: which is the same equation as Eq.

Revised: In the following, we use Eq.

Revised: (16) to derive the distributions from the classiﬁer energy, CLIP energy, and inverse graphics energy deﬁned in Section 3.1.

Revised: Classiﬁer energy:

Revised: Given a classiﬁer P ( ·| x ) and the target class a , we deﬁne the classiﬁer energy as

Revised: E classiﬁer ( x , a ) = − log P ( a | x ) .

Revised: Using Eq.

Revised: (16), we arrive at:

Revised: [Equation]

Revised: which is equivalent to a temperature-adjusted distribution of the original classiﬁer.

Revised: CLIP energy: Using Eq.

Revised: (16), the CLIP energy in Eq.

Revised: (3) is equivalent to

Revised: [Equation]

Revised: Inverse graphics energy: Using Eq.

Revised: (16), the inverse graphics energy in Eq.

Revised: (4) is equivalent to

Revised: [Equation]

Revised: If the geodesic distance d (cid:10) · , · (cid:11) is the Euclidean distance, then p inv-graphics ( ρ | x ) is a Gaussian distribution whose mean is ρ and whose variance depends on the hyperparameter λ inv-graphics ; if the geodesic distance d (cid:10) · , · (cid:11) is the spherical distance (e.g., the distance between pose parameters deﬁned on a unit sphere), then p inv-graphics ( ρ | x ) is a vMF distribution.

Revised: B.2 Equivalence between Image-Space EBM and Latent-Space EBM

Revised: Proposition 1.

Revised: Deﬁne x ∼ p x ( x ) as z ∼ p

Revised: z

Revised: ( z ) , x = G ( z ) and p ( x |C ) as z ∼ p ( z |C ) , x = G ( z ) , where p ( z |C ) is deﬁned as the following EBM:

Revised: [Equation]

Revised: We have

Revised: [Equation]

Revised: In spirit, our proof follows the proof in [50], which follows [19].

Revised: The difference between our proof and that in [50] is that we derive p ( x |C ) from p ( z |C ) while they derived p ( z |C ) from p ( x |C ) .

Revised: Proof.

Revised: Based on Lemma 1 in [19] and Lemma 1 in [50], p ( z |C ) is equivalent to rejection sampling with proposal distribution p z ( z ) and acceptance probability

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: [Equation]

Revised: Since p ( x |C ) is deﬁned as z ∼ p

Revised: z ( z |C ) , x = G ( z ) , p ( x |C ) is equivalent to rejection sampling with proposal distribution p z ( z ) , x = G ( z ) and acceptance probability

Revised: [Equation]

Revised: [Equation]

Revised: Note that the above proposal distribution

Revised: z ∼ p

Revised: z

Revised: ( z ) , x = G ( z ) is the same as p x ( x ) by deﬁnition.

Revised: Based on Lemma 1 in [19] and Lemma 1 in [50], we arrive at:

Revised: [Equation]

Revised: B.3 Derivation of Eq. (9): Approximating EBM with INN

Revised: The full derivation of Eq.

Revised: (9) is given by:

Revised: [Equation]

Revised: B.4 Derivation of Eq. (11): Extension to Generative Models with a Class-Embedding Space

Revised: The full derivation of Eq.

Revised: (11) is given by:

Revised: [Equation]

Revised: y are independent

Revised: [Equation]

Revised: , z

Revised: [Equation]

Revised: ) , y

Revised: [Equation]

Revised: h

Revised: [Equation]

Revised: , x

Revised: [Equation]

Revised: G ( z , y )

Revised: [Equation]

Revised: − log | det

Revised: [Equation]

Revised: ∂f θ ∂

Revised: [Equation]

Revised: ) | − log | det

Revised: [Equation]

Revised: ∂h θ

Revised: [Equation]

Revised: ) |

Revised: [Equation]

Revised: while not converged do

Revised: Sample (cid:15) ∼ N ( 0 , I ) , ρ ∼ p ρ

Revised: [Equation]

Original: All images are resized for visualization.
Revised: ρ [Equation] Map (cid:15) to latent code z = f θ ( (cid:15) , ρ ) 3. Map z to an image x = G ( z ) 4.

Original: 4.6 Inference Latency
Revised: Optimize θ with gradient [Equation] B.5 Extension to Conditional INN Algorithm 3 illustrates how we extend the training algorithm

Revised: (Algorithm 2) to conditional INN, which helps generalize to controls speciﬁed by continuous values ρ .

Revised: The training objective becomes

Revised: [Equation]

Revised: where C ρ means that the control is speciﬁed by value ρ .

Revised: The derivation of Algorithm 3 is given by:

Revised: [Equation]

Revised: B.6 Derivation for Latent-Space Moment Constraint

Original: We compared our PromptGen and the plug-and-play generative model (PPGM) [47] in terms of theinference latency.
Revised: Given a mapping γ : X → R K , moment constraint [7, 36] deﬁnes the target distribution p ( x |C ) as [Equation] [Table] [Equation]

Revised: where µ is the user-speciﬁed constraint.

Revised: Examples are provided in the main text

Revised: [Equation]

Revised: and we omit them here for brevity.

Revised: [7] showed that Eq.

Revised: (33) can be approximated to an arbitrary precision by

Revised: [Equation]

Revised: (34) where ˆ β needs to be computed.

Revised: [36] estimates ˆ β by solving the following regression problem:

Original: PPGM uses Langevin dynamics [75] to optimize over the latent space at inference,while PromptGen samples images in a feed-forward manner.
Revised: [Equation] In [36], sampling x ∼ p x ( x ) is straightforward since they focus on autoregressive language models. In the context of latent-variable vision generative models, where x ∼ p x ( x ) is implicitly deﬁned as z ∼ p z ( z ) , x = G ( z ) , Eq.

Revised: (35) is equivalent to

Revised: [Equation]

Revised: arg min

Revised: [Equation]

Revised: [Equation]

Revised: Finally, based on Proposition 1, we can generalize the moment constraint to the latent space as

Revised: [Equation]

Revised: Note that Eq.

Revised: (36) and Eq.

Revised: (37) are verbatim copies of Eq.

Revised: (7) and Eq.

Revised: (6), respectively.

Revised: Algorithm 4:

Revised: 4.3) while not converged do

Revised: Sample (cid:15) ∼ N ( 0 , I ) , ρ ∼ p ρ ( ρ ) 2.

Revised: Map (cid:15) to latent codes z = f θ ( (cid:15) , ρ ) and z 0 = f

Revised: θ ( (cid:15) , ρ 0 ) 3.

Revised: Map z to an image x = G ( z ) and z 0 to an image x 0 =

Revised: G

Revised: ( z 0 ) 4.

Revised: Optimize θ with gradient

Revised: [Equation]

Revised: B.7 Inverse Graphics Control with Identity Energy

Revised: This subsection provides more detail on how we use an inverse graphics model, DECA [16], to control the pose of faces generated by StyleGAN2 trained on FFHQ 1024 2 , as introduced in Section 4.3.

Original: In this experiment, we used PPGM andour PromptGen to approximate the same EBM with CLIP energy.
Revised: Recall that we use the conditional INN extension, detailed in Algorithm 3, for this experiment. To enable generating different poses of the same identity, we add additional identity energy using the IR-SE50 model [8]. The training algorithm is detailed in Algorithm 4.

Original: {+Speciﬁcally, we generate a canonical latent code z 
+}
Revised: {+Speciﬁcally, we generate a canonical latent code z 
+}

Revised: =

Revised: f

Revised: θ ( (cid:15) , ρ 0 ) for each (cid:15) , where ρ 0 is the canonical pose.

Revised: The latent code z and the canonical latent code z 0 are mapped to the image x

Original: Inference times was estimated onan NVIDIA RTX A4000 GPU.
Revised: and the canonical image x 0 . The identity energy in Eq. (12), copied below, encourages the embeddings of the two images to be similar: [Equation]

Original: Table 4 shows that our PromptGen has the highest performance andefﬁciency.
Revised: Our INN architecture contains 8 blocks. Each block consists of a soft permutation of channels [2], an afﬁne coupling layer [12], and an ActNorm layer [37]. In the afﬁne coupling layer, we model the sub-network as an MLP with one hidden layer, where the hidden dimension is 256 and the non-linear activation is LeakyReLU 0 .

Original: Compared to our PromptGen, PPGM requires 100 ⇥ inference time to achieve comparableresults.
Revised: For the prior distribution p ρ ( ρ ) of the pose parameter ρ in the inverse graphics experiments (Section 4.3), we sample the x -axis and y -axis rotations relative to a canonical pose ρ 0 , whose x -axis and y -axis rotations are 0 . 3 and 0 , respectively.

Original: Moreover, learning f ✓ in PromptGen requires training-time optimization, and this amortizedoptimization is useful when one wants to reuse a controlled distribution many times.
Revised: The relative pose’s x -axis and y -axis rotations are uniformly sampled from [ − (1 / 9) π, (1 / 9) π ] . Each experiment was run on an NVIDIA RTX A4000 GPU (with 16G memory). Our code implementation is based on the PyTorch framework.

Original: [Table] [Table]
Revised: Our code can be found at https://github.com/

Original: When is it equal to the EBM?
Revised: ChenWu98/Generative-Visual-Prompt .

Original: [Equation]
Revised: [Figure] [Table]

Original: Conclusions and Future Work
Revised: C Additional Results for Image Synthesis based on Text Description

Original: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributionalcontrol of  pre-trained generative models.
Revised: This section presents additional results of PromptGen for image synthesis from text, using the CLIP model as control. As the pre-trained generative model, we explore StyleGAN2 trained on AFHQ-Cats [6], LSUN-Churches [81], FFHQ [34], and Landscape-HQ [68].

Original: PromptGen leverages the knowledge of various off-the-shelfmodels and, unlike previous methods, it does not require the availability of these models at inference.
Revised: Figure 11 and Figure 12 present more results of PromptGen with diverse text descriptions.

Original: PromptGen can sample images in a feed-forward manner, which is more efﬁcient than methodsthat require optimization at inference. PromptGen offers the generality for algorithmic design andmodularity for control composition, and it also enables iterative controls. Experiments validate that
Revised: D Additional Results for De-Biasing Generative Models

Original: PromptGen applies to various generative models (StyleGAN2, StyleNeRF, NVAE), control types(continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inversegraphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).
Revised: Table 5 provides quantitative results of de-biasing StyleGAN2 across categorical attributes (Figure 6). Speciﬁcally, we used the pre-trained classiﬁer provided by FairFace [31] to classify the attributes of the generated images. We then report the KL divergence between the attribute distribution of the generated images and the uniform distribution.

Original: Limitation and future work:
Revised: Figure 13 and Figure 14 visualize the de-biasing results. We report the KL divergence between the generated distribution and the uniform distribution. Interestingly, on MetFaces, de-biasing the race results in more sculptures, and we postulate that the reason is that [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: r t i

Revised: [Figure]

Revised: V e r ti ca l m ou t h s t r e t

Revised: c h

Revised: Horizontal mouth stretch

Revised: [Figure]

Revised: E y e c l o s e

Revised: [Figure]

Revised: N o s eti p s t r e t c h

Revised: [Figure]

Revised: E Experiments on Generative 3D Face Models

Revised: 3D face modeling has been an active area of research that has recently gained major interest due to applications in virtual humans, deep faces, and digital actors.

Revised: Existing 3D deep learning generative models build 3D compact representations of shape and appearance capable of modeling non-linearity (e.g., scatter effects, specularities) that is necessary for generating photo-realistic faces.

Revised: Providing a generative model with the capability of generating 3D faces with a particular geometry (e.g., a speciﬁc distance between eyes or nose length) is particularly useful for technical artists when creating new characters.

Revised: However, this problem remains unaddressed, partially due to the lack of publicly available 3D databases labeled with such constraints.

Revised: We refer to this capability as the intra-sample constraint in 3D generative models, and this section shows how our PromptGen framework is able to achieve this by deﬁning a new energy function based on signed distances.

Revised: We conducted the experiments on the FaceScape dataset [80], a large-scale 3D human face dataset consisting of 16 , 940 topologically uniformly registered 3D face meshes along with high-quality textures.

Revised: The dataset contains 847 identities with 20 expressions per identity, and each mesh consists of 26 , 317 vertices of which 68 are 3D landmark vertices covering eyes, nose, mouth, jaw, and eyebrows.

Revised: With some abuse of notation, we denote a 3D face mesh as x ∈ R 3 V (recall that x is used to represent an image in other parts of this paper), where V = 26 , 317 is the number of vertices.

Revised: Given an index l of a vertex (e.g., one of the 68 landmark vertices), the 3D coordinate of this vertex is deﬁned as x [ l ] ∈ R 3 .

Revised: We pre-train a generative mesh model G that maps each latent code z ∼ N ( 0 , I ) to a 3D mesh x .

Revised: PromptGen allows us to obtain geometric control on the 3D meshes.

Revised: Speciﬁcally, given two landmark vertex indices l 1 and l 2 , we deﬁne an intra-sample constraint that enforces the signed distance (along a given direction) between the two vertices to be s .

Revised: To this effect, we deﬁne the signed-distance energy as

Revised: [Equation]

Revised: where d ( · , · ) is the signed-distance (along a given direction), and | · | is the absolute value.

Revised: Figure 15 presents several examples of 3D mesh control, in which random textures are applied to the generated meshes.

Revised: It shows that PromptGen successfully controls the generative 3D mesh model in terms of the mouth stretch, eye openness, nose-tip stretch, and lip forward.

Original: We provide an error analysis in Appendix F and a discussion onsocietal impact in Appendix G. PromptGen is restricted by the pre-trained generative model’s coverage[3, 27].
Revised: As we discussed in Section 5, the controlled distribution depends on (1) the pre-trained generative model’s coverage and (2) the off-the-shelf models used for the control. In this section, we provide some examples of PromptGen failing to generate satisfactory images using the CLIP energy. For the pre-trained generative model, we use StyleGAN2 trained on FFHQ, AFHQ-Wild, or LSUN-Churches.

Revised: [Figure]

Revised: sometimes fails or shows the “reporting bias” that we discuss in Section 4.5.

Revised: For a deeper understanding of these failures and biases, in Figure 17, we provide some image samples from the LAION-400M dataset [63] with CLIP retrieval, using the same text descriptions as this ﬁgure.

Revised: Our ﬁrst observation is that the CLIP model sometimes fails in modeling linguistic negation.

Revised: For instance, the text description photo of a man without beard results in a distribution of photos of men with beard (Figure 16(a)).

Revised: Meanwhile, Figure 16(b) shows that CLIP is capable of modeling the negation of having makeup, but with the “reporting bias” discussed in Section 4.5.

Revised: Moreover, CLIP seems to have difﬁculty in numerical reasoning, and gaining control over the count of speciﬁc objects in a scene tends to be unsuccessful.

Revised: We showed this by specifying photo of a church with three windows, which did not result in the desired speciﬁcation (Figure 16(c)).

Revised: To gain a deeper understanding of the above failures and biases, in Figure 17, we provide some image samples from the LAION-400M dataset [63] with CLIP retrieval, using the same text descriptions as

Revised: Figure 16.

Revised: CLIP retrieval is based on the CLIP embedding similarity between the web images and text descriptions, while the original text below each individual image is not used.

Revised: We observe an impressive consistency between CLIP retrieval and PromptGen: in both Figure 16(a) and Figure 17(a), most images have beard; in both Figure 16(b) and Figure 17(b), all images are female; in both Figure 16(c) and Figure 17(c), some images do not have exactly three windows.

Revised: This consistency suggests that the failures and biases in Figure 16 should be mostly attributed to the CLIP model rather than to our

Original: PromptGen focuses on mode-seeking and -reweighting instead of domain adaptation, and it ispossible to combine PromptGen and domain adaptation of generative models [17].
Revised: PromptGen algorithm. We believe that our observation sheds light on the intricacy of contrastive multi-modal (vision-language) pre-training, which is worthy of being further investigated. [Figure] illustrated in web images and text descriptions, while the original text below each individual image is not used.

Revised: In

Revised: Figure 17(a), most images have a beard; in Figure 17(b), all images are female; in Figure 17(c), some images do not have exactly three windows.

Revised: These observations are consistent with those in Figure 16.

Revised: Another observation is that the control tends to fail when the text description requires sampling from low-density regions of the pre-trained generative model’s output space.

Revised: In other words, the control usually fails if the pre-trained generative model does not cover the mode we are trying to gain control over.

Revised: For example, images faithful to photo of a person yawning and photo of a baby with long hair are not commonly observed in the FFHQ dataset and, hence, these two text descriptions result in degeneration (Figure 18(a)) or weird images (Figure 18(b)).

Revised: Another example is photo of an animal from the side, which is not commonly observed in the AFHQ-Wild dataset, and Figure 18(c) shows that the generated images fail to follow this description.

Revised: Even when the control is successful (e.g., when a complex description is decomposed in Figure 8(b)), sampling from low-density regions results in limited diversity (e.g., the backgrounds in Figure 8(b) look similar to each other).

Original: Also, PromptGendepends on the off-the-shelf models that provide knowledge about the control (Appendix F).
Revised: Finally, we also ran some failure controls using PPGM, showing that PromptGen and PPGM reveal similar failure cases of pre-trained generative models and CLIP. Results are shown in Figure 19. [Figure] [Figure]

Revised: [Figure]

Revised: G Societal Impact

Original: Insubsequent work, it  can be beneﬁcial to explore how to learn energy functions [15, 26] (besides ourmoment constraint used for de-biasing), which may provide ﬁne-grained control with less bias.
Revised: With the improvements in generative models for images, DeepFake technology [75, 72] has become more accessible. Like any new technology, it is a double-edged sword, and it is crucial to research and comprehend the possible advantages and disadvantages of generative models for society. On the positive side, we show that PromptGen can be used to de-bias pre-trained generative models and to reveal biases learned by text-image models (like CLIP), indicating that PromptGen might be a useful tool for fair AI if used appropriately. The efﬁcient inference provided by PromptGen also helps reduce the computational expense, which has a positive impact on the environment.

Revised: Better controllability, however, unavoidably makes it simpler to synthesize targeted pictures, which might have detrimental social effects in creating deceptive media (e.g., DeepFakes) or privacy leaks (e.g., identity-conditioned human face synthesis).

Revised: To battle these cases, there are current technologies that can detect fake media effectively, and we expect companies and users to use these technologies to distinguish what is real from fake.

Revised: We encourage practitioners to consider these risks when using

Revised: PromptGen to develop systems.