Original: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models
Revised: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models

Original: Abstract
Revised: Abstract

Original: Generative models (e.g., GANs, diffusion models) learn the underlying data distribution in an unsupervised manner.
Revised: Generative models (e.g., GANs, diffusion models) learn the underlying data distribution in an unsupervised manner.

Original: However, many applications of interest require sampling from a particular region of the output space or sampling evenly over a range of characteristics.
Revised: However, many applications of interest require sampling from a particular region of the output space or sampling evenly over a range of characteristics.

Original: For efﬁcient sampling in these scenarios, we propose Generative Visual Prompt (PromptGen), a framework for distributional control over pre-trained generative models by incorporating knowledge of other off-the-shelf models.
Revised: For efﬁcient sampling in these scenarios, we propose Generative Visual Prompt (PromptGen), a framework for distributional control over pre-trained generative models by incorporating knowledge of other off-the-shelf models.

Original: PromptGen deﬁnes control as energy-based models (EBMs) and samples images in a feed-forward manner by approximating the EBM with invertible neural networks, avoiding optimization at inference.
Revised: PromptGen deﬁnes control as energy-based models (EBMs) and samples images in a feed-forward manner by approximating the EBM with invertible neural networks, avoiding optimization at inference.

Original: Our experiments demonstrate how
Revised: Our experiments demonstrate how

Original: PromptGen can efﬁciently sample from several unconditional generative models (e.g., StyleGAN2, StyleNeRF, diffusion autoencoder, NVAE) in a controlled or/and de-biased manner using various off-the-shelf models: (1) with the CLIP model as control, PromptGen can sample images guided by text, (2) with image classiﬁers as control, PromptGen can de-bias generative models across a set of attributes or attribute combinations, and (3) with inverse graphics models as control, PromptGen can sample images of the same identity in different poses.
Revised: PromptGen can efﬁciently sample from several unconditional generative models (e.g., StyleGAN2, StyleNeRF, diffusion autoencoder, NVAE) in a controlled or/and de-biased manner using various off-the-shelf models: (1) with the CLIP model as control, PromptGen can sample images guided by text, (2) with image classiﬁers as control, PromptGen can de-bias generative models across a set of attributes or attribute combinations, and (3) with inverse graphics models as control, PromptGen can sample images of the same identity in different poses.

Original: (4) Finally, PromptGen reveals that the CLIP model shows a “reporting bias” when used as control, and
Revised: (4) Finally, PromptGen reveals that the CLIP model shows a “reporting bias” when used as control, and

Original: PromptGen can further de-bias this controlled distribution in an iterative manner.
Revised: PromptGen can further de-bias this controlled distribution in an iterative manner.

Original:  Introduction
Revised:  Introduction

Original: Generative models learn the underlying high-dimensional data distribution and have achieved promising performance on image synthesis [4, 75, 36, 21].
Revised: Generative models learn the underlying high-dimensional data distribution and have achieved promising performance on image synthesis [4, 75, 36, 21].

Original: Though being well praised, they still face two main limitations.
Revised: Though being well praised, they still face two main limitations.

Original: First, since generative models are typically trained in an unsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁc region of the space.
Revised: First, since generative models are typically trained in an unsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁc region of the space.

Original: Second, generative models are prone to inherit the imbalance and bias of training data [60, 31].
Revised: Second, generative models are prone to inherit the imbalance and bias of training data [60, 31].

Original: For instance, StyleGAN2 is more likely to produce images of white individuals, see Figure 1(e).
Revised: For instance, StyleGAN2 is more likely to produce images of white individuals, see Figure 1(e).

Original: Previous works have studied these challenges separately, and typical methods include editing of “style” codes [23, 68, 31] and explicit conditions [43, 69].
Revised: Previous works have studied these challenges separately, and typical methods include editing of “style” codes [23, 68, 31] and explicit conditions [43, 69].

Original: However, these methods are either model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiring all training samples to be labeled for explicit conditions), limiting their generality and practical use.
Revised: However, these methods are either model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiring all training samples to be labeled for explicit conditions), limiting their generality and practical use.

Original: To address the above challenges, this paper advocates a uniﬁed formulation, distributional control of generative models , which enables controllability by incorporating the knowledge of off-the-shelf models (e.g., CLIP [58], classiﬁers, or inverse graphics models [16]).
Revised: To address the above challenges, this paper advocates a uniﬁed formulation, distributional control of generative models , which enables controllability by incorporating the knowledge of off-the-shelf models (e.g., CLIP [58], classiﬁers, or inverse graphics models [16]).

Original: Based on this uniﬁed view, we propose to learn distributions in the latent space of a pre-trained generative model while keeping the pre-trained weights ﬁxed.
Revised: Based on this uniﬁed view, we propose to learn distributions in the latent space of a pre-trained generative model while keeping the pre-trained weights ﬁxed.

Original: Given its conceptual similarity to prompting [42, 85, 86, 30, 22], we term our framework as Generative Visual Prompt (PromptGen).
Revised: Given its conceptual similarity to prompting [42, 85, 86, 30, 22], we term our framework as Generative Visual Prompt (PromptGen).

Original: PromptGen requires no training data , and the only supervision comes from off-the-shelf models that help deﬁne the control.
Revised: PromptGen requires no training data , and the only supervision comes from off-the-shelf models that help deﬁne the control.

Original: Speciﬁcally,[Figure]
Revised: Speciﬁcally,

Original: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated by an invertible neural network (INN).
Revised: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated by an invertible neural network (INN).

Original: Unlike methods that require optimization at inference in
Revised: Unlike methods that require optimization at inference in

Original: EBM sampling [49, 14, 45, 52], PromptGen samples images in a feed-forward manner, which is highly efﬁcient.
Revised: EBM sampling [49, 14, 45, 52], PromptGen samples images in a feed-forward manner, which is highly efﬁcient.

Original: Moreover, PromptGen stands alone at inference, meaning that we can discard the off-the-shelf models, which deﬁne the control, after training.
Revised: Moreover, PromptGen stands alone at inference, meaning that we can discard the off-the-shelf models, which deﬁne the control, after training.

Original: We illustrate the beneﬁts of PromptGen with several experiments.
Revised: We illustrate the beneﬁts of PromptGen with several experiments.

Original: Figure 1 demonstrates our main ﬁndings with StyleGAN2 trained on FFHQ [35] without any labels, while we also show results for
Revised: Figure 1 demonstrates our main ﬁndings with StyleGAN2 trained on FFHQ [35] without any labels, while we also show results for

Original: StyleNeRF [21], diffusion autoencoder [57], and NVAE [75] in Section 4.
Revised: StyleNeRF [21], diffusion autoencoder [57], and NVAE [75] in Section 4.

Original: Figure 1(a) illustrates how we can sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.
Revised: Figure 1(a) illustrates how we can sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.

Original: Figure 1(b) shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) to sample uniformly across all races, de-biasing the pre-trained StyleGAN2.
Revised: Figure 1(b) shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) to sample uniformly across all races, de-biasing the pre-trained StyleGAN2.

Original: Figure 1(d) illustrates that
Revised: Figure 1(d) illustrates that

Original: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.
Revised: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.

Original: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic design and modularity for control composition, but also enables iterative controls when some controls are contingent on others.
Revised: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic design and modularity for control composition, but also enables iterative controls when some controls are contingent on others.

Original: For instance, one may train a text-controlled distribution and then de-bias this distribution.
Revised: For instance, one may train a text-controlled distribution and then de-bias this distribution.

Original: To achieve this, we view the composition of the INN and the generative model as a new “generative model” to be controlled.
Revised: To achieve this, we view the composition of the INN and the generative model as a new “generative model” to be controlled.

Original: Figure 1(f) illustrates that PromptGen reveals “reporting bias” of the CLIP model [58], where “without makeup” is – perhaps surprisingly – positively correlated with female, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control
Revised: Figure 1(f) illustrates that PromptGen reveals “reporting bias” of the CLIP model [58], where “without makeup” is – perhaps surprisingly – positively correlated with female, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Related Work
Revised:  Related Work

Original: Over the past few years, generative models have gained the ability to generate images with high visual quality.
Revised: Over the past few years, generative models have gained the ability to generate images with high visual quality.

Original: A few of the most widely used methods include generative adversarial networks (GANs) [18],
Revised: A few of the most widely used methods include generative adversarial networks (GANs) [18],

Original: VAE [40], invertible neural networks [11, 12]
Revised: VAE [40], invertible neural networks [11, 12]

Original: [Table]
Revised: [Table]

Original: and diffusion models [25, 71].
Revised: and diffusion models [25, 71].

Original: In particular, generative models trained on large amounts of unlabeled data, e
Revised: In particular, generative models trained on large amounts of unlabeled data, e

Original: [Table]
Revised: [Table]

Original: g., BigGAN [4], StyleGANs [35, 36, 34, 64],
Revised: g., BigGAN [4], StyleGANs [35, 36, 34, 64],

Original: Glow [39]
Revised: Glow [39]

Original: [Table]
Revised: [Table]

Original: and diffusion models [51, 10], achieve promising image synthesis results.
Revised: and diffusion models [51, 10], achieve promising image synthesis results.

Original: Despite their success, controllability and de-biasing are still two fundamental challenges generative models face.
Revised: Despite their success, controllability and de-biasing are still two fundamental challenges generative models face.

Original: For controllability, existing methods include explicit conditioning at training [43, 69] or local editing of the learned representation, e.g, “style” codes [1, 56, 46, 68, 80].
Revised: For controllability, existing methods include explicit conditioning at training [43, 69] or local editing of the learned representation, e.g, “style” codes [1, 56, 46, 68, 80].

Original: For de-biasing, existing methods include local editing of “style” codes [60, 31] and importance sampling for either training [20] or inference [29].
Revised: For de-biasing, existing methods include local editing of “style” codes [60, 31] and importance sampling for either training [20] or inference [29].

Original: Existing works study these problems separately, each requiring a speciﬁc design for the task studied.
Revised: Existing works study these problems separately, each requiring a speciﬁc design for the task studied.

Original: On the contrary, PromptGen is a uniﬁed framework for arbitrary controls deﬁned by off-the-shelf models.
Revised: On the contrary, PromptGen is a uniﬁed framework for arbitrary controls deﬁned by off-the-shelf models.

Original: The beneﬁts of unifying tasks have been shown by a recent trend of works in multiple research areas across vision and language [59, 48, 63, 81, 61].
Revised: The beneﬁts of unifying tasks have been shown by a recent trend of works in multiple research areas across vision and language [59, 48, 63, 81, 61].

Original: Moreover, one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen: since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuning can be applied before or after PromptGen training.
Revised: Moreover, one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen: since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuning can be applied before or after PromptGen training.

Original: We leave this exploration to future studies.
Revised: We leave this exploration to future studies.

Original: Previous methods usually sample from EBM [41] with Markov Chain Monte Carlo (MCMC) [73, 76, 14, 19, 13, 45].
Revised: Previous methods usually sample from EBM [41] with Markov Chain Monte Carlo (MCMC) [73, 76, 14, 19, 13, 45].

Original: Among them, plug-and-play generative models (PPGMs) [49] and LACE [52] deﬁne latent-space EBMs.
Revised: Among them, plug-and-play generative models (PPGMs) [49] and LACE [52] deﬁne latent-space EBMs.

Original: However, MCMC requires inference-time optimization, which is inefﬁcient and requires the off-the-shelf models to be available at inference; this is also the criticism for diffusion models, regardless of being gradient-guided [51, 67] or not [25, 26, 50].
Revised: However, MCMC requires inference-time optimization, which is inefﬁcient and requires the off-the-shelf models to be available at inference; this is also the criticism for diffusion models, regardless of being gradient-guided [51, 67] or not [25, 26, 50].

Original: In contrast, PromptGen achieves efﬁcient, feed-forward sampling.
Revised: In contrast, PromptGen achieves efﬁcient, feed-forward sampling.

Original: Table 1 shows a comparison with previous methods.
Revised: Table 1 shows a comparison with previous methods.

Original: Our
Revised: Our

Original: INN training is similar to that proposed by [54] to sample from physical systems, which is later used by [78] to solve inverse problems with two composed INNs.
Revised: INN training is similar to that proposed by [54] to sample from physical systems, which is later used by [78] to solve inverse problems with two composed INNs.

Original: Notably, [54] and [78] do not leverage a low-dimensional latent space, and [78] requires training a separate model for each image sample.
Revised: Notably, [54] and [78] do not leverage a low-dimensional latent space, and [78] requires training a separate model for each image sample.

Original: In this paper, we use INN to modelarbitrary EBMs for various generative models.
Revised: In this paper, we use INN to model

Original: [Figure] views the composition G 
Revised: PromptGen views the composition G  f ✓ as a new “generative model” for iterative control.

Original:  Method
Revised:  Method

Original: Figure 2 illustrates our PromptGen framework.
Revised: Figure 2 illustrates our PromptGen framework.

Original: To begin, the user selects a pre-trained generative model G .
Revised: To begin, the user selects a pre-trained generative model G .

Original: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .
Revised: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .

Original: We train an INN f θ to approximate p ( z |C ) .
Revised: We train an INN f ✓ to approximate p ( z |C ) .

Original: PromptGen views the functional composition G ◦ f θ as a new generative model and can perform iterative control.
Revised: PromptGen views the functional composition G  f ✓ as a new generative model and can perform iterative control.

Original: Algorithm 1 describes the overall procedure.Algorithm 1:
Revised: Algorithm 1 describes the overall procedure.

Original: p z |
Revised: [Table]

Original:  f θ
Revised: Input: control C of the current iteration 2. Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3. Train an INN f ✓ : R d !

Original: R d R approximate p ( z |C 
Revised: R d to approximate p ( z |C ) (Section 3.2) 4. G G  f ✓

Original: Section 3.1 Latent-Space EBM for Distributional Control
Revised: 3.1 Latent-Space EBM for Distributional Control

Original: The plug-and-play generative model [49] was ﬁrst proposed to use a latent-space EBM for controllable image synthesis.
Revised: The plug-and-play generative model [49] was ﬁrst proposed to use a latent-space EBM for controllable image synthesis.

Original: If a ﬁxed generative model is used, then theoretically, any image-space EBM can be viewed as a latent-space EBM, as shown by [19] and [52].
Revised: If a ﬁxed generative model is used, then theoretically, any image-space EBM can be viewed as a latent-space EBM, as shown by [19] and [52].

Original: We deﬁne the latent-space EBM following similar formulation as [49, 19, 52], with some new energy functions.
Revised: We deﬁne the latent-space EBM following similar formulation as [49, 19, 52], with some new energy functions.

Original: We then extend the formulation to incorporate the moment constraint [7], which was adopted for language modeling [37], but unlike [37], we deﬁne the moment constraint in the latent space to accommodate generative vision models.
Revised: We then extend the formulation to incorporate the moment constraint [7], which was adopted for language modeling [37], but unlike [37], we deﬁne the moment constraint in the latent space to accommodate generative vision models.

Original: We deﬁne a control C as M independent properties { y 1 , . . .
Revised: We deﬁne a control C as M independent properties { y 1 , . . .

Original: , y M } , e.g., y 1 can be a text description and y 2 can be an attribute.
Revised: , y M } , e.g., y 1 can be a text description and y 2 can be an attribute.

Original: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)
Revised: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)

Original: [Equation]
Revised: [Equation]

Original: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy are preferred.
Revised: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy are preferred.

Original: Using a pre-trained generative model G : R d → X that maps a latent code z to an image x , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping it to x = G ( z ) .
Revised: Using a pre-trained generative model G : R d ! X that maps a latent code z to an image x , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping it to x = G ( z ) .

Original: Appendix B.2 shows that this EBM is equivalent to the latent-space EBM
Revised: Appendix B.2 shows that this EBM is equivalent to the latent-space EBM

Original: [Equation]
Revised: [Equation]

Original: Latent-space EBM allows us to use any off-the-shelf model to specify the control.
Revised: Latent-space EBM allows us to use any off-the-shelf model to specify the control.

Original: The following are some examples that are discussed in this paper (explained in Appendix B.1):
Revised: The following are some examples that are discussed in this paper (explained in Appendix B.1):

Original: Classiﬁer energy:
Revised: Classiﬁer energy:

Original: Given a classiﬁer P ( ·| x ) and the target class a that we want to sample images from, we deﬁne the classiﬁer energy as E classiﬁer ( x , a ) =
Revised: Given a classiﬁer P ( ·| x ) and the target class a that we want to sample images from, we deﬁne the classiﬁer energy as E classiﬁer ( x , a ) =

Original: − log P ( a | x ) .
Revised:  log P ( a | x ) .

Original: CLIP energy: Using the CLIP model [58], we deﬁne the CLIP energy as the cosine distance between the embeddings of the image and the text t , averaged over L differentiable augmentations [84, 46]:
Revised: CLIP energy: Using the CLIP model [58], we deﬁne the CLIP energy as the cosine distance between the embeddings of the image and the text t , averaged over L differentiable augmentations [84, 46]:

Original: [Equation]
Revised: [Equation]

Original: Inverse graphics energy:
Revised: Inverse graphics energy:

Original: Given an inverse graphics model, f X→P , which infers image parameters (e.g., pose and expression), and the target parameters ρ , we deﬁne the inverse graphics energy as
Revised: Given an inverse graphics model, f X ! P , which infers image parameters (e.g., pose and expression), and the target parameters ⇢ , we deﬁne the inverse graphics energy as

Original: [Equation][Equation]
Revised: [Equation]

Original: where d (cid:104)· , ·(cid:105) is the geodesic distance between the inferred parameters and the target parameters.
Revised: where d h · , · i is the geodesic distance between the inferred parameters and the target parameters.

Original: Moment constraint:
Revised: Moment constraint:

Original: Some controls cannot be directly deﬁned by off-the-shelf models, and the moment constraint [7, 37] is one of them.
Revised: Some controls cannot be directly deﬁned by off-the-shelf models, and the moment constraint [7, 37] is one of them.

Original: Given a mapping γ : X → R K
Revised: Given a mapping  : X !

Original:  (e.g., γ can be a classiﬁer that outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as
Revised: R K (e.g.,  can be a classiﬁer that outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as

Original: [Equation]
Revised: [Equation]

Original: Deviation [Table] distribution
Revised: Deviation from the pre-trained distribution

Original: [Equation]
Revised: [Equation]

Original: where µ is the user-speciﬁed vector.
Revised: where µ is the user-speciﬁed vector.

Original: For example, if we want to generate images that are uniformly distributed across races
Revised: For example, if we want to generate images that are uniformly distributed across races, we may use a race classiﬁer as  and deﬁne µ =  |A|  1 , . .

Original: [Equation]
Revised: [Equation]

Original: we may use a race classiﬁer as γ and deﬁne µ = (cid:0) |A| − 1 , . .

Original: . , |A| − 1 (cid:1) where
Revised: |A|  where

Original: A is the set of races.
Revised: A is the set of races.

Original: It means that we would like to ﬁnd a distribution that 1) stays close to the original, pre-trained generative model G and 2) mitigates G ’s bias.
Revised: It means that we would like to ﬁnd a distribution that 1) stays close to the original, pre-trained generative model G and 2) mitigates G ’s bias.

Original: In this paper, we generalize the moment constraint to the latent space [Equation] and approximate the above objective as (detailed in Appendix B.6):
Revised: In this paper, we generalize the moment constraint to the latent space, and approximate the above objective as (detailed in Appendix B.6):

Original: [Equation]
Revised: [Equation]

Original: Notably, to optimize ˆ β in Eq.
Revised: Notably, to optimize ˆ  in Eq.

Original: [Equation]
Revised: [Equation]

Original: we leverage existing gradient-based optimization tools such as stochastic gradient descent (SGD) and Adam [38].
Revised: we leverage existing gradient-based optimization tools such as stochastic gradient descent (SGD) and Adam [38] [Equation]

Original: [Figure]
Revised: [Figure]

Original: while not converged do
Revised: while not converged do

Original: Sample (cid:15) ∼ N ( 0 , I ) 2.
Revised: Sample ✏ ⇠ N ( 0 , I ) 2.

Original: Map (cid:15) to latent code z = f θ ( (cid:15) ) 3.
Revised: Map ✏ to latent code z = f ✓ ( ✏ ) 3.

Original: Map z to an image x = G ( z ) 4.
Revised: Map z to an image x = G ( z ) 4.

Original: Optimize θ with gradient ∇ θ (cid:16) − log | det( ∂f θ ∂ (cid:15) ) | − log p z ( z ) + E C ( x ) (cid:17)
Revised: Optimize ✓ with gradient r ✓ [Equation]

Original: 3.2 Approximating EBM with Invertible Neural Network
Revised: 3.2 Approximating EBM with Invertible Neural Network

Original: Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcient sampling network.
Revised: Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcient sampling network.

Original: To achieve this, we train a distribution p θ ( z ) that minimizes the KL divergenceD
Revised: To achieve this, we train a distribution p ✓ ( z ) that minimizes the KL divergence

Original:  KL ( p θ ( z ) (cid:107) p ( z |C )) .
Revised: D KL ( p ✓ ( z ) k p ( z |C )) .

Original: Estimating D KL ( p θ ( z ) (cid:107) p ( z |C )) requires easily sampling z ∼ p θ and tractably computing p θ ( z ) .
Revised: Estimating D KL ( p ✓ ( z ) k p ( z |C )) requires easily sampling z ⇠ p ✓ and tractably computing p ✓ ( z ) .

Original: Inspired by [54], we model p θ with an INN f θ that deﬁnes a bijection z
Revised: Inspired by [54], we model p ✓ with an INN f ✓ that deﬁnes a bijection z

Original: = f θ ( (cid:15) ) , which has two merits besides the invertibility: (1) one can easily sample z by sampling (cid:15) ∼ N ( 0 , I ) and mapping it to z = f θ ( (cid:15) ) , and (2) p θ ( z ) has a closed-form solution:
Revised: = f ✓ ( ✏ ) , which has two merits besides the invertibility: (1) one can easily sample z by sampling ✏ ⇠ N ( 0 , I ) and mapping it to z = f ✓ ( ✏ ) , and (2) p ✓ ( z ) has a closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: Based on these properties of INN [Equation] we can rewrite our KL divergence objective D KL ( p θ ( z ) (cid:107) p ( z |C )) as (full derivations in Appendix B.3)
Revised: Based on these properties of INN, we can rewrite our KL divergence objective D KL ( p ✓ ( z ) k p ( z |C )) as (full derivations in Appendix B.3)

Original: the following form:
Revised: the following form:

Original: [Equation]
Revised: [Equation]

Original: Since H N ( 0 , I ) and log
Revised: Since H N ( 0 , I ) and log

Original: Z are independent of θ , our training objective becomesarg min θ
Revised: Z are independent of ✓ , our training objective becomes

Original: [Equation]
Revised: [Equation]

Original: Figure 3 gives an illustration of our process [Equation] and Algorithm 2 describes the algorithmic details.
Revised: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.

Original: Figure 4(g)
Revised: PromptGen in a class-embedding space (Figure 4(g))

Original: Previous works [4, 64] have shown that class conditioning boosts generative models’ performances on ImageNet [62].
Revised: Previous works [4, 64] have shown that class conditioning boosts generative models’ performances on ImageNet [62].

Original: Speciﬁcally, classconditioned generative models map a latent code z and a class embedding y to x = G ( z , y ) .
Revised: Speciﬁcally, classconditioned generative models map a latent code z and a class embedding y to x = G ( z , y ) .

Original: To extend PromptGen to these models, we train an INN h θ to map ξ ∼ N ( µ , σ 2 I ) to y =
Revised: To extend PromptGen to these models, we train an INN h ✓ to map ⇠ ⇠ N ( µ ,  2 I ) to y =

Original: h θ ( ξ ) , where µ and σ are the mean and standard deviation of G ’s class embeddings.
Revised: h ✓ ( ⇠ ) , where µ and  are the mean and standard deviation of G ’s class embeddings.

Original: The motivation for deﬁning the distribution of ξ as N ( µ , σ 2 I ) but not N ( 0 , I ) is that we want the learned INN h θ to be volumepreserving, which is easier to train.
Revised: The motivation for deﬁning the distribution of ⇠ as N ( µ ,  2 I ) but not N ( 0 , I ) is that we want the learned INN h ✓ to be volumepreserving, which is easier to train.

Original: The training objective is to minimize D KL ( p θ ( z , y ) (cid:107) p ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4): 
Revised: The training objective is to minimize D KL ( p ✓ ( z , y ) k p ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4): [Equation]

Original: [Equation]
Revised: [Equation]

Original: where p y ( y ) =
Revised: where p y ( y ) =

Original: N ( y | µ , σ 2 I ) is the estimated class-embedding distribution.
Revised: N ( y | µ ,  2 I ) is the estimated class-embedding distribution.

Original: PromptGen with conditional INN To generalize the control to continuous values, e.g., the scene parameters ρ in Eq.
Revised: PromptGen with conditional INN To generalize the control to continuous values, e.g., the scene parameters ⇢ in Eq.

Original: (4), we condition the INN on ρ .
Revised: (4), we condition the INN on ⇢ .

Original: The condition is modeled by replacing z = f θ ( (cid:15) ) with z = f θ ( (cid:15) , ρ ) .
Revised: The condition is modeled by replacing z = f ✓ ( ✏ ) with z = f ✓ ( ✏ , ⇢ ) .

Original: Space limited, we provide details and derivations in Appendix B.5.
Revised: Space limited, we provide details and derivations in Appendix B.5.

Original: This extension results in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label pairs and is only applicable to explicit condition, which is not capable of modeling EBMs.
Revised: This extension results in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label pairs and is only applicable to explicit condition, which is not capable of modeling EBMs.

Original: When the condition is modeled by the equivariant operations proposed by [79], PromptGen also satisﬁes the homomorphism property [79] in the latent space.
Revised: When the condition is modeled by the equivariant operations proposed by [79], PromptGen also satisﬁes the homomorphism property [79] in the latent space.

Original: [Figure]
Revised: [Figure]

Original: [75], and diffusion autoencoder [57], with text description a photo of a baby.

Original: Figure 4(d) shows PromptGen applied to
Revised: ) shows PromptGen applied to

Original: StyleGAN2 [36] trained on AFHQ-Cats [6], with text description a photo of a British shorthair cat
Revised: StyleGAN2 [36] trained on AFHQ-Cats [6], with text description a photo of a British shorthair cat

Original: Figure 4(e) and Figure 4(f) are PromptGen applied to StyleGAN2 trained on Landscape-HQ [70], with text descriptions a photo of {autumn, winter} scene.
Revised: Figure 4(e) and Figure 4(f) are PromptGen applied to StyleGAN2 trained on Landscape-HQ [70], with text descriptions a photo of {autumn, winter} scene.

Original: Figure 4(g) is the extension to the embedding space of BigGAN [4] on ImageNet [62].
Revised: Figure 4(g) is the extension to the embedding space of BigGAN [4] on ImageNet [62].

Original: All images are resized for visualization.
Revised: All images are resized for visualization.

Original: See Appendix C for results on more datasets and text descriptions.
Revised: See Appendix C for results on more datasets and text descriptions.

Original: [Figure]
Revised: [Figure]

Original:  Experiments
Revised:  Experiments

Original: This section describes the experimental validation of PromptGen.
Revised: This section describes the experimental validation of PromptGen.

Original: See Appendix A for experiments on synthetic data and Appendix C, D, E, and F for additional experiments on images and 3D meshes.
Revised: See Appendix A for experiments on synthetic data and Appendix C, D, E, and F for additional experiments on images and 3D meshes.

Original: Additional experimental details are provided in Appendix B.8.
Revised: Additional experimental details are provided in Appendix B.8.

Original: 4.1 Image Synthesis based on Text Description
Revised: 4.1 Image Synthesis based on Text Description

Original: This experiment illustrates the capability of PromptGen to sample images from a generative model driven by a text description t using the pre-trained CLIP model [58] (the ViT-B/32 version).
Revised: This experiment illustrates the capability of PromptGen to sample images from a generative model driven by a text description t using the pre-trained CLIP model [58] (the ViT-B/32 version).

Original: We used the CLIP energy from Eq.
Revised: We used the CLIP energy from Eq.

Original: (3), with text descriptions such as a photo of a baby.
Revised: (3), with text descriptions such as a photo of a baby.

Original: Figure 5 shows a comparison between PromptGen and two previous CLIP-guided image generation methods, StyleCLIP [56] and StyleGAN2-NADA [17].
Revised: Figure 5 shows a comparison between PromptGen and two previous CLIP-guided image generation methods, StyleCLIP [56] and StyleGAN2-NADA [17].

Original: We observe that PromptGen generates diverse and high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and
Revised: We observe that PromptGen generates diverse and high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and

Original: StyleGAN2-NADA generates baby-like adults.
Revised: StyleGAN2-NADA generates baby-like adults.

Original: These results show that (1) locally editing the latent code (i.e., StyleCLIP) is not always an effective method for controlling generative models (e.g., not all images’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA) is not effective in seeking modes in a generative model.
Revised: These results show that (1) locally editing the latent code (i.e., StyleCLIP) is not always an effective method for controlling generative models (e.g., not all images’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA) is not effective in seeking modes in a generative model.

Original: Figure 4(a), Figure 4(b), and Figure 4(c) show how PromptGen applies to StyleNeRF [21], NVAE [75], and diffusion autoencoder (DiffAE; [57]).
Revised: Figure 4(a), Figure 4(b), and Figure 4(c) show how PromptGen applies to StyleNeRF [21], NVAE [75], and diffusion autoencoder (DiffAE; [57]).

Original: Although these generative models are all trained on FFHQ, baby images sampled from them have distinct characteristics.
Revised: Although these generative models are all trained on FFHQ, baby images sampled from them have distinct characteristics.

Original: Figures 4(d)-4(f) show PromptGen applied to cats and landscape generation.
Revised: Figures 4(d)-4(f) show PromptGen applied to cats and landscape generation.

Original: Figure 4(g) shows the extension (Section 3.2) of our PromptGen to the class-embedding space of
Revised: Figure 4(g) shows the extension (Section 3.2) of our PromptGen to the class-embedding space of

Original: BigGAN [4], a class-conditional GAN trained on ImageNet [62].
Revised: BigGAN [4], a class-conditional GAN trained on ImageNet [62].

Original: We observe that PromptGen helps
Revised: We observe that PromptGen helps

Original: BigGAN generate images with complex text descriptions, which are out of BigGAN’s training image distribution; however, the diversity seems to be limited in this extension.
Revised: BigGAN generate images with complex text descriptions, which are out of BigGAN’s training image distribution; however, the diversity seems to be limited in this extension.

Original: [Figure]
Revised: [Figure]

Original: 4.2 De-Biasing Pre-Trained Generative Models
Revised: 4.2 De-Biasing Pre-Trained Generative Models

Original: An important problem in generative models is to generate fair distributions w.r.t a set of attributes of interest.
Revised: An important problem in generative models is to generate fair distributions w.r.t a set of attributes of interest.

Original: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races and ages.
Revised: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races and ages.

Original: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [33] in terms of categorical attributes, using the moment constraint deﬁned in Eq.
Revised: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [33] in terms of categorical attributes, using the moment constraint deﬁned in Eq.

Original: (5) and Eq.
Revised: (5) and Eq.

Original: As control, we used a classiﬁer trained on FairFace 224 2 [32] as γ .
Revised: As control, we used a classiﬁer trained on FairFace 224 2 [32] as  .

Original: We deﬁned µ = (cid:0) |A| − 1 , . . 
Revised: We deﬁned µ =  |A|  1 , . . .

Original: . , |A| − 1 (cid:1) , where
Revised: , |A|  , where

Original: A is the set of races.
Revised: A is the set of races.

Original: Similar to the energy weights λ i deﬁned in Eq.
Revised: Similar to the energy weights  i deﬁned in Eq.

Original: (2), we propose to rescale the trained ˆ β as λ ˆ β .
Revised: (2), we propose to rescale the trained ˆ  as  ˆ  .

Original: Figure 6 shows that PromptGen de-biases the race and age effectively.
Revised: Figure 6 shows that PromptGen de-biases the race and age effectively.

Original: Existing de-biasing baselines consider binary attributes [31].
Revised: Existing de-biasing baselines consider binary attributes [31].

Original: For a fair comparison with them, we adopted their setting to use binary classiﬁers trained on CelebA [47] for de-biasing and evaluation.
Revised: For a fair comparison with them, we adopted their setting to use binary classiﬁers trained on CelebA [47] for de-biasing and evaluation.

Original: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, we did not use the moment constraint for this experiment.
Revised: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, we did not use the moment constraint for this experiment.

Original: Instead, since PromptGen allows conditional image generation with the classiﬁer energy, we generated the same number of samples conditioned on each attribute or attribute combination.
Revised: Instead, since PromptGen allows conditional image generation with the classiﬁer energy, we generated the same number of samples conditioned on each attribute or attribute combination.

Original: Table 2 shows that PromptGen has competitive performance on de-biasing for attributes and attribute combinations.
Revised: Table 2 shows that PromptGen has competitive performance on de-biasing for attributes and attribute combinations.

Original: [Table]
Revised: [Table]

Original: 0.121 FairGen [72] 4.21 × 10 −7.07 × 10 −– 0.
Revised: 0.121 FairGen [72] 4.21 ⇥ 10 7.07 ⇥ 10 – 0.

Original: 0.00185 FairStyle [31] 3.20 × 10 − – 0.
Revised: 0.00185 FairStyle [31] 3.20 ⇥ 10  – 0.

Original: PromptGen (ours) 1.71 × 10 −1.72 × 10 −0.
Revised: PromptGen (ours) 1.71 ⇥ 10 1.72 ⇥ 10 0.

Revised: StyleGAN

Original: [Table]
Revised: [Table]

Original: Resolution
Revised: Resolution

Original: [Table]
Revised: [Table]

Original: 4.3 Pose-Guided Face Synthesis
Revised: 4.3 Pose-Guided Face Synthesis

Original: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.
Revised: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.

Original: We used the DECA model [16], which infers the parameters of FLAME [44], a parametric facial graphics model.
Revised: We used the DECA model [16], which infers the parameters of FLAME [44], a parametric facial graphics model.

Original: We set ρ ∈ SO (3) as FLAME’s three neck poses and used the conditional INN extension introduced in Section 3.2.
Revised: We set ⇢ 2 SO (3) as FLAME’s three neck poses and used the conditional INN extension introduced in Section 3.2.

Original: To enable generating different poses of the same identity (ID), we propose ID energy using the IR-SE50 model [8].
Revised: To enable generating different poses of the same identity (ID), we propose ID energy using the IR-SE50 model [8].

Original: Speciﬁcally, given a canonical pose ρ 0 , we deﬁne z 0{+
+} =
Revised: Speciﬁcally, given a canonical pose ⇢ 0 , we deﬁne z {+
+} =

Original: f θ ( (cid:15) , ρ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)
Revised: f ✓ ( ✏ , ⇢ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)

Original: [Equation]
Revised: [Equation]

Original: where R is the IR-SE50 model [8] that computes face embeddings.
Revised: where R is the IR-SE50 model [8] that computes face embeddings.

Original: Figure 7 shows that PromptGen generates faces of the same ID in different poses, even without being explicitly trained with poses as conditions.
Revised: Figure 7 shows that PromptGen generates faces of the same ID in different poses, even without being explicitly trained with poses as conditions.

Original: We computed the FID score [24] of each model using Clean-FID [55]; following [36], we did not use the truncation trick when computing the FID score.
Revised: We computed the FID score [24] of each model using Clean-FID [55]; following [36], we did not use the truncation trick when computing the FID score.

Original: Table 3 shows that PromptGen outperforms existing models in terms of the FID score.
Revised: Table 3 shows that PromptGen outperforms existing models in terms of the FID score.

Original: Following [69], we then reported the average
Revised: Following [69], we then reported the average

Original: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.
Revised: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.

Original: Results show that PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.
Revised: Results show that PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.

Original: 4.4 Iterative Distributional Control via Functional Composition
Revised: 4.4 Iterative Distributional Control via Functional Composition

Original: This section discusses an interesting bias that PromptGen reveals about the CLIP model.
Revised: This section discusses an interesting bias that PromptGen reveals about the CLIP model.

Original: Figure 8(a) shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males.
Revised: Figure 8(a) shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males.

Original: This bias should not be attributed to image pre-training data since images contain a bias in the opposite direction, i.e., men are less likely to have makeup.
Revised: This bias should not be attributed to image pre-training data since images contain a bias in the opposite direction, i.e., men are less likely to have makeup.

Original: We argue that this bias should be explained by
Revised: We argue that this bias should be explained by

Original: CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likely to say “a person without makeup” when the person is a female (detailed analysis in Appendix F).
Revised: CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likely to say “a person without makeup” when the person is a female (detailed analysis in Appendix F).

Original: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterative control, enabled by the functional composition in Algorithm 1.
Revised: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterative control, enabled by the functional composition in Algorithm 1.

Original: Speciﬁcally, in the second iteration, we de-biased the gender distribution of G ◦ f θ instead of G , where f θ is the INN learned for the text control.
Revised: Speciﬁcally, in the second iteration, we de-biased the gender distribution of G  f ✓ instead of G , where f ✓ is the INN learned for the text control.

Original: For de-biasing, we used the moment constraint with ˆ β trained for G ◦ f θ .
Revised: For de-biasing, we used the moment constraint with ˆ  trained for G  f ✓ .

Original: Figure 8(b) shows that females and males are uniformly distributed after the moment constraint in the second iteration.
Revised: Figure 8(b) shows that females and males are uniformly distributed after the moment constraint in the second iteration.

Original: (a) PromptGen (iteration 1) with text a photo of a person without makeup.
Revised: (a) PromptGen (iteration 1) with text a photo of a person without makeup.

Original: Gender distribution: female: 81.6%; male: 18.4%.
Revised: Gender distribution: female: 81.6%; male: 18.4%.

Original: (b) PromptGen (iteration 2).
Revised: (b) PromptGen (iteration 2).

Original: Gender de-biasing ( λ =[Table] [Table]
Revised: Gender de-biasing (  =

Original: [Table] [Table] Besides revealing this “reportingbias”, PromptGen can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution
Revised: revealing this “reporting also mitigate this bias via an iterative control

Original: 4.5 Inference Latency
Revised: 4.5 Inference Latency

Original: We compared our PromptGen and the plug-and-play generative model (PPGM) [49] in terms of the inference latency.
Revised: We compared our PromptGen and the plug-and-play generative model (PPGM) [49] in terms of the inference latency.

Original: PPGM uses Langevin dynamics [76] to optimize over the latent space at inference, while PromptGen samples images in a feed-forward manner.
Revised: PPGM uses Langevin dynamics [76] to optimize over the latent space at inference, while PromptGen samples images in a feed-forward manner.

Original: In this experiment, we used PPGM and our PromptGen to approximate the same EBM with CLIP energy.
Revised: In this experiment, we used PPGM and our PromptGen to approximate the same EBM with CLIP energy.

Original: Inference times were estimated on an NVIDIA RTX A4000 GPU.
Revised: Inference times were estimated on an NVIDIA RTX A4000 GPU.

Original: Table 4 shows that our PromptGen has the highest performance and efﬁciency.
Revised: Table 4 shows that our PromptGen has the highest performance and efﬁciency.

Original: Compared to our PromptGen, PPGM requires 100 × inference time to achieve comparable results.
Revised: Compared to our PromptGen, PPGM requires 100 ⇥ inference time to achieve comparable results.

Original: Moreover, learning f θ in PromptGen requires training-time optimization, and this amortized optimization is useful when one wants to reuse a controlled distribution many times.
Revised: Moreover, learning f ✓ in PromptGen requires training-time optimization, and this amortized optimization is useful when one wants to reuse a controlled distribution many times.

Original: [Table][Table]
Revised: [Table]

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributional control of pre-trained generative models, by leveraging the knowledge of other off-the-shelf models.
Revised: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributional control of pre-trained generative models, by leveraging the knowledge of other off-the-shelf models.

Original: Unlike previous approaches, which require optimization for generating each sample at inference
Revised: Unlike previous approaches, which require optimization for generating each sample at inference

Original: [Table]
Revised: [Table]

Original: PromptGen trains an invertible neural network to approximate the latent distribution.
Revised: PromptGen trains an invertible neural network to approximate the latent distribution.

Original: Thus, PromptGen can be viewed as an amortization of the inference-time optimization into a feed-forward neural network, allowing for efﬁcient, feed-forward sampling at inference.
Revised: Thus, PromptGen can be viewed as an amortization of the inference-time optimization into a feed-forward neural network, allowing for efﬁcient, feed-forward sampling at inference.

Original: Further, this amortization also makes PromptGen not require the availability of those off-the-shelf models at inference.
Revised: Further, this amortization also makes PromptGen not require the availability of those off-the-shelf models at inference.

Original: In practice,
Revised: In practice,

Original: PromptGen offers generality for algorithmic design and modularity for control composition, and it also enables iterative controls via functional composition.
Revised: PromptGen offers generality for algorithmic design and modularity for control composition, and it also enables iterative controls via functional composition.

Original: Experiments validate that PromptGen applies to various generative models (StyleGAN2, StyleNeRF, diffusion autoencoder, and NVAE), control types (continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inverse graphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).
Revised: Experiments validate that PromptGen applies to various generative models (StyleGAN2, StyleNeRF, diffusion autoencoder, and NVAE), control types (continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inverse graphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).

Original: Limitation and future work:
Revised: Limitation and future work:

Original: We provide an error analysis in Appendix F and a discussion on societal impact in Section 6.
Revised: We provide an error analysis in Appendix F and a discussion on societal impact in Section 6.

Original: PromptGen is restricted by the pre-trained generative model’s coverage [3, 28].
Revised: PromptGen is restricted by the pre-trained generative model’s coverage [3, 28].

Original: Notably, since PromptGen focuses on mode-seeking and reweighting instead of domain adaptation, the support set of the output distribution will not change after PromptGen learning.
Revised: Notably, since PromptGen focuses on mode-seeking and reweighting instead of domain adaptation, the support set of the output distribution will not change after PromptGen learning.

Original: We ﬁnd it interesting to further explore combining PromptGen and domain adaptation [17] to achieve better generalization and adaptability.
Revised: We ﬁnd it interesting to further explore combining PromptGen and domain adaptation [17] to achieve better generalization and adaptability.

Original: PromptGen depends on the off-the-shelf models that provide knowledge about the control (Appendix F); it can be beneﬁcial to explore how to learn energy functions [15, 27] (besides our moment constraint) for ﬁne-grained control with less bias.
Revised: PromptGen depends on the off-the-shelf models that provide knowledge about the control (Appendix F); it can be beneﬁcial to explore how to learn energy functions [15, 27] (besides our moment constraint) for ﬁne-grained control with less bias.

Original:  Societal Impact
Revised:  Societal Impact

Original: Like any technology, PromptGen has both beneﬁts and drawbacks for society.
Revised: Like any technology, PromptGen has both beneﬁts and drawbacks for society.

Original: On the positive side, we have demonstrated that PromptGen can uncover biases learned by text-image models (like CLIP) and to de-bias text-image models and pre-trained generative models, suggesting that PromptGen may be a helpful tool for fair AI if used properly.
Revised: On the positive side, we have demonstrated that PromptGen can uncover biases learned by text-image models (like CLIP) and to de-bias text-image models and pre-trained generative models, suggesting that PromptGen may be a helpful tool for fair AI if used properly.

Original: The efﬁcient inference of PromptGen also helps reduce the computational expense, giving a positive impact on the environment.
Revised: The efﬁcient inference of PromptGen also helps reduce the computational expense, giving a positive impact on the environment.

Original: However, improved controllability makes it easier to synthesize targeted images; for instance, the creation of deceptive media such as DeepFakes [77, 74] and privacy leakage.
Revised: However, improved controllability makes it easier to synthesize targeted images; for instance, the creation of deceptive media such as DeepFakes [77, 74] and privacy leakage.

Original: To battle these cases, we expect researchers and practitioners to use technologies that can detect fake media and mitigate privacy leakage.
Revised: To battle these cases, we expect researchers and practitioners to use technologies that can detect fake media and mitigate privacy leakage.

Original: We also encourage practitioners to consider these risks when using PromptGen to develop systems.
Revised: We also encourage practitioners to consider these risks when using PromptGen to develop systems.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: The authors would like to thank the anonymous reviewers, Zoltán Ádám Milacski, Jianchun Chen, and Shubhra Aich for their valuable feedback on drafts of this paper.A PromptGen on Synthetic Data
Revised: The authors would like to thank the anonymous reviewers, Zoltán Ádám Milacski, Jianchun Chen, and Shubhra Aich for their valuable feedback on drafts of this paper.

Original: We demonstrate the behavior of our PromptGen with two-dimensional synthetic data, using GAN as the generative model.

Original: This synthetic experiment is illustrated in Figure 9.

Original: Speciﬁcally, we created a “real” distribution that is a Gaussian mixture distribution biased towards one mode.

Original: We derive a closed-form fair classiﬁer based on this distribution, detailed in the purple block in Figure 9.

Original: When a

Original: GAN is trained to approximate this distribution, its outputs are biased towards the over-represented mode, as shown by Figure 9(b).

Original: Figure 9(c) illustrates the controllability experiment: using the fair classiﬁer as control, PromptGen learns a distribution concentrated in a speciﬁc region of the output space.

Original: Figure 9(d) illustrates the de-biasing experiment: using the moment constraint (Section 3.

Original: and Section 4.2), PromptGen upweights the under-represented regions of the output space.

Original: B Method Details and Derivations B.1 Controllability as EBM

Original: In Section 3, we deﬁned a control C as M independent properties { y 1 , . . .

Original: , y M } .

Original: For example, y 1 can be a text description, and y 2 can be an attribute.

Original: In this part, we ﬁrst elaborate on why controllability can be formed as EBMs in Eq.

Original: We then provide concrete examples of the distributions derived from different energy functions deﬁned in Section 3.1.

Original: We denote the image prior as p x ( x ) , the only distribution that can be estimated from data when labels for the control are not provided during generative model pre-training.

Original: Given the control

Original: C = { y 1 , . . .

Original: , y M } , we resort to Bayes’ theorem to rewrite the conditional distribution p ( x |C ) as

Original: [Equation]

Original: For Eq.

Original: (15) to be well-deﬁned, we need to deﬁne p ( y i | x ) for each y i .

Original: In order to incorporate the knowledge of arbitrary off-the-self models (besides image classiﬁers), we deﬁne each p ( y i | x ) as

Original: [Equation]

Original: [Equation]

Original: Eq.

Original: (16) says that p ( y i | x ) is proportional to the exponential of an energy function

Original: E i ( x , y i ) , where y i with lower energy has higher density or mass.

Original: Note that Eq.

Original: (16) deﬁnes a distribution over all possible values of y i instead of all possible values of x .

Original: Combining Eq.

Original: (15) and Eq.

Original: (16), we have p

Original: [Equation]

Original: which is the same equation as Eq.

Original: In the following, we use Eq.

Original: (16) to derive the distributions from the classiﬁer energy, CLIP energy, and inverse graphics energy deﬁned in Section 3.1.

Original: Classiﬁer energy:

Original: Given a classiﬁer P ( ·| x ) and the target class a , we deﬁne the classiﬁer energy as

Original: E classiﬁer ( x , a ) = − log P ( a | x ) .

Original: Using Eq.

Original: (16), we arrive at:

Original: [Equation]

Original: which is equivalent to a temperature-adjusted distribution of the original classiﬁer.

Original: CLIP energy: Using Eq.

Original: (16), the CLIP energy in Eq.

Original: (3) is equivalent to

Original: [Equation]

Original: Inverse graphics energy: Using Eq.

Original: (16), the inverse graphics energy in Eq.

Original: (4) is equivalent to

Original: [Equation]

Original: If the geodesic distance d (cid:10) · , · (cid:11) is the Euclidean distance, then p inv-graphics ( ρ | x ) is a Gaussian distribution whose mean is ρ and whose variance depends on the hyperparameter λ inv-graphics ; if the geodesic distance d (cid:10) · , · (cid:11) is the spherical distance (e.g., the distance between pose parameters deﬁned on a unit sphere), then p inv-graphics ( ρ | x ) is a vMF distribution.

Original: B.2 Equivalence between Image-Space EBM and Latent-Space EBM

Original: Proposition 1.

Original: Deﬁne x ∼ p x ( x ) as z ∼ p

Original: z

Original: ( z ) , x = G ( z ) and p ( x |C ) as z ∼ p ( z |C ) , x = G ( z ) , where p ( z |C ) is deﬁned as the following EBM:

Original: [Equation]

Original: We have

Original: [Equation]

Original: In spirit, our proof follows the proof in [52], which follows [19].

Original: The difference between our proof and that in [52] is that we derive p ( x |C ) from p ( z |C ) while they derived p ( z |C ) from p ( x |C ) .

Original: Proof.

Original: Based on Lemma 1 in [19] and Lemma 1 in [52], p ( z |C ) is equivalent to rejection sampling with proposal distribution p z ( z ) and acceptance probability

Original: [Equation]

Original: where

Original: [Equation]

Original: [Equation]

Original: Since p ( x |C ) is deﬁned as z ∼ p

Original: z ( z |C ) , x = G ( z ) , p ( x |C ) is equivalent to rejection sampling with proposal distribution p z ( z ) , x = G ( z ) and acceptance probability

Original: [Equation]

Original: [Equation]

Original: Note that the above proposal distribution

Original: z ∼ p

Original: z

Original: ( z ) , x = G ( z ) is the same as p x ( x ) by deﬁnition.

Original: Based on Lemma 1 in [19] and Lemma 1 in [52], we arrive at:

Original: [Equation]

Original: B.3 Derivation of Eq. (9): Approximating EBM with INN

Original: The full derivation of Eq.

Original: (9) is given by:

Original: [Equation]

Original: B.4 Derivation of Eq. (11): Extension to Generative Models with a Class-Embedding Space

Original: The full derivation of Eq.

Original: (11) is given by:

Original: [Equation]

Original: y are independent

Original: [Equation]

Original: , z

Original: [Equation]

Original: , y

Original: [Equation]

Original: h

Original: [Equation]

Original: , x

Original: [Equation]

Original: y

Original: [Equation]

Original: − log |

Original: [Equation]

Original: ∂f θ

Original: [Equation]

Original: ) | − log | det

Original: [Equation]

Original: ∂h θ

Original: [Equation]

Original: ) |

Original: [Equation]

Original: while not converged do

Original: Sample (cid:15) ∼ N ( 0 , I ) , ρ ∼ p ρ

Original: [Equation]

Original: ρ

Original: [Equation]

Original: Map (cid:15) to latent code z = f θ ( (cid:15) , ρ ) 3.

Original: Map z to an image x = G ( z ) 4.

Original: Optimize θ with gradient

Original: [Equation]

Original: B.5 Extension to Conditional INN

Original: Algorithm 3 illustrates how we extend the training algorithm

Original: (Algorithm 2) to conditional INN, which helps generalize to controls speciﬁed by continuous values ρ .

Original: The training objective becomes

Original: [Equation]

Original: where C ρ means that the control is speciﬁed by value ρ .

Original: The derivation of Algorithm 3 is given by:

Original: [Equation]

Original: B.6 Derivation for Latent-Space Moment Constraint

Original: Given a mapping γ : X → R K , moment constraint [7, 37] deﬁnes the target distribution p ( x |C ) as

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: where µ is the user-speciﬁed constraint.

Original: Examples are provided in the main text

Original: [Equation]

Original: and we omit them here for brevity.

Original: [7] showed that Eq.

Original: (33) can be approximated to an arbitrary precision by

Original: [Equation]

Original: (34) where ˆ β needs to be computed.

Original: [37] estimates ˆ β by solving the following regression problem:

Original: [Equation]

Original: In [37], sampling x ∼ p x ( x ) is straightforward since they focus on autoregressive language models.

Original: In the context of latent-variable vision generative models, where x ∼ p x ( x ) is implicitly deﬁned as z ∼ p

Original: z ( z ) , x = G ( z ) , Eq.

Original: (35) is equivalent to

Original: [Equation]

Original: arg min

Original: [Equation]

Original: [Equation]

Original: Finally, based on Proposition 1, we can generalize the moment constraint to the latent space as

Original: [Equation]

Original: Note that Eq.

Original: (36) and Eq.

Original: (37) are verbatim copies of Eq.

Original: (7) and Eq.

Original: (6), respectively.

Original: Algorithm 4:

Original: 4.3) while not converged do

Original: Sample (cid:15) ∼ N ( 0 , I ) , ρ ∼ p ρ ( ρ ) 2.

Original: Map (cid:15) to latent codes z = f θ ( (cid:15) , ρ ) and z 0 = f

Original: θ ( (cid:15) , ρ 0 ) 3.

Original: Map z to an image x = G ( z ) and z 0 to an image x 0 =

Original: G

Original: ( z 0 ) 4.

Original: Optimize θ with gradient

Original: [Equation]

Original: B.7 Inverse Graphics Control with Identity Energy

Original: This subsection provides more detail on how we use an inverse graphics model, DECA [16], to control the pose of faces generated by StyleGAN2 trained on FFHQ 1024 2 , as introduced in Section 4.3.

Original: Recall that we use the conditional INN extension, detailed in Algorithm 3, for this experiment.

Original: To enable generating different poses of the same identity, we add additional identity energy using the

Original: IR-SE50 model [8].

Original: The training algorithm is detailed in Algorithm 4.

Original: [-Speciﬁcally, we generate a canonical latent code z 
-]
Revised: [-Speciﬁcally, we generate a canonical latent code z 
-]

Original: =

Original: f

Original: θ ( (cid:15) , ρ 0 ) for each (cid:15) , where ρ 0 is the canonical pose.

Original: The latent code z and the canonical latent code z 0 are mapped to the image x

Original: and the canonical image x 0 .

Original: The identity energy in Eq.

Original: (12), copied below, encourages the embeddings of the two images to be similar:

Original: [Equation]

Original: B.8 Experimental Details

Original: Our INN architecture contains 8 blocks.

Original: Each block consists of a soft permutation of channels [2], an afﬁne coupling layer [12], and an ActNorm layer [39].

Original: In the afﬁne coupling layer, we model the sub-network as an MLP with one hidden layer, where the hidden dimension is 256 and the non-linear activation is LeakyReLU 0 .

Original: For the prior distribution p ρ ( ρ ) of the pose parameter ρ in the inverse graphics experiments (Section 4.3), we sample the x -axis and y -axis rotations relative to a canonical pose ρ 0 , whose x -axis and y -axis rotations are 0 .

Original: 3 and 0 , respectively.

Original: The relative pose’s x

Original: -axis and y -axis rotations are uniformly sampled from [ − (1 / 9) π, (1 / 9) π ] .

Original: Each experiment was run on an NVIDIA RTX A4000 GPU (with 16G memory).

Original: Our code implementation is based on the PyTorch framework.

Original: Our code can be found at https://github.com/

Original: ChenWu98/Generative-Visual-Prompt .

Original: [Figure]

Original: C Additional Results for Image Synthesis based on Text Description

Original: Figure 10 presents additional results of PromptGen for image synthesis from text, using the CLIP model as control.

Original: As the pre-trained generative model, we explore StyleGAN2 trained on AFHQ-Cats [6], LSUN-Churches [83], and FFHQ [35].

Original: D Additional Results for De-Biasing Generative Models

Original: Table 5 provides quantitative results of de-biasing StyleGAN2 across categorical attributes (Figure 6).

Original: Speciﬁcally, we used the pre-trained classiﬁer provided by FairFace [32] to classify the attributes of the generated images.

Original: We then report the KL divergence between the attribute distribution of the generated images and the uniform distribution.

Original: Figure 11 and Figure 12 visualize the de-biasing results.

Original: We report the KL divergence between the generated distribution and the uniform distribution.

Original: [Table]

Original: Interestingly, on MetFaces, de-biasing the race results in more sculptures and we postulate that the reason is that almost all paintings and sketches in MetFaces are for white individuals.

Original: [Figure]

Original: [Figure]

Original: r t i

Original: [Figure]

Original: D.1 Decomposing Complex Control via Energy Composition

Original: Energy composition E C ( x ) = (cid:80) Mi

Original: =1 λ i E i ( x , y i ) in Eq.

Original: 1 allows us to decompose a complex controls into simple ones.

Original: [-For instance, Figure 13(a) shows that y 
-]
Revised: [-For instance, Figure 13(a) shows that y 
-]

Original: [-= photo of a bald black man with beard ( λ 1 = 6000 ) generates good samples of men with beard but several of them have light-skin tone; however, by decomposing it as y 
-]
Revised: [-= photo of a bald black man with beard ( λ 1 = 6000 ) generates good samples of men with beard but several of them have light-skin tone; however, by decomposing it as y 
-]

Original: [-= photo of a bald man ( λ 1 = 1500 ), y 2 = photo of a black man ( λ 1 = 3000 ), and y 
-]
Revised: [-= photo of a bald man ( λ 1 = 1500 ), y 2 = photo of a black man ( λ 1 = 3000 ), and y 
-]

Original: = photo of a man with beard ( λ 1 = 1500 ), we have (on average) darker skin, as shown in Figure 13(b).

Original: Note that the random seed for Figure 13(a) and Figure 13(b) is the same, and hence the identities are preserved and mostly the skin tone has changed.

Original: [Figure]

Original: E Experiments on Generative 3D Face Models

Original: 3D face modeling has been an active area of research that has recently gained major interest due to applications in virtual humans, deep faces, and digital actors.

Original: Existing 3D deep learning generative models build 3D compact representations of shape and appearance capable of modeling non-linearity (e.g., scatter effects, specularities) that is necessary for generating photo-realistic faces.

Original: Providing a generative model with the capability of generating 3D faces with a particular geometry (e.g., a speciﬁc distance between eyes or nose length) is particularly useful for technical artists when creating new characters.

Original: However, this problem remains unaddressed, partially due to the lack of publicly available 3D databases labeled with such constraints.

Original: We refer to this capability as the intra-sample constraint in 3D generative models, and this section shows how our PromptGen framework is able to achieve this by deﬁning a new energy function based on signed distances.

Original: We conducted the experiments on the FaceScape dataset [82], a large-scale 3D human face dataset consisting of 16 , 940 topologically uniformly registered 3D face meshes along with high-quality textures.

Original: The dataset contains 847 identities with 20 expressions per identity, and each mesh consists of 26 , 317 vertices of which 68 are 3D landmark vertices covering eyes, nose, mouth, jaw, and eyebrows.

Original: With some abuse of notation, we denote a 3D face mesh as x ∈ R 3 V (recall that x is used to represent an image in other parts of this paper), where V = 26 , 317 is the number of vertices.

Original: Given an index l of a vertex (e.g., one of the 68 landmark vertices), the 3D coordinate of this vertex is deﬁned as x [ l ] ∈ R 3 .

Original: We pre-train a generative mesh model G that maps each latent code z ∼ N ( 0 , I ) to a 3D mesh x .

Original: PromptGen allows us to obtain geometric control on the 3D meshes.

Original: Speciﬁcally, given two landmark vertex indices l 1 and l 2 , we deﬁne an intra-sample constraint that enforces the signed distance (along a given direction) between the two vertices to be s .

Original: To this effect, we deﬁne the signed-distance energy as

Original: [Equation]

Original: where d ( · , · ) is the signed-distance (along a given direction), and | · | is the absolute value.

Original: Figure 14 presents several examples of 3D mesh control, in which random textures are applied to the generated meshes.

Original: It shows that PromptGen successfully controls the generative 3D mesh model in terms of the mouth stretch, eye openness, nose-tip stretch, and lip forward.

Original: F Error Analysis

Original: As we discussed in Section 5, the controlled distribution depends on (1) the pre-trained generative model’s coverage and (2) the off-the-shelf models used for the control.

Original: In this section, we provide

Original: V e r ti ca l m ou t h s t r e t

Original: c h

Original: Horizontal mouth stretch

Original: [Figure]

Original: E y e c l o s e

Original: [Figure]

Original: N o s eti p s t r e t c h

Original: [Figure]

Original: some examples of PromptGen failing to generate satisfactory images using the CLIP energy.

Original: For the pre-trained generative model, we use StyleGAN2 trained on FFHQ, AFHQ-Wild, or LSUN-Churches.

Original: Our ﬁrst observation is that the CLIP model sometimes fails in modeling linguistic negation.

Original: For instance, the text description photo of a man without beard results in a distribution of photos of men with beard (Figure 15(a)).

Original: Meanwhile, Figure 15(b) shows that CLIP is capable of modeling the negation of having makeup, but with the “reporting bias” discussed in Section 4.4.

Original: Moreover, CLIP seems to have difﬁculty in numerical reasoning, and gaining control over the count of speciﬁc objects in a scene tends to be unsuccessful.

Original: We showed this by specifying photo of a church with three windows, which did not result in the desired speciﬁcation (Figure 15(c)).

Original: To gain a deeper understanding of the above failures and biases, in Figure 16, we provide some image samples from the LAION-400M dataset [65] with CLIP retrieval, using the same text descriptions as

Original: Figure 15.

Original: CLIP retrieval is based on the CLIP embedding similarity between the web images and text descriptions, while the original text below each individual image is not used.

Original: We observe an impressive consistency between CLIP retrieval and PromptGen: in both Figure 15(a) and Figure 16(a), most images have beard; in both Figure 15(b) and Figure 16(b), all images are female; in both Figure 15(c) and Figure 16(c), some images do not have exactly three windows.

Original: This consistency suggests that

Original: [Figure]

Original: and

Original: Figure mostly attributed the CLIP model algorithm believe contrastive multi-modal (

Original: -language) pre-training, which is worthy of being further investigated

Original: Another observation is that the control tends to fail when the text description requires sampling from low-density regions of the pre-trained generative model’s output space.

Original: In other words, the control usually fails if the pre-trained generative model does not cover the mode we are trying to gain control over.

Original: For example, images faithful to photo of a person yawning and photo of a baby with long hair are not commonly observed in the FFHQ dataset and, hence, these two text descriptions result in degeneration (Figure 17(a)) or weird images (Figure 17(b)).

Original: Another example is photo of an animal from the side, which is not commonly observed in the AFHQ-Wild dataset, and Figure 17(c) shows that the generated images fail to follow this description.

Original: Even when the control is successful (e.g., when a complex description is decomposed in Figure 13(b)), sampling from low-density regions results in limited diversity (e.g., the backgrounds in Figure 13(b) look similar to each other).

Original: Finally, we also ran some failure controls using PPGM, showing that PromptGen and PPGM reveal similar failure cases of pre-trained generative models and CLIP.

Original: Results are shown in Figure 18.

Original: [Figure]

Original: [Figure]

Original: [Figure]