Original: D ISCRETE C ONTRASTIVE D IFFUSION FOR C ROSSM ODAL M USIC AND I MAGE G ENERATION
Revised: D ISCRETE C ONTRASTIVE D IFFUSION FOR C ROSSM ODAL M USIC AND I MAGE G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Diffusion probabilistic models (DPMs) have become a popular approach to conditional generation, due to their promising results and support for cross-modal synthesis.
Revised: Diffusion probabilistic models (DPMs) have become a popular approach to conditional generation, due to their promising results and support for cross-modal synthesis.

Original: A key desideratum in conditional synthesis is to achieve high correspondence between the conditioning input and generated output.
Revised: A key desideratum in conditional synthesis is to achieve high correspondence between the conditioning input and generated output.

Original: Most existing methods learn such relationships implicitly, by incorporating the prior into the variational lower bound.
Revised: Most existing methods learn such relationships implicitly, by incorporating the prior into the variational lower bound.

Original: In this work, we take a different route—we explicitly enhance input-output connections by maximizing their mutual information.
Revised: In this work, we take a different route—we explicitly enhance input-output connections by maximizing their mutual information.

Original: To this end, we introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss and design two contrastive diffusion mechanisms to effectively incorporate it into the denoising process, combining the diffusion training and contrastive learning for the first time by connecting it with the conventional variational objectives.
Revised: To this end, we introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss and design two contrastive diffusion mechanisms to effectively incorporate it into the denoising process, combining the diffusion training and contrastive learning for the first time by connecting it with the conventional variational objectives.

Original: We demonstrate the efficacy of our approach in evaluations with diverse multimodal conditional synthesis tasks: dance-to-music generation, text-to-image synthesis, as well as class-conditioned image synthesis.
Revised: We demonstrate the efficacy of our approach in evaluations with diverse multimodal conditional synthesis tasks: dance-to-music generation, text-to-image synthesis, as well as class-conditioned image synthesis.

Original: On each, we enhance the inputoutput correspondence and achieve higher or competitive general synthesis quality.
Revised: On each, we enhance the inputoutput correspondence and achieve higher or competitive general synthesis quality.

Original: Furthermore, the proposed approach improves the convergence of diffusion models, reducing the number of required diffusion steps by more than 35% on two benchmarks, significantly increasing the inference speed
Revised: Furthermore, the proposed approach improves the convergence of diffusion models, reducing the number of required diffusion steps by more than 35% on two benchmarks, significantly increasing the inference speed

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative tasks that seek to synthesize data in different modalities, such as audio and images, have attracted much attention.
Revised: Generative tasks that seek to synthesize data in different modalities, such as audio and images, have attracted much attention.

Original: The recently explored diffusion probabilistic models (DPMs) Sohl-Dickstein et al.
Revised: The recently explored diffusion probabilistic models (DPMs) Sohl-Dickstein et al.

Original: (2015b) have served as a powerful generative backbone that achieves promising results in both unconditional and conditional generation Kong et al.
Revised: (2015b) have served as a powerful generative backbone that achieves promising results in both unconditional and conditional generation Kong et al.

Original: Mittal et al.
Revised: Mittal et al.

Original: ; Lee & Han (2021);
Revised: ; Lee & Han (2021);

Original: Ho et al.
Revised: Ho et al.

Original: (2020); Nichol & Dhariwal (2021); Dhariwal & Nichol (2021);
Revised: (2020); Nichol & Dhariwal (2021); Dhariwal & Nichol (2021);

Original: Ho et al.
Revised: Ho et al.

Original: ; Hu et al.
Revised: ; Hu et al.

Original: Compared to the unconditional case, conditional generation is usually applied in more concrete and practical cross-modality scenarios, e.g. , video-based music generation Di et al.
Revised: Compared to the unconditional case, conditional generation is usually applied in more concrete and practical cross-modality scenarios, e.g. , video-based music generation Di et al.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2022); Gan et al.
Revised: (2022); Gan et al.

Original: (2020a)
Revised: (2020a)

Original: and text-based image generation Gu et al.
Revised: and text-based image generation Gu et al.

Original: ; Ramesh et al.
Revised: ; Ramesh et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Ruan et al.
Revised: ; Ruan et al.

Original: Most existing DPM-based conditional synthesis works Gu et al.
Revised: Most existing DPM-based conditional synthesis works Gu et al.

Original: ; Dhariwal & Nichol (2021) learn the connection between the conditioning and the generated data implicitly by adding a prior to the variational lower bound Sohl-Dickstein et al.
Revised: ; Dhariwal & Nichol (2021) learn the connection between the conditioning and the generated data implicitly by adding a prior to the variational lower bound Sohl-Dickstein et al.

Original: (2015b).
Revised: (2015b).

Original: While such approaches still feature high generation fidelity, the correspondence between the conditioning and the synthesized data can sometimes get lost, as illustrated in the right column in
Revised: While such approaches still feature high generation fidelity, the correspondence between the conditioning and the synthesized data can sometimes get lost, as illustrated in the right column in

Original: Fig.
Revised: Fig.

Original: To this end, we aim to explicitly enhance the input-output faithfulness via their maximized mutual information under the diffusion generative framework for conditional settings in this paper.
Revised: To this end, we aim to explicitly enhance the input-output faithfulness via their maximized mutual information under the diffusion generative framework for conditional settings in this paper.

Original: Examples of our synthesized music audio and image results are given in Fig.
Revised: Examples of our synthesized music audio and image results are given in Fig.

Original: Contrastive methods Oord et al.
Revised: Contrastive methods Oord et al.

Original: ; Bachman et al.
Revised: ; Bachman et al.

Original: (2019); Song & Ermon (2020a) have been proven to be very powerful for data representation learning.
Revised: (2019); Song & Ermon (2020a) have been proven to be very powerful for data representation learning.

Original: Their high-level idea aims to learn the representation z of raw data x based on the assumption that a properly encoded z benefits the ability of a generative model p to reconstruct the raw data given z as prior.
Revised: Their high-level idea aims to learn the representation z of raw data x based on the assumption that a properly encoded z benefits the ability of a generative model p to reconstruct the raw data given z as prior.

Original: This idea can be achieved via optimization of the density ratio p ( x | z ) p ( x )
Revised: This idea can be achieved via optimization of the density ratio p ( x | z ) p ( x )

Original: Oord et al.
Revised: Oord et al.

Original: (2018) as an entirety, without explicitly modeling the actual generative model p .
Revised: (2018) as an entirety, without explicitly modeling the actual generative model p .

Original: While the direct optimization of mutual information via generative models p is a challenging problem to implement and train Song & Ermon (2020b); Belghazi et al.
Revised: While the direct optimization of mutual information via generative models p is a challenging problem to implement and train Song & Ermon (2020b); Belghazi et al.

Original: (2018) in the conventional contrastive representation learning field, we show that this can be effectively done within our proposed contrastive diffusion framework.
Revised: (2018) in the conventional contrastive representation learning field, we show that this can be effectively done within our proposed contrastive diffusion framework.

Original: Specifically, we reformulate
Revised: Specifically, we reformulate

Original: T e x t - t o -I m a g e
Revised: T e x t - t o -I m a g e

Original: C l a ss - t
Revised: C l a ss - t

Original: o -I m a g e
Revised: o -I m a g e

Original: D a n ce - t oM u
Revised: D a n ce - t oM u

Original: s i c
Revised: s i c

Original: Input
Revised: Input

Original: Outputs from Our Contrastive Diffusion
Revised: Outputs from Our Contrastive Diffusion

Original: Outputs w/ Weaker Correspondence the optimization problem for the desired conditional generative tasks via DPMs by analogy to the above embedding z and raw data x with our conditioning input and synthesized output.
Revised: Outputs w/ Weaker Correspondence the optimization problem for the desired conditional generative tasks via DPMs by analogy to the above embedding z and raw data x with our conditioning input and synthesized output.

Original: We introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss, and design two contrastive diffusion mechanisms - step-wise parallel diffusion that invokes multiple parallel diffusion processes during contrastive learning, and sample-wise auxiliary diffusion , which maintains one principal diffusion process, to effectively incorporate the CDCD loss into the denoising process.
Revised: We introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss, and design two contrastive diffusion mechanisms - step-wise parallel diffusion that invokes multiple parallel diffusion processes during contrastive learning, and sample-wise auxiliary diffusion , which maintains one principal diffusion process, to effectively incorporate the CDCD loss into the denoising process.

Original: We demonstrate that with the proposed contrastive diffusion method, we can not only effectively train so as to maximize the desired mutual information by connecting the CDCD loss with the conventional variational objective function, but also to directly optimize the generative network p .
Revised: We demonstrate that with the proposed contrastive diffusion method, we can not only effectively train so as to maximize the desired mutual information by connecting the CDCD loss with the conventional variational objective function, but also to directly optimize the generative network p .

Original: The optimized CDCD loss further encourages faster convergence of a DPM model with fewer diffusion steps.
Revised: The optimized CDCD loss further encourages faster convergence of a DPM model with fewer diffusion steps.

Original: We additionally present our intra- and inter- negative sampling methods by providing internally disordered and instance-level negative samples, respectively.
Revised: We additionally present our intra- and inter- negative sampling methods by providing internally disordered and instance-level negative samples, respectively.

Original: To better illustrate the input-output connections, we conduct main experiments on the novel crossmodal dance-to-music generation task Zhu et al.
Revised: To better illustrate the input-output connections, we conduct main experiments on the novel crossmodal dance-to-music generation task Zhu et al.

Original: (2022), which aims to generate music audio based on silent dance videos.
Revised: (2022), which aims to generate music audio based on silent dance videos.

Original: Compared to other tasks such as text-to-image synthesis, dance-to-music generation explicitly evaluates the input-output correspondence in terms of various cross-modal alignment features such as dance-music beats, genre and general quality.
Revised: Compared to other tasks such as text-to-image synthesis, dance-to-music generation explicitly evaluates the input-output correspondence in terms of various cross-modal alignment features such as dance-music beats, genre and general quality.

Original: However, various generative settings, frameworks, and applications can also benefit from our contrastive diffusion approach, e.g. , joint or separate training of conditioning encoders, continuous or discrete conditioning inputs, and diverse input-output modalities as detailed in Sec. 4.
Revised: However, various generative settings, frameworks, and applications can also benefit from our contrastive diffusion approach, e.g. , joint or separate training of conditioning encoders, continuous or discrete conditioning inputs, and diverse input-output modalities as detailed in Sec. 4.

Original: Overall, we achieve results superior or comparable to state-of-the-art on three conditional synthesis tasks: dance-to-music (datasets:
Revised: Overall, we achieve results superior or comparable to state-of-the-art on three conditional synthesis tasks: dance-to-music (datasets:

Original: AIST++ Tsuchida et al.
Revised: AIST++ Tsuchida et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2021), TikTok Dance-Music Zhu et al.
Revised: (2021), TikTok Dance-Music Zhu et al.

Original: , text-to- image (datasets: CUB200 Wah et al.
Revised: , text-to- image (datasets: CUB200 Wah et al.

Original: (2011), MSCOCO Lin et al.
Revised: (2011), MSCOCO Lin et al.

Original: and class-conditioned image synthesis (dataset: ImageNet Russakovsky et al. (2015)).
Revised: and class-conditioned image synthesis (dataset: ImageNet Russakovsky et al. (2015)).

Original: Our experimental findings suggest three key take-away: 1 Improving the input-output connections via maximized mutual information is indeed beneficial for their correspondence and the general fidelity of the results (see Fig.
Revised: Our experimental findings suggest three key take-away: 1 Improving the input-output connections via maximized mutual information is indeed beneficial for their correspondence and the general fidelity of the results (see Fig.

Original: 1 and supplement).
Revised: 1 and supplement).

Original: 2 Both our proposed step-wise parallel diffusion with intra- negative samples and sample-wise auxiliary diffusion with inter- negative samples show state-of-the-art scores in our evaluations.
Revised: 2 Both our proposed step-wise parallel diffusion with intra- negative samples and sample-wise auxiliary diffusion with inter- negative samples show state-of-the-art scores in our evaluations.

Original: The former is more beneficial for capturing the intra-sample correlations, e.g. , musical rhythms, while the latter improves the instance-level performance, e.g. , music genre and image class.
Revised: The former is more beneficial for capturing the intra-sample correlations, e.g. , musical rhythms, while the latter improves the instance-level performance, e.g. , music genre and image class.

Original: 3 With maximized mutual information, our conditional contrastive diffusion converge in substantially fewer diffusion steps compared to vanilla DPMs, while maintaining the same or even superior performance (approximately 35% fewer steps for dance-to-music generation and 40% fewer for text-to-image synthesis), thus significantly increasing inference speed.
Revised: 3 With maximized mutual information, our conditional contrastive diffusion converge in substantially fewer diffusion steps compared to vanilla DPMs, while maintaining the same or even superior performance (approximately 35% fewer steps for dance-to-music generation and 40% fewer for text-to-image synthesis), thus significantly increasing inference speed.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Diffusion Probabilistic Models.
Revised: Diffusion Probabilistic Models.

Original: DPMs Sohl-Dickstein et al.
Revised: DPMs Sohl-Dickstein et al.

Original: (2015b) are a class of generative models that learn to convert a simple Gaussian distribution into a data distribution.
Revised: (2015b) are a class of generative models that learn to convert a simple Gaussian distribution into a data distribution.

Original: This process consists of a forward diffusion process and a reverse denoising process, each consisting of a sequence of T steps that act as a Markov chain.
Revised: This process consists of a forward diffusion process and a reverse denoising process, each consisting of a sequence of T steps that act as a Markov chain.

Original: During forward diffusion, an input data sample x 0 is gradually “corrupted” at each step t by adding Gaussian noise to the output of step t − 1 .
Revised: During forward diffusion, an input data sample x 0 is gradually “corrupted” at each step t by adding Gaussian noise to the output of step t − 1 .

Original: The reverse denoising process, seeks to convert the noisy latent variable x T into the original data sample x 0 by removing the noise added during diffusion.
Revised: The reverse denoising process, seeks to convert the noisy latent variable x T into the original data sample x 0 by removing the noise added during diffusion.

Original: The stationary distribution for the final latent variable x T is typically assumed to be a normal distribution, p ( x T ) =
Revised: The stationary distribution for the final latent variable x T is typically assumed to be a normal distribution, p ( x T ) =

Original: N ( x T | 0 , I ) .
Revised: N ( x T | 0 , I ) .

Original: An extension of this approach replaces the continuous state with a discrete one Sohl-Dickstein et al.
Revised: An extension of this approach replaces the continuous state with a discrete one Sohl-Dickstein et al.

Original: (2015a);
Revised: (2015a);

Original: Hoogeboom et al.
Revised: Hoogeboom et al.

Original: ; Austin et al.
Revised: ; Austin et al.

Original: (2021), in which the latent variables x 1: T typically take the form of one-hot vectors with K categories.
Revised: (2021), in which the latent variables x 1: T typically take the form of one-hot vectors with K categories.

Original: The diffusion process can then be parameterized using a multinomial categorical transition matrix defined as q ( x t | x t − 1 ) =
Revised: The diffusion process can then be parameterized using a multinomial categorical transition matrix defined as q ( x t | x t − 1 ) =

Original: Cat( x t ; p = x
Revised: Cat( x t ; p = x

Original: t − 1 Q t ) , where [ Q t ] ij = q ( x t
Revised: t − 1 Q t ) , where [ Q t ] ij = q ( x t

Original: =
Revised: =

Original: j | x t − 1 = i ) .
Revised: j | x t − 1 = i ) .

Original: The reverse process p θ ( x t | x t − 1 ) can also be factorized as conditionally independent over the discrete sequences Austin et al.
Revised: The reverse process p θ ( x t | x t − 1 ) can also be factorized as conditionally independent over the discrete sequences Austin et al.

Original: In both the continuous and discrete state formulations of DPMs Song & Ermon (2020c); Song et al.
Revised: In both the continuous and discrete state formulations of DPMs Song & Ermon (2020c); Song et al.

Original: (2020b)
Revised: (2020b)

Original: ; Kingma et al.
Revised: ; Kingma et al.

Original: ; Song et al.
Revised: ; Song et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: ; Vahdat et al.
Revised: ; Vahdat et al.

Original: (2021), the denoising process p θ can be optimized by the KL divergence between q and p θ in closed forms Song et al.
Revised: (2021), the denoising process p θ can be optimized by the KL divergence between q and p θ in closed forms Song et al.

Original: (2020a); Nichol & Dhariwal (2021);
Revised: (2020a); Nichol & Dhariwal (2021);

Original: Ho et al.
Revised: Ho et al.

Original: Hoogeboom et al.
Revised: Hoogeboom et al.

Original: ; Austin et al.
Revised: ; Austin et al.

Original: (2021) via the variational bound on the negative log-likelihood:
Revised: (2021) via the variational bound on the negative log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: Existing conditional generation works via DPMs Gu et al.
Revised: Existing conditional generation works via DPMs Gu et al.

Original: ; Dhariwal & Nichol (2021) usually learn the implicit relationship between the conditioning c and the synthesized data x 0 by directly adding the c as the prior in equation 1.
Revised: ; Dhariwal & Nichol (2021) usually learn the implicit relationship between the conditioning c and the synthesized data x 0 by directly adding the c as the prior in equation 1.

Original: DPMs with discrete state space provide more controls on the data corruption and denoising compared to its continuous counterpart Austin et al.
Revised: DPMs with discrete state space provide more controls on the data corruption and denoising compared to its continuous counterpart Austin et al.

Original: ; Gu et al.
Revised: ; Gu et al.

Original: (2022) by the flexible designs of transition matrix, which benefits for practical downstream operations such as editing and interactive synthesis Tseng et al.
Revised: (2022) by the flexible designs of transition matrix, which benefits for practical downstream operations such as editing and interactive synthesis Tseng et al.

Original: ; Cui et al.
Revised: ; Cui et al.

Original: ; Xu et al.
Revised: ; Xu et al.

Original: We hence employ contrastive diffusion using a discrete state space in this work.
Revised: We hence employ contrastive diffusion using a discrete state space in this work.

Original: Contrastive Representation Learning.
Revised: Contrastive Representation Learning.

Original: Contrastive learning uses loss functions designed to make neural networks learn to understand and represent the specific similarities and differences between elements in the training data without labels explicitly defining such features, with positive and negative pairs of data points, respectively.
Revised: Contrastive learning uses loss functions designed to make neural networks learn to understand and represent the specific similarities and differences between elements in the training data without labels explicitly defining such features, with positive and negative pairs of data points, respectively.

Original: This approach has been successfully applied in learning representations of high-dimensional data Oord et al.
Revised: This approach has been successfully applied in learning representations of high-dimensional data Oord et al.

Original: ; Bachman et al.
Revised: ; Bachman et al.

Original: ; He et al.
Revised: ; He et al.

Original: Song & Ermon (2020a); Chen et al.
Revised: Song & Ermon (2020a); Chen et al.

Original: Many such works seek to maximize the mutual information between the original data x and its learned representation z under the framework of likelihood-free inference Oord et al.
Revised: Many such works seek to maximize the mutual information between the original data x and its learned representation z under the framework of likelihood-free inference Oord et al.

Original: ; Song & Ermon (2020a)
Revised: ; Song & Ermon (2020a)

Original: ; Durkan et al. (2020).
Revised: ; Durkan et al. (2020).

Original: The above problem can be formulated as maximizing a density ratio p ( x | z ) p ( x ) that preserves the mutual information between the raw data x and learned representation z .
Revised: The above problem can be formulated as maximizing a density ratio p ( x | z ) p ( x ) that preserves the mutual information between the raw data x and learned representation z .

Original: To achieve this, existing contrastive methods Oord et al.
Revised: To achieve this, existing contrastive methods Oord et al.

Original: ; Durkan et al.
Revised: ; Durkan et al.

Original: ; He et al.
Revised: ; He et al.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) typically adopt a neural network to directly model the ratio as an entirety and avoid explicitly considering the actual generative model p ( x | z ) , which has proven to be a more challenging problem Song & Ermon (2020b); Belghazi et al.
Revised: (2021) typically adopt a neural network to directly model the ratio as an entirety and avoid explicitly considering the actual generative model p ( x | z ) , which has proven to be a more challenging problem Song & Ermon (2020b); Belghazi et al.

Original: In contrast, we show that by formulating the conventional contrastive representation learning problem under the generative setting, the properties of DPMs enable us to directly optimize the model p in this work, which can be interpreted as the optimal version of the density ratio Oord et al.
Revised: In contrast, we show that by formulating the conventional contrastive representation learning problem under the generative setting, the properties of DPMs enable us to directly optimize the model p in this work, which can be interpreted as the optimal version of the density ratio Oord et al.

Original: Vector-Quantized Representations for Conditional Generation.
Revised: Vector-Quantized Representations for Conditional Generation.

Original: Vector quantization is a classical technique in which a high-dimensional space is represented using a discrete number of vectors, which has proven effective in tasks ranging from data compression to density estimation Gray &
Revised: Vector quantization is a classical technique in which a high-dimensional space is represented using a discrete number of vectors, which has proven effective in tasks ranging from data compression to density estimation Gray &

Original: Olshen (1997); Gray (1984).
Revised: Olshen (1997); Gray (1984).

Original: More recently, Vector-Quantized (VQ) deep learning models employ this technique to allow for compact and discrete representations of music and image data Oord et al.
Revised: More recently, Vector-Quantized (VQ) deep learning models employ this technique to allow for compact and discrete representations of music and image data Oord et al.

Original: (2017); Razavi et al.
Revised: (2017); Razavi et al.

Original: Esser et al.
Revised: Esser et al.

Original: (2021b)
Revised: (2021b)

Original: ; Dhariwal et al.
Revised: ; Dhariwal et al.

Original: Typically, the VQ-based models use an encoder-codebook-decoder framework, where the “codebook” contains a fixed number of vectors (entries) to represent the original high dimensional raw data.
Revised: Typically, the VQ-based models use an encoder-codebook-decoder framework, where the “codebook” contains a fixed number of vectors (entries) to represent the original high dimensional raw data.

Original: The encoder transforms the input x into feature embedding that are each mapped to the closest corresponding vector in the codebook, while the decoder uses the set of quantized vectors z to reconstruct the input data, producing x ′ as illustrated in the upper part of Fig. 2.
Revised: The encoder transforms the input x into feature embedding that are each mapped to the closest corresponding vector in the codebook, while the decoder uses the set of quantized vectors z to reconstruct the input data, producing x ′ as illustrated in the upper part of Fig. 2.

Original: In this work, we perform conditional diffusion process on the VQ space ( i.e. , discrete token sequences) as shown in the bottom part of Fig.
Revised: In this work, we perform conditional diffusion process on the VQ space ( i.e. , discrete token sequences) as shown in the bottom part of Fig.

Original: 2, which largely reduces the dimensionality of the raw data, thus avoiding the expensive raw data decoding and synthesis.
Revised: 2, which largely reduces the dimensionality of the raw data, thus avoiding the expensive raw data decoding and synthesis.

Original: As our approach is flexible enough to be employed with various input and output modalities, the exact underlying VQ model we use depends on the target data domain.
Revised: As our approach is flexible enough to be employed with various input and output modalities, the exact underlying VQ model we use depends on the target data domain.

Original: For music synthesis, we employ a fine-tuned Jukebox Dhariwal et al.
Revised: For music synthesis, we employ a fine-tuned Jukebox Dhariwal et al.

Original: (2020) model, while for image generation, we employ VQ-GAN Esser et al.
Revised: (2020) model, while for image generation, we employ VQ-GAN Esser et al.

Original: (2021b).
Revised: (2021b).

Original: See Sec.
Revised: See Sec.

Original: 4 for further details.
Revised: 4 for further details.

Original: We refer to z , the latent quantized representation of x , as z 0 below to distinguish it from the latent representation at prior stages in the denoising process.
Revised: We refer to z , the latent quantized representation of x , as z 0 below to distinguish it from the latent representation at prior stages in the denoising process.

Original:  M ETHOD
Revised:  M ETHOD

Original: Here we outline our approach to cross-modal and conditional generation using our proposed discrete contrastive diffusion approach, which is depicted in Fig.
Revised: Here we outline our approach to cross-modal and conditional generation using our proposed discrete contrastive diffusion approach, which is depicted in Fig.

Original: In Sec. 3.1, we formulate our Conditional
Revised: In Sec. 3.1, we formulate our Conditional

Original: Discrete Contrastive Diffusion loss in detail, and demonstrate how it helps to maximize the mutual information between the conditioning and generated discrete data representations.
Revised: Discrete Contrastive Diffusion loss in detail, and demonstrate how it helps to maximize the mutual information between the conditioning and generated discrete data representations.

Original: Sec.
Revised: Sec.

Original: 3.2 defines two specific mechanisms for applying this loss within a diffusion model training framework, samplewise and step-wise .
Revised: 3.2 defines two specific mechanisms for applying this loss within a diffusion model training framework, samplewise and step-wise .

Original: In Sec. 3.3, we detail techniques for constructing negative samples designed to improve the overall quality and coherence of the generated sequences.
Revised: In Sec. 3.3, we detail techniques for constructing negative samples designed to improve the overall quality and coherence of the generated sequences.

Original: Given the data pair ( c, x ) , where c is the conditioning information from a given input modality ( e.g. , videos, text, or a class label), our objective is to generate a data sample x in the target modality ( e.g. , music audio or images) corresponding to c .
Revised: Given the data pair ( c, x ) , where c is the conditioning information from a given input modality ( e.g. , videos, text, or a class label), our objective is to generate a data sample x in the target modality ( e.g. , music audio or images) corresponding to c .

Original: In the training stage, we first employ and train a VQ-based model to obtain discrete representation z 0 of the data x from the target modality.
Revised: In the training stage, we first employ and train a VQ-based model to obtain discrete representation z 0 of the data x from the target modality.

Original: Next, our diffusion process operates on the encoded latent representation z 0 of x .
Revised: Next, our diffusion process operates on the encoded latent representation z 0 of x .

Original: The denoising process recovers the latent representation z 0 given the conditioning c that can be decoded to obtain the reconstruction x ′ .
Revised: The denoising process recovers the latent representation z 0 given the conditioning c that can be decoded to obtain the reconstruction x ′ .

Original: In inference, we generate z 0 based on the conditioning c , and decode the latent VQ representation zback to raw data domain using the decoder from the pre-trained and fixed VQ decoder.
Revised: In inference, we generate z 0 based on the conditioning c , and decode the latent VQ representation zback to raw data domain using the decoder from the pre-trained and fixed VQ decoder.

Original: 3. C ONDITIONAL D ISCRETE C ONTRASTIVE D IFFUSION L OSS
Revised: 3. C ONDITIONAL D ISCRETE C ONTRASTIVE D IFFUSION L OSS

Original: We seek to enhance the connection between c and the generated data z 0 by maximizing their mutual information, defined as I ( z 0 ; c ) =
Revised: We seek to enhance the connection between c and the generated data z 0 by maximizing their mutual information, defined as I ( z 0 ; c ) =

Original: (cid:80) z
Revised: (cid:80) z

Original: 0 p θ ( z 0 , c ) log p θ ( z 0 | c ) p θ ( z 0 ) .
Revised: 0 p θ ( z 0 , c ) log p θ ( z 0 | c ) p θ ( z 0 ) .

Original: We introduce a set of negative VQ sequences Z ′ =
Revised: We introduce a set of negative VQ sequences Z ′ =

Original: { z 1 , z 2 , ..., z N } , encoded from N negative samples X
Revised: { z 1 , z 2 , ..., z N } , encoded from N negative samples X

Original: =
Revised: =

Original: { x 1 , x 2 , ..., x N } , and define f ( z 0 , c )
Revised: { x 1 , x 2 , ..., x N } , and define f ( z 0 , c )

Original: =
Revised: =

Original: p θ ( z 0 | c ) p θ ( z 0 ) .
Revised: p θ ( z 0 | c ) p θ ( z 0 ) .

Original: Our proposed Conditional Discrete Contrastive Diffusion (CDCD) loss is:
Revised: Our proposed Conditional Discrete Contrastive Diffusion (CDCD) loss is:

Original: [Equation]
Revised: [Equation]

Original: The proposed CDCD loss is similar to the categorical cross-entropy loss for classifying the positive sample as in Oord et al.
Revised: The proposed CDCD loss is similar to the categorical cross-entropy loss for classifying the positive sample as in Oord et al.

Original: (2018), where our conditioning c and the generated data z 0 corresponds to the original learned representation and raw data, and optimization of this loss leads to maximization of I ( z 0 ; c ) .
Revised: (2018), where our conditioning c and the generated data z 0 corresponds to the original learned representation and raw data, and optimization of this loss leads to maximization of I ( z 0 ; c ) .

Original: However, the loss in Oord et al.
Revised: However, the loss in Oord et al.

Original: (2018) models the density ratio f
Revised: (2018) models the density ratio f

Original: ( z 0 , c ) as an entirety.
Revised: ( z 0 , c ) as an entirety.

Original: In 𝑧 𝑇 𝑧 0 𝑧 𝑡 𝑧 𝑡− ··· ··· ··· ··· 𝑝 𝜃 (𝑧 𝑡−1 |𝑧 𝑡 ) ··· ··· ··· ··· 𝑧 𝑇𝑗 𝑧 𝑡𝑗 𝑧 𝑡−1𝑗 𝑧 0𝑗 ··· 𝑝 𝜃 (𝑧 𝑡−11 |𝑧 𝑡1 ,𝑐) 𝑝 𝜃 (𝑧 𝑡−1𝑗 |𝑧 𝑡𝑗 ,𝑐) 𝑝 𝜃 (𝑧 𝑡−1 𝑁 |𝑧 𝑡 𝑁 ,𝑐)
Revised: In 𝑧 𝑇 𝑧 0 𝑧 𝑡 𝑧 𝑡− ··· ··· ··· ··· 𝑝 𝜃 (𝑧 𝑡−1 |𝑧 𝑡 ) ··· ··· ··· ··· 𝑧 𝑇𝑗 𝑧 𝑡𝑗 𝑧 𝑡−1𝑗 𝑧 0𝑗 ··· 𝑝 𝜃 (𝑧 𝑡−11 |𝑧 𝑡1 ,𝑐) 𝑝 𝜃 (𝑧 𝑡−1𝑗 |𝑧 𝑡𝑗 ,𝑐) 𝑝 𝜃 (𝑧 𝑡−1 𝑁 |𝑧 𝑡 𝑁 ,𝑐)

Original: Transformer-basedDPM block at step 𝑡
Revised: Transformer-basedDPM block at step 𝑡

Original: ConditioningEncoders
Revised: ConditioningEncoders

Original: Conditioning 𝑐
Revised: Conditioning 𝑐

Original: Step-Wise Parallel Diffusion 𝑧 𝑡 𝑝 𝜃 (𝑧 0 |𝑧 𝑡 )
Revised: Step-Wise Parallel Diffusion 𝑧 𝑡 𝑝 𝜃 (𝑧 0 |𝑧 𝑡 )

Original: Auxiliary Transformer-based DPM block at step 𝑡 𝑝 𝜃 (𝑧 01 |𝑧 𝑡 ,𝑐) 𝑝 𝜃 (𝑧 0𝑗 |𝑧 𝑡 ,𝑐) 𝑝 𝜃 (𝑧 0𝑁 |𝑧 𝑡 ,𝑐)
Revised: Auxiliary Transformer-based DPM block at step 𝑡 𝑝 𝜃 (𝑧 01 |𝑧 𝑡 ,𝑐) 𝑝 𝜃 (𝑧 0𝑗 |𝑧 𝑡 ,𝑐) 𝑝 𝜃 (𝑧 0𝑁 |𝑧 𝑡 ,𝑐)

Original: DataEncoder
Revised: DataEncoder

Original: Data Decoder 1n
Revised: Data Decoder 1n

Original: K 2 K 9 ···
Revised: K 2 K 9 ···

Original: GT data data embedding
Revised: GT data data embedding

Original: VQ sequence 𝑧 Codebook reconstructed data
Revised: VQ sequence 𝑧 Codebook reconstructed data

Original: VQ-Based Model
Revised: VQ-Based Model

Original: Conditional Discrete Contrastive Diffusion
Revised: Conditional Discrete Contrastive Diffusion

Original: Sample-Wise Auxiliary Diffusion ··· ··· ··· ··· ··· ··· 2 K 9 ··· 𝑧𝑧 0𝑗 2 K 9 ··· ··· 𝑧 𝑇 ··· ···
Revised: Sample-Wise Auxiliary Diffusion ··· ··· ··· ··· ··· ··· 2 K 9 ··· 𝑧𝑧 0𝑗 2 K 9 ··· ··· 𝑧 𝑇 ··· ···

Original: Conditioning 𝑐
Revised: Conditioning 𝑐

Original: ConditioningEncoders 𝑧 𝑇 𝑧 𝑡 𝑧 𝑡− 𝑧𝑧 𝑇𝑁 𝑧 𝑡𝑁 𝑧 𝑡−1 𝑁
Revised: ConditioningEncoders 𝑧 𝑇 𝑧 𝑡 𝑧 𝑡− 𝑧𝑧 𝑇𝑁 𝑧 𝑡𝑁 𝑧 𝑡−1 𝑁

Original: 𝑧 0𝑁 𝑧𝑧 0𝑁 our case, we demonstrate that the DPMs properties Sohl-Dickstein et al.
Revised: 𝑧 0𝑁 𝑧𝑧 0𝑁 our case, we demonstrate that the DPMs properties Sohl-Dickstein et al.

Original: (2015b)
Revised: (2015b)

Original: ; Ho et al.
Revised: ; Ho et al.

Original: Austin et al.
Revised: Austin et al.

Original: (2021) enable us to directly optimize the actual distribution p θ within the diffusion process for the desired conditional generation tasks.
Revised: (2021) enable us to directly optimize the actual distribution p θ within the diffusion process for the desired conditional generation tasks.

Original: Specifically, we show the connections between the proposed CDCD loss and the conventional variational loss L vb (see equation 1) in Sec. 3.2, and thus how it contributes to efficient DPM learning.
Revised: Specifically, we show the connections between the proposed CDCD loss and the conventional variational loss L vb (see equation 1) in Sec. 3.2, and thus how it contributes to efficient DPM learning.

Original: Additionally, we can derive the lower bound for the mutual information as I ( z 0 ; c ) ≥ log( N ) − L CDCD (see supplement for details), which indicates that a larger number of negative samples increases the lower bound.
Revised: Additionally, we can derive the lower bound for the mutual information as I ( z 0 ; c ) ≥ log( N ) − L CDCD (see supplement for details), which indicates that a larger number of negative samples increases the lower bound.

Original: These two factors allow for faster convergence of a DPM with fewer diffusion steps.
Revised: These two factors allow for faster convergence of a DPM with fewer diffusion steps.

Original: 3. P ARALLEL AND A UXILIARY D IFFUSION P ROCESS
Revised: 3. P ARALLEL AND A UXILIARY D IFFUSION P ROCESS

Original: The CDCD loss in equation 2 considers the mutual information between c and z 0 in a general way, without specifying the intermediate diffusion steps.
Revised: The CDCD loss in equation 2 considers the mutual information between c and z 0 in a general way, without specifying the intermediate diffusion steps.

Original: We propose and analyze two contrastive diffusion mechanisms to efficiently incorporate this loss into DPM learning, and demonstrate that we can directly optimize the generative model p θ in the diffusion process.
Revised: We propose and analyze two contrastive diffusion mechanisms to efficiently incorporate this loss into DPM learning, and demonstrate that we can directly optimize the generative model p θ in the diffusion process.

Original: We present our step-wise parallel diffusion and the sample-wise auxiliary diffusion mechanisms, which are distinguished by the specific operations applied for the intermediate negative latent variables z j 1: T for each negative sample x j .
Revised: We present our step-wise parallel diffusion and the sample-wise auxiliary diffusion mechanisms, which are distinguished by the specific operations applied for the intermediate negative latent variables z j 1: T for each negative sample x j .

Original: The high-level intuition for the parallel and auxiliary designs is to emphasize different attributes of the synthesized data given specific applications.
Revised: The high-level intuition for the parallel and auxiliary designs is to emphasize different attributes of the synthesized data given specific applications.

Original: Particularly, we propose the parallel variant to learn the internal coherence of the audio sequential data by emphasizing the gradual change at each time step, while the auxiliary mechanism focuses more on the sample-level connections to the conditioning.
Revised: Particularly, we propose the parallel variant to learn the internal coherence of the audio sequential data by emphasizing the gradual change at each time step, while the auxiliary mechanism focuses more on the sample-level connections to the conditioning.

Original: Step-Wise Parallel Diffusion.
Revised: Step-Wise Parallel Diffusion.

Original: This mechanism not only focuses on the mutual information between c and z 0 , but also takes the intermediate negative latent variables
Revised: This mechanism not only focuses on the mutual information between c and z 0 , but also takes the intermediate negative latent variables

Original: z j 1: T into account by explicitly invoking the complete diffusion process for each negative sample z j ∈ Z ′ .
Revised: z j 1: T into account by explicitly invoking the complete diffusion process for each negative sample z j ∈ Z ′ .

Original: As illustrated in Fig.(bottom left), we initiate N + 1 parallel diffusion processes, among which N are invoked by negative samples.
Revised: As illustrated in Fig.(bottom left), we initiate N + 1 parallel diffusion processes, among which N are invoked by negative samples.

Original: For each negative sample x j ∈ X ′ , we explicitly compute its negative latent discrete variables
Revised: For each negative sample x j ∈ X ′ , we explicitly compute its negative latent discrete variables

Original: z j 0: T .
Revised: z j 0: T .

Original: In this case, equation 2 is as follows (see supplement for the detailed derivation):
Revised: In this case, equation 2 is as follows (see supplement for the detailed derivation):

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The equation above factorizes the proposed CDCD loss using the step-wise parallel diffusion mechanism into two terms, where the first term corresponds to the original variational bound L vb , and the second term can be interpreted as the negative sum of variational bounds induced by the negative samples and the provided conditioning c .
Revised: The equation above factorizes the proposed CDCD loss using the step-wise parallel diffusion mechanism into two terms, where the first term corresponds to the original variational bound L vb , and the second term can be interpreted as the negative sum of variational bounds induced by the negative samples and the provided conditioning c .

Original: Sample-Wise Auxiliary Diffusion.
Revised: Sample-Wise Auxiliary Diffusion.

Original: Alternatively, our sample-wise auxiliary diffusion mechanism maintains one principal diffusion process, as in traditional diffusion training, shown in Fig.
Revised: Alternatively, our sample-wise auxiliary diffusion mechanism maintains one principal diffusion process, as in traditional diffusion training, shown in Fig.

Original: 2 (bottom right).
Revised: 2 (bottom right).

Original: It contrasts the intermediate positive latent variables z 1: T with the negative sample z j 0 ∈ Z .
Revised: It contrasts the intermediate positive latent variables z 1: T with the negative sample z j 0 ∈ Z .

Original: In this case, we can write the CDCD loss from.
Revised: In this case, we can write the CDCD loss from.

Original: equation 2 as (see supplement for details):
Revised: equation 2 as (see supplement for details):

Original: [Equation]
Revised: [Equation]

Original: As with the step-wise loss, the CDCD-Sample loss includes two terms.
Revised: As with the step-wise loss, the CDCD-Sample loss includes two terms.

Original: The first refers to sampling directly from the positive z 0 at an arbitrary timestep t .
Revised: The first refers to sampling directly from the positive z 0 at an arbitrary timestep t .

Original: The second sums the same auxiliary loss from negative samples z j 0 .
Revised: The second sums the same auxiliary loss from negative samples z j 0 .

Original: This marginalization operation is based on the property of Markov chain as in previous discrete DPMs Austin et al.
Revised: This marginalization operation is based on the property of Markov chain as in previous discrete DPMs Austin et al.

Original: ; Gu et al.
Revised: ; Gu et al.

Original: (2022), which imposes direct supervision from the sample data.
Revised: (2022), which imposes direct supervision from the sample data.

Original: The first term is similar to the auxiliary denoising objective in Austin et al.
Revised: The first term is similar to the auxiliary denoising objective in Austin et al.

Original: Gu et al.
Revised: Gu et al.

Original: Both contrastive diffusion mechanisms enable us to effectively incorporate the CDCD loss into our
Revised: Both contrastive diffusion mechanisms enable us to effectively incorporate the CDCD loss into our

Original: DPM learning process by directly optimizing the actual denoising generative network p θ .
Revised: DPM learning process by directly optimizing the actual denoising generative network p θ .

Original: Final Loss Function.
Revised: Final Loss Function.

Original: The final loss function for our contrastive diffusion training process is:
Revised: The final loss function for our contrastive diffusion training process is:

Original: [Equation]
Revised: [Equation]

Original: L vb is conditioning c related, and takes the form of L t − 1 = D KL ( q ( z t − 1 | z t , z 0 )
Revised: L vb is conditioning c related, and takes the form of L t − 1 = D KL ( q ( z t − 1 | z t , z 0 )

Original: ||
Revised: ||

Original: p θ ( z t − 1 | z t , c )) as in Gu et al.
Revised: p θ ( z t − 1 | z t , c )) as in Gu et al.

Original: (2022), where c included as the prior for all the intermediate steps.
Revised: (2022), where c included as the prior for all the intermediate steps.

Original: L CDCD refers to either the step-wise parallel diffusion or sample-wise auxiliary diffusion loss.
Revised: L CDCD refers to either the step-wise parallel diffusion or sample-wise auxiliary diffusion loss.

Original: Empirically, we can omit the first term in equation 3, or directly optimize L CDCD − Step , in which the standard L vb is already included.
Revised: Empirically, we can omit the first term in equation 3, or directly optimize L CDCD − Step , in which the standard L vb is already included.

Original: The detailed training algorithm is explained in the supplement.
Revised: The detailed training algorithm is explained in the supplement.

Original: 3. I NTRA - AND I NTER -N EGATIVE S AMPLING
Revised: 3. I NTRA - AND I NTER -N EGATIVE S AMPLING

Original: Previous contrastive works construct negative samples using techniques such as image augmentation Chen et al.
Revised: Previous contrastive works construct negative samples using techniques such as image augmentation Chen et al.

Original: ; He et al.
Revised: ; He et al.

Original: (2020) or spatially adjacent image patches Oord et al.
Revised: (2020) or spatially adjacent image patches Oord et al.

Original: In this work, we categorize our sampling methods into intra- and inter- negative samplings as in Fig.
Revised: In this work, we categorize our sampling methods into intra- and inter- negative samplings as in Fig.

Original: For the intra -sample negative sampling, we construct X ′ based on the given original x .
Revised: For the intra -sample negative sampling, we construct X ′ based on the given original x .

Original: This bears resemblance to the patch-based technique in the image domain Oord et al.
Revised: This bears resemblance to the patch-based technique in the image domain Oord et al.

Original: As for the audio data, we first divide the original audio waveform into multiple chunks, and randomly shuffle their ordering.
Revised: As for the audio data, we first divide the original audio waveform into multiple chunks, and randomly shuffle their ordering.

Original: For the inter -sample negative sampling, X ′ consists of instance-level negative samples x
Revised: For the inter -sample negative sampling, X ′ consists of instance-level negative samples x

Original: ′ that differ from the given data pair ( c, x ) .
Revised: ′ that differ from the given data pair ( c, x ) .

Original: In practice, we define negative samples x ′ to be music sequences with different musical genres from x in the music generation task, while x ′ denotes images other than x in the image synthesis task.
Revised: In practice, we define negative samples x ′ to be music sequences with different musical genres from x in the music generation task, while x ′ denotes images other than x in the image synthesis task (in practice, we choose x ′ with different class labels as x ).

Original: Based on our proposed contrastive diffusion modes and negative sampling methods, there are four possible contrastive settings: step-wise parallel diffusion with either intra- or inter-negative sampling (denoted as Step-Intra and Step-Inter ), or sample-wise auxiliary diffusion with either intra- or internegative sampling (denoted as Sample-Intra and Sample-Inter ).
Revised: Based on our proposed contrastive diffusion modes and negative sampling methods, there are four possible contrastive settings: step-wise parallel diffusion with either intra- or inter-negative sampling (denoted as Step-Intra and Step-Inter ), or sample-wise auxiliary diffusion with either intra- or internegative sampling (denoted as Sample-Intra and Sample-Inter ).

Original: Intuitively, we argue that Step-Intra and Sample-Inter settings are more reasonable compared to Step-Inter and Sample-Intra because of the consistency between the diffusion data corruption process and the way to construct negative samples.
Revised: Intuitively, we argue that Step-Intra and Sample-Inter settings are more reasonable compared to Step-Inter and Sample-Intra because of the consistency between the diffusion data corruption process and the way to construct negative samples.

Original: Specifically, the data corruption process in the discrete DPMs includes sampling and replacing certain tokens with some random or mask tokens at each diffusion step Austin et al.
Revised: Specifically, the data corruption process in the discrete DPMs includes sampling and replacing certain tokens with some random or mask tokens at each diffusion step Austin et al.

Original: ; Gu et al.
Revised: ; Gu et al.

Original: (2022), which is a chunk-level operation within a given data sequence similar to the ways we construct intra- negative samples by shuffling the chunk-level orders.
Revised: (2022), which is a chunk-level operation within a given data sequence similar to the ways we construct intra- negative samples by shuffling the chunk-level orders.

Original: In contrast, the sample-wise auxiliary diffusion seeks to provide sample-level supervision, which is consistent with our inter-negative sampling method.
Revised: In contrast, the sample-wise auxiliary diffusion seeks to provide sample-level supervision, which is consistent with our inter-negative sampling method.

Original: In the interest of clarity and concision, we only present the experimental results for Step-Intra and
Revised: In the interest of clarity and concision, we only present the experimental results for Step-Intra and

Original: Sample-Inter settings in Sec.
Revised: Sample-Inter settings in Sec.

Original: 4 of our main paper.
Revised: 4 of our main paper.

Original: The complete results obtained with other contrastive settings and more detailed analysis are included in the supplement.
Revised: The complete results obtained with other contrastive settings and more detailed analysis are included in the supplement.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct experiments on three conditional generation tasks: dance-to-music generation, text-toimage synthesis, and class-conditioned image synthesis.
Revised: We conduct experiments on three conditional generation tasks: dance-to-music generation, text-toimage synthesis, and class-conditioned image synthesis.

Original: For the dance-to-music task, we seek to generate audio waveforms for complex music from human motion and dance video frames.
Revised: For the dance-to-music task, we seek to generate audio waveforms for complex music from human motion and dance video frames.

Original: For the text-to-image task, the objective is to generate images from given textual descriptions.
Revised: For the text-to-image task, the objective is to generate images from given textual descriptions.

Original: Given our emphasis on the input-output faithfulness for cross-modal generations, the main analysis are based on the dance-to-music generation task since the evaluation protocol from Zhu et al.
Revised: Given our emphasis on the input-output faithfulness for cross-modal generations, the main analysis are based on the dance-to-music generation task since the evaluation protocol from Zhu et al.

Original: (2022) explicitly measures such connections in terms of beats, genre and general correspondence for generated music.
Revised: (2022) explicitly measures such connections in terms of beats, genre and general correspondence for generated music.

Original: 4. D ANCE - TO -M USIC G ENERATION
Revised: 4. D ANCE - TO -M USIC G ENERATION

Original: Dataset.
Revised: Dataset.

Original: We use the AIST++ Li et al.
Revised: We use the AIST++ Li et al.

Original: (2021) dataset and the TikTok Dance-Music dataset Zhu et al.
Revised: (2021) dataset and the TikTok Dance-Music dataset Zhu et al.

Original: for the dance-to-music experiments.
Revised: for the dance-to-music experiments.

Original: AIST++ is a subset of the AIST dataset Tsuchida et al.
Revised: AIST++ is a subset of the AIST dataset Tsuchida et al.

Original: (2019), which contains 1020 dance videos and 60 songs performed by professional dancers and filmed in clean studio environment settings without occlusions.
Revised: (2019), which contains 1020 dance videos and 60 songs performed by professional dancers and filmed in clean studio environment settings without occlusions.

Original: AIST++ provide human motion data in the form of SMPL Loper et al.
Revised: AIST++ provide human motion data in the form of SMPL Loper et al.

Original: (2015) parameters and body keypoints, and includes the annotations for different genres and choreography styles.
Revised: (2015) parameters and body keypoints, and includes the annotations for different genres and choreography styles.

Original: The TikTok Dance-Music dataset includes 445 dance videos collected from the social media platform.
Revised: The TikTok Dance-Music dataset includes 445 dance videos collected from the social media platform.

Original: The 2D skeleton data extracted with
Revised: The 2D skeleton data extracted with

Original: OpenPose Cao et al.
Revised: OpenPose Cao et al.

Original: (2017); Cao et al.
Revised: (2017); Cao et al.

Original: (2019) is used as the motion representation.
Revised: (2019) is used as the motion representation.

Original: We adopt the official cross-modality splits without overlapping music songs for both datasets.
Revised: We adopt the official cross-modality splits without overlapping music songs for both datasets.

Original: Implementations.
Revised: Implementations.

Original: The sampling rate for all audio signals is 22.5 kHz in our experiments.
Revised: The sampling rate for all audio signals is 22.5 kHz in our experiments.

Original: We use 2-second music samples as in Zhu et al.
Revised: We use 2-second music samples as in Zhu et al.

Original: for the main experiments.
Revised: for the main experiments.

Original: We fine-tuned the pre-trained Jukebox Dhariwal et al.
Revised: We fine-tuned the pre-trained Jukebox Dhariwal et al.

Original: (2020) for our Music VQ-VAE model.
Revised: (2020) for our Music VQ-VAE model.

Original: For the motion encoder, we deploy a backbone stacked with convolutional layers and residual blocks.
Revised: For the motion encoder, we deploy a backbone stacked with convolutional layers and residual blocks.

Original: For the visual encoder, we extract I3D features Carreira & Zisserman (2017) using a model pre-trained on Kinectics Kay et al.
Revised: For the visual encoder, we extract I3D features Carreira & Zisserman (2017) using a model pre-trained on Kinectics Kay et al.

Original: (2017) as the visual conditioning.
Revised: (2017) as the visual conditioning.

Original: The motion and visual encoder outputs are concatenated to form the final continuous conditioning input to our contrastive diffusion model.
Revised: The motion and visual encoder outputs are concatenated to form the final continuous conditioning input to our contrastive diffusion model.

Original: For the contrastive diffusion model, we adopt a transformer-based backbone to learn the denoising network p θ .
Revised: For the contrastive diffusion model, we adopt a transformer-based backbone to learn the denoising network p θ .

Original: It includestransformer blocks, with each block consisting of full attention, cross attention and feed forward modules, and a channel size of 1024 for each block.
Revised: It includestransformer blocks, with each block consisting of full attention, cross attention and feed forward modules, and a channel size of 1024 for each block.

Original: We set the initial weight for the contrastive loss as λ = 5 e − 5 .
Revised: We set the initial weight for the contrastive loss as λ = 5 e − 5 .

Original: The number N of intra- and inter-negative samples for each GT music sample is 10.
Revised: The number N of intra- and inter-negative samples for each GT music sample is 10.

Original: The visual encoder, motion encoder, and the contrastive diffusion model are jointly optimized.
Revised: The visual encoder, motion encoder, and the contrastive diffusion model are jointly optimized.

Original: More implementation details are provided in the supplement.
Revised: More implementation details are provided in the supplement.

Original: Evaluations.
Revised: Evaluations.

Original: The evaluation of synthesized music measures both the conditioning-output correspondence and the general synthesis quality using the metrics introduced in Zhu et al.
Revised: The evaluation of synthesized music measures both the conditioning-output correspondence and the general synthesis quality using the metrics introduced in Zhu et al.

Original: Specifically, the metrics include the beats coverage score, the beats hit scores, the genre accuracy score, and two subjective evaluation tests with Mean Opinion Scores (MOS) for the musical coherence and general quality.
Revised: Specifically, the metrics include the beats coverage score, the beats hit scores, the genre accuracy score, and two subjective evaluation tests with Mean Opinion Scores (MOS) for the musical coherence and general quality.

Original: Among these metrics, the beats scores emphasize the intra-sample properties, since they calculate the second-level audio onset strength within musical chunks Ellis (2007), while the genre accuracy focuses on the instance-level musical attributes of music styles.
Revised: Among these metrics, the beats scores emphasize the intra-sample properties, since they calculate the second-level audio onset strength within musical chunks Ellis (2007), while the genre accuracy focuses on the instance-level musical attributes of music styles.

Original: Detailed explanations of the above metrics can be found in Zhu et al.
Revised: Detailed explanations of the above metrics can be found in Zhu et al.

Original: We compare against multiple dance-to-music generation works: Foley Gan et al.
Revised: We compare against multiple dance-to-music generation works: Foley Gan et al.

Original: (2020a), Dance2Music Aggarwal & Parikh (2021), CMT Di et al.
Revised: (2020a), Dance2Music Aggarwal & Parikh (2021), CMT Di et al.

Original: (2021), and D2M-GAN Zhu et al.
Revised: (2021), and D2M-GAN Zhu et al.

Original: The first three models rely on symbolic discrete
Revised: The first three models rely on symbolic discrete

Original: MIDI musical representations, while the last one also uses a VQ musical representation.
Revised: MIDI musical representations, while the last one also uses a VQ musical representation.

Original: The major difference between the symbolic MIDI and discrete VQ musical representations lies within the fact
Revised: The major difference between the symbolic MIDI and discrete VQ musical representations lies within the fact

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Vanilla Step-Intra Sample-Inter      5.
Revised: Vanilla Step-Intra Sample-Inter      5.

Original: 120 Diffusion steps T
Revised: 120 Diffusion steps T

Original: F I D s c
Revised: F I D s c

Original: o r e s 19.
Revised: o r e s 19.

Original: VQ-D* Step-Intra
Revised: VQ-D* Step-Intra

Original: Sample-Inter
Revised: Sample-Inter

Original: T h r o u g h p u t ( s a m p
Revised: T h r o u g h p u t ( s a m p

Original: l e s / s ) 1.
Revised: l e s / s ) 1.

Original: that the MIDI is pre-defined for each instrument, while the VQ is learning-based.
Revised: that the MIDI is pre-defined for each instrument, while the VQ is learning-based.

Original: The latter thus enables complex and free music synthesis appropriate for scenarios like dance videos.
Revised: The latter thus enables complex and free music synthesis appropriate for scenarios like dance videos.

Original: Results and Discussion.
Revised: Results and Discussion.

Original: The quantitative experimental results are shown in Tab. 1 and Tab. 2.
Revised: The quantitative experimental results are shown in Tab. 1 and Tab. 2.

Original: Our proposed methods achieve better performance than the competing methods even with vanilla version without contrastive mechanisms.
Revised: Our proposed methods achieve better performance than the competing methods even with vanilla version without contrastive mechanisms.

Original: Furthermore, we find that the Step-Intra setting is more helpful in increasing the beats scores, while the Sample-Inter setting yields more improvements for the genre accuracy scores.
Revised: Furthermore, we find that the Step-Intra setting is more helpful in increasing the beats scores, while the Sample-Inter setting yields more improvements for the genre accuracy scores.

Original: We believe this is due to the evaluation methods of different metrics.
Revised: We believe this is due to the evaluation methods of different metrics.

Original: The beats scores measure the chunk-level ( i.e. , , the audio onset strength Ellis (2007)) consistency between the
Revised: The beats scores measure the chunk-level ( i.e. , , the audio onset strength Ellis (2007)) consistency between the

Original: GT and synthesized music samples Zhu et al.
Revised: GT and synthesized music samples Zhu et al.

Original: (2022), while the genre scores consider the overall musical attributes of each sample sequence in instance level.
Revised: (2022), while the genre scores consider the overall musical attributes of each sample sequence in instance level.

Original: This finding is consistent with our assumptions in Sec. 3.3.
Revised: This finding is consistent with our assumptions in Sec. 3.3.

Original: Convergence Analysis.
Revised: Convergence Analysis.

Original: We also analyze the impact of the proposed contrastive diffusion on model convergence in terms of diffusion steps.
Revised: We also analyze the impact of the proposed contrastive diffusion on model convergence in terms of diffusion steps.

Original: The number of diffusion steps is a significant hyper-parameter for DPMs Sohl-Dickstein et al.
Revised: The number of diffusion steps is a significant hyper-parameter for DPMs Sohl-Dickstein et al.

Original: (2015b)
Revised: (2015b)

Original: ; Nichol & Dhariwal (2021);
Revised: ; Nichol & Dhariwal (2021);

Original: Austin et al.
Revised: Austin et al.

Original: ; Gu et al.
Revised: ; Gu et al.

Original: ; Kingma et al.
Revised: ; Kingma et al.

Original: (2021) that directly influences the inference time and synthesis quality.
Revised: (2021) that directly influences the inference time and synthesis quality.

Original: Previous works have shown that a larger number of diffusion steps usually lead to better model performance, but longer inference times Kingma et al.
Revised: Previous works have shown that a larger number of diffusion steps usually lead to better model performance, but longer inference times Kingma et al.

Original: ; Gu et al.
Revised: ; Gu et al.

Original: We demonstrate that, with the improved mutual information via the proposed contrastive diffusion method, we can greatly reduce the number of steps needed.
Revised: We demonstrate that, with the improved mutual information via the proposed contrastive diffusion method, we can greatly reduce the number of steps needed.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4 (left), we observe that the beats scores reach a stable level at approximately 80 steps, ∼ 35% less than the vanilla DPM that converges in ∼ 120 steps.
Revised: 4 (left), we observe that the beats scores reach a stable level at approximately 80 steps, ∼ 35% less than the vanilla DPM that converges in ∼ 120 steps.

Original: More ablation studies and analysis on this task can be found in the supplement.
Revised: More ablation studies and analysis on this task can be found in the supplement.

Original: 4. C ONDITIONAL I MAGE S YNTHESIS
Revised: 4. C ONDITIONAL I MAGE S YNTHESIS

Original: Dataset.
Revised: Dataset.

Original: We conduct text-to-image synthesis on CUB200 Wah et al.
Revised: We conduct text-to-image synthesis on CUB200 Wah et al.

Original: and MSCOCO datasets Lin et al.
Revised: and MSCOCO datasets Lin et al.

Original: The CUB200 dataset contains images of 200 bird species.
Revised: The CUB200 dataset contains images of 200 bird species.

Original: Each image has 10 corresponding text descriptions.
Revised: Each image has 10 corresponding text descriptions.

Original: The MSCOCO dataset contains 82k images for training and 40k images for testing.
Revised: The MSCOCO dataset contains 82k images for training and 40k images for testing.

Original: Each image has 5 text descriptions.
Revised: Each image has 5 text descriptions.

Original: We also perform the class-conditioned
Revised: We also perform the class-conditioned

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: image generation on ImageNet Deng et al.
Revised: image generation on ImageNet Deng et al.

Original: ; Russakovsky et al.
Revised: ; Russakovsky et al.

Original: Implementation details for both tasks are provided in the supplement.
Revised: Implementation details for both tasks are provided in the supplement.

Original: Evaluations.
Revised: Evaluations.

Original: We adopt two evaluation metrics for text-to-image synthesis: the classic FID score Heusel et al.
Revised: We adopt two evaluation metrics for text-to-image synthesis: the classic FID score Heusel et al.

Original: (2017) as the general measurement for image quality, and the CLIPScore Hessel et al.
Revised: (2017) as the general measurement for image quality, and the CLIPScore Hessel et al.

Original: (2021) to evaluate the correspondence between the given textual caption and the synthesized image.
Revised: (2021) to evaluate the correspondence between the given textual caption and the synthesized image.

Original: For the class-conditioned image synthesis, we use the FID score and a classifier-based accuracy for general and input-output correspondence measurement.
Revised: For the class-conditioned image synthesis, we use the FID score and a classifier-based accuracy for general and input-output correspondence measurement.

Original: We compare against text-to-image generation methods including StackGAN Zhang et al.
Revised: We compare against text-to-image generation methods including StackGAN Zhang et al.

Original: StackGAN++ Zhang et al.
Revised: StackGAN++ Zhang et al.

Original: SEGAN Tan et al.
Revised: SEGAN Tan et al.

Original: (2019), AttnGAN Xu et al.
Revised: (2019), AttnGAN Xu et al.

Original: , DM-GAN Zhu et al.
Revised: , DM-GAN Zhu et al.

Original: (2019), DF-GAN Tao et al.
Revised: (2019), DF-GAN Tao et al.

Original: (2020), DAE-GAN Ruan et al.
Revised: (2020), DAE-GAN Ruan et al.

Original: DALLE Ramesh et al.
Revised: DALLE Ramesh et al.

Original: and VQ-Diffusion Gu et al.
Revised: and VQ-Diffusion Gu et al.

Original: For experiments on ImageNet, we list the result comparisons with ImageBART Esser et al.
Revised: For experiments on ImageNet, we list the result comparisons with ImageBART Esser et al.

Original: (2021a), VQGAN Esser et al.
Revised: (2021a), VQGAN Esser et al.

Original: (2021b),
Revised: (2021b),

Original: IDDPM Nichol & Dhariwal (2021), and VQ-D Gu et al.
Revised: IDDPM Nichol & Dhariwal (2021), and VQ-D Gu et al.

Original: Specifically, VQ-Diffusion Gu et al.
Revised: Specifically, VQ-Diffusion Gu et al.

Original: (2022) also adopts the discrete diffusion generative backbone, which can be considered as the vanilla version without contrastive mechanisms.
Revised: (2022) also adopts the discrete diffusion generative backbone, which can be considered as the vanilla version without contrastive mechanisms.

Original: Additionally, we provide more comparisons with other methods in terms of dataset, model scale and training time in the supplement for a more comprehensive and fair understanding of our proposed method.
Revised: Additionally, we provide more comparisons with other methods in terms of dataset, model scale and training time in the supplement for a more comprehensive and fair understanding of our proposed method.

Original: Results and Discussion.
Revised: Results and Discussion.

Original: The quantitative results are represented in Tab. 3 and Tab. 4.
Revised: The quantitative results are represented in Tab. 3 and Tab. 4.

Original: We observe that our contrastive diffusion achieves state-of-the-art performance for both general synthesis fidelity and input-output correspondence, and the Sample-Inter contrastive setting is more beneficial compared to
Revised: We observe that our contrastive diffusion achieves state-of-the-art performance for both general synthesis fidelity and input-output correspondence, and the Sample-Inter contrastive setting is more beneficial compared to

Original: Step-Intra for the image synthesis.
Revised: Step-Intra for the image synthesis.

Original: This empirical finding again validates our assumption regarding the contrastive settings in Sec. 3.3, where the Sample-Inter setting helps more with the instance-level synthesis quality.
Revised: This empirical finding again validates our assumption regarding the contrastive settings in Sec. 3.3, where the Sample-Inter setting helps more with the instance-level synthesis quality.

Original: Notably, as shown in Fig.
Revised: Notably, as shown in Fig.

Original: 4 (right), our contrastive diffusion method shows model convergence at about 60 diffusion steps, while the vanilla version converges at approximatelysteps on CUB200 Wah et al.
Revised: 4 (right), our contrastive diffusion method shows model convergence at about 60 diffusion steps, while the vanilla version converges at approximatelysteps on CUB200 Wah et al.

Original: (2011), which greatly increases the inference speed by 40%.
Revised: (2011), which greatly increases the inference speed by 40%.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: While DPMs have demonstrated remarkable potential, improving their training and inference efficiency while maintaining flexible and accurate results for conditional generation is an ongoing challenge, particularly for cross-modal tasks.
Revised: While DPMs have demonstrated remarkable potential, improving their training and inference efficiency while maintaining flexible and accurate results for conditional generation is an ongoing challenge, particularly for cross-modal tasks.

Original: Our Conditional Discrete Contrastive Diffusion ( CDCD ) loss addresses this by maximizing the mutual information between the conditioning input and the generated output.
Revised: Our Conditional Discrete Contrastive Diffusion ( CDCD ) loss addresses this by maximizing the mutual information between the conditioning input and the generated output.

Original: Our contrastive diffusion mechanisms and negative sampling methods effectively incorporate this loss into DPM training.
Revised: Our contrastive diffusion mechanisms and negative sampling methods effectively incorporate this loss into DPM training.

Original: Extensive experiments on various cross-modal conditional generation tasks demonstrate the efficacy of our approach in bridging drastically differing domains.
Revised: Extensive experiments on various cross-modal conditional generation tasks demonstrate the efficacy of our approach in bridging drastically differing domains.

Original: Exciting directions for future work include incorporating additional guidance into our contrastive learning process, and extending this work to DPMs operating in a continuous space.
Revised: Exciting directions for future work include incorporating additional guidance into our contrastive learning process, and extending this work to DPMs operating in a continuous space.

Original: Ethics Statement.
Revised: Ethics Statement.

Original: As in other media generation works, there are possible malicious uses of such media to be addressed by oversight organizations and regulatory agencies.
Revised: As in other media generation works, there are possible malicious uses of such media to be addressed by oversight organizations and regulatory agencies.

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: We provide implementation details in the supplement and will release our code and pre-trained models to ensure the reproducibility of this work.
Revised: We provide implementation details in the supplement and will release our code and pre-trained models to ensure the reproducibility of this work.

Original: A M ORE Q UALITATIVE R ESULTS A. G ENERATED M USIC S AMPLES
Revised: A M ORE Q UALITATIVE R ESULTS A. G ENERATED M USIC S AMPLES

Original: For qualitative samples of synthesized dance music sequences, please refer to our anonymous page in the supplement with music samples.
Revised: For qualitative samples of synthesized dance music sequences, please refer to our anonymous page in the supplement with music samples.

Original: In addition to the generated music samples on AIST++ Tsuchida et al.
Revised: In addition to the generated music samples on AIST++ Tsuchida et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: and
Revised: and

Original: TikTok Dance-Music Dataset Zhu et al.
Revised: TikTok Dance-Music Dataset Zhu et al.

Original: (2022), we also include some qualitative samples obtained with the music editing operations based on the dance-music genre annotations from AIST++.
Revised: (2022), we also include some qualitative samples obtained with the music editing operations based on the dance-music genre annotations from AIST++.

Original: Specifically, we edit the original paired motion conditioning input with a different dance-music genre using a different dance choreographer.
Revised: Specifically, we edit the original paired motion conditioning input with a different dance-music genre using a different dance choreographer.

Original: Discussion on Musical Representations and Audio Quality.
Revised: Discussion on Musical Representations and Audio Quality.

Original: It is worth noting that we only compare the overall audio quality with that of D2M-GAN Zhu et al.
Revised: It is worth noting that we only compare the overall audio quality with that of D2M-GAN Zhu et al.

Original: This is due to the nature of the different musical representations in the literature of deep-learning based music generation Gan et al.
Revised: This is due to the nature of the different musical representations in the literature of deep-learning based music generation Gan et al.

Original: (2020a)
Revised: (2020a)

Original: ; Dong et al.
Revised: ; Dong et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: ; Gan et al. (2020b); Aggarwal & Parikh (2021).
Revised: ; Gan et al. (2020b); Aggarwal & Parikh (2021).

Original: There are mainly two categories for adopted musical representations in previous works: pre-defined symbolic and learning-based representations Ji et al.
Revised: There are mainly two categories for adopted musical representations in previous works: pre-defined symbolic and learning-based representations Ji et al.

Original: ; Briot et al.
Revised: ; Briot et al.

Original: For the former symbolic music representation, typical options include 1D piano-roll and 2D MIDI-based representations.
Revised: For the former symbolic music representation, typical options include 1D piano-roll and 2D MIDI-based representations.

Original: While these works benefit from the pre-defined music synthesizers and produce music that does not include raw audio noise, the main limitation is that such representations are usually limited to a single specific instrument, which hinders their flexibility to be applied in wider and more complex scenarios such as dance videos.
Revised: While these works benefit from the pre-defined music synthesizers and produce music that does not include raw audio noise, the main limitation is that such representations are usually limited to a single specific instrument, which hinders their flexibility to be applied in wider and more complex scenarios such as dance videos.

Original: In contrast, the learning-based music representations ( i.e. , musical VQ in our case) rely on well-trained music synthesizers as decoders, but can be used as a unified representation for various musical sounds, e.g. , instruments or voices.
Revised: In contrast, the learning-based music representations ( i.e. , musical VQ in our case) rely on well-trained music synthesizers as decoders, but can be used as a unified representation for various musical sounds, e.g. , instruments or voices.

Original: However, the training of such music encoders and decoders for high-quality audio signals itself remains a challenging problem.
Revised: However, the training of such music encoders and decoders for high-quality audio signals itself remains a challenging problem.

Original: Specifically, high-quality audio is a form of high-dimensional data with an extremely large sampling rate, even compared to high-resolution images.
Revised: Specifically, high-quality audio is a form of high-dimensional data with an extremely large sampling rate, even compared to high-resolution images.

Original: For example, the sampling rate for
Revised: For example, the sampling rate for

Original: CD-quality audio signals is 44.1 kHz, resulting in 2,646,000 data points for a one-minute musical piece.
Revised: CD-quality audio signals is 44.1 kHz, resulting in 2,646,000 data points for a one-minute musical piece.

Original: To this end, existing deep learning based works Dhariwal et al.
Revised: To this end, existing deep learning based works Dhariwal et al.

Original: ; Kumar et al.
Revised: ; Kumar et al.

Original: for music generation employ methods to reduce the number of dimensions, e.g. , by introducing hop lengths and a smaller sampling rate.
Revised: for music generation employ methods to reduce the number of dimensions, e.g. , by introducing hop lengths and a smaller sampling rate.

Original: These operations help to make music learning and generation more computationally tractable, but also introduce additional noise in the synthesized audio signals.
Revised: These operations help to make music learning and generation more computationally tractable, but also introduce additional noise in the synthesized audio signals.

Original: In this work, we adopt the pre-trained JukeBox model Dhariwal et al.
Revised: In this work, we adopt the pre-trained JukeBox model Dhariwal et al.

Original: (2020) as our music encoder and decoder for the musical VQ representation.
Revised: (2020) as our music encoder and decoder for the musical VQ representation.

Original: The adopted model has a hop length of 128, which corresponds to the top-level model from their original work Dhariwal et al.
Revised: The adopted model has a hop length of 128, which corresponds to the top-level model from their original work Dhariwal et al.

Original: Jukebox employsmodels: top-, middle-, and bottom-level, with both audio quality and required computation increasing from the first to the last model.
Revised: Jukebox employsmodels: top-, middle-, and bottom-level, with both audio quality and required computation increasing from the first to the last model.

Original: As an example, in the supplemental HTML page, we provide music samples directly reconstructed from JukeBox using the top-level model we employ in our work, compared to the ground-truth audio.
Revised: As an example, in the supplemental HTML page, we provide music samples directly reconstructed from JukeBox using the top-level model we employ in our work, compared to the ground-truth audio.

Original: While they allow for high-quality audio reconstruction (from the bottom-level model, with a hop length of 8), it requires much more time and computation not only for training but also for the final inference, e.g. , 3 hours to generate a 20-second musical sequence.
Revised: While they allow for high-quality audio reconstruction (from the bottom-level model, with a hop length of 8), it requires much more time and computation not only for training but also for the final inference, e.g. , 3 hours to generate a 20-second musical sequence.

Original: As the synthesized music from the top-level model includes some audible noise, we apply a noise reduction operation Sainburg et al.
Revised: As the synthesized music from the top-level model includes some audible noise, we apply a noise reduction operation Sainburg et al.

Original: However, the overall audio quality is not a primary factor that we specifically address in this work on cross-modal conditioning and generation, as it largely depends on the specific music encoder and decoder that are employed.
Revised: However, the overall audio quality is not a primary factor that we specifically address in this work on cross-modal conditioning and generation, as it largely depends on the specific music encoder and decoder that are employed.

Original: This explains why we report similar MOS scores in terms of the general audio quality.
Revised: This explains why we report similar MOS scores in terms of the general audio quality.

Original: A. S YNTHESIZED I MAGES
Revised: A. S YNTHESIZED I MAGES

Original: We present more qualitative examples for text-to-image synthesis and class-conditioned image synthesis in Fig.
Revised: We present more qualitative examples for text-to-image synthesis and class-conditioned image synthesis in Fig.

Original: 5, Fig. 6, and Fig.
Revised: 5, Fig. 6, and Fig.

Original: B D ETAILED P ROOF AND T RAINING B. L OWER B OUND OF CDCD L OSS
Revised: B D ETAILED P ROOF AND T RAINING B. L OWER B OUND OF CDCD L OSS

Original: We show that the proposed CDCD loss has a lower bound related to the mutual information and the number of negative samples N .
Revised: We show that the proposed CDCD loss has a lower bound related to the mutual information and the number of negative samples N .

Original: The derivations below are similar to those from Oord et al.
Revised: The derivations below are similar to those from Oord et al.

Original: [Equation]
Revised: [Equation]

Original: B. C ONVENTIONAL V ARIATIONAL L OSS
Revised: B. C ONVENTIONAL V ARIATIONAL L OSS

Original: The conventional variational loss L vb is derived as follows Sohl-Dickstein et al.
Revised: The conventional variational loss L vb is derived as follows Sohl-Dickstein et al.

Original: (2015b):
Revised: (2015b):

Original: [Equation]
Revised: [Equation]

Original: B. L vb WITH CONDITIONING PRIOR
Revised: B. L vb WITH CONDITIONING PRIOR

Original: Following the unconditional conventional variational loss, we then show its conditional variant with the conditioning c as prior, which has also been adopted in Gu et al.
Revised: Following the unconditional conventional variational loss, we then show its conditional variant with the conditioning c as prior, which has also been adopted in Gu et al.

Original: [Equation]
Revised: [Equation]

Original: B. S TEP -W ISE AND S AMPLE -W ISE C ONTRASTIVE D IFFUSION
Revised: B. S TEP -W ISE AND S AMPLE -W ISE C ONTRASTIVE D IFFUSION

Original: Below, we show the full derivation for the step-wise parallel contrastive diffusion loss.
Revised: Below, we show the full derivation for the step-wise parallel contrastive diffusion loss.

Original: Given that the intermediate variables from z 1: T are also taken into account in this step-wise contrastive diffusion, we slightly modify the initial notation f ( z 0 , c ) =
Revised: Given that the intermediate variables from z 1: T are also taken into account in this step-wise contrastive diffusion, we slightly modify the initial notation f ( z 0 , c ) =

Original: p θ ( z 0 | c ) p θ ( z 0 ) from Eq.(2) in the main paper to f ( z, c ) =
Revised: p θ ( z 0 | c ) p θ ( z 0 ) from Eq.(2) in the main paper to f ( z, c ) =

Original: p θ ( z 0: T | c ) p θ ( z 0: T ) .
Revised: p θ ( z 0: T | c ) p θ ( z 0: T ) .

Original: [Equation]
Revised: [Equation]

Original: Similarly for the sample-wise auxiliary contrastive diffusion
Revised: Similarly for the sample-wise auxiliary contrastive diffusion

Original: [Equation]
Revised: [Equation]

Original: the loss can be derived as follows:
Revised: the loss can be derived as follows:

Original: [Equation]
Revised: [Equation]

Original: B. C ONDITIONAL D ISCRETE C ONTRASTIVE D IFFUSION T RAINING
Revised: B. C ONDITIONAL D ISCRETE C ONTRASTIVE D IFFUSION T RAINING

Original: The training process for the proposed contrastive diffusion is explained in Algo.
Revised: The training process for the proposed contrastive diffusion is explained in Algo.

Original: C A DDITIONAL E XPERIMENTAL D ETAILS AND A NALYSIS C. D ANCE - TO -M USIC T ASK
Revised: C A DDITIONAL E XPERIMENTAL D ETAILS AND A NALYSIS C. D ANCE - TO -M USIC T ASK

Original: Implementation.
Revised: Implementation.

Original: The sampling rate for all audio signals is 22.5 kHz in our experiments.
Revised: The sampling rate for all audio signals is 22.5 kHz in our experiments.

Original: We use 2-second music samples as in Zhu et al.
Revised: We use 2-second music samples as in Zhu et al.

Original: for our main experiments, resulting in 44,100 audio data points for each raw music sequence.
Revised: for our main experiments, resulting in 44,100 audio data points for each raw music sequence.

Original: For the Music VQ-VAE, we fine-tuned Jukebox Dhariwal et al.
Revised: For the Music VQ-VAE, we fine-tuned Jukebox Dhariwal et al.

Original: (2020) on our data to leverage its pre-learned codebook from a large-scale music dataset (approximately 1.2 million songs).
Revised: (2020) on our data to leverage its pre-learned codebook from a large-scale music dataset (approximately 1.2 million songs).

Original: The codebook size K is 2048, with a token dimension d z = 128 , and the hop-length L is 128 in our default experimental setting.
Revised: The codebook size K is 2048, with a token dimension d z = 128 , and the hop-length L is 128 in our default experimental setting.

Original: For the motion module, we deploy a backbone stacked with convolutional layers and residual blocks.
Revised: For the motion module, we deploy a backbone stacked with convolutional layers and residual blocks.

Original: The dimension size of the embedding we use for music conditioning is 1024.
Revised: The dimension size of the embedding we use for music conditioning is 1024.

Original: For the visual module, we extract I3D features Carreira &
Revised: For the visual module, we extract I3D features Carreira &

Original: Zisserman (2017) using a model pre-trained on Kinectics Kay et al.
Revised: Zisserman (2017) using a model pre-trained on Kinectics Kay et al.

Original: (2017) as the visual conditioning information, with a dimension size of 2048.
Revised: (2017) as the visual conditioning information, with a dimension size of 2048.

Original: In the implementation of our contrastive diffusion model, we adopt a transformer-based backbone to learn the denoising network p θ .
Revised: In the implementation of our contrastive diffusion model, we adopt a transformer-based backbone to learn the denoising network p θ .

Original: It includes 19 transformer blocks, in which each block is consists of full-attention, cross-attention and a feed-forward network, and the channel size for each block is 1024.
Revised: It includes 19 transformer blocks, in which each block is consists of full-attention, cross-attention and a feed-forward network, and the channel size for each block is 1024.

Original: We set the initial weight for the contrastive loss as λ = 5 e − 5 .
Revised: We set the initial weight for the contrastive loss as λ = 5 e − 5 .

Original: The numbers of intra- and inter-negative samples for each GT music sample are both 10.
Revised: The numbers of intra- and inter-negative samples for each GT music sample are both 10.

Original: The AdamW Loshchilov & Hutter (2017) optimizer with β 1 = 0 .
Revised: The AdamW Loshchilov & Hutter (2017) optimizer with β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 96 is deployed in our training, with a learning rate of 4 .
Revised: 96 is deployed in our training, with a learning rate of 4 .

Original: 5 e − 4 .
Revised: 5 e − 4 .

Original: We also employ an adaptive weight for the denoising loss weight by gradually decreasing the weight as the diffusion step increases and approaches the end of the chain.
Revised: We also employ an adaptive weight for the denoising loss weight by gradually decreasing the weight as the diffusion step increases and approaches the end of the chain.

Original: The visual module, motion module, and the contrastive diffusion model are jointly optimized.
Revised: The visual module, motion module, and the contrastive diffusion model are jointly optimized.

Revised: The architecture of adopted motion encoder is shown in Tab. 5, which is the same as in Zhu et al.

Original:  Other than the aforementioned implementation details, we also include the mask token technique that bears resemblance to those used in language modelling Devlin et al.
Revised: [Table] Other than the aforementioned implementation details, we also include the mask token technique that bears resemblance to those used in language modelling Devlin et al.

Original: and text-to-image synthesis Gu et al.
Revised: and text-to-image synthesis Gu et al.

Original: for our dance-to-music generation task.
Revised: for our dance-to-music generation task.

Original: We adopt a truncation rate of 0.
Revised: We adopt a truncation rate of 0.

Original: in our inference.
Revised: in our inference.

Original: MOS Evaluation Test.
Revised: MOS Evaluation Test.

Original: We asked a total of 32 participants to participate in our subjective Mean
Revised: We asked a total of 32 participants to participate in our subjective Mean

Original: Opinion Scores (MOS) music evaluations Zhu et al.
Revised: Opinion Scores (MOS) music evaluations Zhu et al.

Original: ; Kumar et al.
Revised: ; Kumar et al.

Original: (2019), among which 11 of them are female, while the rest are male.
Revised: (2019), among which 11 of them are female, while the rest are male.

Original: For the dance-music coherence test, we fuse the generated music samples with the GT videos as post-processing.
Revised: For the dance-music coherence test, we fuse the generated music samples with the GT videos as post-processing.

Original: We then asked each evaluator to rategenerated videos with a score of 1 (least coherent) to 5 (most coherent) after watching the processed video clip.
Revised: We then asked each evaluator to rategenerated videos with a score of 1 (least coherent) to 5 (most coherent) after watching the processed video clip.

Original: Specifically, the participants are asked to pay more attention to the dance-music coherence in terms of the dance moves corresponding to the music genre and rhythm, rather than the overall music quality, with reference to the GT video clips with the original music.
Revised: Specifically, the participants are asked to pay more attention to the dance-music coherence in terms of the dance moves corresponding to the music genre and rhythm, rather than the overall music quality, with reference to the GT video clips with the original music.

Original: As for the overall quality evaluations, we only play the audio tracks without the video frames to each evaluator.
Revised: As for the overall quality evaluations, we only play the audio tracks without the video frames to each evaluator.

Original: As before, they are asked to rate the overall music quality with a score of 1 (worst audio quality) to 5 (best audio quality). 
Revised: As before, they are asked to rate the overall music quality with a score of 1 (worst audio quality) to 5 (best audio quality). [Table] [Table]

Original: Training Cost.
Revised: Training Cost.

Original: For the dance2music task experiments on the AIST++ dataset, we use 4 NVIDIA
Revised: For the dance2music task experiments on the AIST++ dataset, we use 4 NVIDIA

Original: RTX A5000 GPUs, and train the model for approximately 2 days.
Revised: RTX A5000 GPUs, and train the model for approximately 2 days.

Original: For the same task on the TikTok dance-music dataset, the training takes approximately 1.5 days on the same hardware.
Revised: For the same task on the TikTok dance-music dataset, the training takes approximately 1.5 days on the same hardware.

Original: Complete Results for Contrastive Settings.
Revised: Complete Results for Contrastive Settings.

Original: As discussed in our main paper, there are four possible combinations for contrastive settings given different contrastive diffusion mechanisms and negative sampling methods.
Revised: As discussed in our main paper, there are four possible combinations for contrastive settings given different contrastive diffusion mechanisms and negative sampling methods.

Original: Here, we include complete quantitative scores for different contrastive settings in Tab. 5.
Revised: Here, we include complete quantitative scores for different contrastive settings in Tab. 6.

Original: We observe that all the four contrastive settings, including the Step-Inter and SampleIntra settings that are not reported in our main paper, help to improve the performance.
Revised: We observe that all the four contrastive settings, including the Step-Inter and SampleIntra settings that are not reported in our main paper, help to improve the performance.

Original: As we noted, amongst all the settings, Step-Intra and Sample-Inter are more reasonable and yield larger improvements for intra-sample data attributes ( i.e. , beats scores) and instance-level features ( i.e. , genre accuracy scores).[Table] [Table]
Revised: As we noted, amongst all the settings, Step-Intra and Sample-Inter are more reasonable and yield larger improvements for intra-sample data attributes ( i.e. , beats scores) and instance-level features ( i.e. , genre accuracy scores).

Original: Ablation on Music Length.
Revised: Ablation on Music Length.

Original: Although we use 2-second musical sequences in the main experiments to make for consistent and fair comparisons with Zhu et al.
Revised: Although we use 2-second musical sequences in the main experiments to make for consistent and fair comparisons with Zhu et al.

Original: (2022), our framework can also synthesize longer musical sequences.
Revised: (2022), our framework can also synthesize longer musical sequences.

Original: In the supplementary, we show our generated music sequences in 6seconds.
Revised: In the supplementary, we show our generated music sequences in 6seconds.

Original: The quantitative evaluations in terms of different musical sequence lengths are presented
Revised: The quantitative evaluations in terms of different musical sequence lengths are presented

Original: Tab. 6, where we show better performance when synthesizing longer musical sequences.
Revised: Tab. 7, where we show better performance when synthesizing longer musical sequences.

Original: C. T EXT - TO -I MAGE T ASK
Revised: C. T EXT - TO -I MAGE T ASK

Original: Implementation.
Revised: Implementation.

Original: For the text-to-image generation task, we adopt VQ-GAN Esser et al.
Revised: For the text-to-image generation task, we adopt VQ-GAN Esser et al.

Original: (2021b)
Revised: (2021b)

Original: as the discrete encoder and decoder.
Revised: as the discrete encoder and decoder.

Original: The codebook size K is 2886, with a token dimension d z = 256 .
Revised: The codebook size K is 2886, with a token dimension d z = 256 .

Original: VQGAN converts a 256 × 256 resolution image to 32 × 32 discrete tokens.
Revised: VQGAN converts a 256 × 256 resolution image to 32 × 32 discrete tokens.

Original: For the textual conditioning, we employ the pre-trained CLIP Radford et al.
Revised: For the textual conditioning, we employ the pre-trained CLIP Radford et al.

Original: (2021) model to encode the given textual descriptions.
Revised: (2021) model to encode the given textual descriptions.

Original: The denoising diffusion model p θ has 18 transformer blocks and a channel size of 192, which is a similar model scale to the small version of VQ-Diffusion Gu et al.
Revised: The denoising diffusion model p θ has 18 transformer blocks and a channel size of 192, which is a similar model scale to the small version of VQ-Diffusion Gu et al.

Original: We use λ = 5 e − 5 as  the contrastive loss weight.
Revised: We use λ = 5 e − 5 as [Table] the contrastive loss weight.

Original: Similar to the dance-to-music task, we also use the adaptive weight that changes within the diffusion stages.
Revised: Similar to the dance-to-music task, we also use the adaptive weight that changes within the diffusion stages.

Original: We keep the same truncation rate of 0.86 as in our dance-to-music experiment and in Gu et al.
Revised: We keep the same truncation rate of 0.86 as in our dance-to-music experiment and in Gu et al.

Original: Unlike in the dance-to-music experiments, where we jointly learn the conditioning encoders, both the VQ-GAN and CLIP models are fixed during the contrastive diffusion training.
Revised: Unlike in the dance-to-music experiments, where we jointly learn the conditioning encoders, both the VQ-GAN and CLIP models are fixed during the contrastive diffusion training.

Original: Training Cost.
Revised: Training Cost.

Original: For the text2image task experiments on the CUB200 dataste, the training takes approximately 5 days using 4 NVIDIA RTX A5000 GPUs.
Revised: For the text2image task experiments on the CUB200 dataste, the training takes approximately 5 days using 4 NVIDIA RTX A5000 GPUs.

Original: For the same experiments on the
Revised: For the same experiments on the

Original: MSCOCO dataset, we run the experiments on Amazon Web Services (AWS) using 8 NVIDIA Tesla
Revised: MSCOCO dataset, we run the experiments on Amazon Web Services (AWS) using 8 NVIDIA Tesla

Original: V100 GPUs.
Revised: V100 GPUs.

Original: This task required 10 days of training.
Revised: This task required 10 days of training.

Original: [Table] [Figure] [Table] [Figure]
Revised: [Equation]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: Implementation.
Revised: Implementation.

Original: For the class-conditioned image synthesis, we also adopt the pre-trained VQGAN Esser et al.
Revised: For the class-conditioned image synthesis, we also adopt the pre-trained VQGAN Esser et al.

Original: (2021b)
Revised: (2021b)

Original: as the discrete encoder and decoder.
Revised: as the discrete encoder and decoder.

Original: We replace the conditioning encoder with class embedding optimized during the 
Revised: We replace the conditioning encoder with class embedding optimized during the contrastive diffusion training.

Revised: The size of the conditional embedding is 512.

Revised: Other parameters and techniques remain the same, as in the text-to-image task.

Original: Training Cost.
Revised: Training Cost.

Original: For the class-conditioned experiments on the ImageNet, we use 8 NVIDIA Tesla
Revised: For the class-conditioned experiments on the ImageNet, we use 8 NVIDIA Tesla

Original: V100 GPUs running on AWS.
Revised: V100 GPUs running on AWS.

Original: This task required 20 days of training.
Revised: This task required 20 days of training.

Revised: D F URTHER D ISCUSSION ON THE CDCD LOSS

Revised: In this section, we provide our further discussion on the proposed CDCD loss in terms of various aspects, including its relevance to the existing auxiliary losses, the impact of the CDCD strength, as well as additional experimental results.

Original: [Figure]
Revised: While the diffusion models are typically trained and optimized with the conventional variational lower bound loss L vb as we described in the main paper and Appendix B.2, there are several different types of auxiliary losses proposed to further regularize and improve the learning of diffusion models. Specifically, Dhariwal & Nichol (2021) introduces the idea of classifier based guidance for the diffusion denoising probabilistic models with continuous state space. Classifier-free guidance is proposed in Ho & Salimans (2022).

Revised: In the area with discrete diffusion formulations Austin et al.

Revised: ; Gu et al.

Revised: (2022), an auxiliary loss that encourages the model to predict the noiseless token at the arbitrary step is adopted and proven to help with the synthesis quality.

Revised: Similar to the previous cases, we consider the proposed CDCD loss as a type of auxiliary losses, which seeks to provide additional guidance to better learn the conditional distribution p ( x | c ) .

Revised: Specifically, the classifier-free guidance Ho & Salimans (2022) propose to randomly discard conditioning while learning a conditional diffusion generative model, which bears resemblance to our introduced downsampled contrastive steps in Appendix D.3.

Revised: D. I MPACT OF CDCD S TRENGTH

Revised: We further show the ablation studies on the parameter λ , which is the weight of our proposed CDCD loss that characterize the strength of this contrastive regularizer.

Original: small
Revised: this bird is all black and has a long, pointy beak . this bird is a white and black spotted bird. this bird has a pointy yellow bill, black head and wings, and a yellow throat . this bird has wings that are brown and has an orange bill .

Revised: this bird is black and green in color, with a black beak .

Original: [Figure]
Revised: [Figure]

Revised: Correspondence

Revised: [Table]

Revised: We conduct the dance-to-music generation experiments with different values of λ , and show the results in Tab. 9.

Revised: As we observe from the table that the performance in terms of the beat scores are relatively robust for different λ values ranging from 4 e − 5 to 5 e − 5 .

Revised: At the same time, we empirically observe that with a large value of λ , the model converges faster with less training epochs.

Revised: In case of the image synthesis task, we are rather cautious on the strength of the imposed contrastive regularizer.

Revised: Intuitively, the proposed CDCD loss encourages the model to learn a slightly different distribution for negative samples, which could impose a trade-off between the one for the actual data given a specific conditioning.

Revised: Therefore, while a larger value of λ helps with the learning speed, we empirically set the λ to be 5 e − 5 .

Revised: Note that this value is adapted from the weight for other auxiliary losses in previous works Gu et al.

Revised: D. D OWNSAMPLED C ONTRASTIVE S TEPS

Revised: While we show the performance of complete step-wise contrastive diffusion in the main paper, we discuss here an alternative way to implement the proposed method with less computational cost, by downsampling the contrastive steps in the diffusion process.

Revised: Specifically, we randomly downsampled the steps with the proposed CDCD loss, which shares the similar spirit as in the class-free guidance Ho & Salimans (2022) to randomly drop out the conditioning.

Original: table covered with 
Revised: The experimental results are listed in Tab. 10, where there is little performance drop with downsampled contrastive steps. [Figure] Input

Revised: Outputs

Original: [Figure]
Revised: [Figure]

Revised: Diffusion Model

Revised: Outputs w/ Weaker Correspondence

Original: [Figure] 
Revised: [Figure] [Table] [Figure]