Because the generative model approximates the reverse of the inference process, we need to rethink the inference process in order to reduce the number of iterations required by the generative model. Our key observation is that the DDPM objective only depends on the marginal distributions of the latent variables given the data, but not directly on the full joint distribution. Since there are many different joint distributions that can have the same marginals, we explore alternative inference processes that are non-Markovian. These non-Markovian processes lead to new generative processes but, as we will show, they can be trained with the same objective function as DDPMs. This non-Markovian perspective also applies beyond the Gaussian case, as shown for a discrete case in the appendix. 3.1. Non-Markovian forward processes Let us consider a family of inference distributions that are defined in reverse, from time T back to time 1. The distribution of the latent variable at time t-1 is conditioned not only on the variable at time t, but also on the original data at time 0. The mean of this distribution is carefully chosen to ensure that the marginal distributions remain the same as in the original DDPM framework. The variance of this reverse conditional is controlled by a new parameter, sigma. Unlike the original diffusion process, this new forward process is no longer Markovian. The magnitude of the sigma parameter controls how stochastic the forward process is. When sigma approaches zero, the process becomes almost deterministic; knowing the original data and the latent at time t is enough to know the latent at time t-1. 3.2. Generative process and unified variational inference objective Next, we define a trainable generative process that leverages this new non-Markovian inference process. The intuition is that given a noisy observation at time t, we first predict the original data at time 0. Then, we use this prediction to sample the previous state at time t-1 using the reverse conditional distribution we just defined. The model, epsilon-theta, predicts the noise that was added to an image. From this predicted noise, we can derive a prediction for the "denoised observation," which is our estimate of the original data. We then plug this predicted data into our non-Markovian reverse conditional to define the steps of our new generative process. From this setup, we can define a new variational inference objective, which we call J-sigma. At first glance, it seems that we would need to train a different model for every choice of sigma, since each corresponds to a different objective. However, we show that this is not the case. Theorem 1 states that for any choice of the sigma parameter, the new objective J-sigma is equivalent to the original DDPM objective L-gamma for some set of weights, plus a constant value. The original objective has a special property: the optimal model parameters do not depend on the specific weights used in the objective. This justified using a simplified objective with all weights equal to one in the original DDPM paper. Because our new objective is equivalent to the old one, the same logic applies. Therefore, the simplified objective used by Ho and colleagues can also be used as a surrogate objective for our new, more general variational objective. This means we can use a single, pre-trained DDPM model for a wide range of new generative processes. 4. Sampling from Generalized Generative Processes With the simplified objective, we are not just learning a single generative process for the original Markovian model, but a whole family of generative processes for the non-Markovian forward processes we described, each parameterized by sigma. This allows us to use pre-trained DDPM models and simply change the sigma parameter to find a generative process that is better suited for our needs, without any retraining. 4.1. Denoising Diffusion Implicit Models