Paper: IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
The motivation for image morphing of conceptually different nature images is not clear to me. Why should we study the image morphing of natural images without any human involvement? To my understanding, it may be reasonable to morph images for a certain purpose, e.g., continuously changing the poses or colors, via human involvement. However, how can we guarantee the generated images fit our purpose without any human involvement?

Author response:
Thank you for your thought-provoking question. There is actually one *crucial element of human involvement*, namely the **selection of the endpoint images**. Direct, smooth and realistic image morphing can be used for different downstream purposes depending on the choice of selected images. 

If the selected endpoints directly reflect a certain factor variation, e.g., an object attribute, pose or style, then our method will provide a sequence where this factor is varied continuously, since other variations would hinder the directness of the path. This can be highly beneficial for numerous applications. For example, in controllable generation, as shown in Fig.6 of the revised appendix, we integrate our method with an existing image editing pipeline [1], where our method generates a gradual transition between the original image and its edited version according to certain prompt. With our proposed model adaptation strategy, the transition can reveal a direct & high-fidelity variation that aligns with the editing direction. Thus we believe **our work can also contribute to controllable generation applications in terms of enabling users to control the magnitude of editing or translation**.

Morphing two conceptually different images also has many important usages: 
* **Novel content creation:** Using image morphing techniques to create novel content has been studied for many years [2]. For large-scale generative models, one of the important purposes is to achieve creation of customized novel content that is difficult to see in real world or would require specialized and extensive human involvement [3]. There are two major solutions to achieve novel content generation: a) to design a compositional generative model [4], i.e., combining visual concepts or attributes that exist in a training dataset to generate novel examples, which has been studied by numerous existing works works (e.g., [4,5,6]); and b) to design an interpolative generative model, e.g., the example of interpolating a "lion" and an "eagle" in our work. The former can exponentially expand the control space as more concepts are learned by the generative model, while the latter can potentially provide infinite use of learned concepts. Thus we believe that morphing between different concepts is a highly useful task, and our work is the one of the first attempts.
* **Data augmentation:** As our proposed method can generate **perceptually direct transitions** between endpoint images, for a given dataset, the interpolated samples with our method could be considered as in-distribution images, which can be used as a powerful tool to achieve **semantically meaningful and diverse data augmentation**.
* **Model robustness analysis:** Our work could also be used to generate conceptually-ambiguous samples (e.g., the morph between a "cat" and a "dog") to study the robustness and uncertainty awareness of computer vision models.

Reviewer comment (Theory):
The perceptually-uniform sampling is not reasonable to me. If I am right, morphing should be that given a uniform sequence in time-space, e.g., [0,1], and generating images "uniformly" changing between start and end images. In perceptually-uniform sampling, can we obtain the images corresponding to a given middle time?

Author response:
Perceptually-uniform sampling just finds a continuous mapping from $[0,1] \to [0,1]$ such that the perceptual gap between images $t$ and $t+\delta$ is roughly the same as that of images $t+\delta$ and $t+2\delta$. A middle time (0.5) is still meaningfully defined, and an image can be generated at this time.
In this context, time-uniformity would only be more meaningful than perceptual-uniformity if the underlying space was uniform in some well-defined way.

We provide feedbacks based on our understanding of this question, and more discussion is encouraged.

Reviewer comment (Theory):
The authors argue "adaptive" bottleneck constraint. However, I can not see how the rank is adaptively learned.

Author response:
The rank is adaptively calculated based on the input image pair.

Reviewer comment (Reproducibility):
It would be better to provide an algorithm for both fine-tuning and inference.

Author response:
Thanks for the suggestion. Please refer to Algorithm 1 in the revised appendix.

Reviewer comment (Theory):
In Eq. (5), $\nu$ is a distribution, why it should belong to the data manifold $\mathcal{M}$ which should be the set of samples?

Author response:
Thanks for pointing out the typo. We define $\nu$ as probability measure on manifold $\mathcal{M}$.

Reviewer comment (Theory):
The embedding space of CLIP is locally linear. Since the two semantically distinct text embeddings are not in the global linear space, does it contradict the linear interpolation between the optimized embedding spaces?

Author response:
Thanks for your concern. We avoid this problem by our textual inversion method. Firstly, we set a common prompt **<An image of [token]>** for the inputs no matter their semantic gap, where for semantically distinct pairs, [token] is either a common root class (e.g., "animal") or a concatenation of the pair (e.g., "poker man"). Secondly, we optimize the initial prompt embeddings w.r.t. to the input images. Since two prompt embeddings are initialized with the same common prompt, they stay closed after optimization. We find that this simple method can avoid the nonlinear \& discontinuous issue in CLIP embedding space to some extent, where the usage of a common prompt is a crucial step which is illustrated in Fig.7 of the revised appendix.

Reviewer comment (Theory):
Equation 5 formulates image morphing as walking on the geodesic path in the 2-Wasserstein sense. However, the actual implementation amounts to linear interpolation of CLIP text embeddings, which in no way reflects the theoretical formulation.

Author response:
Equation 5 is formulated following the previous work [1], which is used to reflect the desiderata of image morphing, i.e., the Wasserstein Barycenter problem leads to smoothness and directness of the transition between two images, while the constraint that the morphs should stay on a target data manifold leads to the realism of the morphs. However, it is extremely challenging to apply such formulation to high dimensional real world image data due to computational complexity as well as modeling of manifold; thus, previous work [1] restricts themselves to image data with simple background. We instead utilize alternative, tractable methods, without losing track of the desirable characteristics suggested by the theoretical problem. That is, based on the local linearity of the CLIP embedding space, we perform textual inversion and linear interpolation between inverted text embeddings and latent states which ensures realism and smoothness of interpolation. Our model adaptation strategy then enhances the directness of the morphing process while largely preserve the manifold geometry. Thus, although Equation 5 is not directly solved, we find that our work addresses the same criteria in a more tractable way.

Reviewer comment (Experiments):
It would be helpful if the authors can comment on the robustness of their method relative to the hyper-parameters in model adaptation and sampling.

Author response:
Figure 4 of the main paper shows results of different LoRA ranks. In general, our method achieves reasonable performance across different LoRA ranks, while some ranks generate more visual appealing results than other ranks. If two samples are significantly different in visual appearance, model adaptation with large rank may lead to blurry intermediate samples.  

In terms of conditional guidance scales, we discover that, given fixed LPIPS difference for perceptually-uniform sampling, larger guidance scales are likely to generate longer sequences (less direct morphs). The reconstruction quality also decreases for large guidance scales, but overall the differences are not visually significant. We show an example of sequence generated from different guidance scale in Figure 14 of the revised appendix. Detailed results also report within tables in our appendix.

Reviewer comment (Experiments):
Most of the image pairs used for experiments seem to have certain degree of spatial alignment. It would be helpful to know the rule of thumb for the selection of these image pairs.

Author response:
Thanks for your question. Image pairs selected in our works are collected from three sources: (1) Example image pairs provided in Wang and Golland 2023; (2) Additional images that are manually collected from Internet; (3) Benchmark data for computer vision research. In terms of benchmark evaluation, we select the image pairs based on a simple protocol, i.e., for a given benchmark dataset, we randomly select image pairs whose similarity is under a certain LPIPS threshold, for details, please refer to sec 5.3 in the revised appendix. The intuition behind this process is that, the endpoint images should have certain perceptual connection for the interpolated images between them to be meaningful. In Fig.10 of the revised appendix, we provide a counterexample, showing that morphs between a pair of perceptually unrelated images can be meaningless and benchmark performances on these image pairs are not reliable.

Reviewer comment (Experiments):
In the comparison section, the proposed method is only compared with one previous method, which seems to be not comprehensive. Also, there's only qualitative comparison, while quantitative criterion and user study would better convey the difference.

Author response:
Please refer to general comment 1 for additional comparison with previous methods. Existing automatic morphing tools either create blurred and unrealistic intermediate images or cannot generalize to unseen data, while our work can generate smooth, direct and photorealistic morphs on real world images with large gaps.

Reviewer comment (Experiments):
In the realism criteria and section 3, the authors discussed about manifold and distribution several times, but gave no illustration or discussion in the experiment section.

Author response:
We thank the reviewer for the suggestion. We provide some quantitative comparison with Wang and Golland 2023 in Table 1 of the main paper (with their metrics and hyperparameters). We will defer the user study to future work.

Reviewer comment (Experiments):
Why is data manifold and data distribution discussed in methodologies but not covered in experiments? Would it be possible to visualize the data manifold or distribution with figures or give quantitative experiment results?

Author response:
We discuss the data manifold in the background section since many of the existing morphing works in the literature, which we mention in related work and background, are based upon ideas from manifold geometry and optimal transport to achieve smooth and realistic morphs. However, modeling manifold of high dimensional real world images are challenging, thus, these works restrict themselves to relatively simple image data. Some metrics reported in these works also not applicable for high dimensional image data, therefore, we propose some new metrics. The evaluation metrics proposed in paper are inspired by these works; thus, we mention these concepts to provide some intuition as to the selection of these metrics.

In addition, the design of our method is heavily inspired by the hypothesis that high dimensional image data lie on low dimension manifolds. As pointed out by reviewer meGk, the data prior captured by the pre-trained diffusion model naturally constrains the morphing path on the image manifold and we apply low rank adaptation instead of standard finetuning to preserve the manifold learnt by the pre-trained diffusion model.

To the best of our knowledge, it is challenging to obtain meaningful quantitative measurements related to the manifold or distribution of high dimensional images, so we defer this to future work.

Reviewer comment (Experiments):
Why is the proposed method compared with only one existing diffusion based image interpolation method? Are there other similar works, can they be compared in fair experiment settings and criteria?

Author response:
We note that the compared method is only existing diffusion-based work that is relavent to our targeted task. We provide additional comparison with some non-diffusion methods as illustrated in general comment 1.

Reviewer comment (Evaluation):
The physical meaning of the relative perceptual path diversity (rPPD) score is not clear. Why the score is a ratio that divides the perceptual difference of the endpoint images? Moreover, the accumulated LPIPS difference cannot properly reflect the semantic difference of the input image pair. For example, the morphing for two bears that change a lot in location may be much easier than the morphing for two objects that differ significantly in semantics. To properly choose the LoRA rank, a much simpler metric that depends on CLIP score difference seems to suffice.

Author response:
Our contributions can be summarized as follows: (1) we propose a robust open world automatic image morphing framework with diffusion models (see appendix for comparison with other existing methods); (2) we discover that low-rank adaptation contributes significantly to morphing task; (3) we present an algorithm for perceptually-uniform sampling that encourages the interpolated images to have visually smooth changes; (4) we propose three evaluation criteria, namely smoothness, realism, and directness, with associated metrics to measure the quality of the morphing sequence; and (5) we explore potential applications of morphing in broader domains such as video interpolation, model evaluation as well as data augmentation.

Traditional image morphing is a labor intensive task as it requires involvement from human experts. Thus, creating a morphing sequence with good image quality is costly which restricts the applications of morphing in fewer domains. Some efforts for automatic image morphing have been performed; however, existing tools are only able to handle image pairs with small gaps. We are the first work which can robustly and automatically generate morphing sequences for image pairs with large gaps in semantics, pose, etc. Thus, we believe the technical contributions of our work are significant, as well as its ability to be used easily in applications.

Reviewer comment (Evaluation):
There is no quantitative study for the effectiveness of separate LoRA for unconditional score estimation. Also, there is no smoothness measure for perceptually uniform sampling.

Author response:
Our main contribution is not applying separate LoRA for unconditional score estimation; thus, we do not perform comprehensive evaluation on that. During our experiments, we observed that using separate LoRA for unconditional score estimation performed more stable (compared with randomly discarded conditioning during finetuning) for image inversion, e.g., on CelebA (female-male), maximum end point reconstructed error in terms of LPIPS is **0.137** for separate LoRA unconditional score estimates, the value for randomly discarded conditioning during finetuning is **0.211**. A poor reconstruction may lead to inferior quality; thus, we select separate LoRA for unconditional score estimation. We show two examples of reconstructed images in Fig.13 of the revised appendix. To enhance the difference, we blend real as well as reconstructed images, i.e., $0.5 \times real + 0.5 \times recon$ in Fig. 13 (b). Other approaches could improve reconstruction quality as well, but applying separate LoRA for unconditional score estimates is simple yet effective enough for our task.