Paper: Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators (ICLR-2024)

Reviewer comment (Experiments):
As the author indicated in Sec 4.6, it is hard to create pixel-wise flow map by hand, which I personally will be the biggest limitation for this method to be applied in real-world use-case. I am not exactly how the authors demonstrate all the examples in results section, but from the results, they definitely look a real pixel aligned motion field regarding the input image, which is almost impossible to get in applications. From the motion transfer results, the authors discuss using "fewer recursive denoising steps" to get slightly better results. I wonder that's related to the gap in pixel aligned optical flow. I think it worth providing a careful analysis here to help readers understand the potential gap and insights to address these issues which is how this method used in real-world use-cases. I wonder whether there are also tasks (e.g. video stylization) that can better map the input of this method.

Author response:
**Target Flow Construction**

Generating a target flow field may seem complicated but is actually rather straightforward, as outlined in Appendix A1. All flows in our paper (except for the motion transfer results, which are extracted from real videos), are compositions of simple, elementary flows.  As such, it is relatively straightforward to imagine a UI, similar to existing software such as MS Paint, Keynote, or Illustrator, which allows a user to “paint” a target flow.

Additionally, in Appendix A1 we describe how we create target flows that are pixel-aligned with the input image by using SAM [Kirillov et al. 2023], a simple, fast, and highly user-friendly, off-the-shelf segmentation network. 

We also agree with the reviewer that some flows are hard to construct, which is why we include the motion transfer results. We show that these flows can be obtained by extracting them from existing video. 

&nbsp;

**Recursive Steps in Motion Transfer Results**

We agree that fewer recursive steps being needed for the motion transfer experiments may have something to do with the pixel misalignment. Our hypothesis is that fewer recursive steps helps in this case because we do not want to optimize our guidance loss function too aggressively, given that our flow is somewhat misaligned. This is because we are effectively taking fewer “gradient steps” on our guidance function. We provide a visualization of the extent of misalignment between the extracted motion and the source images in Appendix Section A6 and Figure A9. Overall, motion transfer works because our method optimizes a soft objective, which is controlled partly through the number of recursive steps.

Reviewer comment (Presentation):
A suggestion, not a weakness. The demonstration of flow map can be more clear if provided with the color mapping chat with the flow (motion to color).

Author response:
**Flow Legend**

We have already included a flow legend in the form of a colorwheel in Figure 1, where we show how the hue and value of a color corresponds to the motion we are trying to depict. We opted not to include this legend in each figure due to space considerations, but we have updated the caption of each figure to refer back to this legend to clear up any confusion. We thank the reviewer for pointing this out.

Reviewer comment (Evaluation):
In flow loss discussion on page 6, the authors mentions "Without the flow loss our method is able to move objects to the correct location but often hallucinates things in disoccluded areas". Though it is true from the fig.3, the disocclusion regions looks quite wrong with flow loss, I did not quite understand why it is the case. The disocclusion region should not really impact the optical flow since they don't really have correspondence. One possible explanation is that they could potentially create ambiguity between the input image and synthesize image for the estimated flow, and adding flow floss could drive it to remove that potential ambiguity. I wonder whether that's the potential case?

Author response:
**Role of the Flow Loss**

The reason why removing the flow loss results in hallucinations is because the color loss provides no signal to disoccluded regions, but the diffusion network still denoises in the disoccluded areas. There is nothing preventing the diffusion model from hallucinating what it wants in those areas, in particular, from hallucinating an extra apple or an additional waterfall. We also believe the reviewer’s hypothesis for why the flow loss helps. By adding the additional constraint we encourage the network to remove any potential ambiguity in the flow.

Reviewer comment (Reproducibility):
Both Eq. (2) & (3) seems require the gradient w.r.t. the whole flow estimator if I understand correctly. That could be really slow? I wonder the compute time for each. For eq (3), will it make any difference if using f instead of F(x^{\star}, x) ?

Author response:
**Efficiency of Flow Backpropagation**

Backpropagating through the entire loss function takes up roughly 50% of the total sampling time. Using the target flow in the color loss to circumvent the backpropagation overhead is an interesting idea, but does not work because we have to backpropagate through the flow loss anyways. We do note this trick is still useful, and we use it in our ablations when we try to completely remove the flow loss from our guidance loss.

Reviewer comment (Evaluation):
The proposed method requires dense optical flows as the supervision signal. This is not a user-friendly input, as it is not clear how to obtain such flows without programming. DragGAN [Pan et al. 2023] is more user-friendly; users just need to simply drag a small number of pixels to manipulate images. In addition, DragGAN provide feedback to users almost immediately, while the proposed method is slow to execute.

Author response:
**User Interface**

Generating a target flow field may seem complicated but is actually rather straightforward, as outlined in Appendix A1. All flows in our paper (except for the motion transfer results, which are extracted from real videos), are compositions of simple, elementary flows. As such, it is relatively straightforward to imagine a UI, similar to existing software such as MS Paint, Keynote, or Illustrator, which allows a user to “paint” a target flow.

&nbsp;

**DragGAN**

We see the relationship between our method and DragGAN as a series of tradeoffs. While DragGAN can work for sparse point displacements, they sacrifice precision and control. Our method on the other hand works with dense flows, which can be much more accurate. DragGAN is quite fast, but the tradeoff is that GANs are trained only on narrow datasets, while our method can handle a far wider breadth of images _and_ can be controlled by text prompting.

Reviewer comment (Experiments):
In Figure 5, the authors conduct a comparison experiment with DragGAN on out-of-domain images. However, StyleGANs are trained on narrow domains and DragGAN works better if the input image falls in the training image domain. Although GAN inversion still works for out-of-domain images, I think it would be better if the authors can compare with DragGAN using in-domain images (e.g., human facial images) to make the experiment stronger and more convincing.

Author response:
**Out of Domain DragGAN Results**

The goal of Figure 5 was to illustrate the limitations of DragGAN and how our method can overcome them. We wanted to point out two things. Firstly, that DragGAN only works on narrow domains. And secondly, that for DragGAN to work on real images, it also requires a very good GAN inversion method. This is discussed in section 4.4.

Reviewer comment (Experiments):
I think the proposed method can be easily extended for video generation by providing a sequence of flow guidance. It would make the paper stronger if the authors could provide such an example.

Author response:
**Video Generation**

We thank the reviewer for this idea. We have updated the appendix in the supplemental material to add a sequence of deformations by increasingly larger target flows, which can be found in Figure A10 and Section A7. While the motion edits look good, the images are sampled independently of each other. Because of this the model fills in disocclusions inconsistently, and the frames form a slightly choppy video. We believe more work must be done to address this issue, and leave this for future work.

Reviewer comment (Evaluation):
Unfortunately, all experiments and ablations are done mostly qualitatively. This brings a question whether the results are cherry-picked. I'd like to see more quantitative evaluations. Maybe the authors could evaluate the consistency of predictions over longer videos.

Author response:
We first point out that we quantify the trade-off between CLIP alignment scores and Flow Loss for two different datasets, and two different flow methods in Figure 6. The majority of our results are qualitative because there is no standard benchmark for the task that we consider, and we believe that qualitative results effectively convey both the use cases of our method and the high quality of the samples from our method.

&nbsp;

We have already provided completely random samples using our method in the appendix, in Section A3 and Figure A4. In total, there are 60 images sampled with a set random seed, all of which are of high quality in terms of image content and adherence to the target flow.

Reviewer comment (Experiments):
In all experiments, the authors used optical flow (probably from a video). However, how can this method be used if the end user has only one image and wants to edit it? There is one experiment (Fig. 7), which shows that it can be estimated from a different video (and I appreciate it). However, I'd like to see if a hand-drawn optical flow could be used.

Author response:
All experiments in this paper use “hand-drawn” optical flow, which is described in Appendix section A1. These flows are compositions of simple, elementary flows, making it relatively easy to generate by hand. The only experiments for which we use optical flow from a video is the “motion transfer” results in Figure 7.

Reviewer comment (Theory):
eq. (3): later in Sec. 3.3 it is written that the color loss is masked out in "the occluded regions". However, this is not represented in eq. (3). Please update the equation to be precise.

Author response:
We thank the reviewer for catching this. We have updated equation 4 to be more precise and have included a short explanation.