Paper: Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach (ICLR-2024)

Reviewer comment (Writing):
The authors categorize existing methods into two classes, GAN-based and MAE-based methods, and emphasize that one of their limitations is the multiple running times to achieve the large expansion multiple. However, it is unclear why these two types of methods need to run multiple times. Using GAN or MAE is not the crux.

Author response:
(1) For previous methods, in their training stage, they outpaint images from the given central image 128x128 to 192x192, and the training settings are fixed. As a result, there are 2.25 ($\frac{128}{192}^2$) times more contents are generated. 

(2) However, If we want to generate 57.6 ($\approx(2.25)^{5}$) times more contents, we have to forward the pretrained generative network 5 times ($1$->$f(x,)$->$2.25$-$f(x,)$->$(2.25)^{2}$->resize->$f(x,)$->$(2.25)^{3}$->resize)->$f(x,)$->$(2.25)^{4}$->resize->$f(x,)$->$(2.25)^{5}$). 

(3) The reason for categorizing the existing methods into GAN-based and MAE-based is because GAN-based methods are difficult to converge and MAE-based methods are relatively more efficient due to the dropping of masked patches.

Reviewer comment (Theory):
The authors say "to achieve one-step diffusion-based generation, we propose to use relative positional queries and input sub-images as conditions". This is really strange. Why relative positional queries can reduce the number of diffusion steps?

Author response:
The relative positional queries do not reduce the number of diffusions **timesteps**, but reduce the number of **iterations** for large multiple settings. As described in A1.a, previous methods have to forward the network 5 times to obtain 57.6x times more content. In our methods, for any outpainting multiples $n$, we only forward the network one times $1$->$f(x)$->$(2.25)^{n}$. Note that $f(x)$ includes all the diffusion steps.

Reviewer comment (Presentation):
Some parts of Figure 2 are confusing. For example, in the training part, the relative positional embedding takes the two image views as inputs. But Eq.1 shows that only the relative position (m,n) is needed to generate the embedding. The images are not the inputs of it. In the sampling part, it is unclear what the inputs are. I cannot know which of the original image, the cropped image, and the masked image are the inputs.

Author response:
There are some misunderstandings in the training stage. Actually, in the training stage, we feed the $z_{a}$, noised $z_{b_{t}}$ and relative positional embeddings as input. Please note $z_{b_{t}}$ is relatively equivalent to $x_{t}$ in the original diffusion model, which can be calculated by diffusion forward process (see line 9 in Alg.1 in Appendix B).
Then, **for the sampling stage**, the inputs are the given sub-image (anchor image) $z_{a}$ , a randomly initialized Gaussian noise, and relative positional embeddings.
For better understanding, we have modified the Fig.2. Please check our new version.

Reviewer comment (Writing):
In Eq.2, it seems Xb0 should be Zb0.

Author response:
Thanks for your detailed review. We have corrected the typo in our new version.

Reviewer comment (Theory):
There is a strange point in the proposed model. As shown in Figure 2, the relative position only reflects the position relations of the top-left corners between the two views. In other words, even the anchor view and the relative position are fixed, there can still be many different target views. How does this one-to-many mapping influence the model? Furthermore, in the sampling stage, if the input noise, image and relative position are all fixed, how to ensure the outpainting ratios on four edges are the same as what we expected?

Author response:
We are sorry for the confusion of the figure, and actually, the anchor-to-target is a one-to-one mapping. Since the relative position is one of the **core parts** of our method, we hope our clarification could help your understanding:

Instead of using image-level relative position, we will record the top-left positions for **each patch** of the target view, which means that **the target view is determined**. Take an example:

in the training stage, we take the anchor view as the starting point, suppose the positions of the anchor patches are:
$$\begin{pmatrix}
(0,0)&(0,1)&(0,2)\\\\
(1,0)&(1,1)&(1,2)\\\\
(2,0)&(2,1)&(2,2)\\\\
\end{pmatrix},$$ and the relative positions of the target patches are:
$$\begin{pmatrix}
(-1.6,1.7)&(-1.6,3.4)&(-1.6,5.1)\\\\
(1.4,1.7)&(1.4,3.4)&(1.4,5.1)\\\\
(4.4,1.7)&(4.4,3.4)&(4.4,5.1)\\\\
\end{pmatrix}.$$ From the two positions, we know the top-left **shift bias** of the anchor and target view is $[-1.6-0, 1.7-0]$, the **shift scaling** of the anchor and target view is $[(1.4-(-1.6))/(1-0), (3.4-1.7)/(1-0)]$. Then, the position of the target view is certainly determined. 
We also illustrate the details of how to calculate relative positional (both formulation and illustration) in **Appendix C** in our new version.

Due to the randomness of the relative positions between the anchor view and target view in the **training stage**, the model can learn almost all the positional relations between two views. Then, in the **sampling stage**, for example, if the position input sub-image is:
$$anchorPos=\begin{pmatrix}
(0,0)&(0,w)&(0,2w)\\\\
(h,0)&(h,w)&(h,2w)\\\\
(2h,0)&(2h,w)&(2h,2w)\\\\
\end{pmatrix}.$$
we can set the relative position of the target patches as:
$$
targetPos=\begin{pmatrix}
(-h,-w)&(-h,w)&(-h,3w)\\\\
(h,-w)&(h,w)&(h,3w)\\\\
(3h,-w)&(3h,w)&(3h,3w)\\
\end{pmatrix}.
$$
Then, there are 4x more contents in the newly generated images. Specifically, the outpainting ratio (scaling) of the height is $(h-(-h))/(h-0) = 2$, and the outpainting ratio (scaling) of the width is $(w-(-w))/(w-0)) = 2$. Then, the total outpainting ratio is $2(height)*2(width)=4$. We can also find the sub-image positions $anchorPos$ are in the central area of the calculated relative positions $targetPos$ of the target patches. Correspondingly, the PQDiff can generate images on four edges at the same time.

Reviewer comment (Novelty):
I think the authors should further discuss the potential of their method to other potential applications beyond outpainting to further expand the potential impact of this work.

Author response:
Thanks for your suggestions. We have updated the potential applications in our new version (page 22). We believe our method could be applied in image-inpainting, and super-resolutions if we properly change the relative positional embeddings in the training and sampling stages.

Reviewer comment (Novelty):
Are there other methods using POSITIONAL QUERY? As current existing work part does not mention them (is there were) too much.

Author response:
There are a few works learning about how to construct useful positional embeddings in transformers [1, 2, 3]. However, the motivation behind PQDiff is different from their methods. In the image generation area, to our best knowledge, we are the first to propose relative positional queries (RPE beyond the original region) to outpaint images arbitrarily and continuously.

[1] Shiv V, Quirk C. Novel positional encodings to enable tree-based transformers[J]. NeurIPS 2019.

[2] Chi T C, Fan T H, Ramadge P J, et al. Kerple: Kernelized relative positional embedding for length extrapolation[J]. NeurIPS 2022.

[3] Su J, Lu Y, Pan S, et al. Roformer: Enhanced transformer with rotary position embedding[J]. arXiv preprint arXiv:2104.09864, 2021.

Reviewer comment (Presentation):
Question regarding the wording of the contribution or the presentation in Fig. 1. See Q1.

Author response:
Thank you for the quick reply and constructive suggestions!

We have modified the Fig.1 in our new revised version. Please let us know if you have any further questions.

Reviewer comment (Experiments):
More results are needed to justify the claim of continuous results. See Q2.

Author response:
Thanks for your constructive suggestions. We have added the figure over continuous multiples with increasing multiples in our new version (see Fig.7 in the Appendix).

Reviewer comment (Reproducibility):
Is it precise to say that PQDiff handles inputs with various sizes and ratios while the output size is fixed? Assume the goal is to generate an outpainting image with a multiple of 10x given an input image of size (192, 192). Does PQDiff generate a (192, 192) version of the 10x outpainting image and have to leverage an additional decoder (perhaps super-resolution methods) to resize it to (1920, 1920)?

Author response:
Yes. As we adopt diffusion models, the output sizes are fixed. If the users want to generate images with 10x resolutions, PQDiff has to leverage an additional decoder (e.g., super-resolution).

Reviewer comment (Experiments):
Can the author provide results on more multiple besides (2.5x, 5x, 11.7x) used in other sota to justify their claim of continuous multiples? For example, how is the performance of 99x? My suggestion would be a figure of FID over continuous multiples or spectrums of outpainting images with increasing multiples.

Author response:
Thanks for your constructive suggestions. We have added the figure over continuous multiples with increasing multiples in our new version (see Fig.7 in the Appendix).

Reviewer comment (Evaluation):
What is the quantitative evaluation of Sec 4.2 **Outpainting in an arbitrary position**, i.e. FID under different positions? Does the model prefer certain positions?

Author response:
We conduct a set of experiments when outpainting with **arbitrary positions**. Specifically, we randomly crop the anchor view and target view, and by using RandomCropResize. The crop ratios of the anchor view and target view are $(0.4, 0.4)$ and $(0.9, 0.9)$, respectively. This setting aims to keep PQDiff generating **at least** 2.25 times contents (only when the target view contains the anchor view, the generated region are the same with centered outpainting. Otherwise, PQDiff will generate more regions than centered outpainting). Here are the results:

| Method | IS | FID | Center Outpainting | 
| --- | --- | --- | ---- |
| PQDiff + copy |  3.981 | 20.100 | $\checkmark$ |
| PQDiff (Center position) |  3.849| 29.446 | $\checkmark$ |
| PQDiff (arbitrary position) | 3.781 |30.129 | $\times$ |

Please note that due to the random location, it's difficult to add "copy" operations for arbitrarily outpainting. We find PQDiff (arbitrary position) obtains a bit lower IS score than PQDiff (Center position), which is because the generated region of PQDiff (arbitrary position) is larger than PQDiff (Center position) on average.

Reviewer comment (Presentation):
The generated and input images in Supp. Sec. J (Fig. 11, 12, ...) can be put side by side for clearer comparison.

Author response:
Thank you for your suggestion. We have put the Sec.K (Appendix) side by side for clearer comparison in our new version.