Paper: ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models (ICLR-2024)

Reviewer comment (Evaluation):
Since the dispersed convolution is a dense operation with a larger receptive field, it introduces additional computational costs. How significant is the computational overhead of the method?

Author response:
> Q2. Since the dispersed convolution is a dense operation with a larger receptive field, it introduces additional computational costs. How significant is the computational overhead of the method?

Please refer to our response to reviewer URVR Q2. Thank you!

Reviewer comment (Theory):
Is it possible to find a closed-form solution for the optimization problem in Eq. 4? If so, it would be nice to add it to the paper or supplementary. Otherwise, the method implementation is not entirely training-free (although still pretty cheap), and the claims should be adjusted.

Author response:
> Q4. Is it possible to find a closed-form solution for the optimization problem in Eq. 4?

**There is a closed-form solution for dispersed convolution. One only needs to solve a least square problem and no training is needed.** The detailed method is as follows.

We follow the notation in our main paper. Given a convolution layer with kernel $\boldsymbol{k} \in \mathbb{R}^{r \times r}$ and a target kernel size $r'$. We find a dispersion transform $\mathbf{R} \in \mathbb{R}^{r'^2 \times r^2}$ to get a dispersed kernel $\boldsymbol{k}'$. Consider an input feature map $\boldsymbol{h} \in \mathbb{R}^{n \times n}$, we ignore the bias in convolution, since it will not influence our results. 

Structure-level calibration and pixel-level calibration can form an equation set

$$\left\\{\begin{matrix}  interp_d (f_k(\boldsymbol{h})) = f_{\boldsymbol{k}'}(interp_d(\boldsymbol{h})) \\\ \eta f_{\boldsymbol{k}} (\boldsymbol{h}) = \eta f_{\boldsymbol{k}'}(\boldsymbol{h}). \end{matrix}\right. 
 \tag{1}$$

$ interp_d (f_k(\boldsymbol{h})) \in \mathbb{R}^{nd \times nd}, f_{\boldsymbol{k}} (\boldsymbol{h}) \in \mathbb{R}^{n \times n} $.  The equation set has $(nd)^2 + n^2$ equations. Each equation is made up of the sum of terms $k_{ij}h_{ij}$ (elements in $\boldsymbol{k}$ and $\boldsymbol{h}$) where the coefficient of the terms are linear combinations of $R_{ij}$ (elements in $\mathbf{R}$) and constants. For example, when $n=3, r=3$ and $r'=5$, the 5-$th$ equation in pixel-level calibration is
$$
\begin{matrix} k_{11} h_{11} + k_{12} h_{12} + k_{13} h_{13} + \cdots + k_{33} h_{33} = \\\ (R_{7,1} k_{11} + R_{7,2} k_{12} + R_{7,3} k_{13} + \cdots + R_{7,9} k_{33})h_{11} +  \\\  (R_{8,1} k_{11} + R_{8,2} k_{12} + R_{8,3} k_{13} + \cdots + R_{8,9} k_{33})h_{12} + \\\ \cdots + \\\ (R_{19,1} k_{11} + R_{19,2} k_{12} + R_{19,3} k_{13} + \cdots + R_{19,9} k_{33})h_{33} \end{matrix}
\tag{2}$$
We rearrange the equation and get
$$\begin{align}
(1 - R_{7,1}) k_{11} h_{11} + (-R_{7,2})k_{12} h_{11} + \cdots (1 - R_{19, 9})k_{33}h_{33} = 0 \notag.
\end{align} \tag{3} $$
To ensure this equation for all $k_{ij}h_{ij}$, one can let every coefficient be zero. Getting a linear equation set for the 5-$th$ equation in pixel-level calibration:
$$ \left\\{\begin{matrix} R_{7, 1} = -1 \\\ R_{7, 2} = 0 \\\ \cdots \\\ R_{19, 9} = -1
\end{matrix} \right. \tag{4}$$
We do this for every equation in Eqn.(1) to derive a larger set of linear equations of $R_{ij}$. Note that the equation sets are linear equations of $R_{ij}$ and have no $\boldsymbol{h}$ or $\boldsymbol{k}$, making it applicable to all conv kernels and any input feature. Lets back to general case where $\mathbf{R} \in \mathbb{R}^{r'^2 \times r^2}$. Let $A_{\mathrm{structure}}$ denote the coefficient of linear combination for $R_{ij}$ in structure-level calibration and let $b_{\mathrm{structure}}$ denote the right-hand-side constants in the equation set of structure-level calibration. Similarly, we define $A_{\mathrm{pixel}}, b_{\mathrm{pixel}}$. We construct a least square problem

$$A\mathbf{x} = b, \quad A = \left[\begin{matrix}A_{\mathrm{structure}} \\\ \eta A_{\mathrm{pixel}}\end{matrix} \right], b = \left[\begin{matrix}b_{\mathrm{structure}}  \\\ \eta b_{\mathrm{pixel}}\end{matrix} \right], \mathbf{x} = \left[\begin{matrix} R_{1, 1} \\\ R_{1, 2} \\\ \cdots \\\ R_{r'^2, r^2}\end{matrix}\right]. \tag{5}$$ 

The solution is $\mathbf{x} = (A^T A)^{-1} A^T b$. This can be easily solved by math software, i.e., MATLAB. We have included this part in the supplementary materials.

Reviewer comment (Evaluation):
In the "convolution dispersion" section, a linear transformation R and structural-level calibration are used to enlarge the convolution kernel. Does this step add to the model's computational complexity? If so, by how much?

Author response:
Dispersed convolution leads to slight computation overhead which is due to the quadratic complexity $O(N^2)$ of the convolution operator where $N$ is the kernel size. Therefore, we suggest using dispersed kernels as small as possible. In our case, we enlarge $3\times 3$ convolution kernels to $5 \times 5$. Theoretically, it leads to 2.7x more convolution computation in UNet. However, in practice, not all convolution layers need dispersion, and the parallel computing of GPUs will alleviate that overhead. We compare the overhead with and without $5\times 5$ dispersed convolution on $2048\times 2048$ text-to-image with SD v1.5 on one A100 40G GPU.

||Time per iteration|Memory usage|
|-|-|-|
|w/o dispersion|2.47s|6,482MB|
|w/ dispersion|2.78s|9,874MB|
|Overhead|1.13x|1.52x|

Reviewer comment (Experiments):
In the "noise-damped classifier-free guidance" section, a guidance scale w is mentioned. How is this parameter chosen? Does it require adjustments for different tasks or datasets?

Author response:
The choice of the scale parameter, w, follows the same scale used in the classifier-free guidance (cfg) of the original model. Therefore, there is no need for us to adjust the value of w ourselves. Different tasks and datasets may require different values of w for the cfg. The w in our noise-damped classifier-free guidance (ndcfg) can just remain the same as in the standard cfg.

Reviewer comment (Evaluation):
The authors used FID and KID as the metrics that require downsampling images to 229 × 229, does this imply that these metrics are not very suitable for evaluating high-resolution images?

Author response:
FID and KID can only be used to measure the overall generation quality such as the global structure. Due to the downsampling issue, they are not able to reflect the texture details and definition. 
Thus we add four extra metrics pFID, pKID, sFID, and sKID to address this problem. Please refer to the general response for the results, which show that our method achieves significantly better sFID, and sKID scores and highlight the superiority of our method on the texture details.

Reviewer comment (Experiments):
Has the paper considered comparisons with other recently published methods for high-resolution image generation? Would such comparisons further validate the effectiveness of the model?

Author response:
Thank you for your suggestion. We compare our method with SD XL for high-resolution image generation. Please refer to the general response. The results show that our method via SD 2.1 surpasses the SD XL on the subset of Laion 5B and has a faster inference speed and fewer model parameters.

Reviewer comment (Presentation):
Fig. 5-7 and Table 2-4 could be presented in a former manner.

Author response:
Thank you for your suggestion! We have made revisions to the illustration and also reorganized the order of these components accordingly.

Reviewer comment (Experiments):
More ablation studies could be presented in the main text and limitations are suggested.

Author response:
Thanks for your suggestion! We have replenished more ablations and limitations in the supplementary due to the page limit of the main text. Please refer to Sections D and E in the supplementary materials for detailed information on the ablations and limitations.