Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models
Abstract
During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited region. This leads to a significant waste of computation, especially for minor editing operations.
In this work, we present Spatially Sparse Inference (SSI), a general-purpose technique which selectively performs computation for edited regions and accelerates various generative models, including both conditional GANs and diffusion models.
Our key observation is that users tend to make gradual changes to the input image.
This motivates us to cache and reuse the feature maps of the original image.
Given an edited image, we sparsely apply the filters to the edited regions while reusing the cached features for the unedited regions.
Based on our algorithm, we further propose Sparse Incremental Generative Engine (SIGE) to convert the computation reduction to latency reduction on off-the-shelf hardware.
With 1.2%-area edited regions, our method reduces the computation of DDIM by 7 . 5 × and GauGAN by 18 × while preserving the visual fidelity.
With SIGE, we accelerate the inference time of DDIM by 3 . {+
 ×+} on RTX 3090 and 6 . 6 × on Apple M1 Pro CPU, and GauGAN by 4 . 2 × on RTX 3090 and 14 × on Apple M1 Pro CPU.
[Figure]
[1] by 4 ∼ 6 × and GauGAN [2] by 15 × for the examples shown in the figures while preserving the image quality.
Our method could also be combined with existing model compression methods.
When applied to GAN Compression [3], it further reduces the computation of GauGAN by 47 × .
Introduction
Deep generative models, such as GANs [4, 5] and diffusion models [6, 7, 1], excel at synthesizing photo-realistic images, enabling many image synthesis and editing applications.
For example, users can edit an image by drawing sketches [8, 9], semantic maps [8, 2], or strokes [10].
All of these applications require users to interact with generative models frequently and therefore demand fast inference time. [Figure] Computed
[Figure] Editing [Figure] In practice, content creators often edit images gradually and only update a small image region each time.
However, even for a small edit, recent generative models often synthesize the entire image, including the unchanged regions, which leads to a significant waste of computation.
As a concrete example shown in Figure 2(a), the result of the previous editing has already been computed, and the user further edits 9.4% area. However, the vanilla DDIM [1] needs to generate the entire image to obtain the newly edited regions, wasting 80% computation on the unchanged regions.
A naive approach to address this issue would be to first segment the newly edited regions, synthesize the corresponding output regions, and blend the outputs back into the previous output.
Unfortunately, this method often creates visible seams between the newly edited and unedited regions.
How could we save the computation by only updating the edited regions without losing global coherence?
In this work, we propose Spatially Sparse Inference (SSI), a general method to accelerate deep generative models, including conditional GANs and diffusion models, by utilizing the spatial sparsity of edited regions.
Our method is motivated by the observation that feature maps at the unedited regions remain mostly the same during user editing. As shown in Figure 2(b), our key idea is to reuse the cached feature maps of the previous editing and sparsely update the newly edited regions.
Specifically, given a user input, we first compute the difference mask to locate the newly edited regions.
For each convolution layer in the model, we only apply the filters to the masked regions sparsely while reusing the previous activations for the unchanged regions.
The sparse update can significantly reduce the computation without hurting the image quality.
However, the sparse update involves a gather-scatter process, which often incurs significant latency overheads for existing deep learning frameworks.
To address the issue, we propose Sparse Incremental Generative Engine (SIGE) to translate the theoretical computation reduction of our algorithm to measured latency reduction on various hardware.
To evaluate our method, we automatically create new image editing benchmark datasets on LSUN
Church [11] and Cityscapes [12].
Without loss of visual fidelity, we reduce the computation of DDIM [1] by 7.5 × , Progressive Distillation [13] by 2.7 × , and GauGAN by 18 × measured by MACs * .
Compared to existing generative model acceleration methods [3, 14, 15, 16, 17, 18, 19], our method directly uses the off-the-shelf pre-trained weights and could be applied to these methods as a plugin.
When applied to GAN Compression [3], our method reduces the computation of GauGAN by ∼ 50 × .
See Figure 1 for some examples of our method.
With SIGE, we accelerate DDIM 3.0 × on RTX 3090 GPU and 6.6 × on Apple M1 Pro CPU, and GauGAN 4 . 2 × on RTX 3090 GPU and 38 × on Apple M1 Pro CPU.
Our code and benchmark datasets are available at https://github.com/lmxyy/sige .
Related Work
Generative models.
Generative models such as GANs [4, 5, 20, 21], diffusion models [7, 6, 22], and auto-regressive models [23, 24] have demonstrated impressive photorealistic synthesis capability.
They have also been extended to conditional image synthesis tasks such as imageto-image translation [25, 8, 26, 27], controllable image generation [10, 28, 2], and real image editing [29, 28, 30, 31, 32, 33, 34, 27].
Unfortunately, recent generative models have become increasingly computationally intensive, compared to their recognition counterparts.
For example, GauGAN [2] consumes 281G MACs, 500 × more than MobileNet [35, 36, 37].
Similarly, one key [Figure] limitation of diffusion models [7] is their slow inference time and substantial computational cost.
To generate one image, DDPM requires hundreds or thousands of forwarding steps [7, 22], which is often infeasible in real-world interactive settings.
To improve the sampling efficiency of DDPMs, recent works [1, 38, 39] propose to interpret the sampling process of DDPMs from the perspective of ordinary differential equations.
However, these approaches still require hundreds of steps to generate high-quality samples.
To further reduce the sampling cost, DDGAN [40] uses a multimodal conditional GAN to model each denoising step.
Salimans et al .
[13] propose to progressively distill a pre-trained DDPM model into a new model that requires fewer steps.
Although this approach drastically reduces the sampling steps, the distilled model itself remains computationally prohibitive.
Unlike prior work, our work focuses on reducing the computation cost of a pre-trained model.
Our work is complementary to recent efforts on model compression, distillation, and sampling step reduction of the diffusion models.
Model acceleration.
People apply model compression techniques, including pruning [42, 43, 44, 45, 46, 47] and quantization [42, 48, 49, 50, 51, 52], to reduce the computation and model size of off-theshelf deep learning models.
Recent work applies Neural Architecture Search (NAS) [53, 54, 55, 56, 57, 58, 59] to automatically design efficient neural architectures.
The above ideas can be successfully applied to accelerate the inference of GANs [3, 60, 61, 62, 14, 63, 15, 16, 17, 18, 19, 64].
Although these methods have achieved prominent compression and speedup ratios, they all reduce the computation from the model dimension but fail to exploit the redundancy in the spatial dimension during image editing.
Besides, these methods require re-training the compressed model to maintain performance, while our method can be directly applied to existing pre-trained models.
We show that our method can be combined with model compression [3] to achieve ∼ 50 × MACs reduction in Section 4.1.
Sparse computation.
Sparse computation has been widely explored in the weight domain [65, 66, 67, 68], input domain [69, 70], and activation domain [41, 71, 72, 73].
For activation sparsity, RRN [74] utilizes the sparsity in the consecutive video frame difference to accelerate video models.
However, their sparsity is unstructured, which requires special hardware to reach its full speedup potential.
Several works instead use structured sparsity.
Li et al .
[75] use a deep layer cascade to apply more convolutions on the hard regions than the easy regions to improve the accuracy and speed of semantic segmentation.
To accelerate 3D object detection, SBNet [41] uses a spatial mask, either from a priori problem knowledge or an auxiliary network, to sparsify the activations.
It adopts a tiling-based sparse convolution algorithm to handle spatial sparsity.
Recent works further integrate the spatial mask generation network into the sparse inference network in an end-to-end manner [76] and extend the idea to different tasks [77, 78, 79, 80].
Compared to SBNet [41], our mask is directly derived from the difference between the original image and the edited image.
Additionally, our method does not require any auxiliary network or extra model training.
We also introduce other optimizations, such as normalization removal and kernel fusions, to better adapt our engine for image editing. [Figure]
Method
We build our method based on the following observation: during interactive image editing, a user often only changes the image content incrementally.
As a result, only a small subset of pixels in a local region is being updated at any moment.
Therefore, we can reuse the activations of the original image for the unedited regions.
As shown in Figure 3, we first pre-compute all activations of the original input image.
During the editing process, we locate the editing region by computing a difference mask between the original and edited image.
We then reuse the pre-computed activations for the unedited regions and only update the edited regions by applying convolutional filters to them.
In Section 3.1, we show the sparsity of the intermediate activations and present our main algorithm.
In Section 3.2, we discuss the technical details of how our Sparse Incremental Generative Engine (SIGE) supports the sparse inference and converts the theoretical computation reduction to measured speedup on hardware.
3.1 Activation Sparsity
Preliminary.
First, we closely study the computation within a single layer.
We denote A original l and
A edited l as the input tensor of the original image and edited image of the l -th convolution layer F l , respectively.
W l and b l are the weight and bias of F l .
The output of F l with input A edited l could be computed in the following way due to the linearity of convolution: [Equation] sparse
[Equation] [Table]
, where ∗ is the convolution operator and ∆ A l = A edited l − A original l .
If we already pre-computed all the F l ( A original l ) , we only need to compute W l ∗ ∆ A l .
Naïvely, computing W l ∗ ∆ A l has the same complexity as W l ∗ A edited l .
However, since the edited input shares similar features with the original input given a small edit, ∆ A l should be sparse.
Below, we discuss different strategies to leverage the activation sparsity to accelerate model inference.
Our first attempt was to prune ∆ A l by zeroing out elements smaller than a certain threshold to achieve the target sparsity.
Unfortunately, this pruning method fails to achieve measured speedup due to the overheads of on-the-fly pruning and irregular sparsity pattern.
Structured sparsity.
Fortunately, user edits are often highly structured and localized.
As a result, ∆ A l should also share the structured spatial sparsity , where the non-zero values are mostly aggregated within the edited regions.
Figure 4 justifies this by visualizing | ∆ A l | at the l -th layer with different feature map resolutions.
We then directly use the original image and edited image to compute a difference mask and sparsify ∆ A l with this mask.
3.2 Sparse Engine SIGE
But how could we leverage the structured sparsity to accelerate W l ∗ ∆ A l ?
A naïve approach is to crop a rectangular edited region out of the ∆ A l for each convolution and only compute features for the cropped regions (baseline Crop in Table 2).
Unfortunately, this naïve cropping method works poorly for the irregular edited regions ( e.g ., the example shown in Figure 4).
Tiling-based sparse convolution.
Instead, as shown in Figure 5(a), we use a tiling-based sparse convolution algorithm.
We first downsample the difference mask to different scales and dilate the downsampled masks with extra pixels (1 for diffusion models and 2 for GauGAN). Then we divide [Figure] ⊖
Indices
Indices A edited l A original l Δ A l F l ( A original l )
Conv F l Gather
Scatter
Indices
Indices
A edited l
F l ( A original l )
∆ A l into multiple small blocks of the same size spatially and index the spatial difference mask at the corresponding resolution.
Each block index refers to a single block with non-zero elements.
We then gather the non-zero blocks (we also call them active blocks ) accordingly along the batch dimension and feed them into the convolution F l .
Finally, we scatter the output blocks into a zero tensor according to the indices to recover the original spatial size and add the pre-computed residual
F l ( A original ) back.
The gathered active blocks have an overlap with width 2 for 3 × 3 convolution to ensure the output blocks of the adjacent input blocks are seamlessly stitched together [41].
This pipeline in Figure 5(a) is equivalent to a simpler pipeline in Figure 5(b).
Instead of gathering ∆ A l , we could directly gather A edited l .
The convolution needs to be computed with bias b l .
Besides, we need to scatter the output blocks into F l ( A original l ) instead of a zero tensor.
In this way, we do not need to store A original l anymore, which further saves memory and removes the overheads of addition and subtraction.
Figure 3 visualizes the pipeline.
However, the aforementioned pipeline still fails to produce a noticeable speedup because of extra kernel calls and memory movement overheads in the Gather and Scatter .
For example, the original dense 3 × 3 convolution with 128 channels and input resolution 256 × 256 would take 0.78ms on RTX 3090.
The sparse convolution using pipeline Figure 5(b) on the example at the third row of Figure(15.5% edited regions) needs 0.42ms in total, while the Gather and Scatter costs a significant overhead (0.17ms, about 41%).
To reduce it, we further optimize our SIGE by pre-computing normalization parameters and applying kernel fusion.
Pre-computing normalization parameters.
For batch normalization [81], it is easy to remove the normalization layer during inference time since we can use pre-computed mean and variance statistics from model training.
However, recent deep generative models often use instance normalization [82, 83] or group normalization [84, 85], which compute the statistics on the fly during inference.
These normalization layers create overheads as we need to estimate the statistics from the full-size tensors.
However, as the original and edited images are quite similar given a small user edit, we assume A original l ≈ A edited l . This allows us to reuse the statistics of A original l for the normalization instead of recomputing them for A edited l .
Thus, normalization layers could be replaced by simple Scale+Shift operations with pre-computed A original l statistics.
Kernel fusion.
Although the Gather and Scatter operations do not involve any arithmetic computations, their data movement overhead can not be ignored.
To reduce it, we fuse several kernels with the Gather and Scatter together [41, 86, 87].
Since Scale+Shift and Nonlinearity are both element-wise operations, we can fuse them into
Gather and Scatter .
As a result, these element-wise operations are only applied to the active blocks ( i.e ., edited regions).
Furthermore, we perform the in-place computation to reduce the number of kernel calls and memory allocation overheads.
In Scatter , we need to copy the pre-computed activation F l ( A original l ) .
This copying operation is highly redundant, as most elements from the pre-computed activation do not involve any computation given a small edit and will be discarded in the next Gather .
To reduce the tensor copying overheads, we fuse the Scatter and the following Gather by directly gathering the active blocks from the pre-computed activation F l ( A original l ) and the input blocks to be scattered.
Sometimes, the residual connection in the ResBlock [88] contains a shortcut 1 × 1 convolution to match the channel number of the residual and the ResBlock output.
We also fuse the Scatter in the shortcut branch, main branch, and the residual addition together to avoid the tensor copying overheads in the shortcut Scatter .
Please refer to Appendix A for more details of kernel fusion. [Table]
Experiments
Below we first describe our datasets, models, and evaluation protocols.
We then discuss our main qualitative and quantitative results.
Finally, we include a detailed ablation study regarding the importance of each algorithmic design.
Models.
We conduct experiments on three models, including diffusion models and GAN-based models, to explore the generality of our method.
• DDIM [1] is a fast sampling approach for diffusion models.
It proposes to interpret the sampling process of diffusion models through the lens of ordinary differential equations.
• Progressive Distillation (PD) [13] adopts network distillation [89] to diffusion models.
It progressively distills a pre-trained diffusion model into multiple models that can be sampled efficiently. • GauGAN [2] is a paired image-to-image translation model which learns to generate a high-fidelity image given a semantic label map.
Datasets.
We use the following two datasets in our experiments:
• LSUN Church .
We use the LSUN Church Outdoor dataset [11] and follow the same preprocessing steps as prior works [7, 38].
To automatically generate a stroke editing benchmark, we first use Detic [90] to segment the images in the validation set.
For each segmented object, we use its segmentation mask to inpaint the image by CoModGAN [91] and treat the inpainted image as the original image.
We generate the corresponding user strokes by first blurring the masked regions with the median filter and quantizing it into 6 colors following SDEdit [10].
We collect 454 editing pairs in total (431 synthetic + 23 manual).
We evaluate DDIM [1] and PD [13] on this dataset. • Cityscapes .
The dataset [12] contains images of German street scenes.
The training set and the validation set consist of 2975 and 500 images, respectively.
Our editing dataset has 1505 editing pairs in total.
We evaluate GauGAN [2] on this dataset. Please refer to Appendix B for more details about the benchmark datasets.
Metrics.
Following the previous work [10, 3, 2], we use the standard metrics Peak Signal Noise Ratio (PSNR, higher is better), LPIPS (lower is better) [92], and Fréchet Inception Distance (FID, [Table] [Figure]
lower is better) [93, 94] † to evaluate the image quality on both LSUN Church [11] and Cityscapes [12].
For Cityscapes, we also adopt a semantic segmentation metric to evaluate the generated images.
Specifically, we run DRN-D-105 [95] on the generated images and compute the mean Intersection over Union (mIoU) of the segmentation results.
Generally, if the generated images look more realistic and correspond to the input better, their mIOUs should be higher.
Implementation details.
The total diffusion steps for DDIM and PD are 100 and 8, respectively, and we correspondingly use 50 and 5 steps for SDEdit.
We dilate the difference mask by 5, 2, 5, andpixels for DDIM, PD with resolution 128, PD with resolution 256 and GauGAN, respectively.
Besides, we apply our sparse kernel to all convolutions whose input feature map resolutions are larger than 32 × 32 , 16 × 16 , 8 × 16 and 16 × 32 for DDIM, PD, original GauGAN and GAN Compression, respectively. For DDIM [1] and Progressive Distillation [13], we pre-compute and reuse the statistics of the original image for all group normalization layers [84]. For GAN Compression [3], we pre-compute and reuse the statistics of the original image for all instance normalization layers [82] whose resolution is higher than 16 × 32 .
For all models, the sparse block size for 3 × 3 convolution is 6 and 1 × 1 convolution is 4.
4.1 Main Results
Image quality.
We report the quantitative results of applying our method on DDIM [1], Progressive
Distillation (PD) [13] and GauGAN [2] in Table 1.
We also show some qualitative examples in
Figure 6, comparing our method against the following baselines:
• Patch .
We crop the smallest patch that covers all the edited regions in the model input image, feed the patch into the model, and blend the model output patch into the original output. † We use clean-fid for FID calculation.
[Table]
• 40% Pruning .
We uniformly prune weights of the models without further fine-tuning as our method directly uses the pre-trained weights.
Since the fine-grained pruning is unstructured, it requires special hardware to achieve measured speedup, so we do not report MACs for this baseline.
• 0.19 GauGAN .
We reduces each layer of GauGAN to 19% channels ( 21 × MACs reduction) and train it from scratch.
• GAN Compression [3].
A general purpose method to compress the conditional GANs.
For PSNR and LPIPS, with G.T. means computing the metric with the ground-truth images.
With Orig. means computing the metric with the samples generated by the original model.
On LSUN Church, we only use 431 synthetic images for the PSNR/LPIPS with G.T. metrics, as manual editing does not have ground truths.
For the other metrics, we use the whole LSUN Church dataset ( synthetic + 23 manual).
On Cityscapes, we view the synthetic semantic maps as the original input and the ground-truth semantic maps as the edited input for the PSNR/LPIPS with G.T. metrics, which has 1505 samples.
For the other metrics, we additionally include symmetric editing (view the ground-truth semantic maps as the original inputs and synthetic semantic maps as the edited inputs), which has 3010 samples in total.
For the models with method Patch and Ours , whose computation is editing dependent, we measure the average MACs over the whole dataset.
For DDIM and Progressive Distillation, our method outperforms all these baselines consistently and can achieve results on par with the original model.
The Patch inference fails when the edited region is small as the global context is insufficient.
Although our method only applies convolutional filters to the local edited regions, we could reuse the global context stored in the original activations.
Therefore, our method could perform the same as the original model.
For GauGAN, our method also performs better than GAN Compression [3] with even larger MACs reduction.
When applying our method to GAN Compression, we further achieve on average 40 × MACs reduction with minor performance degradation, beating both 0.19 GauGAN and GAN Comp.
(S) .
Model efficiency.
For real-world interactive image editing applications, inference acceleration on hardware is more critical than the theoretical computation reduction.
To verify the effectiveness of our proposed engine, we measure our speedup of the editing examples shown in Figure 6 on four different devices, including NVIDIA RTX 3090, NVIDIA RTX 2080Ti, Intel Core i9-10920X CPU, and Apple [Table] [Figure]
M1 Pro CPU, with different computational powers.
We use batch size 1 to simulate real-world use.
For GPU devices, we first perform 200 warm-up runs and measure the average latency of the nextruns.
For CPU devices, we perform 10 warm-up runs and 10 test runs, repeat this process 5 times and report the average latency.
The latency is measured in PyTorch 1.7 ‡ .
The results are shown in Table 2.
The original Progressive Distillation [13] can only generate 128 × 128 images, which is too small for real use.
We add some extra layers to adapt the model to resolution 256 × 256 .[-0.
-]
Original means linearly scaling each layer to 50% channels, and we only use this to benchmark our efficiency results.
Crop means for each convolution F l , we directly find a smallest bounding box covering all the masked elements of the input activations A edited l , crop it out, feed it into the F l , and scatter the output patch in F l ( A original l ) .
For fair comparisons, we also pre-compute the normalization parameters for this baseline.
When the editing pattern is like a rectangle, this algorithm could save similar computation with ours ( e.g ., the first example of DDIM in Figure 6).
However, the speedup is still worse than ours on RTX 3090, 2080Ti and Intel Core i9-10920X due to the large memory index overheads in native PyTorch.
When the edited region is far from a rectangle ( e.g ., the third example of DDIM), the cropped patches have much redundancy.
Therefore, even though only 15.5% region is edited, the MACs reduction is only 1.6 × .
With 1 . 2% editing size, our method achieves 7.5 × , 4.6 × and 18 × MACs reduction for DDIM, Progressive Distillation and GauGAN, respectively.
With SIGE, we achieve at most 4.1 × , 2.9 × , 6.0 × and 14 × speedup on RTX 3090, 2080Ti, Intel Core i9-10920X and Apple M1 Pro CPU, respectively.
When applied to GAN Compression, we achieve 9.5 × and 38 × latency reduction on Intel Core i9-10920X and Apple M1 Pro CPU, respectively.
4.2 Ablation Study
Below we perform several ablation studies to show the effectiveness of each design choice.
Memory usage.
The pre-computed activations of the original image require additional memory storage.
We profile the peak memory usage of the original model and our method in PyTorch.
Our method only increases the peak memory usage of a single forward for DDIM, PD, GauGAN, and GAN
Compression by 0.1G, 0.1G, 0.8G and 0.3G, respectively.
Specifically, our method needs to store additional 169M, 56M, 275M and 120M parameters for DDIM [1], PD [13], original GauGAN [2] and GAN Compression [3], respectively, for a single forward.
For the diffusion models, we need to store activations for all iteration steps ( e.g ., 50 for DDIM and 5 for PD).
However, data movement and kernel computation are asynchronized on GPU, so we could store the activations in CPU memory and load the on-demand ones on GPU to reduce the peak memory usage.
Speedup of each design.
Table 3a shows the effectiveness of each kernel optimization we add to
SIGE for DDIM [1] on RTX 2080Ti.
Naïvely applying the tiling-based sparse convolution could reduce the computation by 7.6 × , but the latency reduction is only 1.6 × due to the large memory overheads in Gather and Scatter .
Pre-computing the normalization parameters could remove the latency of normalization statistics calculation and reduce the overall latency to 29.6ms.
Fusing element-wise operations into the Gather and Scatter could remove some redundant operations that are applied to the unedited regions and also reduce the memory allocation overheads (about 9ms).
Fi- nally, fusing the Scatter and Gather to Scatter-Gather and Scatter in the shortcut branch and main branch could further reduce about 1.6ms tensor copying overheads, and achieve 2.9 × speedup.
Experiments with TensorRT.
Real-world DL model deployment always resorts to some optimized libraries and runtimes as their backends to generate the fastest executable.
To demonstrate the effectiveness and extensibility of our SIGE, we also implement our kernels in a widely-used DL backend TensorRT § and benchmark the DDIM latency results on RTX 2080Ti in Table 3b.
Specifically, our speedup ratio becomes more prominent with TensorRT compared to PyTorch, especially for small edits, as TensorRT better supports small convolution kernels with higher GPU utilization than PyTorch.
Conclusion & Discussion
For image editing, existing deep generative models often waste computation by re-synthesizing the image regions that do not require modifications.
To solve this issue, we have presented a generalpurpose method, Spatially Sparse Inference (SSI), to selectively perform computation on edited region, and Sparse Incremental Generative Engine (SIGE) to convert the computation reduction to latency reduction on commonly-used hardware.
We have demonstrated the effectiveness of our approach in various hardware settings.
Limitations. As discussed in Section 4.2, our method requires extra memory to store the original activations, which slightly increases the peak GPU memory usage. It may not work on certain memory-constrained devices, especially for the diffusion models ( e.g ., DDIM [1]), since our method requires storing activations of all time steps. Our engine has limited speedup on [-Apple M
-]convolution with low resolution.
When the input resolution is low, the active block size needs to be even smaller to get a decent sparsity, such as 1 or 2. However, such extremely small block sizes have worse memory locality and will result in low hardware efficiency. Besides, we sometimes observe noticeable boundary between the edited regions and unedited regions in our generated samples of GauGAN [2]. This is because, for GauGAN model, the unedited region will also change slightly when we perform normal inference.
However, since our method does not update the unedited region, there may be some visible seams between the edited and unedited regions, even though the semantic is coherent.
Dilating the difference mask would help reduce the gap.
In most cases, the edit will only update the edited regions.
However, sometimes the edit will also introduce global illumination changes such as shadow and reflection.
For this case, as we only update the edited regions, we cannot update the global changes outside the edited regions accordingly.
Societal impact.
In this paper, we investigate how to update user editing locally without losing global coherence to enable smoother interaction with the generative models.
In real-world scenarios, people could use an interactive interface to edit an image, and our method could provide a quick and high-quality preview for their editing, which eases the process of visual content creation and reduces energy consumption, leading to a greener AI application.
The reduced cost also provides a good user experience for lower-end devices, which further democratize the applications of generative models.
However, our method can be utilized by malicious users to generate fake contents, deceive people, and spread misinformation, which may lead to potential negative social impacts. Following previous work [10], we explicitly specify the usage permission of our engine with proper licenses. Additionally, we run a forensics detector [96] to detect the generated results of our method. On
GauGAN, our generated images can be detected with 97 .
2% average precision (AP).
However, on
DDIM [1] and Progressive Distillation [13], the APs are only 56 .
6% and 52 .
Such low APs is caused by the model differences between GANs and diffusion models, as observed in SDEdit [10]. We believe developing forensic methods for diffusion models is a critical future research direction. Acknowledgment. We thank Yaoyao Ding, Zihao Ye, Lianmin Zheng, Haotian Tang, and Ligeng
Zhu for the helpful comments on the engine design.
We also thank George Cazenavette, Kangle
Deng, Ruihan Gao, Daohan Lu, Sheng-Yu Wang and Bingliang Zhang for their valuable feedback.
The project is partly supported by MIT-IBM Watson AI Lab, Kwai Inc, Sony Corporation, and NSF
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See Appendix D. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Our code and benchmark datasets are available at https://github.com/lmxyy/ sige .
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See Appendix D. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Section 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
[Figure]
As mentioned in Section 3.2, we fuse Scatter and the following Gather into a Scatter-Gather operator and also fuse Scatter in the shortcut, main branch and residual addition together.
The detailed fusion pattern is shown in Figure 7. For simplicity, we omit the element-wise operations ( e.g ., Nonlinearity and Scale+Shift ). Below we include more implementation details of each fusion design.
Please refer to our code for the detailed implementation. Scatter-Gather fusion. When a Scatter is directly followed by a Gather , we could fuse these two operators into a Scatter-Gather to avoid copying the original activation F l ( A original l ) . As shown in Figure 8, we pre-build a Scatter Map to track the data source.
For example, if the data at position ( h, w ) in the Scatter output comes from the original activation, then Scatter Map will store
NULL at ( h, w ) (the gray blocks).
Otherwise, it will store a triple at this position.
The first element of the triple indicates the block ID that the data come from, while the latter two indicate the offsets of the data within the block.
Note that the pre-computation is cheap and only needs to be once for each resolution.
Therefore, in the fused Scatter-Gather , we could use the Scatter Map to index and fetch the data we want directly from either the input blocks or the original activation, given the
Gather indices.
For example, if we want to gather the data at location ( h, w ) , we will look up this position in the Scatter Map.
If it is NULL , we would fetch the data at location ( h, w ) in the original activation.
Otherwise, we would fetch data in the input blocks indicated by the triple.
In this way, we could avoid copying the entire original feature map to reduce the overheads in Scatter .
Shortcut Scatter fusion.
The 1 × 1 convolution in the shortcut branch consumes much less computation than the convolution in the main branch, therefore the overheads of Gather and
Scatter weigh more in the shortcut branch.
We fuse the Scatter in the shortcut branch and main branch along with residual addition together into Scatter with Block Residual to reduce these overheads.
Specifically, we first scatter F l +1 output in the pre-computed F l +1 ( A original l )
[Equation]
) and add the original residual F s ( A original l ) only at the scattered locations correspondingly according to Indices .
Then we calibrate the final output with F s output by adding the residual difference
F s ( A edited l ) − F s ( A original l ) at the scattered locations inplace according to Shortcut Indices .
NULL 0,0, 0,1,0 2,0, 2,1, 0,0,1 0,1, 2,0,1 2,1, 1,0,0 1,1,0 1,0,1 1,1, 3,0,0 3,1,0 3,0,1 3,1,
Scatter Map
Scatter
Gather
Indices , , (1,1) (1,3) (2,1) (4,4)
Indices , , (1,1) (1,2) (2,1) (4,4)
Block 0 Block
[Figure]
2 Block
[Figure]
Block 0 Block
Block 2 Block
Scatter-Gather
Indices , , (1,1) (1,2) (
[Figure]
[Figure]
[Figure]
[Figure]
B Benchmark Datasets
We elaborate more details on how we build the synthetic editing dataset
[Figure]
[Figure]
[Figure]
LSUN Church.
Figure 9(a) shows some examples of our synthetic editing on LSUN Church.
The average edited area of the whole dataset is 13.1%.
The detailed distribution is shown in Figure 10a.
Cityscapes.
We collect 27 foreground object semantic masks from the validation set.
The objects include 4 bicycles, 1 motorcycle, 7 cars, 6 trucks, 3 buses, 5 persons
[Figure]
and 1 train.
Figure 11 shows some visualization of the collected semantic masks.
We generate the editing by randomly pasting one of these objects to the ground-truth semantic maps with augmentation.
The augmentation includes random horizontal flip, resize (scale factor in [0 . 8 , 1 . 2] ), translation ( [ − 32 , 32] for height and [ − 64 , 64] for width).
To make the synthetic editing more reasonable, when the scale factor is larger than 1, the height translation can only be positive, otherwise, it can only be negative.
Figure 9(b) shows some editing examples.
The average editing area of the entire dataset is 4.77%.
The detailed distribution is shown in Figure 10b.
C Additional Results
Dilation hyper-parameter.
We show the results of our method with different dilation sizes on
GauGAN in Figure 12.
Increasing the dilation brings more computations but also slightly improves the image quality.
Specifically, the shadow boundary of the added car fades when increasing the dilation.
We choose dilation 1 as the image quality is almost the same as 20 while delivering the best speed.
Quality results at the edited regions.
In Table 1, we show the quantitative quality results of our method on DDIM [1], Progresive Distillation (PD) [13] and GauGAN [2].
For DDIM and PD, the unedited regions in the generated image keep the same as the input image due to the mask trick [10].
For GauGAN, the generated unedited regions vary across different methods.
In this case, the image quality at the unedited regions will influence the metrics we report in Table 1.
We additionally include the quantitative quality results of GauGAN at the sole edited regions in Table 4.
Our method could still preserve the image quality of original GauGAN and match the performance of GAN
Compression [3].
When applied to GAN Compression, it achieves 40 × MACs reduction on average, outperforming GAN Comp.
(S) with less computation.
This indicates that our method could update the edited regions in high quality without losing global context.
Working with self-attention layers.
Recent generative models often have some attention layers to improve the generated image quality [97, 98].
Such attention layers could model long-range
[Table]
[Figure]
the difference mask very well.
and multi-level dependencies across image regions, which seems to break the local correspondence between the edited regions and changed regions in the generated images.
In Figure 13, we visualize the difference between the input and output activations of a 16 × 16 self-attention layer in the vanilla DDIM model.
The pattern of the input and output activation differences are quite similar and correspond to the difference mask very well.
This shows that the local correspondence of user editing still exists with self-attention layers, which justifies our Spatially Sparse Inference algorithm.
Large editing.
In Table 5 and Figure 14, we show the results of large editing ( ∼ 35% ) using our method.
Specifically, we could achieve at most 1 .
7 × speedup on DDIM, 1 . 5 × speedup on PD and 1 .
7 × speedup on GauGAN without losing visual fidelity.
Furthermore, in many practical cases, users can decompose a large edit into several small edits.
Our method could incrementally update the results instantly when the edit is being created, as described below.
Sequential editing.
In Figure 15, we show the results of sequential editing with our method.
Specifically, One-time Pre-computation performs as well as the Full Model , demonstrating that our method can be applied to multiple sequential editing with only one-time pre-computation in most cases.
Moreover, for extremely large edited regions, we could use SIGE to incrementally update the pre-computed features ( Incremental Pre-computation ) and condition the later editing on the recomputed one.
Its results are also as good as the full model.
Therefore, our method could well address the sequential editing.
Additional visualization.
In Figure 16, we show additional synthetic editing visual results of
DDIM [1] and Progressive Distillation [13] on LSUN Church [11].
In Figure 17, we show additional synthetic editing visual results of GauGAN on Cityscapes [12].
D License & Computation Resources
Here we show all the licenses of our used assets.
The model DDIM [1], Progressive Distillation [13],
GauGAN [2] and GAN Compression [3] is under MIT license, Apache license, Creative Commons
[Table]
[Figure]
license and BSD license, respectively.
SDEdit is under MIT license.
The license of Cityscapes [12] is here.
LSUN Church [11] does not have explicit license.
Since our method does not involve any model training, all our generated results are obtained on a single NVIDIA RTX 3090, which only takes 1 ∼ 2 hours to process all the test images ( ∼ 7 , 000 in total) including both the original models and our method.
We measure the model latency on NVIDIA
RTX 3090, 2080Ti, Intel Core i9-10920X CPU, and Apple M1 Pro CPU.
[Figure]
Original
Original
Original
[Table]
[Figure]
[Figure]