Spatially Sparse Inference for Generative Image Editing
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep generative models excel at synthesizing photo-realistic images and enablevarious image synthesis and editing applications.
However, when editing a photo,existing methods tend to re-synthesize the entire output from scratch, includingthe unedited region, leading to a signiﬁcant waste of computation, especially forminor editing operations.
In this work, we present Spatially Sparse Inference (SSI),a general-purpose speedup technique that selectively performs computation foredited regions and is compatible with different types of generative models.
Ourkey observation is that user editing is often incremental in the interactive setting.
This allows us to pre-compute the feature maps of the original image.
Given anedited image, we sparsely apply the ﬁlters to the edited regions while reusingthe pre-computed features for the unedited regions.
Based on our algorithm, wepropose Sparse Incremental Generative Engine (SIGE) to convert the theoreticalcomputation reduction to latency reduction on commonly-used hardware.
With1.2%-area edited regions, our method reduces the computation of DDIM by 7 . 5 ⇥and GauGAN by 18 ⇥ while preserving the visual ﬁdelity.
With SIGE, we acceleratethe inference time of DDIM by 3 . [-
 ×-]0 ⇥ on RTX 3090 and 6 . 6 ⇥ on Apple M1 Pro,and GauGAN by 4 . 2 ⇥ on RTX 3090 and 14 ⇥ on Apple M1 Pro.
[Figure]
Our method reduces the computation of DDIM [1] by 4 ⇠ 6 ⇥ and GauGAN [2] by 15 ⇥ for the examples shown in the ﬁgures while preserving the image quality.
Our method could also be applied to existing model compression methods.
When applied to GAN Compression [3], we further accelerate GauGAN by 47 ⇥ .
Introduction
Deep generative models ( e.g ., GANs [4] and diffusion models [5, 6]) excel at synthesizing photorealistic images, enabling many image synthesis and editing applications.
For example, users can edit
[Figure]
an image by drawing sketches [7, 8], semantic maps [7, 2], or strokes [9].
All of these applicationsrequire users to interact with generative models frequently and therefore demand fast inference time.
In practice, content creators often edit images gradually and only update a small image region eachtime.
However, recent generative models often synthesize the entire image, including the uneditedregion, even though only a minor edit has been made, leading to a waste of computation.
As aconcrete example shown in Figure 2, the vanilla DDIM [1] generates an entire image to obtain the16.9% edited region, wasting the majority of the computation.
A naive approach to address this issuewould be to ﬁrst segment the edited region, synthesize the corresponding output region, and blend theoutput back into the original image.
Nevertheless, this method often creates visible inconsistencybetween the edited and unedited regions.
How could we save the computation by only updating theedited regions without losing global coherence?
In this work, we propose Spatially Sparse Inference (SSI), a general method to accelerate generativeimage editing by utilizing the spatial sparsity of edited regions.
As shown in Figure 2, our key idea isto pre-compute the feature maps of the original image and then reuse these activations when updatingthe sparse edited regions.
Speciﬁcally, given a user input, we ﬁrst compute the difference maskbetween the original image and edited image to locate the edited regions.
For each convolutionlayer in the model, we only apply the ﬁlters to the masked region sparsely while reusing the originalimage activations for the unedited region.
The sparse update can signiﬁcantly reduce the computationwithout hurting the image quality.
However, this process involves a gather-scatter process whichhas low hardware efﬁciency with existing deep learning frameworks, leading to signiﬁcant latencyoverheads.
To address the issue, we propose Sparse Incremental Generative Engine (SIGE) to translatethe theoretical computation reduction of our algorithm to latency reduction on various hardware.
To evaluate our method, we automatically create new image editing benchmark datasets on LSUN
Church [10] and Cityscapes [11].
Through extensive experiments, our method could reduce thecomputation on DDIM [1] by 7.5 ⇥ , Progressive Distillation [12] by 2.7 ⇥ , and GauGAN by 18 ⇥regarding MACs * , without loss of visual ﬁdelity.
Compared to existing generative model accelerationmethods [3, 13, 14, 15, 16, 17, 18], our method directly uses the pre-trained weights and could beapplied to these methods as a plugin.
When applied to GAN Compression [3], our method reduces thecomputation of GauGAN by ⇠ 50 ⇥ .
See Figure 1 for some examples of our method.
With SIGE, weaccelerate DDIM 3.0 ⇥ on RTX 3090 and 6.6 ⇥ on Apple M1 Pro, and GauGAN 4 . 2 ⇥ on RTX 3090and 38 ⇥ on Apple M1 Pro.
Our code and benchmark datasets will be released upon publication.
Related Work
Generative models.
Generative models such as GANs [4, 19, 20, 21], diffusion models [6, 5,22], and auto-regressive models [23, 24] have demonstrated impressive photorealistic synthesiscapability.
They have also been extended to conditional image synthesis tasks such as image-to-imagetranslation [25, 7, 26, 27], controllable image generation [9, 28, 2], and real image editing [29,28, 30, 31, 32, 33, 34, 27].
Unfortunately, recent generative models have become increasinglycomputationally intensive, compared to their recognition counterparts.
For example, GauGAN [2]consumes 281G MACs, 500 ⇥ more than MobileNet [35, 36, 37].
Similarly, one key limitation ofdiffusion models [6] is their slow inference time and huge computational cost. In order to generateone image, DDPM requires hundreds or thousands of forwarding steps [6, 22], which is ofteninfeasible in real-world interactive settings.
To improve the sampling efﬁciency of DDPMs, recentworks [1, 38, 39] propose to interpret the sampling process of DDPMs from the perspective of ordinary
[Figure] [Figure]
differential equations.
However, these approaches still require hundreds of steps to generate highquality samples.
To further reduce the sampling cost, DDGAN [40] uses a multimodal conditional GAN to model each denoising step.
Salimans et al .
[12] propose to progressively distill a pre-trained DDPM model into a new model that can be sampled in fewer steps.
Although this approach drasticallyreduces the sampling steps, the distilled model itself still remains computationally prohibitive.
Unlikeprior work, our work focuses on reducing the computation cost of a pre-trained model.
Our work iscomplementary to recent efforts on model compression and distillation.
Model acceleration.
People apply model compression techniques, including pruning [41, 42, 43, 44,45, 46] and quantization [41, 47, 48, 49, 50, 51], to reduce the computation and model size of off-theshelf deep learning models.
Recent work applies Neural Architecture Search (NAS) [52, 53, 54, 55,56, 57, 58] to automatically design efﬁcient neural architectures.
The above ideas can be successfullyapplied to accelerate the inference of GANs [3, 59, 60, 61, 13, 62, 14, 15, 16, 17, 18, 63].
Althoughthese methods have achieved prominent compression and speedup ratios, they all reduce the computation from the model dimension but fail to exploit the redundancy in the spatial dimension during imageediting.
Besides, these methods require re-training the compressed model to maintain performance,while our method can be effortlessly applied to existing pre-trained models.
We show that our methodcan be combined with model compression [3] to achieve ⇠ 50 ⇥ MACs reduction in Section 4.1.
Sparse computation.
Sparse computation has been widely explored in the weight domain [64, 65, 66, 67], input domain [68, 69], and activation domain [70, 71, 72, 73].
For activationsparsity, RRN [74] utilizes the redundancy in consecutive video frames to accelerate video models.
However, their difference sparsity is unstructured, which requires special hardware to reach itsfull speedup potential.
Several works instead use structured sparsity.
Li et al .
[75] use a deeplayer cascade to apply more convolutions on the hard regions than the easy regions to improve theaccuracy and speed of semantic segmentation.
SBNet [70] uses a spatial mask, either from a prioriproblem knowledge or an auxiliary network, to sparsify the activations.
It adopts a tiling-basedsparse convolution algorithm to handle spatial sparsity.
More recent works further integrate the maskgeneration network into the sparse execution network in an end-to-end manner [76] and extend theidea to different tasks [77, 78, 79, 80].
Our method refers SBNet’s sparse kernel implementation.
Compared to SBNet [70], our mask is directly derived from the difference between the originalimage and the edited image.
Additionally, our method reuses the pre-trained weights and does notrequire any auxiliary network or extra model training.
We also introduce other optimizations suchas normalization removal and kernel fusions to better adapt the kernels for image editing.
Method
We build our method based on the following observation: during interactive image editing, a useroften only changes the image content incrementally.
As a result, only a small subset of pixels in a local region is being updated at any moment.
Therefore, we should be able to reuse the activationsof the original input image for the unedited regions.
Figure 3 illustrates our method.
We ﬁrstpre-compute all activations of the original input image.
During the editing process, we locate theediting region by computing a difference mask between the original and edited image.
We then reusethe pre-computed original activations for the unedited region and only update the edited region byapplying convolutional ﬁlters to it.
In Section 3.1, we demonstrate the sparsity of the intermediateactivations and present our main algorithm.
In Section 3.2, we discuss the technical details of howour Sparse Incremental Generative Engine (SIGE) supports the sparse inference and converts thetheoretical computation reduction to measured latency reduction on hardware.
3.1 Activation Sparsity
Preliminary.
First, we closely study the computation within a single layer.
We denote A original l and
A edited l as the input tensor of the original image and edited image to the l -th convolution layer F l ,respectively.
W l and b l are the weight and bias of the F l .
The output of F l with input A edited l could becomputed in the following way due to the linearity of convolution:
[Equation]
where ⇤ is the convolution operator and  A l =
A edited l  A original l .
If we already pre-computed allthe F l ( A original l ) , we only need to compute W l ⇤  A l .
Naïvely, computing W l ⇤  A l has the samecomplexity as W l ⇤ A edited l .
However, since the edited input shares similar features with the originalinput given a small edit,  A l should be sparse.
Below, we discuss different strategies to leverage theactivation sparsity to accelerate model inference.
Our ﬁrst attempt was to prune  A l by zeroing out elements smaller than a certain threshold to achievethe target sparsity.
Unfortunately, it is quite difﬁcult for this pruning method to achieve measuredlatency reduction due to the overhead of on-the-ﬂy pruning and irregular sparsity pattern.
Structured sparsity.
Fortunately, user edits are often highly structured and localized.
As a result, A l should also share the structured spatial sparsity , where the non-zero values are mostly aggregatedwithin the edited regions.
We then directly use the original image and edited image to compute adifference mask and sparsify  A l with this mask.
3.2 Sparse Engine SIGE
But how could we leverage the structured sparsity in our kernel design?
A naïve approach is tocrop a rectangular edited region out of the input feature map for each convolution and only computefeatures for the cropped regions.
Unfortunately, this naive cropping method does not work well forthe irregular edited regions ( e.g ., the third row of Figure 5).
Tiling-based sparse convolution.
Instead, as shown in Figure 4(a), we use a tiling-based sparseconvolution algorithm.
We ﬁrst downsample the difference mask to different scales and dilate thedownsampled masks with extra pixels (1 for diffusion models and 2 for GauGAN).
Then we divide A l into multiple small blocks of the same size spatially and index the spatial difference maskat the corresponding resolution.
Each block index refers to a single block with non-zero elements. We then gather the non-zero blocks (we also call them active blocks) accordingly along the batchdimension and feed them into the convolution F l .
Finally, we scatter the output blocks into a zerotensor according to the indices to recover the original spatial size and add the pre-computed residual F l ( A original ) back.
The gathered active blocks have an overlap of 2 pixels for 3 ⇥ 3 convolution toensure the output blocks of the adjacent input blocks are seamlessly stitched together.
This pipeline in Figure 4(a) is equivalent to a simpler pipeline in Figure 4(b).
Instead of gathering A l , we could directly gather A edited l .
The convolution needs to be computed with bias b l .
Besides,we need to scatter the output blocks into F l ( A original l ) instead of a zero tensor.
In this way, we do notneed to store A original l anymore, which saves more memory and reduces the overheads of addition andsubtraction.
Figure 3 shows the visualization of the pipeline.
[Figure]
Indices Indices
A edited l
A original l  A l
F l ( A original l )
Conv F l Gather
Scatter
Indices Indices
A edited l
F l ( A original l )
However, the aforementioned pipeline still fails to produce a noticeable speedup because of kernelcalling and memory movement overheads in the Gather and Scatter .
For example, the originaldense 3 ⇥ 3 convolution with 128 channels and input resolution 256 ⇥ 256 would take 0.78ms on
RTX 3090.
The sparse convolution using pipeline Figure 4(b) on the example at the third row of
Figure 5 (15.5% editing size) needs 0.42ms in total, where the Gather and Scatter take 0.17ms(about 41%) overheads, which cannot be ignored.
To reduce the overhead, we further optimize our
SIGE by pre-computing normalization parameters and applying kernel fusion.
Pre-computing normalization parameters.
For batch normalization [81], it is easy to remove thenormalization layer during inference time since we can use pre-computed mean and variance statisticsfrom model training.
However, recent deep generative models often use instance normalization [82,83] or group normalization [84, 85], which tends to compute the statistics on the ﬂy during inference.
These normalization layers create overheads as we need to compute the statistics of the full-sizetensors during inference.
However, as the original image and the edited image are quite similar givena small user edit, we assume A original l ⇡ A edited l , which allows us to reuse the statistics of A original lfor the normalization instead of recomputing them for A edited l .
Thus, normalization layers could bereplaced by simple Scale+Shift operations with pre-computed A original l statistics.
Kernel fusion.
Although the Gather and Scatter operations do not involve any arithmeticcomputations, their overheads can not be ignored.
To reduce them, we fuse several kernels with the
Gather and Scatter together [70, 86, 87].
Since Scale+Shift and Nonlinearity are all element-wise operations, we can fuse them into
Gather and Scatter .
In this way, these element-wise operations are only applied to the activeblocks ( i.e ., edited regions).
Furthermore, we perform the in-place computation to reduce the kernelcalling and memory allocation overheads.
In Scatter , we need to copy the pre-computed activation F l ( A original l ) .
This copying operation haslots of redundancy, as the majority of elements from the pre-computed activation do not involveany computation given a small edit and will be discarded by the next Gather .
To reduce the tensorcopying overheads, we fuse the Scatter and the following Gather together by directly gatheringthe edited blocks from the pre-computed activation F l ( A original l ) and the input blocks to be scattered.
Sometimes, the residual connection in the ResBlock [88] contains a shortcut 1 ⇥ 1 convolution tomake the channel number of residual equal to the ResBlock output.
We also fuse the Scatter inthe shortcut branch, main branch, and the residual addition together to avoid the tensor copyingoverheads in shortcut Scatter .
Please refer to our appendix for more details.
Experiments
Below we ﬁrst describe our datasets, models, and evaluation protocols.
We then discuss our mainqualitative and quantitative results.
Finally, we include a detailed ablation study regarding theimportance of each algorithmic design.
Models.
We conduct experiments on three models, including diffusion models and GAN-based models, to explore the generality of our method
• DDIM [1] is a fast sampling approach for diffusion models.
It proposes to interpret the sampling
process of diffusion models through the lens of ordinary differential equations.
• Progressive Distillation (PD) [12] adopts network distillation [89] to diffusion models.
It progressively distills a pre-trained diffusion model into multiple models that can be sampled efﬁciently.• GauGAN [2] is a paired image-to-image translation model which learns to generate a high-ﬁdelityimage given a semantic label map.
[Table] [Table]
Datasets.
We use the following two datasets in our experiments
• LSUN Church .
We use the LSUN Church Outdoor dataset [10] and follow the same preprocessingsteps as prior works [6, 38].
To automatically generate a stroke editing benchmark, we ﬁrst useDetic [90] to segment the images in the validation set.
For each segmented object, we use itssegmentation mask to inpaint the image by CoModGAN [91] and treat the inpainted image as theoriginal image.
We generate the corresponding user strokes by ﬁrst blurring the masked regionwith the median ﬁlter and quantizing it into 6 colors following SDEdit [9].
We collect 454 editingpairs in total (431 synthetic + 23 manual).
Please refer to our appendix for more details regardingthis benchmark.
We evaluate DDIM [1] and PD [12] on this dataset.• Cityscapes .
The dataset [11] contains images of German street scenes.
The training set and thevalidation set consist of 2975 and 500 images, respectively.
Our editing dataset has 1505 editingpairs in total.
Please refer to our appendix for more details.
We evaluate GauGAN [2] on this dataset.
Metrics.
Following the previous work [9, 3, 2], we use the standard metrics Peak Signal Noise Ratio(PSNR, higher is better), LPIPS (lower is better) [92], and Fréchet Inception Distance (FID, loweris better) [93, 94] † to evaluate the image quality on both LSUN Church [10] and Cityscapes [11].For Cityscapes, we also adopt a semantic segmentation metric to evaluate the generated images.
Speciﬁcally, we run DRN-D-105 [95] on the generated images and compute the mean Intersectionover Union (mIoU) of the segmentation results.
Generally, if the generated images look more realisticand correspond to the input better, their mIOUs should be higher.
Implementation details.
The total diffusion steps for DDIM and PD are 100 and 8, respectively,and we correspondingly use 50 and 5 steps for SDEdit.
We dilate the difference mask by 5, 2, 5,and 2 pixels for DDIM, PD with resolution 128, PD with resolution 256 and GauGAN, respectively.
Besides, we apply our sparse kernel to all convolutions whose input feature map resolutions are largerthan 32 ⇥ 32 , 16 ⇥ 16 , 8 ⇥ 16 and 16 ⇥ 32 for DDIM, PD, original GauGAN and GAN Compression, 212 respectively. For all models, the sparse block size for⇥ 3 convolution is 6 and 1 ⇥ 1 convolution is 4.
4.1 Main Results
Image quality.
We report the quantitative results of applying our method on DDIM [1], Progressive
Distillation (PD) [12] and GauGAN [2] in Table 1.
We also show some qualitative examples in
Figure 5.
We compare our method against the following baselines:
• Patch .
We crop the smallest patch that covers all the edited regions in the model input image, feed
the patch into the model, and blend the model output patch into the original output. [Table] [Figure]
• 40% Pruning .
We uniformly prune 40% weights of the models without further ﬁne-tuning, asour method directly uses the pre-trained weights.
Since the ﬁne-grained pruning is unstructured, itrequires special hardware to achieve measured speedup, so we do not report MACs for this baseline.
• 0.19 GauGAN .
We uniformly reduces each layer of GauGAN to 19% channels and train it fromscratch.
• GAN Compression [3].
A general purpose method to compress the conditional GANs.
For PSNR and LPIPS, with G.T. means computing the metric with the ground-truth images.
WithOrig. means computing the metric with the samples generated by the original model.
On LSUNChurch, we only use 431 synthetic images for the PSNR/LPIPS with G.T. metrics, as manual editingdoes not have ground truths.
For the other metrics, we use the whole LSUN Church dataset (431synthetic + 23 manual).
On Cityscapes, we view the synthetic semantic maps as the original input andthe ground-truth semantic maps as the edited input for the PSNR/LPIPS with G.T. metrics, which has1505 samples.
For the other metrics, we additionally include symmetric editing (view the ground-truthsemantic maps as the original inputs and synthetic semantic maps as the edited inputs), which has3010 samples in total.
For the models with method Patch and Ours, whose computation is editingdependent, we measure the average MACs over the whole dataset.
For DDIM and Progressive Distillation, our method outperforms all these baselines consistently and can achieve results on par with the original model.
The Patch inference fails when the editedregion is small as the global context is insufﬁcient.
Although our method only applies convolutionalﬁlters to the local edited regions, we could reuse the global context stored in the original activations.
Therefore, our method could perform the same as the original model.
For GauGAN, our methodalso performs better than GAN Compression [3] with even larger MACs reduction.
When applyingour method to GAN Compression, we further achieve on average 40 ⇥ MACs reduction with minorperformance degradation, beating 0.19 GauGAN by a large margin. [Table]
Model efﬁciency.
For real-world interactive image editing applications, inference acceleration onhardware is more critical than the theoretical computation reduction.
To verify the effectiveness of ourproposed engine, we measure the speedup of the editing examples shown in Figure 5 on four differentdevices, including NVIDIA RTX 3090, NVIDIA RTX 2080Ti, Intel Core i9-10920X CPU, and Apple
M1 Pro, with different computational powers.
We use batch size 1 to simulate real-world use.
For GPU devices, we ﬁrst perform 200 warm-up runs and measure the average latency of the next 200runs.
For CPU devices, we perform 10 warm-up runs and 10 test runs, repeat this process 5 times andreport the average latency.
The latency is measured in PyTorch 1.7 ‡ .
The results are shown in Table 2.
The original Progressive Distillation [12] can only generate 128 ⇥ 128 images, which is too smallfor real use.
We add some extra layers to adapt the model to resolution 256 ⇥ 256 [-.0.
-]{+. 0.
+}
Originalmeans linearly scales each layer to 50% channels, and we only use this to benchmark our latencyresults.
Crop means for each convolution F l , we directly ﬁnd a smallest bounding box covering allthe masked elements of the input activations A edited l , crop it out, feed it into the F l , and scatter theoutput patch in F l ( A original l ) .
For fair comparisons, we also pre-compute the normalization parametersfor this baseline.
We skip the kernel fusion optimization as this baseline is based on highly-optimized
PyTorch implementation.
When the editing pattern is like a rectangle, this algorithm could savesimilar computation with ours ( e.g ., the ﬁrst two examples of DDIM in Figure 5).
However, thespeedup is still worse than ours on RTX 3090 and 2080Ti due to the large memory index overheadsin native PyTorch.
When the editing region is far from a rectangle ( e.g ., the third example of DDIM),the cropped patches have much redundancy.
Therefore, even though only 15.5% region is edited, the MACs reduction is only 1.6 ⇥ .
With 1 . 2% editing size, our method achieves 7.5 ⇥ , 4.6 ⇥ and 18 ⇥ MACs reduction for DDIM, Progressive Distillation and GauGAN, respectively.
With SIGE, weachieve at most 4.1 ⇥ , 2.9 ⇥ , 6.0 ⇥ and 14 ⇥ speedup on RTX 3090, 2080Ti, Intel Core i9-10920X
[Table]
Optimizations
[Table]
[Figure]
[Table]
[Figure]
and Apple M1 Pro, respectively.
When applied to GAN Compression, we achieve 9.5 ⇥ and 38 ⇥latency reduction on Inter Core i9-10920X and Apple M1 Pro, respectively.
4.2 Ablation Study
Below we perform several ablation studies to show the effectiveness of each design choice.
Memory usage.
The pre-computed activations of the original image require additional memorystorage.
We proﬁle the peak memory usage of the original model and our method in PyTorch.
Ourmethod only increases the peak memory usage of a single forward for DDIM, PD, GauGAN, and GAN
Compression by 0.1G, 0.1G, 0.8G and 0.3G, respectively.
Speciﬁcally, our method needs to storeadditional 169M, 56M, 275M and 120M parameters for DDIM [1], PD [12], original GauGAN [2]and GAN Compression [3], respectively, for a single forward.
For the diffusion models, we need tostore activations for all iteration steps ( e.g ., 50 for DDIM and 5 for PD).
However, data movementand kernel computation are asynchronized on GPU, so we could store the activations in CPU memoryand load the on-demand ones on GPU to reduce the peak memory usage.
Speedup of each design.
Table 3a shows the effectiveness of each kernel optimization we add to SIGE for DDIM [1] on RTX 2080Ti.
Naïvely applying the tiling-based sparse convolution couldreduce the computation by 9.1 ⇥ , but the latency reduction is only 1.6 ⇥ due to the large memory overheads in Gather and Scatter .
Pre-computing the normalization parameters could remove the latencyof normalization statistics calculation and reduce the overall latency to 29.6ms.
Fusing element-wiseoperations into the Gather and Scatter could remove some redundant operations that are appliedto the unedited region and also reduce the memory allocation overheads (about 9ms).
Finally, fusingthe Scatter and Gather to Scatter-Gather and Scatter in the shortcut branch and main branchcould further reduce about 1.6ms overheads of tensor copying in Scatter , and achieve 2.9 ⇥ speedup.
Comparisons with TensorRT.
Real-world DL model deployment always resorts to some optimizedlibraries and runtimes as their backends to generate the fastest executable.
To demonstrate theeffectiveness and extensibility of our SIGE, we also implement our kernels in a widely-used DLbackend TensorRT § and benchmark the DDIM latency results on RTX 2080Ti in Table 3b.
Speciﬁcally,our speedup ratio is not diluted but more prominent with TensorRT compared to PyTorch, especiallywhen the editing size is small.
This is because TensorRT has better support for the small convolutionkernel and could make better use of GPU capacity than PyTorch.
Conclusion
In an interactive image editing setting, existing deep generative models often waste computation byre-synthesize the image regions that do not require modiﬁcations.
To solve this issue, we presenta general-purpose method, Spatially Sparse Inference (SSI), that selectively performs computationfor edited regions.
Based on SSI, we propose Sparse Incremental Generative Engine (SIGE) toconvert the theoretical computation reduction to latency reduction on commonly-used hardware.
Ourapproach demonstrates strong performance in various settings.
Speciﬁcally, our proposed SIGEallows us to accelerate the inference time of DDIM by 3 . [-
-]{+0 ⇥ on RTX 3090 and 6 . 6 ⇥ on Apple M
+}
Pro, and GauGAN by 4 . 2 ⇥ on RTX 3090 and 14 ⇥ on Apple M1 Pro.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.• Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Appendix.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix.(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Appendix.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes] See Appendix.
[Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table]
information or offensive content?
[Yes] See Appendix.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]