D EEP S INGLE
I MAGE M ANIPULATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Image manipulation has attracted much research over the years due to the popularity and commercial importance of the task.
In recent years, deep neural network methods have been proposed for many image manipulation tasks.
A major issue with deep methods is the need to train on large amounts of data from the same distribution as the target image, whereas collecting datasets encompassing the entire distribution of images is impossible.
In this paper, we demonstrate that simply training a conditional adversarial generator on the single target image is sufﬁcient for performing complex image manipulations.
We ﬁnd that the key for enabling single image training is extensive augmentation of the input image and provide a novel augmentation method.
Our network learns to map between a primitive representation of the image (e.g. edges and segmentation) to the image itself.
At manipulation time, our generator allows for making general image changes by modifying the primitive input representation and mapping it through the network.
We extensively evaluate our method and ﬁnd that it provides remarkable performance.
I NTRODUCTION
Images capture a scene at a speciﬁc point in time.
Viewers often wish the scene had been different e.g. that objects were arranged differently.
Due to the popularity of this task, it has been the focus of much research and also of many companies and products e.g. Instagram and Photoshop.
Deep learning methods have signiﬁcantly boosted performance of image manipulation methods for which large training datasets can be obtained e.g. super-resolution or facial inpainting.
User-captured photographs follow a long tailed distribution.
Some classes of photographs are very common e.g. faces or cars.
On the other hand a large proportion of photographs capture a rare object class or conﬁguration.
Training deep learning methods that capture the entire distribution images can be very hard, particularly for generative models that are slow and tricky to train.
Training models on just the target image is emerging as an alternative to training deep models on large image datasets.
Although this is counter-intuitive as deep learning methods typically require many training samples, single-image methods have recently demonstrated some promising results.
In this paper, we introduce a novel method for training deep conditional generative models from a single image.
The objective differs from popular single-image methods e.g. Deep Image Prior and
SinGAN that focus on unconditional image manipulation.
The training image is ﬁrst represented with a primitive representation, which can be unsupervised (an edge map, unsupervised segmentation), supervised (segmentation map, landmarks) or a combination of both.
We use a standard adversarial conditional image mapping network to learn to map between the primitive representation and the image.
In order to extend the training set (which simply consists of a single image), we perform extensive augmentations.
The choice of augmentation method makes a signiﬁcant difference to the method’s performance.
We ﬁnd the crop-and-ﬂip augmentations typically used in conditional image generation are insufﬁcient for providing a sufﬁciently rich training distribution.
We propose to use a thin-plate-spline (TPS) augmentation method and show that it is key to the success of our method.
After training, we are able to perform challenging image manipulation tasks by modifying the primitive representation.
Our method is evaluated extensively and displays remarkable results.
Our contributions in this paper:
A general purpose approach for training conditional generators from a single image.
Proposing a TPS-based augmentation for conditional image generation, and demonstrating its importance for single image training.
A primitive representation allowing concurrent low and high-level image editing.
Extensive evaluations showing remarkable visual performance, and the introduction of a new benchmark enabling quantitative evaluation of single image manipulation.
R ELATED W ORK
Classical image manipulation methods: Image manipulation has attracted research for decades from the image processing, computational photography and graphics communities.
It would not be possible to survey the scope of this corpus of work in this paper.
We refer the reader to the book by
Szeliski (2010) for an extensive survey, and to the Photoshop software for a practical collection of image processing methods.
A few notable image manipulation techniques include: Poisson Image
Editing (P´erez et al., 2003), Seam Carving (Avidan & Shamir, 2007), PatchMatch (Barnes et al.) and
ShiftMap (Pritch et al., 2007).
Learning a high-resolution parametric function between a primitive image representation and a photo-realistic image was very challenging for pre-deep learning methods.
Deep conditional generative models: Image-to-image translation maps images from a source domain to a target domain, while preserving the semantic and geometric content of the input images.
Most image-to-image translation methods use Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) that are used in two main scenarios: i) unsupervised image translation between domains (Zhu et al., 2017a;
Kim et al., 2017; Liu et al., 2017; Choi et al., 2018) ii) serving as a perceptual image loss function (Isola et al., 2017; Wang et al., 2017; Zhu et al., 2017b).
Existing methods for image-to-image translation require many labeled image pairs.
Several methods e.g Dekel et al.
(2017), are carefully designed for image manipulation, however they require large datasets which are mainly available for faces or interiors and cannot be applied to the long-tail of images.
Non-standard augmentations: Conditional generation models typically use crop and ﬂip augmentations.
Classiﬁcation models also use chromatic and noise augmentation.
Recently, methods have been devised for learning augmentation for classiﬁcation tasks e.g. AutoAugment (Cubuk et al., 2018).
Mounsaveng et al.
(2019) learned warping ﬁelds for augmenting classiﬁcation networks.
Thin-plate-spline transformation have been used in the medical domain e.g. Tang et al.
, but they are used for training on large datasets rather than a single sample.
Zhao et al.
(2019) learned augmentations for training segmentation networks from a single annotated 3D medical scan (using a technique similar to Kanazawa et al.
however they require a large unlabeled dataset of similar scans which is not available in our setting.
TPS has also been used as a way of parametrizing warps for learning dense correspondences between images e.g. Han et al.
(2018) and Lee et al.
Learning from a single image: Although most deep learning works use large datasets, seminal works showed that single image training is effective in some settings.
Asano et al.
(2019) showed that a single image can be used to learn deep features.
Limited work has been done on training image generators from a single-image.
Deep Image Prior (Ulyanov et al., 2018), retargeting (Shocher et al., 2018a) and super-resolution (Shocher et al., 2018b) from a single image.
Recently, the seminal work of Shaham et al.
(2019), presented a general approach for single image generative model training.
However these methods are mainly focused on unconditional image manipulations (or very limited conditional manipulation).
Our method is focused on conditional manipulations and is able to affect signiﬁcantly more elaborate changes to images including to large objects in the scene.
D EEP SIM: L EARNING C ONDITIONAL G ENERATORS FROM A S INGLE I MAGE
We propose DeepSIM, a conditional generative adversarial network (cGAN) for learning to map from a primitive representation (e.g. edges, segmentation) to the image.
The approach has several objectives: i) single image training ii) ﬁdelity - the output should reﬂect the primitive representation iii) appearance - the output image should appear to come from the same distribution as the training image.
We present a novel augmentation method that allows standard cGAN architectures to achieve these objectives.
The particular type of augmentations used is critical as they provide a prior over the data.
Having a good prior is important as the training set consists of merely a single image.
3. M ODEL :
Our model design follows standard practice for cGAN models (particularly Pix2PixHD (Wang et al., 2017)).
Let us denote our training image pair ( x, y ) where y ∈ R d x × d y × 3 is the input image ( d x and d y are the number of rows and columns) and x ∈ R d x × d y × d p is the corresponding image primitive ( d p is the number of channels in the image primitive).
We learn a generator network G : R d x × d y × d p → R d x × d y × 3 , which learns to map input image primitive x to the generated image G ( x ) .
The ﬁdelity of the result is measured using the VGG perceptual loss (cid:96) perc : ( R d x × d y × 3 , R d x × d y × 3 ) → R (Johnson et al., 2016) , which compares the differences between two images using a set of activations extracted from each image using a VGG network pre-trained on the ImageNet dataset (we follow the implementation in Wang et al. We therefore write the reconstruction loss (cid:96) rec :
[Equation]
Conditional GAN loss: Following standard practice, we add an adversarial loss which measures the ability of a discriminator to differentiate between the (primitive, generated image) pair ( x, G ( x )) and the (primitive, true image) pair ( x, y ) .
The conditional discriminator D : ( R d x × d y × d p , R d x × d y × 3 ) → [0 , 1] is implemented using a deep classiﬁer which maps a pair of primitive and corresponding image into the probability of the two being a groundtruth primitive-image pair. D is trained adversarially against the generator network G .
The loss of the discriminator ( (cid:96) adv ) is:
[Equation]
The combined loss (cid:96) total is the sum of the reconstruction and adversarial losses, weighted by a constant α :
[Equation]
Primitive
[Table]
Primitive
[Table]
Image
[Table] 3. A UGMENTATIONS :
When large datasets exist, ﬁnding the generator G and conditional discriminator D that optimize (cid:96) total under the empirical data distribution can result in a strong generator G . However, as we only have a single image pair ( x, y ) , this formulation severely overﬁts.
This has the negative consequence of G not being able to generalize to new primitive inputs.
In order to generalize to new primitive images, the size of the training dataset needs to be artiﬁcially increased so as to cover the range of expected primitives.
Conditional generative models typically use simple crop and shift augmentations.
We will later show (Sec. 4) that this simple augmentation strategy however will not generalize to primitive images with non-trivial changes.
We ﬁrst construct a non-smooth warp t by selecting a set of pixel locations ( i, j ) and perturbing them by a uniformly distributed shifts ( d i , d j ) ∼ ( U ( − d x L , d x L ) , U ( − d y L , d y L )) s.t. t ( i, j ) = ( i + d i , j + d j ) (we use L = 10 ). In order to have a simple transformation, we do not do this for every pixel but only for a set of 4 equispaced pixels along each axis of the image.
We learn a warp f : ( R d x , R d y ) →
( R d x , R d y ) using a thinplate-spline (TPS) to smooth the random transformation t into a more realistic warp ( i (cid:48) , j (cid:48) )
= f ( i, j ) .
f is parameterized using a thin-plate spline (see Duchon (1977) for more details).
The TPS loss
L TPS consists of two terms, a loss that encourages f to be similar to t while the other encourages smoothness:
[Equation]
Where f xx , f xy , f yy denote the second order partial derivatives of f .
λ is a regularization constant determining the smoothness of the warp.
The optimization over the warp f can be performed very efﬁciently e.g. Donato & Belongie (2002).
We denote the distribution of random TPS that can be generated using the above procedure as Ω .
3. O PTIMIZATION :
At training time, we sample random warps from the distribution Ω .
We use each random warp f ∼ Ω to transform both the input primitive x and image y to create a new training pair ( f ( x ) , f ( y )) (where we denote f ( x )( i, j ) = x ( i (cid:48) , j (cid:48) ) where ( i (cid:48) , j (cid:48) )
= f ( i, j ) ). We optimize the generator and discriminator adversarially to minimize the expectation of the loss (cid:96) total under the empirical distribution of random TPS warps: [Equation]
We implemented the conditional GAN using the Pix2PixHD architecture.
We kept the same hyperparameters as in the ofﬁcial repository except changing the number of iterations to 16000 .
3. P RIMITIVE IMAGES :
In order to be able to edit the image, we condition our generator on a representation of the image that we call the image primitive.
The required properties of the image primitives are being able to precisely specify the required output image and the ease of manipulation by image editor.
These two objectives are in conﬂict, although the most precise representation of the edited image is the edited image itself, this level of manipulation is impossible to achieve by a human editor, in fact simplifying this representation is the very motivation for this work.
Two standard image primitives used by previous conditional generators are the edge representation of the image and the semantic instance/segmentation map of the image.
Segmentation maps provide information on the high-level properties of the image, but give less guidance on the ﬁne-details.
Edge maps provide the opposite trade-off.
To achieve the best of both worlds, we propose a novel primitive, which includes both the edge and segmentation maps combined together (we dub this representation - ”super primitive”).
The advantages of this primitive representation will be shown in Sec. 5.
Note that although Pix2PixHD proposes using a combination of segmentation maps and the edges of the instance maps, their representation is different from ours.
The instance segmentation edges merely provide a method for overcoming the indexing issue present in instance maps, but do not provide the high-resolution of ﬁne details in the edge maps as in our proposed approach.
Our editing procedure is illustrated in App.
B.
E XPERIMENTS 4. Q UALITATIVE EVALUATION
We present many results of our method in the main paper and the appendix.
In Fig.
1, our method is able to generate very high resolution images from single image training.
In the top row we are able to perform ﬁne changes to the facial images from edge primitives e.g. raising the nose and ﬂipping the eyebrows.
In the second row, we show complex shape transformations by using segmentation primitives.
Our method was able to add a third wheel to the car and convert its shape into a sports car.
This shows the power of the segmentation primitive, enabling major changes to the shape using simple operations.
On the bottom row, we show that our method can perform free-form changes, completely changing the shape of an image while preserving ﬁne texture.
In Fig.
we demonstrate the advantages of our single image model by comparing to two models that were trained on a large dataset.
We present edges-to-image translation results on two different shoes.
We can see that Pix2PixHD-MI (Pix2PixHD that was trained on the entire edge2shoes dataset, where ”MI” is an acronym for ”Multi Image”) is unable to capture the correct identity of the shoes as there are multiple possibilities for the appearance of the shoe given the edge image.
BicycleGAN is able to take as input both the edge map and guidance for the appearance (style) of the required shoe.
Although it is able to capture the general colors of the required shoe, it is unable to capture the ﬁne details of the shoes (e g. shoe laces and buckles).
We believe that the loss of information is a general disadvantage of training on large datasets, a general mapping function becomes less specialized and therefore less accurate on individual images.
In Fig.
we present results on a paint-to-image task.
Our method was trained to map from a rough paint image to an image of a tree, while SinGAN was trained using the authors’ best practice.
We can see that SinGAN outputs an image which is more similar to the paint than a photorealistic image.
Our method is able to change the shape of the tree to correspond to the paint while keeping the appearance of the tree and background as in the training image.
4. Q UANTITATIVE EVALUATION
As previous single image generators have mostly operated on unconditional generation, there are no established datasets and metrics to conduct such an evaluation.
We propose a new video-based benchmark for conditional single image evaluation over datasets spanning different scenes and primitives.
A single frame from each video is designated for training, where the network is trained to map the primitive image to the designated training frame.
The trained network is then used to map from primitive to image for all the other frames in the video and compute the prediction error using LPIPS (Zhang et al., 2018) and ﬁdelity using SIFID (Shaham et al., 2019).
We use all 16 video segments from the Cityscapes dataset (Cordts et al., 2016) provided by the code in vid2vid (Wang
A visual evaluation on a few frames from the Cityscapes dataset can be seen in Fig.
We compare our method to the results of Pix2PixHD-SIA, where ”SIA” stands for ”Single Image Augmented” e g a Pix2PixHD model that was trained on a single image using random crop-and-ﬂip warps but not TPS.
We can observe that our method is able to synthesize very different scene setups from those seen in training, including different numbers and positions of people.
Our method outperforms signiﬁcantly in terms of ﬁdelity and quality than Pix2PixHD-SIA indicating that our proposed TPS augmentation is critical for single image conditional generation.
Quantitative evaluations on Cityscapes and LRS2 are provided in Tab. 2 and Tab. 3.
We report LPIPS and SIFID for each of the 5 LRS2 sequences and also for the average of 16 Cityscapes videos.
We can observe that our method signiﬁcantly outperforms Pix2PixHD-SIA in all comparisons.
A NALYSIS
Why use super-primitive(SP)?
The choice of different input primitives conveys different information.
Segmentations capture high-level aspects of the image while edge maps capture the low-level of the image better.
Pix2PixHD proposed combining instance and semantic segmentation maps, however, this does not provide low-level details.
We propose a new primitive, ”super primitive” (SP), that combines their strengths.
Fig.
2 compares the three primitives.
The edge representation is unable to capture the eye, presumably as it cannot capture its semantic meaning.
The segmentation is unable to capture the details in the new background regions creating a smearing effect.
SP is able to capture the eye as well as the low-level textures of the background region, showing its strong representational power.
In Fig.
3 we present more manipulation results using SP.
In the center column, we present [Table] [Table] image reorganization results, where the positions of the rightmost cars were switched.
As the objects were not of the same size, some empty image regions were ﬁlled using small changes to the edges.
A more extreme result can be seen in the rightmost column, the car on the left was removed.
This created a large empty image region.
By ﬁlling in the missing details using edges, our method was able to successfully complete the background.
An ablation is presented in App.
E, showing the superiority of the SP representation.
More comparisons are presented in App.
D.
Is the cGAN loss necessary?
We evaluated removing the cGAN loss used by our method (and
Pix2PixHD), and using a VGG perceptual loss (the difference between the activations of a VGG network of output and groundtruth).
The results on Cars are presented in App.
F, we conclude that for such high-res images the cGAN is a better perceptual loss.
We experimented with the VGG loss at lower resolutions, the results are reasonable, however the images are not as sharp as the cGAN loss.
Can methods trained on large-scale datasets generalize to infrequently occurring images?
We present examples where this is not the case.
Fig.
4 showed that BicycleGAN did not generalize as well as
Pix2PixHD-MI for new (in-distribution) shoes.
We show that in the more extreme case, where the image lies further from the source distribution used for training, current methods fails completely.
We manually labelled the semantic and instance segmentation maps of the Cars image, and input it into a Pix2PixHD pre-trained by the authors on the Cityscapes dataset which contains street scenes of cars, roads and building.
The results are presented in Tab. 4.
We can see that a state-of-the-art method pre-trained on a large dataset does not generalize well to out-of-distribution inputs whereas our method, which is trained on a single image can perform well in such cases.
Augmentation for single image manipulation: Although we are the ﬁrst to propose single-image training for manipulation using extensive non-linear augmentations, SinGAN can be seen as implicitly being an augmentation-based approach (for unconditional generation).
In the ﬁrst pyramid level it creates a low-resolution high-level random image generator, by mapping noise to low-res images.
The later stages can be seen as a conditional generator which learns to map low-to high resolution images.
Critically, it relies on a set of “augmented” input low-res images generated by the ﬁrst stage
GAN. So although this is not explicit in SinGAN, we think it also uses augmentations. On the other hand, Deep Image Prior indeed does not use any form of augmentation. [Table]
results.
Runtime : As with other methods that train deep neural networks based on a single image, per-image training times are higher than when trained on large datasets.
Our runtime is a function of the neural architecture we use and the number of iterations.
Here speciﬁcally, we use the Pix2Pix-HD framework which has roughly comparable runtime to SinGAN for the same image size.
The precise runtime depends on image size and the number of iterations used.
When running all experiments on the same hardware (NVIDIA RTX-2080 Ti), smaller images e.g. the “Balloons” image showcased by SinGAN (size: 248X186) take SinGAN 50 minutes while DeepSIM (ours) takes 63 minutes.
Runtime scales with the size of the image, so that large images such as the Cars image (640X320) take SinGAN 195 minutes while ours takes 185 minutes.
Although this is not particularly fast, it is a general characteristic of many deep single image manipulation methods and not a particular issue of DeepSIM.
We are optimistic that runtime optimizations in future work can signiﬁcantly cut the runtime in all such methods, but this is not our focus.
Failure modes: Learning from a single image is very challenging and although our method makes encouraging progress, it has failure cases.
We highlight three main failure modes (Fig.
8): i) generating unseen objects - the network is trained on just a single pair and is not aware of any other objects.
When the manipulation requires generation of unseen objects, the network can do so incorrectly.
ii) background duplication - when adding an object onto new background regions, the network can erroneously copy some background regions that originally surrounded the object, which may appear out-of-context.
iii) interpolation in empty regions - as no guidance is given in empty image regions, the network hallucinates details, sometimes incorrectly.
See App.
C for further analysis.
C ONCLUSION
We proposed a novel method for training conditional generators from a single training image based on thin-plate-spline augmentations.
We demonstrated that our method is able to perform complex image manipulation at high-resolution.
Single image methods have signiﬁcant potential, they preserve image ﬁne-details to a level not typically achieved by previous methods trained on large datasets.
One limitation of single-image methods (including ours) is the requirement for training a separate network for every image, which can be expensive over a large dataset.
Speeding up training of single-image generators is an important and promising direction for future work.
A A DDITIONAL R ESULTS
We present additional results of our method, DeepSIM using the SP on a range of manipulations on different images
Training SP
Training Image
Input SP
Output Image
Changing the Posture of the Squirrel
Training SP Training Image Input SP Output Image
Changing the Shape of the Tail
Removing the Fork
Joining the Hamburger Halves
[Table]
Making the Beak Longer
Changing the Position of the Wings
Removing the Handcuffs
Removing the Right Hand
[Table]
SP
[Table]
Image
Changing the Hat and Making the Face Longer
Making the Body Wider
Moving the Tree
Changing the Shape of the Tree
[Table]
Changing the Shape of the Lake
Adding More Lakes
Removing the Teeth of the Top Lama
Changing the Shape of the Ears of the Top Lama
[Table]
B A S TEP -B Y -S TEP D EMONSTRATION OF E DITING THE P RIMITIVE
Performing complex manipulations by our method is quite easy.
In this ﬁgure we present a step-bystep example of editing a primitive representation using ”Paint”.
It simply requires sampling the required color and painting over the primitive image.
One may also ”borrow” edges from other areas of the image to ﬁll in empty spaces.
[Figure]
C E MPTY SPACE INTERPOLATION
In this section we stress test our method’s ability to handle regions with little guidance.
In this example, the nose of the cat was shifted progressively downwards, forcing the network to interpolate the missing space.
We observe the network synthesizes attractive images for moderate empty regions, however, as the empty region gets larger, the network looks for similar regions to ﬁll the newly created void.
These regions will often be areas which exhibit low amounts of detail in the primitive representation.
In our case we can notice that for larger shifts, the empty space becomes greener until eventually it inpaints a background patch.
We conclude that at a certain point, the network fails to learn the spatial relationship among objects in the image (i.e. that the background can not be placed on the cat’s face) and satisﬁes the given constraint using neighboring information (as was analysed above).
D V IDEO F RAMES
We present a qualitative comparison between different primitives on two frames from the LRS datasets.
Although all primitives generate surprisingly good results, given the training on just a single image, super-primitive generates cleaner outputs with fewer artifacts.
Training Primitive
Training Image
Input Primitive
Output Image
Ground Truth
Training Primitive
Training Image
Input Primitive
Output Image
Ground Truth
[Figure]
E A BLATION OF SP FOR THE C ARS IMAGE
We present an ablation of the superprimitive (SP) representation for the Cars image.
In Fig.
10, we present results for a manipulation on the Cars image using edge-only, segmentation-only and SP.
We can see that SP generates attractive artifact free results.
F A N ABLATION OF THE LOSS OBJECTIVE
We compare the results of our method, DeepSIM, using the original cGAN loss as in the base
Pix2PixHD architecture vs. non-adversarial losses - the simple L 1 loss and the percetual loss based on the difference of VGG activations.
We can see that on this image both non-adversarial losses fail.
Note that at lower resolutions non-adversarial losses do indeed succeed but do not generate results of comparable sharpness of the cGAN loss.
Additionally, we performed the experiment with the cGAN but without the VGG perceptual loss, the results are presented below.
It can be seen that without the
VGG loss, the results are reduced in quality and contain grainy artifacts.
Training
Perceptual Loss
L1 Loss Ours w/o VGG Ours w/ VGG [Figure]
G TPS GENERALIZATION IMPROVEMENT
Let us consider the train and test edge-image pairs presented in Fig.
We input each edge map through an ImageNet-trained ResNet50 network and computed the activations at the end of the third residual block.
For each pixel in the activation grid of the test image, we computed the nearestneighbor (1NN) distance to the most similar activation of the train image.
We then performed 50 TPS augmentations to the training image, and repeated the 1NN computation with the training set now containing the activations of the original training image and its 50 augmentation.
Let us compare the 1NN distances presented in Fig.
11 with and without TPS augmentations.
Naturally, the 1NN distance decreased for the TPS-augmented training set due to its larger size.
More interestingly, we can see that several face regions which prior to the augmentations did not have similar patches in the input, now have much lower distance (while more signiﬁcant changes might not be possible to describe by TPS).
In Fig.
11, we present the results of our method when trained on the training edge-image pair (shown in the leftmost column) and evaluated on the test edge.
We can see that the prediction error ( L 1 difference between ResNet50 activations of the predicted and the true test image) appears to be strongly related to the 1NN distance with TPS-augmentations.
This gives some evidence to the hypothesis that the network recalls input-output pairs seen in training.
It also gives an explanation for the effectiveness of TPS training, namely increasing the range of input-output pairs thus generalizing to novel images.
H TPS E XAMPLES
We present several examples of original and TPS augmented images and primitives.
We can see that
TPS introduces complex deformations to the samples, allowing much more expressive edits than when using simple ”ﬂip and crop” augmentations.
Original
TPS TPS
TPS
Original TPS TPS TPS