D EFECT T RANSFER GAN: D IVERSE DEFECT SYNTHESIS FOR DATA AUGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Large amounts of data are a common requirement for many deep learning approaches.
However, data is not always equally available at large scale for all classes.
For example, on highly optimized production lines, defective samples are hardly acquired while non-defective samples come almost for free.
The defects however often seem to resemble each other, e.g., scratches on different products may only differ in few characteristics.
In this work, we propose to make use of the shared characteristics by transferring a stylized defect-speciﬁc content from one type of background product to another.
Moreover, the stochastic variations of the shared characteristics are captured, which also allows generating novel defects from random noise.
These synthetic defective samples enlarge the dataset and increase the diversity of defects on the target product.
Experiments demonstrate that our model is able to disentangle the defect-speciﬁc content from the background of an image without pixel-level labels.
We present convincing results on images from real industrial production lines.
Also, we show consistent gains of using our method to enlarge training sets in classiﬁcation tasks.
I NTRODUCTION
Automated Visual Inspection (AVI) is vital for quality control in modern production lines.
Despite the fact that AVI has been studied for decades, it remains a challenging task with many open research questions await to be answered.
One of the main challenges in data-driven AVI is the acquisition of suitable training data.
This is for two reasons: First, collecting a vast amount of labelled data is usually labor-intensive and time-consuming.
In many cases, even experts are required to identify where and what to look for.
However, the acquired label information is task-speciﬁc and cannot be reused or transferred to a new task in most cases.
Thus, the tedious labelling process must be repeated for each new product, even if its defect is similar to other products in people’s eyes.
Second, in real-world scenarios such as highly optimized production lines, a more severe problem emerges: data imbalance.
Only very few defective parts are produced by design.
Moreover, the acquired anomaly images from a single product are lacking diversity and may not capture the full defect distribution.
Training a robust deep neural network model in such conditions is very challenging.
Since collecting sufﬁcient real-world defective samples is impractical, algorithms to synthesize required images became a focus in research.
Image synthesis through Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) has shown promising performance in recent years.
But it also requires large amounts of balanced data which are not available in most industrial use cases, in particular for irregular defect patterns and large variation.
Therefore, GANs tend to overﬁt to the training examples when trained with little data (Karras et al., 2020a).
In this work, we tackle these issues by exploiting cross-domain information: we ﬁrst deﬁne two sets of domains —foreground domains and background domains.
The foreground domain describes a set of images that contains a speciﬁc foreground content to be grouped into a distinctive category, and each content has a different style .
The background domain instead is considered as a group of images that shares similar structural appearance over the whole image.
For example, we can set foreground domains as defect types and background domains as product types while the styles of defects indicate their artistic looks such as light or heavy strokes.
Building upon StarGAN v2 (Choi et al., 2020), the concept underlying this work is to transfer and generate foreground contents with a variety of styles across different background domains, as illustrated in Figure 1.
The contributions of this work are three-fold: First, we introduce Defect Transfer GAN (DT-GAN), a model that learns transferring existing foreground content and generating novel contents onto different backgrounds at the same time.
In the real-world scenario, it allows defect inspection networks to learn from a variety of synthetic defective images by composing the foreground defects together with various non-defective images from different products.
Second, DT-GAN is able to disentangle the foreground defect-speciﬁc content and the defect-irrelevant background in a weakly-supervised manner.
Third, extensive experiments show that our method can generate diverse and real-looking defective samples even for products with only 20 real defective images.
These defective images generated by DT-GAN boost the performance in defect inspection networks signiﬁcantly.
R ELATED W ORK
GANs have shown their power in many computer vision tasks such as image synthesis (Luˇci´c et al., 2019), style translation (Johnson et al., 2016), super-resolution (Ledig et al., 2017), image impainting (Pathak et al., 2016) and many other applications.
To quantify the performance of GANs, visual quality and the diversity of generated images are considered as two of the most important criteria.
Recent models address these requirements either by dedicated loss functions (Mao et al., 2019b;
Yang et al., 2019) or architectural design (Brock et al., 2019).
StyleGAN v2 (Karras et al., 2020b), the latest state-of-the-art model in image synthesis, introduces stochastic variation in image generating process by adding per-pixel noise after each convolution.
However, it is non-trivial to adapt the model to transform given input images due to the design of the generator.
In contrast, image-to-image translation methods (Isola et al., 2017) provide a way to recover the connection between inputs and the generated images while encouraging diversity.
For example, Zhu et al.
(2017b)
and
Huang et al.
(2018) impose consistent mappings in latent space to achieve the goal.
Some approaches (Ma et al., 2019; Park et al., 2019) use reference images as guidance to generate diverse outputs.
Mokady et al.
(2020) further extends the translation task from styles to contents.
It learns to identify a speciﬁc content in a given input (e.g., a speciﬁc pair of glasses) and transfer it to the target image.
However, aforementioned methods only consider the translation between two domains and their extension to multiple domains is non-trivial.
Surface defect detection is one of the important tasks in real-world industrial manufacturing.
It aims at identifying and classifying defects with the help of machine vision.
Traditional methods (Ngan et al., 2011) build models upon hand-crafted feature extractors, which are unstable and outperformed by deep learning based models.
However, the performance and generalization ability of deep learning approaches are restricted due to limited number of defective samples in real-world scenarios.
Data augmentation aims to enrich the training dataset by introducing different kinds of invariance for the model to capture.
Several recent works (Niu et al., 2020; Zhang et al., 2021) have proposed to adopt GANs as a data augmentation method to generate realistic defective samples.
Among them, Defect-GAN (Zhang et al., 2021) tries to capture the stochastic variation within defects by mimicking the defacement and restoration processes.
However, it still learns a deterministic mapping between inputs and outputs while DT-GAN achieves multi-modality by varying styles.
Moreover, our method can generate realistic defects with sophisticated patterns copied from real-world defective samples.
[Figure]
M ETHODOLOGY
Our primary aim is to perform unpaired image-to-image translation across multiple foreground domains within a single model.
In our use case, the foreground domains refer to the defect types, which means we want to achieve translations between different types of defects while the background remains unaffected.
We assume that there is always an adequate amount of normal samples (e.g., non-defective) available, while anomaly samples are rare and hard to acquire.
3. P ROPOSED F RAMEWORK
Our framework builds on StarGAN v2, a multimodal image-to-image translation model.
Given an input image x ∈ X and an arbitrary domain y ∈ Y , StarGAN v2 generates a domain speciﬁc style code in a learned style space and outputs an image that is stylized to ﬁt the domain of y .
Its network architecture consists of four modules: a generator, a mapping network, a style encoder and a discriminator.
We modify and extend all four modules (see Figure 2) and describe the key differences in details as below.
Style-Content Separation.
Given a latent code z and a domain y , the mapping network M (Figure 2(b)) generates a style code s = M y
( z ) and a domain speciﬁc content c = M y ( z ) in different branches.
It is worth mentioning that M y here denotes an output of M corresponding to the domain y .
This feature allows our method to separately model the structural appearance (i.e. content) and its artistic looks (i.e. style), which is essential because applying different styles to the same content enriches the diversity of outputs.
By randomly sampling z from a standard normal distribution and y from all available foreground domains, M is able to produce diverse style codes and domain speciﬁc contents.
The encoder E (Figure 2(c)) extracts the style code s =
E y ( x ) and the domain speciﬁc content c = E y ( x ) from an given image x , which reﬂect the characteristics of reference images instead of randomly sampled noise.
Foreground/Background ( FG / BG ) Disentanglement.
The generator G (Figure 2(a)) translates an input image x into an output image G ( x , e s , e c ) according to given domain speciﬁc style code e s and content e c , which are provided either by the mapping network M when generating from random noise or by the style-content encoder E when transferring an existing content from a reference image. To achieve a FG / BG disentanglement, we split the channels of the three-dimensional feature map (i.e., H × W × C ) at the bottle neck of G into two parts. The model is then forced to encode the background into the ﬁrst channels and the domain speciﬁc content ˆ c into the latter channels by classiﬁcation losses as discussed in Section 3.2. ˆ c can then be replaced with content e c from the target domain.
The adaptive instance normalization (AdaIN) (Huang & Belongie, 2017) is then used to inject e s into e c during the decoding process while the background BG G ( x ) is decoded separately. StarGAN v2 learns FG and BG together which leads to a conditional relationship between both. Our disentanglement and separate encoding break this conditioning and therefore enable our method to freely combine FG and BG as well as learn the full variation of FG content. Finally, BG G ( x ) and e c are concatenated together and then fused before output.
Multi-task discriminator with auxiliary classiﬁers. The discriminator D (Figure 2(d)) is a multitask discriminator with two auxiliary classiﬁers: a foreground domain classiﬁer and a background domain classiﬁer.
This feature strengthens the disentanglement of FG and BG by ﬁrst ensuring the input image x contains a domain speciﬁc content that can be recognized by the foreground domain classiﬁer independent of the background.
Later, each branch D y in the multi-task discriminator D is trained to determine if an image x is a real image of its foreground domain or a fake image G ( x , s , c ) generated by G .
Apart from that, one extra branch BG cls is attached to decide whether the background information of the input images is well preserved.
Content Transfer.
Mokady et al.
(2020) introduced a concept that a model should be able to identity the difference between two domains when one of the domains contains a feature that the other does not have.
We refer to this concept as ‘anchor’ and extend to multiple domains ( > 2) by the FG / BG disentanglement, the multi-task discriminator and the foreground content classiﬁer in D .
We treat domain Normal as the anchor domain i.e. set the domain speciﬁc content to zero, because a normal image has no domain speciﬁc content in our deﬁnition.
As a result, we can now transfer contents between all combination of FG and BG domains (see Figure 10).
Compared to StarGAN v2, our method not only models style codes and contents separately but also disentangles the foreground and background of an image in a weakly-supervised manner.
These features allow explicit control over output images by combining desired style codes and contents from one of the subnetworks with the input images.
Therefore, it leads to higher variance regarding the location, structural pattern and artistic style of defects in the synthetic images of DT-GAN.
3. T RAINING O BJECTIVES
Given an image x ∈ X , its original foreground domain y ∈ Y and its background domain p ∈ P , the following objectives are used to train our framework.
Adversarial loss.
In the training phase, a noise vector z ∈ Z and a target foreground domain e y ∈ Y are sampled randomly.
Both of them are fed to M , producing a target style code e s and a target content e c as follows:
e s , e c = M e y ( z ) .
Goal of the training is to ensure that e s and e c are sampled from the distribution over styles and contents of the target domain e y .
The generator G then combines an image x with e s and e c and learns to generate an output image G ( x , e s , e c ) that is indistinguishable from real images in the target domain e y .
We encourage this behavior by using an adversarial loss same as in Choi et al.
[Equation]
where D y and D e y are the output branches of D that correspond to the source domain y and the target domain e y , respectively.
Style-content reconstruction loss.
Similar to StarGAN v2, to enforce the generator G takes the style code e s and the domain speciﬁc content e c into consideration during the generation process, we employ a style-content reconstruction loss
[Equation]
This objective urges the style-content encoder E to recover e s and e c from G ( x , e s , e c ) .
Here, the stylecontent encoder E learns a mapping from an image to its style and content domains, which allows
G to synthesize an image with given s and c from reference images at test time.
Diversity loss.
In order to further boost the diversity of output images from G , we introduce a loss that encourages diversity as follows: for a pair of random latent codes z 1 and z {+
+} we compute
e s i , e c i = M e y
( z i ) for i ∈ { 1 , 2 } and enforce a different outcome of the generator G for differently mixed style and content input pairs:
[Equation]
where m, n ∈ { 1 , 2 | m 6 = n } and o ∈ { 1 , 2 } .
Driven by this term, the generator G is forced to discover meaningful style features and contents that eventually lead to diversity in generated images.
We ignore the denominator k z 1 − z 2 k 1 of the original diversity loss (Mao et al., 2019a) for stable training as in StarGAN v2.
Cycle consistency loss.
To ensure that the generated image G ( x , e s , e c ) preserves the domaininvariant properties of its input image x , we impose the cycle consistency loss (Zhu et al., 2017a)
[Equation]
where ˆ s , ˆ c = E y ( x ) is the extracted style code and domain speciﬁc content of the input image x , and y is the original domain of x .
By learning to reconstruct the input image x with given style code ˆ s and content ˆ c , the generator G is then further encouraged to disentangle the background, the domain speciﬁc content and the style code.
Content consistency loss.
Besides the cycle consistency loss, we apply another constraint to enforce that the detached domain speciﬁc content from G is consistent with the one retrieved from E according to
[Equation]
where ˆ c = E y ( x ) , e c = E e y ( x ) , FG G ( x ) and FG G ( G ( x , e s , e c )) are the pop-out domain speciﬁc content from input image x and generated image G ( x , e s , e c ) , respectively.
Classiﬁcation losses.
We employ two classiﬁcation losses: the ﬁrst one is the foreground content classiﬁcation loss
[Equation]
which aims to ensure that the domain speciﬁc content is properly encoded and carries enough information from the target domain
The second one is the background classiﬁcation loss
[Equation]
where p is the corresponding background type of x real and x fake .
With the help of this objective, the generator G learns to preserve the domain-invariant characteristics of its input image x while dissociating the foreground domain speciﬁc part.
Full objective.
Our full objective functions can be summarized as
[Equation]
where λ sty , λ ds , λ cyc , λ con cyc , λ FG cls and λ BG cls are the hyperparameters for each term.
E XPERIMENTS
We evaluated the images generated by DT-GAN through a series of experiments both quantitatively and qualitatively.
Finally, we demonstrate the beneﬁts of our generated images when being used as data augmentation for a defect classiﬁcation task on limited data.
Dataset.
All experiments were performed on a real industrial dataset: a Surface Defect Inspection (SDI) dataset that contains three different kinds of products from production lines and samples from each product are classiﬁed into three mutually exclusive classes: Normal , Scratch and Spot .
All of the images are grayscale.
Detailed statistics of the dataset are summarized in Appendix
A. Note that only the training set was used in GAN training, the test set was left untouched for ﬁnal evaluation in classiﬁer training.
For a fair comparison, all images were resized to 128 ×resolution for both GAN training and classiﬁer training, which was also the highest resolution used in the baselines for image generation.
For comparison, we also conducted experiments on the widely used MVTec Anomaly Detection dataset (Bergmann et al., 2019) in Appendix E.4.
4. D EFECT G ENERATION
Baselines.
As discussed in Section 3, DT-GAN can either use the mapping network to randomly generate styles and defects, or it can use the style-content encoder to extract both from reference images.
We refer to these cases as ‘latent-guided’ and ‘reference-guided’, respectively.
Since the two ways of guidance are fundamentally different, we evaluated them against two sets of baselines:
Our reference-guided image generation was compared to Mokady et al.
(2020) and
StarGAN v2, because both of them can perform a reference-guided translation.
Note that Mokady et al.
(2020) can only translate between two domains while StarGAN v2 and DT-GAN can achieve multi-domain translation within a single model.
Images generated through the latent-guided part of DT-GAN were compared to state-of-the-art GANs in image synthesis: BigGAN (Brock et al., 2019) and StyleGAN v2 (Karras et al., 2020b).
We set BigGAN to condition on defect types during training while StyleGAN v2 was trained unconditionally.
All baselines were trained from scratch with the public implementations provided by the authors 1 .
4.1. Q UANTITATIVE E VALUATION
Metrics.
We employed the commonly used frechet inception distance (FID) (Heusel et al., 2017) to evaluate both the visual quality and the diversity of the generated images.
We also report the kernel inception distance (KID) (Binkowski et al., 2018) which is a more stable metric for small sets of images like our SDI dataset.
Lower FID and KID scores indicate better performance.
Both scores are shown in Table 1.
We observe that methods like BigGAN and StyleGAN v2, which perform defect synthesis purely based on latent codes, generally provide unsatisfactory results on the SDI dataset, presumably due to the small number of defective samples that were available.
These methods then struggle to capture the complex and irregular patterns of defects.
We also experimented with augmentation methods for GAN training (Karras et al., 2020a; Zhao et al., 2020) but did not ﬁnd a consistent improvement (see Appendix E.2).
We thus only report the best scores.
Reference-guided synthesis methods like Mokady et al.
and StarGAN v2 seem to generate more realistic images.
The scores of StarGAN v2 on a single product are omitted here because generating images with speciﬁed background is not possible due to its network design—the product type changes in output images, which we refer to as ‘identity-shift’.
As seen in Table 1, our method achieves better scores in all cases.
We believe this is due to the fact that our method allows free combination of foreground defects and backgrounds, making the generated images more diverse even with a small number of training samples. [Table]
[Table]
Q UALITATIVE E VALUATION
We present a qualitative comparison with the baseline methods in latent-guided image synthesis in Figure 3.
To make a fair comparison, we trained StyleGAN v2 and BigGAN on each product separately to have control on background products.
Note however, that images from DT-GAN were always obtained from a single model.
We can see that some generated samples from StyleGAN v2 do not contain clear defects [Table] and samples from BigGAN present abnormal grid patterns.
Both methods do not take images as inputs but generate synthetic images according to a given latent code which contains information for both FG and BG .
This conditioning leads to limited diversity in the output images.
On the other hand, StarGAN v2 performs translation based on input images but suffers from the same entanglement issue. Thus, it fails to preserve the background, which results in artifacts or identity-shift in its outputs.
Our network architecture that disentangles foreground and background seems to mitigate these issues.
See Appendix E.4 for more images.
[Figure] [Figure]
illustrated in Figure method
Ablation study.
We visually demonstrate the effect of each component we added to DT-GAN compared to StarGAN v2 in Figure 5, using the examples of both latent- and reference-guided image synthesis from Normal to Scratches . The quantitative evaluation can be found in Appendix E.3.
Column (a) corresponds to StarGAN v2 and highlights the drawback of entangled FG / BG again (i.e. the identity-shift in the background).
We ﬁrst tackle this problem by modeling the style code and foreground content explicitly and feeding them separately to the generator.
This leads to a better preservation of the background structure in column (b) for the reference-guided subnetwork, but not for the latent-guided synthesis on the bottom of Figure 5.
Thus, we add a foreground classiﬁer in the discriminator in (c) to ensure the output image contains the desired foreground content (scratch).
Similarly, we introduce a background classiﬁer to the discriminator in column (d).
Note that the additional product type labels can be acquired automatically from production lines.
For column (e), we add the separate decoders for foreground and background in the generator which are fused only in the end.
This enhances the preservation of background characteristics like lighting even more.
Imposing an additional penalty for foreground content extracted from a normal sample as described in Section 3.1 leads to another visual improvement of the foreground edges for reference-guided synthesis in column (f).
Finally, inspired by StyleGAN, we incorporate adaptive noise injection to the mapping network, which signiﬁcantly boosts the performance of our latentguided image synthesis as shown in column (g).
Styling.
We visually demonstrate the effect of style codes in our method by randomly sampling those and combining them with ﬁxed reference background and foreground images in Figure 6, where a variety of artistic styles can be seen on the output columns.
[Figure]
[Figure]
We also evaluated our method as a data augmentation method for defect classiﬁcation on the SDI dataset.
We deﬁned one task ‘general’, where the classiﬁer was trained on images from all products at once, while task ‘single product’ only used the subset of images for one product.
Besides, we incrementally varied the amount of real Normal data available for classiﬁer training: 4500, 6600, 12000 and 18600.
In the case of defective images, all of them were always used due to the small amount unless otherwise speciﬁed.
As backbone we used a ResNet-50 (He et al., 2016a) with ImageNet pretrained weights.
For experiments with synthetic data, we attached an auxiliary domain classiﬁer to the network through a Gradient Reversal Layer (Ganin & Lempitsky, 2015).
[Table]
Since the SDI dataset is highly imbalanced, we oversampled the minority classes (Ling et al., 1998) unless the data was balanced through synthetic images.
Additionally, we always applied traditional data augmentation techniques like random horizontal ﬂips, jittering and lighting (Shorten & Khoshgoftaar, 2019) except where noted.
All following results were evaluated by the achieved error rates over ﬁve runs with different random seeds.
Effectiveness of synthetic data.
We ﬁrst compare classiﬁer performance for no augmentation ( NoAug ), traditional data augmentation ( Trad-Aug ), and a combination of traditional augmentation with synthetic images for GAN methods including DT-GAN.
We also introduce a stronger backbone, EfﬁcientNet-b4 (Tan & Le, 2019), to demonstrate that our results are not conﬁned to a speciﬁc network.
Table 2 shows that our method is the only one that improves performance for both backbones, presumably due to the combination of high visual image quality and diversity in our samples.
[Table]
Impact of dataset size.
Motivated by the limited availability of data in real-world production scenarios, we therefore evaluated DT-GAN for data augmentation on a subset of the full SDI dataset ( All ), which only contains 20 defective samples in product A for each defect type ( 20A ).
In this case,
DT-GAN was also trained on the reduced subset.
As shown in Table 3, there is a clear improvement when synthetic images from DT-GAN are used as data augmentation, even for the extremely limited data subset.
Further results on single product classiﬁers can be found in Appendix E.1.
[Table]
Cross-domain effect.
We hypothesized that limited data can be counteracted by transferring defects across multiple background products, if there are at least some defects that occur on multiple products (See Appendix E.1 for further discussion).
We tested this approach by comparing the performance of classiﬁers trained on synthetic images with defects from a speciﬁc source ( vA , vB , vC ) to classiﬁers trained on images with defects from all products ( vABC ).
As we can see in Table 4, the best performances are reached by the models that take over defects from other products.
We interpret this as support for our hypothesis and its practical usefulness.
C ONCLUSION
We propose a novel method, DT-GAN, which allows diverse defect synthesis both by generating from randomly sampled noise and by following the guidance of given reference images.
Due to explicit style-content separation and FG / BG disentanglement, DT-GAN achieves higher image ﬁdelity, better variance in defects and full control over background and foreground while being sample-efﬁcient.
We demonstrated the feasibility and beneﬁts of DT-GAN on a real industrial defect classiﬁcation task and the results show our method provides consistent gains even with limited data and boosts the performance of classiﬁers compared to state-of-the-art image synthesis methods.
For future investigation, we aim to represent defects more explicitly (e.g., localization) to improve the explainability of the model and also enhance the model transferability to unseen products.
R EPRODUCIBILITY S TATEMENT
We aim for full reproducibility by publishing the source code and dataset with the ﬁnal version of the paper.
Besides, we provide descriptions of the training details in Appendix B, the evaluation setup in Appendix C and the network architecture in Appendix D
A T HE S URFACE D EFECT I NSPECTION D ATASET
The Surface Defect Inspection (SDI) dataset consists of 20,414 images at 128 × 128 resolution.
It contains three background domains—product A , product B and product C , each can be further classiﬁed into three foreground domains— Normal , Scratches and Spots .
Figure 7 shows example images of the SDI dataset.
To be noticed that the dataset is highly imbalanced not only between normal and defective samples but also between different products as shown in Table 5.
This sets a more challenging task when training deep neural networks like GANs and downstream classiﬁers.
For each foreground and background domains, we randomly select 50 images for a joint validation/test set, which is then further split into separate sets in the ratio of 3:7, and use all remaining images as training sets for GAN and classiﬁer training.
We present the distribution of the training set when training DT-GAN in Table 6.
Note that the normal samples used in GAN training are only a subset of all available samples in Normal and we keep the rest of them for generating defective samples at test time.
For classiﬁer training, we show the statistics in Table 7, where the number of normal samples involved in classiﬁer training increase incrementally.
The validation set is used to select the best model classiﬁer training while the test set is left untouched until the ﬁnal evaluation.
Both of the validation and test set are inaccessible by DT-GAN.
[Table] [Table]
[Table]
B T RAINING D ETAILS
DT-GAN.
We follow the training scheme as described in StarGAN v2 with minor modiﬁcations.
To ﬁt the model on a single Nvidia GTX TITAN X, the batch size is reduced to four while the model is still trained for 100,000 iterations.
The training time is about three and a half days on the dedicated
GPU with the modiﬁed network architecture 2 and loss functions mentioned in Section 3 in PyTorch (Paszke et al., 2017).
We set λ sty = 1, λ ds = 1, λ cyc = 1, λ con cyc = 1, λ cls = 1 and λ BG cls = 1 for the
SDI dataset.
All other design choices remain the same as in StarGAN v2.
Classiﬁers.
We train all the classiﬁers that use ResNet-50 as backbone for 100 epochs with the
SGD optimizer (Ruder, 2016) and batch size 256.
The initial learning rate is 0.001, momentum is 0.9 and weight decay is 1e-4.
A learning rate scheduler is set to reduce the learning rate by factor of 0.1 when the validation loss stops decreasing for 5 epochs.
The same setting also applies to
EfﬁcientNet-b4, except the batch size is reduced to 128.
Although DT-GAN can synthesize realistic defective samples, we notice that there still exists a domain gap between the generated samples and the real samples.
To explore the full potential of the generated samples, we attach an auxiliary
[Figure]
source classiﬁer to distinguish between synthetic and real samples.
Then, this classiﬁer is connected to the backbone (e.g. ResNet-50) through a Gradient Reversal Layer.
With the help of the Gradient
Reversal Layer, the backbone is forced to extract the shared features between synthetic and real samples, which ensures all training samples are effectively learned.
We design a two-layer perceptron that connects to the average pooling layer in ResNet-50 as shown in Figure 8.
Note that the usual fully connected layer after the average pooling in ResNet-50 remains the same and is not affected by the extra branch we added.
Inspired by Chen et al.
(2018), a threelayer perceptron is used for EfﬁcientNet-b4 instead as shown in Figure 9.
Its layers are initialized with a random normal distribution, where the standard deviation is set to 0.01 for the ﬁrst two layers and 0.05 for the output layer.
The biases for all layers are set to 0.
C E VALUATION SETUP
Generated samples from DT-GAN.
DT-GAN requires images as input for generating synthetic data.
At test time, we translated each Normal image in the SDI dataset into four defective images: two with Scratches and two with Spots .
The translations were performed by two subnetworks: by the mapping network M using random noise (‘latent-guided’) and by the style-content encoder
E using a reference image (‘reference-guided’).
We ﬁrst randomly sampled one latent code for each defective foreground domain.
Similarly, we also randomly sampled one reference image from the training set for each defective foreground domain.
The corresponding style codes and defect contents were then produced by the two subnetworks respectively and fed to the generator for target image generation.
We conducted classiﬁcation experiments separately on images generated from the two subnetworks and a mixture set of both (i.e. 50% from each subnetwork).
Experiments show consistent gains of using synthetic images generated from DT-GAN (Table 8).
We observe that the latent-guided synthetic images in general perform better than the reference-guided one, while the mixture set provides more stable results with regard to the standard deviation.
Presumably the mixture set beneﬁts from the combination of samples from reference-guided synthesis, which are well aligned with the original defect distribution, and the samples from latent-guided synthesis, i.e. from random noise, which adds novel but plausible defects to the dataset.
In the main text, we report the results of the mixture set for all experiments, including the quantitative evaluation of DT-GAN.
[Table]
Frech´et inception distance (FID) and Kernel inception distance (KID).
We used the feature vectors from the last average pooling layer of the ImageNet pretrained Inception-V3 to calculate both scores.
For each test image from the Normal domain, we translated it into a synthetic defective image of each defect domain.
The style codes and contents for the translation were acquired in two ways: by randomly sampling from the standard normal distribution and by randomly sampling a reference image from the train set of a defect domain.
To calculate the FID and KID score, we generated 4000 defective samples per product per defect domain for each way of guidance, and formed the mixture set by randomly sampling 2000 images per product per defect domain from each way.
The reported FID and KID scores were then computed between the defective images in the training set and the mixture set of synthetic defective images.
The same procedure was applied when computing scores on single product subsets of the SDI dataset.
For example, for product A, we calculated the scores between the defective image of product A in the training set and the mixture set of synthetic defective images of product A.
D N ETWORK A RCHITECTURE
In this section, we provide the architectural details of all four modules in DT-GAN.
[Table] [Table]
Generator (Table 9) .
For the SDI dataset, the encoder part of the generator consists of three downsampling blocks and two intermediate blocks (Table 9 (a)), all of them are pre-activation residual units (He et al., 2016b).
Then the encoded feature map is split channel-wise into background (Table 9 (b)) and foreground (Table 9 (c)).
Both of them are then carried through separate decoders.
We use the instance normalization (IN) and the adaptive instance normalization (AdaIN) as indicated.
The style code is injected into all AdaIN layers to modulate the afﬁne transformations.
Note that
[Table]
AdaIN is only used in the foreground decoder.
The outputs of both decoders are only fused in the end (Table 9 (d))
[Table]
Mapping Network (Table 10) .
The mapping network consists of four shared linear layers (Table(a)) and two separate branches: one for generating style codes (Table 10(b)) and one for contents (Table 10(c)).
Each of them is further divided into K output branches, where K denotes the number of domains.
The dimension of the input, the output style code and the output content is set to 16, 64, and 16 × 16 × 64, respectively.
The latent code is sampled from the standard normal distribution.
Note that we apply per-pixel noise after each convolution in the content branch, which we have observed to increase the diversity of generated defects signiﬁcantly (cf.
Figure 5 (g)).
Style-Content Encoder (Table 11) .
The style-content encoder consists of a CNN (Table 11 (a)) with two branches (Table 11 (b) and (c)) as in the mapping network.
Each branch has K outputs, where
K is the number of domains.
Three pre-activation residual blocks are shared among two branches, followed by a speciﬁc structure for each branch.
The output dimension D in Table 11 is set to 64, which denotes the dimension of the style code.
Discriminator (Table 11) .
The discriminator is a multi-task discriminator with two auxiliary classiﬁers for the foreground content and the background.
The structure is almost identical to the stylecontent encoder, except D is set to 1 for real/fake classiﬁcation.
The background classiﬁer acts in parallel to ﬁnal linear layer in Table 11 (b) and provides the logits for background classiﬁcation.
The foreground classiﬁer instead acts on top of the output in Table 11 (c) and four more pre-activation residual layers are applied to encode the content into logits for foreground content classiﬁcation.
E A DDITIONAL R ESULTS E. A DDITIONAL R ESULTS ON THE SDI DATASET
We provide additional reference-guided image synthesis results on the SDI dataset in Figure 10.
We demonstrate all the possible transfers among all foreground domains.
Both style codes and contents are extracted from the reference images.
To be noted that DT-GAN can append and remove foreground defects not only onto Normal samples but also to defective samples.
For example, in the ﬁfth column of Figure 10, the original scratch in the source image is removed and only the defects from the reference images are presented in the output images.
Besides, we present additional evaluations showing the effectiveness of our synthetic data according to Table 3.
As seen in Table 12, the synthetic images from DT-GAN also boost the performance in single product classiﬁers, where the classiﬁers were trained on the subset of images for one product ( A , B , C ) instead of the full dataset ( ABC ).
As discussed in Section 4.2, we assumed that the data-insufﬁciency problem can be mitigated by transferring defects across multiple background products.
To examine if this assumption holds, we compared the performance of classiﬁers trained on synthetic images with defects from a speciﬁc source ( vA , vB , vC ) to classiﬁers trained on images with defects from all products ( vABC ).
The results on the cross-domain effect with regard to different sizes of the training set are shown in
Table 13.
We again notice that using our synthetic data is beneﬁcial.
Moreover, in most cases the performance is further improved by exploiting cross-domain information (i.e. by transferring defects from other products).
We interpret this as support for our assumption and the practical usefulness of our method in the real-world scenario.
The case of cross-domain image synthesis when the desired combination is not presented in the training set is covered in the study on the MVTec Anomaly
Detection dataset (Bergmann et al., 2019) (see Appendix E.4). [Table]
[Table]
E. A DDITIONAL FID AND KID RESULTS ON THE SDI DATASET
We provide additional results in the case of training GANs with augmentation methods in Table
[Table]
Augmentation methods like ADA (Karras et al., 2020a) or DiffAug (Zhao et al., 2020) are proposed to adapt GAN training to limited data.
We applied these augmentation methods to StyleGAN v2 and BigGAN, because these state-of-art image synthesis methods are not optimized for small dataset.
However, incorporating the augmentation methods in training GANs on the SDI dataset is not always beneﬁcial.
The performance of StyleGAN v2 is largely degraded when using ADA, potentially due to the conﬂict between augmentation methods and the decentralized location of defects—in the SDI dataset, defects can occur anywhere on the surface.
This is in contrast to datasets that were used to evaluate the aforementioned augmentation methods in GANs, where the objects are centralized (e
[Table]
g., ImageNet (Deng et al., 2009), Cifar (Krizhevsky & Hinton, 2009)) and their attributes (e [Table] g., beard, eye glasses in CelebA (Liu et al., 2015)) only occur in speciﬁc images parts.
[Table]
A BLATION STUDY WITH REGARD TO FID AND KID SCORES
We report the FID and KID scores of the ablation study in Table 15.
We notice that both subnetworks show positive correlation to each modiﬁcation except for structural change as in (a) and (e) .
Among the two subnetworks, the reference-guided subnetwork outperforms the latent-guided one in the beginning, which is due to the fact that transferring existing contents is easier than generating them from random noise.
This effect is also observed in Figure 5.
However, the performance of the latentguided subnetwork improves signiﬁcantly after applying per-pixel noise injection.
The subnetwork can now output non-deterministic foreground contents even for a ﬁxed input vector which results in better visual quality and higher diversity of generated defects.
In the main text, the scores of the mixture set are reported [Table] [Table] E.
[Table]
A DDITIONAL RESULTS ON THE MVT EC A NOMALY D ETECTION DATASET
The MVTec Anomaly Detection dataset (Bergmann et al., 2019) contains 15 different object and texture categories for anomaly detection. The dataset is formed of non-defective image for training and both non-defective and defective images with various kinds of defects for testing. The pixel-level annotations of all defective images are also provided. It is worth noting that the MVTec Anomaly
Detection dataset is relatively small scale in number of images, where the number of training images is ranging from 60 to 391.
Moreover, the number of defective images for each defect category in the test set is varying only from 8 to 30, which is relatively limited considering the sophisticated pattern of defects.
We conducted image synthesis experiments on a subset of MVTec Anomaly Detection dataset, where we selected four texture categories: Carpet , Leather , Wood and Tile for our targeted scenario i.e. surface defects.
Furthermore, we aggregated some of the original defect types deﬁned in the MVTec Anomaly Detection dataset into scratches and spots according to their visual appearance.
We then simply added the subset of the MVTec Anomaly Detection dataset to the training set together with the SDI dataset for training DT-GAN.
Details of the resulting dataset are shown in
Table 17.
Note that the small scale of available data posts a major challenge for training generative models.
Quantitative Evaluation.
We present additional quantitative results on the subset of the MVTec
Anomaly Detection dataset in Table 16, following the same evaluation setup as described in Appendix C. As shown in Table 16, our method achieves the best scores in Carpet and Wood , which supports our claim that DT-GAN generates synthetic images with higher ﬁdelity and more diverse defect.
However, we also observe that StyleGAN v2 seems to outperform our method in Leather and Tile .
Please note that FID and KID are not optimized to evaluate such a small dataset, there the results should only be interpreted together with the qualitative results.
Note the we again omit the FID and KID of StarGAN v2 because it is not cable of generating images for a speciﬁed product due to the ‘identity-shift’, which is also explained in detail in the qualitative evaluation.
[Table]
Qualitative Evaluation.
For qualitative results, we again discuss the ‘latent-guided’ and ‘referenceguided’ synthesis separately.
We present the ‘latent-guided’ image synthesis results of StyleGAN v2 in Figure 11 and Figureand BigGAN in Figure 13 and Figure 14.
The results are acquired by training one model for each product and then generating 16 images from randomly sampled latent codes from each of them.
As pointed out in Section 4.1.2, both methods can not adapt well on small dataset.
They suffer from model collapsing and show signs of overﬁtting by generating images similar to the training data.
For example, StyleGAN v2 generates images either with no clear defect or identical to the training set (e.g., Leather in Figure 11 and Product B in Figure 12).
The overﬁtting we observe here also explains the better FID and KID scores in Table 16.
For Tile , we can see clear signs of mode collapse in the generated Tile images of StyleGAN v2.
Similarly, BigGAN produces images with single mode and abnormal patterns (e.g., grid structure and gray edges).
Unlike StyleGAN v and BigGAN, StarGAN v2 and our method both require images as input (i.e. Source ).
Therefore, we randomly sampled two Normal images and applied eight defects which are generated from randomly sampled latent codes to each of them.
As seen in Figure 15 and Figure 16, StarGAN v fails to preserve the background from the given input images due to the highly entangled FG and BG .
Also it fails to generates legit and diverse defects without separately modeling the style and the content.
In contrast to aforementioned methods, our DT-GAN produces images with higher ﬁdelity and more diversity in defect patterns as shown in Figure 17 and Figure 18.
We believe this again prove the importance of style-content separation and FG / BG disentanglement, which we introduce in Section 3.1.
For ‘reference-guided’ image synthesis, the results of Mokady et al.
(2020) are shown in Figureand Figure 20 while the results of StarGAN v2 are in Figure 21 and Figure 22.
We can observe a clear shift in color in all the outputs
Moreover, Mokady
) can only transfer content between two domains.
In order to perform translation from a non-defective sample to a defective one, we trained a model for each type of defect and for each product.
This sums up to be 13 models ( Scratches and Spots for 6 categories and Scratches only for Tile ).
The results from the intended use within one background domain can be found on the diagonal and are marked in red in both Figure 19 and Figure 20.
We still show the images that we feed in images from other background domains.
As expected, the model then fails to preserve the background of given source images and introduce artifacts to the outputs.
Similarly, StarGAN v2 does not preserve the background from the input images.
Without style-content separation and FG / BG disentanglement, we observe that StarGAN v2 encodes the background characteristics together with the foreground content of the reference images, which results in identity-shits in its output images.
Moreover, the output images either show no clear defect or contain abnormal patterns which sabotage the ﬁdelity.
On the contrary, our method can faithfully transfer the foreground content of reference images across given background of different products as shown in Figure 23 and Figure 24, which demonstrate the effectiveness of the style-content separation and FG / BG disentanglement we introduced in
Section 3.1.
It is also worth noting that our method can perform cross-domain image synthesis even the desired combination is not presented in the training set.
We demonstrate this on product Tile , which only has images with Scratches but no Spots .
As shown in Figure 18 and Figure 24, DT-GAN can generated spots one given Tile images.
However, this kind of transformation is most useful when the desired combination is reasonable for the downstream applications.
Limitation and Future Work.
We have demonstrated the feasibility of the proposed DT-GAN by incorporating more products from the MVTec Anomaly Detection dataset in our training procedure.
Intensive experiments have shown that the generated images from DT-GAN yielded better results compared to the baseline image synthesis methods.
However, we noticed that despite the diverse patterns of the generated defects, DT-GAN tends to apply the styles learned from the SDI dataset also to the samples from the MVTec Anomaly Detection dataset.
For example, we can observe some ”halo” effects in Leather and Wood in Figure 18 and some of the generated scratches in Figureand Figure 23 are rather weakly pronounced.
We hypothesize this can be counteracted by explicitly localizing the defect and enforcing the model to learn conditional relationships between ‘styles’ and different backgrounds.
We aim to address these issues in future work.
[Table]