Paper: Few-shot Hybrid Domain Adaptation of Image Generator (ICLR-2024)

Reviewer comment (Experiments):
One notable concern is the lack of diverse domain results in the paper. The primary focus of the paper revolves around attributes like age, smile, and sketch, which are maintained throughout the paper. This limited scope may not provide sufficient evidence to establish the method's overall validity.

Author response:
**The results of more domains**

Consistent with prior work for few-shot generative domain adaptation (like CDC[1], RSSA[2], and DoRM[3]), our experimental data encompasses both global style ($\textit{sketch}$ and $\textit{baby}$) and local attributes ($\textit{smile}$ and $\textit{sunglasses}$). The combination of these domains demonstrates the effectiveness of our method since they encompass all types of generative domain adaptation.

To provide more comprehensive evidence of validity, we conduct additional experiments on $\textit{Raphael}$ and $\textit{Caricature}$ for both single and hybrid domain adaptation. As shown in Fig. 15 of Appendix, the results integrate the characteristics from multiple target domains and maintain robust consistency with source domain, which further demonstrates the effectiveness of our method.

Reviewer comment (Experiments):
The paper overlooks discussions and validation concerning the advantages and disadvantages of employing few-shot domain adaptation techniques with discrimination other than StyleGAN-NADA, as seen in methods such as DualStyleGAN and 3DAvatarGAN.

Author response:
**The advantages and disadvantages of few-shot domain adaptation techniques with discrimination**

Few-shot domain adaptation techniques with discrimination (like DoRM, DualStyleGAN and 3DAvatarGAN) excel in generating images closely resembling the style of the target domain as its discriminator tends to memorize training images. However, they suffer from the notorious issue of model collapse. Especially in few-shot scenarios, they easily overfits to the target images, compromising the diversity of generated images. Additionally, they are challenging to extend into an end-to-end hybrid domain adaptation approach. Their approach to hybrid domain often involves separately training multiple models and interpolating style codes, necessitating multiple model size and training time.

Reviewer comment (Novelty):
Attributes like smile and age can be manipulated using existing StyleGAN editing methods. It's challenging to discern why hybrid domain adaptation is necessary in these cases.

Author response:
**Why is hybrid domain adaptation necessary in these cases**

Image editing indeed enables local adjustments resembling the original domain, like $\textit{smile}$ and $\textit{baby}$. However, in this scenario, our method holds several advantages. 

Firstly, our objective is to generate images with attributes of target domain while maintaining considerable diversity, which can be applicable in scenarios like data collection. Image editing, on the other hand, requires an original image as input, rendering it impractical for such applications. 

Moreover, hybrid domain adaptation has the capability to generate images from hybrid domain containing multiple target attributes like $\textit{baby with the style of sketch}$ or $\textit{smiling person with the style of sketch}$. Image editing, however, lacks the ability to perform such global stylistic modifications, limiting its broader applications.

Reviewer comment (Experiments):
To ensure that the original generator's attributes are preserved, a useful approach could involve comparing the editability before and after the domain adaptation task. For instance, evaluating the ability to edit attributes like pose using a 2D generator. Such evaluation is lacking in the paper. This also raises the question of how the method would perform in a 3D GAN setting.

Author response:
**The editability before and after the domain adaptation**

We conduct the editing on the images generated by original and adapted model for both single and hybrid domain adaptation. As shown in Fig. 17 of Appendix, the results indicate the adapted generator maintains similar editability like pose to the original generator. This verifies that the our method effectively preserves original generator's attributes. 

**Results in 3D GAN setting**

We conducted experiments using the popular 3D-aware image generation method, EG3D[4]. Specifically, we replace the discriminator as we did in Fig. 2 for both single and hybrid domain adaptation. As shown in Fig. 18 of Appendix, we adapt the pre-trained generator from FFHQ to $\textit{sunglasses}$ and the hybrid of $\textit{sunglasses}$ and $\textit{smile}$ with 10-shot training images per domain. We can observe that the results effectively integrate the attributes and preserve the characters and poses of source domain.

Reviewer comment (Novelty):
It is worth noting that there are other relevant papers in the field, such as "Mind the GAP," which employ similar loss mechanisms to constrain domains. Additionally, "StyleCLIP" introduces directional loss terms based on CLIP.

Author response:
**Comparisons with $\textit{Mind the GAP}$ and $\textit{StyleCLIP}$**

While Mind the GAP has a similar loss with direction loss, one term of our proposed directional subspace loss, their motivations differ. Mind the GAP is proposed for one-shot domain adaptation, thus representing the entire target domain using the embedding of a single image. In contrast, for the few-shot setting, we propose representing the entire domain using the subspace formed by the embeddings of multiple images. Furthermore, we utilize pre-trained encoders to obtain separated subspaces corresponding to different domains, enabling us to accomplish hybrid domain adaptation. 

The direction loss in StyleCLIP is computed based on the cosine similarity between the generated images and textual prompts within the CLIP embedding space. It does not depend on the source image or domain. In practice, this loss leads to adversarial solutions and sees no benefit from maintaining diversity as depicted in Style-NADA[5]. A mode-collapsed generator producing only one image may be the best minimizer for the distance to a given textual prompt. Differently, our direction loss aims to preserve more characteristics from source domain and maintains its diversity.

Reviewer comment (Novelty):
I’m unsure about why this paper claims to introduce Hybrid generative DA, after acknowledging that prior work studying this problem exists (eg. DoRM). I understand that DoRM necessitates training target-specific models, but do they not also present some results on hybrid generative DA (eg. Figure 1 of their paper)?

Author response:
**Why we claim to introduce Hybrid generative DA**

Prior techniques like style-mixing [1] can generate images from hybrid domain via interpolating latent codes. DoRM is just one of those methods. However, they lack a detailed investigation of HDA, which is merely mentioned as an additional feature. For example, they fail to provide systematic definition and proper evaluation metrics to distinguish good from bad.

With regarding to the method in achieving HDA, they encounter two primary issues:

(1) DoRM is not an end-to-end pipeline to address HDA. They primarily focus on single domain adaptation, necessitating the separate training and interpolating multiple models to accomplish HDA. This may not be the right way to approach this task. 

(2) DoRM necessitates intricate tuning of hyperparameters that interpolate between the source and target domains. As demonstrated in the DoRM paper, they assign a weight of 0.005 to the $\textit{baby}$ domain during single domain adaptation. Varying these hyperparameters significantly affects the outcomes. This complexity and sensitivity in parameter tuning make HDA's tuning more intricate and demanding.

Reviewer comment (Experiments):
The approach requires manually tuning $\alpha_i$, and does not provide a principled method for hyperparameter selection. How sensitive is the approach to this parameter?

Author response:
**How sensitive is our method to the domain coefficient $\alpha$**

In our method for hybrid domain adaptation, this parameter controls the composition ratio of the attribute from each domain. As depicted in A.4 of Appendix, we use $\alpha_i = 0.5$ for most experiments without the need for complex and intricate adjustments. To further explore the sensitivity, we conduct the study for simple traversal of $\alpha_i$. As shown in the Fig. 14 of Appendix, the attributes of generated images transit smoothly between domains. Our method produces the similar attribute blending effect when $\alpha_i \in$ \{0.4, 0.5, 0.6\} .

Reviewer comment (Experiments):
The paper would be strengthened by a discussion and additionally, a comparison to recent methods for diffusion model personalization (eg. Dreambooth). While the methods would not necessarily be directly comparable, it would be a valuable point of reference.

Author response:
**Comparison to methods for diffusion model personalization**

Current trend in customized text-to-image models like DreamBooth aim to mimic the appearance of subjects in a given reference set. Similar to ours, DreamBooth fine-tunes a pre-trained generator for the personalization. However, there are two key differences between personalization and HDA.

(1) DreamBooth aims to retain the individuals from the training images, requiring similar individuals across the training set. Differently, our goal is to acquire the attributes of target domain like $\textit{baby}$ and preserve consistency with the source generator, ensuring that images generated from the same noise exhibit similar individuals.

(2) Dreambooth utilizes text-to-image generator which necessitates intricate and laborious adjustments of prompts to synthesize images with specific attributes. Additionally, certain attributes are challenging to accurately describe using text, such as artistic paintings. Conversely, our adapted model adeptly preserves the domain-specific attributes of reference images without the need for intricate prompt engineering.

Reviewer comment (Novelty):
The paper would benefit from a more detailed conceptual comparison to Nitzan et al., CVPR 2023: particularly, their approach of repurposing dormant directions in latent space for a new domain, seems quite related to the proposed approach.

Author response:
**Conceptual comparison to Domain Expansion**

Domain Expansion aims to expand the pre-trained generator to have the capacity to generate images from multiple domains. To the end, it proposes to repurpose dormant directions in latent space for the new domains. While our approach involves training images from multiple target domains, our objective is to create a unseen composition of given target domains that integrates attributes from them. For example, given training images from $\textit{baby}$ and $\textit{sketch}$, Domain Expansion aims to expand the generator to have the capacity to generate images from $\textit{baby}$ or $\textit{sketch}$. Differently, we aim to adapt the generator to unseen hybrid domain $\textit{baby with the style of sketch}$.

Reviewer comment (Reproducibility):
The training time comparisons in Table 2 lack sufficient details: were the baseline numbers obtained by running publicly available code? Is it a completely apples-to-apples comparison, i.e. have all necessary parameters (eg. number and type of GPUs) been controlled for?

Author response:
**Details of the training time comparisons**

In Table 2, our comparison about training time is fair. Specifically, we measure all the time on a single NVIDIA TITAN GPU. Due to the absence of open-source code for DoRM, we implement it following their description in the paper, based on the official implementation of StyleGAN2-ADA. The Seq method is also developed in a similar manner. As for StyleNADA, we utilize their open-source code for training. For an apples-to-apples comparison, we set the same batch size as 4 and the same resolution as $256 \times 256$.

Reviewer comment (Novelty):
(Major) The paper does not provide strong arguments as to why hybrid domain adaptation is a meaningful task. It is not clear why the current approaches and benchmarks, e.g. domain adaptation and domain generalisation, are not sufficient to address the problem presented. These points need further elaboration to motivate the paper.

Author response:
**Why is hybrid domain adaptation a meaningful task**

Current generative domain adaptation approaches typically employ the discriminator to discern whether generated images belong to the target domain. When we require to generate images with integrated attributes, they need to collect images from the hybrid target domain (e.g., a $\textit{smiling}$ $\textit{baby}$ with the style of $\textit{sketch}$ in Fig. 1). However, these images tend to be more difficult to collect compared with single domain in real-world scenarios. In contrast, HDA only requires collecting data for several independent target domains rather than for each possible combination as mentioned by **Reviewer 2zhA**. Under such circumstances, HDA offers greater flexibility and versatility to adapt the generator to more composite and expansive domain.

Reviewer comment (Experiments):
(Major) results are reported using 10 shots, while it would be interesting to see results using different numbers of shots, e.g. 1-shot and 5-shot.

Author response:
**Results using different numbers of shots**

We perform experiments on lower shots (5-shot and 1-shot) as suggested using various datasets. As shown in Fig. 11 of Appendix, the results of 5-shot are close to those of 10-shot, which integrates the attributes and maintains the consistency with source domain. Although multiple attributes have been learned, the results of 1-shot exhibit relatively lower cross-domain consistency compared with 10-shot and 5-shot.  This is because when there is only one reference image per domain, the subspace in our directional subspace loss degenerates into a single point (see Fig. 2). Then distinct generated images corresponding to different noise tend to converge towards the same image in CLIP's embedding space, which comprises cross-domain consistency as depicted in Section 3.3.

Reviewer comment (Experiments):
It is unclear whether the methodology is applicable to the generation of conditioned image styles. For example, generating an image that is conditioned on one or more styles of the target domain, rather than an image that spans all attributes of the target domains. It could discussed how the proposed methodology generalises to conditional generation.

Author response:
**Results of conditional generation**

As shown in Fig. 12 of Appendix, we conduct the experiments on conditional generation. Specifically, we collect 10-shot images with $\textit{red hair}$ and $\textit{sunglasses}$. Then we use masks to separate these attributes and adapt the generator with masked images for $\textit{red hair}$ and $\textit{sunglasses}$ respectively. We can observe that the generated images possess the corresponding attribute for both single DA and hybrid DA. Simultaneously, these images also maintain consistency with source domain.

Reviewer comment (Experiments):
A study of the effect of different pre-trained image encoders would add value to the paper.

Author response:
**Effect of different pre-trained image encoders**

As shown in Fig. 13 of Appendix, we conduct experiments on pre-trained Swin and Dinov2 to explore the impact of different image encoders on generated images. Their results all achieve HDA, indicating that our method is agnostic to different pre-trained image encoders. Although they exhibit slight stylistic differences, these are due to their different approaches to extract features into separated subspaces, as depicted in Fig. 8 of Appendix. To converge to the exact realization of the target domain, our method employs the ensemble technique that exploits both Swin and Dinov2. As shown in the figure, the results closely resembles the attributes of the target domain while maintaining the best consistency with source domain.

Reviewer comment (Experiments):
How would the proposed generator compare with well-known generator models such as DALLE, Imagen and etc.?

Author response:
**Compare with DALL-E and Imagen**

DALL-E and Imagen are both text-to-image models known for their ability to generate images based on natural language descriptions. They exhibit the advantage of autonomously producing realistic images with specific attributes while maintaining considerable diversity. However, in comparison to our generator, they exhibit two primary drawbacks. 

(1) They necessitate intricate and laborious adjustments of prompts to generate images with specific attributes. Additionally, certain attributes are challenging to accurately describe using text, such as artistic paintings. Conversely, our generator can preserve the domain-specific attributes of reference images without the need for intricate prompt engineering.

(2) It is hard to control the characters within images via prompts for them. In contrast, our adapted model maintains strong consistency with the source generator, preserving the same characters from the same noise.

Besides, ours differs from the current trend in customized text-to-image models like DreamBooth. DreamBooth aims to retain the individuals from the training images, requiring similar individuals across the training set. Differently, our goal is to acquire the attributes of target domain like $\textit{baby}$ and preserve consistency with the source generator, ensuring that images generated from the same noise exhibit similar individuals.