P ROTOTYPE MEMORY AND ATTENTION MECHANISMS FOR FEW SHOT IMAGE GENERATION
A BSTRACT
Recent discoveries indicate that the neural codes in the superﬁcial layers of the primary visual cortex (V1) of macaque monkeys are complex, diverse and supersparse.
This leads us to ponder the computational advantages and functional role of these “grandmother cells."
Here, we propose that such cells can serve as prototype memory priors that bias and shape the distributed feature processing during the image generation process in the brain.
These memory prototypes are learned by momentum online clustering and are utilized through a memory-based attention operation.
Integrating this mechanism, we propose Memory Concept Attention ( MoCA ) to improve few shot image generation quality.
We show that having a prototype memory with attention mechanisms can improve image synthesis quality, learn interpretable visual concept clusters, and improve the robustness of the model.
Our results demonstrate the feasibility of the idea that these super-sparse complex feature detectors can serve as prototype memory priors for modulating the image synthesis processes in the visual system.
I NTRODUCTION
Recent neurophysiological ﬁndings based on calcium imaging have revealed that many neurons in the superﬁcial layers of V1 are strongly tuned to complex local patterns, rather than simple oriented edges or bars.
These complex neurons exhibit far stronger responses (a 3-5 fold increase) when exposed to their preferred patterns than when exposed to simple grating and bar-like stimuli.
The high degree of speciﬁcity in these neurons’ selectivity suggests that they might serve as speciﬁc pattern detectors (Tang et al., 2018a).
Due to the selectivity towards complex stimuli, the population response of these V1 neurons is extremely sparse.
Only 4–6 out of roughly 1000 neurons respond strongly to any given pattern or natural stimulus (Tang et al., 2018b).
This ﬁnding is reminiscent of an earlier study that found similarly sparse encoding of concepts in the hippocampus (Quiroga et al., 2005).
Recent data in V4 also indicate the existence of cells with a similarly high degree of sparsity * .
Thus, we theorize such cells should exist in every level of the hierarchical visual system.
We called these highly selective sparsely-responding feature detectors "grandmother neurons" to highlight their possible explicit encoding of speciﬁc prototypes, even though in reality, a prototype is likely represented by a sparse cluster of neurons rather than a single cell.
Neurons in different layers of each visual area exhibit different degrees of response sparsity, complementing one another in various functions.
The observations of this diverse set of super-sparsely responding feature detectors raise the following question: what are the possible computational beneﬁts and rationales of having such neurons in the early visual cortex?
In this paper, we hypothesize that these "grandmother neurons" can serve as a prototype memory prior to modulate the process of image synthesis.
Image synthesis is a central theme in a number of hierarchical models of the visual system, including interactive activation and predictive coding (McClelland & Rumelhart, 2020; Grossberg, 1987; Mumford, 1992; Rao & Ballard, 1999; Lee &
Mumford, 2003), and is hypothesized to take place through the top-down feedback connections between areas in the visual cortex.
These priors allow the synthesis process to look beyond the current spatial context and utilize the prototype memories learned and accumulated over time.
Thus, the "grandmother cells" serve as structural conceptual priors in a memory attention process during image generation.
To emphasize the importance of having an attention mechanism that operated beyond the current image representation, we name our proposed memory-based attentional process Memory Concept
Attention ( MoCA ).
MoCA is a module that can be plugged into any layer of pre-existing generator architectures in the GAN framework.
We test our model by performing extensive experiments using the state-of-the-art StyleGAN2 (Karras et al., 2020b) and a newly proposed few-shot image generator FastGAN (Liu et al., 2021).
Our experiments demonstrate the utilization of prototype information accumulated within the semantic clusters during training can improve few-shot image generation on Animal-Face Dog (Si & Zhu, 2012), 100-Shot-Obama (Zhao et al., 2020), ImageNet100 (Russakovsky et al., 2015), COCO-300 (Lin et al., 2014), CIFAR10 (Krizhevsky et al., 2009) and
Caltech-UCSD Birds (CUB)
(Welinder et al., 2010).
Additionally, we also ﬁnd that generators with
MoCA can resist a certain degree of injected noise corruption during testing time, suggesting that attending to a structured memory prior during generation can improve the robustness of the model.
Our goal here is to explore the utility of prototype cells as memory priors in a standard computer vision image generation task, with a view to gaining some insights into the advantages of having these "grandmother" neurons in the visual cortex at a functional level.
R ELATED W ORK
Visual Concept Learning Visual concepts, deﬁned as intermediate level semantic features, have been shown to be effective in overcoming misclassiﬁcation of objects due to occlusion when used in a feedforward voting scheme (Wang et al., 2017).
Explicit representations of visual concepts using prototype neurons might also serve as effective reconﬁgurable parts for building compositional machines (Bienenstock et al., 1997; Geman et al., 2002; Zhu & Mumford, 2007).
In this paper, we explore the use of visual concepts in the form of prototype memory priors to provide temporal spatial and temporal contextual modulation with an attention mechanism for the complex compositional task of image generation.
Self-Attention
The attention mechanism (Vaswani et al., 2017) in deep learning is popular in
Natural Language Processing (NLP) and is also known as "non-local networks" in the vision community (Wang et al., 2018).
Zhang et al. (2019a) introduced self-attention into generative models in the
Self-Attention GAN.
Since then, adding self-attention into Generative Adversarial Networks has become a standard practice.
Brock et al.
(2018) and Esser et al.
(2020) demonstrate the beneﬁts of using self-attention in high-ﬁdelity image synthesis models.
However, current self-attention mechanisms in
GANs only utilize contextual information within the same image to modulate the activation.
In this work, we propose the use of a memory cache of intermediate-level visual conceptual prototypes to provide additional modulation in addition to the standard self-attention.
In our work, the activation mechanism attends not only to its spatial context within the image itself, but also attends to a cached set of memory prototypes accumulated over time.
Prototype Memory Mechanism A memory bank can capture diverse features over a long time period, and has been shown to be effective in other domains.
Wu et al.
(2018) utilizes a memory bank in contrastive learning to obtain more diverse negative samples to contrast with.
Caron et al.
(2020) also uses a memory queue during training to accumulate representative negative samples.
He et al.
(2020) shows that the stability of the features accumulated in the memory bank can be improved using a momentum-updated encoder, a strategy we will also use in learning our prototypes.
SimGAN (Shrivastava et al., 2017) introduces the image pool trick which uses a buffer to store previously generated samples, in order to make the discriminator focus not only on the current training batch, but also improve itself based on memory.
The major innovation of our work is the proposal, inspired by recent neurophysiological ﬁndings, that we should have memory banks at intermediate levels, and, in principle, every level of the visual hierarchy.
While early memory bank work stored images at the instance level for object recognition, which is not useful for image generation, we argue that memory banks at lower levels of the hierarchy can store prototypes of parts and sub-parts which are useful for image generation.
Few-shot image generation, in particular, can beneﬁt from memory mechanisms that support ﬂexible composition and decomposition of parts, particularly when data are limited.
Few-Shot Prototypes Learning Few-shot learning refers to performing computer vision tasks when training data are very limited.
One of the popular ideas in the few-shot learning literature is to form distinct prototypes from the training set and use them during testing (Snell et al., 2017).
Although
MoCA also forms prototypes during the training stage and uses them during inference, there are two important distinctions between our work and Snell et al.
(2017): (1) Snell et al.
(2017) forms prototypes at the instance level whereas MoCA’s prototypes are generated at the intermediate parts level.
(2) Snell et al.
(2017) simply selects the closest prototype to obtain discrete class prediction,
MoCA, on the other hand, employs an attention process to continuously modulate the activation features based on the prototypes, hence can be applied to a broader set of tasks, including image synthesis, which predicts continuous pixel values.
Few-Shot Image Generation Few-shot image generation task is a challenging task as GANs are very data-hungry and inefﬁcient.
Unconditional image generation is especially difﬁcult among the current few-shot image generation schemes.
Different solutions have been developed in the literature.
Recent works such as DiffAug (Zhao et al., 2020), StyleGAN-Ada (Karras et al., 2020a) propose differentiable augmentation to avoid overﬁtting in the discriminator.
InsGen (Yang et al., 2021) proposes to use a contrastive learning objective to enhance the adversarial loss in the few-shot generation setting.
These works mostly proposed methods to improve the discriminators so that they can provide better error signals to the generator, but do not speciﬁcally address the generator architecture.
Another line of research proposed generator architectures to avoid mode collapse.
The state-of-the-art architecture for few-shot image generation (hundreds of real images) is FastGAN (Liu et al., 2021).
Here, we propose a new architectural change on the generator side and make comparison against FastGAN architecture extensively in the experiment section.
M ETHODS
Our key contribution is the introduction of a novel prototype-based memory modulation module to improve the generator network of a GAN.
The activation from the previous layer is modiﬁed by two attention processes: 1) contextual modulation with Memory Concept Attention ( MoCA ), and 2) spatial contextual modulation within the generated image itself ( Self-Attention ).
Our module takes a feature map in the GAN hierarchy as input combines the results of these two mechanisms to modulate the feature map for further downstream processing.
A high-level overview of our model is shown in Figure 1.
Formally, we denote the input to our MoCA layer to be the activation of a particular layer, A ∈ R n × c × h × w .
The output produces a modulation H ∈ R n × c × h ×
w that update A into ˆ A .
To allow the information to be ﬂexibly modulated, we ﬁrst transform A by 1 × 1 convolutions into a lowerdimensional space via functions θ ( · ) , φ ( · ) , and ψ ( · ) , where { θ ( A ) , φ ( A ) , ψ ( A ) } ∈ R n × ˜ c
× h × w .
P ROTOTYPE M EMORY L EARNING
In this section, we discuss the organization of our prototype memory that is used as one route of modulation.
Our prototype concept memory is arranged hierarchically into semantic cells and prototype cells.
As shown in Figure 2, each semantic cell is cluster mean representative of a cluster of prototype cells.
Formally, suppose our memory P consists of M semantic cells P
= { K 1 , K 2 , ..., K M } .
For each of the semantic cells K i , there are T of prototype cells { E ( i ) 1 , E ( i ) 2 , E ( i ) 3 , ..., E ( i ) T } stored in the memory cell associated with K i , where E ( i ) j and K i both ∈ R ˜ c
and K i =
( (cid:80) Tj =1 E ( i ) j ) /T is the mean of the stored prototype cells.
These prototype cells come from the feature maps in previous iterations after being transformed via a momentum updated context encoder ˜ φ ( · ) and are updated in the memory at the end of every training iteration.
˜ φ ( · ) is a momentum counterpart of φ ( · ) and its parameter is updated as shown in Equation 1 with the momentum parameter m .
˜ φ ( · ) does not change as rapidly as φ ( · ) so that the prototype learned are more stable, accumulating information beyond the current training batch.
The rationale is similar to that in (He et al., 2020)).
[Equation]
After transformed by ˜ φ ( · ) to a low-dimensional space, the activation at each hypercolumn (pixel location) in the feature map is assigned to its closest semantic cluster and replaces an existing prototype cell in the memory bank of that cluster.
We use a random replacement policy to prevent the prototype cells from all collapsing to trivial solutions.
The update is done in a batch synchronized fashion, i.e. we update K i as the cluster mean of the i th cluster of prototypes after the entire memory is updated based on the recent batch.
3. M EMORY C ONCEPT A TTENTION
As we described in section 3.1, our memory is a cache of hypercolumn activation patterns gathered across the entire training set as well as from different training periods.
By organizing them via semantic cells ( K i s), we can route more diverse information into the ﬁnal activation while still maintaining high data efﬁciency, hence yielding higher quality image synthesis task.
In this section, we present the detailed operation given an existing memory P and the input feature transformation { θ ( A ) , φ ( A ) , ψ ( A ) } ∈ R n × ˜ c
× h × w .
During the memory attention process, an activation column a ∈ R ˜ c
from θ ( A ) ﬁrst selects the closest semantic cell K i and retrieves the associated prototype cell matrix E ( i ) ∈ R ˜ c
× H from memory, where each column j of E ( i ) is a prototype cell E ( i ) j ∈ R ˜
c
belonging to the cluster of the selected semantic cell K i .
The next step is to attend to the activation column a with the prototype cell matrix E ( i ) .
First, a similarity score s is calculated as s =
[ E ( i ) ] (cid:48) a .
A nonlinear softmax normalization is then applied to s to obtain the normalized attention weight β where for each entry β t ( t = { 1 , 2 , ..., T } ) Equationapplies.
[Equation]
Using the (softly) normalized attention weight β , we can build the retrieved information from memory h m =
E ( i ) β ( h m ∈ R ˜ c ) for the activation column a .
By applying the same operation to the activation a at every spatial location as well as to every image in the batch, we obtain H m ∈ R n × ˜ c
× h × w .
3. S PATIAL C ONTEXTUAL A TTENTION
While the memory prior is important, spatial contextual information also plays a role in the activation modulation.
Therefore, we also use a non-local network to implement spatial contextual modulation in the same layer.
Speciﬁcally, we ﬁrst compute a afﬁnity map between θ ( A ) and φ ( A ) , denoting as
S =
[ θ ( A )] T φ ( A ) .
Each row of S is then normalized via softmax to allow a rather sparse attention weight ˆ S to be computed.
ˆ S is then multiplied with φ ( A ) to get the spatial contextual modulation tensor H s ∈ R n × ˜ c
× h × w .
3. I NTEGRATE TWO ROUTES OF MODULATION
Finally, we integrate the retrieved information from memory H m and the spatial contextual modulation H s by element-wise addition, H = H m ⊕ H s , which is then transformed back to the original feature space via the 1 × 1 convolution O ( · ) .
A learnable parameter γ is multiplied to the multiplication as weight and add back to the input activation, i.e. ˆ A = γO ( H ) + A , which is then output into the next layer in the generator.
E XPERIMENTS
The proposed MoCA module is a generator architecture improvement and is not speciﬁc to the discriminator’s training techniques.
Note that some of the existing works on unconditional few-shot image generation (DiffAug (Zhao et al., 2020), ADA (Karras et al., 2020a), etc.) improve the quality of image synthesis by focusing on the discriminator side.
For example, DiffAug proposes to perform augmentation of the generated image before being fed into the discriminator to prevent discriminator’s overﬁtting.
Although this line of work is valuable, it is orthogonal and complementary to MoCA, as MoCA is a generator architecture that can be trained under all of these discriminator training techniques.
In the following experiments, we use a MoCA enhanced generator with either DiffAug or ADA augmented discriminators.
Datasets and Metrics We validate the performance improvements of adding the MoCA module for few-shot image synthesis with 6 datasets.
Namely, we perform comparisons between the baseline architecture and the MoCA-enhanced architecture on Animal-Face Dog (Si & Zhu, 2012),
100Shot-Obama (Zhao et al., 2020), ImageNet-100 (Russakovsky et al., 2015), COCO-300 (Lin et al., 2014), CIFAR10 (Krizhevsky et al., 2009) and Caltech-UCSD Birds (CUB)
(Welinder et al., 2010).
Animal-Face Dog consists of 389 dog images while 100-Shot-Obama contains 100 Obama faces with different expressions.
We use 256x256 resolution for both of these datasets across all experiments.
To further challenge the few-shot image generation task, we select 100 and 300 images from one class of ImageNet ("Jeep" category) and MSCOCO ("Train" category) respectively to compose the ImageNet-100 and COCO-300 datasets (the details of the selected image list can be found in the Appendix 8.5).
In addition, we also conduct experiments on CUB (5,990 256x256 resolution wild bird images) and CIFAR10 datasets (60,000 32x32 images span across 10 categories).
For evaluation metrics, we use a widely adopted metric, Fréchet Inception Distance (FID) (Heusel et al., 2018) to measure the quality of generated images.
For most of the experiments, except the one for the evaluation of model robustness against noise, the standard deviation of FID are often less than 1% relatively and much smaller than the performance improvement.
To strengthen conﬁdence in the evaluation of the image synthesis performance, we also evaluate the Kernel Inception Distance (KID) metric (Bi ´ nkowski et al., 2018), which is found to be more descriptive for the few-shot image generation tasks.
Baselines To demonstrate that MoCA is not limited to a certain backbone architecture, we add
MoCA to two state-of-the-art architectures, FastGAN
(Liu et al., 2021) and StyleGAN2 (Karras et al., 2020b).
Of note, FastGAN is a state-of-the-art architecture that is speciﬁcally targeted on image synthesis in the extremely low data regime.
It is an important baseline architecture that provides reasonable few-shot synthesis given only hundreds of real images and as few as 15 GPU hours of training time.
Additionally, we also add MoCA to StyleGAN2, which is a more powerful and generic model but requires more data and computing resources.
We will showcase that the MoCA layer can be useful for generators in both architectures in Section 4.1.
The implementations of the baseline models are based on their ofﬁcial publicly available PyTorch implementations.
We follow the best training conﬁguration reported in the original papers to train our models.
4. F EW - SHOT I MAGE S YNTHESIS P ERFORMANCE
Here, we present the results of the experiments on 4 small datasets, each of which has only hundreds of real images (Animal-Face Dog (Si & Zhu, 2012), 100-Shot-Obama (Zhao et al., 2020), ImageNet100 (Russakovsky et al., 2015), and COCO-300 (Lin et al., 2014)).
Qualitative results are shown in
Figure 3.
Meanwhile, we show quantitative comparisons between the best performance of MoCA and baseline models for each dataset in Table 1 (FastGAN) and Table 2 (StyleGAN2).
Note that the
KID is shown on the scale of 10 − 3 .
As shown in the results, we observe consistent improvements in terms of image synthesis quality in this difﬁcult low data regime when adding MoCA to the generator.
In terms of FID score, we observe that, using the FastGAN base architecture, MoCA can bring a 5.8% improvement on Animal Face Dog, 13.8% improvement on Obama, 21.7% improvement on
ImageNet-100 and 12.4% improvement on the COCO-300 dataset (Table 1).
Similar improvements are also observed with KID metrics.
To further test MoCA’s enhancement on more powerful model, we run experiments with a StyleGAN base architecture.
Using the FID evaluation metric, we observe a 5.1% improvement on Animal Face
Dog dataset, 8.1% improvement on Obama dataset, 14.1% improvement on ImageNet-100 dataset and 17.3% improvement on COCO-300 dataset.
Note that ImageNet-100 and COCO-300 dataset are very challenging as they contain much more diverse scenes than Animal Face Dog and Obama.
To compensate for the model size of the StyleGAN2 base architecture, we downsize ImageNet-100 and
COCO-300 to a 64x64 image size when training using StyleGAN2 and MoCA-StyleGAN2.
From the results, we observe that MoCA-StyleGAN2 consistently synthesizes better images compared to the baseline despite the image size.
Additionally, we also observe that different augmentation methods
[Table]
[Table] [Table]
on the discriminator side do not affect MoCA’s improvement, as FastGAN based models use DiffAug whereas StyleGAN2 based models employ the ADA technique.
4. A BLATION STUDIES
In this section, we present ablation studies for MoCA layer compared with self-attention alone and the importance of the momentum head design.
Additional ablation studies on the design of memory clustering organization used in MoCA can be found in Appendix 8.1, and more visualization analysis can be found in Appendix 8.6.
Self-Attention v.s. MoCA
On both StyleGAN2 and FastGAN, we compare the performance of
MoCA with a standard self-attention module to verify that the performance improvement of our model is not only due to the self-attention block, as several previous works have highlighted the advantage of applying self-attention on GANs to capture non-local dependency (Zhang et al., 2019b; Brock et al., 2018).
Table 3 shows that in spite of the baseline architecture, MoCA consistently outperforms the standard Self-Attention module, suggesting that MoCA module plays an irreplaceable role in our method.
Importance of Momentum Update Mechanism We also study the effect of different ways to update the concept encoder.
Table 4 shows that the models’ FID on multiple datasets generally improves when using momentum to update the concept encoder.
With momentum, the FID of
FastGAN on larger and diverse datasets like AnimalFace Dog and CUB is notably better.
This supports our motivation for using a momentum-updated encoder to build a more generalized memory bank that contains broader information beyond the current training batch.
[Table]
P ROTOTYPE C ONCEPT A NALYSIS
Here, we demonstrate that MoCA can learn in distinct semantic concepts in an unsupervised manner, represented by the "grandmother cells" or their representatives, during adversary learning for image synthesis.
Semantic Concepts in MoCA
We assigned a semantic cluster to each hypercolumn in thex 16 feature map during a feedforward generation pass (cluster selection mechanism is described in Method Section 3.2).
We label the receptive ﬁelds or projection ﬁelds of all the hypercolumns associated with each chosen prototype semantic cluster as white bounding boxes in the generated image, with some examples shown in Figure 4.
We can observe that different clusters encode distinct semantic concepts.
For examples, Cluster 0 tends to be associated with train rails.
Cluster 2 covers area of uniform color such as sky and ground.
Cluster 17 pays attention to the side of the train.
Cluster 8 focus on the trains themselves, particularly the front.
Along each row, we observe that the inﬂuence of MoCA for a speciﬁc image can be decomposed into different clusters.
Some clusters are more popular (e.g. Cluster 0, 2, 8) while other clusters are used more selectively (for example, cluster 17 for the third image).
Understanding Prototype Cells To understand the prototype cells that are cached inside each cluster of MoCA, we identiﬁed the generated image patch in the receptive ﬁeld of any hypercolumn whose activation is closest to the prototype cell’s memory in the assigned cluster of that hypercolumn from 100 image generations.
Figure 5 show that the image patches closest to a prototype memory are similar visually, and that prototypes belonging to the same cluster are semantically related but distinct in visual features.
We can see that different prototype cells are specialized as sub-parts of the semantic clusters they belong to.
For example in Cluster 0, prototype 20 is a cached activation for rail or train tracks, but other prototypes include the top of the trains.
Similar observations can also be made for Cluster 2 and 8 (Figure 5).
These three clusters are queried frequently and therefore likely contain diverse but semantically related prototype memories.
The prototype memories resemble the "visual concepts" of (Wang et al., 2017) and can potentially be used as visual concepts to implement hierarchical compositional system for image synthesis.
4. R OBUSTNESS AGAINST NOISE
We evaluate the noise robustness of our proposed memory concept attention module and compare it against the standard self-attention module.
The evaluation is conducted on FastGAN models trained without any injection of noise into the intermediate layers.
In the evaluation stage, we inject Gaussian noise of different magnitudes with variance from 0 to 1 into the input feature map of the MoCA layer and standard self-attention layer.
Our results (Table 5) show that, without the help of external memory, self-attention is more vulnerable to the noise attack, while MoCA is generally less sensitive to noise.
We suspect that with MoCA, the feature map can attend to previously-stored concepts in the memory bank to retrieve noise-free part information, which alleviates the inﬂuence of noise perturbation.
We also ﬁnd that, no-cluster MoCA (All prototypes belong to the same cluster and jointly inference every activation A ij , i.e. M=1) is more robust under higher levels of noise.
We hypothesize that the feature map can attend to more concepts in no-cluster MoCA, increasing the chances of obtaining a correct bias.
[Figure]
[Table]
C ONCLUSION
In this paper, we introduced a module called MoCA, which can be inserted into any layer of a hierarchical neural network such as a GAN to improve image synthesis.
MoCA caches part prototype memories over time, dynamically updated using a momentum encoder, and has the ability to modulate the continuous response of intermediate layers via an attention mechanism.
In contrast to earlier memory bank studies which stored representation of the entire images as prior in the latent space for object recognition, the prototypes in MoCA are remembered parts and sub-parts of objects, that can be extracted at every level of the hierarchy, and ﬂexibly composed for image synthesis.
Our work was inspired by the discovery of a diverse set of highly selective complex feature detectors in V1, which correspond to the prototype cells in MoCA.
The selection of different prototype clusters via semantic cells could correspond to switching circuits mediated by inhibitory neurons.
While we do not claim that there is any serious correspondence between our model and neural mechanisms or circuits, the effectiveness of MoCA might still provide some insights to the roles of these "grandmother neurons" at a functional level.
E THICS S TATEMENT
This study investigates a novel proposal for understanding the computational advantages of having prototype-like “grandmother” neurons in the visual cortex, namely, that these prototype neurons can serve as auxiliary memory priors to modify both the embedding space of the self-attention mechanism as well as the activation patterns of neurons during the image synthesis process.
While the motivation is for advancing our understanding of contextual modulation in neural systems, our work also results in a novel mechanism for contextual modulation in deep learning systems, particularly in image generation, and thus has a broader positive impact on machine learning, particularly on few shot learning.
However, image generation technology can be misused for misinformation, with adverse social impact, e.g. the creation of "deepfakes."
On the other hand, advancing our understanding of image generation is also crucial for combating deepfakes and other misuses of image generation technology by bad actors.
R EPRODUCIBILITY S TATEMENT
Our project code can be found at https://github.com/Crazy-Jack/MoCA_release . A CKNOWLEDGEMENT This work was supported by an NSF grant CISE RI 1816568 awarded to T.S.L. This work is also partially supported by the start-up fund provided by CMU Mechanical Engineering Department.
A PPENDIX 8. A BLATION : I MPORTANCE OF M EMORY C LUSTERING O RGANIZATION
[Table]
The second aspect we study is the impact of the clustering mechanism on the overall model performance.
We ﬁnd that for MoCA, a single concept pool with sufﬁciently large size, can still has similar performance compared against MoCA with multiple clusters.
As shown in Table 6, the performance of nocluster MoCA containing 8192 concepts inside, is at the similar level as MoCA with 32 clusters.
We hypothesize this is because, although there is no clustering mechanism to enforce the memory bank learning diverse concepts in no-cluster MoCA, given a very large pool of concepts, it is still able to memorize diverse and useful concepts.
Despite the ﬁnding that no-cluster MoCA can have similar performance to cluster MoCA with a sufﬁciently large pool, we argue that using the clustering mechanism is still superior.
Uusing the clustering mechanism greatly reduces the amount of vectors involved in the attention calculation, which enables us to build a larger memory bank for complex datasets
8. M O CA ON LS-GAN
To strengthen our conclusion on MoCA, we have also conducted experiments on the least-square
GAN (LS-GAN)(Mao et al., 2017) framework which has a different loss and architecture compared to FastGAN and StyleGAN.
We found MoCA can still provide a certain amount of improvement, consolidating the advantages of the MoCA layer.
[Table]
8. L IMITATION AND FAILURE
Despite MoCA can bring in consistent improvement on most of the dataset with limited data
[Table]
we also observe case where MoCA is less effective.
We found that adding MoCA on StyleGAN for training
Grumpy-cat leads to a slight performance decrease and leads to small improvement on FastGAN.
This suggests that when the underlying dataset is less diverse and the base network is quite large
[Table]
MoCA cached concepts can potentially cause more distraction during generation, leading to performance setbacks
[Table]
8. A LGORITHM
Here, we present the algorithm that is presented in Figure 1 and Figure 2 in the main text.
Algorithm
Result:
Updated activation ˆ A after MoCA and Self-Attention; Updated Memory P .
A previous ; for spatial A ) do
Choose prototype semantic cell K
i ← argmin
i
||
K i − a || 22 ;
Retrieve Prototype Component Cells for K from memory; end A m do Choose
Randomly
; end
8. E XPERIMENTAL DETAILS
In this section we provide details for how we implement MoCA on FastGAN and StyleGAN 2 respectively.
For reference, we provide our code and image ID list of MSCOCO, ImageNet subset used in the experiments at https://anonymous.4open.science/r/MoCA-D372/README.md .
All the datasets used in experiments are publicly available and for non-commercial usage only.
Among them, CIFAR-10 † is attributed to Krizhevsky (2009), Obama, Animal Face, Grumpy Cat ‡ are attributed to Zhao et al.
(2020), and Caltech-UCSD Birds-200-2011 § (CUB) is attributed to Wah et al.
We insert MoCA layer between the last but one residual block and the last residual block of the generator.
Given the input feature map A of the last but one residual block f ( l − 1) , the last residual block f ( l ) and MoCA module Φ , suppose the original image generation procedure on the last two blocks is:
[Equation]
Then the feedforward procedure including MoCA can be deﬁned as:
[Equation]
where ˆ x is the generated image.
Please note that, to enable that MoCA can be ﬂexibly integrate into arbitrary layer, MoCA will not change the channel numbers and resolution of the feature map.
8.5. F AST GAN
Network Architecture and Optimization
We use exactly the Fast-GAN architecture Liu et al.
(2021) as the backbone.
MoCA layer is inserted after the 64x64 resolution layer for all experiments run for CUB, AnimalFace Dog, Obama and Grumpy-cat.
We use Adam optimizer for training and 0.0002 for learning rate with β 1 = 0 .
5 and β 2 = 0 .
Differentialbe augmentation is used during training.
We utilize the loss proposed in (Liu et al., 2021) for stable and optimal training.
For all our experiments, 100K iteration is run to perform fair comparsion.
Computational Resource With FastGAN and MoCA, it takes about 15 hours to train 100K iteration on a single GTX-1080Ti GPU.
8.5. S TYLE GAN
Network Architecture and Optimization
We use exactly same architecture from Karras et al.
(2020a) as the backbone.
We use Adam (Kingma & Ba, 2017) optimizer with a learning rate of 0 .
0025 and β 1 = 0 , β 2 = 0 .
We adopt the same training conﬁguration as the Ofﬁcial
PyTorch implementation of StyleGAN 2 (including Adaptive discriminator augmentation, path length regularization , lazy regularization (Karras et al., 2020b), and non-saturating logistic loss (Goodfellow et al., 2014)).
For the training on the large dataset (CIFAR-10), we train models for 200k iterations of gradient updates, and 100k iterations for few-shot datasets.
Computational Resource We train StyleGAN2-based model with 4 RTX-2080Ti GPUs each run.
The training of each model takes roughly 20 - 35 hours to complete, depending on the image resolution.
8. M EMORY CONCEPT ANALYSIS
[Figure]
M
Feature map [ch, h, w] [ch, M] [M, h, w] transpose
Affinity to M prototypes  which prototype has the largest affinity?
Index of closest prototypes [h, w] qu e r y t h e i - t h c h a nn e l
i
i i
M pick 3 most frequent indexes affinity heatmap w.r.t i-th prototype [h , w] i
[Figure]
We generate 50,000 images from random noise for CIFAR-10, and 1,000 images for 100-shot Obama.
For each feature map and speciﬁc concept cluster i , we count how many pixels’ closest cluster (measured by the magnitude of the dot product) is i , and repeat this process for all clusters.
Then we sum the count and pick top-K clusters which have the most pixels, and visualize their afﬁnity heatmap.
To interpret the vector embeddings (concepts) stored in each concept cluster, we perform three kinds of visualization: afﬁnity heatmap w.r.t.
different concept clusters, images’ binary mask w.r.t.
different concept clusters, and t-SNE visualization (van der Maaten & Hinton, 2008) of the feature map before and after attending with memory concept.
The detailed information retrieval process is shown in Figure 6.
We use StyleGAN2+MoCA’s results for visualization, as they perform the best on most of the dataset.
Semantic concepts In Figure 7a and Figure 7b, we show the resulting afﬁnity maps computed as discussed above (detailed schematic of visualization procedure is shown in Figure 6).
Note that we select clusters and images for demonstration purpose.
Results on CIFAR10 using MoCA-StyleGAN (32x32 resolution) and Obama dataset using MoCA-FastGAN (256x256 resolution) are acquired.
We ﬁnd that these afﬁnity maps demonstrate that concepts formed during MoCA training cover crucial and diverse patterns for image synthesis.
As shown in Figure 7a and Figure 7b, patterns of different clusters are usually semantic meaningful and very distinct.
On CIFAR-10, we notice that cluster 11 stores concepts related to the sky, while cluster 13 stores concepts related to the frame of trucks.
Cluster 7 favors the white space while cluster 18 has a "crush" on animals’ body and head.
Similarly, on the other dataset, different clusters also focus on very different patterns.
For instance, on 100-shot
Obama, cluster 3 stores the concepts closely related to ties, and cluster 9 contains concepts related to the light reﬂection on the face.
[Figure]
meanings although not trained with category supervision.
Prototype attention We study the internal dynamics of MoCA by using t-SNE to visualize the changing trend of feature map before and after attending with concepts in the memory.
We generate 10,000 images from random noise on CIFAR-10 and extract the feature maps before and after
MoCA layer.
Figure 9a shows that the activation patterns (blue dots) from previous layer, after the θ transform, are effectively inﬂuenced by MoCA (green box in Figure 1) into a representation (red dots) that are pulled towards or stretched by certain familiar prototype exemplars (green dots).
Image synthesis as concepts assembling By visualizing the afﬁnity to the prototype of each pixel in the feature map, we acquire a high-level understanding of what concepts different clusters are trying to capture.
We can also further investigate the relationship between different concepts and the image synthesis process.
We decompose the image into different binary masks with regard to top-3 clusters.
The decomposition is shown in Figure 8.
Concretely, given a speciﬁc cluster C k and a feature map F , we generate the binary mask w.r.t.
C k by querying whether pixels’( F ij ) closest cluster (highest afﬁnity) is C k or not, where white denotes true and black denotes false.
We pick the top-3 clusters for each image and visualize their binary mask as shown in Figure 8.
For most images, concepts from the top-2 clusters are usually related to foreground and background information , while the third cluster contains concepts related to high-frequency details in an image.
This demonstrate that, with MoCA, image synthesis can also be viewed as a process of retrieving different concepts from memory and assembling them together to create a realistic image.
Implicit inﬂuence of MoCA the attention mechanism Interestingly, we also observe that having the MoCA module can in turn sharpen the functional activities of the horizontal interactions.
To visualize the functional connectivity of the horizontal connections, we plot the normalized rank-order afﬁnity score of the self-attention map in the 16x16 resolution layer of the converged StyleGAN trained on CIFAR10.
[Figure]
[Figure]
In Figure 9b, the purple curve represents the rank-ordered curve without installing the MoCA and the red represents that with MoCA.
Clearly, MoCA sharpens the functional connectivity among the horizontal connections, even though we do not explicitly modify the self-attention process in the MoCA layer, indicating that our prototype memory is implicitly inﬂuencing the horizontal connections.
8. M ETRIC EVALUATION
For all models, we evaluate metric using the last 3 snapshots of models (with an interval of 10k gradient updates) and report the best one.
We train all StyleGAN-based models for 200k iterations and all FastGAN-based models for 100k iterations.
To measure the model performance on Animal-Face
Dog (Si & Zhu, 2012), 100-Shot-Obama (Zhao et al., 2020), ImageNet-100 (Russakovsky et al., 2015), COCO-300 (Lin et al., 2014), we sample 3 , 000 noise vectors from standard Gaussian and calculate FID and KID score of the generated images.
On CIFAR10 (Krizhevsky et al., 2009) and
Caltech-UCSD (Welinder et al., 2010), we sample 50 , 000 respectively to adjust for number of training data.
8. G ENERATED IMAGES AND THEIR NEAREST NEIGHBOR IN THE DATASET
Although MoCA proposes a memory module, our improved results are not coming from trivially memorizing the training images.
First, the metrics we are using (FID and KID scores) take the generation diversity in to consideration.
We sample over 3,000 or 50,000 images during the evaluation process, however the entire training dataset only contains hundreds.
Second, to qualitatively test if
MoCA only perform generation by memorizing the training images, we ﬁnd the nearest neighbor of generated images based on the Learned Perceptual Image Patch Similarity metric (LPIPS) (Zhang et al., 2018).
The feature extractor used here is a pretrained VGG16 (Simonyan & Zisserman, 2015).
In Figure 10, we see that on the left is some images randomly chosen from the MoCA-FastGAN generation, and on the right is top-3 closest training images based on the LPIPS (Zhang et al., 2018) similarity metrics.
We can clearly observe that the generated images are different from the training images.
And more interestingly, we can see that the generated images indeed incorporate some parts level information and combine those parts level examples to compose new images.
For example, the generated train example on the second row (left) may have the similar train head as the second training image, but resembles a similar train side view (the shape) as the ﬁrst training image.
The overall direction of the rails also have looks similar as the ﬁrst image but it replace the rain shelter on the boarding platform with an open sky with some electric cables hanging over.
This further consolidates with the observation made in Section 4.3 that MoCA works at modulating the parts level generations.
Overall, the results in Figure 10 suggests MoCA’s generations are not only memorization of the training images but rather combine different parts to compose a new instance.
[Figure] Top 3 Nearest Neighbors in the dataset
8. V ISUALIZATION D ETAILS
Details of Receptive Field Localization In order to generate the receptive ﬁeld of each hypercolumn (white bounding box in Figure 4), we calculate the receptive ﬁeld according to the network architectures (Liu et al., 2021).
The ﬁnal image is of 256 x 256 resolution.
For a single hyper-column in the layer we install MoCA (layer 16 x 16), there are 4 up-sampling layers (each with factor of 2) and 7 convolution layers (each has 3x3 kernel size, 1 stride and 1 padding), resulting in a 66 xreceptive ﬁeld size.
The location is traced based on the recursive derivation of the convolution and up-sampling operations (Goodfellow et al., 2016).
Details of Visualizing the Prototypes Since our prototypes only inﬂuence the activation via attention, it is not straightforward to visualize a prototype.
However, since our prototypes are used to modulate the hyper-column weighted by the normalized similarity between the θ ( A ij ) and the prototype vector, we can consider hyper-columns that have high cosine similarity with the prototype vector in the projected space as an approximation of that prototype because if they are similar, there will be a large weight during attention process.
Since each hyper-column would correspond to a patch generation in the image, we can visualize the corresponding patch and get a sense of what the individual prototype cell is representing for.