R ETRIEVER : L EARNING C ONTENT -S TYLE R EPRESENTATION AS A T OKEN -L EVEL B IPARTITE G RAPH
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper addresses the unsupervised learning of content-style decomposed rep- resentation
We ﬁrst give a deﬁnition of style and then model the content-style representation as a token-level bipartite graph.
An unsupervised framework, named
Retriever , is proposed to learn such representations.
First, a cross-attention module is employed to retrieve permutation invariant (P
I.) information, deﬁned as style, from the input data.
Second, a vector quantization (VQ) module is used, together with man-induced constraints, to produce interpretable content tokens.
Last, an innovative link attention module serves as the decoder to reconstruct data from the decomposed content and style
, with the help of the linking keys.
Being modal-agnostic, the proposed Retriever is evaluated in both speech and image domains.
The state-of-the-art zero-shot voice conversion performance conﬁrms the disentangling ability of our framework.
Top performance is also achieved in the part discovery task for images, verifying the interpretability of our representation.
In addition, the vivid part-based style transfer quality demonstrates the potential of Retriever to support various fascinating generative tasks.
Code will be released upon acceptance of the paper.
I NTRODUCTION
Human perceptual systems routinely separate content and style to better understand their observations (Tenenbaum & Freeman, 2000).
In artiﬁcial intelligence, a content and style decomposed representation is also very much desired.
However, we notice that existing work does not have a uniﬁed deﬁnition for content and style.
Some deﬁnitions are dataset-dependent (Chou & Lee, 2019;
Ren et al., 2021), while some others have to be deﬁned on a certain modality (Lorenz et al., 2019; Wu et al., 2019).
We wonder, since content-style separation is helpful to our entire perception system, why is there not a uniﬁed deﬁnition that applies to all perception data?
In order to answer this question, we must ﬁrst study the characteristics of data.
The data of interest, including text, speech, image, and video, are structured.
They can be divided into standardized tokens, either naturally as words in language and speech or intentionally as patches in images.
Notably, the order of these tokens matters.
Disrupting the order of words can make a speech express a completely different meaning.
Reversing the order of frames in a video can make a stand-up action become a sit-down action.
But there is also some information that is not affected by the order of tokens.
For example, scrambling the words in a speech does not change the speaker’s voice, and a frame-shufﬂed video does not change how the person in the video looks.
We notice that the content we intuitively think of is affected by the order of tokens, while the style is usually not.
Therefore, we could generally deﬁne style as token-level permutation invariant (P.I.) information and deﬁne content as the rest of the information in structured data.
However, merely dividing data into two parts is not enough.
As Bengio et al.
(2013) pointed out, if we are to take the notion of disentangling seriously, we require a richer interaction of features than that offered by simple linear combinations.
An intuitive example is that we could never generate a colored pattern by linearly combining a generic color feature with a gray-scale stimulus pattern.
Inspired by this, we propose to model content and style by a token-level bipartite graph, as Figureillustrates.
This representation includes an array of content tokens, a set of style tokens, and a set of links modeling the interaction between them.
Such a representation allows for ﬁne-grained access and manipulation of features, and enables exciting downstream tasks such as part-level style transfer.
[Figure]
A set of style tokens    ... ... ...
N         10 1113 14 15 16 1719 20 21 22
To be, or not to be:
That is the question: 1 2 35
In this paper, we design a modal-agnostic framework, named Retriever , for learning bipartite graph representation of content and style.
Retriever adopts the autoencoder architecture and addresses two main challenges in it.
One is how to decompose the deﬁned content and style in an unsupervised setting.
The other is how to compose these two separated factors to reconstruct data.
To tackle the ﬁrst challenge, we employ a cross-attention module that takes the dataset-shared prototype as query to retrieve the style tokens from input data (Carion et al., 2020).
A cross-attention operation only allows the P.I. information to pass (Lee et al., 2019), which is exactly what we want for style.
On the other content path, we employ a vector quantization (VQ) module (van den Oord et al., 2017) as the information bottleneck.
In addition, we enforce man-induced constraints to make the content tokens interpretable.
To tackle the second challenge, we innovate the link attention module for the reconstruction from the bipartite graph.
Speciﬁcally, the content and style serve as the query and value, respectively.
Links between the content and style are learnt and stored in the linking keys.
Link attention allows us to retrieve style by content query.
As such, the interpretability is propagated from content to style, and the entire representation is friendly to ﬁne-grained editing.
We evaluate Retriever in both speech and image domains.
In the speech domain, we achieve state-of-the-art (SOTA) performance in zero-shot voice conversion, demonstrating a complete and precise decomposition of content and style.
In the image domain, we achieve competitive results in part discovery task, which demonstrates the interpretability of the decomposed content.
More excitingly, we try part-level style transfer, which cannot be offered by most of the existing contentstyle disentanglement approaches.
Vivid and interpretable results are achieved.
To summarize, our main contributions are three-folds: i)
We provide an intuitive and modalagnostic deﬁnition of content and style for structured data.
We are the ﬁrst to model content and style with a token-level bipartite graph.
ii)
We propose an unsupervised framework, named
Retriever , for learning the proposed content-style representation.
A novel link attention module is designed for data reconstruction from content-style bipartite graph.
iii)
We demonstrate the power of Retriever in challenging downstream tasks in both speech and image domains.
R ELATED W ORK
Content-style decomposed representation can be approached in supervised or unsupervised settings.
When style labels, such as the speaker labels of speeches (Kameoka et al., 2018; Qian et al., 2019;
Yuan et al., 2021) and the identity labels of face images (Mathieu et al., 2016; Szab´o et al., 2018;
Jha et al., 2018; Bouchacourt et al., 2018; Gabbay & Hoshen, 2020), are available, latent variables can be divided into content and style based on group supervision.
Recently, there has been increased interest in unsupervised learning of content and style.
Since there is no explicit supervision signal, the basic problem one must ﬁrst solve is the deﬁnition of content and style.
We discover that all existing deﬁnitions are either domain-speciﬁc or task-speciﬁc.
For example, in speech domain, Chou & Lee (2019) assume that style is the global statistical information and content is what is left after instance normalization (IN).
Ebbers et al.
(2021) suggest that style captures long-term stability and content captures short-term variations.
In image domain, the deﬁnition is even more diverse.
Lorenz et al.
(2019) try to discover the invariants under spatial and appearance transformations and treat them as style and content, respectively.
Wu et al.
(2019) deﬁne
[Figure]
content as 2D landmarks and style as the rest of information.
Ren et al.
(2021) deﬁne content as the most important factor across the whole dataset for image reconstruction, which is rather abstract.
In this work, we attempt to ﬁnd a general and modal-agnostic deﬁnition of content and style.
Style transfer is partially related to our work, as it concerns the combination of content and style to reconstruct data.
AdaIN (Huang & Belongie, 2017; Chou & Lee, 2019) goes beyond the linear combination of content and style (Tenenbaum & Freeman, 2000) and proposes to inject style into content by aligning the mean and variance of the content features with those of the style features.
However, style is not separated from content in this line of research.
Liu et al.
(2021) touch upon the part-based style transfer task as we do.
They model the relationship between content and style by a one-one mapping.
They follow the common deﬁnition of content and style in the image domain as shape and appearance, and try to disentangle them with hand-crafted data augmentation methods.
Besides, the term “style” is often seen in image generative models, such as StyleGAN (Karras et al., 2019).
However, the style mentioned in this type of work is conceptually different from the style in our work.
In StyleGAN, there is no concept of content, and style is the whole latent variable containing all the information including the appearance and shape of an image.
Following StyleGAN,
Hudson & Zitnick (2021) employ a bipartite structure to enable long-range interactions across the image, which iteratively propagates information from a set of latent variables to the evolving visual features.
Recently, researchers have become interested in disentangling content and style from the latent variables of StyleGAN (Alharbi & Wonka, 2020; Kwon & Ye, 2021).
However, they only work for well-aligned images and are hard to be applied to other modalities.
C ONTENT -S TYLE R EPRESENTATION FOR S TRUCTURED D ATA
In this section, we provide deﬁnitions of content and style for structured data, introduce the framework for content-style decomposition, and propose the token-level bipartite graph representation.
3. D EFINITION OF C ONTENT AND S TYLE
The data of interest is structured data that can be tokenized, denoted by X
=
[ x 1 , x 2 , ..., x n ] .
We think of text, speech, and image, among many others, as structured data.
Each token x i can be a word in text, a phoneme in a speech, or a patch in an image.
These data are structured because non-trivial instances (examples of trivial instances are a silent speech or a blank image) are not able to keep their full information when the order of tokens is not given.
Inspired by this intuition, we deﬁne style of X as the information that is not affected by the permutation of tokens, or permutation invariant (P.I.) information.
Content is the rest of information in X .
3. C ONTENT -S TYLE S EPARATION
The information in a piece of structured data is carried either in the content or in the style.
By deﬁnition, style can be extracted by a P.I. function f P.I. ( · ) , which satisﬁes f ( π ( X )) =
f ( X ) , where π ( · ) represents permutation of tokens.
To achieve content-style decomposition, we naturally adopt an autoencoder architecture, as shown in Figure 2.
The bottom path is the style path which implements f P.I. ( · ) .
We shall ﬁnd a powerful P.I. function which will let all the P.I. information pass.
The top path is responsible for extracting the content, but the challenge is that there does not exist a function which only lets pass the non-P.I. information.
Therefore, we employ in the content path a permutation-variant function, which will let all information pass, including the P.I. information.
[Figure]
C r o ss - A tt e n t i o n
Q
K V
[Figure]
Encoder
L i n k A tt e n t i o n Q K
Decoder
V
Output Tokens
Style Tokens
Content Tokens
Content Encoder
Structural Constraint x 𝑳 𝒅 Linking
[Figure]
S e l f - A tt e n t i o n
[Figure]
K
V
Pre
[Figure]
Obviously, if we do not pose any constraint on the content path, as shown in Figure 2 (a), the style information will be leaked to the content path.
To squeeze the style information out of the content path, an information bottleneck f B.N. ( · ) is required.
A perfect bottleneck, as shown in Figure 2 (c), can avoid the style leak while achieving perfect reconstruction.
But an imperfect bottleneck, being too wide or too narrow, will cause style leak or content loss, as shown in Figure 2 (b) and Figure(d), respectively.
3. T OKEN - LEVEL B IPARTITE G RAPH R EPRESENTATION OF C ONTENT AND S TYLE
After dividing the structured input X into content feature C and style feature S , we continue to explore how the relationship between C and S should be modeled.
Ideally, we do not want an all-to-one mapping, in which the style is applied to all the content as a whole.
Nor do we want a one-to-one mapping where each content token is associated with a ﬁxed style.
In order to provide a ﬂexible way of interaction, we propose a novel token-level bipartite graph modeling of C and S .
The token-level representations of C and S are C
= [ c 1 , c 2 , ..., c n ] and S =
{ s 1 , s 2 , ..., s m } , respectively.
Note that there is a one-one correspondence between x i and c i , so the order of the structured data is preserved in the content C .
While the order is preserved, the semantic meaning of each c i is not ﬁxed.
This suggests that the bipartite graph between C and S cannot be static.
In order to model a dynamic bipartite graph, we introduce a set of learnable linking keys
K = { k 1 , k 2 , ..., k m } .
The linking keys and style tokens form a set of key-value pairs { ( k i , s i ) } mi =1 .
Our linking key design allows for a soft and learnable combination of content and style.
The connection weight between a content token c
i and a style token s j is calculated by w i,j = r
θ ( c i , k j ) , where r θ is a learnable linking function parameterized with θ .
For a content token c
i , its content-speciﬁc style feature now can be calculated by s θ ( c i ) =
(cid:80) mj
=
ˆ w i,j s j , where ˆ w i,j are normalized weights.
T HE R ETRIEVER F RAMEWORK 4. O VERVIEW
This section presents the design and implementation of the Retriever framework, which realizes the bipartite graph representation of content and style as described in Section 3.
Retriever , as shown in Figure 3, is an autoencoder comprising of four major blocks, two of which implement the most important style retrieval functions.
This is why the name Retriever is coined.
Retriever is modal-agnostic.
The input of the framework is an array of input tokens, denoted by X =
[ x 1 , x 2 , .., x n ] ∈ R n × d , obtained by modal-speciﬁc tokenization operations from raw data.
The ﬁrst major component is a pre-processing module to extract the feature F ∈ R n × d from
X .
We implement it as a stack ( L e ) of non-P.I. transformer encoder blocks (Vaswani et al., 2017) to keep all the important information.
Then F is decomposed by the content encoder f B.N. ( · ) and the style encoder f P.I. ( · ) .
f B.N. ( X ) is implemented by vector quantization to extract content
C =
[ c 1 , c 2 , ..., c n ] .
Structural constraint can be imposed on C to improve interpretability.
A cross-attention module is employed as f P.I. ( X ) to retrieve style S
= { s 1 , s 2 , ..., s m } .
At last, the decoder, implemented by the novel link attention module, reconstructs X from the bipartite graph.
C r o ss - A tt e n t i o n q k v
T o k e n
M i x e r
FF N nxd mxd s mxd s
F
Z iZ i
A dd &
N o r m
A dd &
N o r m
A dd &
N o r m x L s
L i n k A tt e n t
i
o
n q k v S e l f - A tt e n t i o n
FF N nxd nxd Y iY
i
A dd &
N o r m
A dd &
N o r m
A dd &
N o r m x L d ( K, S
[Figure]
mxd s
[Figure]
l u
[Figure]
C ONTENT E NCODER
Content encoder should serve two purposes.
First, it should implement an information bottleneck to squeeze out the style information.
Second, it should provide interpretability such that meaningful feature editing can be enabled.
Information bottleneck.
Vector quantization (VQ) is a good candidate to implement information bottleneck (Wu et al., 2020), although many other choices can easily ﬁt into our framework as well.
VQ maps data from a continuous space into a set of discrete representations with restricted information capacity, and has shown strong generalization ability to multiple data formats, such as image (van den Oord et al., 2017; Razavi et al., 2019;
Esser et al., 2021), audio (Baevski et al., 2020a;b) and video (Yan et al., 2021).
We use product quantization (Baevski et al., 2020a;b), which represents the input F with a concatenation of G groups of codes, and each group of codes is from an independent codebook of V entries.
To encourage all the VQ codes to be equally used, we use a batch-level VQ perplexity loss, denoted as L V Q .
Please refer to Appendix A for more details.
When the bottleneck capacity is appropriate, the content can be extracted as C = VQ( F ) , where C ∈ R n × d c .
Interpretability.
Priori knowledge can be imposed on the content path of Retriever to increase the interpretability of feature representations.
In this work, we introduce structural constraint L SC to demonstrate this capability.
Other prior knowledge, including modal-speciﬁc ones, can be added to the framework as long as they can be translated into differentiable loss functions.
The structural constraint we have implemented in Retriever is a quite general one, as it reﬂects the locality bias widely exists in natural signals.
In image domain, we force the spatially adjacent tokens to share the same VQ code, so that a single VQ code may represent a meaningful object part.
In speech domain, we discourage the VQ code from changing too frequently along the temporal axis.
As such, adjacent speech tokens can share the same code that may represent a phoneme.
The visualization in the next section will demonstrate the interpretability of Retriever features.
4. S TYLE E NCODER WITH C ROSS -A TTENTION
We have deﬁned style as the P.I. information that can be extracted by a P.I. function from the structured data.
Cross-Attention, which is widely used in set prediction task (Carion et al., 2020) and set data processing such as 3D point cloud (Lee et al., 2019), is known to be a P.I. function.
It is also a powerful operation that can project a large input to a smaller or arbitrary target shape.
Previously,
Perceiver (Jaegle et al., 2021b) and Perceiver IO (Jaegle et al., 2021a) have used this operation as a replacement of the self-attention operation in Transformers to reduce the complexity.
However, cross-attention operation does not preserve data structure.
In order to compensate for this, additional positional embedding is implemented to associate the position information with each input element.
Conversely, our work takes advantage of cross-attention’s non-structure-preserving nature to retrieve the P.I. style information.
A key implementation detail is that position information should not be associated with the input tokens.
Otherwise, there will be content leaks into the style path.
To implement the style encoder, we follow previous work (Carion et al., 2020; Lee et al., 2019) to learn m dataset-shared prototypes Z 0 ∈ R m ×
d s as seed vectors.
Figure 4 (a) shows the implementation details of the style encoder, which is a stack of L s cross-attention blocks, token mixing layers, and feed-forward network (FFN) layers.
The ﬁnal output is the retrieved style S ∈ R m × d s .
4. D ECODER WITH L INK A TTENTION
In an autoencoder architecture, the decoder is responsible for reconstructing data from latent features.
In learning a content-style representation, the ability of the decoder limits the way content and style can be modeled.
Therefore, decoder design becomes the key to the entire framework design.
In order to support the bipartite graph representation of content and style, we innovatively design link attention by unleashing the full potential of multi-head attention.
Unlike self-attention and cross-attention, link attention has distinct input for Q , K , and V .
In the Retriever decoder, content tokens are queries and style tokens are values.
The keys are a set of learnable linking keys K ∈ R m × d s which are paired with the style tokens to represent the links in the content-style bipartite graph.
Such a design allows us to retrieve content-speciﬁc style for data reconstruction, enabling ﬁne-grained feature editing and interpretable style transfer.
Figure 4 (b) shows the implementation details of the Retriever decoder.
We ﬁrst use a convolutional layer directly after VQ to aggregate the contextual information as Y 0 =
Conv( C ) .
Then the style is injected into content for L d times.
In each round of style injection, we add a self-attention layer before the link attention layer and append an FFN layer after it.
The element order in K and
S does not matter, as long as k i and s i are paired.
The output of the decoder is then detokenized to reconstruct the raw data, whose difference with the input raw data is used as the reconstruction loss.
4. L OSS F UNCTIONS
Our ﬁnal loss function L sum consists of three components: the reconstruction loss L rec , the VQ diversity loss L V Q , and the structural constraint loss L SC (implementation details described in
Appendix D).
It is computed by:
[Equation]
where λ rec , λ V Q , and λ SC are hyper-parameters controlling the weights of the three losses.
E XPERIMENTS
We evaluate the Retriever framework in both speech and image domains.
Due to the differences in tasks, datasets, and evaluation metrics, we organize the experiments in these two domains in two subsections.
We use visualizations or quantitative results to: i) demonstrate the effect of contentstyle separation and the interpretability of content; ii) illustrate how the bipartite graph representation conveys the interpretability of content to style and supports ﬁne-grained style transfer.
5. S PEECH D OMAIN 5.1. T RAINING R ETRIEVER FOR S PEECH S IGNALS
Retriever for speech signals is trained with the entire 44-hour CSTR VCTK Corpus (Veaux et al., 2017) containing 109 speakers.
We use the pre-trained CPC network (Rivi`ere et al., 2020) followed by a depth-wise convolution to perform tokenization.
In Retriever , the content encoder is implemented by a VQ with two groups ( G = 2 ) and V = 100 .
A truncated neighborhood crossentropy loss is used as L SC to enforce the structural constraint, which is only applied to group #0 .
For the style encoder, we set the number of style tokens to 60.
At the decoder, the content is ﬁrst processed by a 1D depth-wise convolution with kernel size 31 before being passed to link attention.
In the detokenization step, we resample the output log-Mel spectrum to 80Hz and feed it into Parallel
WaveGAN vocoder (Yamamoto et al., 2020) to generate the waveform.
We apply L1 reconstruction loss on log-Mel spectrum as L rec .
More details can be found in Appendix F.1.
5.1. V ISUALIZATION OF C ONTENT -S TYLE R EPRESENTATION
To understand how Retriever models the content, the style, and the bipartite graph between them, we visualize the VQ codes and the decoder link attention map for a 2.5s-long test utterance in
Figure 5 (a).
Each dot represents one token and the token hop is 10ms.
Retriever encodes content with VQ, and we further impose structural constraint to VQ Group #0 .
We ﬁnd that this group sil sp
S
Z
IH N
M
NG
M
UW
N AA R AH
UH
F V EH DV AY
AO AW
IY UH UHER
W
AY EH AH
AA
AE
AW NG TH
K
G
ZHH
ZH ZH
P B IY
Y
EY
B
D
G sp sil
SH
CH JH
AE UH
S
Z
AW
EH AY
ER
R R
T OW L
AO AW OY
UH
EY
AY EH
IY
W
UWR
NG
M
N
OY
AO W
[Figure]
[Table] [Table]
of code
demonstrates good interpretability as desired.
Codes for adjacent tokens are highly consistent, and the changes of codes are well aligned with ground-truth phoneme changes.
The VQ codes of group #1 exhibit a different pattern.
They switch more frequently as they capture more detailed residual variations within each phoneme.
We further study the interpretability of content by phoneme information probing (implementation described in Appendix F.3), which is used to show the information captured by the discovered content units (Chorowski et al., 2019).
We measure the frame-level phoneme classiﬁcation accuracy with or without context and compare with the baseline (CPC + k-means).
Results in Table 1 show that the two groups of VQ codes discovered in our unsupervised framework align surprisingly well with the annotated phoneme labels.
Notably, group #0 codes capture most of the phoneme information and group #1 codes are less interpretable.
This demonstrates how the structural constraint helps to increase the interpretability of content.
In decoder link attention, all the content tokens within the same phoneme tend to attend to the same style vector, while tokens from different phonemes always attend to different style vectors.
We further visualize the co-occurrence map between the ground-truth phonemes and the style vectors in Figure 5 (b) (calculation described in Appendix B).
We see that the phonetic content and style vectors have a strong correlation.
Interestingly, each style vector encodes the style of a group of similarly pronounced phonemes, such as the group of ‘M’, ‘N’, and ‘NG’.
This observation conﬁrms that content-speciﬁc style is actually realized in Retriever features.
5.1. Z ERO -S HOT V OICE C ONVERSION T ASK
Zero-shot voice conversion task converts the source speaker’s voice into any out-of-training-data speaker’s while preserving the linguistic content.
We follow the setting of previous work to randomly select 1,000 test utterance pairs of 18 unseen speakers from the CMU Arctic databases (Kominek & Black, 2004).
Each conversion is given ﬁve target utterances to extract the style.
Both objective and subjective metrics are used to evaluate conversion similarity (SV accuracy, SMOS) and speech quality (MOSNet score, MOS).
More details of the four metrics are in Appendix F.2.
Existing approaches fall into two categories: content-style disentanglement approaches (Qian et al., 2019; Yuan et al., 2021; Chou & Lee, 2019; Wu et al., 2020), and deep concatenative approaches (Lin et al., 2021b;a).
Methods in the ﬁrst category use a shared style embedding for all the content features while methods in the second category try to ﬁnd the matching style for each content frame, resulting in high computational complexity.
Our method provides a similar level of style granularity as the second class of approaches.
But our bipartite graph representation is more compact and effective as it is capable of retrieving content-speciﬁc style with a light-weight link attention operation.
Table 2 shows the comparison between our Retriever-based method and four SOTA methods.
The superior performance demonstrates the power of the content-style bipartite graph representation.
Audio samples are provided in the supplementary material.
5.1. A BLATION S TUDY ON R ETRIEVER
Based on the zero-shot voice conversion task, we carry out ablation studies on the VCTK + CMU
Arctic dataset to better understand Retriever .
Style Encoder.
Unlike most previous work, which are limited to learning a single speaker style vector, our method can learn an arbitrary number of style tokens.
When the number of style tokens increases from 1 to 10 and 60, the SV accuracy increases from 81.3% to 94.3% and 99.4%, showing the beneﬁts of ﬁne-grained style modeling.
[Table]
Content Encoder.
We study how the capacity of the information bottleneck affects the separation of content and style.
Compared to the proper bottleneck setting ( G = 2 , V = 100 ), speech quality signiﬁcantly drops when the bottleneck is “too narrow” ( G = 1 , V = 10 ) and conversion similarity drops when the bottleneck is “too wide”( G = 8 , V = 100 ), as shown in Table 3.
This indicates content loss and style leakage when the bottleneck is too narrow and too wide, respectively.
Decoder.
To illustrate the advantages of our link attention-based decoder, we replace it with the traditional AdaIN module for experiment.
As the AdaIN module only takes a single vector as input, we ﬂatten the 60 style tokens produced by Retriever into a very long vector.
Results in Tableshow that, although the amount of information provided to the decoder is the same, AdaIN-based decoder has a signiﬁcant performance drop.
This conﬁrms the value of link attention.
5. I MAGE D OMAIN
The image domain is one of the ﬁrst domains to carry out the research on content-style disentanglement.
Conventionally, shape is treated as content and appearance is treated as style.
Interestingly, our uniﬁed deﬁnition of content and style aligns with the intuitive understanding of images.
5.2. T RAINING R ETRIEVER FOR I MAGES
To tokenize the image, we downsample it by 4 times using a stack of convolutions.
For VQ module, we set G to 1, and V is dependent on the dataset.
The convolution after VQ operation is of kernel size 3 × 3 .
We detokenize the output tokens to images by convolutions and PixelShufﬂes (Shi et al., 2016).
For the reconstruction loss L rec , we apply a typical perceptual loss (Chen & Koltun, 2017).
For the structural constraint L SC , we adopt a geometric concentration loss (Hung et al., 2019).
We choose two commonly used datasets: Celeba-Wild (Liu et al., 2015) and DeepFashion (Liu et al., 2016).
Images in Celeba-Wild are not aligned and each face has ﬁve landmark coordinates.
The full-body images from DeepFashion are used.
See Appendix E for more details and experiments.
5.2. U NSUPERVISED C O - PART S EGMENTATION
[Table]
Unsupervised co-part segmentation aims to discover and segment semantic parts for an object.
It indicates whether the content representations are semantically meaningful.
Existing methods (Hung et al., 2019; Liu et al., 2021) rely heavily on hand-crafted dataset-speciﬁc priors, such as background assumption and transformations.
We choose SCOPS (Hung et al., 2019) and Liu et al.
(2021) as baselines and follow the same setting to use landmark regression on Celeba-Wild dataset as a proxy task for evaluation.
Please refer to the Appendix E.2 for details.
The content code of Retriever is visualized in Figure 6 (a) for the Celeba-Wild dataset and in
Figure 8 for the DeepFashion dataset.
By setting a small V in Retriever , the encoded content only preserves the basic shape information as segmentation maps.
Each content token is consistently aligned with a meaningful part among different images.
When compared to the state-of-the-art
[Figure] [Figure]
methods, as Table 4 shows, Retriever is one of the top performers on this task, even without any hand-crafted transformations or additional saliency maps.
[Figure]
Part-level style transfer requires disentanglement of shape and appearance at part-level, which is particularly challenging in an unsupervised manner.
Previous works (Lorenz et al., 2019; Liu et al., 2021) propose to use one-one matching between shape and appearance, which is not ﬂexible and leads to unrealistic results.
Our content-style bipartite graph separates content and style at part-level and enables a more ﬂexible content-speciﬁc style modeling.
We can perform part-level style transfer by calculating the co-occurrence map between content and style tokens.
See Appendix B for more details.
Both image-level and part-level transfer results are shown in Figure 6, 7, and 8.
Retriever is capable of the explicit control of local appearance.
Even without any supervision for the highly unaligned image pairs, our method can transfer the appearance to a target shape with high visual quality.
See Appendix E.4 for more results.
C ONCLUSION
In this paper, we have designed and implemented an unsupervised framework for learning separable and interpretable content-style features.
At the core of the proposed Retriever framework are the two retrieval operations powered by the innovative use of multi-head attention.
Cross-attention is used as the style encoder to retrieve style from the input structured data, and link attention is used as the decoder to retrieve content-speciﬁc style for data reconstruction.
We have demonstrated that structural constraints can be integrated into our framework to improve the interpretability of the content.
This interpretability is further propagated to the style through the innovative bipartite graph representation.
As a result, the proposed Retriever enables a couple of ﬁne-grained downstream tasks and achieves superior performance.
As for the limitations, we have discovered in experiments that different tasks on different datasets need different model settings.
We currently lack theoretical guidance for determining these settings, and we plan to work on it in the future.
A B ACKGROUD
Attention.
Given n query vectors, each with dimension d q : Q ∈ R n × d q , an attention function
Attn( Q , K , V ) maps queries Q to outputs using m key-value pairs K ∈ R m ×
[Equation]
q , V ∈
[Equation]
m × d v ,
[Equation]
where dot product QK T ∈ R n × m measures the similarity between queries and keys
softmax( · ) normalizes this similarity matrix to serve as the weight of combining the m elements in V .
Multi-Head Attention.
Vaswani et al.
(2017) propose multi-head attention, denoted as MHA( · , · , · ) .
It projects Q , K , V into h different groups of query, key, and value vectors, and applies attention function on each group.
The output is a linear transformation of the concatenation of all attention outputs:
[Equation]
where W Qi , W Ki ∈ R d q × dqh , and W V i ∈ R
d v × dvh are the projection matrix for query, key, and value in the i -th head, W O ∈ R d v × d q is the linear transformation that fuses all the heads’ outputs.
Self-Attention.
Given an input array X ∈ R n × d x , a self-attention SA( · ) or multi-head self-attention
MHSA( · ) operation takes X as query, key, and value:
[Equation]
Cross-Attention.
Given input arrays X ∈ R n × d x , Y ∈ R m × d y , a cross-attention CA( · , · ) or multihead cross-attention MHCA( · , · ) operation takes X as query, and takes Y as key and value:
CA( X , Y
[Equation]
Property: (multi-head) cross-attention operation is P.I. to Y , that is,
[Equation]
where P m is the set of all permutations of indices { 1 , ..., m }
Permutation Invariant Property of Style Encoder For the i -th style encoder block, we assume its input Z i − 1 is P.I. to F .
The output of the multi-head cross-attention layer is ˆ Z i =
MHCA( Z i − 1 , F ) .
According to the property of multi-head cross-attention, we have ˆ Z i is permutation invariant to F .
The following residue connection combines ˆ Z i with Z i − 1 , both of which are P.I. to F .
Thus the combination is P.I. to F .
The following token mixing layer and feed-forward layer don’t take F as input.
Because their inputs are P.I. to F , their outputs are also P.I. to F .
To summarize, if Z i − 1 is P.I. to F , then Z i is also P.I. to F .
It is obvious that Z 0 is P.I. to F since Zis ﬁxed during the forward path.
Therefore, the output of the whole style encoder is P.I. to F .
Vector Quantization.
For efﬁciency, we use product quantization (Baevski et al., 2020a;b).
The quantized feature c i is the concatenation of G embeddings e 1 , e 2 , ..., e G ∈ R d c /G , which are looked up from the corresponding G codebooks E 1 , E 2 , ...,
E G ∈ R V
×
d c /G , where V is the number of entries in each codebook.
To make VQ operation differentiable, Gumbel-softmax (Jang et al., 2017) is adopted: Firstly, each input token is mapped to logits l ∈ R G × V .
Then, e g is calculated by a weighted sum of codewords in codebook E g , and the weight for adding the v -th code in group g is given by:
[Equation]
where τ is temperature, n g,v = − log( − log( u g,v )) , and u g,v ∼ U (0 , 1) is independently sampled for all subscripts.
We use the straight-through estimator (Jang et al., 2017), treating the forward and backward path differently.
During the forward path, the one-hot version of the above weight one hot (arg max k w g,k ) is used, and in the backward path, the gradient of the original w g,k is used.
To encourage all the VQ codes to be equally used, we use a batch-level VQ perplexity loss:
[Equation]
where p g indicates the probability calculated by softmax( l g ) averaged across all the tokens in batch,
H ( · ) calculates the entropy of a discrete distribution.
B V ISUALIZATION OF LEARNED C ONTENT -S TYLE B IPARTITE G RAPH
Retriever models the content-style bipartite graph by introducing the novel link attention module.
The graph’s edges are represented by the link attention map.
To understand what content is linked to which style, we gather the statistics of the link attention map as the content-style cooccurrence map.
To show content information in a 2-d map, the content is categorized by either the ground-truth labels, or the discovered parts or units.
Speciﬁcally, the co-occurrence statistics of content category c and style token s , denoted as m c,s , is calculated as follows:
[Equation]
where D indicates the set composed of all the content tokens in the target dataset.
A ( x, s ) is the link attention amplitude between content token x and style token s .
I ( · ) is the characteristic function.
It outputs 1 if its input is true and outputs 0 if its input is false.
l ( x ) gives the content category of token x .
For each style token, we consider the content category that shows the strongest co-occurrence with it to be the major content category that it serves.
For the convenience of ﬁnding the pattern inside the co-occurrence map, we normalize on both content category axis and style token axis, then sort the style tokens so that their major content category index is in the ascending order.
In audio domain, we categorize content tokens using ground-truth phoneme labels and calculate the co-occurrence map using the ﬁrst link attention.
The content-style co-occurrence map is shown in
Figure 9.
sil sp
S
Z
IH N
M
NG
M
UW
N
AA R AH
UH
F V EH DV
AY
AO
AW
IY
UH
UH ER
W
AY
EH AH
AA
AE
AW NG TH
K
G
ZHH
ZH ZH
P B IY
Y
EY
B
D
G sp sil
SH
CH
JH
AE UH
S
Z
AW
EH
AY
ER
R R
T OW
L
AO
AW OY
UH
EY
AY
EH
IY
W
UW R
NG
M
N
OY
AO W
P h o n
[Figure]
In image domain, we categorize content tokens using the discovered parts and calculate the cooccurrence map using the third link attention.
The part indices together with content-style cooccurrence map are shown in Figure 10.
C S TYLE S PACE V ISUALIZATION
In order to provide an intuitive impression about what the styles decomposed by our framework are, we present visualizations of some style spaces produced by t-SNE (Van der Maaten & Hinton, 2008). We see the clustered patterns in many style spaces which correspond to meaningful attributes like person gender and clothes color. C. S PEECH D OMAIN
We visualize the style vectors extracted from 500 randomly-selected utterances of ten unseen speakers from Librispeech test-clean split.
In Figure 11, we show the latent space of four typical style vectors, which correspond to the style of nasal consonant (NG), voiceless consonant (T), voiced consonant (Z), and vowel (IH).
Different speakers are labeled in different colors.
We see that different speakers are clustered into different clusters.
C. I MAGE D OMAIN
We examine the style vector spaces on Deepfashion dataset, and ﬁnd that some style vector spaces are separately or collectively correlated with the human-deﬁned attributes.
For example, we ﬁnd that the style vector #43 is highly correlated with the color of the clothes.
Figure 12 shows the t-SNE visualization of this vector space, as well as some images in each cluster.
It is clear that images are clustered solely based on the color of the clothes despite of various model identiﬁcations, standing poses, and styles of the dress.
As this dataset also has the gender attribute labelled, we further look for the ‘gender vector’ in the decomposed style vector spaces.
Interestingly, we ﬁnd 14 vector spaces that are highly correlated to the gender attribute.
We concatenate these 14 vectors and show the t-SNE visualization in Figure 13.
We can see a clear boundary between male and female models despite of the their color of clothes or standing poses.
D S TRUCTURAL C ONSTRAINT D. I MAGE DOMAIN
To make the content representation interpretable, we can apply additional structural constraint.
Inspired by the observation that pixels belonging to the same object part are usually spatially concentrated and form a connected region, we follow Hung et al.
(2019) to use a geometrical concentration loss as the structural constraint to shape the content representation to part segments.
In practice, we denote the pixels assigned to the ﬁrst entry in the VQ codebook as background pixels and the other pixels as the foreground pixels.
Given the logits L ∈ R V
×
H
×
W that assign each pixel to the parts (entries in VQ codebook), for foreground pixels:
[Equation]
where c kh and c kw are the coordinates of the v -th part center along axis h , w and z k is a normalization term.
c kh , c kw and z k can be calculated as follows:
[Equation]
For the background pixels, different from previous methods using background loss (Liu et al., 2021) or saliency map to bound the region of interested object, we do not add any constraint.
Thus L SC =
L fg
D. A UDIO DOMAIN
To discover phoneme-like units in speech, the discrete representation should not switch too frequently along the time axis.
Therefore, we penalize the code switch based on the calculation of neighborhood cross-entropy.
[Equation]
where N is the length of the input sequence, CE( · ) calculates the cross-entropy between discrete probability and given label, p i denotes the predicted discrete probability before Gumbel-softmax, and z i is the highest-probability code on position i .
However, L CE also penalizes the true phoneme switch, where the input acoustic feature really changes dramatically.
To solve this problem, we use a truncated version of L CE as our structural constraint:
[Equation]
where γ is a hyper-parameter indicating the truncation threshold.
For the true phoneme switch, high L CE is expected, thus the loss will be truncated, while within the same phoneme, low L CE is expected, and L SC behaves like the original CE loss.
As such, the truncated loss is more suitable for discovering phoneme-like units.
In our experiment, we set γ = ln( V ) = ln(100) .
E I MAGE : A DDITIONAL D ETAILS E. I MPLEMENTATION D ETAILS
We ﬁrst show the tokenization module in Table 5 and the detokenization module in Table 6.
For the layer numbers, we set L e , L s , L d to 6 , 3 , 6 , respectively.
The hyper-parameter settings of each styleencoder layer and decoder layer are shown in Table 7 and Table 8, where d FFN indicates the hidden dimension of the feed-forward network, and n head is the number of heads used in the multi-head attention modules.
Speciﬁcally, for the decoder, we replace the FFN with Mix-FFN to improve the image quality.
The Mix-FFN can be formulated as:
[Equation]
where x in is the input feature.
The Gumbel-Softmax temperature τ in VQ is annealed from 2 to a minimum of 0.01 by a factor of 0.9996 at every update.
For the Celeba-Wild dataset, we resize the input to 128 × 128 .
For DeepFashion dataset, we resize the input to 192 × 96 .
For the number of style tokens, we use 512 .
We summarize the training setting in Table 9.
Our model is implemented with Pytorch and trained on 4 Nvidia V100 GPUs
[Table] [Table]
[Table] [Table] [Table]
[Table]
E. D ETAILS ABOUT E VALUATION ON C O - PART S EGMENTATION
For the evaluation on co-part segmentation, we follow the setting in Liu et al.
and Hung et al.
The proxy task for Celeba-Wild is landmark regression.
We ﬁrst convert part segmentations into landmarks by taking part centers as in Eq.
Then we learn a linear regressor to map the converted landmarks to ground-truth landmarks and evaluate the regression error on test data. E. A BLATION S TUDY
[Table]
Do the style encoder and decoder impact the co-part segmentation?
In this work, we ﬁnd VQ module is suitable for unsupervised co-part segmentation, as it is a cluster center among the whole dataset.
Here we evaluate that if the style branch further helps the co-part segmentation.
As shown in Table 10, with a style branch, the discovered part segmentation is more consistent.
Moreover, we compare our link attention decoder with AdaIN operation and ﬁnd that our design is better for co-part segmentation task.
This further veriﬁed that our content-style bipartite graph is a powerful representation with high interpretability.
On using different structural constraint weight.
We further carry out experiments with different loss weight for structural constraint.
Without the structural constraint, the parts learned are scattered all over an image.
As the weight for structural constraint getting larger, the part segmentation becomes tighter.
With too large weight, the regions lose ﬂexibility and fail to cover the parts.
On using different VQ diversity loss weight.
We also test our model’s sensitivity to λ vq .
As long as λ vq is not set to zero or too large (e.g., λ vq = 1), the model has a fairly stable landmark regression error even when λ vq is adjusted from 0.01 to 0.3.
[Figure]
On using different tokenization module.
Different tokenization modules bring additional bias.
Besides a stack of convolution layers to tokenize the input image, we also experiment with the
PatchMerge operation in ViT (Dosovitskiy et al., 2021), which is a 4 × 4 convolution layer with stride 4 in our case.
As shown in Table 10, using convolution layers as the tokenization module can help the preprocessing transformer blocks inside Retriever work better, which is also observed by a concurrent work (Xiao et al., 2021). [Figure]
On using different downsampling rate.
In this work, we mainly use 4 × downsampling rate.
For the input 128 × 128 image, we ﬁrst tokenize it to 32 × 32 .
Here we experiment with additional different downsampling rates: 8 × and 16 × .
In addition to the convolution layer for 4 × setting, we add additional convolution layers for further downsampling.
We do not try 2 × due to the limited computational resource.
As shown in Table 10, for the selected downsampling rate, larger one results in worse performance.
We also provide visual comparison in Figure
[Figure]
E. M ORE QUALITATIVE RESULTS
Co-part segmentation.
We ﬁrst provide more visualization on Celeba-Wild in Figure 17.
Besides, we also apply our method on CUB dataset (Wah et al., 2011).
CUB is a dataset with 11, 788 photos of 200 bird species in total.
This dataset is challenging due to the pose of the birds is of large diversity.
We follow (Liu et al., 2021) select the photos from the ﬁrst 3 categories, and get aboutphotos.
The results are shown in Figure 18.
Part-level style transfer.
Besides the results in the main paper, we also apply our method on a high-resolution dataset: CelebA-HQ.
We resize the images to 256 × 256 .
We show the part-level style transfer results in Figure 19 for mouth, nose, and eye.
Zero-shot image style transfer. We ﬁnd our model capable of generalizing into unseen image datasets. In Figure 20, we use our model trained on CelebA-Wild to transfer artistic style from an artistic image. The transfer result keeps the pose and shape of the content images (the ﬁrst row), while adopting the tone and appearance of the style image (the ﬁrst column on the left).
Visual comparison.
We ﬁnd our method enables more natural style transfer compared with the previous work.
Here we show side-by-side visual comparison on Deepfashion dataset in Figure 21 and 22.
In image-level style transfer, the results of Lorenz et al.
(2019) suffer from signiﬁcant deformation artifact.
In contrast, such artifact is not observed in our results.
In part-level style transfer, we show the results of head transfer.
Again, our results are more natural and contain more image details.
[Figure]
[Figure]
F A UDIO : A DDITIONAL D ETAILS F. I MPLEMENTATION DETAILS
For the tokenization module, we follow Lin et al.
(2021a), using s3prl toolkit 1 to extract the CPC feature, and use a depth-wise convolutional layer with kernel size 15 after the CPC feature.
The depth-wise convolutional layer is trainable, while the CPC model is ﬁxed during training.
For the detokenization module, we use publicly available implementation of Parallel WaveGAN 2 pretrained on LibriTTS.
L e , L s , L d are set to 0, 3, 4, respectively.
The dimension of content representation is d c = 512 .
Following Qian et al.
(2020), we add auxiliary quantized normalized f 0 feature into content for better voice quality.
The Gumbel-Softmax temperature τ is annealed from 2 to a minimum of 0.01 by a factor of 0.9996 at every update.
The hyper-parameter settings of each style-encoder layer and decoder layer are shown in Table 11 and Table 12, where d FFN indicates the hidden dimension of the feed-forward network, and n head is the number of heads used in the multi-head attention modules.
The decoder output is converted to 80-d log-Mel spectrum with an MLP whose hidden dimension is 4096.
The training hyper-parameter setting is listed in Table 13.
Each training sample is a random 4-second audio clip sampled from the training dataset.
For utterances that are shorter than 4s, we pad zero at the end of the utterance.
The code is implemented with Pytorch, and training takes 5 hours on 4 Nvidia V100 GPUs.
[Table] [Table]
[Table]
F. E VALUATION METRICS OF ZERO - SHOT VC TASK
For objective metrics, conversion similarity is measured by Resemblyzer speaker veriﬁcation systemby calculating speaker veriﬁcation accuracy (SV Accu) between converted speech and corresponding target utterance, as done in the previous work (Lin et al., 2021b).
The conversion is considered successful if the cosine similarity between Resemblyzer speaker embedding of converted speech and target utterance exceeds a pre-deﬁned threshold.
The threshold is decided based on equal error rate (EER) of the SV system over the whole testing dataset.
The SV accuracy is the percentage of successful conversions.
The objective speech quality metric is estimated by MOSNet (Lo et al., 2019).
It takes converted speech as input and outputs a number ranging from 1 to 5 as the measurement of speech naturalness.
Both metrics are higher the better.
For subjective metrics, we conduct two tests to calculate the mean opinion score of conversion similarity (denoted as SMOS), and speech quality (denoted as MOS).
For conversion similarity, each subject is asked to listen to the target utterance and converted utterance, and judge how conﬁdent they are spoken by the same speaker.
For speech quality, each subject is asked to listen to utterances that are randomly chosen from converted speech and real speech, and determine how natural they sound.
Both test results are scored from 1 to 5, the higher, the better.
We randomly sampleutterances from the test set for both tests.
Each utterance is evaluated by at least 5 subjects.
The score is then averaged and reported with 95% conﬁdence intervals.
F. P HONEME INFORMATION PROBING : TRAINING AND TESTING
We conduct phoneme information probing to test how much phoneme information is encoded in the learned discrete content tokens, so as to demonstrate their interpretability.
We experiment with two settings, one considers the contextual information (denoted as “Frame w/ context”), and the other considers only single-frame information (denoted as “Frame”).
For these two settings, a 1d convolutional layer with kernel size 17 or a linear layer is used as the probing network respectively, to predict ground-truth phoneme label.
For the input of the probing network, each frame is represented by the one-hot vector of the corresponding code.
For the experiment involving two groups of code, the one-hot vector of both groups are concatenated on the channel axis.
For the k-means clustering of CPC feature, the cluster number is set to 100, which is the same as the number of codebook entries per VQ group.
We train the probing network on LibriSpeech train-clean-100 split and test it on LibriSpeech test-clean split.
Adam optimizer is used with β 1 = 0 .
9 , β 2 = 0 .
Learning rate and batch size are set to 0.00005 and 30, respectively.
Each training and testing sample is a random 2-second segment from the dataset.
We drop the utterance that is shorter than 2 seconds.
In Table 1, we report the test accuracy after the convergence of the training.
F. D ETAILS IN SYSTEM COMPARISON
We choose four SOTA systems for comparison.
Two content-style disentanglement approaches are AutoVC (Qian et al., 2019) 4 , and
AdaIN-VC
(Chou & Lee, 2019) 5 .
Two deep concatenative approaches are FragmentVC (Lin et al., 2021b) 6 , and S2VC (Lin et al., 2021a) 7 .
All these methods have ofﬁcially released their code, together with the pretrained models also on VCTK dataset.
We adopt these models and test them on our randomly sampled 1000 conversion pairs in CMU Arctic dataset.
F. I NFERENCE SCALABILITY ON ZERO - SHOT VC TASK
[Table]
Table 14 shows the inference scalability of our method on VCTK + CMU Arctic dataset.
“# target” indicates the number of available target utterances for inference.
We see that
Retriever performs quite well in the case that only one target utterance is given.
As the available target utterance increases, the conversion similarity keeps increasing up to 99.6%, and it has no negative effect on speech quality, indicating the extracted style becomes more accurate when seeing more target samples.
F. P ERFORMANCE ON L IBRI S PEECH DATASET
LibriSpeech dataset (Panayotov et al., 2015) is more diverse than VCTK + CMU Arctic in the aspect of emotion, vocabulary, and speaker number, thus closer to real-world application.
In this experiment, the whole train-clean-100 split containing 251 speakers is used for training, and testclean split containing 40 speakers is used for testing.
When testing, the conversion source-target pairs are built as follows: for each utterance in the test set, we treat it as the source utterance, and assign it one target utterance that is randomly sampled from the test set.
In this way, each utterance in the dataset serves as the source for one time.
Objective similarity metric is measured for using different style token numbers, as shown in Figure 23.
The conversion similarity increases when using more style tokens, and the performance saturates at nearly 100%.
To be speciﬁc, we achieve 98.4% SV accuracy and 3.13 MOSNet score when using 60 style tokens, demonstrating that our method can be generalized into more complicated scenarios and potentially can be used in the real-world application. [Figure] [Table]
To test the effectiveness of structural constraint and VQ diversity loss
[Table]
we do ablation study on the corresponding loss weights, λ V Q and λ SC .
The SV accuracy and MOSNet scores achieved at different parameter settings are shown in Table 15.
We observe that the absence of either loss term leads to noticeable speech quality (MOSNet score) drop.
We further test the model’s sensitivity to these two loss weights and we are able to empirically conclude that our framework is not sensitive to the selection of these two hyper-parameters.
When λ SC increases from 0.002 to 0.1 (by 50 times),
SV accuracy only ﬂuctuates by ± 0 .
45% and MOSNet score only ﬂuctuates by ± 0 .
When λ V Q increases from 0.01 to 2.0 (by 200 times), SV accuracy only ﬂuctuates by ± 0 .
05% and MOSNet score only ﬂuctuates by ± 0 .
In a nutshell, both structural constraint and VQ diversity loss are necessary, and our model is not sensitive to the selection of the corresponding loss weights in a reasonably large range.