Visual Concepts Tokenization
Anonymous Author(s)
AffiliationAddress email
Abstract
Obtaining the human-like perception ability of abstracting visual concepts fromconcrete pixels has always been a fundamental and important target in machinelearning research fields such as disentangled representation learning and scenedecomposition.
Towards this goal, we propose an unsupervised transformer-based
Visual Concepts Tokenization framework, dubbed VCT, to perceive an image intoa set of disentangled visual concept tokens, with each concept token respondingto one type of independent visual concept.
Particularly, to obtain these concepttokens, we only use cross-attention to extract visual information from the imagetokens layer by layer without self-attention between concept tokens, preventinginformation leakage across concept tokens.
We further propose a Concept Disentangling Loss to facilitate that different concept tokens represent independent visualconcepts.
The cross-attention and disentangling loss play the role of induction andmutual exclusion for the concept tokens, respectively.
Extensive experiments onseveral popular datasets verify the effectiveness of VCT on the tasks of disentangled representation learning and scene decomposition.
VCT achieves the state ofthe art results by a large margin.
Introduction
Despite the remarkable success of deep learning in various vision tasks, such as classification [21, 11]
detection [35, 7], segmentation [32, 40], it still suffers from the requirement of a tremendous amountof training data [44], low robustness and generalization [43, 19] and lack of interpretability [1].
Thosetraditional vision tasks learn the visual concepts, such as semantics and object localization, fromartificially predefined guidance.
On the contrary, human is capable of extracting abstract conceptsfrom concrete visual signals, then using those visual concepts to understand and depict the worldcomprehensively.
Towards learning visual concepts from observations, Bengio et al.
[4] proposedisentangled representation learning to discover the visual concepts as the explanatory factors hiddenin the observed data.
To achieve disentangled representation, the following works, based on VAE[22, 5, 28, 27, 8] or GAN [9], rely on probabilistic-based regularization of the latent space.
However,it has been fundamentally proved to be impossible if only relying on probability constrains [30].
Introducing inductive bias is necessary to solve the identifiability issue.
Besides, those methods oftenfail in complex scenes with multiple objects.
To address those complex scenes, another branch oflearning visual concepts aims to spatially decompose a scene image into different object regionsrepresented as different segmentation masks [18, 6].
Typically, these methods rely on explicit spatialdecomposition and can not learn global visual concepts.
In this paper, we are particularly interestedin finding a general way to learn visual concepts from pixels, which covers the aforementioned twobranches, disentangled representation learning and scene decomposition.
Pos
ConceptPrototypes
I m a g e T
o k
e n i z e r
ImageTokens
ConceptTokens
ImageQueries
I m a g e
D e t o k e n i
z e r
ImageTokens
S e l f - A tt e n t i o n
Q ×𝐿 𝐸
Concept Tokenizer
A dd & N o r m
FF N
A dd & N o r m
C r o ss - A tt e n t i o n
A dd & N o r m
FF N
A dd & N o r m
K,V
Pos
K,V ×𝐿 𝐷
Q C r o ss - A tt e n t
i o n
A dd & N o r m
FF N
A dd & N o r m
S e l f - A tt e n t i o n
A dd & N o r m
FF N
A dd & N o r m
S e l f - A tt e n t i o n
A dd & N o r m
FF N
A dd & N o r m
Concept Detokenizer
Originated from Natural Language Processing (NLP), tokenization means the process of demarcatinga string of input characters into sections named tokens.
For the visual signal, we propose anunsupervised transformer-based approach, Visual Concept Tokenization (VCT), to extract the visualconcept inside a given image as a set of tokens, serving as a general solution for visual conceptlearning.
For example, given an image of a 3D scene [27], our VCT can represent it into the objectshape token, object color token, object scale token, background color token, floor color token and posetoken.
Before introducing the details of VCT, we would like first to discuss the requirements of thetokens to achieve a good representation of the visual concepts contained in an image.
Those tokensshould satisfy the following three conditions: ( i ) completeness , one can reconstruct the image withthose tokens; ( ii ) disentanglement , different tokens should represent independent visual concepts,and each token should only reflect one kind of visual concept variation; ( iii ) disorder , consideringthe disordered nature of concepts, the ranking order of tokens should not carry any information.
Werefer to the tokens as concept tokens if they satisfy the above three conditions.
Note that the concepttokens are different from image tokens, i.e., the input of vision transformers, which are simply thegrid feature obtained by a CNN backbone [23] or patch features from a linear projection layer [11].
Those image tokens are typically entangled in terms of abstract information and positionally sensitive.
To obtain concept tokens from images, we build a novel transformer-based architecture, as shownin Figure 1, consisting of a Concept Tokenizer and a Concept Detokenizer.
The Concept Tokenizerabstracts visual concepts from image tokens, and the Concept Detokenizer reconstructs image tokensto meet the completeness condition.
In the Concept Tokenizer, using learnable concept prototypes asquery, we use cross-attention to induct information from image tokens layer by layer independently,without any interference between tokens of concept part, to meet the disentanglement requirement.
Itis worth noticing that, although the concept prototypes are dataset-level shared, different from otherworks containing learnable dataset-level queries with interaction (self-attention), such as DETR [7],
Perceiver [24], Retriever [42], there is no self-attention across our concept queries or tokens.
Finally,considering disorder, since there is no interference between the concept tokens, the order of the tokenscarries no information in the tokenization process.
In addition, no position embedding is added toconcept tokens to maintain this disorder nature for the Concept Detokenizer.
To further facilitate the disentanglement of concept tokens, inspired by DisCo [36] which achievesdisentanglement via contrasting the visual variations produced by a pretrained generative model,we propose a Concept Disentangling Loss to encourage the mutual exclusivity between the visualvariations caused by modifying different concept tokens.
Specifically, we modify a concept tokenand result in image variation.
After feeding to the Concept Tokenizer, by minimizing the Concept
Disentangling Loss, the image variation should only affect the modified concept token.
We verify the effectiveness of VCT on disentanglement and scene decomposition tasks, and bothachieve state-of-the-art performance with a significant improvement.
Surprisingly, we find that visualconcept tokens are well aligned with language representations by simply adopting the CLIP [33]image encoder as the image tokenizer.
Therefore, we can control the image content using text input.
Our main contributions can be summarized as:
• We present a general solution to extract visual concepts from concrete pixels, which canachieve disentangled representation learning and scene decomposition.•
We build an unsupervised framework, including Concept Tokenizer and Detokenizer, torepresent an image into a set of tokens, and each token reflects a visual concept.•
We propose a Concept Disentangling Loss to facilitate the mutual exclusivity of the visual
concept tokens.
Related Works
Image Tokenization For visual inputs, the previous methods to get image tokens can be dividedinto region-based, grid-based and patch-based.
For region-based, each token responds to an objectregion predicted by a detector [2].
For grid-based, each token is a spatially 1 × 1 vector taking fromthe extracted feature of a CNN backbone [23].
For the patch-based, the image is first divided intopatches, and each token corresponds to the feature extracted from one patch [11].
Our visual concepttokenization is different from the previous works due to it is the process of extracting high-levelvisual concepts from the image tokens produced by previous image tokenization methods.
We canadopt the grid-based or patch-based image tokenization as the module before concept tokenization.
The region-based one needs an extra detector to get object-level semantics, which is redundant andnot desired in our framework.
We refer to the reverse process of image tokenization and concepttokenization as image detokenization and concept detokenization, respectively.
Disentangled Representation Learning Disentangled representation learning is introduced in
Bengio et al.
[4], which aims to discover the hidden explanatory factors hidden in the observeddata.
Each dimension of the disentangled representation corresponds to one independent factor.
Following that, there are some VAE-based works that achieve disentanglement [22, 5, 28, 27, 8]by relying on probability-based regularizations on the latent space.
Locatello et al.
[30] prove thatonly these regularizations are not enough for disentanglement, and extra conductive bias on modeland data is required.
Following works explore an alternative way for disentangled representation,including leveraging pretrained generative model [36] or symmetry properties modeled with grouptheory [41].
To the best of our knowledge, our VCT is the first Transformer based framework,which takes advantage of the cross attention mechanism to discover the factor as well as forbid theentanglement across the concept tokens.
This specific architecture serves as network inductive bias.
VCT significantly improves the disentanglement ability, especially on some challenging datasets.
Scene Decomposition Scene decomposition, also referred to as object-centric representation, aimsto decompose a scene into objects, each with spatial segmentation masks [6].
Different from previousdisentangled representation learning methods, which typically only consider one main object, thevisual concepts on scene decomposition tasks focus on the object level representation.
One canachieve scene segmentation with explicit pixel-level supervision [32, 40].
To get rid of the supervision,
MONet [6] proposes an attention network to infer the mask and representation of each object froman image in an autoregressive way.
Locatello et al.
[31] propose a slot attention module to insertthe image feature, which is extracted by a CNN backbone, into slots.
This process is similar butfundamentally different from the cross-attention [3] we used for concept tokens.
The key is that slotattention applies a softmax normalization in the slot dimension.
The slots compete and interact witheach other when including the image feature.
The previous works need an explicit pixel-level maskspecifically designed for the decomposition task.
However, our concept token representation is ageneral and abstract high-level visual representation, and we would like to show that without anydedicated design, VCT can achieve scene decomposition and represent each object as a concept token.
We also provide a solution to get a pixel-level mask in the experiment.
Some other works explore theloss functions, such as energy-based loss [12] and contrastive loss [3].
Our Concept Disentangling
Loss is based on the manipulating concept tokens, and is specifically designed for VCT.
Visual Concept Tokenization
Given an image set { x i } , our target is to represent each image x
i as a set of M concept tokens of
D dimension C i ∈ R M × D , where each row of C i represents a concept token, and each token canreflect only one visual concept contained in the image set, such as object color red, background colorgreen, object shape cube.
We first present an overview of VCT, then introduce the Concept Tokenizer,
Concept Detokenizer, and Concept Disentangling Loss sequentially.
3.1 Overview of VCT
The overview framework of VCT is shown in Figure 1.
An image x i is first be tokenized into anumber of N image tokens, Z i ∈ R N × D , using an image tokenizer I T , where each row of Z i is animage token.
The image tokenizer I T could be a pre-trained or randomly initialized module, suchas a pretrained VQ-VAE [38] encoder, CLIP [33] image encoder, CNN-based encoder.
Then eachimage token is added with a positional embedding to include 2D spatial information in the sameway as [24].
Then a Concept Tokenizer V T , as a key component of VCT, takes the image tokens Z ias input, as well as a set of M concept prototypes P ∈ R M × D .
P is learnable and dataset-shared,and each row of P is a prototype of concept attribute, such as object color, background color, objectshape.
With prototype P , Concept Tokenizer V T extracts the visual concepts from image tokens Z i ,resulting in the set of concept tokens
C i
= V T ( P, Z i ) .
For example, the object in two images x i andx j are red and green.
p 1 (first row of P ) is the color prototype.
The corresponding extracted concepttokens c 1 i (first row of C i ) and c 1 j (first row of C j ) represent the red and green colors, respectively.
Then the concept tokens C i are fed into a Concept Detokenizer V D .
The Concept Detokenizer V Dreconstructs the image tokens
Z i , with an extra input of dataset-shared learnable image queries
Y ∈ R N × D .
The reconstructed image tokens are decoded into pixels via an image detokenizer I D ,which is an inversing module of image tokenizer I T .
For example, one can choose a pre-trained
VQ-VAE encoder and decoder for the image tokenizer and detokenizer, respectively.
3.2 Concept Tokenizer
Given an image x i , the Concept Tokenizer V T extracts the concept tokens C i from the image tokens
Z i and prototypes P .
The detailed implementation of Concept Tokenizer V T is shown in Figure 1.
To satisfy the aforementioned requirements for visual concept representation in Section 1, we adopttwo different types of attention for the image tokens and concept prototype tokens, respectively.
Specifically, for the image tokens, we adopt a standard Transformer layer using self-attention toprocess image tokens, followed by a feed-forward network (FFN) layer.
For the concept part, toinduct information from image tokens and prevent interference between concept tokens, we adoptcross-attention ( Q, K, V ) without following self-attention.
In the cross-attention operation,
Q is the tokens from the concept part, K and V are the tokens from the image part.
For thefirst layer, the cross-attention block takes concept prototypes P as Q , and image tokens
Z i as
K and V , resulting in cross-attention ( P, Z i , Z i ) .
Each cross-attention block is followed byan FFN.
To provide pathways for local and global visual concepts flexibly, we use a stack of
L E layers of the aforementioned self-attention & FFN, cross-attention & FFN layers to extractvisual concepts from image tokens layer by layer.
It is worth noting that the tokens in the conceptpart are only used as queries, and there is no interaction between those queries.
Consequentially,the encoding process of each token is independent:
c ji = V T ( P, Z i ) j = V T ( p j , Z i ) .
Note that
V T ( π ( P ) , Z i ) =
π ( V T ( P, Z i )) , where π is shuffle function, disorder nature is satisfied for Concept
Tokenizer.
This is the key to ensuring disentanglement, serving as a network inductive bias.
3.3 Concept Detokenizer
Given the concept tokens C i , the Concept Detokenizer V D reconstructs the image tokens Z i .
Thedetailed implementation of Concept Detokenizer is shown in Figure 1.
As a reverse procedureof Concept Tokenizer, for the Concept Detokenizer we follow a symmetry design of the Concept
Tokenizer.
Similar to the concept prototypes P used to query concept tokens in the Concept Tokenizer,
[Figure] [Figure]
[Figure]
Detokenizing & Tokenizing a a a b b b −
Concepts
DisentanglingLoss
[Figure]
3.4 Concept Disentangling Loss
The Concept Tokenizer structure can ensure there is no interference between the concept tokens.
Weuse the Concept Disentangling Loss to encourage the mutual exclusivity of concept tokens.
Generally,the Concept Disentangling Loss is a cross-entropy loss on the image variation caused by manipulationsof the concept tokens.
We demonstrate the process of calculating Concept Disentangling Loss in
Figure 2.
The two steps are: ( i ) producing image variations by substituting one concept token, ( ii )identifying the image variations.
Firstly, we introduce the process of producing image variations.
Weintroduce image variation by replacing specific concept token.
Specifically, we introduce the detailedimplementation given a batch of images { x i } B .
For image x i , we randomly select another imagex j , j ̸ = i .
The related two sets of concept tokens are C i and C j , respectively.
We randomly select anindex l, 0 ≤ l < K , and replace c li (the l -th token of C i ) with c lj , resulting in ˆ C i .
Then we decode C iand
ˆ C
i to images via Concept Detokenizer V D and image detokenizer I D sequentially.
The imagevariation between the reconstructed image pair x ′ i =
I D ( V D ( C i )) and ˆ x ′ i
= I D ( V D ( ˆ C i )) is causedby the aforementioned replacing operation ( c li → c lj ).
The next step is to identify which concepttoken is replaced.
We first encode the images x ′ i and ˆ x ′ i to get concept tokens via image tokenizer I Tand Concept Tokenizer V T .
Then the concept token variation is
[Equation]
The Concept Disentangling Loss is
[Equation]
where norm (∆ C ) means to calculate the ℓ 2 norm for each row, i.e., the norm of the variation ofeach token, resulting in a vector of K dimension.
l is the ground truth.
It is a one-hot vector of Kdimension, with the replaced dimension set to 1.
To prevent disentangling loss from sacrificing thereconstruction quality, we only optimize disentangling loss w.r.t.
Concept Tokenizer.
3.5 Total Loss
Besides the aforementioned Concept Disentangling Loss, we also need the reconstruction loss L rec toreconstruct the image tokens.
We can choose the formulation of L rec according to the choice of image
[Figure]
tokenizer and detokenizer.
For example, when using VQ-VAE, we can predict the quantized label ofeach reconstructed image token and use cross-entropy between the predicted label and ground truthlabel as the reconstruction loss.
For other image tokenizer and detokenizer, we adopt MSE betweenthe image and reconstructed image as reconstruction loss.
Together with the Concept Disentangling
Loss L dis , the total loss is L
=
L rec + λ dis L dis , where λ dis is the hyper-parameter.
We set λ dis =
1and adopt VQ-VAE for L rec in all the experiments.
Experiments
[Figure]
4.1 Disentanglement Results
In order to verify the disentanglement of the learned visual concepts of our framework, we conductexperiments on the task of disentangled representation learning.
Datasets Following [36], we conduct the experiments on the public datasets below, which are popularin disentangled representation literature: Shapes3D [27] is a dataset of 3D shapes generated from 6factors of variation.
MPI3D [17] is a 3D dataset recorded in a controlled environment, defined by 7factors of variation, and Cars3D [34] is a dataset of CAD models generated by color renderings from3 factors of variation.
We follow the literature to resize images to 64x64 resolution.
Baselines & Metrics The baselines contain four different types:
The VAE-based baselines are
FactorVAE [27], and β -TCVAE [8].
The GAN-based baseline is InfoGAN-CR [29].
For pre-trained
GAN-based baselines
, we adopt GANspace (GS) [20], LatentDiscovery (LD) [39], ClosedForm(CF)
[37], DeepSpectral (DS) [26] and DisCo [36].
For the concept-based method, we use Energy
Concepts (COMET) [13] as our baseline.
We follow [30] to conduct our experiments with differentrandom seeds.
We have 25 runs for each method.
Four popular and representative metrics are used inour experiments: FactorVAE score [27], the DCI [14], β -VAE score [22], and MIG [8].
However,since concept-based disentangled representations are vector-wise, we follow [13] to perform PCA aspost-processing on the representation and evaluate the performance with these metrics.
Quantitative Results Table 1 shows the comparison between VCT and other SOTA methods underdifferent disentanglement metrics.
VCT achieves superior performance with a large margin, whichdemonstrates the disentanglement ability of our framework.
The baseline methods include fourcategories.
The VAE-based and GAN-based methods suffer from the traded-off between generationand disentanglement [36].
The pre-trained GAN-based methods are limited by the latent space of
GAN.
COMET adopts energy-based modeling, which is more suitable for composition variations,e.g., scene decomposition.
However, our method does not have these limitations.
In addition, sinceour disentanglement is conducted in a learned space, which reduces the difficulty of the problem.
Qualitative Results Different from dimension-wise disentanglement methods, concept token isvector-wise disentanglement.
Therefore, we can not do the latent traversals as the dimension-wisedisentanglement methods did.
We swap the concept tokens of different images instead.
As Figure3 shows, VCT is capable of learning pure factors.
Note that our framework achieves not only
(mean ± std, higher is better).
VCT achieves the state of the art performance with a large margin in almost all the cases compared to all of the baselines.
Especially on the MPI3D dataset.
For MIG and
BetaVAE metrics and other details of experiments, please see Appendix A
[Table]
pure disentanglement but also has high-quality reconstruction.
For results on real-world datasets
CelebA/MSCOCO/KITTI, please see Appendix B. Besides, VCT can combine with pretrained GANsfor image editing (details are presented in Appendix B).
4.2 Ablation Study
We conduct the ablation study on disentangled representation learning of Shapes3D from the followingfive aspects.
For ease of conducting experiments, we ensure 15 runs for each setting
[Table] [Table]
Image Tokenizer Analysis Since VCT conductsdisentanglement on the latent space of an autoencoder, the choice of initialization of encoder influence the performance of the framework.
Westudy the following choices of encoder: randomly initialized naive autoencoder (AE), pretrained autoencoder (pre-trained AE), and pretrained VQ-VAE.
The usage of pre-trained VQVAE reduced the difficulty of the problem.
Table2 presents the results of different types of imagetokenizers.
The results show that the model withpre-trained VQ performs best as expected.
In addition, VCT still works using a randomly initialized autoencoder, which demonstrates that VCTdoes not rely on the pre-trained models.
Besides, VCT can still disentangle the latent spacethat is highly entangled (a pre-trained naive autoencoder).
Even with the Patch tokenizer in
ViT [10], VCT still works.
Concept Disentangling Loss
We study the effectiveness of Concept Disentangling Loss by removing it, denoted as “wo L dis .”
The results in
Table 2 demonstrate that even without Concept Disentangling Loss, our framework still can learn adisentangled representation to a considerable extent, no matter when AE or VQ-VAE is used.
Concept Tokenizer As we stated in Section 3, the interference between concepts is vital for VCT.
Inorder to verify it, we add a self-attention after each cross-attention block in Concept Tokenizer, whichis denoted as “w/ self-attention.”
Table 2 demonstrate that the model catastrophic fails.
In addition,
[Figure]
to verify that only optimizing Concept Tokenizer w.r.t L dis is effective, we train VCT without stopgradient when computing L dis and the performance significant drops, which is denoted as “wodetach.”
Finally, since Concept Tokenizer already satisfied the disorder requirement, adding positionembedding on our framework (“w/ pos embedding”) only has little influence on the performance.
Concept Detokenizer In order to verify the effectiveness of the Concept Detokenizer design, weuse a CNN and transformer (self-attention) to replace our Concept Detokenizer, we find that theperformance drops significantly, especially on the CNN decoder.
We posit the reason is that thenon-symmetric architecture and directly decoding from concept tokens make VCT less disentangled.
Sensitive Analysis
The batch size and the number of concepts affect the sample diversity of computing
L dis .
We also explore of training with different batch sizes and different numbers of concepts.
Asshown in Table 2, we see that the batch size and concept number slightly influence VCT on thecondition that they are larger than the number of ground truth factors (see details in appendix B).
Notethat VCT works well with a small batch size (32), and the performance only slightly drops even witha batch size of 16.
Since we set λ dis = 1 for all experiments, VCT is robust to the hyper-parameters.
4.3 Scene Decomposition
Next, in this section, we verify the ability of VCT to decompose a scene into object-level representations.
Note that our requirements of concept tokens are applicable to both object-level and factor-levelrepresentations.
Therefore, which kind of concept VCT learns is determined by data.
Different from
COMET [13], there is no need to change the framework to bias the model to learn objects.
Weevaluate the decomposition ability of our framework on CLEVR [25] and Teris [18] dataset.
Finally,we verify the assumption that the type of concept is driven by the data on the Objects-Room [16]dataset, which has both objects and global factors.
The datasets used here are all public.
Decomposition Given a scene image, VCT represents a single object inside the scene with a singleconcept token and thus spatially decomposes the scene image into objects.
See the details in Appendix
C. Figure 4 and 5 (c) illustrate that an individual object is encoded in a single concept token both on
CLEVR and Tetris .
Quantitative Evaluation Note that we can not obtain masks from VCT directly.
In order to have aquantitative comparison, we decode a explicit mask as did in [31, 18] (see Appendix C for detail).
We present the qualitative results of masks in Figure 4.
Our framework achieves an ARI [18] of0.923 and a mean segmentation covering [15](MSC) of 0.760.
COMET [13], MONET [6], and Slot
Attention [31] obtain ARI of 0.916, 0.873, and 0.818, and obtain MSC of 0.713, 0.701, and 0.750.
Objects Recombination As demonstrated in Figure 5, VCT can add an object from one scene toanother, remove an object from a scene, and combine different objects from two scenes by simplyreplacing tokens.
We can even generate an image with more objects in one scene than the datasetimages, which is out-of-distribution.
Please see the details in Appendix C. Compared to COMET and
MONET ( n times inference of decoder for n objects), for an image with any number of objects, VCTonly needs to inference one time.
[Figure]
[Figure] [Figure]
Objects & Properties Decomposition Since the proposed requirements of concept tokens areboth applicable for object-level representation and property factor variation, what information VCTencodes is data-driven.
To verify this, we experiment on Objects-Room.
As Figure 3 shows, visualtokens represent both object and property (background color and floor color).
In contrast, as shownin Appendix C, COMET has difficulty learning factors like background colors.
4.4 Language Aligned Disentanglement
As we stated in Section 1, VCT tokenizes the visual concepts, like tokenization in NLP.
In order todemonstrate the benefit of such tokenization.
In this section, we utilize the pre-trained CLIP model topresent such goodness of our method.
Specifically, we use the pre-trained CLIP image encoder as theimage tokenizer.
Our experiments were mainly conducted on Shapes3D.
CLIP Text Encoder-based Editing CLIP aligns the text and image information.
Therefore, the textencoder is also disentangled when we disentangle the image encoder.
In order to verify this, we inputa text prompt to derive concept tokens and replace the corresponding token of a image.
Then we canedit the image by decoding.
From Figure 6 (a), we see that the images are edited accordingly.
CLIP Text Encoder-based Decoding Since language is concept-level, like tokenizing a sentence(partitioning a sentence to words), VCT partitions information into individual concepts.
Therefore,
VCT transforms images into tokens similar to NLP.
With VCT, we use the text encoder and VCT todecode the text prompt into an image.
As shown in Figure 6 (b), VCT successfully differentiates theconcept of ball/cube/cylinder (row 1) and small/None/big (row 2).
See details in Appendix C.
Conclusion
In this paper, we demonstrate a general visual concept learning framework VCT to encode data intoa set of tokens, which is a unified architecture to tackle disentangled representation learning andscene decomposition.
We propose an attention-based tokenizer for the induction of information anda disentangling loss for encouraging the exclusivity of different tokens.
VCT can be deployed tothe intermediate representation to learn visual concepts.
Although VCT is unsupervised, and theinformation encoded is data-driven, little hyper-parameter tuning is needed.
Currently, we verify theeffectiveness of VCT on small datasets.
In future, we would like to scale up VCT to learn plentifulvisual concepts in large-scale datasets.
The potential negative societal impacts are malicious uses.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Please see Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please seeSection 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] The code of
reproducing will be public upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Please see Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix A
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 3.1,
Section 4.1, and Section 4.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]