I LLITERATE DALL · E L EARNS TO C OMPOSE
Anonymous authors
Paper under double-blind review
A BSTRACT
DALL · E has shown an impressive ability of composition-based systematic generalization in image generation.
This is possible because it utilizes the dataset of text-image pairs where the text provides the source of compositionality.
Following this result, an important extending question is whether this compositionality can still be achieved even without conditioning on text.
In this paper, we propose an architecture called Slot2Seq that achieves this text-free DALL · E by learning compositional slot-based representations purely from images, an ability lacking in DALL · E. Unlike existing object-centric representation models that decode pixels independently for each slot and each pixel location and compose them via mixture-based alpha composition, we propose to use the Image GPT decoder conditioned on the slots for a more ﬂexible generation by capturing complex interaction among the pixels and the slots.
In experiments, we show that this simple architecture achieves zero-shot generation of novel images without text and better quality in generation than the models based on mixture decoders.
I NTRODUCTION
Unsupervised learning of compositional representation is a core ability of human intelligence (Yuille & Kersten, 2006; Frankland & Greene, 2020).
Observing a visual scene, we perceive it not simply as a monolithic entity but as a geometric composition of key components such as objects, borders, and space (Kulkarni et al., 2015; Yuille & Kersten, 2006; Epstein et al., 2017;
Behrens et al., 2018).
Furthermore, this understanding of the structure of the scene composition enables the ability for zero-shot imagination , i.e., composing a novel, counterfactual, or systematically manipulated scenes that are signiﬁcantly different from the training distribution.
As such, realizing this ability has been considered a core challenge in building a human-like AI system (Lake et al., 2017).
DALL · E (Ramesh et al., 2021) has recently shown an impressive ability to systematically generalize for zero-shot image generation.
Trained with a dataset of text-image pairs, it can generate plausible images even from an unfamiliar text prompt such as “avocado chair” or “lettuce hedgehog”, a form of systematic generalization (Lake & Baroni, 2018; Bahdanau et al., 2018) in the text-to-image domain.
However, from the perspective of compositionality, this success is somewhat expected because the text prompt already brings the composable structure.
That is, the text is already discretized into a sequence of concept modules, i.e., words, which are composable, reusable, and encapsulated.
DALL · E then learns to produce an image with global consistency by smoothly stitching over the discrete concepts via an Image GPT (Chen et al., 2020) decoder.
Building from the success of DALL · E, an important question is if we can achieve such systematic generalization for zero-shot image generation only from images without text.
This would require realizing an ability lacking in DALL · E: extracting a set of composable representations from an image to play a similar role as that of word tokens.
The most relevant direction towards this is object-centric representation learning (Greff et al., 2019; Locatello et al., 2020; Lin et al., 2020b;
Greff et al., 2020b).
While these can obtain a set of slots from an input image and reconstruct the same image from the slots, we argue that its systematic generalization ability to handle arbitrary reconﬁguration of slots is signiﬁcantly limited due to the way it decodes the slots.
In this paper, we propose a model that can achieve composition-based systematic generalization for image generation like DALL · E but by learning compositional representations from images instead of providing a text prompt.
We call this model, Illiterate-DALL · E. To do this, we ﬁrst introduce the slot-decoding dilemma and the pixel independence problem observed in the pixel-mixture decoder of traditional object-centric representation models.
Then, we hypothesize that achieving compositional systematic generalization in image generation requires not only compositional slots but also a powerful expressive decoder that can model complex interactions among the slots and other pixels.
To this end, we propose the slot-to-sequence (Slot2Seq) decoder, which is an Image GPT decoder that takes the extracted slots as input instead of text.
In experiments, we show that this simple architecture achieves the desired properties.
The main contribution of the paper is to show for the ﬁrst time that the compositional systematic generalization, shown in DALL · E, can be achieved without text.
We also show that making a powerful and expressive decoder work with slot representations is crucial in achieving this.
In this regard, we introduce a new architecture, Illiterate-DALL · E, and a new slot decoding framework, Slot2Seq.
The paper also introduces a new dataset, CLEVR-Mirror, and other tasks useful to evaluate the systematic generalization ability in image generation.
Lastly, the experiment results strongly support the beneﬁts of our model, and suggest that future object-centric models may beneﬁt from building on the advantages of the proposed framework alongside the pixel-mixture decoder.
P RELIMINARIES 2. O BJECT -C ENTRIC R EPRESENTATION L EARNING WITH P IXEL -M IXTURE D ECODER
A common framework for learning object-centric representations is via a form of auto-encoders (Locatello et al., 2020; Burgess et al., 2019; Lin et al., 2020b).
An encoder takes an input image to return a set of object representation vectors or slot vectors { s 1 , . . .
, s N } = f φ ( x ) .
A decoder then reconstructs the image by composing the decoded result of each slot ˆ x = g
θ
( s 1: N ) .
To encourage the emergence of object concepts in the slots, the decoder usually uses an architecture implementing an inductive bias about the scene composition such as the pixel-mixture decoders.
Pixel-mixture decoders are the most common approach to construct an image from slots.
This method assumes that the ﬁnal image is constructed by the pixel-wise weighted mean of the slot images.
Speciﬁcally, each slot n generates its own slot image µ n and its corresponding alpha-mask π n .
Then, the slot images are weighted-summed with the alpha-mask weights as follows:
[Equation]
where
[Equation]
and
[Equation]
where i ∈ [1 , HW ] is the pixel index with H and W the image height and width, respectively.
2. L IMITATIONS OF P IXEL -M IXTURE D ECODERS
The pixel-mixture decoder has two main limitations.
The ﬁrst is what we term as the slot-decoding dilemma .
As there is no explicit incentive for the encoder to make a slot focus on the pixel area corresponding to an object, it is usually required to employ a weak decoder for g RGB θ such as the
Spatial Broadcast Network (Watters et al., 2019) to make object concepts emerge in the slots.
By limiting the capacity of the slot-decoder, it prevents a slot from modeling multiple objects or the entire image and instead incentivizes the slot to focus on a local area that shows a statistical regularity such as an object of a simple color.
However, this comes with a side effect: the weak decoder can make the generated image blur the details.
When we use a more expressive decoder such as a CNN to capture such details, the object disentanglement tends to fail with encoder producing slots that capture the whole image, hence a dilemma (See Appendix D.1).
The second problem is the pixel independence .
In pixel-mixture decoders, each slot’s contribution to a generated pixel, i.e., π n,i and µ n,i , is independent of the other slots and pixels.
This may not be an issue if the aim of the slots is only to reconstruct the input image.
However, it becomes an issue when we consider that an important desired property for object representations is to use them as concept modules i.e., to be able to reuse them in an arbitrary conﬁguration in the same way as word embeddings can be reused to compose an arbitrary sentence.
The lack of ﬂexibility of the decoder due to the pixel independence, however, prevents the slots from having such reconﬁgurability.
For example, when decoding an arbitrary set of slots, the rendered image would look like a mere superposition of individual object patches without global semantic consistency , producing a
Frankenstein-like image as the examples in Section 5 shall show.
In contrast, this zero-shot global semantic consistency for concept reconﬁguration is achieved in DALL · E by using the Image GPT decoder.
2. I MAGE GPT AND DALL · E
Image GPT (Chen et al., 2020a) is an autoregressive generative model for images implemented using Transformer (Vaswani et al., 2017; Brown et al., 2020).
For computational efﬁciency, Image
GPT ﬁrst down-scales the image of size H × W by a factor of K using a VQ-VAE encoder (van den
Oord et al., 2017).
This turns an image x into a sequence of discrete image tokens { z i } where i indexes the tokens in a raster-scan order from 1 to HW/K 2 .
The transformer is then trained to model an auto-regressive distribution over this token sequence by maximizing the log-likelihood (cid:80) i log p θ
( z i | z <i ) .
During generation, the transformer samples the image tokens sequentially conditioning on the previously generated tokens, i.e., ˆ z i ∼ p θ (ˆ z i | ˆ z <i )
where ˆ z i is the image token generated for the position i in the sequence.
Lastly, a VQ-VAE decoder takes all the generated tokens and produces the ﬁnal image ˆ x .
Crucially, due to the autoregressive generation of z i , unlike the pixel-mixture decoder, the generated pixels are not independent.
The autoregressive generation powered by the direct memory access of the transformer makes Image GPT one of the most powerful decoder models in image generation and completion tasks.
However, Image GPT does not provide a way to learn high-level semantic representations such as object-centric representations that can provide a way to semantically control image generation.
DALL · E (Ramesh et al., 2021) shows that the image generation of Image GPT can be controlled by conditioning on a text prompt.
In particular, it shows the impressive ability of zero-shot generation with global semantic consistency for rather an arbitrary reconﬁguration of the words in the text prompt.
DALL · E can be seen as a conditional Image GPT model, ˆ z i ∼ p
θ (ˆ z i | ˆ z <
i , c 1: N ) , where c 1: N are the text prompt tokens.
The key limitation from the perspective of this work is that it requires supervision in terms of text-image pairs.
This means that it does not need to learn the compositional structure by itself since the discrete concepts are inherently provided by the prompt text even though it learns the word embeddings provided the structure.
Therefore, it is important to investigate if we can make a text-free DALL · E by learning to infer such compositional structure and representation only from images.
I LLITERATE DALL · E WITH S LOT -T O -S EQUENCE D ECODER
In this section, we propose a method that can provide global semantic consistency in the zeroshot image generation setting without text by learning object slots from images.
To do this, we shall lift the use of inductive biases in the slot decoder about object composition by making use of a transformer (Vaswani et al., 2017) as our image decoder while replacing the text prompt with a concept library constructed from image slots.
Our central hypothesis is that a powerful autoregressive decoder such as Transformer after training with a sufﬁciently diverse set of raw images should learn the rules of composition without domain speciﬁcations.
Obtaining Image Tokens using DVAE.
To make the training of transformer computationally feasible for high-resolution images, we ﬁrst downscale the input image x of size H × W by a factor of
K using Discrete VAE (Im et al., 2017).
To do this, we split the image into K × K -sized patches resulting in T patches where T = HW/K 2 .
We provide each patch x i as input to an encoder network f φ to return log probabilities (denoted as o i ) for a categorical distribution with V classes.
With these log probabilities, we use a relaxed categorical distribution (Jang et al., 2016) with a temperature τ to sample a relaxed one-hot vector z soft i .
We then decode these soft vectors to obtain patch reconstructions.
This process is summarized as follows:
[Equation]
By minimizing an MSE reconstruction objective for the image patches i.e. L DVAE =
(cid:80)
Ti =1 (˜ x i − x i ) 2 , we train the DVAE encoder f φ ( · ) and the decoder g θ ( · ) networks.
Inferring Object Slots.
To infer the object slots from a given image x , we ﬁrst use the DVAE encoder as described above to obtain a discrete token
z i for each patch i .
Next, we map each token z i to an embedding by using a learned dictionary.
To incorporate the position information into these patch embeddings, we sum them with learned positional embeddings.
This results in an embedding u i which now has both the content and the position information for the patch.
These embeddings u 1: T are then given as input to a Slot-Attention encoder (Locatello et al., 2020) with N slots.
[Equation]
This results in N object slots s 1: N and N attention maps A 1: N from the last reﬁnement iteration of the Slot-Attention encoder.
Reconstruction using Transformer.
To reconstruct the input image, we decode the slots s 1: N and ﬁrst reconstruct the DVAE tokens ˆ z 1: T using a Transformer.
We then use the DVAE decoder g θ to decode the DVAE tokens and reconstruct the image patches ˆ x i and therefore the image ˆ x .
[Equation]
i
[Equation]
Transformer
[Equation]
where ˆ u i =
Dictionary φ (ˆ z i ) + p φ,i and
ˆ
o
i,v is the log probability of token v among the V classes of the categorical distribution represented by ˆ o i .
To train the transformer, the slot attention encoder and the embeddings, we minimize the cross-entropy of predicting each token
z i conditioned on the preceding tokens
z <
i and the slots s 1: N .
Let the predicted log-probabilities for the token at position i be ¯
o i =
Transformer θ ( u <i ; s 1: N ) .
Then the cross-entropy training objective can be written as
L ST = (cid:80)
Ti =1 CrossEntropy ( z i , ¯ o i ) .
Learning Objective and Training.
The complete training objective is given by L
= L ST + L DVAE and all modules of our model are trained jointly.
We apply a decay on the DVAE temperature τ from 1.0 to 0.1 at the start of the training.
We also apply a learning rate warm-up for the parameters of
Slot-Attention and Transformer at the start of the training as we found that this led to stable training.
Multi-headed Slot Attention.
For representing images, we found that slots of standard Slot Attention encoder (Locatello et al., 2020) can suffer in expressiveness when representing objects with complex shapes and textures.
This is because the slots collect information from the input cells via dot-product attention in which the attended input values are pooled using a simple weighted sum.
As such, this pooling method can be too weak for representing complex objects.
To address this, we propose an extension of Slot Attention called Multi-headed Slot Attention in which each slot attends to the input cells via multiple heads.
Thus each slot can attend to different parts of the same object.
When the slot state is computed, the attended values of different heads are concatenated which allows these partial object features to interact more ﬂexibly and produce a signiﬁcantly better object representation.
Full details of the implementation and an ablation experiment to show the beneﬁts of multiple heads are provided in Appendix A.3.
3. V ISUAL C ONCEPT L IBRARY
The above model extracts a set of concepts or slots belonging to a particular image whereas an intelligent agent would build a library of reusable concepts from diverse experience.
In DALL · E, a vocabulary of word embeddings plays the role of this library and provides reusable concepts.
Similarly, to build a library of visual concepts from the dataset, we use the following simple approach based on K -means clustering to construct a library of reusable visual concepts.
To do this, (i) collect
N slots S i
=
s i 1 , . . .
, s iN } and their attention maps A i
=
{ A i 1 , . . .
, A iN } from each image i in the training dataset.
(ii) Run K -means clustering on S =
(cid:83) i S i with cosine similarity between slots as a distance metric where K is the number of total concepts in the library.
If the object position is more important for building the concept library, cluster the slots using IOU between the attention maps as a similarity metric.
(iii) Use each cluster as a concept and the slots assigned to the cluster as the instantiations of the concept.
(iv) To compose an arbitrary image, choose the concepts from the library and randomly choose a slot for each concept and build a slot prompt similar to a text prompt in DALL · E. (v) Provide this prompt to the decoder to compose and generate the image.
R ELATED W ORK
Compositional Generation via Object-Centric Learning.
Self-supervised object-centric approaches (Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Kabra et al., 2021; Greff et al., 2017; Engelcke et al., 2020; 2021; Eslami et al., 2016; Crawford & Pineau, 2019b; Lin et al., 2020b; Jiang & Ahn, 2020; Kosiorek et al., 2018; Jiang et al., 2019; Crawford & Pineau, 2019a; Lin et al., 2020a;
Deng et al., 2021; Anciukevicius et al., 2020; von K ¨ ugelgen et al., 2020; Wu et al., 2021) typically perform mixture-based alpha compositing for rendering with contents of each slot decoded independently.
Savarese et al.
(2021) and Yang et al.
(2020) rely on minimizing mutual information between predicted object segments as a learning signal while Yang et al.
(2021) leverage optical ﬂow.
However, these approaches either cannot compose novel scenes or require specialized losses for object discovery unlike ours which uses a simple reconstruction loss.
DINO (Caron et al., 2021) combining Vision Transformer (Dosovitskiy et al., 2021) and a self-supervised representation learning objective (Chen et al., 2020b; He et al., 2020; Grill et al., 2020; Touvron et al., 2021; Hinton, 2021; van den Oord et al., 2018) discovers object-centric attention maps but, unlike ours, cannot compose novel scenes.
TIMs (Lamb et al., 2021; Goyal et al., 2021) with an Image GPT decoder show object-centric attention maps but, unlike ours, do not investigate compositional generation.
Compositional Generation using GANs.
Chai et al.
(2021) show compositional generation using a collage of patches encoded and decoded using a pretrained GAN.
However, the collage needs to be provided manually and object discovery requires a specialized process unlike our model.
Bielski & Favaro (2019) and Chen et al.
(2019) show object discovery from given images by randomly re-drawing or adding a random jitter (Voynov et al., 2021) to the foreground followed by an adversarial loss.
As these models use alpha compositing, their compositional ability is limited.
Some works (Donahue et al., 2016; Donahue & Simonyan, 2019) infer abstract representations for images that emphasize the high-level semantics, but these, unlike ours, cannot be used for compositional image generation.
Niemeyer & Geiger (2021), Nguyen-Phuoc et al.
; Chen et al.
(2016), van
Steenkiste et al.
(2020), Liao et al.
(2020) and Ehrhardt et al.
(2020) introduce GANs which generate images conditioned on object-wise or factor-wise noise and optionally on camera pose.
Lacking an inference module, these cannot perform compositional editing of a given image unlike ours.
While
Kwak & Zhang (2016) provide an encoder to discover objects, the decoding relies on alpha compositing.
Reed et al.
(2016a) show compositional generation in GANs and improved generation quality (Johnson et al., 2018;
Hinz et al., 2019) via an object-centric or a key-point based pathway for scene rendering.
However, these require supervision for the bounding boxes and the keypoints.
Compositional Generation with Latent Variable Models and Text.
Early approaches focused on disentangling the independent factors of the observation generating process using β -VAE (Higgins et al., 2017) or ensuring that independently sampling each latent factor should produce images indistinguishable from the training distribution (Kim & Mnih, 2018; Kumar et al., 2017; Chen et al., 2018).
However unlike ours, these approaches providing a monolithic single vector representation can suffer in multi-object scenes due to superposition catastrophe (Greff et al., 2020a; Jiang & Ahn, 2020).
Another line of approaches learn to map text to images to compose novel scenes (Ramesh et al., 2021;
Reed et al., 2016b; Li et al., 2019; Higgins et al., 2018) or conditional image generation
[Table]
[Figure] [Figure]
(Sohn et al., 2015; Yan et al., 2016; Johnson et al., 2018; Ashual & Wolf, 2019; Bar et al., 2021;
Herzig et al., 2020; 2019; Du et al., 2020).
However such approaches rely on supervision.
E XPERIMENTS
In experiments, we evaluate the beneﬁts of Slot2Seq over pixel-mixture decoder in: 1) generating novel images from an arbitrary conﬁguration of slots, 2) image reconstruction, and 3) generating novel images not belonging to the training image distribution.
As the baseline, we compare with an unsupervised object-centric framework having a slot attention encoder like ours but using a mixture decoder for decoding instead of Slot2Seq.
Hereafter, we refer to this model as simply Slot Attention (Locatello et al., 2020).
We evaluate the models on four datasets: 3D Shapes (Burgess & Kim, 2018), CLEVR-Mirror which we develop from the CLEVR (Johnson et al., 2017) dataset by adding a mirror in the scene (see Appendix E.1), Shapestacks (Groth et al., 2018), Bitmoji (Graux, 2021) and Textured-MNIST.
For brevity, we shall refer to CLEVR-Mirror as simply CLEVR.
In each dataset, a scene can be described in terms of composable entities such as blocks, walls, ﬂoors, hair, or face.
Our models take only raw images without any other supervision or annotations.
5. C OMPOSITIONAL I MAGE G ENERATION
We evaluate how effectively the known slots can be reused to generate novel scenes.
For this, we evaluate the quality of image generations produced by our model from arbitrarily composed slot prompts.
To build these prompts, we ﬁrst build a concept library using K -means as described in
Section 3.1.
In 3D Shapes and Bitmoji, we apply K -means clustering on the slots.
In 3D Shapes, this results in clusters for wall , object and ﬂoor and in Bitmoji for dress , hair , neck , eyes , face and collar .
These clusters make up our concept library and are visualized in Appendix C. To build novel prompts from this library, we simply pick one slot from each cluster.
In CLEVR and Shapestacks datasets, to build a prompt, it is more natural to pick the object positions ﬁrst and then sample a slot from the library for that position on the canvas.
Hence, for CLEVR, we pick a set of random positions keeping a certain distance between the picked positions.
For Shapestacks, we pick a set of random positions in a vertical tower-like conﬁguration.
In this way, we generate 40000 prompts for each dataset and render the corresponding images.
We then evaluate the quality of the images by computing the FID score with respect to the true images.
To support this metric, we also report the training curves of a CNN discriminator that tries to classify between real and model-generated images.
If generated images are close to the true data distribution, then it should be harder for the discriminator to discriminate and the resulting training curve should converge more slowly.
From Table 1a and Figure we note that our compositional generations are signiﬁcantly more realistic than the generations from the mixture decoder.
The qualitative samples shown in Figures 3, 4, 5 and 6 show that our decoder can compose the desired image accurately and make an effective reuse of the known slots.
Additional qualitative results are provided in Appendix C. We also note that for visually complex Shapestacks and Bitmoji datasets, the improvement in the FID score provided by our model is much larger than that for the visually simpler CLEVR and 3D Shapes datasets.
These results provide evidence that i) our model resolves the slot-decoding dilemma and consequently, it beneﬁts from the ﬂexibility of the Image GPT decoder that can render the ﬁne details such as ﬂoor texture in Shapestacks and complex facial features in Bitmoji while the mixture decoder suffers.
Notice that the attention segments of the visualized slots effectively focus on the speciﬁc object regions of the respective images that were provided during concept library creation.
This emergent object localization occurs despite the powerful decoder, thus resolving the dilemma.
Because our slots attend on the DVAE tokens, hence our attention is patch-level and our attention segments may appear coarser than those of Slot Attention in which slots attend on the CNN feature cells having the same resolution as the input image.
ii) Because the mixture decoder has the pixel-independence assumption , we see in Figure 3 that it suffers when slots are combined in arbitrary conﬁgurations.
In 3D Shapes, the object shadows are controlled by the ﬂoor component of the mixture and are either too large or too small for the 3D object.
This is because after arbitrarily choosing a different 3D object, the ﬂoor slot has no knowledge that the object which casts the shadow has changed and it continues to draw the shadow mask of the source image which has now become obsolete in the new slot conﬁguration.
Slot Attention suffers even more severely in Bitmoji as the composed mixture components and masks are mutually incompatible in the new slot conﬁguration and produce an inaccurate alpha composition.
In contrast, Slot2Seq is robust to this problem because the slots and pixels can interact.
For instance, in 3D Shapes, as the pixels of the ﬂoor are being rendered, the predictor is aware of the object that will cast the shadow via the previous pixels and the input slots and can thus predict a globally consistent pixel value.
5. R ECONSTRUCTION Q UALITY
A natural test about whether an auto-encoder learns accurate representations of the given input is through an evaluation of the reconstruction error.
For this, we report two metrics: 1) MSE to evaluate how well the reconstructed image preserves the contents of the original image and 2) FID score to show how realistic is the reconstructed image with respect to the true image distribution.
We compute these metrics on a held-out set and report in Table 1b.
We ﬁnd that our model outperforms Slot
Attention in all datasets in terms of FID which shows that the image quality of our model is better
[Table]
due to its powerful decoder.
In MSE, our model performance is comparable for synthetic datasets such as 3D Shapes and CLEVR while we outperform on the visually more complex Shapestacks,
Bitmoji and Textured-MNIST.
The performance gain from our model, in both MSE and FID, increases as the visual complexity of the dataset increases.
Despite the better MSE of the mixture decoder in 3D Shapes and CLEVR, the gap is not severe given the broader limitations of the mixture decoder in handling novel slot conﬁgurations.
5. G ENERALIZATION TO O UT - OF -D ISTRIBUTION (OOD) S LOT R ECONFIGURATIONS
Because the Slot2Seq decoder is based on Transformer and does not have an explicit inductive bias about scene composition, we test whether it can generalize in composing slots when their conﬁgurations are different from the training distribution.
For this, i) in Shapestacks, we compose slot prompts to render two towers while only single-tower images were shown during training.
ii)
In
CLEVR, we compose prompts to render fewer or more objects than were shown during training.
iii)
Lastly, in Bitmoji, we annotated the gender of 500 images and used it to construct 1600 OOD slot prompts by composing hair and face only from opposite genders.
By doing this, we change the distribution of the slot conﬁgurations with respect to the training distribution.
Note that such slot prompts were already a part of the Bitmoji compositions in Section 5.1.
However, with this experiment, we evaluate the speciﬁc OOD subset of those prompts.
We show the results in Figures 4,and 7 and a comparison of FID score with respect to the mixture decoder in Table 2.
From these, we note that our decoder is able to generalize to the OOD slot conﬁgurations and it also outperforms the mixture decoder in image quality.
D ISCUSSION
We presented Illiterate DALL · E, a model for text-free zero-shot imagination by learning slot representations only from images.
Our model combines the best of DALL · E and object-centric representation learning via the proposed decoder Slot2Seq .
It provides a signiﬁcant improvement over pixel-mixture decoders in reconﬁgurability of slots for zero-shot imagination.
These results suggest that future object-centric models may beneﬁt from building on the advantages of the proposed framework alongside the pixel-mixture decoder.
The proposed Slot2Seq, with its Transformer-based implementation, can also be seen as a graphics engine that learns to mimic the physical laws governing the rendering of the image through its exposure to a large dataset.
Thus to render a scene, the only information that the Transformer requires and requests from the encoder is the abstract object-level what and where in the form of slots.
Knowing these objects, the Transformer then ﬁlls in the rest and implicitly handles the mechanisms that give rise to the observed object appearance, occlusions, shadows, reﬂections, and transparency.
An interesting future direction is to consider that an intelligent agent may collect new experience indeﬁnitely, and thus our approach of building the library of reusable concepts can be made more formal by learning discrete concepts and growing this library using online clustering (Liberty et al., 2016) or Bayesian non-parametric clustering (Neal, 2000).
Another direction is to learn a slot-level prior for fast density modeling and sampling scenes at the representation level.
E THICS S TATEMENT
For future applications, negative societal consequences should also be considered.
The current model does not generate images realistic enough to deceive humans if used with malicious intent.
However future versions of the model after scaling up its size and the training dataset can have such potential.
Despite these considerations, these consequences are not imminent
I LLITERATE DALL · E USING S
[Figure]
A. DVAE
Motivation.
Our motivation for using DVAE is to quantize each patch (4x4 pixels in size) in the image as a discrete token and to ensure that the discrete token can accurately reconstruct the patch via the DVAE decoder.
We decided to use DVAE (Ramesh et al., 2021) over VQ-VAE (van den Oord et al., 2017) because DVAE is simpler than VQ-VAE and requires only a simple reconstruction loss to train while VQ-VAE requires additional loss terms.
Furthemore, because DALL-E also uses
DVAE, it becomes a more natural choice to adopt.
Omitting the KL Term.
In training the DVAE, we only use the reconstruction loss i.e. a simple
MSE reconstruction objective rather than additionally using a KL term to impose a prior on the inferred tokens.
This choice is made because we found the training to be more stable when we used smaller β coefﬁcients (such as 0.04 or smaller) for the KL term including setting β to 0.
This suggested that the KL term did not play a useful role in our model and hence setting the β coefﬁcient to 0 is a more simpler choice to adopt.
This is equivalent to removing the KL term.
A. M ULTI - HEADED S LOT A TTENTION
In Slot-Attention (Locatello et al., 2020), each slot undergoes several rounds of interactions with the input image in order to collect the contents for the slot representation.
In each iteration, this interaction of a slot with the input image is performed using dot-product attention with slot as the query.
The value returned by this attention is a simple weighted sum of the input cells being attended by the slot.
This weighted sum as a pooling method and method of interaction between the attended cells can be insufﬁcient for properly representing the attended cells.
This is especially the case when the object has a complex shape and texture.
To address this, we propose a multi-headed extension of slot attention.
In this, we replace the dot-product attention with a multi-headed dot-product attention.
By doing this, each slot attends to the different parts of the same object simultaneously and the result of the attention is concatenated.
As this concatenated result passes through the RNN layers, these features for different parts of the same object can interact ﬂexibly and result in a more rich encoding of the attended object.
Recall that in standard slot attention, because there is only one attention head, different parts of the same object can only interact via the weighted sum which serves as a weak form of interaction as our results below shall indicate.
Implementation.
We implement iterations of the Multi-headed Slot Attention as follows.
For a given iteration, we ﬁrst project the slots S
= s 1: N from the previous iteration into query vectors for each head.
Similarly, we project the input cells U
= u 1: T into key and value vectors for each head.
[Equation]
Next, we compute unnormalized attention proportions for each head m as by taking a dot-product as follows:
[Equation]
where D K is the size of the key vectors.
The attention weights for the input cells for each slot n and each head m is given by taking a softmax over both the N slots and the M heads.
[Equation]
We then obtain the attention result for a given slot n as follows.
[Equation]
where W O is an output project matrix.
With this update vector, the slot state is updated as follows as done in the standard Slot Attention (Locatello et al., 2020).
[Equation]
This results in the update slot that is given to the next iteration.
Results.
We apply Multi-headed Slot Attention in the Bitmoji dataset for our model.
We found signiﬁcant improvement in the MSE when we used 4 heads as compared to the standard Slot Attention (equivalent to 1 head).
This is reported in Table 3.
[Table]
We also noted that when using mixture decoder, using multi-headed slot attention does not help because the performance bottleneck is the Spatial Broadcast decoder (Watters et al., 2019) of the mixture decoder.
This comparison is shown in Table 4.
Limitations.
While multi-head attention helps information in the slots, in datasets with simple objects, having multiple heads can harm disentanglement of objects into different slots.
This can occur because as each slots becomes more expressive, the network may be incentivised to collect information about multiple objects into the same slot.
Therefore for our experiments on 3D Shapes,
CLEVR and Shapestacks datasets, we used regular slot attention module with 1 head.
Because
Bitmoji can have complex hair and face shapes, we used 4-headed slot attention module for this dataset in our model.
[Table] [Table]
A. A BLATION OF A RCHITECTURE
To better justify our design we perform two additional experiments that we describe here.
In terms of architectural components
[Table]
our model can be seen as VQ-Input + Slot Attention + Transformer
Decoder while our baseline (Locatello et al., 2020) can be seen as CNN Input + Slot Attention
+
Mixture Decoder.
Hence, the components that differ between our model and the baseline are i) providing VQ-Input and ii) using a Transformer decoder.
We analyze the effect of these individual components by performing two ablations.
In the ﬁrst ablation, we take our model and replace the
VQ-Input with CNN Input.
We refer to this ablation as CNN Input + Transformer Decoder .
In the second ablation, we take our model and replace the Transformer decoder with the mixture decoder.
We refer to this ablation as VQ-Input + Mixture Decoder .
We report the results in Table 5.
We also visualize the object attention maps of the ablation models in Figure 26.
B H YPERPARAMETERS AND C OMPUTATIONAL R EQUIREMENTS
We report the hyperparameters and the computational requirements for training our model in Table 6.
[Table]
C A DDITIONAL Q UALITATIVE R ESULTS
D A DDITIONAL D ISCUSSION D. S TRONG D ECODER IN M IXTURE L IKELIHOOD M ODELS M AKE S LOTS C APTURE M ULTIPLE O BJECTS
One of the attractive properties of our model is that our Transformer decoder produces images with signiﬁcantly better detail than Slot Attention.
Slot Attention fails to render such details largely because of Spatial Broadcast Decoder that is used to render each object component.
Spatial Broadcast
Decoder decodes images by mapping a pixel coordinate and the provided slot vector to RGB pixel values and a mask using an MLP.
This MLP prefers to learn simple mappings from pixel coordinate to RGB values.
At the start of the training, this function tends to be a constant or a linear function of pixel coordinate and only after more training does this function achieve a more complex input-output behavior.
Hence, when a Spatial Broadcast Decoder is used, the decoder of each object component is biased in early training to prefer modeling segments without much RGB variation within the object patch.
This property is important to guide the training of Slot Attention encoder to learn to disentangle separate objects and it has been exploited in several prior works on unsupervised object-centric representation learning.
To understand why this occurs, consider a grayscale image that contains two balls, one colored white with pixel value 1.0 and the other colored grey with pixel value 0.5.
Now consider the case when each slot represents one object in the scene.
In this case, the decoder for the white ball simply needs to output a value 1.0 for all pixel coordinates and this is a constant function.
Similarly, for the grey ball, the decoder needs to output a value 0.5 for all pixel coordinates.
In contrast, consider the case where a single slot represents both the white and the grey balls.
In this case, the decoder needs to map the pixels coordinates for the white ball to 1.0 and the pixel coordinates for the grey ball to 0.5 within the same function.
This function is not longer a constant and is therefore much more complex than the functions that arose when each slot modeled separate objects.
Furthermore, as objects might be positioned randomly, this mapping needs to quickly change when the positions of the balls change in the scene depending on information from the slot representation.
This makes the mapping even more complex when more than one objects are modeled by the same slot.
Hence, the network naturally prefers to model and decode different objects via different slots.
As a result, the encoder is incentivized to cluster the pixels of the input image into objects.
While this property could be utilized in simple datasets which have objects without much RGB variation and detail within the object, it can fail to render images properly in datasets such as Shapestacks with ﬂoor with complex textures or Bitmoji with complex face details and the shape of the hair.
While one may argue that this could be ﬁxed simply by replacing the Spatial Broadcast Decoder with a more powerful decoder for the object components.
However, because weak decoder was encouraging the disentanglement of objects as argued above, hence by the same argument, when a powerful decoder is used, a single slot no longer has the incentive to model one object.
Thus each slot can try to represent and model more than one object in the scene.
We tested this empirically by replacing the Spatial Broadcast Decoder of Slot Attention with a powerful CNN decoder.
We found that this causes the slots to capture multiple objects and in some cases a single slot may try to model all the contents of the scene as a single object segment.
This is shown in Figure 22.
Hence for unsupervised object discovery, this historically presented a trade-off between decoding capacity and the ability of the model to disentangle the objects.
In contrast, our work shows that such a trade-off can be eliminated if Transformer is used as the decoder.
Our model can not only detect objects without supervision but also render their ﬁne details during decoding.
D. O BJECT D ISCOVERY AND C OLOR B IAS
Another problem that mixture decoders suffer from is what we term as the color bias .
In mixture decoder, each object slot is commonly decoded using the spatial broadcast decoder to achieve a decoding capacity bottleneck.
In this decoder, to decode each pixel, the pixel coordinate and the slot vector are concatenated and given to a decoder MLP to predict the pixel values for that coordinate.
Hence, when two pixels are close to each other, the decoder MLP generates very similar pixel outputs for them.
This is because the inputs to the MLP are very similar to each other for these two pixels.
Given that an MLP, in general, prefers to learn smooth and linear (or constant) functions ﬁrst before learning more complex functions, hence the output object rendered by a given slot tends to have a uniform or smoothly changing colors.
As a consequence, this also biases the object-discovery process to also combine similar colors into a single slot and separate the regions with different colors into different slots.
Consequently, this design creates the problem of color bias which arises in two forms when dealing with visually complex images: i)
First, if a single object is made up of multiple colors, the object may get split into multiple slots to handle the complexity of rendering multiple colors.
ii)
Second, if there are multiple objects having a similar color, those objects can get merged and become represented as a single slot.
In visually simple scenes having different objects with separate colors, this might not appear to be a problem.
But in more natural scenes in which single objects can be made up of multiple colors and many objects can have similar colors, this becomes a problem and harms the quality of the discovered objects.
This phenomenon is demonstrated in the object attention maps of mixture decoder in Figures 23, 25 and 24.
In comparison, because our decoder is not biased towards simple or uniform colors, hence our object discovery process is more robust to the color bias issue.
E A DDITIONAL E XPERIMENT D ETAILS E. CLEVR-M IRROR
In this work, we introduce a new dataset, CLEVR-Mirror as an extension of the standard CLEVR dataset.
In comparison to standard CLEVR, in this dataset, we also introduce a mirror into the scene.
CLEVR-Mirror is designed to see whether a model can obtain the ability to model complex global consistency by learning systematic relationship between local components.
The standard CLEVR dataset is not proper for this purpose because the scene can easily be modeled without learning this global structure.
CLEVR-Mirror requires learning relational knowledge such as the size change of reﬂected objects relative to the distance of the actual object to the mirror and occlusion among objects.
For example, as shown in Figure 5, an object can be occluded in the mirror even if it is not in the non-mirror region.
E. C LUSTERING BASED ON A TTENTION M APS
In CLEVR and Shapestacks, we build a concept library based on object position.
For this, we apply K -means but using attention maps of the slots as described in Section 3.1.
Thus each cluster corresponds to certain region in the image.
For building slot prompts in CLEVR, we randomly pick
N clusters without replacement.
By doing this, we picked N random locations where we want to place objects.
In Shapestacks, we randomly pick N clusters in a vertical tower conﬁguration.
We then randomly pick a slot from each cluster to construct the prompt.
In these we note that Slot Attention suffers because of color bias as the parts of the digit and the parts of the background tend to combined into slot
In contrast, in our model the attention maps for the are and are than