L EARNING W HAT AND W HERE : D ISENTANGLING L OCATION AND I DENTITY T RACKING W ITHOUT S UPERVISION
A BSTRACT
Our brain can almost effortlessly decompose visual data streams into background and salient objects.
Moreover, it can anticipate object motion and interactions, which are crucial abilities for conceptual planning and reasoning.
Recent object reasoning datasets, such as CATER, have revealed fundamental shortcomings of current vision-based AI systems, particularly when targeting explicit object representations, object permanence, and object reasoning.
Here we introduce a self-supervised LOCation and Identity tracking system (Loci), which excels on the CATER tracking challenge.
Inspired by the dorsal and ventral pathways in the brain, Loci tackles the binding problem by processing separate, slot-wise encodings of ‘what’ and ‘where’.
Loci’s predictive coding-like processing encourages active error minimization, such that individual slots tend to encode individual objects.
Interactions between objects and object dynamics are processed in the disentangled latent space.
Truncated backpropagation through time combined with forward eligibility accumulation significantly speeds up learning and improves memory efficiency.
Besides exhibiting superior performance in current benchmarks, Loci effectively extracts objects from video streams and separates them into location and Gestalt components.
We believe that this separation offers a representation that will facilitate effective planning and reasoning on conceptual levels.
I NTRODUCTION
Human perception is characterized by segmenting scenes into individual entities and their interactions [4; 13; 25].
This ability poses a non-trivial challenge for computational models of cognition [37; 84]: the binding problem [79].
Visual features need to be selectively bound into single objects, segregated from the background, and encoded by means of compressed stable neural attractors [5; 44; 46; 73].
Recent years have seen revolutionary progress in the ability of connectionist models to operate on complex natural images and videos [31; 50; 65].
Yet, neural network models still do not fully solve the binding problem [37].
Indeed, recent work on synthetic video-based reasoning datasets, like CLEVR, CLEVRER, or CATER [32; 43; 93], suggests that state-of-the-art systems [16; 89; 92] still struggle to model fundamental physical object properties, such as hollowness, blockage, or object permanence—concepts that children learn to master in the first few months of their lives [2; 63].
In a comprehensive review on the binding problem in the context of neural networks, Greff et al.
[37] define three main challenges for solving the problem: representation , segregation , and composition .
Representation refers to the challenge to effectively represent the essential properties of an object, including its appearance and potential dynamics.
We will refer to these properties as the ‘Gestalt’ of an object [48; 86; 87].
Moreover, the individual objects’ locations and motions dynamics should be disentangled from their Gestalt to enable compositional recombinations.
Meanwhile, the representations should share a common format to enable general purpose reasoning.
Segregation describes the challenge to extract particular objects from a perceived scene.
This extraction should be done contextand task-dependently to identify the currently relevant entities.
As a result, a good segregation should enable effective dynamic predictions of the whole, rather than only the parts.
Finally, composition characterizes the challenge to develop object representations that enable meaningful re-combinations of object properties—particularly those that facilitate the prediction of object interaction dynamics.
As a result, compositional representations enable conceptual reasoning about object properties as well as relations and interactions between objects.
We introduce a novel Loc ation and I dentity tracking system.
While observing videos, Loci disentangles object identities (‘what’) from their spatial properties (‘where’) in a fully unsupervised, autoregressive manner.
It is motivated by our brain’s ventral and dorsal processing pathways [67; 80].
Loci’s key contribution lies in how object-specific information is disentangled and recombined:
(i) Loci fosters slot-respective object persistence over time via a novel combination of slotspecific input channels, temporal slot-interactive predictions via self-attention [83] followed by GateL0RD-RNN [39], and object permanence-oriented loss regularization.
(ii) Our slot-decoding strategy combines object-specific Gestalt codes with parameterized Gaussians in a, to the best of our knowledge, novel manner.
This combination fosters the emergent explication of an object’s size, its position, and current occlusions.
(iii)
We improve sample and memory efficiency by training Loci’s recurrent modules by means of time-local backpropagation combined with forward propagation of eligibility traces.
As a main result, we observe superior performance on the CATER benchmark: Loci outperforms previous methods by a large margin with an order of magnitude fewer parameters.
Additional evaluations on moving MNIST, an aquarium video footage, and on the CLEVRER benchmark underline Loci’s contribution towards the self-organized, disentangled identification and localization of objects as well as the effective processing of object interaction dynamics from video data.
R ELATED W ORK
Previous work by [59] has emphasized that, in general, unsupervised object representation learning is impossible because infinitely many variable models are consistent with the data.
Inductive biases are thus necessary to ensure the effective learning of a system that segregates a visual stream of information into effective, compositional representations [37].
Accordingly, we review related work in the light of the binding problem and their relation to the proposed Loci system.
Representation A powerful choice of an encoding format is the formulation of ‘slots’, which share the encoding module but keep the codes of individual objects separate from one another.
To ensure a common format between the slot-wise encodings, typically, slot-respective encoder modules share their weights [8; 60; 83].
To assign individual objects to individual slots, though, the system needs to break slot symmetry.
Recurrent neural networks have been used to disentangle encodings or assignments [11; 28; 29; 35; 60].
Other mechanisms explicitly separate spatial slot locations [20; 42; 54], which we also do in Loci.
However, instead of treating every spatial location as a potential object, each slot has a spotlight, which is designed to approximate the object’s center.
To further foster a compositional object representation, Loci enforces disentanglement of ‘what’ from ‘where’ by separating an object’s Gestalt code—mainly representing shape and surface pattern—from its location, size (visual extent), and priority (current visibility with respect to other objects).
This stronger disentanglement and more complex ‘where’ representation is related to work that models selective visual attention, realizing partially size-invariant tracking of one particular entity, such as a pedestrian [22; 45; 72].
Advancing this work, Loci tracks multiple objects in parallel, imposes interactive, object-specific spot-lights, and enables more compressed, object-specific appearance representations due to its novel way of combining ‘what’ and ‘where’ for decoding.
Segregation Segregating object instances from images is traditionally solved via bounding box detection [58; 74], where more advanced techniques extract additional masks for instance segmentation [15; 21; 40].
Through slot-attention mechanisms, recent unsupervised approaches partition image regions into separate slots to represent distinct objects [36; 61].
To segregate objects from images, Burgess et al.
[10] and von Kügelgen et al.
[85] combine a soft attention (mask) approach with encoder, recurrent attention, and decoder modules, where slots compete for encoding individual objects.
Loci pursues a similar approach but segregates objects even further to encourage objectrespective ‘what’ and ‘where’ representations, where the latter additionally disentangle location, size, and approximate depth.
Slot-respective masks compete via softmax attention to actively minimize the prediction error of the visual content.
Moreover, a pre-trained background model separates potentially interesting from uninteresting regions.
While we keep the background modeling rather simple in this work, more advanced techniques may certainly be applied in the near future [26; 68; 69; 82].
Composition Compositional reasoning in our model builds on two modules, which process objectto-object interactions and object dynamics.
Object-to-object interactions are modelled using MultiHead Self-Attention (MHSA) [83], in close relation to [27; 47].
Others have employed message passing neural networks to simulate object interactions [6; 18; 41]. Another promising approach uses
Neural Production Systems to dynamically select a learnable rule to model object interactions [23].
Object dynamics are modelled using a recurrent GateL0RD module [39].
GateL0RD is designed to apply latent state updates sparsely, which encourages the maintenance of stable object encodings over long periods of time.
Previous approaches have also employed recurrent structures to propagate slotwise encodings forward in time [42; 49].
Although some previous works have combined recurrent structures with attention [33; 34], recent slot-attention frameworks tend to adopt fully auto-regressive designs based on transformers without explicit internal state maintenance [27; 47; 64].
Tracking models While our primary goal is to separate object location and Gestalt representations, extracted object locations likely facilitate object tracking; a wide area of research on its own [91].
State-of-the-art object tracking methods rely on features extracted via attention modules, which are typically applied autoregressively on individual frames [19; 64].
Again, these approaches do not explicitly foster a separation of ‘what’ from ‘where’, which may limit their applicability and accuracy in distractive environments, where humans still maintain high tracking skills, as shown in [55].
Loci maintains separate object encodings and thus copes with the presence of distractors more readily.
The tracking model proposed in [55] is similar in spirit to Loci, extracting object encodings and propagating them with a competition mechanism and a recurrent module.
However, to model complex objects their model relies on additional supervision.
Disentangling movement and appearance is a common principle in video models with a notion of optical flow [57].
But optical flow based methods are not designed to deal with occlusions as they only represent parts of the scene that are currently visible.
In contrast, Loci is able to maintain the encoding of an object even under occlusion.
The CATER benchmark We evaluate Loci mainly on the CATER challenge [32].
Previous
SOTA methods on this challenge, like Multi-Hopper or OpNet [76; 96], have reused well-established neural network building blocks.
Others have attempted to build their system on top of a supervised pre-trained bounding-box-based object detector [15; 97].
Most recently, current best results were reported using a video-level TFCNet [95], which makes efficient use of spatio-temporal convolutions on long videos.
In contrast, Loci effectively combines an entity-oriented representational format with several kinds of interactive neural processing modules.
While Loci disentangles location from identity when trained in a fully self-supervised manner, it can be further fine-tuned with supervision to minimize location-specific prediction error.
Due to the employed inductive biases, Loci yields superior performance with a fraction of the number of parameters used in other models.
M ETHODS
The core idea of Loci involves two major aspects: First, distributing the visual input across recruitable slot-modules, which try to explain the ongoing scene dynamics in a compositional, mutually exclusive manner.
A similar principle was shown to be a powerful inductive bias for emergent time series decomposition [70].
Second, splitting the encoding of a moving entity within a slot-module into its ‘what’ and ‘where’ components.
These components are represented as a latent Gestalt vector (‘what’), which can intuitively be understood as a symbolic encoding of an object’s appearance, including its shape, color, size, texture, and other kinds of visual properties, and a latent spatial code (‘where’), which explicates location, size, and relative priority [67; 80].
Moreover, we use the Siamese network approach implemented in slot-attention [60], sharing weights across slots.
In fact, Loci employs the k generates output Gestalt vectors G k and position codes P k .
dynamics and interactions are then processed based on these pairs of latent Gestalt and location encodings.
We predict the future states of latent encodings in a transition module and allow cross-slot interactions via multi-head self-attention.
Figure 1 shows the main components of Loci. Information processing in one iteration unfolds from left to right.
A detailed algorithmic description is provided in Appendix B.
Here, we provide an overview and then detail the loss computation and training procedure.
3. L OCI O VERVIEW
The slot-wise encoder is based on a conventional ResNet architecture.
The input consists of eight main features, which encode the image frame I t at current iteration t , the prediction error map E t , a common background mask ˆ M tbg as well as, specifically for every slot k , a content mask ˆ M tk , a position map ˆ Q tk , and maps for both the other masks ˆ M t,sk and the other positions ˆ Q t,sk (cf.
Figure 1).
The computation path of Gestalt and position for an individual slot is tree-shaped, starting with a shared ResNet-trunk after which the path is split into four pathways, which compute Gestalt, location, size, and priority.
This encoder design encourages that separately moving entities are encoded in individual ResNet activity dynamics as well as that each slot encodes ‘what’ (Gestalt) and ‘where’ (location, size, and priority) of the slot-encoded moving entity separately but conjointly.
Thereby, the error map E t encourages active error minimization, following the principle of predictive coding fostering attractor dynamics [44; 73].
Appendix B specifies further details.
The transition module then predicts location P tk and Gestalt dynamics G tk as well as interactions between slots.
The module is loosely based on the architecture of a transformer-like encoder [83], where several multi-head attention layers are stacked together with residual feed forward layers in
[Figure]
[Figure]
t 1 , ˆ M t 1 , ˆ M t,s 1 ,ˆ Q t 1 ,ˆ Q t,sˆ
[Figure]
tK , ˆ M tK , ˆ M t,sK ,ˆ Q tK ,ˆ Q t,sK ...
[Figure]
Slot module K ...
ˆ R [-t + 
 [Figure]-]bg , ˜ M
bg ˆ Q t + 1ˆ R t + 1 1 , ˜ M t + 1ˆ Q t + 1 K ˆ R t + 1 K , ˜ M t + 1 K ...
P o s t p r
o ce ss i n g ( d a t a f u s i o n )
t+1 I t
+ˆ R t + 1 , E t + 1 , ˆ M t + 1 bg ˆ R t + 1 1 , ˆ M t + 1 1 , ˆ Q t + 1ˆ R t + 1 K
ˆ M t + 1 K , ˆ Q t + 1 K ...
[Figure]
between.
We replace the residual feed forward layers with residual GateL0RD [39] layers.
GateL0RD is a recent gated recurrent neural network module, which is very well-suited to learn and distinctively encode interaction events, which have been described as “structured, describable, memorable units of experience” [3], cf.
In accordance with our entity-focused processing approach, we apply a Siamese GateL0RD version, operating on the individual slot level while receiving information form other slots via the attention layers.
At the end of the transition module, the Gestalt code is pushed through a binarization layer, inspired by the principle of vector quantization [81].
This layer enforces an information bottleneck and thus contributes to the development of disentangled entity codes.
The slot-wise decoder recombines the ‘what’ and ‘where’ output from the transition module.
To do so, first the potential influence of each Gestalt code G k is computed over the output range by means of a 2d isotropic Gaussian parameterized by the position code P k , yielding density maps Q k .
Next, a priority-
ˆ z -based attention is applied to account for the fact that only one slot-object can be visible at any location (transparent objects are left for future work, see Algorithm-2 in Appendix B).
As a result, when two slots cover the same location, the one with the lower priority will have its feature maps set to zero.
The rest of the decoder is based on a conventional ResNet, which increases the resolution back to the video size.
Like the encoder, the decoder shares weights across all slots.
Eventually, the decoder outputs predictions of RGB entity reconstructions ˆ R t +1 k and individual mask predictions ˜
M t + k for each slot k .
The masks from all slots and from the background are then combined to construct the final output image prediction ˆ R t +1 and competitive mask predictions ˆ M t +1 k .
To fully reconstruct the image, Loci uses its simple background module, which generates background image estimates ˆ R bg and a background mask ˆ M bg .
In case of a static background, a Gaussian mixture model over the whole training set is used and a flat background mask set to the bias value θ bg .
For more dynamic backgrounds, we employ a simple auto encoder.
3. T RAINING
Loci is trained using a binary cross-entropy loss ( L BCE ) pixel-wise on the frame prediction applying rectified Adam optimization [RAdam, cf. 56].
Several regularizing losses are added to foster object permanence.
Additionally, to speed-up learning, we use truncated backpropagation trough time and enhance the gradients with an e-prop-like accumulation of previous neural activities [7].
Empirical evaluations showed that backpropagating the gradients through the networks’ inputs creates instabilities in the training process.
Thus, we detach the gradients for the latent states.
As a result, the only part of the network that needs to backpropagate gradients trough time are the GateL0RD layers.
Here, we found that using the described combination of e-prop and backpropagation is not only comparable in terms of network accuracy, it also greatly decreases training wall-clock time, as it allows the use of truncated backpropagation with length 1, effectively updating the weights after each input frame (see supplementary material for details).
Another important aspect for successful training is the use of a warmup phase, where we mask the target of the network with a foreground mask computed with a threshold τ :
[Equation]
where ˆ R bg is the background model estimate detailed above.
A black background is used instead of ˆ R bg to construct the next frame prediction during this phase. The foreground mask together with the zeroed background enforces the network to focus on reconstructing the foreground objects only.
This encourages the usage of the position-constrained encoding and decoding mechanism.
After about 30 000 updates—when the network has sufficiently learned to use the position encodings—we switch from the masked foreground reconstruction to the full reconstruction.
Object Permanence Loss To encourage object permanence, an additional loss is computed based on
Equation 2, which favors a slot that keeps encoding the same object, even if the object is temporarily occluded and thus invisible:
[Equation]
where P k and G k denote location and Gestalt encoding in slot k , D k refers to the RGB part of the decoder network, while ˆ G tk denotes the Gestalt code averaged around the last time step in which the entity was visible, and M tk denotes the mask of slot k
at time step t .
As a result, L o is only applied when the object becomes invisible, which is the case when M tk is approaching zero.
Time Persistence Loss
A second mechanism to enforce object permanence and to also regularize the network towards temporal consistent slot encodings, is a time persistent regularization loss:
[Equation]
where again D k refers to the RGB part of the decoder network and p 0 is the center position in the image.
L t essentially penalizes large visual changes in the decoded object between two consecutive time steps.
Position Change Loss
In order to encourage the network to predict small slot-position changes over time, a simple L 2 regularization loss is based on the position change between two time steps:
[Equation]
Supervision Loss Finally, for the experiments that are using a supervised target object location signal for fine-tuning, detailed in Equation 6, we added a gating network Φ that operates on the latent
Gestalt codes G tk before binarization and predicts a single softmax probability, which is used to decide which entity corresponds to the Snitch in the CATER dataset.
The location of the selected entity is then used in an L 2 -loss to foster regression to the target location provided in the dataset.
[Equation]
The final loss for the network results from adding the individual loss components, denoting the unsupervised and supervised losses, respectively:
[Equation]
E XPERIMENTS & R ESULTS
We performed unsupervised training in all experiments.
Only for the CATER challenge we additionally evaluated a version where we fine-tuned the network via supervision using Equation 6.
4. CATER-T RACKING C HALLENGE
In the CATER tracking challenge (cf.
Figure 2), the task is to locate a unique yellow object, called
Snitch, at the end of a video sequence.
During the video, different geometric objects rotate, move,
[Table]
frameframeframe
[Table]
[Figure]
and hop in a scripted random way.
By doing so, cone objects can engulf other objects and move them to another location before releasing them again, which can lead to situations where the Snitch remains hidden in the last frame of the video.
Therefore, the challenge is not only to recognize containment events, but also to track the specific cone that contains the Snitch.
For classification purposes, the 3D
Snitch position is partitioned into a 6 × 6 grid, resulting in a total of 36 classes.
In order to account for the fact that a small location error could lead to miss-classification, a common reported metric is the L 1 grid distance.
We additionally report the continuous L 2 distance with grid length 1, such that the distance is comparable to, but more informative than the L 1 grid distance.
As described in Girdhar & Ramanan [32], we split the dataset with a ratio of 70:30 into a training and test set and further put aside 20% of the training set as a validation set, leaving 56% of the original data for training.
To blend-in the supervision loss, we first set the supervision factor µ s = 0 .
01 for the first 4 epochs, fostering mostly unsupervised training, and then set µ s = 0 .
1 for the duration of the training process and to µ s = 0 . 3 during the last epochs, to give the Snitch location a weak pull towards the target location.
In order to produce labels from the purely unsupervisedly trained network, we train a separate small classifier with around 17k parameters, which only operates on the latent states of the trained Loci network with a correction and a gating network.
The correction network computes a residual for the location of each entity.
The gating network computes softmax probabilities, which are used to select the location belonging to the Snitch object similar to Equation 6.
To prevent supervised gradient flow, the classifier network module is trained once the unsupervised training has finished.
Eventually, the location and Gestalt codes are computed for the whole dataset and extracted into a separate data file, from which the classifier module is trained, again using a 70:30 train/test split.
As shown in Table 1, Loci not only surpasses all previous methods by a large margin—achieving a top 1 accuracy of 90.7% and a top 5 accuracy of 98.5% with an L 1 distance of 0.14—it also surpasses most of the existing work when the main architecture is trained purely unsupervised.
Loci learns object permanence merely by means of inductive bias and regularization.
This is shown in Figure 3, where Loci keeps the representation of the contained Snitch in memory.
While the shape details blur over time, importantly, the locating is correctly tracked to reconstruct the Snitch once it is revealed.
Figure 4 also demonstrates the tracking performance in case of several co-occurring occlusions and double contained situations, where two cones are covering the Snitch.
Here, the location output of the supervised gating network is marked in the video frames.
In additional studies (detailed in Appendix A.3.5), we furthermore investigate Loci’s object permanence and also demonstrate the effective disentanglement of position, Gestalt, size, and priority codes, which underlines Loci’s ability to separate position and Gestalt of objects from a scene without supervision.
4. M OVING MNIST
The moving MNIST (MMNIST) challenge is a dataset for video prediction [77].
The task is to predict the motion of two moving digits, with a maximum size of 28 × 28 pixels, moving independently inside a 64 × 64 pixel window and bouncing off the walls in a predictable manner.
While the dataset is usually implemented by simply bouncing the entire 28 × 28 MNIST sub-image within the 64 ×window, we first crop each digit to its actual appearance in order to obtain a realistic bounding box of the digit and thus to generate more realistic bouncing effects (once the actual number touches the border, instead of its superficial bounding box).
This removes the undesired bias in the dataset to remember each digit’s individual bounding box, individually, which is then the only way to correctly predict a bounce off a wall.
Instead, now the network can predict the bounce based on the pixel information alone.
While the task was originally formulated to predict the next 10 frames after receiving the firstframes, we also evaluate the ability to generate temporal closed-loop predictions for up to 100 frames after being only trained to predict 10 frames.
We compare Loci to the state-of-the-art approach
PhyDNet, also designed to disentangle ‘what’ from ‘where’; more specifically: Physical dynamics from unknown factors such as the digit’s Gestalt code.
While using the original code provided by the authors, we did not reach the same reported performance of PhyDNet using our unbiased
MMNIST dataloader.
Nevertheless, PhyDNet reaches a high accuracy for the 10 frame prediction, which is only slightly topped by Loci.
For extended temporal predictions, though, PhyDNet quickly dissolves the digits, while Loci preserves the Gestalt of each digit over the 100 predicted frames, as
[Table]
shown in Figure 5.
Table 2 shows a qualitative comparison between Loci and PhyDNet: While Loci consistently outperforms PhyDNet for the structural similarity index measure (SSIM, [90]), PhyDNet has a lower MSE than Loci after 30 frames.
This might be due to the blurring of PhyDNet, which accounts for the uncertainty in the position estimate at the cost of the digits shape.
4. O THER D ATASETS & S UPPLEMENTARY M ATERIAL
Apart from a video footage for the main experiments and further algorithmic details on Loci we provide additional insights and tests in the supplementary material: We evaluate the real world tracking performance of Loci on a ten hour aquarium footage found on YouTube Additionally, we examine Gestalt preservation and indicators of intuitive physics in closed loop predictions on the
CLEVRER dataset [93].
Furthermore, several ablation studies are provided.
C ONCLUSION AND F UTURE W ORK
We presented Loci—a novel location and identity disentangling artificial neural network, which learns to process object-distinct position and Gestalt encodings.
Loci significantly exceeds current state-of-the-art architectures on the CATER challenge, while requiring fewer network parameters and less supervision.
Loci thus offers a critical step towards solving the binding problem [37; 79; 84].
We particularly hope that the mechanisms proposed in this work bear potential to enrich the field of object representation learning and highlight the importance of well-chosen inductive biases.
Currently, Loci operates on static background estimates and cameras, which we denote as its main limitation.
We intend to extend the background model to incorporate rich and varying backgrounds more flexibly in future work.
Potential avenues for this extension may include an explicit, separate encoding of ego-motion and depth [52].
Furthermore, image reconstructions may be fused with subsequent image inputs in an information-driven manner.
We also expect to create even more compact object-interaction encodings following theories of event-predictive cognition and related conceptions and models in computational neuroscience [12; 14; 30; 78].
Moreover, we are excited to explore the potential of Loci to address problems of intuitive physics and causality [66; 71; 75], seeing that Loci offers a suitable compositional structure [37; 51] to enable symbolic reasoning.
Finally, we hope that Loci will also be well-combinable with reinforcement learning and model-predictive planning in system control setups to pursue active, goal-directed environmental interactions.
A CKNOWLEDGEMENT
This work received funding from the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – EXC number 2064/1 – Project number 390727645 as well as from the Cyber Valley in Tübingen, CyVy-RF-2020-15.
The authors thank the
International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Manuel
Traub and Matthias Karlbauer and the Alexander von Humboldt Foundation for supporting Martin
Butz and Sebastian Otte.
We also thank Simon Frank for the Gestalt-Code analysis in Figure
A A PPENDIX
As additional content, we first provide details on further evaluations we conducted with Loci.
In order to evaluate the tracking performance of Loci in a real world example, we trained it on ahour aquarium footage found on YouTube.
The task poses the additional challenge to cope with backgrounds that are not fully stationary.
The results are shown in Figure 6 and demonstrate that
Loci is able to track 15+ objects in a complex real world environment.
The Gestalt preserving performance of Loci for closed loop predictions are also demonstrated exemplary on the CLEVRER dataset in Figure 7.
Here, the effects of collisions of different geometric objects are predicted into the future.
While the location deviates visibly over time, Loci is able to preserve the Gestalt code also for more complex objects in a closed loop setup, which is considered specifically challenging for RNNs.
A. CATER E VALUATION D ETAILS
In order to produce the results from Table 1 we trained five networks with different initial seeds and and then evaluated each network five times with different initial seeds.
In Table 1 we reported the mean results over the five evaluation runs from our best performing network (Network 3 from
Table 3).
A. B ACKGOUND M ODEL
In order to compute an background model for datasets with a simple static background like CATER,
CLEVR or CLEVRER we use a simple Gaussian Mixture Model.
The specific function used is createBackgroundSubtractorMOG2 from OpenCV [9] which we use with a learningrate of 0.
to compute an background image based on the training set.
For more complex backgrounds like in the aquarium example where the camera is still static, we use an autoencoder ResNet that encodes the input image into a latent vector with the same size as the Gestalt code.
This latent background code is then run through an residual GateL0rd in order to capture temporal dynamics, like the chancing water level, and is then run through an ResNet decoder.
The background autoencoder is pretrained using a L1 reconstruction loss in order to focus on the most dominant features of the background and to not care so much about foreground objects.
[Table]
[Table]
[Table]
t =t =t =t = t =t =
t =
A. A BLATIONS
Using the binary cross-entropy with non binary targets, while producing valid gradients, gives little insights into the network’s actual performance.
In order to better compare different designs and to also take the objects into focus, we use a modified L 2 -loss for our ablation studies that is computed based on Equation 9:
[Equation]
Here the MSE is masked with the error between the input and the background.
As a result, we get a much higher error when the network produces a tracking error compared to a background reconstruction error.
The L 2 object loss takes into account instances, where, for example, the network overlooks an object or, makes a mistake in the prediction of the movement of an object.
It thus offers itself as a good metric for comparing the performance of different architectures during training.
A.3. G ATE L0RD VS LSTM
As shown in Figure 8, using GateL0RD within the predictor network significantly increases the
L 2 object -Loss.
Thus, it appears that GateL0RD’s piece-wise constant latent state regularization mechanism indeed suitably biases the network towards assuming object permanence
. A.3. I NPUT CHANNELS
In Figure 10, we show the networks performance in terms of L 2 object when we zero out different input channels.
[Table] [Table]
A.3. R EGULARIZATION LOSSES
In Figure 11, we show the networks performance in terms of L 2 object without certain regularization losses.
A.3. E- PROP VS BACK - PROPAGATION THROUGH TIME
In Figure 9 we compared the L 2 object loss for truncated back-propagation through time with different sequence lengths against a version where we used back-propagation together with e-prop.
In all experiments the networks are trained on the full Cater sequence with 300 frames, which are fed into the network sequentially.
We then updated and afterwards detached the gradients using truncated back-propagation through time with different time intervals.
Using e-prop informed gradients not only drastically sped up training time, but it was also more sample-efficient while achieving the same performance as truncated back-propagation through time with an interval of three time steps.
Especially interesting is the experiment where we tested truncated back-propagation with a sequence length of one, without using e-prop’s informed gradients.
In this case, the performance is significantly worse than when using e-prop’s informed gradients.
[Figure]
GateL0rd
[Figure]
[Figure]
A.3. L ESION STUDIES
In order to quantify the effect of the Gestalt and position codes, we conducted an ablation by adding normally distributed noise with standard deviations σ ∈ [0 , 0 .
1 , ..., 1 . 0] to the Gestalt and position codes provided by the encoder–right before they were forwarded to the transition module—and calculated the resulting accuracies and errors, which are reported in Figure 12 and Figure 13 for the
CATER and moving MNIST benchmarks, respectively.
Furthermore, exemplary images of according position, priority, size, and Gestalt code manipulations are presented in Figure 14 and Figure 5, again for the CATER and moving MNIST benchmarks.
Effectively, manipulating the according codes result in changes in position, priority, size, or Gestalt code, demonstrating the disentangled encoding of object features in the respective latent codes.
In particular, while the Gestalt-code manipulation results in changes of the actual numbers in the moving MNIST experiment under conservation of positions (first vs. third row of Figure 5), the position-code manipulation alters the position of the numbers in space without modifying the object shape (first vs. fourth row of Figure 5).
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
A. G ESTALT C ODE I NVESTIGATION
Here we investigate the latent landscape of the Gestalt codes learned by Loci when trained on the
CATER challenge.
To do this, we first selected a fixed amount (450) Gestalt codes created by Loci using the CATER dataset.
We then clustered these Gestalt codes using a Gaussian Mixture model and then performed a dimensionality reduction using t-SNE.
In some of the found clusters mainly the color of the objects was different, while in others the color and the shape varied.
The images from Figure 16 where created by using a a Principal component analysis (PCA) to calculate the main axes of variance in different clusters.
A random Gestalt code was chosen from a clusters and was modified by subtracting or adding a portion of the axes or in other words, the Gestalt code was used as a starting point and then new Gestalt codes were sampled walking down and up the selected axes.
These new Gestalt codes are not binary anymore but were still clipped at 0 and 1, to create meaningful inputs to the decoder.
We were able to identify an axis, that only varies the color of the objects, while the shape stays the same, and one axis, which only varies the shape, while the color stays the same.
Since these axes are disentangled they can be used in an additive manner to simultaneously change the color and shape of objects.
A. O BJECT P ERMANENCE E VALUATION
In order to analysed Loci’s object permanence abilities we evaluated how the number of time steps the Snitch was hidden affects Loci’s ability to correctly locate it.
As shown in Figure 17, where the tracking accuracy is plotted for progressively extended periods of Snitch containment.
For both the supervised and unsupervised tracking, the location error remains constant after an initial increase, even over extended time spans.
A. E - PROP FOR G ATE L0RD
GateL0RD is defined in Equation 10 till Equation 15 following Gumbsch et al.
[39] with the gating network g , the candidate network r , and the cell state c .
Next, we detail how we applied e-prop in
GateL0RD.
A.6.
[Equation]
gating network
[Equation]
candidate
[Equation]
(consequent cell
[Equation]
[Equation]
output factor p )
[Equation] [Equation]
output factor o
[Equation] [Equation]
(resulting prediction at time t )
[Equation]
A.6. E - PROP UPDATES : core derivative
and eligiblity
determination
[Equation]
(cid:26)
[Equation]
if x ≤(1 − Λ( x ) 2 ) otherwise (pseudo-derivative of gate) (
[Equation]
gating network forward eligibility propagation:
[Equation]
candidate network forward eligibility propagation:
[Equation]
Note that the partial derivatives ∂E∂θ j,i in Equation 18 and Equation 22 address the current time step t only.
Also note that ∂E∂c tj and ∂E∂r tj carry true gradient information of the current time step, backprogated through the feedforward connections of the architecture.
In contrast, original e-prop uses local approximations of the learning signal only, which does not allow to stack multiple layers without losing the exact local gradient.
[Table]
B L OCI A LGORITHM
Loci processes a sequence of RGB video-encoding images I ∈ R T × H × W × 3 .
Processing is mostly done slot-wise, whereby the system is initialized with a variable number of K processing slots.
Its main components consist of a slot-wise encoder , a transition module , and a slot-wise decoder .
Moreover, a background processing module is implemented.
The slot-wise encoder is implemented by a tree-structured ResNet-based processing encoder (see Figure 18).
The transition module processes slot-wise temporal dynamics and between-slot interaction dynamics (see Figure 19).
The slot-wise decoder is again implemented by a ResNet (see Figure 20).
For simple backgrounds, we use a
Gaussian Mixture Model to obtain a default background estimate ˆ R bg , which is used for the whole training set.
For more complex backgrounds we use an additional Auto-Encoder Module.
In the remainder, we denote scalar values by lower-case letters, tensors by upper-case letters, and vectors by bold letters.
Moreover, we denote slot-specific activities with a subscript k ∈ 1 , .., K and time by the superscript t .
We drop t for temporary values.
We now first define data and neural encoding sizes and types used throughout Loci’s processing pipeline.
We then specify neural activity initialization.
Finally, we detail the unfolding overall processing loop.
B. S LOT - WISE ENCODER Inputs
The encoder inputs at each time step t consist of:
• RGB input image
I t ∈
R
H
× W × 3 , • MSE map E t ∈
R H
× W × 1 (pixel-wise mean squared error between I t and ˆ R t ), • Slot-specific RGB image reconstructions
ˆ R tk ∈ R H
× W × 3 , • Slot-specific mask predictions ˆ M tk ∈ R
H
× W × 1 , • Slot-specific mask complements ˆ M t,sk = (cid:80) k ′ ∈{
1 ,..,K }\ k ˆ M tk ′ • Slot-specific isotropic Gaussian position map predictions ˆ Q tk ∈ R H × W , • Slot-specific Gaussian position map complements ˆ Q t,sk = (cid:80) k ′ ∈{ 1 ,..,K }\ k ˆ Q tk ′ • Background mask ˆ M tbg ∈ R H × W , which is equivalent to 1 − (cid:80) k ∈{ 1 ,..,K } ˆ M tk
Outputs Based on these inputs, the slot-wise encoder network generates latent codes, which are forwarded to the transition module:
• Slot-specific Gestalt codes G tk ∈ R D g , • Slot-specific position codes P tk ∈ R 4 encode an isotropic Gaussian ( µ tk , σ tk ) and a slotpriority code z tk ,
where D g denotes the size of the Gestalt code and µ t k ∈ R 2 .
B. T RANSITION MODULE
A transition module is used to process interaction dynamics within and between these slot-respective codes and creates a prediction for the next state, which is fed into the decoder.
The input to the transition module equals G t , P t .
It is processed across slots and per slot in the respective layers:
Multi-Head Attention predicts slot interactions (across slots), while GateL0RD predicts slot-specific dynamics (per slot).
In our main CATER implementation we use two attention layers with two heads each with GateL0RD layers in between.
Outputs The outputs of the transition module ˆ G t +1 k and ˆ P t +1 k have the same size as its inputs.
Additionally, recurrent, slot-respective hidden states ˆ H tk are maintained in the time-recurrent GateL0RD layers:
• Slot-specific position codes ˆ P tk ∈ R 4 , • Slot-specific Gestalt codes ˆ G tk ∈ R D g , • Slot-specific GateL0RD-layer-respective hidden states ˆ H tk ∈ R D h ,
where D h denotes the size of the recurrent latent states.
B. S LOT - WISE DECODER
Inputs The outputs of all slots from the transition module ˆ P t +1 and ˆ G t +1 then act as the input to the decoder.
Outputs The output of the decoder includes the slot-respective masks and RGB reconstructions:
• Slot-specific mask outputs ˆ M t + k ∈ R H × W , • Slot-specific RGB image reconstructions ˆ R t +1 k ∈ R
H
× W × 3 ,
which are then used as part of the input at the next iteration as specified above.
We generate the combined reconstructed image ˆ R t +1 by summing all slot estimates ˆ R t +1 k and the background estimate ˆ R bg weighted with their corresponding masks ˆ M t +1 k and ˆ M t +
bg , as specified further in Algorithm 1.
B. S EQUENCE I NITIALIZATION
At time step t = 1 we determine the network’s inputs based on randomly generated, fictive position and Gestalt estimates for each slot k , which are then fed through the Loci decoder module.
Initial position and Gestalt codes ˆ P 1 k and ˆ G 1 k are sampled from an isotropic Gaussian distribution
N Position ( µ p , σ p ) and a factorized Gaussian distribution N Gestalt ( µ g , σ g ) with learnable parameters µ p ∈ R 3 , σ p ∈ R and µ g ∈ R D g , σ g ∈ R D g , respectively.
The third position code value of ˆ P 1 k , that is, the Gaussian standard deviation ˆ σ 1 k , is set to 1 / width , where width denotes the number of pixels in a row, effectively setting ˆ σ 1 k to one pixel distance.
The fourth position code value, that is, the priority value ˆ z 1 k , is set to its index value ˆ z 1 k ← k , inducing an ordered priority, which biases initial random slot assignments and thus bootstraps initial slot-assignment progress.
Based on the initial codes, we generate estimates of the output mask ˆ M 1 k , reconstruction ˆ R 1 k and
Gaussian positions Q 1 k by calling the slot-wise decoder (see Algorithm 1 line 19 and following): ˆ M 1 k , ˆ R 1 k , Q 1 k ← SlotWiseDecoder ( ˆ P 1 k , ˆ G 1 k ) .
We finally determine the first RGB image reconstruction ˆ R 1 .
The hidden states of the recurrent neural network GateL0RD are initialized to zero, that is, H 1 k ← 0 .
B. M AIN PROCESSING LOOP
Loci first runs the main processing loop for ten time steps with the first video image of a particular image sequence.
It thereby bootstraps the objects into individual slots, somewhat similar to previous slot-attention approaches [60].
After the ten initial time steps, Loci keeps re-initializing positions P tk of a slot k to random values (as specified above) given that max ( ˆ M tk ) has been smaller than 0 .
5 for all time points until t .
This induces an initial active search process.
For longer video sequences, such as the Aquarium footage, this search process was also used for invisible slots, which increased the number of tracked objects, but negatively influenced object permanence.
Note that the prediction error is calculated across the three RGB channels.
It determines the MSE between the input I t and the static background R bg , multiplied per Hadamard-product with the forth square root MSE between the input I t and the predicted input ˆ R t .
The fourth square root emphasizes small differences, encouraging accurate encodings of individual entities.
In the transition module we apply a single trainable parametric bias neuron alpha, as proposed in Bachlechner et al.
[1], instead of layer-normalization.
Alpha is initialized to zero.
Its current
Algorithm
I t − ˆ R t (cid:17) 2 , axis = ’rgb’ (cid:19) 8: compute complements ˆ M t,sk , ˆ Q t,sk 9: # Slot-wise encoder: 10:
S tk ← Encoder Trunk
( I t , E t , ˆ R tk , ˆ M tk , ˆ M t,sk , ˆ Q tk , ˆ Q t,sk , ˆ M tbg ) 11:
G tk ← Encoder Gestalt ( S tk ) 12:
P ′ tk ← Encoder Position ( S tk ) 13:
P tk ← concat [ µ tk , σ tk , z tk ] ← concat [ Encoder µ ( P ′ tk ) , Encoder σ ( P ′ tk ) , Encoder z
( P ′ tk )] 14: # Transition module: 15: ˆ G t +1 , ˆ P t +1 ,
H t +1 =
TransitionModule ( G t , P t , H t ) 16:
# Gestalt binarization: 17: ˆ G t +1 ← Sigmoid ( ˆ G t +1 ) 18: ˆ G t +1 ← ˆ G t +
+ ˆ G t
+1 (1 − ˆ G t +1 )
N (0 , 1) 19: # Slot-wise decoder: 20: # p encodes all pixel positions normalized to [ − 1 , 1] , width the number of pixels in a row 21: ˆ Q t +1 k ← exp (cid:18) − ( p − ˆ µ t +1 k )2max ( 1 width , ˆ σ t +1 k )(cid:19) 22: decode k ← Priority-Based-Attention ( ˆ G t
+1 k , ˆ Q t +1 k , ˆ z t +1 ) 23:
ˆ R t +1 k , ˜ M t +1 k ← Decoder Trunk ( decode k ) 24: # Post-processing: 25: [ ˆ M t + , . . .
, ˆ M t +1 K , ˆ M t +1 bg ] ← softmax ( concat [ ˜ M t + , . . . ,
˜ M t +1 K , ˜ M bg ] , axis = ’K’ ) 26: ˆ R t +1 ← sum ( concat (cid:104)
ˆ R t +11 , .., ˆ R t +1 K , ˆ R bg (cid:105) ◦
ˆ M t +1 , axis = ’K’ )
27: end for 28: return [ ˆ R 1 . . .
ˆ R T ] value is multiplied with the output vector before the residual parts of the transition module.
These alpha-residuals enforce the predictor to initially compute the identity function.
The mechanism bootstraps Loci’s learning progress by initially focusing it on developing decoder-suitable Gestalt encodings.
[Figure]
RELU [Figure]
RELU + x x + x + x
RELU linear, 100 -> [Figure] linear,
[Figure]
->
[Figure]
RELU + x
GateL0rd
[Figure]
C P OTENTIAL N EGATIVE S OCIETAL I MPACTS
Here we take potential negative social impacts of Loci into consideration.
Orientated on the NeurIPS
Ethics Guidelines we identified two topics where Loci could potentially be used in an harmful way:
Could Loci directly facilitate injury to living beings?
At its current stage it is unlikely that Loci could directly be used in any weapons systems.
However, when developing Loci further, its unsupervised nature allows training on huge amounts of unlabeled data and then fine tune a weapons systems to identify and track a specific target.
Could Loci help develop or extend harmful forms of surveillance?
Potentially yes; Loci already works well with static cameras and backgrounds.
While we did not evaluate Loci’s performance on pedestrians, it is likely that Loci could be trained to track pedestrians in a surveillance setting.
Supervised fine-tuning could then potentially be used to identify a specific target group of people.
While these potential misuses of Loci are concerning, they are an unavoidable byproduct of advancing the field of (unsupervised) object tracking as a whole.
Seeing that current object tracking systems are mostly black boxes, particularly when it comes to deciphering how and why they track certain entities,
Loci may enable better control over what is tracked and where exactly tracking may be applied.
Hopefully, Loci can thus be used to mitigate unwanted tracking biases, or, at least, to facilitate the identification of such tracking biases.