S CALABLE O BJECT -O RIENTED S EQUENTIAL G ENERATIVE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
The main limitation of previous approaches to unsupervised sequential objectoriented representation learning is scalability.
Most of the previous models have been shown to work only on scenes with a few objects.
In this paper, we propose SCALOR, a probabilistic generative model for SCALable sequential
Object-oriented Representation.
With the proposed spatially-parallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude more number of objects compared to the previous state-of-the-art model.
Besides, we introduce the background model so that SCALOR can model complex background along with many foreground objects.
We demonstrate that SCALOR can deal with crowded scenes containing nearly a hundred objects while modeling complex background jointly.
Importantly, SCALOR is the ﬁrst unsupervised object-representation model demonstrating its working in natural scenes containing several tens of moving objects.
I NTRODUCTION
Unsupervised structured representation learning for visual scenes is a key challenge in machine learning.
When a scene is properly decomposed into meaningful entities such as foreground objects and background, we can beneﬁt from numerous advantages of symbolic representation.
These include interpretability, sample efﬁciency, the ability of reasoning and causal inference, as well as compositionality and transferability for better generalization.
In addition to symbols, another essential dimension is time.
Objects, agents, and spaces all operate under the governance of time.
Without accounting for temporal developments, it is often much harder if not impossible to discover certain relationships in a scene.
Among a few methods that have been proposed for unsupervised object-oriented representation learning of temporal scenes, SQAIR (Kosiorek et al., 2018) is by far the most complete model.
As a probabilistic temporal generative model, it can learn object-wise structured representation while modeling underlying stochastic temporal transitions in the observed data.
Introducing the propagation-discovery model, SQAIR can also handle dynamic scenes where objects may disappear or be introduced in the middle of a sequence.
Although SQAIR provides promising ideas and shows the potential of this important direction, a few key challenges remain, limiting its applicability merely to synthetic toy tasks that are far simpler than typical natural scenes.
The ﬁrst and foremost limitation is scalability.
Sequentially processing every object in an image,
SQAIR has a fundamental limitation in scaling up to scenes with a large number of objects.
As such, the state-of-the-art remains at the level of modeling videos containing only a few objects such as MNIST digits per image.
Considering the complexity of typical natural scenes as well as the importance of scalable unsupervised object perception for applications such as self-driving systems, it is thus a challenge of the highest priority to scale robustly to scenes with a large number of objects.
Scaling up object-attention capacity is an important problem because it allows us to maximize the modern parallel computation that can maximize search capacity, unlike humans who can attend only a few objects at a time, resulting in time-consuming sequential scanning.
The
AlphaGo system (Silver et al., 2017) is an example demonstrating the power of such parallel search (attention) beyond human capacity.
Second, SQAIR conspicuously lacks any form of background modeling and thus only copes with scenes without background, whereas natural scenes usually have a complex background.
Thus, a temporal generative model that can deal with complex backgrounds along with many foreground objects is an important step toward natural video scene understanding.
In this paper, we propose a model called SCAL able Sequential O bject-Oriented R epresentation (SCALOR), which resolves the above key limitations and hence can model complex videos with several tens of moving objects along with complex backgrounds, eventually making the model applicable to natural videos .
In SCALOR, we achieve scalability with respect to the object density by parallelizing both the propagation and discovery processes, reducing the parallel time complexity per scene image to O (1) from O ( N ) with N the number of objects in an image.
We also observe that the serial object processing in SQAIR based on an RNN not only increases the computation time but also deteriorates discovery performance.
To this end, we propose a parallel discovery model with much better discovery capacity and performance.
Temporally predicting and detecting trajectories of objects, SCALOR can also be regarded as a generative tracking model.
In our experiments, we show that SCALOR can model videos with nearly one hundred moving objects along with a complex background on synthetic datasets.
Furthermore, we evaluate and demonstrate SCALOR on natural videos as well with tens of objects with a complex background.
The contribution of this work are: (i)
We propose the SCALOR model that signiﬁcantly improves (two orders of magnitude) the scalability with regard to the object density.
It is applicable to nearly a hundred objects with comparable computation time to SQAIR, which scales only to a few objects.
(ii) We propose parallelizing the propagation–discovery process by introducing the propose–reject model, reducing the time complexity to O (1) .
(iii) SCALOR can model scenes with complex background.
(iv) SCALOR is the ﬁrst probabilistic model demonstrating its working not only on natural images but also at a signiﬁcant complexity level of tens of objects and background.
P RELIMINARIES : S EQUENTIAL A TTEND I NFER R EPEAT (SQAIR)
Before introducing our model, we ﬁrst review the SQAIR model.
SQAIR models an observed sequence of images x = x 1: T by assuming that the observation x t at time t is generated from a set of object latent variables z
O t = { z t,n } n ∈O t with O t a set of objects present at time t .
Each latent variable z n for an object n consists of the factors ( z pres n , z where n , z what n ) , which represent the existence, pose, and appearance of the object, respectively.
SQAIR also assumes that an object can disappear or be introduced in the middle of a sequence.
To model this, it introduces the propagation– discovery model.
In propagation, a subset of currently existing object is propagated to the next time step and those not propagated (e.g., because an object disappears) are deleted.
In discovery, after deciding how many objects D t will be discovered, new D t objects are introduced into the scene.
Combining the propagated P t and discovered D t , we obtain the set of currently existing objects O t .
The complete process can be formalized as:
[Equation]
Here, we use z fg t to denote z O t .
Due to the intractable posterior, SQAIR is trained through variational inference with the following posterior approximation:
[Equation]
SQAIR is trained using an importance-weighted autoencoder (IWAE) objective (Burda et al., 2015).
The VIMCO estimator (Mnih & Rezende, 2016) is used to backpropagate through the discrete random variables while using the reparameterization trick (Kingma & Welling, 2013; Williams, 1992) for continuous variables.
SQAIR has two main limitations in scalability.
First, for propagation,
SQAIR relies on an RNN.
Thus, the propagation is performed sequentially by conditioning on previously processed objects.
Second, the discovery is also sequential because it uses RNN-based discovery based on AIR (Eslami et al., 2016).
Consequently, SQAIR has O ( |O t | ) time complexity per step t .
In previous work (Crawford & Pineau, 2019), it is demonstrated that this sequential discovery can easily fail beyond the scale of a few objects.
Moreover, SQAIR lacks any model for the background and its temporal transitions which is important in modeling natural scenes.
T HE P ROPOSED M ODEL : SCALOR 3. G ENERATIVE P ROCESS
SCALOR assumes that an image x t is generated by background latent z bg t and foreground latent z fg t .
The foreground is further factorized into a set of object representations z fg t = { z t,n } n ∈O t .
In
SCALOR, we represent an object by z t,n = ( z pres t,n , z where t,n , z what t,n , z depth t,n ) .
The depth representation, which is missing in SQAIR, helps model object occlusion.
The foreground mask m t,n obtained from z what t,n is used to distinguish background and foreground.
The appearance representation z
what t,n is a continuous vector representation (e.g., as in VAE), and z where t,n is further decomposed into position z pos t,n and scale z scale t,n .
We adopt the propagation–discovery model from SQAIR, but improve it in such a way to resolve the scalability problem.
The generative process of SCALOR can be written:
[Equation]
[Table]
[Equation]
where z t =
( z bg t , z fg t ) .
As shown, the generation process is decomposed into four modules: (i) propagation, (ii) discover, (iii) background transition, and (iv) rendering.
The propagation in SCALOR is modeled as follows:
[Equation]
where p ( z pres t,n | z <t,n ) is a Bernoulli distribution with parameter β t,n .
The distributions of what and where are deﬁned only when the object is propagated.
To implement this, for each object n we assign a tracker RNN denoted by its hidden state h t,n .
The RNN is updated by input z t,n for all t where the object n is present in the scene.
The parameter β t,n is obtained as β t,n = f nn ( h t,n ) .
If z pres t,n = 0 , the object n is not propagated and the tracker RNN is deleted.
Importantly, unlike the
RNN-based sequential propagation in SQAIR, the propagation in SCALOR is fully parallel.
Discovery by Proposal-Rejection.
The main contribution in making our model scalable with respect to the number of objects is our new discovery model that consists of two phases: proposal and rejection .
In the proposal phase, we assume that the target image can be divided into H × W grid cells, and we propose an object latent variable ˜
z t,h,w per grid cell.
This proposal phase can be written as:
[Equation]
In the rejection phase, our goal is to reject some of the proposed objects if a proposed object largely overlaps with a propagated object.
In our model, each object representation contains a mask variable m t,n , which is used to make the rejection decision.
Speciﬁcally, if the overlap between the mask of a proposed object and that of a propagated one is over a threshold τ , we reject the proposed object.
This procedure can be described as (i) ˜
z D t ∼ p (˜ z
D t | z P t ) and (ii) z D t = f accept (˜ z D t , z P t , τ ) and thus z D t ⊆ ˜
z D t .
Although we use a deterministic function for the rejection, it can be a design choice to implement this as a stochastic decision.
While one rationale behind this design is to reﬂect an inductive bias from physics that two objects cannot coexist in the same position, we shall also see later while discussing the inference procedure further reasons as to why this design is effective.
The ﬁnal discovery model can be written as p ( z D t | z P t ) =
p (˜ z
D t | z P t ) (cid:81) HWh,w =1 p ( z D t,h,w | z P t , ˜
z D t , τ ) , where the acceptance model is p (
[Equation]
[Equation]
[Equation]
[Equation]
Proposal independent to the propagation also worked well in our experiments.
Background Transition.
Unlike SQAIR, SCALOR is endowed with a background model.
The background image is encoded into a D -dimension continuous vector z bg t from the background transition p ( z bg t | z bg <t , z fg t ) .
The background RNN encodes the temporal transition of background images.
Rendering.
The implementation of the rendering process is the same as in SPAIR (Crawford &
Pineau, 2019), except that we process the objects in parallel.
[Table]
="Kfe2l2iGdo8ph
[Table] [Table]
="
[Table]
AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexGQY9BLx4jmAckS5idzCZjZmeWeQhhyT948aCIV/
[Table]
jJNmDJhY0FFXddHdFKWfa
[Table]
+
[Table]
+
[Table] [Table]
[Table]
+
[Table]
+
[Table]
/ex6K14OUzx/AH3ucP0UyPSQ==</latexit>
Propagation Mask
MaskEncoder
MaskDecoder
ImageEncoder
Bg Decoder
Bg Encoder
TrackerRNNs
Rejection Mask Accepted Proposal Proposal
[Table]
Image
Reconstruction
AcceptRejectTest
ObjectDecoder
MaskDecoder
Proposal Mask
A
B
C
D
[Table]
+SvZkr</latexit>
3. L EARNING AND I NFERENCE
Due to the intractability of the true posterior distribution p ( z 1: T | x 1: T ) we train our model using variational inference with the following posterior approximation:
[Equation]
Posterior Propagation q ( z P t | z <t , x ≤ t ) is similar to the propagation in generation, except that we now provide observation x ≤ t through an RNN encoding.
Here, the propagation for each object n is done by q ( z t,n | z <t,n , x ≤ t ) using attention a t,n = f att ( x ≤ t ) on the feature map for object n .
To compute the attention, we use the previous position z pos t − 1 ,n as the center position and extract half the width and height of the convolutional feature map using bilinear interpolation.
This attention mechanism is motivated by the observation that only part of the image contains information for tracking an object, and an inductive bias that objects cannot move a large distance within a short time span (i.e., objects do not teleport).
Posterior Discovery.
The posterior discovery also consists of proposal and rejection phases.
The main difference is that we now compute the proposal in spatially-parallel manner by conditioning on the observations x ≤ t , i.e., q (˜ z D t | z P t , x ≤ t ) =
(cid:81) HWh,w =1 Q (˜
z D t,h,w | z P t , x ≤ t ) .
Here, the observation x ≤ t is encoded into the feature map of dimensionality H × W × D using a Convolutional LSTM (Xingjian et al., 2015).
Then, from each feature, we obtain ˜
z
D t,h,w .
Importantly, this is done in parallel over all the feature cells h, w .
A similar approach is used in SPAIR (Crawford & Pineau, 2019), but it infers the object latent representations sequentially and thus is difﬁcult to scale to a large number of objects (Anonymous, 2019).
Even if this spatially-parallel proposal plays a key role in making our model scalable, we also observe a challenge due to this high-capacity of the discovery module.
The problem is that the discovery module tends to dominate the propagation module and thus most of the objects in an image are explained by the discovery module, i.e., objects can be re discovered at every time step while nothing is propagated.
Why the model should tend to explain an image through discovery while suppressing propagation?
First, the model does not care where—either from discovery or propagation—an object is sourced from as long as it can make accurate reconstruction.
Second, the propagation performs a much harder task than the discovery.
For the propagation to properly predict, the tracker n needs to learn to ﬁnd the matching object from the image containing many objects.
Even if we use attention for this, we observe that this is particularly difﬁcult during the early training phase.
On the contrary, the discovery module does not need to solve such difﬁcult association problem because it only performs local image-to-latents encoding without associating latents of previous time step.
Therefore, it is much easier for the discovery encoder to produce latents that are more accurate than those from propagation.
If we limit the capacity of the discovery module and sequentially process objects like in SQAIR, we may mitigate this problem because the propagation module is naturally enforced to explain what the discovery module cannot, but this approach is difﬁcult to scale.
We employ two methods to resolve this problem.
First, we simply bias the initial network parameter so that it has a high propagation probability at the beginning of the training.
This helps the model try to explain the observation ﬁrst through propagation and then the discovery module takes the remaining as the training proceeds.
The second method is our proposal-rejection mechanism, which is implemented the same way as in the generation process.
This prevents the discovery model from redundantly explaining what is already explained by the propagation module.
The ﬁnal posterior discovery model can be written as:
[Equation]
where the acceptance model is p ( z D t,
[Equation] [Equation]
z
[Equation]
Posterior Background.
The posterior of the background q ( z bg t | z fg t , x t ) is conditioned on the input image and currently existing objects.
The existing objects describe what part of the image should already have been dealt with by the foreground object models, so that the remaining should be explained by the background module.
Training.
We train our model by maximizing the following evidence lower bound L ( θ, φ ) =
[Equation]
We use the reparameterization trick (Williams, 1992; Kingma & Welling, 2013) for continuous random variables such as z what , and the Gumbel-Softmax trick (Jang et al., 2016) for discrete variables such as z pres .
We found that our proposed model works well and stably with these simpler training methods than what is used in SQAIR, i.e., VIMCO and IWAE.
R ELATED W ORK
AIR (Eslami et al., 2016) decomposes scenes into latent variables representing the appearance, position and size of the underlying object by making use of a recurrent neural network.
SPAIR (Crawford & Pineau, 2019) takes a similar approach but applies spatially invariant attention to extract local feature maps.
Although SPAIR provides better scalability than AIR, it is still limited as it performs sequential inference on objects.
Scene-mixture models (Greff et al., 2017; Van Steenkiste et al., 2018;
Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019) decompose scenes into a collection of components, each being a full-image level representation.
Although these models allow decomposition of the input image into components, multiple objects can be in a component and the obtained representation does not contain explicit features like position and scale, etc.
SQAIR (Kosiorek et al., 2018), discussed in Section 2, extends the ideas proposed in AIR to temporal sequences.
DDPAE (Hsieh et al., 2018) is another sequential generative model for object-oriented learning, which models each object in the scene with an appearance and a position vector.
The model assumes the appearance of an object to be ﬁxed and thus shares the content vector across different time steps.
NEM (Greff et al., 2017) and RNEM (Van Steenkiste et al., 2018) introduce a spatial mixture model to disentangle the scene into multiple components representing each entity.
Since each component generates a full scene image, the latent representations are not interpretable.
On another front, Tracking-By-Animation (He et al., 2019) introduces a deterministic model to tackle the task of object tracking in an unsupervised fashion.
Overall, there has numerous amount of works on object tracking from the computer vision community using the same “bounding box”-representation approach proposed in SCALOR (Kosiorek et al., 2017; Ning et al., 2017; Nam & Han, 2016;
Tao et al., 2016), but such methods use provided labels to tackles the problem of object tracking and thus are usually fully or semi-supervised and not probabilistic models.
Therefore they do not model object-wise representation.
[Table]
E XPERIMENTS
Since we cannot provide all the ﬁgures in the following section, some ﬁgures, denoted by “.A”, are moved to the Appendix.
Furthermore, We have also created a project webpage with additional qualitative examples and videos of SCALOR.
5. T ASK 1: L ARGE -S CALE MNIST AND D S PRITES S HAPES
We ﬁrst evaluate our model on a dataset of moving dSprites shapes and a dataset consisting of moving MNIST digits.
In all experiments, objects move in a constrained environment but the processed trajectory images cover only a 64 × 64 partial view of the center of the whole scene.
Therefore, while there is a ﬁxed number of objects in the environment at each time step, only a subset of them are visible in the observed image and others are outside the observed view.
We experiment on ﬁve different scale settings.
In each setting, the number of objects in each trajectory is sampled uniformly from an interval [ min , max ].
Each scale-setting is speciﬁed with a triplet ( min , avg , max ) , where min and max are as we mentioned and avg represents the average number of visible objects in all the trajectories in that setting.
The ﬁve settings investigated in this paper are referred to as Very
Low Density (VLD)
[(2, 2.9, 4)], Low Density (LD) [(10, 9.2, 12)], Medium Density (MD) [(24, 21.9, 27)], High Density (HD) [(50, 54.5, 64)] and Very High Density (VHD)
For example, in the Medium Density setting, there are always between 24 to 27 objects in the overall environment, while only ∼ 22 of them are visible on average in each viewpoint frame.
Experiment 1: Measuring Tracking Performance.
The aim of this experiment is to evaluate the ability to track moving objects in a simple environment without background.
Table 1 includes various performance measures, encompassing standard object tracking metrics such as Multi Object
Tracking Accuracy (MOTA) and Multiple Object Tracking Precision (MOTP) as well as Precision and Recall of the inferred bounding boxes (Bernardin & Stiefelhagen, 2008).
While MOTA and
MOTP measure the accuracy and consistency of the inferred bounding boxes over time, precision and recall measure the number of false positive or negatives, regardless of the associated ids.
Note that MOTP only considers the dissimilarity between the true positives and their ground truth targets, i.e. only the correctly matched trajectories are used for calculating the error.
And MOTA considers all error terms including false negative, false positive and id swaps.
For computing the MOT metrics, we chose the distance threshold to be the maximum possible Euclidean distance so that all objects are considered at the association phase.
MAE is mean average error which measures the absolute difference between the number of predicted objects and the actual number of objects, normalized by the latter.
We compare the performance of the proposed model to SQAIR in the VLD setting for both the
MNIST and dSprites datasets, and LD setting for dSprites.
Note that we are not able to make SQAIR work on other settings.
As shown, SCALOR can outperform SQAIR in all these settings, obtaining signiﬁcantly higher accuracy and recall, as SQAIR either misses some objects or misidentiﬁes distinct objects as one.
SCALOR also has lower values of counting MAE in comparison to
[Table]
SQAIR, showing that SCALOR can infer the correct number of objects in the scene more accurately.
Furthermore, We observe that increasing the number of objects in the scene does not signiﬁcantly impede SCALOR’s tracking ability, which demonstrates the strength of SCALOR when applied to images with a high number of objects.
SCALOR is able to achieve relatively high precision and recall even for scenes containing ∼ 100 objects.
Table 2 provides a comparison of SCALOR to SQAIR and VRNN in terms of the reconstruction error (MSE) and negative log-likelihood (NLL).
NLL is computed per pixel for the thole sequence.
Figures 3.A and 4.A demonstrate the qualitative performance of SCALOR on dSprites (HD) and
MNIST (MD), respectively.
As for the moving MNIST dataset, we did not explore the High Density and Very High Density scenarios, as the resolution of MNIST digits became too small, making digits unidentiﬁable for that setting.
To better show the consistency of the inferred
Experiments on Frequent Dense Discovery, Generalization to Unseen Data, Very High Density, and Handling Occlusion: We also perform experiments to demonstrate and evaluate various other abilities of SCALOR.
Speciﬁcally, we show that SCALOR can deal with (i) frequent discovery of many objects happening in the middle of a sequence, (ii) generalization to unseen data, (iii) very high density of about 100 objects, and (iv) occlusions.
For the experiment settings and results on each of these experiments, refer to sections E.1 to E.4 in the Appendix respectively.
Although we describe these results in the Appendix due to the space constraint, we note that these results are also our major contributions.
5. T ASK 2: R EAL -W ORLD D ATASET T RACKING
This section considers the effectiveness of SCALOR when provided with noisy real-world videoframes, which previous work has not been able to handle.
The challenges faced in this setting can be signiﬁcantly more difﬁcult to overcome, due to the inherent difﬁculty compared to synthetic data.
Speciﬁcally, we consider the Crowded Grand Central Station dataset (Zhou et al., 2012), which was collected from CCTV cameras at New York City’s Grand Central Station.
Due to the complexity of pedestrian behavior, the dataset can be considered a mix of both low density and high density objects settings.
Figure 6.A shows the qualitative results of the model at inference time.
SCALOR performs reasonably well on this pedestrian tracking dataset, maintaining consistent temporal trajectories.
As shown in Figure 6.A(c), the background module infers the background composition and reconstructs the extracted background correctly.
As for the object detection, SCALOR succeeds in accurate pedestrian detection and tracking.
It could also output the foreground mask produced by z what , which can represent the segmentation of the shape of individual pedestrians, as shown in Figure 6.A(d).
We draw extracted trajectories in Figure 6.A(e) for each pedestrian in the natural scene.
Trajectories in different colors correspond to different pedestrian ids, which is given by the identity of the inferred latent variable of each object.
Figure 7.A shows qualitative results of conditional generation.
The last 5 frames are generated images conditioned on the ﬁrst 5 observations.
Starting from the 6th frame, the latent transition of the propagation trackers is modeled by the sequential prior network.
During generation, a different prior of z pres is introduced in the discovery module for capturing new objects emerging in the scene (see
[Figure]
t
[Figure]
SCALORNoBiasNoRejectionNoBiasandRejection
[Figure]
Appendix G for details). As shown in the ﬁgure, for the trajectories, the model tends to generate the same directions as aligned in the previous frames.
This also applies to newly generated objects from the discovery phase, the movement of which appears highly consistent as shown in Figure 7.A(f).
Although the trajectory movements are generated consistently, the appearance of some objects in the generation phase sometimes varies across different time frames.
Noticeably, in the generated sequence the segmentation mask for each object tends to deform into a different shape.
This may stem from the imperfection during the inference that makes the learning of appearance transition to be difﬁcult.
More qualitative results are given in Appendix E.5.
Since the ground truth trajectories of the Grand Central Station Dataset were no longer available at the time the experiment was conducted, we instead used negative log-likelihood (NLL) as the evaluation metric, and compared our model with two baselines to show the data modeling capacity, a sequential VAE and a Recurrent Latent Variable Model (VRNN).
The sequential VAE has one latent variable z of dimensionality 64, and a sequential prior p ( z t | z <t ) .
It is similar to our background module with the same number of latent variables for encoder and decoder.
As for the VRNN baseline, we implement a VRNN with 128 as the dimensionality of the LSTM hidden state and latent variable z and chose a convolutional network as the image encoder and decoder.
The NLL value for our model is 28.30, and for the VAE and VRNN baseline it is 27.59 and 27.79, respectively.
As we can see, as a discrete representation learning model that learns a semantic structured representation,
SCALOR can still attain a comparable level of generation quality to methods with continuous latent representation.
5. A BLATION S TUDY AND C OMPUTATIONAL E FFICIENCY C OMPARISON
In this section, we present the quantitative result on the ablation study and computational efﬁciency comparison.
The ablation study experiment is upon the rejection mechanism and initial bias of propagation.
We measure the rate of propagation for objects discovered at the previous timestep, as well as the total number of discovery happens in the sequence.
The result is shown in Fig.
2 (a) and (b).
As we can see in the ﬁgure, the introduction of rejection mechanism signiﬁcantly increases the propagation rate and reduce the number of discovery.
The computational efﬁciency comparison use inference latency and time of convergence as metrics and compare with generative model SQAIR.
Fig.
2 (c) and (d) shows the comparison result.
As is shown, SCALOR outperforms
SQAIR signiﬁcantly on inference latency and convergence time.
For detailed analysis for both experiments refer to section F in Appendix.
C ONCLUSION
We introduce SCALOR (SCALable Sequential Object-Oriented Representation), a scalable sequential object-oriented representation learning model via generation.
We make the model deal with two orders of magnitude more objects than previous models.
To do this, we combine the spatiallyparallel discovery with the proposal-rejection mechanism.
The proposed model can also deal with complex background.
Consequently, we signiﬁcantly improve the state-of-the-art, which was at the level of dealing with a few simple objects without background, and demonstrate that SCALOR is the ﬁrst probabilistic model that can model natural scenes with several tens of objects with complex background.
An interesting future direction is to introduce more structure to the background and modeling relationships among objects and background.
B Q UALITATIVE R ESULTS
In this section we show the qualitative results of the experiments.
C A DDITIONAL EXPERIMENT
Additional Experiment 1: Frequent Dense Discovery.
This experiment evaluates the ability to discover many newly introduced objects across many time-steps.
This is important because in many applications only key-frames of a video are available, i e., frames at which signiﬁcant changes happen.
In the object tracking domain, an example of a key-frame is one in which many objects get introduced in the same frame due to a sudden change of the observer’s view point.
Our model can handle such challenging situations where there is a need to discover many objects in a single frame, as the powerful convolutional encoders of the discovery module enable us to discover many objects in each single frame.
Figure 8 in the Appendix shows one such instance, where 10-15 objects are introduced at the ﬁrst, fourth, and seventh time-steps, respectively, although it is still possible for objects to move in and out of the scene at other time-steps as well.
Additional Experiment 2: Complex Moving Background Separation.
Another interesting yet challenging aspect of natural video processing is the presence of a complex moving background.
SCALOR is able to separate such complex moving scenes from the foreground objects, as the background inference network is explicitly designed for that purpose.
Figure 5.A provides one such
[Figure]
instance, where the model is provided with a dynamic environment consisting of moving objects on top of a complex moving background.
Figure 5.A(c) shows the inferred bounding boxes, while Figure 5.A(d) represents the inferred complex background.
SCALOR is able to disentangle foreground objects from a complex moving background, while tracking them accurately and consistently.
Even if some objects share similar colors to regions of the background, SCALOR is still able to separate them as the foreground correctly.
Table 1 also provides tracking performance of the model for this setting as well to gain a better understanding of how more complex images affect the tracking quality.
Interestingly, we observe that it attains a comparable degree of quality to the default nobackground setting.
These results are provided in Table 1 under “SCALOR – BG”.
MSE error for this case is 0.0099.
Additional Experiment 3: Generalization Ability.
We conduct three sets of experiments to evaluate the ability to generalize to settings unseen at training time.
In the ﬁrst experiment, we investigate generalization ability to longer sequences, where the model is trained on trajectories of lengthwhile being tested on trajectories of length 16.
In the second experiment,we evaluate the generalizability to more crowded scenes, where the model is trained on scenes containing 10–15 objects
[Figure]
while it is tested on scenes containing 50–60 objects.
This can be beneﬁcial in real-world applications, as sometimes the training data is only available for less crowded instances of an environment, e.g. times of the day where streets are less crowded, while the model should be able to generalize to more crowded settings as well.
The third experiment is intended to assess how well the model generalizes to unseen objects.
The model is trained only on moving MNIST images containing digitsto 5, while being tested on images containing digits 6 to 9.
Figures 9.A and 10.A in the Appendix show samples from these experiments.
“SCALOR – LG” and “SCALOR – CG” in Table.1 provide tracking results for the “length generalization” and “count generalization” experiments discussed here.
Furthermore, measured MSE is 0.0078 and 0.0081 respectively.
D A LGORITHM D ETAILS
This section contains details of the algorithms used in discovery, propagation and image rendering processes.
E A PPENDIX : A DDITIONAL E XPERIMENTAL R ESULTS E. F REQUENT D ENSE D ISCOVERY
Figure 8.A shows sample images obtained in experiment 2 (Frequent Dense Discovery).
SCALOR is able to accurately introduce new bounding boxes for the newly discovered objects.
10–15 objects are introduced at random places at certain time-steps.
Although the performance is reasonable, we noticed some limitations when the newly introduced objects overlap signiﬁcantly with the already present objects.
In such cases, sometimes the propagation detects the newly introduced object as part of the previously overlapping object and infers a new z what for the already present object rather than discovering a new object.
E. G ENERALIZATION TO UNSEEN DATA
This section presents the results obtained during generalization experiments.
Figure 9.A shows the ability of SCALOR to scale to longer time steps.
While the model has been shown only 8 time steps during training, it is tested on sequences of length 16 at test time.
Figure 10.A (a) shows the generalizability of the model to unseen objects when it is trained on scenes containing only MNIST digits of shape 1–5 but tested on scenes containing digits 6–9.
Figure 10.A (b) demonstrates bounding boxes of a sample sequence on a scene containing 50–60 objects while it was trained on scenes containing only 10–15.
E. V ERY H IGH D ENSITY
Figure 11.A shows some samples from the Very High Density experiment.
This experiment places 100–120 objects in the overall environment, 99 of which are visible at every time step on average.
However, because of memory efﬁciency, the model only contains 64 cells for discovery at each time step.
Interestingly, it is shown that the model tries its best to identify as many objects as possible in the ﬁrst time step, however, due to the lack of sufﬁcient number of cells, it will discover whatever object remains in the second time step.
Furthermore, in the case that too many objects are densely packed in one region of the space, this delayed discovery might again happen due to a lack of a sufﬁcient number of cells in that region of the space.
E. A BILITY TO HANDLE OVERLAP AND OCCLUSION
Figures 12.A and 13.A show samples of the dSprites (HD) and MNIST(MD) settings in which multiple objects overlap each other and severe occlusion happens.
As shown, the identities of the objects are preserved and identity swaps do not occur between them even if several objects overlap with each other.
E. Q UALITATIVE RESULTS FROM G RAND C ENTRAL S TATION D ATASET
We provide qualitative results in Figures 14.A to 19.A. Figures 14.A to 16.A show the examples for inference.
Rows from top to bottom are input sequence, overall reconstruction, extracted background, foreground segmentation mask with IDs, center position from each z pos t,n latent, and extracted trajectories based on the transition of center position.
Figures 17.A to 19.A show the examples from conditional generation, where images starting from column 6 are generation results (after red line).
Rows from top to bottom are conditional generations of the overall images, conditional generations of background, extracted conditional generation of each segmentation mask, and conditional generation of trajectories.
F D ETAILED A NALYSIS OF A BLATION S TUDY AND C OMPUTATIONAL E FFICIENCY C OMPARISON
Ablation Study We perform an ablation study on the proposed method to show the importance of the rejection mechanism and the initial bias of z pres in propagation step.
Here, we apply the model on the moving dSprites dataset and evaluate it using two metrics.
First metric is the propagation rate, where we measure the probability of an object being propagated after being discovered at the previous time step.
The second metric is the number of discovered objects in the whole sequence.
Figure 2 (a) (b) shows the quantitative result for both metrics.
With the use of rejection, the propagation rate converges to a value of 99%, meaning the model is able to learn to correctly propagate existing objects.
While, in the model without rejection, the propagation rate decreases to 20%.
The curve corresponding to the number of discovered objects in Figure 2 (b) further explains the behavior of the model with regards to the propagation rate.
As is shown, the model without rejection attains a higher discovery rate.
This is because when the propagation module is unable to work properly, the model has to explain the objects by re-discovering the same objects consistently at every timestep, in other words the discovery module dominates.
In contrast, model having the rejection phase obtains a relatively low discovery rate since the discovery module only needs to detect new objects in the scene.
We further compare models trained with and without some initial bias on z pres in the propagation step.
As shown in the ﬁgure, the initial bias encourages propagation in the early training iterations, which allows the model with rejection to converge faster.
Note that, in all ablation experiments, the attention mechanism in propagation is replaced by a full-image attention, i e. the propagation module takes the entire image as input.
We found that with the introduction of a local attention mechanism, the propagation rate can reach up to 92% even without the rejection step, but this value shows high variance over different training examples and does not decrease during training.
Computational Efﬁciency We also empirically compare the computational efﬁciency of SCALOR to SQAIR, presented in Figure 2.
More speciﬁcally we measure (a) inference time when model is provided with different number of objects in a scene and (b) how fast the model is trained.
In (a) we provide a hypothetical situation where there are N objects in the ﬁrst frame and the model should discover and propagate them accordingly.
Number of cells in discovery module of SCALOR as well as discovery module capacity in SQAIR are set to match this number.
Figure 2(c) demonstrates how much time one forward step takes as the number of objects/discovery capacity increases from 4 to 64.
SQAIR’s speed decreases linearly as both its discovery and propagation mechanisms process objects sequentially.
In contrast, SCALOR does not suffer from such phenomena as discovery and propagation are done in parallel.
Figure 2 (d) shows MSE Convergence error vs. training time when both models are trained on the MNIST VLD setting.
SCALOR is not only able to converge to a better minimum, but it does so orders of magnitude faster.
G M ODEL A RCHITECTURE D ETAILS
In this section, we provide additional details of the architecture and hyperparameters for the
SCALOR model on pedestrian detection.
For one input frame, the network uses a fully convolutional image encoder to obtain a H × W feature map.
The feature map is input into a convolutional
LSTM to model the sequential information along the sequence.
The convolutional LSTM is shared across discovery module and propagation module for updating the tracker hidden state, and proposing new objects.
The discovery module and propagation module share the same glimpse encoder and decoder.
The encoder has a convolutional network followed by one fully connected layer, while the glimpse decoder uses a fully convolutional network with sub-pixel layer (Shi et al., 2016) for upsampling.
The background module shares a similar structure with the glimpse encoder and decoder, while having a 4-dimensional input but producing a 3-dimensional output.
We use GRUs for trackers in propagation and all other temporal prior transition networks.
We choose a batch size of 20 for the natural scene experiments and a batch size of 4 (due to GPU memory limitation) for MNIST/dSprites experiments.
learning rate is ﬁxes at 4e-5 for natural image experiments and 1e-4 for dSprites/MNIST experiments and we use RMSProp for optimization during training.
The standard deviation of the image distribution is chosen to be 0.1.
The prior for all
Gaussian posteriors are set to standard normal.
In pedestrian tracking dataset, we constrain the range of z scale so that the inferred width can vary from 5.2 pixels to 11.7 pixels, and the height can vary from 12.0 to 28.8, and both with a prior of the middle value in discovery.
Similarly, we constrain z scale on synthetic dataset so that it can vary from 0 to twice/ or 1.5 × the actual object size depending on the experiment.
In our implementation, z pos is model as the deviation of position instead of the global position coordinate.
The prior for z pres in discovery is set to be 0.1 at the beginning of training and to quickly anneal to 1e-3 for natural image experiments and 1e-8 for dSprites/MNIST experiments.
The temperature used for modelling z pres is set to be 1.0 at the beginning and anneal to 0.5 after 20k iterations.
Since it is also interesting to see new objects introduce into the scene during generation.
We assume a distribution shift on z pres to allow more objects to be generated in discovery phase, i.e. we assume the prior for new object appearing to be 0.1.
The rest of the architecture details are described in the following tables.
Name
Value
Comment z depth dim z scale dim for x and y axis
z
what dim z pres dim z pos dim for x and y axis glimpse shape (64 , 64) for o , α image shape (128 , 128)
H
W
Image Encoder
Layer
Size/Ch.
Stride
Norm./Act.
Input
Conv 4 ×
GN(4)/CELU
Conv 4 ×
GN(8)/CELU
Conv 4 ×
GN(8)/CELU
Conv 4 ×
GN(8)/CELU
Conv 1 ×
GN(8)/CELU
Mask Encoder
Layer
Size/Ch.
Stride
Norm./Act.
Input
Conv 4 ×
GN(4)/CELU
Conv 4 ×
GN(8)/CELU
Conv 4 ×
GN(8)/CELU
Conv 4 ×
GN(8)/CELU
Conv 1 ×
GN(8)/CELU
Glimpse Encoder
Layer
Size/Ch.
Stride
Norm./Act.
Input
Conv 4 ×
GN(4)/CELU
Conv 4 ×
GN(8)/CELU
Conv 4 ×
GN(8)/CELU
Conv 4 ×
GN(16)/CELU
Conv 4 ×
GN(16)/CELU
Linear
Glimpse Decoder
Layer
Size/Ch.
Stride
Norm./Act.
Input
Conv
[Table]
GN(16)/CELU
Conv 1 ×
GN(16)/CELU
ConvSub(4)
GN(8)/CELU
Conv 3 ×
GN(8)/CELU
Conv 1 ×
GN(8)/CELU
ConvSub(4)
GN(8)/CELU
Conv 3 ×
GN(8)/CELU
Conv 1 ×
GN(8)/CELU
ConvSub(2)
GN(8)/CELU
Conv 3 ×
GN(8)/CELU
Conv 1 ×
GN(8)/CELU
ConvSub(2)
GN(4)/CELU
Conv 3 ×
GN(4)/CELU
Conv 3 ×
GN(4)/CELU
[Table] [Table]
[Table] [Table]
[Table] [Table]