SCALOR: S CALABLE O BJECT -O RIENTED S EQUENTIAL G ENERATIVE M ODELS
A BSTRACT
Scalability in terms of object density in a scene is a primary challenge in unsupervised sequential object-oriented representation learning.
Most of the previous models have been shown to work only on scenes with a few objects.
In this paper, we propose SCALOR, a probabilistic generative model for learning SCALable
Object-oriented Representation of a video.
With the proposed spatially-parallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude more number of objects compared to the previous state-of-the-art models.
Besides, we introduce a background module that allows SCALOR to model complex dynamic backgrounds as well as many foreground objects in the scene.
We demonstrate that SCALOR can deal with crowded scenes containing up to a hundred objects while jointly modeling complex dynamic backgrounds.
Importantly, SCALOR is the ﬁrst unsupervised object-representation model demonstrating its working in natural scenes containing several tens of moving objects. https://sites.google.com/view/scalor/home
I NTRODUCTION
Unsupervised structured representation learning for visual scenes is a key challenge in machine learning.
When a scene is properly decomposed into meaningful entities such as foreground objects and background, we can beneﬁt from numerous advantages of abstract symbolic representation.
These include interpretability, sample efﬁciency, the ability of reasoning and causal inference, as well as compositionality and transferability for better generalization.
In addition to symbols, another essential dimension is time.
Objects, agents, and spaces all operate under the governance of time.
Without accounting for temporal developments, it is often much harder if not impossible to discover certain relationships in a scene.
Among a few methods that have been proposed for unsupervised learning of object-oriented representation in temporal scenes, SQAIR (Kosiorek et al., 2018) is by far the most complete model.
As a probabilistic temporal generative model, it can learn object-wise structured representation while modeling underlying stochastic temporal transitions in the observed data.
Introducing the propagation-discovery model, SQAIR can also handle dynamic scenes where objects may disappear or be introduced in the middle of a sequence.
Although SQAIR provides promising ideas and shows the potential of this important direction, a few key challenges remain, limiting its applicability merely to synthetic toy tasks that are far simpler than typical natural scenes.
The ﬁrst and foremost limitation is scalability.
Sequentially processing every object in an image,
SQAIR has a fundamental limitation in scaling up to scenes with a large number of objects.
As such, the state-of-the-art remains at the level of modeling videos containing only a few objects such as MNIST digits per image.
Considering the complexity of typical natural scenes as well as the importance of scalable unsupervised object perception for applications such as self-driving systems, it is thus a challenge of the highest priority to scale robustly to scenes with a large number of objects.
Scaling up object-attention capacity is an important problem because it allows us to maximize the modern parallel computation that can maximize search capacity, unlike humans who can attend only a few objects at a time, resulting in time-consuming sequential scanning.
The AlphaGo system (Silver et al., 2017) is an example demonstrating the power of such parallel search (attention) beyond human capacity.
The second limitation is that SQAIR conspicuously lacks any form of background modeling and thus only copes with scenes without background, whereas natural scenes usually have a dynamic background.
Thus, a temporal generative model that can deal with dynamic backgrounds along with many foreground objects is an important step toward natural video scene understanding.
In this paper, we propose a model called SCAL able Sequential O bject-Oriented R epresentation (SCALOR), which resolves the aforementioned key limitations and hence can model complex videos with several tens of moving objects along with dynamic backgrounds, eventually making the model applicable to natural videos.
In SCALOR, we achieve scalability with respect to the object density by parallelizing both the propagation and discovery processes, reducing the parallel time complexity per scene image from O ( N ) to O (1) with N the number of objects in an image.
We also observe that the serial object processing in SQAIR which is based on an RNN not only increases the computation time but also deteriorates discovery performance.
To this end, we propose a parallel discovery model with superior discovery capacity and performance.
SCALOR can also be regarded as a generative tracking model since not only it detects object trajectories but is also able to predict trajectories into the future.
In our experiments, we demonstrate that SCALOR can model videos with nearly one hundred moving objects along with a dynamic background on synthetic datasets.
Furthermore, we showcase SCALOR on natural-scene videos containing tens of objects with a dynamic background.
The contributions of this work are: (i)
We propose the SCALOR model that signiﬁcantly improves (two orders of magnitude) scalability in object density.
It is applicable to nearly a hundred objects while maintaining comparable computation time to SQAIR.
(ii) We propose to parallelize the propagation–discovery process by introducing the propose–reject model, reducing the time complexity to O (1) .
(iii) SCALOR can model scenes with dynamic background.
(iv) SCALOR is the ﬁrst probabilistic model demonstrating its working on signiﬁcantly complex natural scenes containing tens of objects as well as background.
P RELIMINARIES : S EQUENTIAL A TTEND I NFER R EPEAT (SQAIR)
SQAIR models a sequence of images x = x 1: T by assuming that observation x t at time t is generated from a set of object latent variables z
O t = { z t,n } n ∈O t with O t the set of objects present at time t .
Latent variable z t,n corresponding to an object n consists of the factors ( z pres t,n , z where t,n , z what t,n ) , which represent the existence, pose, and appearance of the object, respectively.
SQAIR also assumes that an object can disappear or be introduced in the middle of a sequence.
To model this, it introduces the propagation–discovery model.
In propagation, a subset of currently existing objects is propagated to the next time-step and those not propagated (e.g., moved out of the scene) are deleted.
In discovery, after deciding how many objects D t will be discovered, D t objects are newly introduced into the scene.
Combining the propagated set P t and discovered set D t , we obtain the set of currently existing objects O t .
The complete process can be formalized as:
[Equation]
Here, we use z fg t (with “fg” standing for foreground) to denote z O t because in SQAIR there is no latent for background.
Due to the intractable posterior, SQAIR is trained through variational inference with the following posterior approximation:
[Equation]
SQAIR is trained using the importance-weighted autoencoder (IWAE) objective (Burda et al., 2015).
The VIMCO estimator (Mnih & Rezende, 2016) is used to backpropagate through the discrete random variables while using the reparameterization trick (Kingma & Welling, 2013; Williams, 1992) for continuous variables.
SQAIR has two main limitations in terms of scalability.
First, for propagation, SQAIR relies on an RNN which sequentially processes each object by conditioning on previously processed objects.
Second, the discovery is also sequential because it uses RNN-based discovery based on AIR (Eslami et al., 2016).
Consequently, SQAIR has time complexity of O ( |O t | ) per step t .
In Crawford & Pineau (2019b), the authors demonstrated that this sequential discovery can easily fail beyond the scale of a few objects.
Moreover, SQAIR lacks any model for background and its temporal transitions, which is important in modeling natural scenes.
SCALOR
In this section, we describe the proposed model, SCALOR. We ﬁrst describe the generative process along with the proposal-rejection mechanism to prevent propagation collapsing, and then the inference process and learning. 3. G ENERATIVE P ROCESS
SCALOR assumes that an image x t is generated by background latent z bg t and foreground latent z fg t .
The foreground is further factorized into a set of object representations z fg t = { z t,n } n ∈O t .
In
SCALOR, we represent an object by z t,n = ( z pres t,n , z where t,n , z what t,n ) .
The appearance representation z
what t,n is a continuous vector representation (e.g., as in VAE), and z where t,n is further decomposed into the center position z pos t,n , scale z scale t,n , and depth z depth t,n .
The depth representation, which is missing in SQAIR, represent the relative distance to the camera between objects and helps modeling object occlusion.
The foreground mask m t,n obtained from z what t,n is used to distinguish background and foreground.
We adopt the propagation–discovery model from SQAIR, but improve it in such a way to resolve the scalability problem.
The generative process of SCALOR can be written as:
[Equation]
where z t =
( z bg t , z fg t ) .
As shown, the generation process is decomposed into four modules: (i) propagation, (ii) discovery, (iii) background transition, and (iv) rendering.
The propagation in SCALOR is modeled as follows:
[Equation]
where p ( z pres t,n | z <t,n ) is a Bernoulli distribution with parameter β t,n .
The distributions of what and where are deﬁned only when the object is propagated.
To implement this, for each object n we assign an object-tracker RNN denoted by its hidden state h t,n .
The RNN is updated by input z t,n for all t where the object n is present in the scene.
The parameter β t,n is obtained as β t,n = f mlp ( h t,n ) .
If z pres t,n = 0 , the object n is not propagated and the tracker RNN is deleted.
Importantly, unlike the
RNN-based sequential propagation in SQAIR, the propagation in SCALOR is fully parallel.
Discovery by Proposal-Rejection
The main contribution in making our model scalable with respect to the the number of objects is our new discovery model that consists of two phases: proposal and rejection .
In the proposal phase, we assume that the target image can be covered by H × W latent grid cells, and we propose an object latent variable ˜
z t,h,w per grid cell.
This proposal phase can be written as:
[Equation]
In the rejection phase, our goal is to reject some of the proposed objects if a proposed object largely overlaps with a propagated object.
We realize this by using the mask variable m t,n .
Speciﬁcally, if the overlapping area between the mask of a proposed object and that of a propagated object is over a threshold τ , we reject the proposed object.
This procedure can be described as (i) proposal: ˜
z D t ∼ p (˜ z
D t | z P t ) and (ii) accept-reject: z D t = f accept-reject (˜ z D t , z P t , τ ) .
In this way, the ﬁnal discovery set z D t is always a subset of the proposal set ˜
z D t , i.e., z D t ⊆ ˜
z D t .
Although we use a deterministic function for the rejection, it can be a design choice to implement this as a stochastic decision.
While one rationale behind this design is to reﬂect an inductive bias of a Gestalt
[Table]
"Kfe2l2iGdo8ph/MEHW/TMNkFtOM=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexGQY9BLx4jmAckS5idzCZjZmeWeQhhyT948aCIV//Hm3/jJNmDJhY0FFXddHdFKWfa+P63V1hb39jcKm6Xdnb39g/Kh0ctLa0itEkkl6oTYU05E7RpmOG0kyqKk4jTdjS+nfntJ6o0k+LBTFIaJngoWMwINk5q9WTKre6XK37VnwOtkiAnFcjR6Je/egNJbEKFIRxr3Q381IQZVoYRTqelntU0xWSMh7TrqMAJ1WE2v3aKzpwyQLFUroRBc/X3RIYTrSdJ5DoTbEZ62ZuJ/3lda+LrMGMitYYKslgUW46MRLPX0YApSgyfOIKJYu5WREZYYWJcQCUXQrD88ipp1arBRbV2f1mp3+RxFOEETuEcAriCOtxBA5pA4BGe4RXePOm9eO/ex6K14OUzx/AH3ucP0UyPSQ==</latexit>
Propagation Mask
MaskEncoder
MaskDecoder
ImageEncoder
Bg Decoder
Bg Encoder
TrackerRNNs
[Table]
Accepted Proposal Proposal
[Table]
Reconstruction
AcceptRejectTest
ObjectDecoder
MaskDecoder
Proposal Mask
A
B
C
D
[Table]
+SvZkr</latexit> principle saying that two objects cannot coexist in the same position, we shall also see later further reasons as to why this design is effective. The ﬁnal discovery model can then be written as: [Equation] τ ) , where the acceptance model is:
[Equation]
Background Transition Unlike SQAIR, SCALOR is endowed with a background model. The background image is encoded into a D -dimension continuous vector z bg t from the background transition p ( z bg t | z bg <t , z fg t ) . The background RNN encodes the temporal transition of background images. Rendering The implementation of the rendering process is the same as in SPAIR (Crawford &
Pineau, 2019b), except that we process the objects in parallel.
Implementation details are shown in
Appendix A.
3. L EARNING AND I NFERENCE
Due to the intractability of the true posterior distribution p ( z 1: T | x 1: T ) , we train our model using variational inference with the following posterior approximation:
[Equation]
Posterior Propagation q ( z P t | z <t , x ≤ t ) is similar to the propagation in generation, except that we now provide observation x ≤ t through an RNN encoding.
Here, the propagation for each object n is done by q ( z t,n | z <t,n , x ≤ t ) using attention a t,n = f att ( x ≤ t ) on the feature map for object n .
To compute the attention, we use the previous position z pos t − 1 ,n as the center position and extract half the width and height of the convolutional feature map using bilinear interpolation.
This attention mechanism is motivated by the observation that only part of the image contains information for tracking an object, and an inductive bias that objects cannot move a large distance within a short time span (i.e. objects do not teleport).
Posterior Discovery
The posterior discovery also consists of proposal and rejection phases.
The main difference is that we now compute the proposal in spatially-parallel manner by conditioning on the observations x ≤ t , i.e., q ( ˜ z D
[Equation]
| z P t , [Equation] ≤ t ) .
Here, the observation x ≤ t is encoded into the feature map of dimensionality H × W × D using a Convolutional LSTM (Xingjian et al., 2015).
Then, from each feature we obtain ˜
z D t,h,w .
Importantly, this is done over all the feature cells h, w in parallel.
A similar approach is used in SPAIR (Crawford & Pineau, 2019b), but it infers the object latent representations sequentially and thus is difﬁcult to scale to a large number of objects (Lin et al., 2020).
Even if this spatially-parallel proposal plays a key role in making our model scalable, we also observe another challenge due to this high-capacity of the discovery module.
The problem is that the discovery module tends to dominate the propagation module and thus most of the objects in an image are explained by the discovery module, i.e., objects are re discovered at every time-step while nothing is propagated.
We call this problem propagation collapse .
Why the model should tend to explain an image through discovery while suppressing propagation?
First, the model does not care where—either from discovery or propagation—an object is sourced from as long as it can make accurate reconstruction.
Second, the propagation performs a much harder task than the discovery.
For the propagation to properly predict, tracker n needs to learn to ﬁnd the matching object from an image containing many objects.
Although the propagation attention play an importance role in balancing the discovery and propagation, we found it does not eliminate the problem of re-discovery, and without rejection its effectiveness varies across different experiment settings.
On the contrary, the discovery module does not need to solve such difﬁcult association problem because it only performs local image-to-latents encoding without associating latents of previous time-step.
Therefore, it is much easier for the discovery encoder to produce latents that are more accurate than those inferred from propagation.
If we limit the capacity of the discovery module and sequentially process objects like in SQAIR, we may mitigate this problem because the propagation module is naturally enforced to explain what this weakened discovery module cannot. This approach, however, cannot scale.
We employ two techniques to resolve the aforementioned problem.
First, we simply bias the initial network parameter so that it has a high propagation probability at the beginning of the training.
This helps the model prefer to explain the observation ﬁrst through propagation.
The second technique is our proposal-rejection mechanism, which is implemented the same way as in the generation process.
This prevents the discovery model from redundantly explaining what is already explained by the propagation module.
The posterior for discovery model can be written as:
[Equation]
where the acceptance model is p accept ( z D
[Equation]
[Equation]
[Equation]
| z
[Equation]
Posterior Background
The posterior of the background q ( z bg t | z fg t , x t ) is conditioned on the input image and currently existing objects.
Here, we provide the foreground latents so that the remaining parts in the image can be explained by the background module.
Training We train our model by maximizing the following evidence lower bound L ( θ, φ ) =
[Equation]
We use the reparameterization trick (Williams, 1992; Kingma & Welling, 2013) for continuous random variables such as z what , and the Gumbel-Softmax trick (Jang et al., 2016) for discrete variables such as z pres .
We found that our proposed model works well and robustly with these simpler training methods than what is used in SQAIR, i.e., VIMCO and IWAE.
R ELATED W ORK
Different approaches have been taken to tackle the problem of unsupervised scene representation learning.
Object-oriented models such as AIR (Eslami et al., 2016) and SPAIR (Crawford & Pineau, 2019b) decomposes scenes into latent variables representing the appearance, position and size of the underlying objects. While AIR makes use of a recurrent neural network, SPAIR applies spatially invariant attention to extract local feature maps.
Although the latter provides better scalability than AIR, it is still limited as it performs sequential inference on objects.
SQAIR (Kosiorek et al., 2018), discussed in Section 2, extends the ideas proposed in AIR to temporal sequences.
On the other hand, scene-mixture models (Greff et al., 2017; Van Steenkiste et al., 2018; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019) decompose scenes into a collection of components, each being a full-image level representation.
Although such models allow decomposition of the input image into components, components are not object-wise disentangled as multiple objects can be in the same component. Furthermore the obtained representation does not contain explicit interpretable features [Figure] u r a c y
Tracking and Detection Accuracy
[Figure] like position and scale, etc. SPACE (Lin et al., 2020) combines both of the above approaches by using object detection for foreground and mixture decomposition for background.
It improves upon
SPAIR by parallelizing the latent inference process. DDPAE (Hsieh et al., 2018) is another object-oriented sequential generative model that models each object with an appearance and a position vector.
The model assumes the appearance of an object to be ﬁxed and thus shares the content vector across different time-steps.
NEM (Greff et al., 2017) and RNEM (Van Steenkiste et al., 2018) introduce a spatial mixture model to disentangle the scene into multiple components representing each entity.
Since each component generates a full scene image, the latent representations are not interpretable.
Tracking-By-Animation (He et al., 2019) introduces a deterministic model to tackle the task of object tracking in an unsupervised fashion.
Furthermore, there is numerous amount of work on object tracking from the computer vision community using the same “bounding box”-representation approach proposed in SCALOR (Kosiorek et al., 2017; Ning et al., 2017; Nam & Han, 2016;
Tao et al., 2016), but such methods use provided labels to tackles the problem of object tracking and thus are usually fully or semi-supervised and not probabilistic objectoriented models.
We also note that, Crawford & Pineau (2019a) has independently and concurrently developed a similar architecture to ours. This model also emphasizes the scalability problem with a similar idea motivated by parallelizing SPAIR and extending it to sequential modeling. Main differences are the usage of proposal-rejection mechanism and the background modeling that makes our model work on complex natural scenes.
E XPERIMENTS
In this section, we evaluate SCALOR quantitatively and qualitatively in comparison to SQAIR for various tasks. These tasks can be split into two classes.
The ﬁrst tasks consist of synthetic videos of Moving MNIST/dSprites and dynamic backgrounds. The second tasks contain a natural scene CCTV footage of walking pedestrians in New York Grand Central Station.
5. T ASK 1: L ARGE -S CALE MNIST AND D S PRITES S HAPES
We ﬁrst evaluate our model on datasets of moving dSprites shapes as well as moving MNIST digits.
In all experiments, the image sequence covers a 64 × 64 partial view of the center of the whole environment.
Therefore, while there is a ﬁxed number of objects in the environment at each timestep, only a subset of them are visible in the observed image.
Environment size is customized for each setting in a way that objects can conveniently move of the viewpoint completely and re-enter whithin a few time-steps.
so that We test on ﬁve different scale settings.
In each setting, the number of objects in each trajectory is sampled uniformly from an interval [ min , max ].
Each scale setting is speciﬁed with a triplet ( min , avg , max ) , where min and max are as mentioned and avg represents the average number of visible objects in the trajectories in that setting.
The ﬁve settings are referred as Very Low Density (VLD) [(2, 2.9, 4)], Low Density (LD)
[(8, 8, 11)], Medium Density (MD) [(18, 20, 24)], High Density (HD) [(50, 55, 64)] and Very High Density (VHD)
For example, in the MD setting, there are always between 18 to 25 objects in the overall environment, while only about 20 of them are visible on average in each frame.
Experiment 1: Tracking Performance
The experiment aims to evaluate the tracking performance in an environment without background.
We evaluate with the following tracking metrics: Multi- [Figure] [Figure]
Object Tracking Accuracy (MOTA), Precision-Recall of the inferred bounding boxes (Bernardin & Stiefelhagen, 2008), the reconstruction mean square error, and the counting mean average error (CountMAE).
CountMAE measures the difference between the number of predicted objects and the actual number of objects, normalized by the latter. While MOTA measures the tracking accuracy and consistency of the bounding boxes, precision-recall measures the number of false-positive and negatives, regardless of the associated ids.
For computing the MOT metrics, we choose the Euclidean distance threshold to be twice the object size in each setting.
Figure 2 shows the quantitative results of SCALOR compared to baselines. More detailed quantitative results are shown in Appendix B.
We compare the performance of the proposed model with SQAIR in the VLD setting for both
MNIST and dSprites datasets, and LD setting for dSprites.
Note that we were not able to make
SQAIR work on other settings due to the high object density.
As shown in Figure 2, SCALOR outperforms SQAIR in all these settings, obtaining signiﬁcantly higher accuracy and recall. SQAIR either misses some objects or misidentiﬁes distinct objects as one.
In addition, SCALOR has lower values of counting MAE in comparison to SQAIR, showing that SCALOR can infer the number of objects in the scene more accurately.
Furthermore, we observe that increasing the number of objects in the scene does not signiﬁcantly impede SCALOR’s tracking ability, which demonstrates the strength of SCALOR when applied to images with a high number of objects.
SCALOR is able to achieve relatively high precision-recall even for scenes containing about 100 objects.
Note that in
VHD case, the number of objects (about 90) in the ﬁrst time-step exceeds the number of detection grid cells (8 × 8) the discovery model has.
Thus, the model can only detect up to 64 objects at the ﬁrst time-step and detects the rest in the following time-steps, this results in an offset on the tracking and detection accuracy in the VHD case. This is demonstrated in Figure 10 in Appendix C. Figure 3 demonstrates qualitative performance of SCALOR on dSprites and MNIST (HD). Bounding boxes with distinct ids are represented by distinct colors to clarify tracking consistency.
Discovered objects are emphasized by white circles.
As shown in Figure 3(a), the discovery module of SCALOR can identify newly introduced objects and put them in the propagation list while the propagation module keeps tracking existing objects. Figure 3(c) shows object-wise rendering of inferred z what latent variable. For clear visualization, object-wise rendering is shown only for a subset of objects present in all the time-steps. SCALOR infers consistent object-wise representation even when objects are largely occluded.
Experiment 2: Inference with Dynamic Background
Another interesting yet challenging setting is the presence of a dynamic background.
As we can see in Figure 4, SCALOR is able to decom- [Figure] [Figure] [Figure]
pose the video sequence to a set of foreground objects as well as the dynamic background. While the background module is able to model the background dynamic, it can also model the dynamics of the whole environment including the moving objects. This usually brings competition between background module and foreground module on explaining the foreground part. However, the background module and foreground module cooperate properly in SCALOR.
We believe this is because of our modeling that the background module obtains information from the foreground part by conditioning on the foreground variable.
Table 1 also provides the tracking performance of the model for this setting to show how complex images affect the tracking quality. We observe that it achieves comparable performance to the default no-background setting. These results are provided in Table 1 under “SCALOR – BG”. Experiment 3: Future Time Prediction/Generation from Scratch
This section showcases the generation ability of SCALOR via two experiments.
The ﬁrst experiment is a test for the conditional generation.
Here, the model is provided with the ﬁrst 5 frames and then asked to generate the next 5 frames.
Latent variables at each time-step are sampled independently from the prior distribution of the propagation step conditioned on latent variables from previous time-step.
Because the focus is on the conditional generation of background and objects, discovery prior of z pres is manually set to zero.
Figure 5 shows a generated sample. The ﬁrst 5 frames (before the red line) correspond to reconstruction while the next 5 frames represent conditional generation 1 . As we can see in Figure 5, SCALOR can generate dynamic background and object trajectories that are reasonably consistent.
The second experiment showcases video generation from scratch.
In this setting, the model generates all latent variables from the discovery prior at the ﬁrst time-step.
For the following time-steps, latent variables are sampled by conditioning on the latent variables of the previous time-steps.
Object generation probability for each grid cell at the ﬁrst time-step, i.e. z pres 1 ,h,w , is set to 0.2.
Figureshows one generated sample. At the ﬁrst time-step where all variables are sampled from a ﬁxed prior, the background is generated reasonably, but the objects are generated only partially. This behavior is expected because newly introduced objects are mostly partially observed at image boundaries. This makes the inference network learns z what representation corresponding to the partial views.
Furthermore, since z what prior is a standard Gaussian, it is hard to model the multi-modal object appearance. Yet, it is interesting to see that the conditional generation of each object’s
z what gradually converges to a consistent complete view of an object.
This demonstrates that in our model, the probability of z what t ≥ 2 in propagation can actually achieve multi-modality when conditioned on z whatat the ﬁrst timestep. Furthermore, it shows that the propagation prior network is capable of maintaining a consistent representation of the object along the sequence as is reﬂected in the dataset. Therefore, SCALOR is capable of generating objects with independent appearance and behavior. [Figure]
5. T ASK 2: R EAL -W ORLD S CENARIO
This section considers the performance of SCALOR on real-world natural video, which previous models have not been able to handle due to the scalability issue and lack of a background model.
Compared to synthetic data, the challenges in this setting are signiﬁcantly more difﬁcult.
Speciﬁcally, we consider the Crowded Grand Central Station dataset (Zhou et al., 2012), which was collected from CCTV cameras at the Grand Central Station in New York City.
Due to the complex pedestrian behavior, the density of the dataset can be considered a mix of LD and HD.
Figure 7 shows the tracking result on a sample sequence.
We can see SCALOR performs reasonably well on this pedestrian tracking dataset by maintaining consistent temporal trajectories.
As shown in Figure 7(c), the background module infers the background component and reconstructs the extracted background correctly.
As for object detection, SCALOR succeeds in accurate pedestrian detection and tracking.
Furthermore, the foreground mask produced by z what provides the segmentation of each individual pedestrian, as shown in Figure 7(d).
We draw tracking trajectories in Figure 7(e) for each pedestrian in the natural scene.
Trajectories in different colors correspond to different pedestrian ids inferred by the identity of the latent variable of each object.
Additional ﬁgures and the dataset details are provided in Appendix D.
Figures 14 in Appendix D shows the conditional generation.
The ﬁrst 5 frames are the inference reconstruction while the last 5 frames are model generation.
Starting from the 6th frame, the latent transition of the propagation trackers is modeled by the sequential prior network.
In the generation process, a different prior of z pres is introduced in the discovery module to introduce new objects emerging in the scene (see Appendix E for more details).
As shown in the ﬁgure, the model tends to generate movement aligned with the previous frames.
This also applies to newly generated objects from the discovery phase as shown in Figure 14(f).
Although the trajectories are consistent, the appearance of generated objects fail to maintain their consistency across different time frames.
Noticeably, in the generated sequence the segmentation mask for each object tends to deform into a different shape.
This may stem from the imperfection during the inference that makes the learning of appearance transition difﬁcult.
Additional ﬁgures for generation are provided in Appendix D.
The ground truth trajectories of the Grand Central Station Dataset were not available at the time this experiment was conducted. We instead use negative log-likelihood (NLL) to compare our model with two baselines, a sequential VAE and a Recurrent Latent Variable Model (VRNN).
The sequential VAE has one latent variable z of dimension 64, and a sequential prior p ( z t | z <t ) .
It is similar to our background module with the same number of latent variables for encoder and decoder.
As for the VRNN, we implement a VRNN with 128 dimensions of the LSTM hidden state and latent variable z and choose a convolutional network as the image encoder and decoder.
The NLL value for our model is 28.30, and for the VAE and VRNN baseline, it is 27.59 and 27.79, respectively.
While learning a highly structured representation, SCALOR can still obtain a comparable level of generation quality. Training iteration
[Figure] k) P r o p a {+g a t i o n r a t e SCALORNoAttentionNoRejectionNoRejNoAtt 1 2 3 4 5 6 7 
+}
Training iteration (k) N u m b e r o f d
i s c
o v e r y 0 10 20 30 40 50 60 0.
T
Number of Objects
I n f e r e n ce L a t e n c y ( i n s ) SCALORSQAIR 0 10 20 30 40 0. 4.0 · 10  Training Time (in Hours)
M e a n
S qu a r e
E rr o r [Figure] 5. A BLATION S TUDY AND C OMPUTATIONAL E FFICIENCY C OMPARISON Ablation Study: We perform an ablation study on the rejection mechanism and the propagation attention mechanism.
Here, we train our model on the moving dSprites dataset where the number of objects varies from 6 to 36.
We use two metrics to evaluate different architecture. The ﬁrst metric is the propagation rate. Using this, we measure how long an object is propagated.
Since, by design, objects always stay in a scene for more than one frame, the propagation rate can be regarded as a proxy to measure the success rate of tracking through propagation. The second metric counts the number of discoveries occurred in a whole sequence. If the discovery module dominates, a high value will be observed. As we can see in Figure 8 (a), with no rejection (Rej) or attention (Att) in propagation, the propagation rate goes down to 30% in early training iterations and keeps decreasing as training goes on.
In this setting, we found that the model re-discovers objects in every frame without tracking them properly. This is also shown in Figure 8 (b) where it has a signiﬁcantly larger number of discoveries.
With the propagation attention mechanism, the propagation rate increases to 95%. This is because the attention mechanism allows the propagation to track objects more accurately.
It balances the inference capacity of discovery and propagation. Thus, it prevents the discovery module from propagation collapse.
However, as we can see in Figure 8 (b) the re-discover problem still exist in this setting. The introduction of rejection, however, makes the propagation rate converge to 1 even without propagation attention. The propagation attention together with the rejection produced more accurate boxes. Computational Efﬁciency:
In Figure 8 (c) and (d), we measure the inference time, (c), and training convergence time(d).
For measuring the inference time, we provide a hypothetical situation containing N objects in the ﬁrst frame. The model is desired to discover and propagate them accordingly. Furthermore, we set the discovery capacity of SQAIR to be the same as the number of discovery grid cells in SCALOR. Figure 8(c) demonstrates how much time one forward step takes, as the number of objects/discovery capacity increases from 4 to 64.
SQAIR’s speed decreases linearly as both its discovery and propagation mechanisms process objects sequentially.
In contrast, SCALOR does not suffer from such phenomena as discovery and propagation are done in parallel. Figure(d) shows MSE convergence vs. training time when both models are trained on the MNIST VLD setting. SCALOR converges to a lower MSE than SQAIR and it does so orders of magnitude faster.
C ONCLUSION
We introduce SCALOR, a probabilistic generative model aimed at modeling a video of crowded dynamic objects.
With the proposed parallel discovery-propagation and proposal-rejection mechanism, we improve the capacity of object density from a few objects to up to a hundred objects.
Unlike previous models, the proposed model can also deal with dynamic backgrounds.
These contributions consequently makes our model applicable, for the ﬁrst time in this line of research, to natural scenes.
An interesting future direction is to introduce more structure to the background and modeling interactions among objects and the background.
B A DDITIONAL Q UANTITATIVE RESULT
Table 1 includes Multi Object Tracking Accuracy (MOTA) as well as precision-recall of the inferred bounding boxes (Bernardin & Stiefelhagen, 2008). Table 2 provides a comparison of SCALOR to SQAIR and VRNN in terms of the reconstruction error (MSE) and negative log-likelihood (NLL). NLL is computed per pixel across the whole sequence. [Table]
[Table] C A DDITIONAL EXPERIMENTS ON D S PRITES AND MNIST C. F REQUENT D ENSE D ISCOVERY
This experiment evaluates the ability to discover many newly introduced objects across time-steps.
This is important because in many applications only key-frames of a video are available, i.e., frames at which signiﬁcant changes happen.
An example of a key-frame is a sudden change of the observer’s viewpoint.
This change of viewpoint introduces many new objects in the frame.
Figure 9 shows one such instance.
In this setting, 10-15 objects are introduced at the ﬁrst, ﬁfth, and ninth time-steps, respectively.
As is shown SCALOR is able to discover many new objects in each frame.
C.
[Figure]
H IGH D ENSITY
Figure 10 shows samples from the Very High Density experiment.
This experiment places 90– objects in the overall environment, around 90 of which are visible at every time-step on average.
The discovery module contains 8 × 8 detection grid cells and thus can only detect up to 64 objects.
Interestingly, as is shown, the model will identify as many objects as possible in the ﬁrst time-step and discover the remaining in the second time-step.
Furthermore, there exist the case that too many objects are densely packed in one local region of the space. For those regions, SCALOR will split the detection task and detects all objects through multiple frames.
C. A BILITY TO HANDLE OVERLAP AND OCCLUSION
Figure 11 shows a sample of the MNIST settings where objects move towards each other more aggressively. In this setting, there is a higher chance of overlapping and occlusion.
As shown, the identity of the objects are preserved when objects overlap with each other. [Figure] [Figure] [Figure]
We conduct three sets of experiments on generalization. In the ﬁrst experiment, we investigate generalization on longer sequences. In this setting, the model is trained on trajectories of lengthwhile being tested on trajectories of length 20. In the second experiment, we evaluate generalization in more crowded scenes.
In this setting, the model is trained on 15–20 objects and tested on 50– 60 objects.
The third experiment is the generalization of unseen objects for tracking.
The model is trained only on moving MNIST images containing digits 0 to 5 while being tested on images containing digits 6 to 9.
Figures 12 and 13 show samples from these experiments.
“SCALOR – LG” in Table.1 also provides tracking results for the “length generalization” setting.
D E XPERIMENT D ETAIL AND A DDITIONAL Q UALITATIVE RESULTS ON G RAND C ENTRAL S TATION D ATASET
For natural-scene experiments, we spatially split the video into 8 parts and create a dataset of 400k frames in total.
We choose the ﬁrst 360k frames for training and 40k frames for testing.
Since the movement of pedestrians is too slow under 25 fps in the original video, we treat every other 7 frames as two consecutive frames in the dataset sequence.
The length of the input sequence is 10, and each image is resized to 128 × 128 .
We ﬁrst provide the generation result mentioned in Section 5.2 in Figure 14. And We provide additional qualitative results in Figures 15 to 20.
Rows from top to bottom are input sequence, overall reconstruction, extracted background, foreground segmentation mask with IDs, center position from each z pos t,n latent, and extracted trajectories based on the transition of center position.
Additionally,
Figures 18 to 20 show the examples of conditional generation, where images starting from time-step 6 (after the red line) are generated sequences.
Rows from top to bottom are conditional generations of the overall images, conditional generations of background, extracted conditional generation of each segmentation mask, and conditional generation of trajectories.
[Figure]
E M ODEL A RCHITECTURE D ETAILS
In this section, we provide additional details of the architecture and hyperparameters used for pedestrian detection.
When a new frame is provided, the network uses a fully convolutional encoder to obtain a H × W feature map.
The feature map is input into a convolutional LSTM to model the sequential information along the sequence.
The convolutional LSTM is shared across discovery module and propagation module.
The discovery module and propagation module also share the z what encoder and decoder.
The encoder has a convolutional network followed by one fully connected layer, while the glimpse decoder uses a fully convolutional network with sub-pixel layer (Shi et al., 2016) for upsampling.
The background module shares a similar structure with the z what encoder and decoder. It take a 4-dimensional input, i.e. RGB and foreground mask, and output a 3-dimensional image.
We use GRUs in propagation trackers and in prior transition networks.
We choose a batch size of 20 for the natural scene experiments and a batch size of 16 for MNIST/dSprites experiments.
Learning rate is ﬁxed at 4e-5 for natural image experiments and 5e-4 for dSprites/MNIST experiments. And we use RMSProp for optimization during training.
The standard deviation of the image distribution is chosen to be 0.1 for natural experiments and 0.2 for toy experiments.
The prior for all Gaussian posteriors are set to standard normal.
In pedestrian tracking dataset, we constrain the range of z scale so that the inferred width can vary from 5.2 pixels to 11.7 pixels, and the height can vary from 12.0 to 28.8, and both with a prior of the middle value in discovery.
Similarly, we constrain z scale on synthetic dataset so that it can vary from half or 1.5 × the actual object size.
The z pos variable in propagation phase is modeled as the deviation of position from the previous time-step instead of the global coordinate.
The prior for z pres in discovery is set to be 0.1 at the beginning of training and to quickly anneal to 1e-3 for natural image experiments and 1e-4 for dSprites/MNIST experiments.
The temperature used for modelling z pres is set to be 1.0 at the beginning and anneal to 0.3 after 20k iterations.
Full details of these architecture will be released along with our code.