O BJECT P URSUIT : B UILDING A S PACE OF O BJECTS VIA D ISCRIMINATIVE W EIGHT G ENERATION
A BSTRACT
We propose a framework to continuously learn object-centric representations for visual learning and understanding.
Existing object-centric representations either rely on supervisions that individualize objects in the scene, or perform unsuper vised disentanglement that can hardly deal with complex scenes in the real world
To mitigate the annotation burden and relax the constraints on the statistical com plexity of the data, our method leverages interactions to effectively sample diverse variations of an object and the corresponding training signals while learning the object-centric representations.
Throughout learning, objects are streamed one by one in random order with unknown identities and are associated with latent codes that can synthesize discriminative weights for each object through a convolutional hypernetwork.
Moreover, re-identiﬁcation of learned objects and forgetting prevention are employed to make the learning process efﬁcient and robust.
We perform an extensive study of the key features of the proposed framework and analyze the characteristics of the learned representations.
Furthermore, we demonstrate the capability of the proposed framework in learning representations that can improve label efﬁciency in downstream tasks.
Our code and trained models are made publicly available at: https://github com/pptrick/
Object
Pursuit
I NTRODUCTION
What are human infants and toddlers learning while they are manipulating a discovered object?
And, how do such continual interaction and learning experiences, i.e., objects are discovered and learned one by one, help develop the capability to understand the scenes that consist of individual objects?
Inspired by these questions, we aim for training frameworks that enable an autonomous agent to continuously learn object-centric representations through self-supervised discovery and manipulation of objects, so that the agent can later use the learned representations for visual scene understanding.
A majority of object-centric representation learning methods focus on encoding images or video clips into disentangled latent codes, each of which explains an entity in the scene, and together they should reconstruct the input.
However, without explicit supervision and more sophisticated inductive biases beyond parsimony, the disentanglement usually has difﬁculties aligning with objects, especially for complex scenes.
We leverage the fact that an autonomous agent can actively explore the scene, and propose that the data collected by manipulating a discovered object can serve as an important source for building inductive biases for object-level disentanglement.
In our proposed framework, whenever an object is discovered by the agent, a dataset containing images and instance masks of this object can easily be sampled via interaction compared to annotating all the objects.
Theoretically speaking, any function of the images induced by the discovered object could be a representation of the object.
For example, let φ be an encoder implemented by a neural network, and let x be the image of an object, we can say that φ (x) is a representation of the object.
Similarly, the encoder itself can also be a representation of this object since φ = arg min φ L ( φ, x) , i.e., φ is the output of an optimization procedure that takes the object’s images as input.
We employ network weights as the object-centric representations.
Speciﬁcally, the proposed method learns an object-centric representation from the data collected by manipulating a single object, through learning a latent code that can be translated into a neural network.
The neural network is produced by a discriminative weight generation hypernetwork and is able to distinguish the represented object from anything else.
In order to learn representations for objects that stream in one by one, the proposed framework is augmented with an object re-identiﬁcation procedure to avoid learning seen objects.
Moreover, we hypothesize that object representations are embedded in a low-dimensional manifold, so the proposed framework
ﬁrst checks whether a new object can be represented by learned objects; if not, the new object will be learned as a base object serving the purpose of representing future objects, thus the name object pursuit.
Furthermore, the proposed framework deals with the catastrophic forgetting of learned object representations by enforcing the hypernetwork to maintain the mapping between the learned representations and their corresponding network weights.
In summary, our work makes the following contributions: 1) we propose a novel framework named object pursuit that can continuously learn object-centric representations using training data collected from interactions with individual objects, 2) we perform an extensive study to understand the pursuit dynamics and characterize its typical behaviors regarding the key design features, and 3) we analyze the learned object space, in terms of its succinctness and effectiveness in representing objects, and empirically demonstrate its potential for label efﬁcient visual learning.
R ELATED W ORK
Object-centric representation learning falls in the ﬁeld of disentangled representation learning (Higgins et al., 2016;
Kim & Mnih, 2018; Press et al., 2019; Chen et al., 2018b; Karras et al., 2019;
Li et al., 2020; Locatello et al., 2020a; Zhou et al., 2021).
However, object-centric representations require that the disentangled latents correspond to objects in the scene.
For example, (Eslami et al., 2016; Kosiorek et al., 2018) model image formation as a structured generative process so that each component may represent an object in the generated image.
One can also apply inverse graphics (Yao et al., 2018;
Wu et al., 2017) or spatial mixture models (Greff et al., 2017; 2019; Engelcke et al., 2020b) to decompose images into interpretable latents.
Monet (Burgess et al., 2019) jointly predicts segmentation and representation with a recurrent variational auto-encoder.
Capsule autoencoders (Kosiorek et al., 2019) are proposed to decompose images into parts and poses that can be arranged into objects.
To deal with complex images or scenes, (Yang et al., 2020; Bear et al., 2020) employ motion to encourage deomposition into objects.
Besides motion, (Klindt et al., 2021) shows that the transition statistics can be informative about objects in natural videos.
Similarly, (Kabra et al., 2021) infers object latents and frame latents from videos.
Slot-attention (Locatello et al., 2020b; Jiang et al., 2020) employs the attention mechanism that aggregates features with similar appearance, while Giraffe (Niemeyer & Geiger, 2021) factorizes the scene using neural feature ﬁelds.
Even though better performance is achieved with more sophisticated network designs, scenes with complex geometry and appearance still lag.
As shown in (Engelcke et al., 2020a), the reconstruction bottleneck has critical effects on the disentanglement quality.
Instead of relying on reconstruction as a learning signal, our work calls for interactions that stimulate and collect training data from complex environments.
Rehearsal-based continual learning.
In general, continual learning methods can be divided into three streams: rehearsal-based, regularization-based, and expansion-based.
The rehearsal-based method manages buffers to replay past samples, in order to prevent from forgetting knowledge of the preceding tasks.
The regularization-based methods learn to regularize the changes in parameters of the models.
The expansion-based methods aim to expand model architectures in a dynamic manner.
Among these three types, rehearsal-based methods are widely-used due to their simplicity and effectiveness (L¨uders et al., 2016; Kemker & Kanan, 2017; Rebufﬁ et al., 2017; Cha et al., 2021;
von Oswald et al., 2019; Riemer et al., 2018; Lopez-Paz & Ranzato, 2017; Buzzega et al., 2020; Aljundi et al., 2019; Chaudhry et al., 2020; Parisi et al., 2018; Lopez-Paz & Ranzato, 2017).
Samples from previous tasks can either be the data or corresponding network activations on the data.
For example, (Shin et al., 2017) proposes a dual-model architecture where training data from learned tasks can be sampled from a generative model and (Draelos et al., 2017; Kamra et al., 2017) propose sampling in the output space of an encoder for training tasks relying on an auto-encoder architecture.
ICaRL Rebufﬁ et al.
(2017) allows adding new classes progressively based on the training samples with a small number of classes, while (Pellegrini et al., 2020; Li & Hoiem, 2017) store activations volumes at some intermediate layer to alleviate the computation and storage requirement.
Co 2 L (Cha et al., 2021) proposes continual learning within the contrastive representation learning framework, and
(Balaji et al., 2020) studies continual learning in large scale where tasks in the input sequence are not limited to classiﬁcation.
Similar to the forgetting prevention component in our framework, von
Oswald et a task-conditioned hypernetwork to rehearse the task-speciﬁc weight realizations.
refer
(Parisi
Delange et al a
Hypernetwork.
The goal of hypernetworks is to generate the weights of a target network, which is responsible for the main task (Ha et al., 2016; Krueger et al., 2017; Chung et al., 2016; Bertinetto et al., 2016; Lorraine & Duvenaud, 2018; Sitzmann et al., 2020; Nirkin et al., 2021).
For example, (Krueger et al., 2017) proposes Bayesian hypernetworks to learn the variational inference in neural networks and (Bertinetto et al., 2016) proposes to learn the network parameters in one shot.
HyperSeg (Nirkin et al., 2021) presents real-time semantic segmentation by employing a U-Net within a
U-Net architecture, and (Finn et al., 2019) applies hypernetwork to adapt to new tasks for continual lifelong learning.
Moreover, (Tay et al., 2020) proposes a new transformer architecture that leverages task-conditioned hypernetworks for controlling its feed-forward layers, whereas (Ma et al., 2021) proposes hyper-convolution, which implicitly represents the convolution kernel as a function of kernel coordinates.
Hypernetworks have shown great potential in different meta-learning settings (Rusu et al., 2018; Munkhdalai & Yu, 2017; Wang et al., 2019), mainly due to that hypernetworks are effective in compressing the primary networks’ weights as proved in (Galanti & Wolf, 2020).
M ETHOD
We consider an agent that can explore the environment and manipulate objects which are discovered in an unknown order.
Suppose there are N objects in the scene, each of which randomly appears in an image x ∈ R H
× W × 3 , whose ground-truth instance segmentation mask is
y ∈ R H
× W × N .
One can train a deep neural network that maps an image x to its mask y with a dataset D = { (x i , y i ) } that consists of such paired training samples.
However, sampling from the joint distribution p(x , y) can be extremely time-consuming, e.g., someone may have to manually draw the instance masks for every object in an image.
On the other hand, sampling from the marginals can be much more accessible through interactions.
Let D k be the dataset collected by observing an image x
i and the corresponding binary mask of the k -th object y ki ∈ R H × W , i.e., D k = { (x i , y ki ) } ∼ p(x , y k ) , which is the marginal distribution obtained by integrating out other objects’ masks in y .
The goal of the proposed object pursuit framework is to learn object-centric representations from the data collected by continuously sampling the marginals.
Next, we detail the representations used for objects (as illustrated in Fig. 1), and how we can learn them without catastrophic forgetting.
3. R EPRESENTING O BJECTS VIA D ISCRIMINATIVE W EIGHT G ENERATION
In order to represent an object, one can compute any functions of the data produced with this object.
For example, the encoding of an image containing a speciﬁc object that can be used to reconstruct the input image.
Here we take a conjugate perspective instead of asking the representation to store information of an object that is good for reconstruction.
We propose that the object-centric representation of an object shall generate the mechanisms for performing certain downstream tasks on this object, e.g., distinguishing this object from the others.
Let φ be a segmentation network with learnable weights θ that maps an image to a binary mask, i.e., φ : Θ × R H × W × 3 → R H × W .
Moreover, let ψ : ζ → Θ be the mapping from the latent space ζ to the weights of the segmentation backbone φ .
We deﬁne the object-centric representation of an object o as a latent z o ∈ ζ , such that:
[Equation]
where the expectation is computed according to p(x , y o ) , i.e., the marginal distribution of object o , and ∆ is a similarity measure between the prediction from φ and the sampled mask y o .
In other words, z o is a representation of object o , if the network weights generated from z o are capable of predicting high-quality instance masks regarding the object under the corresponding marginal distribution.
The threshold τ is a scalar parameter that will be studied in the experiments.
Now we detail the proposed object pursuit framework, which uniﬁes object re-identiﬁcation, succinctness of the representation space, and forgetting prevention, for continuously learning object representations.
3. O BJECT P URSUIT
Given the deﬁnition of object-centric representations in Eq.
1, our goal is to construct a lowdimensional manifold to embed objects in the input space ζ of the weight generation hypernetwork ψ .
We conjecture that the low-dimensional manifold can be spanned by a set of base object representations.
More explicitly, we instantiate two lists z and µ , which store the representations of the base objects and the embeddings of the learned objects, respectively.
We denote z t − 1 =
{ z i } mi = and µ t − 1 = { µ i } ni =1 ( n ≥ m , with n the number of learned objects and m the number of base objects, up to time t − 1 ) as the constructed lists after encountering a ( t − 1) -th object.
Note that µ i has the same dimension as the number of base object representations.
Similarly, we denote ψ t − 1 as the corresponding hypernetwork parameters.
As discussed, when the t -th object o t is discovered, a dataset D t = { (x j , y tj ) } can be easily sampled from the marginal distribution p(x , y t ) through interactions.
However, such object might already be seen previously.
Thus, it is necessary to apply re-identiﬁcation to avoid repetitively learning the same object.
According to the deﬁnition in Eq. 1, object o t will be claimed as a seen or learned object if the following condition is true ( | · | is the cardinality of a set):
[Equation]
with z i = µ i · z t − 1 .
In this case, object o t will be assigned the identity i ∗ that achieves the maximum value.
Otherwise, if Eq. 2 is not valid, o t is considered as an object that has not been learned.
Learning base object representations.
An object o t that can not be identiﬁed with the list of learned objects µ t − 1 can potentially serve as a base object whose representation should be added to the list of base representations z .
To ensure that object o t qualiﬁes as a base object, we propose the following test which checks whether o t can be embedded in the current manifold spanned by z t − 1 :
[Equation]
where µ ∗ is the optimal embedding for object
o t regarding z t − 1 under the (cid:96) 1 regularizer to encourage sparsity.
If the ﬁrst term of Eq.
3 passes the threshold τ with the representation µ ∗ T z t − 1 , then we consider o t as an object that should not be added to the list of bases since it can already be represented by the existing base objects.
Next, if o t does not fall on the manifold spanned by z t − 1 , a joint learning of the representation of o t and the hypernetwork ψ shall be performed so that a new base object representation can be added to the list.
However, since updating the hypernetwork could result in catastrophic forgetting of the previously learned object representations, it is also necessary to constrain the learning process, and the training loss is:
[Equation]
where the ﬁrst two terms help to ﬁnd a good representation for object o t under the sparsity constraint, and the third term enforces that the updated weight generation hypernetwork maintains the previously learned object representations.
The value of the negative scalar coefﬁcients α, β will be detailed in the experiments.
Backward redundancy removal.
The last but not the least component of the proposed object pursuit framework is to have a backward redundancy check.
Since the weight generation hypernetwork is updated to ψ t = ψ ∗ with Eq. 4, there may now exist an embedding µ ∗ (computed using Eq. 3) that re-certiﬁes object o t as an object falls on the manifold spanned by z t − 1 under ψ t .
If this is true, we set z t =
z t − 1 , otherwise, z ∗ is added to the list of base object representations since object o t is now conﬁrmed as a base object.
In some rare cases, object o t might be hard to learn, e.g., z ∗ may not satisfy the criterion described in Eq.
1 under the current hypernetwork ψ t .
In this case, we simply toss away this object so that it can be better learned in the future as the pursuit process evolves.
The proposed object pursuit framework is also summarized in Algorithm.
E XPERIMENTS
We target the learning scenario where a scene consists of multiple objects, each of them can be discovered and manipulated through interactions.
The objects are learned one by one in a continuous manner but with unknown orders.
There are two main aspects of the whole pipeline, i.e., data collection by sampling the marginals of individual objects and construction of the object-centric representations with Object Pursuit .
We focus on continuous object-centric representation learning, and thus orient our study on the behavior and characteristics of the proposed object pursuit algorithm.
We also perform experiments on one-shot and few-shot learning, and show the potential of the learned object-centric representations in effectively reducing supervisions for object detection.
Next, we brief our data collection process.
4. S ETUP
Data collection.
To learn diverse objects from variant positions and viewing angles, we collect synthetic data within the iThor environment ((Kolve et al., 2017)), which provides a set of interactive objects and scenes, as well as accurate modeling of the physics.
We collect data of 138 different objects to generate their images and masks.
The 138 objects are divided into 52 pretraining objects, 60 train objects for the pursuit process, and 25 test unseen objects.
To focus on the representation learning part, we abstract the interaction policy, and the data collection procedure of a single object can be summarized as follows: 1) Randomly set the positions of all the objects in the scene.
Calculate all available camera positions and viewing angles from which the target object (to be learned) is visible so that the sampling is effective.
The camera position, yaw angle, and pitch angle change within the range of 0.4 (grid size), 4 ◦ and 30 ◦ respectively.
For each camera position and viewing angle, we collect a 572 × 572 RGB image and a binary mask of the target object.
4) Repeat (1-3) for all objects in the stream.
Please check Fig.
2 for the sampled data.
[Table] [Table]
Network implementation
In our experiment, we use Deeplab v3+ (Chen et al., 2018a) as the segmentation network φ , which consists of 3 parts: a backbone to encode features at different levels, an aspp module, and a decoder to predict the segmentation probability per pixel.
We use resnet as the backbone (encoder), whose weights are ﬁxed both in the pretraining and the pursuit process.
The weights of the aspp module and the decoder are generated by the convolutional hypernetwork ψ .
For each convolution layer in the aspp module and the decoder, ψ takes object representation z as input, and predict weights of the convolution kernel using an upsampling convolution block.
The input representation z ﬁrst expanded to a 1024-dim vector by a linear mapping and resized to a 1 × 1 × 32 × 32 tensor.
After going through several upsampling blocks, each of which consists of an upsampling followed by a convolution and a leaky Relu, the 1 × 1 × 32 × 32 tensor turns into the output kernel weight.
For other network weights like ’running mean’ and ’running var’ in batch normalization, the hypernetwork linearly maps representation z to generate them.
Training details.
For the similarity measure ∆ , we use the dice score proposed in (Milletari et al.).
In addition to ∆ , we ﬁnd that it will be beneﬁcial to add an extra binary cross-entropy term when learning base object representations using Eq. 4.
Note, to deal with imbalanced foreground and background sizes, we also put a weighting on the entropy terms that correspond to the object so the learning can be more efﬁcient.
The sparsity constraint α is set to − 0 .
2 , − 0 .
1 for Eq. 3 and Eq.respectively, and β = − 0 .
04 for all our experiments.
To improve the convergence, we also warm up the hypernetwork using the pretraining objects.
During pretraining, each mini-batch contains training data from one object, and we randomly choose which object to use in the next batch.
In backpropagation, we update the hypernetwork ψ and representation z for each object.
When the pretraining is done, we perform a redundancy check to get rid of the objects that can be represented by others.
For simplicity, this check is performed in sequential order, and we are left with a set of base object representations to carry out the following studies.
4. O N THE R EPRESENTATION Q UALITY M EASURE
The learning dynamics and the output of Algorithm.
1, i.e., the lists of base object representations z and the learned objects µ , together with the weight generation hypernetwork ψ , are primarily affected by the representation quality measure τ introduced in Eq.
For example, τ controls whether an object will be claimed as seen, and it also determines whether or not an object falls on the manifold spanned by the current base object representations.
We study each of them in the following.
4.2. R E - IDENTIFICATION
As described in Eq. 2, when an object is discovered, it will be ﬁrst checked against the learned objects and re-identiﬁed if the maximum expected similarity passes τ .
To examine how the quality measure τ inﬂuences the re-identiﬁcation process, we run multiple object pursuit processes with different τ ’s. All runs are performed with the same training object order so that the only variant is the value of τ .
For evaluation, we preserve a separate set of 25 objects (unseen test objects) that never appear during training.
Note, among these unseen test objects, there are also objects that are similar to the training ones.
And we use 27 objects (seen test objects) from the warp-up joint training described above to check the re-identiﬁcation accuracy.
First, we check how τ affects the re-identiﬁcation for seen objects.
As reported in Tab. 1, if an object is learned and added to the object list µ , it will be claimed as seen by Eq. 2, and the re-identiﬁcation accuracy is always one.
This is true for τ varying between 0.5 and 0.8, which demonstrates the robustness of the re-identiﬁcation process against τ for objects learned.
Second, we check the behavior of the re-identiﬁcation component for unseen objects under different τ ’s along the pursuit process.
In Tab. 2, we can observe that as more and more objects are learned during the pursuit, the unseen objects that are claimed as seen from the re-identiﬁcation process also increase.
This observation is consistent across different τ ’s. Furthermore, the rate of unseen objects identiﬁed as seen converges at the end of the pursuit process, but at different levels for different τ ’s, i.e., the converged rate is lower for larger τ .
It may seem incorrect if an unseen object is claimed as seen by the reidentiﬁcation component.
However, if we examine the unseen objects (see
Fig. 3), we can see that it is quite natural for these unseen objects to be labeled as seen, because they are similar to one or multiple objects in the object list µ .
This is indeed a desired characteristic since representing or learning an object that is similar to existing ones may not be informative.
Moreover, one can adjust τ to tune the similarity level.
For example, if one insists on learning an object similar to previously seen objects, increasing the value of τ should work as evidenced by the converged rates for τ ’s in
Tab. 2.
In a nutshell, the representation quality measure τ has little effect on the re-identiﬁcation recall and accuracy for learned objects.
Yet, it controls the granularity of the learned representations by modulating the rate of unseen objects that would be identiﬁed as learned ones
. 4.2. S UCCINCTNESS AND E XPRESSIVENESS
τ .
Please see the enclosing description for the meaning of the metrics and corresponding analysis.
[Table]
We want to study how the representation quality measure τ affects the overall learning dynamics in terms of the succinctness and expressiveness of the learned base object representations.
By checking Eq. 2, Eq 3 and also the previous experiment, we conjecture that if τ is small, objects similar to the learned ones will be more easily identiﬁed as seen and certiﬁed as on the manifold.
If so, the number of objects that will be used for learning the base representations may also be small, thus increasing the succinctness of the ﬁnal representations.
Conversely, when τ increases, we would expect that more objects will contribute to the base representations, thus increasing the expressiveness.
We like to check if the observations align with our conjecture and how such behavior affects the quality of the learned bases.
To facilitate the analysis, we propose to check the following quantities: 1) | z | /N , which is the portion of objects that contribute to base representations; 2) | µ | /N , which is the portion of learnable objects that are added to the object list µ ; 3) R e , rate of objects that are conﬁrmed unseen but can be expressed by the base object representations; 4)
R f , rate of objects to be learned as base representations, which are later considered as redundant or unqualiﬁed; 5) A µ , segmentation accuracy on learned objects.
We report the above metrics across different τ ’s in Tab. 3.
As expected, a larger τ generally encourages more objects to be learned as base representations.
For example, the number of base objects learned is much larger when τ increases from 0.5 to 0.6 (ﬁrst row).
This is also evidenced by the third row of Tab. 3, which shows that the probability of an unseen object to be expressed by base rep- resentations will decrease as τ increases, creating more attempts to learn objects as bases.
However, the number of learnable objects, i.e., a base object or an object that falls on the manifold spanned by the bases, attains the maximum at a medium value τ = 0 .
6 (second row).
The underlying reason is two-fold: First, a very small τ means that many objects will be identiﬁed as seen and thus discarded to save computation; Second, a very large τ can make the qualiﬁcation of an object representation extremely difﬁcult such that it will be put aside for future learning.
The latter is also supported by the metric R f shown in the fourth row, i.e., the probability that an object will be considered redundant or unqualiﬁed after learning as a base object will increase as τ becomes large.
Lastly, when checking the quality of the base representations in expressing a common set of learned objects, we can see that the segmentation accuracy correlates with τ in a positive manner (ﬁfth row).
In general, τ directly impacts the quality of the base representations for learned objects, but its effect on the number of base representations produced by the pursuit procedure is not monotone.
Within a moderate range, we can increase τ to encourage learning more base representations, however, we may not want τ to be too large that only a few objects are qualiﬁed as base representations.
4.2. L ABEL E FFICIENCY
[Table]
Besides the training dynamics, we evaluate the usefulness of the learned object base representations in terms of how it facilitates learning the representation of a new object with only a few annotations.
For comparison, we also perform learning of the object representations over the entire representation space.
Training is similar to Eq.
The quality of the learned object representations is measured by their segmentation accuracy on test data.
As reported in Tab. 4, the quality of the few-shot learned representations increases as τ gets large, which aligns with our observation in the previous section that the expressiveness of the learned object base representations highly correlates with τ .
However, note that there is a slight drop in performance when τ increases from 0.
to 0.8 (fourth and ﬁfth column).
The reason is that as τ gets really large, it also becomes much easier to omit objects that can not pass the quality test.
As a result, the hypernetwork, which translates the representation to network weights, also gets less trained.
Thus, when tested on new objects, the performance may not match that of the trained objects for the same set of base representations, suggesting again that a moderate τ is needed to balance between the succinctness and generalization of the learned base representations.
The above observation does not hold for the representations learned over the full space.
Moreover, when comparing the performance within the low data regime, we can see that those object representations found on the manifold outperform those found in the entire space by a large margin.
For example, the new object representations found with the learned bases under τ = 0 .
7 outperform their counterparts by 57 .
5% , 29 . 2% for the 1-shot, 5-shot, and 10-shot settings, respectively.
This demonstrates the potential of using the learned base representations to help reduce the supervision needed to learn a new object.
Also, it conﬁrms that the learned base representations are meaningful since the manifold spanned by them provides a good regularity for learning unseen objects.
4. O RDER OF T RAINING O BJECTS
[Table]
The proposed object pursuit algorithm learns object representations in a stream, so we also check how the learning dynamics vary when the order of training objects changes.
We ﬁx τ to 0.6 and run ten pursuit processes with random training object order.
We reported the mean and standard deviation of the metrics proposed in Tab. 3.
As observed in
Tab. 5, the pursuit process is robust to the training object order.
4. F ORGETTING P REVENTION
[Table]
In this section, we want to check if the forgetting prevention term in Eq.
4 is effective and how it affects the pursuit dynamics.
We run pursuit processes with different values of the coefﬁcient β , where the quality measure τ and training object order is ﬁxed.
Segmentation accuracy A µ and forgetting rate γ f (i.e., how many objects falls under the quality measure after the process is ﬁnished) in Tab.
6 demonstrate the effectiveness of the forgetting prevention term: when | β | decreases, the segmentation accuracy drops, and the forgetting rate increases; when | β | vanishes, the forgetting rate reaches 97% , which means that the hypernetwork almost forgets all the object representations it previously learned.
We can also observe that both | z | /N and | µ | /N increase when | β | decreases.
This is due to the fact that when the hypernetwork forgets what are learned, any incoming object will be unlikely to be considered as seen, nor to be expressed by current bases.
So the hypernetwork tends to learn them as new base objects, which causes | z | /N to increase.
This is also evidenced by the drop in R e , which is rate of new objects that are certiﬁed as on the object manifold.
Furthermore, without the constraint of the forgetting prevention term, it is more likely to get higher accuracy in learning a new object, which decreases the number of unqualiﬁed objects.
Since the number of redundant objects and unqualiﬁed objects both drop when | β | decreases, | µ | /N increases.
Thus, in order to reduce computational cost and enforce learning meaningful representations, one would like to apply a relatively large | β | during the pursuit process.
We can also observe that as | β | changes from 0.02 to 0.1, R f increases monotonically, this is because the forgetting prevention constraint affects the quality of the learned representations, since less freedom is available when | β | is extremely large.
Consequently, fewer objects will be qualiﬁed with a good representation measured by τ .
On the other hand, R f is also high when beta is set to 0.
The reason is that when learning a new object without the constraint of the forgetting prevention term, the hypernetwork tends to overﬁt, thus making it easier for this new object to be considered as redundant, i.e., it can be expressed by the existing base representations, even though the learned representation will be forgotten by the network right after the current learning episode.
4. M ORE R ESULTS
In the appendix, we also provide ablation studies on how the sparsity constraints in Eq.
3 and Eq.affect the learning dynamics and the quality of the learned representations.
By examining the most relevant base objects for a novel object that can be expressed by the base representations (Fig. 4), we can qualitatively see that high-level concepts are learned within the representation space as objects that share similar geometry or appearance will be more correlated than others.
For curiosity, we also test the usefulness of the base object representations on real-world video objects.
As demonstrated in Fig.
7, the learned base representations can capture well the representations of real-world objects with a single learning example even if they are trained on synthetic data.
C ONCLUSION
We demonstrate that the proposed object pursuit framework can be used for continuously learning object-centric representations from data collected by manipulating a single object.
The key designs, e.g., object re-identiﬁcation, forgetting prevention, and redundancy check, all contribute to the quality of the learned base object representations.
We also show the potential of using the learned objectcentric representations for tasks at a low-annotation regime.
Especially, the learned object manifold provides a meaningful and effective prior on objects, which can facilitate downstream tasks that require object-level reasoning.
As inspired by an initial attempt on the real-world data (Fig. 7), we would also like to check the proposed object pursuit algorithm in the real world.
For example, we can train an autonomous agent to collect data from the natural 3D environment with a more efﬁcient interaction policy, and then test the learned object representations on real-world compositional visual reasoning tasks.
These are in our future research agenda.
Acknowledgement Toyota Research Institute provided funds to support this work.
It is also supported by a Vannevar Bush Faculty Fellowship and a grant from the Stanford HAI Institute.
Ethics Statement.
The proposed object pursuit framework aims at learning object representations for object-centric visual reasoning tasks.
Currently, the experiments are performed in simulation, which is publicly available and comes with a proper license.
However, how to use the learned representations could be an issue, and we will explicitly state the guideline on how to use our code and trained models ethically.
In our future research, when data collection in the real world is involved, we will consult the university ethics review committee for advice.
However, in the current form, we do not observe any signiﬁcant concerns.
Reproducibility Statement.
Our code, training data, and learned models are made publicly available.
We add detailed comments in the code so that the implementation can be easily understood.
For a preview of the implementation, please refer to the attached code in the supplementary materials.
A A PPENDIX A. A LGORITHM
Here is the Object Pursuit algorithm we describe in the method section.
In this algorithm, we ﬁrst check if the object is seen according to Eq. 2.
If the object is seen, we directly move on to the next object, otherwise, we start to check if the object can be represented by current bases according to Eq.
If it can be expressed, we add this object to the object list µ and move on to the next object, otherwise, we need to train for a new z and update the hypernetwork using loss function Eq. 4.
After training, we start a second redundancy check, with the same criterion Eq.in the ﬁrst check.
If the object can be represented by bases, we just add it to the object list µ , otherwise, we consider it a new base and add it to both base list z and object list µ .
The algorithm runs in a loop to simulate that objects are continually showing up and learned by our system.
Algorithm
Result:
A set of object representations µ T = { µ i } Ni =1 , z T = { z i } Bi =1 , and the hypernet ψ T , with T ≥ N ≥ B initialization:
z 0 = µ 0 = ∅ , ψ 0 is randomly initialized; while t ≤ T do
Sample D t = { (x j , y tj ) } ∼ p(x , y t ) ;
Check if object o t is in µ t − 1 (parameter τ ); if YES then µ t = µ t − 1 ; z t =
z t − 1 ; ψ t = ψ t − 1 ; else
Check if object o t can be represented using z t − 1 (parameter τ ); if YES then µ t = µ t − 1 (cid:83) µ o t ; z t =
z t − 1 ; ψ t = ψ t − 1 ; else
Training for z o t and ψ (cid:48) under the constraint of all objects in µ t − 1 ; ψ t = ψ (cid:48) ;
Check if z o t can now be approximated by z t − 1 ; if YES then
; else end end end end
A. A BLATION ON THE S PARSITY C ONSTRAINTS
[Table]
1. L-1 norm coefﬁcient on µ
In this ablation study, we aim to understand whether Lnorm on combination coefﬁcient µ in Eq.
3 affects an object falls on the manifold or not.
From Tab.
7 we can see that when L-1 norm coefﬁcient | α | change from 0.0 to 0.2 (absolute value), R e increases, which means more object can be expressed by current bases, causing the number of base objects ( | z | /N ) to decrease.
It shows that if we properly increase the constraint on µ and promote the sparsity of µ , it may be easier to ﬁnd a µ that can express the object.
However, if | α | exceeds a certain limit, e.g. 0.5, too much constraint added to µ (which makes µ harder to change), ﬁnding a µ to express an object becomes difﬁcult, thus the number of objects that can be expressed ( R e ) decreases, as shown in Tab. 7.
We can also see that R f ﬁrst drops then increases when | α | gets bigger.
Our reasoning is that, even though more objects can not be expressed by the bases at the ﬁrst redundancy check and have to be learned as new representations when | α | increases, the chances that they are unqualiﬁed or redundant after training also increase, making the base number continually decreases.
2. L-1 norm coefﬁcient on z
[Table] [Table]
In this ablation study, we focus on how L-1 norm on z affects object pursuit.
We change the coefﬁcient α and evaluate the pursuit process.
As Tab. 8 shows, when | α | increases, the number of objects that can be expressed by bases ( R e ) also increases, causing the base number ( | z | /N ) to decrease.
This is because regularization on z prevents the object representations from getting too far from each other, thus making z distribute more uniformly.
Since z is well distributed, it may be easier to ﬁnd a coefﬁcient µ that can express an object.
We also ﬁnd it in our experiment that when | α | gets bigger, more objects will be considered as unqualiﬁed due to their low training accuracy, especially when | α | = 0 .
It shows that more constraints on z may cause it harder to ﬁnd a proper z to represent an object during training, thus decrease the accuracy.
It also explains why the number of learnable objects ( | µ | /N ) decreases when | α | change from 0.1 to 0.5.
A. U NDERSTAND THE B ASE R EPRESENTATIONS
Fig.
4 shows some visualization results of unseen objects and the corresponding active base objects.
The results are from the pursuit process.
When an unseen object is detected, and if it can be expressed by the current bases, we then ﬁnd the combination coefﬁcient µ through Eq. 3.
In Fig.
4, we show three examples that the unseen objects can be expressed by base objects, and the combination coefﬁcient values are shown below the corresponding bases.
We show the top 5 bases that have the highest coefﬁcient value among all the bases.
In the ﬁrst example (the 1st row), the base object ’DishSponge 1’ has the highest coefﬁcient value in expressing a green cup (Cup 3).
We conjecture that ’Cup 3’ and ’DishSponge 1’ share a similar color, and green objects are rare in this set of bases, causing the DishSponge’s coefﬁcient to be the highest one. ’
Dumbell 1’ may share a similar shape with ’Cup 3’, since they are both thin in the middle and thick in the end.
The second example (the 2nd row) shows that if there is a base object (Bowl 7) that is evidently more similar to the target object (Bowl 10) than other bases, its coefﬁcient value may be the highest.
Here Bowl 7 and Bowl 10 are similar in both color and shape, but they are different in shape at the bottom of the bowl.
In the third example (the 3rd row), the black cup (Cup 2) and the black pot (Pot 1) share the same color, while the black cup (Cup 2) and the glass cup (Cup 1) share a similar shape.
These results show that the object representation space learned through object pursuit characterizes some high-level concepts (e.g. color, shape) that help better understanding the objects.
An unseen object tends to be represented by a base object with similar color or shape.
Although in some cases, understanding the base representations is not as easy as the examples shown in Fig.
4, this regular pattern is still evident in most cases.
A. M ORE V ISUAL R ESULTS ON S YNTHETIC D ATA
Fig.
5 shows more segmentation results from the experiments.
In these examples (the 1st row and the 2nd row), ’bowl’ and ’kettle’ are two unseen objects which are not in the object list µ and the base list z .
To represent an unseen object, we ﬁx the hypernetwork and try to ﬁnd a combination coefﬁcient µ ∗ to express the target object with the bases, according to Eq. 3.
The unseen object, which resides on the manifold (the object space) with latent code z can be segmented by the segmentation network generated from its representation with the hypernetwork.
Note, we test the segmentation network on the validation dataset.
Even if the unseen object is expressed by base objects which may share some similarity with it, the unseen object can still be segmented accurately from various backgrounds in different positions and angles, as Fig.
5 shows.
It veriﬁes that the learned representations, together with the hypernetwork, preserve discriminative object-centric knowledge.
A. R EAL -W ORLD D ATA C OLLECTION AND L EARNING
We propose that our framework object pursuit can be used with autonomous agents that explore the environment and interact with objects.
Through interactions, an agent could learn object-centric representations, and this is an unsupervised setting since no human annotations are required when the agent interacts with objects in the physical scene.
Currently, we perform the major experiments on synthetic data due to the lack of a robot to perform the data collection in the real world.
Moreover, we like to have a proof of concept before carrying out real-world experiments, which could involve an enormous amount of research funds and effort, which is out of the scope of our current work.
However, we are conﬁdent that transitioning from the synthetic environment to the real world is highly practical as all the technical components that need to be used in the real world are ready.
In order to evaluate how practical it is for a robot to perform the pursuit process in the real world, we need to look into two aspects.
First, how possible it is to obtain instance masks of the manipulated object.
Second, how efﬁcient the learning can be given the objects that need to be learned.
Next, we demonstrate how it is possible to get the instance masks of a real-world object through interaction.
And we discuss the second aspect in the following section.
A.5. C OLLECT D ATA IN THE R EAL -W ORLD VIA I NTERACTION
In our setting, a robot manipulates only one object at a time and learns its representation.
Two cues can be used to obtain the instance mask of the object being manipulated.
Robotic arm localization and motion.
It is easy to know where the robotic arm is within the view through calibration.
Also, motion segmentation is a well-studied topic on real-world objects in the literature (Charig Yang et al., 2021;
Yang et al., 2019).
A possible pipeline is to ﬁrst apply motion segmentation on the images, and this should give the masks of both the robotic arm and the object as they are both moving.
To remove the portion of the robotic arm, the agent can treat its arm as the ﬁrst object to learn, so that it knows how to segment its arm in the future.
By subtracting the arm, it now has the mask of the manipulated object.
Fig.
6 shows an example of getting instance masks from a video of object manipulation without supervision.
In this example, the background in the video is static (but the motion segmentation algorithm we employed can also work when there is ego-motion), and we ﬁrst segment the cube together with the robot hand using motion segmentation (Charig Yang et al., 2021).
Then we remove the robot hand from the segmentation mask, using “densecrf” (Kr¨ahenb¨uhl & Koltun, 2011) as post-processing.
As mentioned, there are other substitutes to exclude the hands, such as using a speciﬁcally trained segmentation model of the robotic arm.
The segmentation process can run at 5 frames per second.
And the results show the practicality of obtaining instance masks during manipulating objects in the real world.
A.5. R EAL -W ORLD O BJECT P URSUIT
To evaluate the efﬁciency and robustness of learning on real-world images or videos, we conduct experiments on two large-scale real-world datasets, i.e., Youtube VOS (Xu et al., 2018) and CO3D (Reizenstein et al., 2021).
YouTube-VOS We train and evaluate our framework on the Youtube-VOS dataset, which contains 65 categories.
Each category may appear in multiple video sequences, and there may be multiple instances of the same category in a video sequence.
We traverse all the categories, and for each category, we sample one video sequence from all sequences containing a single instance of this category to serve as the object we pursuit (Note this is to stimulate the interaction that should happen in the real world).
The number of frames in a video sequence typically ranges from 20 - 36.
For each video sequence, we repeat the frames many times so that the effective dataset size is around 500 samples.
For each frame, we apply horizontal ﬂip augmentation and random crop augmentation.
Speciﬁcally, we crop each frame with the same ratio along the x and y-axis, which is a random number between 0.6 and 1.
Finally, we resize the frame to 320x180 (The origin size is 1280x720).
CO3D We also test our framework on CO3D. Since processing the original dataset is too timeconsuming (18,619 objects of 50 MS-COCO categories), we randomly select 285 objects fromobject classes: apple, banana, backpack, baseball bat, bench, bicycle, book, and bottle as training objects.
The data of each object contains a video sequence that shows different views of the object.
We randomly crop each frame to a square image, then apply horizontal ﬂip augmentation.
After augmentation, the effective dataset size is about 200 samples per object.
Each frame is at the resolution 256x256.
Other
training details.
For both datasets, we initialize the hypernet with the model pretrained on synthetic data, with no initial bases and objects.
We set different quality measures τ (0.5, 0.6, 0.7, and 0.8) to evaluate our framework on real-world data.
Other settings are the same as the synthetic data experiment.
[Table] [Table]
Pursuit dynamics.
Tab. 9 and Tab.
10 show pursuit dynamics under different τ on YouTube-VOS and CO3D datasets, respectively.
The real-world data experiments share similar patterns with the synthetic data experiments: when τ gets bigger, A µ (average segmentation accuracy) and R f (proportion of unqualiﬁed and redundant objects) gets bigger, while R e (proportion of expressed objects) ﬁrst increases then decreases.
A higher τ excludes objects with poor training accuracy, thus improving the overall segmentation performance.
On the other hand, a higher τ could make an object easier to be considered as unqualiﬁed, which would be skipped and would not be learned by our system.
The number of bases and qualiﬁed objects both increase with τ , because fewer objects would be claimed as seen when τ increases, increasing the number of learned objects.
The similar patterns between real-data and synthetic data show that the conclusions we made from Tab. 3 can generalize to the real-world domain and transfer between different datasets.
Learning efﬁciency.
The running time of our framework depends on the threshold τ .
Generally, a smaller τ leads to a shorter running time since objects are easier to be expressed by bases or recognized as seen objects, which will save the time of learning base object representations.
In the synthetic data experiments, τ = 0 .
5 could ﬁnish running 67 objects in about 10 hours, while τ = 0 .
needs 1.5 days.
In the real-world experiments on youtubeVOS and CO3D datasets, our framework can learn around 80 objects per day under τ = 0 .
5 and 40 objects per day under τ = 0 . 8 .
Conclusion on real-world practicality.
With these experiments, we can conclude that the observations we make in the synthetic environment transfer to real-world data.
Moreover, two key factors that are directly related to how practical it is to run object pursuit in the real world are checked to be positive.
First, collecting data with object instance masks through interaction in the real world is practical, as veriﬁed by the effectiveness of the proposed pipeline in the previous section.
Second, the pursuit framework is robust on the real-world data, and the efﬁciency is good enough to perform learning in the real world.
For example, in a house with hundreds of objects, the training can be ﬁnished within three weeks and requires no human supervision, which is far more time-consuming and complicated in practice.
A. A DDITIONAL R ESULTS ON R E -I DENTIFICATION
In section 4.2.1, we demonstrate the impact of the quality measure τ on seen objects and unseen objects separately during re-identiﬁcation.
To further elaborate on the impact of τ , in this section, we report precision and recall on both seen objects and unseen objects (collectively) on the scale of all testing objects.
For unseen objects, we deﬁne recall as the fraction of correctly identiﬁed unseen objects among all the unseen objects, and precision is deﬁned as the fraction of correctly identiﬁed unseen objects among all the objects we identify as unseen.
Same for seen objects.
[Table]
[Table]
Tab. 11 and Tab.
12 report recall and precision on unseen objects and seen objects, taking all testing objects into consideration.
We collect the number of objects our model identiﬁes as seen or unseen from the re-identiﬁcation experiment introduced in Section 4.2.1, then compute recall and precision.
From these two tables, the precision of unseen objects and the recall of seen objects are 1.0 under all τ , which shows that a seen object can always be identiﬁed correctly.
This is crucial to our framework : if a seen object is not identiﬁed as seen, it would be problematic since we have to learn that object repeatedly, unlimitedly enlarging the object list µ .
On the other hand, the recall of unseen objects and the precision of seen objects are lower than 1.0, showing that some unseen objects could be identiﬁed as seen.
By visualizing the data (Fig. 3), we ﬁnd this situation happens only when the unseen object has substantial similarities.
, e.g., in terms of color or shape, to a seen object.
It has no signiﬁcant impact on the training process, since these “seen but actually unseen” objects contribute very little information to the learned object representations.
Therefore, our model learns object representations in a way that prevents learning the same or highly similar objects , improving efﬁciency.
We also ﬁnd that the recall of unseen objects and the precision of seen objects get larger when τ gets bigger.
It is because a larger τ increases the bar of determining an object as seen, according to
Eq. 2.
Therefore, a difference would make the model consider two as different objects.
As τ gets bigger, the learned representation becomes more discriminative, at the cost of generalization, which is a characteristic that allows us to tune the system depending on the actual need.
A. E VALUATION ON B ASE AND N ON - BASE O BJECTS
Tab. 3 shows pursuit dynamics, including segmentation accuracy and re-identiﬁcation rate with different τ for base and non-base objects.
We take all objects in the object list µ into consideration.
However, these objects are added to the object list in two different ways: some can be expressed by current bases (non-base), while others are trained and accepted as a new base (base).
For the former, we simply ﬁnd combination coefﬁcients µ without updating the hypernet, optimizing the model in much smaller parameter space than the latter.
This may cause different pursuit dynamics in these two situations.
This section analyzes the segmentation accuracy and re-identiﬁcation rate to promote the understanding on this aspect.
[Table]
As reported in Tab. 13, for both base and non-base objects, A µ shows the average segmentation accuracy, and R reid shows the proportion of the objects that are correctly re-identiﬁed as seen objects.
We report A µ and R reid under different τ .
For τ = 0 .
8 , we found that all objects that appeared in the pursuit sequence could not be expressed by bases at ﬁrst and would be learned as a new base, since the threshold τ is too high for an object to be considered as being expressed by bases.
In this case, we could not compute
A µ and R reid for non-base objects.
For other τ , the re-identiﬁcation rate R reid stays stably at 1.0, showing that if a base or non-base object has been encountered, it could be re-identiﬁed correctly.
Tab. 13 shows that A µ of both base and non-base objects increases when τ gets bigger, which shares a similar pattern with Tab. 3.
We also ﬁnd that A µ on non-base objects could be lower than A µ on base objects.
It indicates that updating the hypernet and training as a new base can perform better than simply combining bases due to a larger degree of freedom.
This difference can be reduced by increasing the capacity of the hypernetwork.
In conclusion, the re-identiﬁcation performance is stable and accurate on both base and non-base objects, and the segmentation accuracy increases with τ .
Furthermore, the segmentation accuracy of base objects is generally higher than non-base objects.
A. O NE -S HOT L EARNING ON R EAL D ATA
[Table]
We perform one-shot learning on the DAVIS 2016 dataset (Perazzi et al., 2016), a video object segmentation dataset in the real scene.
Under the one-shot learning scheme, we ﬁx the hypernet and the bases, initialize the combination coefﬁcient µ with only one training sample (ﬁrst frame in the sequence).
From µ , we can get the representation z for the training object, generate the parameters of a segmentation network using the hypernet, then evaluate the segmentation accuracy.
We ﬁrst conduct pre-training on training dataset, acquiring an updated hypernet and new bases.
We then test one-shot learning on the DAVIS evaluation set, which contains 20 video sequences.
We use the ﬁrst frame as initialization and evaluate the rest frames.
Finally, we compare our results with a set of semi-supervised video object segmentation works on the DAVIS benchmark, using the Jaccard index (IoU) as the evaluation criterion.
[Table]
Tab.
14 shows the quantitative evaluation on
DAVIS.
We report Jaccard Mean (average Jaccard score for all test objects), Jaccard Recall (average Jaccard score for test objects whose score is higher than 50.0), and Jaccard Decay (evaluate the Jaccard score decay by time).
We compare our framework with video object segmentation methods that use both appearance and temporal information, such as SEA (Avinash Ramakanth & Venkatesh Babu, 2014), HVS (Grundmann et al., 2010) and JMP (Fan et al., 2015).
We also compare our work with OSVOS (Caelles et al., 2017), a recent work of one-shot learning on video without temporal information.
To make a fair comparison, we implement OSVOS using the same structure as the primary network in our framework, and remove the post-processing.
As shown in Tab. 14, our method outperforms these four methods on both Jaccard mean and recall.
Speciﬁc experiment settings are described in A.9.
Generally speaking, our method is comparable to the state- of-the-art on one-shot object segmentation learning, though there is enough space for us to push the performance.
Furthermore, in terms of learning efﬁciency, our method performs much better, as shown in Tab. 15.
Since we only need to optimize the coefﬁcient µ with the size of | z | , the number of optimizable parameters of our method is much smaller than that of OSVOS.
So our framework is more efﬁcient in storage if the trained networks need to be transmitted and used elsewhere.
Another advantage of our framework is that the one-shot learning is much faster, which shows the potential of using our framework for real-time applications.
Fig.
7 shows some visualization results of one-shot learning on the DAVIS dataset.
Although only one sample is provided during one-shot training, our model can predict the masks on subsequent frames.
In some sequences, for example, the ’dog’ sequence showed in the 4th row of Fig.
7, the viewing angle and the object’s shape vary signiﬁcantly between frames, making it challenging to predict subsequent frames only based on the ﬁrst frame without any object prior.
It could be inferred from the results that our model contains useful object-centric priors that help segment objects in subsequent frames.
A. D ETAILS O F R EAL -W ORLD O NE -S HOT L EARNING
To achieve better performance on one-shot learning tasks, we explore one-shot learning on realworld datasets.
We ﬁnd several critical factors that would signiﬁcantly affect the one-shot learning performance from these experiments.
Section A.8 shows the ﬁnal result we acquire on one-shot learning.
In this section, we’ll show how we reach the ﬁnal score step by step.
A.9. P RETRAINING : S TREAMED L EARNING V . S . J OINT L EARNING
We propose two ways to learn objects’ representation: streamed learning and joint learning.
In streamed learning, objects are learned one by one in a certain order, as the algorithm algo.1 shows.
To prevent catastrophic forgetting problems in streamed learning, we add memory constraints with a coefﬁcient β , as Eq.
4 shows.
In joint learning, objects are learned together.
Speciﬁcally, suppose we have n objects in total, we jointly update the hypernet and n representations ( z ) during training.
Each mini-batch of training data contains data of a randomly selected object.
Different objects appear alternately, and the representation z corresponds to the current object is updated.
Forgetting problems would not occur in joint training.
In this experiment, the hypernet predicts the parameters of the whole segmentation network.
We train it on the DAVIS training set, which contains 30 objects.
The experiment found that ’joint learning’ reaches higher average segmentation accuracy (52.96) than streamed learning (49.72).
There’re two possible reasons.
First, we ﬁnd that the average validation accuracy of joint training (85.6) is higher than streamed learning (81.2), suggesting that joint learning acquires better base representations that would help represent a novel object.
Second, although the memory constraint of streamed learning helps the model remember previous knowledge, it also restricts the searching range, which would cause negative effects on ﬁnding representation for a novel object.
We’ll use joint learning for pre-training in the following experiments to reach better performance.
A.9. N ETWORK A RCHITECTURE
As the experiment section 4.1 introduces, we use an encoder-decoder network as the primary network (segmentation network).
In the last experiment (A.9.1), the hypernetwork predicts the parameters of both the encoder and the decoder.
However, there’s another option: the hypernetwork predicts the parameters of the decoder only.
In this setting, the encoder updates with the hypernetwork in pre-training, and is ﬁxed in one-shot learning.
In this experiment, we ﬁnd that such architecture differences would signiﬁcantly affect the one-shot performance.
For pre-training, we use ’joint learning’, as we mention in A.9.1.
Other settings are the same as A.9.1.
When the hypernetwork predicts the decoder only, we ﬁnd that the test score (58.95) is much higher than predicting the whole network (52.96).
In our implementation, the encoder has more network parameters than the corresponding hypernet that generates its parameters.
The hypernet could only generate a subset of all possible parameters of the encoder.
The limitation of the searching range in parameter space may cause the output result to fall into local extrema.
Since the encoders under these two settings are both ﬁxed during one-shot learning and share the same structure, we assume that an independently trained encoder that is not predicted by the hypernet can better extract useful high-level features for the downstream processing.
We’ll use this setting in the following experiments.
A.9. P RETRAIN D ATA
In our previous experiments, we train the hypernet and object representations on the DAVIS training set, which contains 30 objects only.
In this section, we expand our training dataset with Youtube
VOS (Xu et al., 2018).
In total, we train our network with 694 real-world objects; each object contains a 60-90 frames video sequence.
Compared to the previous test score (58.95), training on our extended dataset reaches a much higher score (64.45).
When our model is trained on more objects, it learns more prior knowledge about objects, thus performing better on the one-shot learning task.
Training with more objects would help our model better discriminate the shared object properties that could be learned by the hypernet and the independent properties that should be saved in the object’s representation z .
One-shot training is performed by searching the optimal representation either on the manifold spanned by the base objects, or over the entire representation space.
[Table]
We also conduct experiments to evaluate the effect of the bases.
In previous experiments, we represent novel objects over base object representations; in this part, we also perform learning of the object representations over the entire representation space.
Tab.16 shows that searching over base objects representation space performs much better than searching over full representation space on both testing accuracy and training efﬁciency.
We show a similar pattern in 4.2.3.
In this experiment, we construct the base representation space through joint training onobject data.
In the ’full representation space’ setting, we use the same hypernet as the ’over base representation’ setting.
The result probably suggests that the base representation space, as a subspace of the full representation space, contains objects’ prior knowledge and shares some common information about objects.
When a novel object comes, it is more likely to accurately represent it in this sub-space than in the full representation space.
Searching over the full representation space would make the searching process slower and possibly fall into local maxima.
A.9. F INAL R ESULT
[Table]
The ﬁnal result is shown in Tab.17.
The best score is acquired when we pre-train our model on 694 objects (YoutubeVOS + DAVIS) using joint training, and the hypernet predicts the parameters of the decoder only.
The
Jaccard score increases from 49.7 to 64.5 by modifying the critical factors.