Paper: Unsupervised motion segmentation in one go: Smooth long-term model over a video (ICLR-2024)

Reviewer comment (Reproducibility):
The paper needs to clearly describe the procedure for B-Spline estimation that is required to perform the update of the loss. Concretely, the argmin of eq. 9 is not described. Moreover, more details are required to state and explain how B-splines are used to model the 12 parameters of the motion model. This is important to both understanding the paper and being able to reproduce the main proposal in the paper.

Author response:
First, as we described in Section 3.1 of the paper, B-Splines are modelling the evolution of the parameters of the motion model over time. In practice, it means that the resulting motion model for a given time $t$ is a weighted average of the close control points (each control point is a quadratic model as given in eq.1 of the paper). The degree of the splines and the frequency of control points along the temporal axis dictate the weights of this average. Figure 2 of the paper illustrates the evolution of motion parameters over time.

As written in eq.2 of the paper, the parametric models induced over time by the splines are a linear combinations of quadratic parametric models. Thus, we can optimize eq.6 to obtain the argmin in eq. 9, which is a convex problem. In practice, we use a quasi-Newton method, more specifically Limited-memory BFGS, as mentionned in Section A.3 - Optimization Implementation - of the Appendix (now Section A.4 in the updated Appendix). 

We forgot to mention in the submitted paper that the code will be made available on GitHub upon paper acceptation, which will allow reproducibility of the results. In addition, we have included a repeatability study in Section A.7 of the updated Appendix.

Reviewer comment (Theory):
It would be beneficial to provide more details on why loss in eq. 7 is a reasonable approximation of KL terms of ELBO. This is not discussed.

Author response:
We have written a detailed description of the mathematical development motivating eq.7. Please refer to the new Section A.1 of the updated Appendix in the supplementary file.

Reviewer comment (Theory):
Equations 6, 8, and 9 might have the wrong sign. The loss is minimized (Eq. 9); however, the loss in eq. 6 is negative because the sum includes terms that are all positive, and the sign is then flipped. This would then increase the l1 error between parametric estimates and input flow, which seems to be the opposite of what is desired.

Author response:
Yes, you are right about the wrong sign. This was a typo. We noticed it just after the submission deadline, and it was too late to fix it.

Reviewer comment (Theory):
Given that both terms of ELBO (eq 4.) are approximated, and their signs seem to have flipped (as written in the paper), it is not clear if the resulting loss function of eq. 8 is still a lower bound.

Author response:
Yes, it is a lower bound, since the sign was unfortunately wrong (see W4). Again, the detailed description of the mathematical development mentioned above (Section A.1 of the updated Appendix in the supplementary file) clarifies this point.

Reviewer comment (Novelty):
Ye et al. [A] similarly use B-splines for modelling motion and deformations and similarly evaluate segmentation tasks on, e.g., DAVIS. It would be interesting to discuss more the differences in the approaches and provide some comparisons.

Author response:
We will add this reference. There are strong differences between this work and ours. They use B-splines to model the deformation of the image stripes over time. We use B-splines to model the temporal component of the dense flow. Moreover, their model is optimized at test time on each individual video, which is very time consuming, while we optimize our motion model only at training time. We do not need any motion model at test time.

Reviewer comment (Evaluation):
The paper claims the method is fast at inference time. Does this include the estimation of optical flow?

Author response:
The computation time at test time of our motion segmentation method is very low (210 fps on average). It does not include the computation of the optical flow, since it is just the input of our method. The computation time for the optical flow estimation depends of course on the optical flow method used. Recent methods are relatively fast.

Reviewer comment (Novelty):
‘Motion segmentation methods often proceed frame by frame’ is not entirely accurate. Earlier methods, especially before the era of deep learning, were designed to process entire videos. This is supported by a rich body of literature. Additionally, recent methods, such as "Deformable Sprites for Unsupervised Video Decomposition" (CVPR 2022), also operate on the entire video.

Author response:
This statement was not intended to be categorical (use of “often”), and we implicitly referred to the deep learning era (we explicitly did it in the related work section). We are aware of previous classical methods processing the sequence, e.g., by taking the sequence as a space-time volume and providing “tubes”. We will make this point clearer in the abstract and introduction. We will add in Section 2 the mentioned reference (please see Q1 in the response to reviewer 3).

Reviewer comment (Evaluation):
The statement claiming, "We have not only… but also achieved some kind of tracking," is vague. The FBMS-59 dataset is explicitly designed for segmenting and tracking each instance. If the proposed method operates on video sequences and performs temporal association, it would be important to be evaluated under the original settings.

Author response:
Our method processes the entire video in one go with one unique segment labeling over the sequence (this is what we meant by “kind of tracking”). It is a more demanding goal, but when achieved, offers more possibilities. We also evaluated our method under this sequence setting for FBMS59 as reported in Table 3, column “Linear Assignment”, by using the official evaluation method of DAVIS2017 for both datasets.

Reviewer comment (Experiments):
The paper predominantly compares with single-frame methods (e.g., EM, MoSeg, DivA, CIS) that do not consider temporal consistency. So the proposed method is expected to yield higher results when using an entire video batch. An exception is the comparison with OCLR. OCLR performs better while the paper argues that OCLR is not fully unsupervised since it trains on synthetic data with labels. Still, the proposed method also trains on synthetic data, where labels are typically sufficient and readily available, and not using the labels is no longer an advantage. The performance of the proposed method is notably worse than OCLR, especially considering that it only compares with the 'flow-only' version of OCLR. If the paper aims to assert the advantage of not using labels, it should directly train on generic real data. See (2.) in Questions.

Author response:
The use of manual annotation in OCLR to build the training set actually provides a competitive advantage, as previously outlined by the authors of DivA, since the annotation concerns the segmentation itself. On our side, the purpose of using the synthetic FT3D dataset is completely different.  We do not leverage any annotation or segmentation ground-truth. Our (demanding) goal by using FT3D was to demonstrate the high generalization power of our model. We train it once for all on the optical flow fields computed on FT3D, and then, apply it on new unseen datasets. We will add comments on that point to make it clearer.

Motion segmentation at sequence level does not necessarily guarantee a better score on benchmarks where evaluation is done at frame level and does not enhance temporal consistency. In fact, it is more demanding, while at the same time representing a highly beneficial added value for downstream tasks.

We took OCLR “flow-only” for a fair comparison, since we have developed a motion segmentation method, not a VOS one as such. We use VOS benchmarks for evaluation in the absence of benchmarks for optical flow segmentation.

Reviewer comment (Novelty):
The proposed method is essentially an extension of ST-MS, claiming to introduce temporal consistency over the entire video sequence. However, if I understand it right, the temporal consistency in equation (7) remains constrained to neighboring consecutive frames, similar to ST-MS. The primary distinction lies in using B-spline to model long-term motion changes and a different network architecture.

Author response:
The differences with ST-MS are significant. First, they concern the core ingredients: motion model and network architecture. Second, the introduction of the B-spline temporal motion model is quite novel and decisive. Thanks to this motion model and the use of transformers, we were able to design an end-to-end learning-based method, in contrast to ST-MS that requires a complex postprocessing for temporal linkage.

Reviewer comment (Evaluation):
The incorporation of transformers is only valuable if it leads to significant improvements, which do not appear to be impressive (particularly on FBMS-59).

Author response:
Transformers helped in defining an end-to-end method, which is essential beyond improving performance. Adding the transformer to the latent space of the Unet allows for interaction between distant features in the input, and thus a better long-term segmentation. Furthermore, we only incorporated a transformer decoder on the downsampled latent space of the Unet, which induces a limited additional computational load.

Reviewer comment (Reproducibility):
Details such as training speed, hardware requirements for training, training batch size, and the size of the training dataset are not provided in the paper. These details are essential information for reproduction.

Author response:
Requested details on the training stage will be added in Section 4.1. Details on the size of each dataset along with the split into training and test were given in Section A.1 of the Appendix (now Section A.2 of the updated Appendix). We forgot to mention in the submitted paper that the code will be made available on GitHub upon paper acceptation, allowing reproducibility. In addition, we have included a repeatability study in Section A.7 of the updated Appendix.

Reviewer comment (Experiments):
How is K decided? Is it possible to show results with different K's?

Author response:
K is set by the user before training. In fact, results with different K values are provided (K=2 and K=4 in Table 2, K=3 in Table 3).

Reviewer comment (Evaluation):
In Table 1, using more frames leads to worse results, which is counterintuitive.

Author response:
At first glance, it could be counterintuitive, but it is due to the way the evaluation is performed in these benchmarks (see W5).

Reviewer comment (Evaluation):
In Figure 3, the motorcycle and rider fly together and should share the same motion pattern. Why are they segmented into different regions?

Author response:
Yes, the motorcycle and rider fly together, but they undergo slightly different motions as perceived in the flow fields. The motorcycle wheel turns a little. Furthermore, our method operates on the vector field, not on its HSV color representation.

Reviewer comment (Writing):
What exactly is the motion model? It seems to be some kind of linear decomposition of the optical flow. If so, please clarify and denote all notation in Eq(2), for instance what is \( n \)?

Author response:
The parametric motion model is a polynomial expansion in the coordinates x and y of any point in the image. There is one polynomial for each component of the 2D flow vector. Polynomial expansion is a usual approximation of any unknown function.  It is linear w.r.t. to the polynomial coefficients. However, it is not a linear decomposition of the optical flow for polynomial of degree greater than 2. In eq.2, $n$ is the degree of B-spline (we take $n=3$ as specified at the beginning of Section 3.1 of the paper), $t$ denotes time. The other notations are already defined just above or just below eq.2.

Reviewer comment (Presentation):
Can you include labels for the axes in Figure 2? I assume the x-axis represents the frame index while the y-axis represents the linear coefficients.

Author response:
The $x$-axis in Fig.2 is the frame index in the sequence and the $y$-axis the value of each parameter of the motion model (six parameters for the polynomial of degree 2 representing the horizontal motion component). This will be added in the figure caption.

Reviewer comment (Theory):
What is the form of the prior \( p(z, \theta) \)? It is not quite clear. In Eq(7), a regularizer that penalizes differences in subsequent motion model predictions is added, but it is not clear how this relates to the KL divergence between posterior and prior.

Author response:
We have written a detailed description of the mathematical development that motivates eq.7 including the expression of the prior $p(z,\theta)$. Please refer to the new Section A.1 of the updated Appendix in the supplementary file.