Paper: Where is the Invisible: Spatial-Temporal Reasoning with Object Permanence (ICLR-2024)

Reviewer comment (Experiments):
Looking at a high level, the current method mimics the classical Kalman and Particle filter very closely (predict future location based on past observations and update the posterior based on current observations). Given the similarities, I believe comparing to such classical methods is necessary. I'm sure with the power of deep networks, the proposed method can outperform such classical methods but it is essential to know the gap in performance. Are the deep networks even necessary or does Kalman filter just solve the synthetic dataset?

Author response:
We applied the traditional Kalman filter method to conduct experiments both on the LA-CATER and iVOT datasets, and the experimental results showed that the traditional filter method cannot effectively understand the spatial temporal relationship under uncertain noise distribution and nonlinear system. That is the Kalman filter only predicts the short-term trajectories, and can not conditioned on the object-object relationship. However, the objects in containment may disappear in the long term.

- Comparison of the traditional Kalman filter method and our proposed method on the LA-CATER dataset and the iVOT dataset using mean IOU.

|                     | LA-CATER | iVOT |
|---------------------|---------------------|-----------------|
| Kalman filter               | 0.6978               | 0.374           |
| QQ-STR(Ours)        | 0.8278               | 0.583           |

Reviewer comment (Experiments):
Authors demonstrate results on two real world benchmarks but I believe some more experiments are necessary to properly understand the contributions. RAM (Tokmakov et.al.) show results on KITTI benchmark. The ID-Switch problem in Multi-Object Tracking (MOT) is a result of models failing to understand object permanence. If authors claim their method is good at reasoning object permanence, it is essential to report results on these tracking datasets. I don't expect the authors to show state-of-the-art performance on Tracking/MOT but showing that this work improves some metric, like reducing the number of ID switches, is a good indicator of this method working on real world data. I strongly recommend authors to perform these experiments and report results on these tasks and datasets.

Author response:
We agree that the evaluation of more real-world datasets is necessary. However, we notice that the object-object relationship in most existing real-world datasets is too simple to evaluate the object permanence reasoning. The invisible objects usually are just occluded by other objects or out of view, hardly contained by other objects. As a result, the invisible objects usually are shortly occluded by other objects, and the reasoning is unnecessary in most cases. However, in this paper, we focus on the ability to reason the localization of invisible objects, partially in some complex relationships, e.g., co-occurring containments and occlusions. Hence, this motivated us to collect new datasets in our daily activities that contain complex transitions of the object relationship. We argue that it will be our future work to collect more diverse videos with complex object relationships to evaluate the object permanence, rather than on those existing video tracking datasets, as the focus of the two domains are different.

Reviewer comment (Experiments):
A few design decisions haven't been ablated to understand their significance. Why did authors choose to go with diffusion models for object tracking? Why not go with the well tested trackers like SORT[2] or DeepSORT[3]? Or some recent state of the art MOT trackers [4]? I think this experiment is necessary to justify the use of diffusion models.

Author response:
The main purpose of using the diffusion model is to predict the trajectories of invisible objects under certain spatial constraints rather than serving as a tracker itself. Under high uncertainty, the possible trajectories of the invisible target can be viewed as a noisy Gaussian distribution that represents a blurred area controlled by the container trajectory. As the uncertainty decreases, the distribution gradually approximates the ground truth distribution for generating the correct trajectory. Therefore, we regard the trajectory prediction task as a condition-based generative task and apply the diffusion model. The relationship experiments in Section 4.5 demonstrate the effectiveness of diffusion models for trajectory prediction. 

The MOT trackers also apply a similar tracking-by-detection framework. They focus on solving the data association problem for multi-object identification, where the objects may be of similar appearance and will shortly occluded by others. In this case, most MOT trackers assume the object keeps the linear dynamic (they usually track multiple pedestrians), and only apply some simple models to predict the short-term trajectory when the object is occluded by another. However, such short-term occlusion is a simple case in the study of object permanence. 
We find that the trajectory of the object highly depends on the relationship to other objects. If the object is only occluded by others, it will keep its own movements. In contrast, if it is contained in some object, its trajectory will depend on the transition of the container.
That is why we need to build a relation-conditioned trajectory prediction for tracking invisible objects. The results of the Kalman Filter-based tracker also show that the simple predictor can not handle the complex relationship and long-term invisibility.

Reviewer comment (Writing):
Some detailed designs are not explained clearly. For example: (1) For the object detection model, the classification of the general detection task is to identify different categories of objects, while the proposed method seems to need an object-level classification, but the paper did not mention much about this. (2) For the shiver error in the correction module, there is no explanation of what happens if the size of the object itself changes in adjacent frames, such as aspect ratio change or scale change.

Author response:
> *Q4: Some detailed designs are not explained clearly...* 
> 
A4: We are sorry that we did not describe the details exhaustively and caused your misunderstanding. We have revised the manuscript to explain more details. For convenience, we introduce the details in the following: 

*Object Detection*: For the experiments on the CATER and LA-CATER datasets, due to the limited types of objects included in the dataset, the vision module can be implemented by an object detector. For a fair comparison, we use the same object detector as previous works (OPNet[1], AAPA[2], and RAM[3]), a Faster RCNN network pre-trained on the COCO dataset, which is introduced in Section 4.2 of the paper. For the experiments in real scenes, we apply AutoMatch[4] as the object detector. During tracking, the search area in the next frame depends on the estimated target location (detection position or predicted position) in the current frame. This rule is widely used in visual tracking models. In this case, the adjustment of the search area will affect the performance of the tracker once it disappears.
Hence, in our framework, once the confidence of the detection result is lower than a threshold of $\lambda_{disappear}$, we will assume the target has disappeared and predict the trajectory of the invisible target to guide the adjustment of the search area synchronously. In the same way, when the confidence of an object that originally disappeared is greater than $\lambda_{reappear}$, we will think that the object appears again. The proposed method focuses on predicting the spatial and motion relationships of objects in the invisible state, rather than vision-based object tracking.

*Correction Module*:  Thanks for pointing this out. Since the frame rates of the LA-CATER synthetic dataset and the iVOT real-world dataset used are both 30fps, the scale of objects in adjacent frames changed slightly in most cases. If the scale changes dramatically, the correction module may result in unreasonable modifications. It will be our future work to extend our framework on videos with violent shaking.

Reviewer comment (Experiments):
One of the main contributions of this paper is to propose a new dataset; however, there are very few descriptions of this new dataset. Besides, there is also a lack of experimental results of SOTA methods on the proposed dataset iVOT (Table3).

Author response:
> Q5: *One of the main contributions of this paper is to propose a new dataset, however, there are very few descriptions of this new dataset. Besides, there is also a lack of experimental results of SOTA methods on the proposed dataset iVOT (Table 3).*
> 
A5: We appreciate your suggestions. We have added more details and results about the new dataset. To be specific, the proposed iVOT(invisible Object Tracking) dataset is collected by Intel Realsense D435i which has an RGB frame resolution of 1920\*1080, a depth output resolution of 1280\*720, and a frame rate of 30. The data are recorded in indoor rooms (e.g., bedroom and living room), including diverse human-object interaction activities, where the objects frequently become invisible in the video. These cases are normal in our daily activities but are challenging to the state-of-the-art methods. For example, a man picks up a toy into a box and moves the box to another place, where the model needs to reason the object-object relation to localize the target object. More details on this dataset are elaborated in Appendix A.5. We also add more results of SOTA methods and visualizations in Appendix A.6.

- Comparison of SOTA methods testing on the iVOT dataset using mean IOU.

| Method       | mIoU  |
|--------------|-------|
| AutoMatch[4]    | 0.491 |
| RTS[5]          | 0.513 |
| ToMP-50[6]      | 0.527 |
| ToMP-100[6]     | 0.532 |
| QQ-STR(Ours) | 0.583 |

Reviewer comment (Writing):
The paper is a bit hard to read since: (1) The definition and use of some variables and formulas seem a bit complicated and contain some typos. For example, the definition of hand positions H_i, the use of t/T/k, inconsistent use of superscript and subscript on different formulas, and so on. (2) Figures 1 and 2 are not connected tightly with the text, and the cases shown in the figures are not well explained in the text.

Author response:
> *Q6: The paper is a bit hard to read...*
> 
A6: Thanks for your valuable and helpful suggestions. We have corrected relevant ambiguities or inconsistencies in the revision. We highlight the main update in blue.

Reviewer comment (Presentation):
No qualitative results: There is no single image in the main paper visualizing the tracking results of the proposed method. Please consider showing multiple frames (which can be manually selected) of an object sequence and the tracked bounding boxes for QQ-STR and the closest baseline method (say RAM). This will tell us the improvements QQ-STR brings over the existing methods. If space permits, consider adding a failure case visualization highlighting the limitations. Note that the supplemental consists of two short videos (3 secs from Liang et al and 5 secs from LA-CATER) showing the results of the proposed method but no comparison to the baseline. It is a missed opportunity to not show video results on the collected iVOT dataset in the main paper or supplementary.

Author response:
We agree with your comment that qualitative results are important to demonstrate the effectiveness of our method. We have added representative sequences on the iVOT datasets in the manuscript, as shown in Appendix A.6. These sequences compare the qualitative results of different methods and illustrate the advantages of our approach.

Reviewer comment (Experiments):
Evaluation on the CATER dataset: Sec. 2 argues that the CATER dataset was not used in the evaluation as it only has classification and relation labels. However, in comparison to LA-CATER (the substitute), the CATER dataset is more widely used by current methods and has a more evolved list of quantitative performance of related methods. The results of the proposed method QQ-STR can still be evaluated on the CATER dataset, similar to the CATER-Snitch localization task Table 1 of [1].

Author response:
We apologize for the confusion caused by our argument. We have evaluated our model on both the LA-CATER and the CATER datasets. Due to space limitations, we reported the results on the CATER dataset in Appendix A.3. We have clarified this point in the manuscript. In summary, our method outperforms most of the baselines in the Top 1 Accuracy and achieves comparable results to [1] (slightly lower in the Top 1 Accuracy but higher in the Top 5 Accuracy).

Reviewer comment (Presentation):
More descriptive method figures: Fig.1 and Fig.2 treat the proposed modules, VM, SRR, and SRA, as black boxes and tell nothing about the method. Please consider adding details about these components and provide insights into the inner workings of these blocks. Method figures are excellent visual tools to quickly convey the key ideas to the reader, which takes a while if it is only text-based (which is the case currently).

Author response:
We appreciate your valuable suggestions. We have revised our manuscript to provide more details about each module of our method. Please refer to Appendix A.1 (Fig.4, Fig.5, Fig.6) for the detailed illustrations and explanations of our method.

Reviewer comment (Reproducibility):
Many important details are not well explained or missing. For example, hand positions are extracted as in Sec. 3.3, but how it is being used is never mentioned in latter sections.

Author response:
We apologize for the lack of clarity and detail in our original manuscript. We have improved our manuscript to provide more details about our method. In particular, we have explained how the hand positions are used in the Spatial Relation Predictor (Section 3.4). Empirically, we regard the hand as a special object that never be contained by other objects in our framework. If we treat the hand as an ordinary object, it may cause ambiguity between containments and occlusions.

Reviewer comment (Reproducibility):
Some parts don't make much sense. For example: When generating spatial relationship graphs, do you distinguish occlusion and containment? If not, why?

Author response:
When generating a spatial relation graph, we distinguish three types of spatial relationships: occlusion, direct containment, and no direct relationship. We use different edge categories to represent these relationships.

Reviewer comment (Experiments):
Some parts don't make much sense. For example: When generating the graphs, why do you only use object trajectories without identity information? The object characteristic should affect occlusion and containment, in my opinion.

Author response:
The visual module in our framework has already obtained the trajectories of visible objects by ID matching (using the same Faster RCNN network as OPNet and RAM on the LA-CATER dataset, and the AutoMatch tracker on real-world datasets). In this paper, the relation graphs focus on the relationship between invisible objects and visible objects. However, there is no identity information for invisible objects, so our method only uses historical trajectories without identity information when generating spatial relation graphs. The main goal of our work is to infer the trajectories of invisible objects by constructing the spatial relations of both visible and invisible objects, rather than using visual features to track visible objects.

Reviewer comment (Experiments):
The method needs to "enumerate all possible spatial relations in the first frame and form the candidates PG1." This may cause some problems when there are too many objects.

Author response:
We agree that the number of candidate relationship graphs may grow exponentially, so we only keep the most likely $K$ spatial relation graphs at each time step, as described in the Evaluation Module in Section 3.5. In practice, our method achieves an average processing speed of $100ms$ per frame when the size of the candidate set K=16. We have also tested the cases where the size of candidate sets is larger or smaller, and the processing speed of our method is about $30ms$ per frame (K=1) and $250ms$ per frame (K=64). We have updated the manuscript accordingly.

Reviewer comment (Experiments):
Despite a relatively straightforward main idea of estimating occlusion / containment status based on past trajectories, there are a lot of twists and tweaks involved, for example, lots of hyperparameters and those correction stages, making the whole system over-complicated and vague to understand. The effects of most of them are unclear, and may hinder the generalizability to other datasets.

Author response:
Compared to the end-to-end methods, our method has the following advantages: 
1) Generalizability. The pre-trained model can provide generalizable state representation to make our framework easily transferable to unseen datasets. We have reported the cross-domain generalization of our method compared with the baselines. The models are trained on LA-CATER and tested on iVOT. Our approach demonstrates superior generalization capabilities compared to OPNet. 
2) Modularity. Each module (e.g., object detection, action recognition, trajectory prediction) can be easily replaced by state-of-the-art methods, thus our model can benefit from the advances of each component. 
3) Interpretability. Our framework consists of modular components that can be individually analyzed and evaluated. For example, we can visualize the relationship graph and the forecasted trajectory produced by each module, which can help us identify the potential weaknesses and limitations of our approach. 
 
To evaluate the generalization ability of our method to different datasets, we performed cross-dataset experiments, where we trained our model on the synthetic LA-CATER dataset and tested it on the real-world iVOT dataset. In this experiment, we use OPNet for comparison. The AutoMatch model is a part of the vision module for localizing the visible objects. Results show that our proposed method has stronger generalization ability and performance than OPNet.

- Comparison of the cross-domain generalization using mean mIoU. We train the models on the LA-CATER dataset and evaluate the performance on the iVOT dataset. We also report the results of OPNet as a reference.


|                     | Trained on LA-CATER | Trained on iVOT |
|---------------------|---------------------|-----------------|
| AutoMatch[1] | 0.491               | 0.491           |
| OPNet[2]               | 0.476               | 0.535           |
| QQ-STR(Ours)        | 0.554               | 0.583          |