Paper: Offline Tracking with Object Permanence (ICLR-2024)

Reviewer comment (Experiments):
In tracklet association, there appears to be a division between the utilization of map and motion data. However, it's worth considering their complementary contributions. For instance, map information could enhance the accuracy of motion association. Separating these aspects might lead to a loss of crucial information.

Author response:
Indeed, the fusion of the two features can theoretically improve performance. That is also exactly what we did. In the map branch of the Re-ID model, the map feature is fused with the motion feature via spatial attention [1]. So the map branch takes both motion information and map information as inputs.

Reviewer comment (Experiments):
Appearance features could serve as a pivotal factor in tracklet association. Even when an object becomes temporarily occluded and untrackable, its re-emergence could still benefit from utilizing appearance data for accurate association.

Author response:
Thanks for your suggestion! We have also tried to extract appearance features from point clouds for Re-ID. We used the same technique for point cloud aggregation as in [2], only that we used CenterPoint [3] as the feature extractor. Then we applied contrastive learning to learn the appearance affinity between the appearance features of tracklets in a latent space. However, the result is not satisfying so we eventually abandoned this. We think this is because the model can only implicitly extract the object size information from the point cloud and learns the affinity based on this, which is not accurate. The location and motion information in point clouds are also provided by the motion feature. That is also the reason why the appearance feature extracted from the point cloud is mainly used to refine the bounding box size in [2,4]. However, we are planning to explore a more effective way to extract it so that it can help tracklet association. A more trivial way is to use extract appearance feature from camera images, which contains much richer appearance information for Re-ID.