Paper: ZeroFlow: Scalable Scene Flow via Distillation (ICLR-2024)

Reviewer comment (Experiments):
For the full point cloud setting, what is the exact number of points in average? Have we applied ground point removal so the number of points has been significantly reduced. I know there is a discussion in 3.1, but I want to confirm the actual number of points and settings in the later experiments.

Author response:
# Numbers on dataset point cloud sizes

We provide plots of the full point cloud size distribution for AV2 and Waymo datasets after ground point removal and region clipping (102.4m x 102.4m area around the ego vehicle) in the Supplemental, Section A. The average number of points per point cloud is 52,871.6 points for AV2 and 79,327.8 points for Waymo Open.

Reviewer comment (Novelty):
The innovation and contribution of the paper mainly lie in the new application of the distillation strategy, and I suggest the authors provide further analysis or insights into the scene flow estimation task. I think the achieved performance of the proposed approach is highly dependent on the quality and effectiveness of the existing works. The authors combine two previously published works, using NSFP to generate pseudo labels and training the FastFlow3D model using these labels.

Author response:
Thank you for your review.
# Novelty

Michael Black’s reviewer guidelines [1] eloquently argue that "the simplicity of an idea is often confused with a lack of novelty when exactly the opposite is often true. Such reviews lead researchers to make an idea appear more complex so that reviewers will find it of higher value."

Indeed, our novelty comes from the fact that our simple and post-hoc obvious pipeline produces state-of-the-art results. To quote q16v: "This distillation idea is very simple, but previous work on point-cloud-based scene flow methods hasn't tried the idea yet. The paper demonstrates that this simple idea is working and demonstrates that it's scalable by using large-scale diverse unlabeled data." This result also produces "valuable insights to the autonomous vehicle industry" (pGhB).

# Performance of our system is the result of its components

Our performance cannot be strictly attributed to the components plugged into our *Scene Flow via Distillation* framework — we benchmark ZeroFlow and outperform both NSPF and FastFlow3D. 

But, broadly, we agree that we benefit from high quality components! Indeed, a feature of our *Scene Flow via Distillation* pipeline is, given a better test-time optimization method, a better supervised scene flow network architecture, or more unlabeled data, it can leverage these component improvements to produce an even better final method *for free*.

Reviewer comment (Novelty):
In Sections 3.2 and 3.3, the author provides a detailed description to help understand the proposed method, but it is difficult for me to detect the author's new in-depth thinking about this work from these descriptions.

Author response:
Thank you for your review.
# Novelty

Michael Black’s reviewer guidelines [1] eloquently argue that "the simplicity of an idea is often confused with a lack of novelty when exactly the opposite is often true. Such reviews lead researchers to make an idea appear more complex so that reviewers will find it of higher value."

Indeed, our novelty comes from the fact that our simple and post-hoc obvious pipeline produces state-of-the-art results. To quote q16v: "This distillation idea is very simple, but previous work on point-cloud-based scene flow methods hasn't tried the idea yet. The paper demonstrates that this simple idea is working and demonstrates that it's scalable by using large-scale diverse unlabeled data." This result also produces "valuable insights to the autonomous vehicle industry" (pGhB).

Reviewer comment (Novelty):
Section 3 contains many statements on existing work, such as 3.1. This section might benefit from providing a more in-depth analysis of combining the strengths of optimization-based and feed-forward methods.

Author response:
# More analysis of of the combined strengths of optimization-based and feed-forward methods
We refer the reviewer to Section 3.1, which analyzes the combined strengths of optimization-based and feed forward methods. Optimization methods produce high quality flow, but have high computational cost, whereas feed-forward methods are fast but limited by the scale of their supervised labels. Combining these two techniques gives us a formula for scaling that allows our combined method to exceed the performance of its components and achieve state-of-the-art results. We would gladly appreciate suggestions for further analysis.

Reviewer comment (Evaluation):
The quantitative results in Table 2 on Waymo Open are less competitive, making the method's performance on this dataset not very convincing.

Author response:
# Waymo Open (Table 2) results not competitive

ZeroFlow provides a label-free, scalable training pipeline. We expect that, relative to human supervision *on the same data*, ZeroFlow's pseudolabeling approach will perform worse.

This is reflected in ZeroFlow's performance on Argoverse 2 (Table 1); ZeroFlow 1x performs worse than its supervised counterpart, FastFlow3D. However, when scaling up training to include the unannotated Argoverse 2 LiDAR dataset, ZeroFlow exhibits meaningfully superior performance compared to the human supervised counterpart (ZeroFlow 3X vs FastFlow3D).

This same pattern is also reflected in Table 2 for Waymo Open. Waymo Open does not provide any additional unlabeled data beyond their annotated dataset, preventing us from scaling up ZeroFlow further; however, our performance results on the data we do have are actually quite positive. Supervised FastFlow3D has a very similar Threeway EPE on both datasets, while ZeroFlow 1x on Waymo Open is actually closer in performance to FastFlow3D than on AV2 where we successfully demonstrated that more raw data produces state-of-the-art performance.

Reviewer comment (Presentation):
The overall style of the article is more like a technical report than a research paper, and I feel that it contains more engineering skills.

Author response:
# This is a technical report not a research paper; mostly engineering

Our paper provides non-trivial, fundamental insights: our straight-forward pipeline shows that simply training a supervised model with imperfect pseudo-labels can *exceed* the performance of perfect human labels given sufficient unlabeled data. We think this is itself surprising, but we also think it has significant implications for the problem of scene flow estimation in general: *point cloud quantity and diversity is more important than perfect flow label quality for training feedforward scene flow estimators*. We translate this into an empirical scaling law that describes the log-log relationship between dataset size and final Threeway EPE performance.

We also present a novel scene flow estimation analysis technique. To our knowledge, the residual plots in Section 4.4 are the first attempt at visualizing the residual *distribution* of scene flow estimators. We think these plots provide useful insights to practitioners and researchers. For example, open world object extraction [2] must be able to threshold and  cluster motion vectors together to extract the entire object. Understanding the *distribution of flow vectors* is essential for identifying extraction heuristics. .

Furthermore, we provide extensive scientific inquiries on the impact of not only method component choices (e.g. Teacher quality — Section 4.5), but of the data itself (e.g. Dataset diversity — Section 4.3). These inquiries provide important insights to practitioners and researchers on what are fruitful directions to explore.        

[1] Michael Black. Novelty in science: A guide to reviewers.
[2] Najibi et al. Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving. European Conference on Computer Vision. ECCV 2022.

Reviewer comment (Evaluation):
In Table 1, how can ZeroFlow XL 3X (Ours) outperform the NSFP w/ Motion Comp baseline as ZeroFlow's accuracy can be bounded by the accuracy of pseudo GT (i.e., NSFP)? Would it be also possible to include the accuracy of pseudo GT in Table 1?

Author response:
# Why does ZeroFlow's scaled up performance surpass the teacher's performance?

ZeroFlow's student-teacher distillation setup enables the student to surpass the teacher because neural networks are good at learning from noisy labels [1]. As we show in Figure 5, as ZeroFlow's pseudolabeled dataset is scaled up (1x, 3x), the variance of its error distribution shrinks because the student gets better at predicting the *expected value* of the NSFP motion estimates conditioned on the observation of motion. This lower variance expected value estimation leads to lower Threeway EPE estimates compared to even NSFP itself.
# Pseudolabel performance in Table 1

 As shown in Table 1, NSFP (our pseudolabeler) achieves 0.068 EPE.

Reviewer comment (Evaluation):
In Fig. 4, why is FastFlow3D better than 'Ours' when the dataset size is less than 100%? I thought ZeroFlow's backbone was based on FastFlow3D, but what change made it underperform FastFlow3D?

Author response:
# Why does FastFlow3D outperform ZeroFlow when the dataset size is less than 100%?

FastFlow3D is trained using ground truth flow provided via human annotations. ZeroFlow is trained using pseudolabel flow provided via NSFP. NSFP is an imperfect pseudolabeler, so even though ZeroFlow and FastFlow3D share the same network architecture and see the same point clouds, ZeroFlow's supervision is from inferior quality pseudolabels. Therefore,  performance is expected to be worse than its human supervised counterpart.

Reviewer comment (Experiments):
(Just a remark) By the way, there is a recent paper [a] that significantly improves the runtime performance of NSFP by 10 or 30 times (varying different datasets). The proposed approach can reduce the compute cost for pseudo GT generation by using this faster method.  

[a] Fast Neural Scene Flow, ICCV 2023

Author response:
# Faster teacher method

Thank you for this reference! We will look into this as a faster teacher!