Paper: Differentiable Sensor Layouts for End-to-End Learning of Task-Specific Camera Parameters (ICLR-2024)

Reviewer comment (Experiments):
[Generalization in diverse applications] The experiments in this paper are limited to specific tasks such as semantic segmentation and multi-label classification on facial attributes, so it is unclear how well the proposed method would generalize to other computer vision tasks.

Author response:
Our method aims to exploit spatial bias in the input data distribution to enhance a specific computer vision task and as such the learned sensor layouts are highly tuned to that data/task combination. This is especially useful for tasks like semantic segmentation for autonomous driving, because the scene structure has that clear spatial bias.
Common datasets and benchmarks for more low-level computer vision tasks like denoising or deblurring do not have a sufficient spatial bias, presumably since the images are taken from a very broad distribution. Nevertheless, if a spatial bias is present, we expect our method to be applicable to larger computer vision pipelines. However, there is no simple way to assess the amount of spatial bias.

Reviewer comment (Experiments):
[Exp. on computational cost] In Sec. 2 (in paragraph [End-to-end Optimization of the ISP pipeline]), the authors mention that the proposed model can reduce the size of the network and the training time, so further experiments on the computational cost of the proposed method are needed.

Author response:
This is a typo, thank you for pointing it out. Our method reduces inference time, because once the sensor is built, our method does have no computational overhead while allowing the network to operate on a lower resolution. As the FLOPS scale roughly linearly with the number of pixels, lower resolutions can have a more favorable accuracy / FLOPS tradeoff than higher resolutions. We added this tradeoff to Tab. 4 in the appendix.

Reviewer comment (Experiments):
[Comparison with non-uniform] The authors conduct experiments comparing their method to other method (Zhao et al., 2017) using a uniform layout, and additional experiments comparing their method to other method using non-uniform layout are needed. (Marin et al., 2019)

Author response:
The comparison to (Marin et al. 2019) would be unfair, since we try to solve a different problem. (Marin et al. 2019) compute non-uniform downsampling patterns on a per-image basis. They require a high resolution image (and thus also a high resolution sensor) to compute these patterns. On the other hand our approach aims to optimize a low resolution (but fixed) sensor layout in a data driven fashion by exploiting the inherent spatial biases in a dataset.
Our method only needs high resolution images for the differentiable sensor simulation during training. During inference (with a built sensor) we do not need high resolution images at all anymore and can directly feed the sensor output to the downstream network - without any computational overhead.

Reviewer comment (Experiments):
[Exp. on different object size] In Sec. 5 (in paragraph [Semantic Segmentation]), the authors argue that rectangular layout is learned to put more pixels towards the left and right edges because of a higher density of small objects on the sidewalks and to confirm this effect, experimental results based on the class of small objects near the horizon or accuracy in dense area is required to support this effect.

Author response:
We now compute average per-pixel accuracies over the test set which we visualize in Fig. 6 in Appendix C. The visualizations show that performance indeed increases in regions of higher densities compared to the uniform layout, which in the case of the rectangular layout lies at the horizon line and towards the left and right image edges.

Reviewer comment (Evaluation):
The authors say that the rectilinear layouts outperformed curvilinear layouts in all experiments because of curvilinear layouts’ limited adaptability in the image corners, more detailed explanation of this part is needed.

Author response:
The worse performance due to limited adaptability of the curvilinear layout is a hypothesis based on the numerical results we have gotten. Yet, this hypothesis is difficult to verify, particularly because the simulation is based on a high resolution rectangular grid, which could introduce a natural bias to rectangular pixels. We made this aspect more clear in the revised version of our paper in the experimental section and also in the very last section of Appendix C.

Reviewer comment (Experiments):
In Sec. 5 (in paragragh [Semantic Segmentation]), are there any experiments on accuracy by dense area or class to demonstrate the effectiveness of the learned pixel layout? (as commented in 6. Weakness [Exp. on different object size])

Author response:
Please refer to our comment above. We added visualizations of per-pixel accuracies in Fig. 6 in Appendix C.

Reviewer comment (Reproducibility):
How are the Deeplabv3+, PSPNet, SegNetXt trained? It seems the reported performance of Deeplabv3+ on the original image is lower than the original paper. Is this the reproduced result following official GitHub repo parameters? If the original model is not trained properly, then it is hard to distinguish if the performance boost is from extra fine-tuning (more training epochs) or the change of sensor parameters. I would encourage the author to provide more details.

Author response:
We give details on the training procedure for all baselines in Appendix B. Specifically, we choose the same (default) hyperparameters for the training process of both the uniform baselines and their corresponding experiments with a learned sensor layout.

The network and evaluation in the original Deeplabv3+ paper operates on the full resolution of cityscapes, which is 2048x1028, while the inputs to the networks in our method by design have much lower resolutions (i.e. 256x128).

Reviewer comment (Experiments):
Is the designed hardware sensor in Sec. 4 evaluated in simulation or the real world?

Author response:
All evaluations in the paper are made on simulated data. We make this clearer in the beginning of Chapter 5 in the revised manuscript.

Reviewer comment (Evaluation):
What is the inference/training speed advantage of using the proposed method? The author claimed the speed advantage in the introduction. More quantitative results are needed to justify this argument.

Author response:
The speed advantage mainly comes from the lower resolution. The sensor simulation is only needed during training. During inference (with a built sensor) the sensor image can be sent directly to the downstream network with no computational overhead.
Concurrent methods like (Marin et al. 2019) and (Jin et al. 2021) downsample on a per-image basis, thus still requiring high resolution sensors (which require more energy) and introducing runtime overhead (computing the sampling positions + the downsampling operation itself).

Reviewer comment (Presentation):
What is the meaning of the red and green arrows in Figure 1? Some captions would help.

Author response:
Thank you for the suggestion. The green arrows indicate the data flow from the simulation input and the sensor simulation parameters through the whole end-to-end pipeline while the red arrows indicate the gradient flow back to the parameters in the backward pass. We specified the meaning of the arrow colors by adding a legend to Fig. 1.

Reviewer comment (Reproducibility):
Will the framework change the camera parameters for each sample? Or the parameters are learned from a dataset and once learned, it is fixed for evaluation and inference? The training pipeline for the whole system is still vague. A high-level description of the general framework would be also needed. Please specify.

Author response:
The latter: The camera parameters are learned from data and fixed at inference time. We emphasize this aspect in our revised paper in the first section of Chapter 5.


A high level description of the training pipeline for the entire framework is specified in the original paper in Chapter 3 and with more details Appendix A and B. Specifically, we use high-resolution real images to simulate lower resolution sensor output. This simulation step additionally depends on differentiable parameters that specify the sensor layout. Since standard networks cannot deal with non-uniform layouts, as they have explicit notion of distance between pixels, we reinterpret the non-uniform sensor simulation output as if it was on a uniform grid. This leads to a deformed image like the bird in Fig. 1., with areas of high pixel density being enlarged. This deformed image is fed into a task-dependent network, e.g. a segmentation network. Since the output of that network (the segmentation mask) is also deformed and has lower resolution, we resample it to be able to compare it to ground truth. Because every step in this pipeline is differentiable, we can propagate the gradients back to both the task-dependent network and the layout parameters. The code for this process will be made publicly available upon acceptance.