Paper: Loci-Segmented: Improving Scene Segmentation Learning (ICLR-2024)

Reviewer comment (Experiments):
Since the proposed method is built upon Loci, I think there should be more comparisons between Loci and Loci-s in the experimental section.

Author response:
The original Loci cannot be trained with dynamic backgrounds (it uses a Gaussian-Mixture-Model to extract one single background for training- and test set) and therefore a direct comparison is not applicable.

Reviewer comment (Experiments):
The authors mentioned that instead of the residual structure used in Loci, the encoder and decoder subnetworks in Loci-s have been revamped to adopt a ConvNeXt-like architecture. I wonder how much performance improvement is brought by this structure change.

Author response:
ConvNeXt has proven to be the superior residual CNN by the original paper (A ConvNet for the 2020s) we therefore adapted this superior design since it also nicely fits with our Hyper-Convolution approach. Although it would certainly be an interesting ablation we kindly ask the reviewer to consider that it might be out of scope (and energy wasteful) for us to re-ablate already established designs.

Reviewer comment (Evaluation):
The proposed method incorporates some additional input information for performance boost, e.g. the segmentation input (seg), depth map (sd). How much is the time cost?

Author response:
There is no real additional cost besides training the additional segmentation network. All methods (rnd), (reg) and (seg) are used in a warm-up phase where the first frame is shown during 10 cycles that involve only the encoder and decoder. The computational cost of the segmentation network is comparable to one additional slot in Loci-s.

Reviewer comment (Writing):
Overall this paper is very difficult to follow. The "Loci" method, on which this is based, is never quite made clear on its own, and then every subsequent section makes big changes to the architecture without much motivation, and without a connecting story or high-level idea.

Author response:
- We polished the overall presentation (see answer to all reviewers) and apologize for the mediocre accessibility of some of the novel components in the previous version of the paper.

Reviewer comment (Experiments):
The section on the "Background Module" never mentions this, but the abstract and the section on "Segmentation Preprocessing" describe the background module as "pre-trained", apparently for a segmentation task that "distinguishes foreground entities from the background context". My guess is that much of the performance gain is coming from this.

Author response:
- The Background module indeed is trained supervised using  ground truth foreground masks. We now clarify this in the methods. Clearly, slot attention methods like Loci greatly benefit from an accurate background module.

Reviewer comment (Writing):
What are Gestalt codes?

Author response:
- Gestalt-Codes are a compressed latent vector that encodes position invariant object properties needed to reconstruct the object at any location determined by the Position-Code. We also make this more clear now in our section on the (base) Loci Architecture.

Reviewer comment (Writing):
What are the two predictions about object positions? The paper says "we introduce a dedicated background processing module that generates both predictions about object positions as pixel densities".

Author response:
- We clarified the sentence about "pixel densities". It should now be much clearer that the Uncertainty Network as part of the Background Module computes a foreground mask, which is learned supervised.

Reviewer comment (Writing):
The paper mentions using something called "GateL0RD units" but these are never really described.

Author response:
- GateL0RD units are LSTM-like recurrent cells with an even stronger shielding. We now further clarify this in the paper.

Reviewer comment (Writing):
Section 2.1 focuses on improving Loci's "object tracking abilities", but the earlier section (describing Loci) never mentioned any object tracking happening, and tracking is never mentioned again. What is the idea here?

Author response:
- The main objective of the original Loci paper was object tracking trough occlusions, but since Loci-s focuses on object segmentation we removed the statements regarding object tracking.

Reviewer comment (Writing):
The paper mentions that the decoder "reconstructs the predicted scene via slot-wise density maps as object masks." What are slot-wise density maps?

Author response:
- Slot-wise density maps are the instance segmentation masks computed by each slot. We clarify this now.

Reviewer comment (Reproducibility):
The paper briefly mentions an "L0 loss on gate openings" but it is not clear what ground truth is used for this loss. Is it maybe just a regularization term, penalizing the L0 norm?

Author response:
- The L0 loss on gate openings is a regularization loss that punishes any gate values except zero with the same gradient. But since this inner loop is not relevant for the segmentation task we moved it to the appendix. This also puts the depth input into a more consistent story line.

Reviewer comment (Presentation):
Section 2.2 introduces a depth input and an equation to normalize it, but it is not clear where this fits with the inner loop described in the previous section.

Author response:
- The L0 loss on gate openings is a regularization loss that punishes any gate values except zero with the same gradient. But since this inner loop is not relevant for the segmentation task we moved it to the appendix. This also puts the depth input into a more consistent story line.

Reviewer comment (Experiments):
For Table 2 it would be great to clarify what dataset these experiments happen in, and what the metrics are.

Author response:
- We added descriptions for the datasets and metrics from the compositional scene understanding paper.