Paper: Sparling: Learning Latent Representations With Extremely Sparse Activations (ICLR-2024)

Reviewer comment (Theory):
**(Identifiability of the latent factors of variation)**. Not having any guarantees on the identifiability of the latent variables jeopardizes the proposed method, which seeks to find meaningful/interpretable latent variables without any explicit supervision. When is a given amount of data enough to guarantee that the proposed method can recover all the latent factors? And, are there any other specific requirements needed to have disentangled factor of variations as discussed in [2]?

Author response:
This paper is empirical in nature and validates its empirical hypothesis in three different domains. While it would of course be better to have a theoretical result as well, we believe that it is out of the scope of the paper to provide one. A theoretical guarantee would also be inherently limited to the kinds of mathematical objects that could be easily analyzed within a theoretical framework, whereas here we demonstrate that the technique works empirically on a g network that takes the form of a multi-layer residual convolutional network and an h network that is a complex LSTM/transformer architecture.

Reviewer comment (Writing):
Section 3.3 is hard to parse and does not clearly highlight the connection between the information bound and SPARLING.

Author response:
We have rewritten the ending of this section to better highlight the connection. Let us know if you would like further clarification.

Reviewer comment (Experiments):
The baselines considered are not totally fair. While the Sparling coefficient \(t\) is annealed during the training process to alleviate optimization challenges, the coefficient used for L1 loss is set only once, likely preventing L1 loss from learning sparse representations due to the same optimization challenges. Could we see ablations where the L1 coefficient is also annealed similar to the Sparling coefficient? In particular, is the improved sparsity a result of the spatial sparsity layer, or just the annealing scheme?

Author response:
An initial analysis, using an exponentially annealing L1 parameter (specifically, starting at 0.1 and increasing by 33% every time accuracy hits the threshold), shows that this performs slightly less well than our MT technique (about 5% error vs 1.7-3.8% for ours). We will analyze this more completely with different annealing schedules in future, but do not have sufficient time to properly implement this for this particular rebuttal cycle. Even if this were to work, it would not allow us to target specific sparsity values as easily (e.g., if you for one reason or another wanted to consistently keep two parts of your input at different sparsities), and it would require an additional hyperparameter to tune in the form of the initial L1 loss coefficient.

(To clarify, in Sparling, t is not annealed during the training process, rather it is targeted to a quantile of the output of the g network, specified by delta; delta is annealed.)

Reviewer comment (Experiments):
Does Sparling work for non-convolutional architectures (e.g. MLP)?

Author response:
Sparling requires some notion of locality for the g layer, but the h layer can be arbitrary. In our case, we used transformers because we were producing sequential outputs, but this is not a necessary feature of the system. In future work, we will explore different h architectures.

Reviewer comment (Evaluation):
In Figure 6, the retrained \(\hat{h}\) outperforms the end-to-end non sparse network. How is this the case? Why is the performance different at all from the Sparling network which trains the motif model and prediction head end-to-end?

Author response:
We do not believe that this apparent improvement reflects a genuine advantage to the Retrained technique over end-to-end training. The small difference that can be seen in the diagram is not statistically significant, and we have clarified this in the paper. The numerical difference is also small enough to be explained by minor technical details of convergence criterion in training. In the paper we do not claim Retrained is better than Non-Sparse, merely that it is not worse in two domains, and only slightly worse in the third.

In the Sparling setting, we are imposing an information bottleneck to force g to have semantically meaningful outputs. The same g is used in the Retrained setting, but since the bottleneck is removed, the network h is able to take advantage of the additional information that would not have been allowed past the bottleneck.

This becomes relevant because of the presence of noise. As there is noise, there are often motifs that might be real or might be noise, just at the threshold of detection. In a model with no bottleneck, the g model can pass information about these potential motifs through and disambiguate later in the process. However, in the presence of a strong sparsity bottleneck, the model cannot do this without passing through too many nonzero activations. As we describe in the paper, this requires the model “commit” to choosing one or the other. This results in degraded performance.

Reviewer comment (Theory):
What is the motivation for computing the spatial sparsity layer channelwise?

Author response:
There are two alternatives that come to mind.

The first, using a threshold at each position and channel, would result in a very large number of thresholds and be quite noisy. We make the assumption of homogeneity across positions in the input, as do most techniques dealing with 1d or 2d inputs.

The second, using a single threshold across all channels, is something we did explore. You can find our analysis in Appendix D. A summary is that this approach leads to substantially higher confusion error on average, while leading to lower false positive error. Confusion error is the error we are most concerned with, so we decided to use per-channel threshold setting.

Reviewer comment (Theory):
The requirement of $g^*$ being necessary for the final prediction is quite harsh for general settings. While this might be suitable for OCR, it is fairly rare to have localized and independent predictive features in an input.

Author response:
See https://openreview.net/forum?id=zgHamUBuuO&noteId=tJhJMRIRWh for our response.

Reviewer comment (Writing):
The writing (especially figure captions) can be made more clear.

Author response:
We have taken a pass over the paper and clarified/elaborated the captions. If you have any more specific feedback, we would be happy to incorporate it.

Reviewer comment (Presentation):
Could you elaborate Fig 2? Do the letters represent specific shapes or the digits themselves?

Author response:
Each letter represents a separate channel of the post-Sparse layer of our neural network. For example, in the DigitCircle example, the A channel represents the first channel out of 10, the C channel represents the third, the D channel the fourth, etc. In practice, for this network, once trained, you can think of the C channel as representing the 6 digit: however, this is an emergent property of the network rather than something that it was specifically trained to do. This property, where the channels end up recognizing the true intermediate features, is the main claim we are trying to establish with our experiments. I have updated the caption to clarify this.