Paper: Information-Ordered Bottlenecks for Adaptive Dimensionality Reduction (ICLR-2024)

Reviewer comment (Experiments):
The method depends on static hyperparameters: weights and k-max, with only two models of weights distribution proposed (geometric and linear).

Author response:
The model is designed to be insensitive to $k_{\rm max}$. So long as $k_{\rm max}$ is sufficiently large, the bottleneck compression . This assumption is used and tested in Section 6, where we estimate intrinsic dimensionalities and show that the log-likelihood gain asymptotes at a fixed $k<k_{\rm max}$.

We have also exhaustively tested different weighting schemes within the family of common statistical distributions (Uniform, Geometric, Poisson, Normal, etc.). We had found that IOB AE was largely insensitive to the form and hyperparameters. Instead, it was much more sensitive to the training procedure (i.e. stochastic vs deterministic summation of Eq 2). As a result, minding the length constraints, we did not include this testing in our main paper. However, if it should satisfy the reviewer's concern, we would be happy to include a section on hyperparameter tuning/sensitivity in the Appendix.

Reviewer comment (Presentation):
The organisation into several sections is confusing.

Author response:
We ask for further details on any confusing aspects of the organization, so that we may improve the quality of the work.

Reviewer comment (Experiments):
Could the hyperparameters be learned or at least adjusted automatically?

Author response:
Refer to the previous point.

Reviewer comment (Theory):
It is claimed in the abstract that this paper introduces a "novel theory for estimating global intrinsic dimensionality," but I cannot find any theoretical analysis related to this claim.

Author response:
In this sentence, we are referring to our usage of Wilks Theorem and nested model comparison to establish an intrinsic dimensionality (ID) of our learned likelihood model. To our knowledge, this is the first application of this theory to bottlenecked autoencoders for the purpose of ID estimation. We acknowledge that the usage of the phrase 'novel theory' might be misleading, insinuating rigorous theoretical guarantees of ID convergence. Convergence of IDs is extremely difficult to show generally, as the definition of ID is inherently variable and  hyperparameter-dependent, and thus we do not provide evidence in this regard. 

We would be happy to change this term to 'novel methodology' from 'novel theory,' if this would satisfy the reviewer's concern.

Reviewer comment (Writing):
Regarding Eq. 2, it is unclear (i) whether $b_i$ is shared by all $f_{\theta}^i$ where $i\ge k$, and (ii) whether all the $b_i$ are updated at the same time.

Author response:
We believe there is some significant confusion here. $b_i$ is not a variable but instead a deterministic function, without any tunable weights and, therefore, is not being updated at  all. $b_k(z)$ is our bottleneck operation and simply works by intaking a real-valued vector $z\in\mathbb{R}^n$ and outputting the same vector $z$, except all components $z_i=0$ for which $i>k$.

Reviewer comment (Evaluation):
Fig. 3 shows that Separate AEs are comparable with the proposed method. Does this mean that Separate AEs can perform the same function, i.e., delivering ordered latent embedding, as the proposed IOB?

Author response:
As pointed out in the paper, separate autoencoders are only provided as a baseline for "optimal compression" given the neural architecture. By applying an IOB bottleneck, we are training a 'constrained' autoencoder, enforcing an ordered representation which is not strictly necessary for compression optimality. The purpose of including Seperate AEs is to show that this additional constraint doesn't significantly weaken our capacity for compression.

The Separate AEs are strictly **not using ordered latent embeddings**. For example, an AE with $k=10$ and an AE with $k=11$ can have entirely distinct latent embeddings (in theory), because their weights are not shared. This violates the nested model comparison assumption necessary to make our ID estimates in Section 6. Also, it becomes extremely computationally expensive to train separate AEs for every possible bottleneck as the latent dimensionality increases. For our MS-COCO example, we would have needed to train 384 separate AEs to equal the performance of one IOB AE. This is another key advantage of our IOB approach, that we can get a computationally tractable ordered latent representation with this objective.

Reviewer comment (Novelty):
The motivation of this paper is not clear. The authors argue that the current compression methods, like PCA, and Auto-Encoder need dimension parameters artificially, however, the threshold k_max in IOB also depends on empirical.

Author response:
The IOB model is specifically designed to be insensitive to the choice of $k_{\rm max}$. As shown in Fig 3 and used as a key assumption in our ID estimation in Section 6, the IOB compression is expected to asymptote at a given $k<k_{\rm max}$ when we reach the ID of the dataset. So long as we set a $k_{\rm max}$ sufficiently large, the compression below this limit should reach an optimal configuration.

We also note that the primary focuses of this paper is in ID estimation, integration with pre-trained models, and data exploration. The compression capacity of IOBs is simply a mechanism to achieve this end, which needed to be shown first. We acknowledge that some language in the abstract claiming 'near-optimal' compression might be misleading. This, of course, was meant with respect to Separate AEs, but could insinuate some further information-theoretic analysis of compression. We would be happy to tone the language down in the abstract and throughout the paper, if this would satisfy the reviewers concern.

Reviewer comment (Theory):
Lacking of theoretical contribution. Directly masking top-k elements of latent variables seems too simple, and easy to discard semantic information. I hope the authors can provide analytic evidence of how to compress data without data distortion.

Author response:
Reiterating the previous point, the primary theoretical contribution is in proposing a new method for ID estimation and not in providng information-theoretic guarantees on compression capacity. We delibirately did not include analysis of information compression  because it was previously explored in the proposal of Nested Dropout [1] and we did not want to copy their analysis.

We also insist that the simplicity of the solution is a strength and not a weakness, as it allows for easy implementation of IOBs and simple integration into existing SOTA pre-trained models.

Reviewer comment (Experiments):
In experiments, why not compare current model compression methods, such as Distillation, Quantization, and Pruning?

Author response:
The goal of IOBs is not to develop a compressed model representation, but instead to develop an ordered compression scheme amenable to the experiments in ID estimation and data exploration in Sections 6 and 7. As a result, we did not consider model compression methods such as Distillation, Quantization, and Pruning. Respecting the page limit, we likely will not have room for this in a further revision of the manuscript. However it is certainly one of our priorities for further applications of IOBs in future applications.

Reviewer comment (Novelty):
The algorithmic novelty is not clear. The core of IOB formulation and training seems to be based on Nested Dropout and the unit-sweeping introduced in that work.

Author response:
**The algorithmic novelty is not clear...**

We acknowledge that the core of the approach has been considered in many ML subfields. The idea is, of course, a very simple extension of normal autoencoders (which we argue is a strength!), and thus it is unsuprising that others have considered this before. However, we remark that our progenitor papers (Nested Dropout, Triangular Dropout, SparC-IB) have never been cited together, and that this is the first paper providing a universal approach to this topic, unifying the previous methods.

Also, we insist that our paper includes many novel elements that have never been explored in this IOB-like context, to the best of our knowledge. These include:
 * Integration with pre-trained models, including the SOTA semantic, multimodal compression models. (Fig 2c and 2d).
 * Intrinsic dimensionality (ID) estimation (Table 1)
 * Latent embedding interpolation (Fig 4a)
 * Heterogenous dataset interpretation through complexity ranking. (Fig 4b)

These topics are the focus of our paper and differ greatly from the investigations in previous works (e.g. regularization and information compression analyses).

Reviewer comment (Novelty):
Few key approaches on ordered dimension reduction have not been discussed, for example [1] where a stochastic latent dimension and ordered sparsity has been applied through a Bayesian prior, where IOB seems to be a special case of it where the weighting is fixed to be linear or geometric.

Author response:
**Few key approaches on ordered dimension reduction have not been discussed...**

We thank the reviewer for making us aware of this previous work on Sparc-IB's. They are correct that our approach and this previous work share several properties, notably including a masked autoregressive prior on the latent vector. However, a key difference between the approaches is that for a given forward pass in Sparc-IB, the imposed bottleneck dimension $d_n$ is sampled from a categorical prior $\pi_{n,k}(X)$ which is a learned function of the input $X_n$. In our case, the categorical prior is fixed and the information pass-through is a continuous function of dimensionality. With its latent prior, Sparc-IB also includes elements of local ID estimation, although it is not thoroughly explored and benchmarked in their work, nor is it backed by statistical hypothesis testing, as in our approach.

Our model could be cast as a Sparc-IB with specific choices of weighting distributions and priors and with a $d_n$ distribution which is purely dominated by a fixed prior.  The authors of Sparc-IB did not consider this configuration of their model, nor did they compare to previous approaches as in our study, so we insist that the analysis presented in this work remains valuable.

We will certaintly include a reference and discussion of the Sparc-IB in the paper revision. We hope that this will satisfy the reviewers concern without discounting the qualtiy of our novel contributions (listed above). We believe an exhaustive comparison of our method and Sparc-IB is a crucial topic for follow-up work.