Paper: On convex decision regions in deep network representations (ICLR-2024)

Reviewer comment (Theory):
The concept of measuring convexity in NNs appears to be somewhat circular. It doesn’t seem like the convexity in NNs spontaneously emerges from the data or is an unexpected outcome; rather, it seems intentionally designed. Take the "graph convexity" prominently featured in the paper, for example. If we view the final softmax layer as the classifier (setting the decision boundary) and the feature vector inputted into this final layer as a data point on a manifold, it's almost always a given for a 100% accurate NN to possess a graph-convex decision boundary, owing to the intrinsic convexity of the softmax function. Essentially, the core function of the feature extractor is to map the data onto a (potentially) high-dimensional space where a simple linear or convex layer like the softmax can efficiently segregate data with varied labels, a strategy evidently seen in methods like SVM. Hence, the design of convexity seems more of an intentional architectural choice rather than a novel, data-driven emergence.

Author response:
Thank you for the comment: 
> "The concept of measuring convexity in NNs appears to be somewhat circular. It doesn’t seem like the convexity in NNs spontaneously emerges from the data or is an unexpected outcome; rather, it seems intentionally designed."

Author reply:  The convexity is not entered by design for NN! As we note in the paper,  the last layer is convex by the convexity of softmax. Also, we argue that the internal attention/softmax layers can induce convexity.  And so can ReLU layers. However, they can also lead to much more complex decision regions, so by design, there is a *possibility for convexity*, not a necessity. Testing the hypothesis, we find experimentally that convexity is quite pervasive in the networks, and it is increasing with label-based fine-tuning.  This is highly non-trivial and aligned with findings in human learning.

Reviewer comment (Presentation):
The paper, dense with detailed proofs and extensive explanations, feels somewhat overloaded. Elements like Theorem 2 and 3 seem somewhat simple, potentially to individuals with an initial graduate-level understanding. A more streamlined approach, possibly through referencing standard convex optimization textbooks for such foundational concepts, could enhance the paper's readability and focus.

Author response:
We appreciate your comment: 
>"The paper, dense with detailed proofs and extensive explanations, feels somewhat overloaded.".  

Author reply: Please note that all proofs were delegated to the appendix.  We agree that the paper introduces several new concepts, as needed when proposing a conceptually new way of analysing NNs

Reviewer comment (Theory):
The connection between human representation and NN convexity isn’t entirely lucid. While there’s a notable reference to numerous papers on human representation in the introduction, the subsequent sections don’t seem to elucidate a substantial link beyond a generalized notion of convexity. It leaves room for doubt regarding whether the human-related assertions are primarily theoretical embellishments or if they hold significant, intrinsic connections and inspirations.

Author response:
Thank you for the comment: 
> "The connection between human representation and NN convexity isn’t entirely lucid."

Author reply: The cognitive psychology literature suggests that convexity enables generalization as we detail in the introduction.  In our work and conclusion, we find a similar result for NNs. We claim this is highly non-trivial and hope that you agree this is a worthy ICLR contribution.

Reviewer comment (Experiments):
The experiments predominantly focus on the evaluations of convexity. An expansion of the experimental design to include algorithmic innovations that utilize these analysized insights to enhance training and learning processes could be quite valuable. Questions like whether explicitly incorporating convex boundaries during NN training could expedite convergence remain unexplored. Without such practical explorations and demonstrations, the applicability and impact of these observations seem somewhat limited.

Author response:
We appreciate your comment: 
> "The experiments predominantly focus on the evaluations of convexity. An expansion of the experimental design to include algorithmic innovations that utilize these analysed insights to enhance training and learning processes could be quite valuable. Questions like whether explicitly incorporating convex boundaries during NN training could expedite convergence remain unexplored. Without such practical explorations and demonstrations, the applicability and impact of these observations seem somewhat limited."

Author reply: Our contribution is mainly conceptual and we maintain that ICLR can present work that is not engineering of new architectures and algorithms.  We ask a fundamental scientific question ("Are NN decision regions convex"?) and we find significant evidence in favor of the hypothesis. Also, we find convexity is linked to generalization. We hope you will consider raising your grade to reflect the value of these findings.

Reviewer comment (Presentation):
The organization of the paper's content is unreasonable. For instance, the introduction section dedicates substantial space to discussing the relationship between convexity and generalization, while neglecting to mention the specific objectives of the paper. This section should be reorganized by moving background information to the related work or appendix while focusing on introducing the main content and contribution of the paper.

Author response:
We believe that an integral part of paper is bridging the gap between the notion of convexity in human and machine learned representations - a topic that to our knowledge has not before been investigated. The motivation for our research is grounded in extensive research within the field cognitive science, and we believe it necessary to introduce the new vocabulary to properly motivate our research question, namely “Are generalizable, grounded decision regions implemented as convex regions in machine-learned representations?” Our contributions are stated on page 2 at the end of the introduction.

Reviewer comment (Experiments):
The experimental setup in the paper has significant issues. To demonstrate the claimed relationship between convexity and generalization, the paper should provide comparisons of convexity and performance among multiple different models within the same domain, thus proving that convexity indeed facilitates generalization. However, the paper only conducts experiments with a single model for each modality, which fails to reveal the impact of convexity on generalization.

Author response:
We very much appreciate the input and concerns of the reviewer:

First,  we apologize for the typographical mistakes, they have been corrected!

Second, following your suggestion we have performed a set of new experiments. In particular, we have added results for two more models for the human activity domain and for the text domain. The findings are consistent with our earlier results: Higher pre-trained convexity is associated with improved downstream accuracy. The human activity data shows a major effect, while the text data models show more modest differences (see new appendix E). 

We hope you find these results as interesting as we do and that you will consider raising your score.

Reviewer comment (Evaluation):
This paper fails to provide any meaningful conclusions. Despite emphasizing the relationship between convexity and generalization, as mentioned above, the existing experiments fail to demonstrate this point or draw any valuable conclusions, rendering the contributions of the paper trivial.

Author response:
Our most important conclusion is that “we find evidence that the higher convexity of a class decision region after pretraining is associated with the higher level of recognition of the given class after fine-tuning in line with the observations made in cognitive systems, that convexity supports few-shot learning.”
Additionally, we hope that this new approach will inspire future investigations of machine learned representations rooted in properties known to be found in human learned representation spaces.

Reviewer comment (Evaluation):
Some experiments raise questions. For example, Figure 3 employs t-SNE to visualize the feature space, but due to the inherent instability of t-SNE, this experiment lacks persuasiveness. In Figure 5, why are there multiple points for each modality? And why is the vertical axis using recall rate instead of accuracy?

Author response:
Experiments

    a. T-SNE is indeed an unstable method for dimensional reduction. We would like to emphasise that the method is only deployed for visualisation purposes to give an intuition into the organisation of the latent space. No conclusions are drawn based on the t-SNE plots.  

    b. We apologise for the lack of clarity in the description of the figure. We will update the description to make it more clear. In the figure, we are looking at the recall rate for each individual class in the fine-tuned model and plotting it vs. the convexity of this class in the pretrained model. The number of points for each model is therefore equal to the number of classes in the domain under investigation for said model.

Reviewer comment (Theory):
Although the authors mention that one of their contributions is proving the stability of convexity to relevant latent space re-parametrization and provide a formal proof in Appendix A, I was unable to find it. Appendix A only mentions the robustness to affine transformations, which is different from re-parametrization. Furthermore, in 2.1.3, there is an anonymous citation. From the context, it seems that the authors may have inadvertently revealed personal information in this citation.

Author response:
Proof in Appendix A on the stability of convexity to relevant latent space re-parametrization
We mean to say that the convexity measures are invariant to relevant re-parametrization of the latent space induced by weight initialisations and other sources of randomness in training. We provide proof that the Euclidiean convexity measure is invariant to affine transformations and that the graph convexity measure is invariant to isometries and scaling. We will update the text to “relevant latent space transformations”.