Paper: Simple mechanisms for representing, indexing and manipulating concepts (ICLR-2024)

Reviewer comment (Writing):
Section 6 very briefly explains a link between the proposed approach and transformer architectures, due to the use of an attention mechanism within the manifold. This connection with transformers should probably be made more explicit, as it is not easy to grasp from Section 6.

Author response:
**Connection to Transformer**: We have added a figure (Figure 3) to explain the connection of our proposed learning architecture to transformer architectures. On a high level, a random feedforward network can contain the signature $S(x) = \phi(x) \phi(x)^\top$ for each token and attention can be thought of as combining tokens from the same concept together to obtain $M(X)$ and the subsequent feedforward network can be used to obtain $T(X)$ using $M(X)$ because $T(X)$ is the null space of $M(X)$ and it can be obtained by taking by repeated multiplication of $(I-M(X))$. We have added this discussion to the updated paper.

Reviewer comment (Experiments):
The major limitation is the missing of experiments.

Author response:
We agree with the reviewer that the present paper lacks experimental work. The paper aimed to take a first-principles approach to the challenging problem of formalizing the recognition of concepts and deduce the implications of such a formalism. While there are several empirical investigations of such questions for existing architectures and methods, the formalism of concepts is quite poorly understood, and it would help to develop theoretical formalism first without being bogged down by the noise in data, variations in architectures, training methods, and datasets. Designing extensive experiments and developing the framework for real-world datasets will be a substantial body of work by itself, just as developing the formalism here was.

Reviewer comment (Experiments):
If the relation to Transformer can be established, can the new method outperform Transformers in some tasks?

Author response:
As the reviewer points out, our formalism does suggest a modified transform architecture that includes a separate concept storage unit (dictionary of concepts) and a concept discovery unit, which is much smaller than today’s giant transformers. See Figure 2 in the modified pdf. Thus, our theory formalizes the importance of retrieval-augmented transformer architectures.

Reviewer comment (Writing):
The writing of "Proposition 1.1. (Informal summary of results)" is strange. By "Proposition" we expect formal results, but followed by "(Informal summary of results)".

Author response:
Proposition 1.1 is to describe the main results before going into the detailed statements presented later in the paper. We can change the name ‘Proposition 1.1’ to “Informal summary of results”.

Reviewer comment (Theory):
Think of the character recognition (MNIST). There are ten concepts: 0, 1, 2, ..., 9. Given images of characters x_1, x_2, ..., each image has the size of 28X28 pixels. How should the kernel signature be defined?

Author response:
We note that the kernel signature is defined specific to a dataset and the choice of the kernel phi. For MNIST, one could view each image as a point cloud where each point represents (pixel-coordinates, pixel-value). You could now compute the kernel signature for each image $M(x) = \mathbb{E}[ \phi(x) \phi(x)^\top ]$ where expectation over a group of related points that attend to each other. As explained in Section 3.1, the attention will cause points from the same continuous curve to attend to each other. If the image had only one curve (e.g., an image of number “0”), then this would result in a signature for that curve. An image containing multiple curves (such as the image of number “7”) would result in two signatures at the second layer. The third layer will group these two curves into one signature (see section 3.2) and this signature can be used to identify similar images.

Reviewer comment (Theory):
In the mechanism behind the learning method, the attention is simply based on the cosine similarity. Does this have enough representation power to distinguish manifolds located at the same orientation but different distances to a reference manifold?

Author response:
The learning method uses a previously computed representation (such as $\phi(x)$) and applies attention on $\phi(x)$ based on cosine similarity (see figure 3 for more details where the attention is actually over $S(x)$ which is $\phi(x) \phi(x)^\top $. Thus, $S(x).S(y) = (\phi(x).\phi(y))^2 \ )$. As the reviewer points out, cosine similarity cannot distinguish manifolds with the same orientation but different distances but since we are using the kernel transform, they can be distinguished by a suitable kernel (for example, if the distance to a reference point is included as one of the features in the kernel).

Reviewer comment (Experiments):
It is unclear whether this framework is particularly useful, either for understanding concept acquisition on currently-used transformer architectures, or for learning in their proposed architecture. For example, there are good reasons to believe that Capsule-Nets should learn to hierarchically decompose complex objects into re-usable parts, but most of the existing literature on Part-Whole hierarchies has encountered great difficulties in being applied to non-toy datasets when training with e.g. gradient descent. As such, the lack of experiments makes it unclear that a reader is well-served to work through this theoretical paper, as it is not immediately obvious that theoretical guarantees around inner-product proximity and signature computation will reflect the learning of disentangled concept manifolds in practice.

Author response:
We thank the reviewer for their time and effort in providing feedback on our work. We agree with the reviewer that the present paper lacks experimental work. The paper aimed to take a first-principles approach to the challenging problem of formalizing the recognition of concepts and deduce the implications of such a formalism. While there are several empirical investigations of such questions for existing architectures and methods, the formalism of concepts is quite poorly understood, and it would help to develop theoretical formalism first without being bogged down by the noise in data, variations in architectures, training methods, and datasets. Designing extensive experiments and developing the framework for real-world datasets will be a substantial body of work by itself, just as developing the formalism here was.

Reviewer comment (Theory):
Which constraints, if any, does the assumption of a polynomial manifold impose on the form of the input vectors which this framework could model? In particular, are there reasons to expect the assumption of linear manifolds associated with concepts to hold in practical settings, where e.g. the inputs, $x_{t}$, might be features from a complex neural network?

Author response:
Regarding the question about polynomial manifolds, the reviewer is right that this is an assumption that constraints on the inputs. However, the embedding mapping is meant to capture them and allows for wider applicability. 

Regarding the question about polynomial manifolds, the reviewer is right that we are constraining vector representations of different instances of a concept to change smoothly and lie in a manifold however, we believe that even if this is not true in the input layer as one computes signatures up the layers hierarchically one gets smoother representations . Although we have argued this  concretely only for simple concepts such as circles,  rectangles, and stick figures we believe this should be true more generally.

Reviewer comment (Experiments):
Although the analyses and demonstrations are comprehensive, the authors do not provide any experiments to support their theories or how their theories can be applied to existing models.

Author response:
We thank the reviewer for their time and effort in providing feedback on our work. We agree with the reviewer that the present paper lacks experimental work. The paper aimed to take a first-principles approach to the challenging problem of formalizing the recognition of concepts and deduce the implications of such a formalism. While there are several empirical investigations of such questions for existing architectures and methods, the formalism of concepts is quite poorly understood, and it would help to develop theoretical formalism first without being bogged down by the noise in data, variations in architectures, training methods, and datasets. Designing extensive experiments and developing the framework for real-world datasets will be a substantial body of work by itself, just as developing the formalism here was.

We hope our response resolves the reviewer's concerns, and we are happy to discuss further to clarify any further concerns.