Paper: Block-operations: Creating an Inductive Bias to Route Data and Reuse Subnetworks (ICLR-2024)

Reviewer comment (Presentation):
The paper may have been hastily prepared, as indicated by the presence of inappropriate tables and figures, minor typographical errors, and a lack of overall organization.

Author response:
**On "Could you address the concerns raised about inappropriate tables and figures, as well as minor typographical errors in the paper?"**: We will be happy to fix any errors we find, but is unclear to us in what sense the tables and figures are inappropriate. Could you provide an example?

Reviewer comment (Writing):
The authors mention that "there is no inductive bias that would cause FNNs to keep activation patterns for the same concept consistent throughout the network." This motivation should serve as the crux of the paper, yet it lacks a clear and thorough discussion.

Author response:
**On "there is no inductive bias that would cause FNNs to keep activation patterns for the same concept consistent throughout the network."**: We refer to Csordás et al. (2020) for a thorough explanation.

Reviewer comment (Novelty):
The authors state, "In this way, we attempt to construct a neuro-symbolic system using a purely connectionist approach, as suggested by Greff et al. (2020) as a possible solution to the binding problem." This raises questions about whether the authors are merely adopting an existing solution, and it is unclear what the unique proposition and specific contributions of this paper to the field are.

Author response:
**On "What, then, is the unique proposition of this paper?"**: The concept of block operations is entirely novel. Greff et al. (2020) made a very generic call for research in this direction, while block operations are a quite specific solution.

Reviewer comment (Theory):
Could you provide more context and a thorough discussion regarding the statement that "there is no inductive bias that would cause FNNs to keep activation patterns for the same concept consistent throughout the network"? How does this serve as the motivation for your research, and why is it important?

Author response:
**On "there is no inductive bias that would cause FNNs to keep activation patterns for the same concept consistent throughout the network"**: If a concept is not represented consistently then the network can not learn reusable subnetworks, because it it not possible to reuse subnetworks that expect different input formats. Imagine if you wrote a computer program where numbers are represented as int, float, double, etc. at different parts of the algorithm even if they are all integers, so that you end up needing to write a separate function for each of them.

Reviewer comment (Novelty):
The paper mentions the attempt to construct a neuro-symbolic system using a purely connectionist approach, as suggested by Greff et al. (2020). Can you clarify how your approach differs from or builds upon this suggestion? What novel elements or contributions does your paper bring to the concept of a neuro-symbolic system?

Author response:
**On "What novel elements or contributions does your paper bring to the concept of a neuro-symbolic system?"**: There are a lot of different possible neuro-symbolic systems. Our main novelty are block-operations. They are unique in that they are designed to satisfy all the goals outlined in section 3.1., which has not previously been done.

Reviewer comment (Presentation):
Could you provide a clearer and more structured organization of the paper to improve its readability and comprehension for readers who may not be experts in the field?

Author response:
**On "Could you provide a clearer and more structured organization of the paper"**: We are uncertain what parts are unclear, but can make improvements that are pointed out to us.

Reviewer comment (Novelty):
What specific issues or challenges in the field of Feed Forward Neural Networks (FFNNs) are you addressing with your proposed concept of block-operations and the Multiplexer? How does this concept enhance FFNNs, and what practical applications or benefits can be derived from it?

Author response:
**On "What specific issues or challenges..."**: The goals outlined in section 3.1. can be achieved with block-operations. This makes networks able to learn to reuse concepts more easily and improves modularity. We expect gains in performance and generalization on tasks that rely on modularity.

Reviewer comment (Experiments):
Are there any empirical results or experiments that support the claims made in the paper regarding the performance and effectiveness of the Multiplexer compared to traditional FFNNs? Providing evidence would strengthen the paper's arguments.

Author response:
**On "Are there any empirical results or experiments"**: We focused on synthetic tasks so far because our focus was on proving the soundness of the underlying logic (block-operations), not on beating SOTA with a specific architecture. Our experiments confirm the presence of the properties we were looking for. Running practical tests are next steps in our research.

Reviewer comment (Novelty):
How do the concepts of "block-operations" and "Multiplexer" relate to the broader field of neural network architectures, and what potential implications do they have for various applications beyond the scope of the paper?

Author response:
**On "How do the concepts of "block-operations" and "Multiplexer" relate to the broader field of neural network architectures..."**: Existing neural networks of all types can be modified to make use of block-operations, which may improve their performance on tasks that require modularity.

Reviewer comment (Presentation):
Could you address the concerns raised about inappropriate tables and figures, as well as minor typographical errors in the paper? How can these issues be resolved to enhance the overall presentation of your research?

Author response:
**On "Could you address the concerns raised about inappropriate tables and figures, as well as minor typographical errors in the paper?"**: We will be happy to fix any errors we find, but is unclear to us in what sense the tables and figures are inappropriate. Could you provide an example?

Reviewer comment (Novelty):
The paper lacks soundness (and presentation quality though that may not be a cause for rejection). I'm not sure that the block-operations / multiplexers are doing something qualitatively different from an FNN with residual connection. The behaviour of the multiplexer can very well be simulated by a standard FNN, so I'm not exactly sure what the inductive bias is / where the improvements are coming from. At the very least, it would be good to know how well baselines were tuned / how do results change with slightly bigger models.

Author response:
**On residual connections**: Residual connections do not satisfy the criteria outlined in Section 3.1. as our design goals. They only operate on the entire tensor and can not rearrange parts of the tensor independently of each other. This causes overlap and interference between the neurons used to represent different concepts.

**On model sizes**: We did test models of various sizes and compared performance. As pointed out in the paper, the details on this are in the appendix. Testing performance at even larger model sizes would not be very practical as the tasks are deliberately quite simple and do not require large models.

Reviewer comment (Novelty):
The paper also claims to introduce "block operations" to split activations into multiple blocks. This is definitely not new. For instance, even in ordinary self-attention, hidden states are divided into multiple blocks which are separately consumed by distinct attention heads.

Author response:
**On the novelty of block operations**: Other architectures have certainly used mathematically similar operations, but not for the same purpose. Please read Section 3.1. to understand the purpose of these operations. Saying that block operations are not new is like saying that self-attention was not a novel concept when it was first introduced because the softmax function had been used by other papers before.

Reviewer comment (Experiments):
The specific datasets chosen also seem to be extremely toy-ish for compositional generalization / modularity. Would the proposed approach work on e.g. MNIST (still toy-ish but larger scale than the datasets in the paper).

Author response:
**On the example tasks**: The tasks are deliberately simple in order to test for the presence of certain properties, which we experimentally confirm. Testing MNIST would not be conducive to our goals: MNIST is a task that relies largely on statistical correlations and requires no reuse of concepts in different contexts. SMFRs help specifically for tasks that rely on modularity. We have run tests on MNIST before and got the results we expected: The SMFR learns not to use routing and learns to simulate an FNN instead. It ends up with the same performance as an FNN. SMFRs only achieve better performance on a task where modularity is useful.

Reviewer comment (Experiments):
The main weakness is the quality and design of the experiments, which failed to convince me about the validity of the architecture, and the clarity of the described method.

Author response:
**On the experiments**: The experiments are deliberately designed to be a minimal test of the attributes we want to test. Our goal is to test if the concept works as intended, not to beat SOTA. You are correct that the arithmetic tasks are very simple and can be solved largely by memorization. We did also run tests on 2 digits and noticed the same patterns, but since these take much longer to run we focused on the 1-digit version afterwards. While 1-digit arithmetic can more easily be solved with memorization than 2-digit arithmetic, this does not actually affect the validity of our results. Routing and module reuse are just as relevant for memorization tasks as for other types of tasks.

Reviewer comment (Experiments):
The arithmetic tasks are all using 1-digit numbers. This limits the total number of possible training examples in the order of 100. This can result in very easy memorization, and it is also unclear if it provides enough "motivation" for the network to learn to route. The simplicity could be an explanation for why the difference is relatively small between FFN and SMFR in Tab 1. The original paper by Csordás et al had 2-digit inputs, which resulted in ~10k possibilities, which is significantly higher.

Author response:
**1-digit arithmetic**: We agree that a low task complexity like this would normally be cause for concern, but we believe that it is not important in this case: For experiments 1 and 2 we care about interference and reuse between tasks and the complexity of the task should not influence greatly how well the network is able to do this. We did run a few trials with 2-digits and those did confirm that we get the same results. We did not run enough trials with 2 digits to report them, because those experiments take longer to run. We can run some more trials with 2 digits if necessary, but we will only be able to run a small number of experiments by the deadline.

Reviewer comment (Experiments):
The authors sometimes report performance on all hyperparameters averaged together. This unnecessarily presents their results weaker than they are, and it also makes it harder to understand the actual performance gain. I don't see the utility in averaging in bad hyperparameters. For example at the end of P6, they report an average of "0.205 OOD accuracy", and then "...For these architectures, 75% of trials achieved 100% OOD accuracy". So what is the mean accuracy of the architecture with the best hyperparameters then? Could you please report the mean (and if possible the standard deviation) on the best hyperparameter configuration, as usually done in the literature? Similarly, on page 7, it says "there was a range of values where all experiments converged with 100% accuracy on all iterations", yet in Fig 4 I don't see any method that is 100% on all iterations.

Author response:
**On how to report performance**: You criticize that we present our results as weaker than they are. Isn't that a good thing? Most papers report the best hyperparameters because it makes their results look better, but this also contributes to replication issues. We deliberately took a pessimistic view, under the assumption that in real-life tasks you won't be able to test all possible hyperparameters. We show that performance is good even so, when averaging over many hyperparameters.

Reviewer comment (Reproducibility):
Some details are unclear: For example, if the weights are shared in the Multiplexer, FNNR, and inside the "Derived Blocks" in FNNR. This should be described by equations in the main paper.

Author response:
**Weight sharing**: No weights are shared. Each of the blue blocks marked "FNN" in the image is just that: A single FNN.

Reviewer comment (Presentation):
In FNNR block in Fig.3, and in the description below it, it mentions that "the FNNR module uses the input blocks of the Multiplexer as its extra input". What are the standard, non-extra inputs? What is the *output* of the Multiplexer used for?

Author response:
**inputs/outputs**: The Multiplexer takes M blocks and produces N blocks. The FNNR takes the concatenation of both the M original blocks and the N blocks produced by the Multiplexer as input. In this way it can condition both on the original input before interpolation and reordering and on the interpolated blocks produced by the multiplexer.

Reviewer comment (Theory):
In the explanation of why SMFR performs better on Addition/Multiplication, at the bottom of P5, it says: "... because the softmax used by the Multiplexer is also commutative". What does this mean? The output of the softmax depends on the logits, and those on the specific input blocks unless the gating weights are shared between them. This does not seem to be the case, as far as I can tell from Fig 2 and 3, and the related text.

Author response:
**On "... because the softmax used by the Multiplexer is also commutative"**: We mean the following: If the network learns (for example) to calculate the softmax as f(x,y, y)=0.5*x+0.5*y+0.0*z where x and y are some blocks, then this function will be commutative with respect to x and y. This would allow the network to solve commutation-based tasks even for OOD data, i.e. even for values of x and y that were not seen in that order during training. While this is a good thing, it is not what we want to test. We therefore make it harder for ourselves by using gumbel-softmax, which does not have this property. This lets us be more confident that the performance gain indeed comes from the routing abilities of the SMFR.

Reviewer comment (Evaluation):
**Semantic Grouping**: The paper claims that block operations group the neurons into semantic units, but the grouping is actually done arbitrarily. The paper doesn’t provide evidence for semantic grouping or disentanglement to properties etc, quantitatively or qualitatively.

Author response:
**Semantic Grouping**: We should clarify what we meant. As you say, The neurons are not grouped semantically in advance. However, the network has an inductive bias that causes the groups to take on semantically distinct roles. The experiments confirm this assumption.

Reviewer comment (Writing):
**Clarity**: The writing, clarity and overall presentation could be improved significantly. This is especially the case up to page 5. Instead of having coherent sections with conjunctions and transitions, the paper is organized as lists of bullets or disconnected paragraphs, and in many cases not enough details are provided on each of them. Personally I think this form of succinct writing is less suitable for a paper, and would encourage the authors to work on the writing further both in order to increase its flow and coherency, and also to extend it to include more details. In the questions/suggestions section I refer to specific content that is missing through the different sections.

Author response:
**Clarity**: Not sure what to say here. Different people have different preferences? We have had positive feedback about the bullet-point style before.

Reviewer comment (Writing):
**Terminology**: Multiple terms are used without being defined clearly and precisely enough: e.g. routing, uniform representations, representation-preserving mappings, copy gate. A notable additional example is that of negative interference, a term that is mentioned repeatedly through the paper but isn’t explained or illustrated. A further discussion, formal definitions or concrete examples could help clarify the mentioned terms.

Author response:
**Terminology**: We would have liked to give more details on the things you mentioned, but we are constrained by the page limit. Explaining new ideas took priority over explaining things.

Reviewer comment (Experiments):
**Datasets**: The experiments in the paper are performed over synthetic discrete data only. I would suggest either exploring multiple modalities or real-world data to strengthen the empirical evaluation.

Author response:
**Datasets & Baselines**: The focus of this paper is on the core idea of Block Operations. Since the SMFR replaces the FNN, we only compare with it and run only tests that evaluate what we want to test. Real life tests are left as future work. As pointed out in the paper, even if SMFRs in their current state of development should perform poorly, our results show that the underlying idea of block-operations works as intended and is worth exploring further.

Reviewer comment (Experiments):
**Baselines**: The experiments compare the new approach only to FNNs as baselines. Comparing to other approaches, such as recurrent networks and transformers, is in my opinion critical.

Author response:
**Datasets & Baselines**: The focus of this paper is on the core idea of Block Operations. Since the SMFR replaces the FNN, we only compare with it and run only tests that evaluate what we want to test. Real life tests are left as future work. As pointed out in the paper, even if SMFRs in their current state of development should perform poorly, our results show that the underlying idea of block-operations works as intended and is worth exploring further.

Reviewer comment (Novelty):
**Novelty**: The authors contrast between self-attention and the new multiplexers idea, mentioning the strength of multipliers allowing for different “inputs and outputs”. These could be modeled by cross attention (which is broadly used too but isn’t mentioned in the paper). I think both a discussion of the conceptual differences as well as empirical head-to-head comparison between the proposed idea and cross attention will be very important.

Author response:
**Novelty**: Attention models of all types are mathematically similar to SMFRs, but they have one crucial difference that makes a comparison pointless: Attention models are still connected to each other with fully-connected layers. These re-introduce the difficulties in data transfer that the SMFR sets out to solve.

Reviewer comment (Novelty):
**Background & Related Works**: Not enough context is provided on the current issues of FNNs, the prior attempts to address them, what are their specific limitations that the paper seeks to address. For instance, where the paper mentions that a prior work “calls for additional research on suitable inductive biases” – what prior inductive biases have been explored in the literature, how is your approach different from them, etc. Likewise, the related works section mentions Capsule Networks, Routing Networks and Recurrent Independent Mechanisms, but provides a particularly brief description of them, and doesn’t in which ways they are similar or different than the new approach. I would also recommend discussing slot attention and its extensions.

Author response:
**Background & Related Work**: The reason we do not talk about these in detail is because none of them address the design goals mentioned in Section 3.1. We just mention that they exist to show that we are aware of them, because several of them look similar at first glance but they all fail to achieve the goals outlined in 3.1, usually because they were designed for different purposes.

Reviewer comment (Experiments):
**Implicit Grouping**: It could be that FNNs perform some degree of grouping implicitly, even without externally imposing the grouping through inductive biases. In hierarchical deep networks it’s likely to be the case. It would be therefore good to either show empirically that FNNs do not do that effectively enough on the tasks that you study, or rephrase/tone down the paper’s claims about FNNS.

Author response:
**Implicit Grouping**: As shown by Csordás et al. (2020), even if implicit groupings arise, they are not maintained throughout multiple layers of the network.

Reviewer comment (Theory):
The paper says: “fully-connected layers treat all neurons as independent of each other”. I disagree with that on several levels. Since they have non-linearities, FNNs can model arbitrarily complex functions that do account for various subtle relations between the activations. They do the opposite of treating neurons as independent factors, and in fact research efforts on e.g. disentanglement aim to encourage them to treat latent factors more independently then what they do by default. In another level, models that are based on slots, tokens, key-value, or in the case of transformers heads, already incorporate inductive biases for grouping of activations into larger units. It would be good to discuss these in the paper.

Author response:
**On “fully-connected layers treat all neurons as independent of each other”**: We will phrase this differently. What we meant was that neurons are treated as independent by the learning process. I.e. their index within the tensor has no effect. Once learning has taken place, the meanings they have taken on are not independent, as you say. The existing models you mention that do group neurons into larger layers perform these groupings for different reasons that do not address our design goals in section 3.1.

Reviewer comment (Theory):
Section 3 says: “there is no inductive bias that would cause FNNs to keep activation patterns for the same concept consistent throughout the network.”. That’s a very strong statement that I’m not sure if there’s a way to prove. I would recommend rephrasing it.

Author response:
**On “there is no inductive bias that would cause FNNs to keep activation patterns for the same concept consistent throughout the network.”**: This is a finding of Csordás et al. (2020).

Reviewer comment (Writing):
The last sentence in the abstract isn’t clear: “Adapting more complex architectures than the FNN to make use of them could lead to…” <-- are them the FNNs that are altered, or the current proposed idea, or future structural extensions?

Author response:
**About the abstract**: "Them" refers to block-operations.

Reviewer comment (Writing):
In section 3: “more dense” -> “denser”.

Author response:
**Typos**: The phrase "investigate in how far" is valid English and means "investigate to what extent". "More dense" should indeed be replaced with "denser". Thank you for pointing this out!