Paper: Multilinear Operator Networks (ICLR-2024)

Reviewer comment (Experiments):
In mu-layer, is it possible to use low-rank matrices multiplication for A? How will it perform

Author response:
A1：We confirm that such a factorization is possible and we are thankful to the reviewer for suggesting this idea. We factorize the matrix $A$ of our model as $A = RQ$ of rank $k$. We try out different ranks $k$ and report the results below as conducted on a 6M parameter model trained on CIFAR10: 

|                    | Baseline(ours)   | Design 1 | Design 2 | Design 3 | Design 4 | Design 5 | Design 6 |
| ------------------ | ---------------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Rank of matrix $RQ$   | - | 64       | 64       | 64       | 64       | 32       | 32       |
| Rank of $BD$ | 64               | 32       | 64       | 16       | 8        | 16       | 8        |
| Acc                | 88.1             | 87.1     | 85.2     | 86.4     | 86.2     | 86.1     | 85.6     |

Reviewer comment (Experiments):
Is it feasible to employ low-rank matrix multiplication for A in the mu-layer? How would this alternative approach perform?

Author response:
A1：We confirm that such a factorization is possible and we are thankful to the reviewer for suggesting this idea. We factorize the matrix $A$ of our model as $A = RQ$ of rank $k$. We try out different ranks $k$ and report the results below as conducted on a 6M parameter model trained on CIFAR10: 

|                    | Baseline(ours)   | Design 1 | Design 2 | Design 3 | Design 4 | Design 5 | Design 6 |
| ------------------ | ---------------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Rank of matrix $RQ$   | - | 64       | 64       | 64       | 64       | 32       | 32       |
| Rank of $BD$ | 64               | 32       | 64       | 16       | 8        | 16       | 8        |
| Acc                | 88.1             | 87.1     | 85.2     | 86.4     | 86.2     | 86.1     | 85.6     |

Reviewer comment (Experiments):
In the ODE experiments, could the authors offer more detailed quantitative results and compare them with other state-of-the-art methods, such as NeuralODE?

Author response:
A4: We are thankful to the reviewer for the suggestion. Following existing ODE works quantitative evaluation, we conduct additional quantitative evaluation on image classification tasks (MNIST, CIFAR10 as in [1,2]). We follow the setting of [2] and conduct an image classification experiment. The model used consists of less than 0.2M parameters to make a fair comparison. The results in the table below indicate that  MoNet surpasses ANODE in CIFAR10 and SVHN, while it scores between NODE and ANODE in MNIST with accuracies larger than 96%.


| Dataset | NODE      | ANODE     | PolyODE   |
| ------- | --------- | --------- | --------- |
| MNIST   | 96.4%±0.5 | 98.2%±0.1 | 97.2%±0.1 |
| CIFAR10 | 53.7%±0.2 | 60.6±0.4  | 67.1%±0.2 |
| SVHN    | 81.0%±0.6 | 83.5%±0.5 | 84.1%±0.5 |

We show the same main evaluation tasks as [2] in appendix F to compare with those two works. We have presented training time and loss curves similar to those in the referenced work, and in the tasks provided by ANODE, our advantage lies in faster convergence and can restore to symbolic representation.

If the reviewer has any remaining concerns, we would be happy to address them. 

## References

[1] Chen, et al. “Neural ordinary differential equations”, NeurIPS’18.

[2] Dupont, et al. “Augmented neural odes, NeurIPS’19.

Reviewer comment (Reproducibility):
In the ODE experiment setup, is it necessary to use patch embeddings? Could you please provide additional details about the data processing and experimental settings for this specific task?

Author response:
A2: We do not need patch embedding, since the data in the ODE model process time series. The setup for this experiment differs slightly from the image recognition setup, since now our input is a vector. Our data preprocessing uses standard practices in the ODE literature. For the experiment in sec. 4.4, we generated discrete data points using the underlying ground-truth formula. We add noise to the points and use those to train the model. Lastly, our model uses few Mu-layers, while the spatial shift is also not used in this case.

Reviewer comment (Reproducibility):
In solving ODE, could the authors provide more details regarding how the model recovers the right-hand side of the Lotka-Volterra formula?

Author response:
A3：We utilize MoNet to approximate the derivatives in our system of ordinary differential equations (ODEs). MoNet is specifically designed to output the rate of change of the state variables, namely dx/dt and dy/dt (i.e., the two rates of change that Lotka-Volterra includes). The architecture of MoNet generates a polynomial function of the input variables x and y, with the coefficients of this polynomial being determined by the learnable matrices W_i. Through the training process, MoNet learns the "optimal" values of these matrices W_i from the given data, effectively learning the dynamics described by the ODE. Once training has converged, the learned weights W_i are directly used to construct the specific forms of dx/dt and dy/dt.

Reviewer comment (Novelty):
Virtually no motivation for the work aside from making polynomial networks have better generalization. We are given a sentence in the introduction "Interpretability and encryption..." with no reasons at all given as to why the proposed model architecture would be any more amenable to interpretation or encryption, and certainly no such results/analysis. Even the most academic work needs motivation. Getting closer to reaching state-of-the-art performance on some task alone is not enough of a motivation for a method if there are no tangible benefits over existing methods (and especially when it seems there are disadvantages in memory usage as covered below).

Author response:
A3: We respectfully disagree with the reviewer. The community does value diverse architectures and in fact much of the progress observed can be (partly) attributed to the architecture design, which manifests a different inductive bias on the task at hand. 
In this work, we propose an orthogonal direction to neural architecture design that does not follow the pattern of stacking linear/convolutional layers and non-linear activation functions. We believe that by showing that such an alternative can perform better or on par with standard baselines, we contribute a valuable direction in the search for the optimal deep architecture. As such, even though, as Table 2 suggests, our model outperforms both ResNet 50 and ViT, we believe our contribution lies beyond this improvement in performance.

Reviewer comment (Theory):
Similarly, why are we interested in getting rid of activation functions? No motivation is given for this explicitly, and the only reason I can think of given the interpretability word is to remove non-linearity in the mapping from input to output. A sentence in the conclusion backs this up: "which avoids the requirement for activation functions or other non-linear mappings". However, the "multilinear" operation in this work is itself still a non-linear mapping of input to output (a polynomial of degree 2), making this statement false.

Author response:
A4: The mapping we use is multilinear. We use the standard terminology as used in the tensor literature that extends beyond the field of machine learning. The links between multilinear algebra and signal processing have long been established and use a standard terminology, as can be verified in [12-14]. For instance, an interested reader can find further information on the seminal review paper of Kolda and Bader [11]. 

However, in the revised text we clarify in the introduction that we are using this multilinear terminology and provide links for the interested reader. 

In addition, to avoid any misunderstanding, we have also clarified the statement in the conclusion that we can avoid the activation functions, instead of all non-linear mappings.

Reviewer comment (Novelty):
The background is *way* too short and doesn't give near enough information on the current work's context in polynomial network literature.

Author response:
A9: We appreciate the remark from the reviewer and the effort to improve our work. We have included a new section in the appendix (sec. B in the revised manuscript), where we provide additional background specifically for polynomial networks.

If the reviewer has any remaining concerns, we are happy to provide further clarifications. The revised text is denoted in red for readability.

Reviewer comment (Novelty):
While the method is detailed enough, with much more detail in the appendix on some of the most interesting bits, the paper doesn't explain much at all what specifically is different from current polynomial networks, seemingly assuming the reader should know this. It is up to the authors to explain how their method fits into the current research literature in the background and method and compare their method explicitly with the closest existing work in literature.

Author response:
A1: The difference from previous polynomial networks lies in our goal to design polynomial networks (PNs) that are competitive to the state-of-the-art networks *without relying on activation functions*. To the best of our knowledge, the core works that have explored PNs without activation functions are [2, 16, 17]. 

To achieve that, we are inspired by the modern setup of considering the input as a sequence of tokens. The token-based input is widely used across a range of domains and modalities that last few years. The proposed modules (Mu-Layer and Poly-block) are customized to correspond to this form of data, which differs from the modules proposed previously. We note that this approach has not been considered by previous PNs. Furthermore, we propose additional modules, such as the pyramid patch embedding, which have not been considered before, especially in the context of polynomial networks.

We highlight that a byproduct of our design (when compared to previous PNs) is that our method relies on matrix multiplications instead of convolutional layers. Arguably, this results in a weaker inductive bias as pointed out in the related works of MLP-based models. This can be particularly useful in domains outside of images, e.g., in the ODE experiments.

Reviewer comment (Experiments):
Requires 4 A100 GPUs to train for a batch size of 448 according to the authors (and they say they maximize batch size). This means that the maximum batch size on each GPU is 112. This is **much more VRAM usage** than the baselines being compared to that are achieving similar accuracy on ImageNet1K - for example, ViT B/16 can fit with a batch size of 256 on a single A100, and ResNet-50 a batch size of 512 on a single A100. Note it is not clear if these are 40GB or 80GB A100 models.

Author response:
A2: We are thankful to the reviewer for helping us clarify this potential misunderstanding. **MONet does not consume more VRAM**. We kindly remind the reviewer that the reported batch size in the paper without “the total batchsize” usually refers to the batch size used per single GPU, and this terminology is commonly employed in similar works as a common practice [1-3]. 

However, to avoid any further misunderstanding, we have changed the text to refer explicitly to the batchsize “per GPU”. In our experiments, we use a batchsize of 448 per GPU, which is larger than the batchsize of 256 the reviewer reports for ViT. Overall, the total batchsize of 1792 is used in our case.

Reviewer comment (Presentation):
Bold figures in captions are misleading, and do not it turns out always represent the best result as you might expect! For example, in Table 2.

Author response:
A8: We respectfully disagree with the reviewer. As typically done in the literature, we conduct experiments in different categories of architectures. The categories are often determined by the number of parameters, where the “small” category uses less than 15 million parameters, medium models with less than 40 million, while the “large” category exceeds 40 million parameters. 

However, to increase the readability of our paper, we have indicated the small models in green color, medium models in red color and the larger models in blue color.

Reviewer comment (Theory):
Fundamentally, it's unclear why the MONet should generalize well to other tasks if the Poly-block is only capturing up to fourth degree interactions. For instance, for general MLPs, results such as Cybenko's theorem tells us the MLP should capture many families of Lebesgue integrable functions but here the limit is restricted to functions that are captured by compositions of fourth degree approximations and skip connections. The surprising capacity of the MONet is not adequately discussed.

Author response:
A1: We thank the reviewer for the remark. Let us clarify why the total network results in a higher-degree expansion. We do mention in the text that the Poly-block captures up to 4th degree interactions as the reviewer mentions. However, we also note that a single network is composed of several Poly-blocks, usually tens of those. 

*Simple case with two Poly-blocks*: The output of the first Poly-block (which is 4th degree interactions of its inputs) of the network is directly an input to the next Poly-block. The output of the second Poly-block is again 4th degree interactions of the output of the first block (which was already 4th degree of the inputs). Therefore, the output of the second Poly-block with respect to the input (e.g., image) captures up to 16th degree interactions. 

To generalize from two Poly-blocks to a sequence of those as used in our experiments, we notice that the output of $N$ sequential blocks is $4^N$. Given than $N>10$, we assume that this is a sufficient number to capture high-degree interactions. In practice, we observe that in our experiments this degree is sufficient for a competitive accuracy. 

Nevertheless, we agree with the reviewer that it remains an open and interesting problem of identifying the degree of interactions required by the total network to tackle each task. We have inserted a related note in sec. 3 to inform the reader, since we do believe this is an interesting future step. The note is marked with red color for visual clarity.

Reviewer comment (Evaluation):
The numbers reported for the P-Nets [1] do not appear to be accurate. With >11 parameter P-Nets, the achievable accuracy on CIFAR-10 is 94.5% and ImageNet ~77% Top-1 accuracy/ ~94% Top-5 accuracy. The exact accuracies reported in [1] would either closely-defeat or match the performance of the MONets with similar or drastically fewer numbers of parameters. Please address this discrepancy.

Author response:
Please let us clarify. **There are no restrictions placed on $\Pi-$Nets**. 
However, there is a difference in the experimental settings. The following models are used: 
$\Pi-$Net (Table 2): This refers to the small model (equivalent to ResNet-18) WITHOUT activation functions. 
Hybrid $\Pi-$Net (Table 2): This is also a small model WITH activation functions.

We elaborate on these points below:
The numbers referred to by the reviewer (e.g., 94.5% on cifar10) are reported by the authors when **activation functions are used**. For instance, in the ImageNet experiment in [1] (page 7, sec. 4.2) they identify that “the second order of each residual block is normalized with a hyperbolic tangent unit”, i.e., a tanh activation function is used in each block. However, if we compare with $\Pi-$Nets without activation functions, then we obtain those scores reported in our Table. In fact, the authors of [1] have conducted some of those experiments in the supplementary material, e.g., in Table 1 [here](https://openaccess.thecvf.com/content_CVPR_2020/supplemental/Chrysos_P-nets_Deep_Polynomial_CVPR_2020_supplemental.pdf). This is also explicitly identified by the authors of [1] in the code release of the paper. In this [table](https://github.com/grigorisg9gr/polynomial_nets/tree/master/classification-NO-activation-function#imagenet-1k) they elaborate on the accuracy reached when using activation functions and when you do not use activation functions. In conclusion, *there is no restrictions placed  on our reported results on $\Pi-$Nets; we simply report $\Pi-$Nets without activation functions*. 

Note that our original manuscript identified that $\Pi-$Nets do not use activation functions in Table 2 (in the second-to-last column). The models with activation functions were notified as `hybrid $\Pi-$Nets’. However, to clarify this point even further, we have inserted a footnote that explicitly identifies that we are using the $\Pi-$Nets without activation functions. 

Furthermore, the accuracy of 77% on ImageNet (in [1]) involves a ResNet-50 model and the equivalent $\Pi-$Net WITH activation functions. That would be under the category of medium models in our case, where the accuracy of our model is 81.3%.

Reviewer comment (Experiments):
Is there some intentional restriction placed on the P-Net method? I believe you have pulled the numbers from [2], which is also in discrepancy with the first work (by the same authors, no less). Similarly, the ResNet 50 numbers are reported strangely. It can be easily fine-tuned to achieve 78.8%/94.5% Top1/5 accuracies, respectively, but this is not in concordance with Table 2.

Author response:
A3: We use the accuracy as reported in [A], for a fair comparison. To avoid any confusion, we have replaced this with the accuracy reported in the original ResNet paper (now the accuracy on Table 2 is 77.2%). We do agree that additional techniques, such as finetuning, can improve the results further, however we would need to apply similar techniques to all methods for a fair comparison. Our evaluation is the standard and fair evaluation for all different architectures. 

Nevertheless, we are thankful for the suggestion. To increase the transparency on the reported numbers, we added sec. K.3 (appendix), where we explain how we obtained the core compared results on the ImageNet experiment. 

If the reviewer has any remaining questions or concerns, we are happy to address them. 

[A] S2-mlp: Spatial-shift mlp architecture for vision, WACV'22.