Paper: Connected Hidden Neurons (CHNNet): An Artificial Neural Network for Rapid Convergence (ICLR-2024)

Reviewer comment (Experiments):
It seems that the same learning rate was used for both the proposed architecture and the standard MLP. This seems inappropriate: log-likelihood (cross-entropy) loss yields higher gradients than the mean squared loss, as such one usually picks smaller learning rate value for cross-entropy. The comparison between architectures will only be fair if the hyperparameters are optimised separately for each, the proposed model and the MLP.

Author response:
Hyperparameters: Regarding the hyperparameters, our hypothesis was that because of the set of weights referred to as $w_2$ in the paper, the model should outperform FNN in complex datasets. We had conducted additional experiments and concluded that CHNNet outperformed FNN in complex datasets like the FMNIST and EMNIST datasets.

Reviewer comment (Writing):
The authors do not always adhere to correct formatting standards. A few grammatical mistakes are present in the paper. Please see below for the suggested list of corrections:  
- “disconnected from biological reality. (Akomolafe, 2013).” - please remove the full stop prior to the reference.  
- “model depicted a noticeable increase” - model exhibited  
- “While Hopfield use” - While Hopfield used  
- “UNet++(Zhou et al., 2018), “ - add a space in front of the bracket  
- “Though the forward propagation mechanism of the proposed model echoes the forward propagation mechanism of conventional RNNs, in conventional RNNs, the activations of the hidden neurons, obtained from the prior time step, are used to calculate the output of the hidden layer, whereas in CHNNet, the current pre-activations of the hidden neurons are used to calculate the output of the hidden layer.” - this sentence is too long, please break it up into three separate sentences.  
- “equation 3 and equation 4” - Equation (3) and Equation (4)  
- I suggest that the contributions are listed in present tense.

Author response:
Formatting: We will follow the suggested formatting in the camera-ready version of the paper if it gets accepted.

Reviewer comment (Theory):
Regarding the statement *“the difference between the cost of CHNNet, generated at two sequential time steps, is greater than that of the conventional FNN; that is, CHNNet converges faster than the conventional FNN.”* — is this really a valid statement? From the loss landscape perspective, a higher difference in loss values implies “steeper” gradients. Whether or not that would lead to faster convergence is not easily answered. With very steep gradients one might experience higher oscillations and stronger sensitivity to the learning rate value. The authors should only state what they know for certain: that the proposed approach provably yields higher gradients (which may lead to faster convergence).

Author response:
We will state the suggested statement about the steeper gradient in the camera-ready version of the paper if it gets accepted.

Reviewer comment (Evaluation):
The p-values show that the difference in performance is not statistically significant for the proposed model and the MLP. What about convergence speed? Can that be quantified and statistically evaluated for significance? Also, what about runtime efficiency—in absolute terms, when the number of parameters is comparable, which architecture is quicker to train? The authors should evaluate and/or comment on this.

Author response:
The p-values show that the difference in accuracy gained by the models is not statistically significant in many cases. However, our main goal was to get faster convergence than FNN, and in some experiments, we also got performance gains compared to FNN. Further, with an equal number of parameters in CHNNet and FNN, the number of computations per epoch is nearly equal as well. Thereby, the training time for CHNNet and FNN with an equal number of parameters will be nearly equal as well.

Reviewer comment (Theory):
However, the mathematical formulation of the recurrence thus occurring in the networks seems incorrect, and equivalent to a feed-forward neural network.

Author response:
The recurrent formulation will make the model a variant of RNN architectures. However, CHNNet is different from RNN, as discussed in section 1.1, and is in fact similar to a conventional FNN with intra-connected hidden layers. The novelty of the model lies in the introduction of a new set of weights, responsible for the rapid convergence as discussed in section 3.3 of the paper, and the approach used to connect the hidden neurons.

Reviewer comment (Experiments):
Why have you avoided a recurrent formulation?

Author response:
The recurrent formulation will make the model a variant of RNN architectures. However, CHNNet is different from RNN, as discussed in section 1.1, and is in fact similar to a conventional FNN with intra-connected hidden layers. The novelty of the model lies in the introduction of a new set of weights, responsible for the rapid convergence as discussed in section 3.3 of the paper, and the approach used to connect the hidden neurons.

Reviewer comment (Theory):
It appears that the mathematical model for this new model reduces to the conventional feedforward network without intra-hidden layer connections. Equations (1) and (2) seem to suggest that $H^{[l]}$ is a linear transformation of input $A{[l-1]}$. If you compose linear transformations, you get a linear transformation. Specifically, if you put equation (2) in equation (1), you will get a new linear transformation that is similar to using no intra-hidden layer connection.

Author response:
The proposed architecture is in fact a conventional FNN with intra-connected hidden layers. The novelty of the model lies in the introduction of a new set of weights, responsible for the rapid convergence as discussed in section 3.3 of the paper, and the approach used to connect the hidden neurons.

Reviewer comment (Experiments):
I think the experiments are insufficient. The number of training epochs before convergence does not give the full picture. The claim of rapid convergence fails to account for the overhead per epoch due to the extra intra-layer computations. Also, I think the authors should compare this work with the use of skip connections.

Author response:
We omitted experiments with skip connections, as skip connections can be added to both CHNNets and conventional FNNs. As a result, skip connections can benefit both CHNNets and FNNs equally.  
With an equal number of parameters in CHNNet and FNN, the number of computations per epoch is nearly equal as well. Additionally, in section 4.4, we conducted experiments with CHNNet and FNN having nearly equal numbers of parameters and showed that CHNNet outperformed FNN in such setups as well.

Reviewer comment (Novelty):
Please can you provide more information on how this model architecture differs from the conventional feedforward network with an intra-connection layer, especially with the simplification of plugging equation (2) into equation (1).

Author response:
The proposed architecture is in fact a conventional FNN with intra-connected hidden layers. The novelty of the model lies in the introduction of a new set of weights, responsible for the rapid convergence as discussed in section 3.3 of the paper, and the approach used to connect the hidden neurons.

Reviewer comment (Experiments):
The limited number of datasets tested on strikes me as the primary experimental weakness of this paper: given the similarities between all of the MNIST and MNIST-adjacent datasets, it's a little hard to tell to what extent these results suggest a more general conclusion, vs just a property of training FFNNs on MNIST and similarly-structured datasets.

Author response:
We had conducted additional experiments on the Abalone, Iris, Boston Housing, and Breast Cancer datasets and got similar results to those shown in the paper. However, due to page restrictions, we could not add the results to the paper.

Reviewer comment (Theory):
I think the convergence speed proof could have done with a little more explanation to make the intuitions of the steps in the chain more clear.

Author response:
We will add more explanation to the convergence proof in the camera-ready version of the paper if it gets accepted.

Reviewer comment (Experiments):
This is more a question than a suggestion, but given the central role MLPs are currently playing in modern transformer architectures, it would be interesting to see whether this approach leads to better performance/faster convergence in that setting (i.e. as a way to modify the parameterization of internal MLPs).

Author response:
We appreciate your suggestion and will work with modern transformer architectures in the future.