Paper: Scalable Monotonic Neural Networks (ICLR-2024)

Reviewer comment (Evaluation):
The authors provide many experiments but it seems that the proposed SMNN has a similar performance to LMN. There are 5 datasets. SMNN wins on 2 of them, loses on 2 of them, and is comparable on the last one.

Author response:
Certainly, we acknowledge that our method demonstrated performance on par with existing methods. While achieving lower performance could be considered a weakness, we would like to emphasize that comparable performance should not be perceived as a weakness, in our perspective. While striving for better performance is always desirable, please understand that attaining the best performance in the field of monotonic neural networks is not the sole objective of this research. Our method, as highlighted in Table 1, presents advantages over existing techniques, maintaining competitive performance levels. Given that our approach supports gradient descent learning without the need for additional modifications (such as weight normalization or weight absolutization) or the imposition of mathematical constraints such as the Lipschitz constraint, we see our method as a valuable addition to the field of monotonic neural networks because it will be suitable for expanding research into different complex network architectures for future works, such as neural additive models or convolution structure.

Reviewer comment (Writing):
On page 2, the authors claim that the assurance of monotonicity is not always guaranteed as a regularization method. This is true but I think this should not be treated as the drawbacks of regularization methods. The monotonicity usually comes from experience and may not be the truth. When we prefer to believe the experience, we can use methods in the first group to guarantee monotonicity. But when the experience contradicts collected data, then regularization provides a chance to balance between past experience and new data. Thus, I think it is more suitable to say these two groups of methods have their applications in the real world rather than treating not guaranteed monotonicity as a drawback.

Author response:
Q1: We deeply appreciate your valuable feedback. As we totally agree with your comment, we have made the following modification.
Before: However, this method has some drawbacks. It requires additional processes to enforce and verify monotonicity, and the assurance of monotonicity is not always guaranteed as a regularization method.
After: However, it requires additional processes to enforce and verify monotonicity. As a regularization method, this method does not always ensure monotonicity. Therefore, in scenarios where the guarantee of monotonicity is essential, opting for the approaches with customized architecture may be preferable.

Reviewer comment (Theory):
Authors provide two definitions for partial monotonicity, one using function values and one using partial derivatives. However, it seems that the definition using function values is not correct. I think the correct one should be "The function $f(\boldsymbol{x})$ is partially monotonically non-decreasing on $\boldsymbol{x}_m$ iff $\forall i, x_i \leqslant x_i' , \forall j \neq i , x_j = x_j' \Rightarrow f(\boldsymbol{x}) \leqslant f(\boldsymbol{x}')$. Take a 2-dimensional case as an example. Let the 1st coordinate be the monotonic feature, and the 2nd is not. Let $f(x_1 , x_2) = x_1 + x_2^2$. Then $f$ is partially monotonic according to the definition using partial derivatives but is not partially monotonic according to the definition using function values.

Author response:
Q2: Again, we deeply appreciate your valuable feedback. As we totally agree with your comment, we have made the modification on the definition using function values. Please refer to the revised manuscript.

Reviewer comment (Writing):
In Eq. (5), $h_{con}$ should be $h_{conf}$.

Author response:
Q4: We have modified the equation by following your comment. Thank you.

Reviewer comment (Experiments):
In the current SMNN structure, both the exp unit and conf unit use the ReLU-n activation function for the purpose of universal approximation and aligning output magnitudes. But in the ReLU unit, ReLU is used rather than ReLU-n. I am wondering what will happen if all these three units use the ReLU-n activation function. I think a unified activation function makes the structure more succinct.

Author response:
Q5: Thank you for raising an interesting issue. Please refer to the newly created Appendix C1 on this matter. Additionally, we have inserted the following sentence into the middle of the 'Network structure’ paragraph.
`` To maintain a consistent activation function throughout the network, it is acceptable to use the ReLU-n function for the ReLU unit (See Appendix C1.).’’

Reviewer comment (Theory):
In theorem 1, $x$ belongs to $\boldsymbol{x}_m$. To my understanding, $\boldsymbol{x}_m$ is a vector rather than a set, which indicates that the belonging relation is not rigorously defined. I think $x$ is a coordinate of the vector $\boldsymbol{x}_m$. A suitable way is to define the subscript set of $\boldsymbol{x}_m$ as $M$, and then use $x_i$ with $i \in M$.

Author response:
Q6: We believe that your comments have significantly improved our paper. We have made modifications to Theorem 1 based on this feedback.

Reviewer comment (Experiments):
There seems to be a big focus on scalability, but it’s unclear how this approach is better than current state-of-the-art models in this respect. The experiments do not compare against other models, and for good reason, I imagine; there is little difference in the scalability of the proposed method and existing works in the literature. So, this begs the question of why scalability is a central thesis in the paper.

Author response:
We have performed the experiments for scalability comparison. Please refer to the next paragraph. We genuinely appreciate your insightful feedback. While it is a fact that our method exhibits scalability when compared to benchmark methods, the primary reason for our use of the term 'scalability’ is the absence of prior claims in this particular aspect within existing research. We believed that introducing this term could contribute to differentiating our work. Furthermore, as demonstrated, our method remains scalable with respect to both increasing numbers of monotonic features and expanding network sizes. We are encouraged by the agreement on this scalability aspect expressed by some reviewers. We wish your perspective on extending the term 'scalability’ in a broader context. Our method's generic nature, as highlighted by other reviewers, positions it well for future research endeavors exploring diverse and complex network architectures, such as neural additive models or convolution structures. Your thoughtful consideration of these aspects will be highly appreciated.
Indeed, creating a fair experimental environment for scalability comparison, considering different learning ways, structures, and feasibility based on the number of monotonic inputs, poses challenges in ensuring identical network sizes for training/test times. Nevertheless, we made efforts to assess the speed of our method in comparison to benchmarking methods, and the results are presented in the table below. While this is not a comprehensive comparison, it is evident that our method was faster than others based on the available comparisons (Notice that verifying time is not test time but additionally required time only for Certified MNN. We did not perform the comparison of test times as it is unnecessary because all methods except for COMET require a similar small inference time.). We hope this outcome addresses your concern. As we have already demonstrated the scalability of our method empirically in Section 4.1, we have chosen not to include the table below in the revised manuscript.
|Dataset|Method|# Parameter|# Monotonic Features|Training Time (s)|Verifying Time (s)|
|:---:|:---:|:---:|:---:|:---:|:---:|
|Auto-MPG|Certified MNN|$11006$|$3$|$99.88\pm13.37$|$28.83\pm12.89$|
||COMET|$421$|$3$|$--$|$--$|
||HLL|$14301$|$3$|$113.64\pm1.89$|$--$|
||LMN|$14025$|$3$|$22.20\pm1.31$|$--$|
||Constrained MNN|$14025$|$3$|$26.04\pm1.20$|$--$|
||SMNN (Ours)|$14193$|$3$|$\mathbf{17.00\pm0.91}$|$--$|
|Heart Disease|Certified MNN|$7318$|$2$|$27.16\pm6.71$|$11.31\pm6.71$|
||COMET|$785$|$2$|$--$|$--$|
||HLL|$7788$|$2$|$16.56\pm0.92$|$--$|
||LMN|$7617$|$2$|$10.04\pm1.17$|$--$|
||Constrained MNN|$7617$|$2$|$12.04\pm0.17$|$--$|
||SMNN (Ours)|$7905$|$2$|$\mathbf{9.40\pm0.91}$|$--$|

Reviewer comment (Novelty):
Overall, this paper feels like yet another approach to the inductive bias of monotonicity in neural networks. It has no big flaws, but the advantages do not appear to be substantial compared to current methods. The approach is simple enough and is worth considering, but the paper is slightly below the acceptance threshold.

Author response:
We appreciate your thoughtful evaluation of our research and the reasoning behind the assigned score. Your decision is respected, and we value your perspective. While we may not contest your viewpoint regarding the relative superiority of our SMNN method compared to the most recent state-of-the-art methods in terms of performance and training time (although we firmly believe in its speed and accuracy), we would like to underscore the distinctiveness brought about by the trainability of our method using conventional gradient descent. This feature positions our method as scalable, enabling extension to ongoing research, e.g. development of a neural additive model using the SMNN structure. This work is currently in progress, aiming to create a model that combines the advantages of a neural additive model, such as interpretability (currently challenging for LMN or Constrained MNN), with monotonicity. Regardless of the outcome of the acceptance/rejection at the ICLR conference, we remain committed to advancing our research series, including SMNN and its extensions to various models, through (hopefully) participation in this conference and others, as well as through journal publications. We deeply appreciate the time and effort you dedicated to reviewing our paper and providing insightful comments.

Reviewer comment (Writing):
Could you clarify what this sentence means: “Nevertheless, it should be noted that an optimally trained network is not always guaranteed with respect to the Lipschitz constant λ, particularly for larger constants”?

Author response:
Q1: Intended meaning of the sentence is as follows. Due to the constraint imposed by $\lambda$ in LMN, when the Lipschitz constant $\lambda$ is smaller than the intrinsic data monotonicity, it is impossible to sufficiently learn the given data using LMN because the inherent monotonicity is greater than $\lambda$ (LMN cannot approximate any function with a gradient exceeding the given Lipschitz constant). As you mentioned, it seems that the original sentence does not accurately convey the intended meaning. Therefore, in order to clearly convey the meaning, we have revised the sentence as follows.
``Nevertheless, it is important to note that if the Lipschitz constant \lambda remains smaller than the inherent monotonicity scale within data, LMN may struggle to effectively learn from the data. This limitation arises because LMN cannot approximate any function that holds a gradient exceeding the given Lipschitz constant.’’

Reviewer comment (Writing):
Missing slash for $\exp$ instead of $exp$ in the proof of theorem 1 on page 5.

Author response:
N1: Thank you for the comment. We have made modifications to the exponential function by following your feedback.

Reviewer comment (Experiments):
The scaling of compute time with respect to the number of monotonic features cannot really be extrapolated from varying the dimensions from 1 to 20. It's likely that at this scale, the execution time is dominated by constant factors, and it's unclear how it would actually scale to "high-dimensional monotonic features", compared to Certified MNN or COMET for instance. Due to the architecture, it is likely that the scaling time per training step would scale up as well as a regular MLP; however, optimization issues could happen and slow down convergence, for instance. It would be more effective to demonstrate the scalability of this method by applying it on a problem too big for Certified MNN or COMET.

Author response:
We acknowledge your concern. In response, we conducted an additional experiment involving the expansion of the model in equation (11). Specifically, we created a 220-dimensional dataset, excluding the output feature, consisting of 20 uniform distribution-based dummy inputs and 200 monotonic inputs achieved by expanding each group of features from 5 to 50 features. Notably, in this experiment, the dummy features exert much less influence during training compared to our original experiment. The training time and MSE results are shown in the table below. We observed the consistent trend of training times remaining constant while MSE decreases as the number of monotonic inputs increases. Please consider that the intention behind the scalability test using equation (11) was to demonstrate not only the constancy of training time but also the improvement in MSE performance with the gradual increase in monotonic inputs. We believe that this additional experiment adequately addresses your concern. However, since the original experiment effectively conveyed our intended message and presenting the equation for the 220-dimensional dataset might be challenging in the paper, we have chosen not to replace the original experiment with this new one.
|# Monotonic Features|# Parameter|MSE|Time (s)|
|:---:|:---:|:---:|:---:|
|$m=10$|$29414$|$0.204\pm0.12$|$78.76\pm2.22$|
|$m=20$|$29977$|$0.216\pm0.12$|$78.40\pm2.57$|
|$m=50$|$31657$|$0.175\pm0.09$|$78.24\pm1.51$|
|$m=100$|$34457$|$0.117\pm0.09$|$79.08\pm2.06$|
|$m=200$|$40057$|$0.013\pm0.01$|$78.84\pm2.01$|

Reviewer comment (Evaluation):
In Tables 2 and 3, statistical ties should be bolded and be assessed by a statistical test taking into account the variance of both distributions. Assuming the ± numbers indicate 95% confidence intervals, there should be many more ties, and the conclusions about SMNN outperforming all other methods on regression tasks do not really hold. For instance, on AutoMPG, 7.44±1.20 and 7.58±1.20 should clearly overlap.

Author response:
We appreciate the constructive feedback. To enhance the statistical clarity of our results, we have made revisions to Tables 2 and 3, accounting for statistical ties. The values following the `±’ symbol in these tables indicate the standard deviation based on 25 repetitions of our experiments. As the references do not explicitly provide details on experimental configurations such as the number of repetitions, we lack the necessary information to conduct statistical tests for benchmarking methods. Consequently, we have defined a statistical tie when, for both methods, one mean falls within the range (mean±std) of the other. Instances of statistical ties by this definition are marked with a $\dagger$ symbol.

Reviewer comment (Experiments):
The COMPAS dataset is used despite ethical concerns, since these features were used by the original COMPAS system to produce a score that was unfair and biased. This work uses the dataset despite not explicitly aiming at producing unbiased decisions, or examining critically the trained system.

Author response:
We acknowledge the ethical concerns surrounding the use of the COMPAS dataset (we already mentioned this in Appendix A2.), particularly given its history of being utilized in ways that produced biased and unfair outcomes. While our work does not explicitly focus on producing biased/unbiased decisions or critically examining the trained system from a fairness perspective, we used the COMPAS dataset as a means to test and demonstrate the technical capabilities of our model in a widely recognized context. Please also notice that this dataset has been used in several published papers for the same purpose with ours. Nevertheless, if it is still unable to avoid ethical concerns, we will definitely stop using the data and remove its results.

Reviewer comment (Presentation):
The circled "+" signs in Figure 1 are confusing if the operation is concatenation, not addition. Maybe it would be clearer to have both arrows point directly at the next "Exp unit"? (Idem for the fc layer)

Author response:
Thank you for your comment. The `+’ in a circle symbol generally represents concatenation in the related literature. To enhance understanding, Fig. 1 has been modified. The meaning of the symbol has been noted in the bottom right corner of the figure. We would be grateful if you could confirm whether this modification has clearly resolved the issue.

Reviewer comment (Presentation):
Why are Table 2 and Table 3 separate? Both have a mix of regression and classification datasets, but they have a different set of methods, and those in Table 3 do not report parameter counts.

Author response:
As mentioned in the main text, Tables 2 and 3 include experimental results from the existing benchmark studies. Therefore, the tables have been separated for entry (not dividing the tables based on classification/regression). In Table 3, the original benchmark literature 'COMET’ [Sivaraman et al., 2020] was published without specifying the number of parameters. And the studies 'LMN’ [Nolte et al., 2022] and 'Constrained MNN’ [Runje & Shankaranarayana., 2023] also missed the number of parameters. Therefore, unfortunately, we could only include the performances of benchmark methods for comparison. Please refer to the benchmark literatures.

Reviewer comment (Writing):
Not clear at first what "subjected to denoising" (S. 4.1) means.

Author response:
We have modified the sentence as follows. ``Fig.3(c) presents an interesting observation where the noise of the test dataset was eliminated.’’

Reviewer comment (Presentation):
Clarify what the "±" numbers in the table mean: standard deviation or confidence interval?

Author response:
We appreciate the constructive feedback. To enhance the statistical clarity of our results, we have made revisions to Tables 2 and 3, accounting for statistical ties. The values following the `±’ symbol in these tables indicate the standard deviation based on 25 repetitions of our experiments. As the references do not explicitly provide details on experimental configurations such as the number of repetitions, we lack the necessary information to conduct statistical tests for benchmarking methods. Consequently, we have defined a statistical tie when, for both methods, one mean falls within the range (mean±std) of the other. Instances of statistical ties by this definition are marked with a $\dagger$ symbol.

Reviewer comment (Experiments):
The small scale of experiments is a limitation. Given that the ambition of the algorithm is to be "scalable", I think it should try to demonstrate scaling where some other algorithms are limited.

Author response:
Sig.: For the proposed method only, please refer to our answer to W2(Quality). To address other reviewer’s concern, we have performed an additional experiment for training time comparison. Please refer to our answer to Q4 for the reviewer tddM. The limitations of the three techniques (Certified MNN, COMET, HLL) are apparent in terms of scalability, as observed in the comparative experiment. In the context of scalability, we find that the other three methods (LMN, Constrained MNN, SMNN) exhibit comparable characteristics. However, when considering scalability in the sense of extensibility, the proposed method stands out as sufficiently generic (unlike LMN and Constrained MNN), making it suitable for future research exploring various complex network architectures, such as neural additive models or convolution structures, as you pointed out.

Reviewer comment (Experiments):
To disentangle the effects of the connectivity pattern from those of monotonous parametrization, I'd be curious to see the performance of a network composed only of Exponential units (all fully connected), but where the non-monotonous inputs would be duplicated as [x_¬m, -x_¬m]. Is that something you tried?

Author response:
Q: This is indeed an interesting and good idea, we think so. By following your idea, to address your curiosity, we have performed the experiments based on the Auto-MPG and Heart Disease datasets. The results below illustrate the comparability between your idea and our method.
SMNN : Auto-MPG(MSE) 7.44 ± 1.20, Heart disease(Test Acc) 0.88 ± 0.04
Reviewer's suggestion : Auto-MPG(MSE) 7.44 ± 1.57, Heart disease(Test Acc) 0.86 ± 0.04
(Now, our defense ;)) However, there seems to be a trade-off in terms of scalability and model interpretability, as non-monotonic features would need to be manipulated and duplicated. While your idea is innovative, it is necessary to conduct a further analysis of the impact of feature duplication during the learning process.

Reviewer comment (Novelty):
Only performs on-par as the existing methods for enforcing monotonicity in neural networks

Author response:
Certainly, we acknowledge that our method demonstrated performance on par with existing methods. While achieving lower performance could be considered a weakness, we would like to emphasize that comparable performance should not be perceived as a weakness, in our perspective. While striving for better performance is always desirable, please understand that attaining the best performance in the field of monotonic neural networks is not the sole objective of this research. Our method, as highlighted in Table 1, presents advantages over existing techniques, maintaining competitive performance levels. Given that our approach supports gradient descent learning without the need for additional modifications (such as weight normalization or weight absolutization) or the imposition of mathematical constraints such as the Lipschitz constraint, we see our method as a valuable addition to the field of monotonic neural networks because it will be suitable for expanding research into different complex network architectures for future works, such as neural additive models or convolution structure.

Reviewer comment (Reproducibility):
How does one incorporate other activations/non-linear operators in this architecture?

Author response:
We appreciate the constructive feedback regarding the extension for the activation function. In order to address this concern, we added the following sentences at the end of the paragraph of `Activation functions’,
`` Alternative activation functions offering the similar benefit, including both convexity and concavity, can be substituted for the ReLU-$n$ function (See our ablation study in Appendix C1.).’’
and the ablation study in Appendix C1.

Reviewer comment (Experiments):
How do you select the parameter n in ReLU-n activation?

Author response:
Thank you for your valuable feedback. Indeed, the parameter $n$ in ReLU-$n$ is a hyperparameter to be predetermined as we mentioned in Appendix A1. We consistently set it to 1 throughout all our experiments, because the primary purpose with this activation function was to utilize the convex and concave properties, making detailed tuning of this value unnecessary. Nonetheless, to empirically investigate whether the performance of our method holds across different values of $n$, we conducted an ablation study and integrated its outcomes in Appendix C2 of the revised manuscript.

Reviewer comment (Evaluation):
Do you have any comparison on train/test time for the baselines with the SMNN architecture? Does it take longer to converge compared to existing methods?

Author response:
Indeed, creating a fair experimental environment for comparison, considering different learning ways, structures, and feasibility based on the number of monotonic inputs, poses challenges in ensuring identical network sizes for training/test times. Nevertheless, we made efforts to assess the speed of our method in comparison to benchmarking methods, and the results are presented in the table below. While this is not a comprehensive comparison, it is evident that our method was faster than others based on the available comparisons (Notice that verifying time is not test time but additionally required time only for Certified MNN. We did not perform the comparison of test times as it is unnecessary because all methods except for COMET require a similar small inference time.). We hope this outcome addresses your concern. As we have already demonstrated the scalability of our method empirically in Section 4.1, we have chosen not to include the table below in the revised manuscript.
|Dataset|Method|# Parameter|# Monotonic Features|Training Time (s)|Verifying Time (s)|
|:---:|:---:|:---:|:---:|:---:|:---:|
|Auto-MPG|Certified MNN|$11006$|$3$|$99.88\pm13.37$|$28.83\pm12.89$|
||COMET|$421$|$3$|$--$|$--$|
||HLL|$14301$|$3$|$113.64\pm1.89$|$--$|
||LMN|$14025$|$3$|$22.20\pm1.31$|$--$|
||Constrained MNN|$14025$|$3$|$26.04\pm1.20$|$--$|
||SMNN (Ours)|$14193$|$3$|$\mathbf{17.00\pm0.91}$|$--$|
|Heart Disease|Certified MNN|$7318$|$2$|$27.16\pm6.71$|$11.31\pm6.71$|
||COMET|$785$|$2$|$--$|$--$|
||HLL|$7788$|$2$|$16.56\pm0.92$|$--$|
||LMN|$7617$|$2$|$10.04\pm1.17$|$--$|
||Constrained MNN|$7617$|$2$|$12.04\pm0.17$|$--$|
||SMNN (Ours)|$7905$|$2$|$\mathbf{9.40\pm0.91}$|$--$|

Reviewer comment (Reproducibility):
Why are parameters missing for XGBoost and Isotonic methods in Table 2?

Author response:
As mentioned in the main text, the results in Table 2 came from the references (Liu et al., 2020 ; Nolte et al., 2022; Runje & Shankaranarayana, 2023). The paper where these experimental results for XGBoost and Isotonic were first presented is [Liu et al., 2020] and the number of parameters for XGBoost and Isotonic were missing in the original paper. This may be because they are not based on neural networks. We could not include the number of parameters for those methods.