Paper: Smooth Min-Max Monotonic Networks (ICLR-2024)

Reviewer comment (Novelty):
I am not entirely sure about the novelty of this idea of smoothening non-smooth neurons to address the problem of vanishing gradients or silent neurons in the context of monotonic networks. The main idea of this work of using LogSumExp to act as a smooth approximation while preserving monotonicity does not seem too non-trivial due to its popularity in statistic modelling. However, I am not familiar with the line of work with monotone networks thus I will defer this discussion to other reviewers.

Author response:
**Weakness 1:** To the best of our knowledge, our idea is new in the context of monotonic neural networks. This is indeed remarkable, given the many papers building on Sill’s work coming up with complicated alternatives.

Reviewer comment (Evaluation):
While the empirical comparisons are sufficient, they do not provide evidence (especially after accounting the error bars) to suggest that SSM has significant advantage over existing approaches. It is then unclear why practitioners should prefer SSMs over LMNs or XGBoost.

Author response:
**Weakness 2:** The results in Table 1 show statistically significant better performance of SMM compared to all other methods. The results in Table 1 may be regarded as the most relevant ones, because the experimental setup is very controlled and focussed on the monotonicity aspect. 

We are very sorry that we failed to make it clear in our text why one should prefer SSMs over LMNs or XGBoost in practice.

XGBoost vs SMM: XGBoost models are piecewise constant. If you want a smooth model (e.g., in a natural science application), then you should prefer SMM or LMN over XGBoost. If you have a larger (deep) learning system and you need a monotonic module and would like to train end-to-end, then XGBoost is not a good option and you should pick a neural network such as LMN or SMM. If you do not have these constraints and tabular data, then XGBoost is a choice that is difficult to beat by any neural network approach - except perhaps SMMs as shown in Table 1.

SMM vs LMN: “LMNs share many of the desirable properties of SMMs. Imposing an upper bound on the Lipschitz constant, which is required for LMNs and can be optionally added to SMMs, can act as a regularizer and supports theoretical analysis of the neural network. However, a wrongly chosen bound can limit the approximation capabilities. Our experiments show that there are no reasons to prefer LMNs over SMMs because of generalization performance and efficiency. Because of the high accuracies obtained without architecture tuning, the more general asymptotic approximation results, and the simplicity of the min-max approach, we prefer SMMs over LMNs.” 

Perhaps this was too modest, so we would like to add: Over all experimental results in our study, SMM performs (most often significantly) better than LMN. While SMMs are very robust w.r.t. to hyperparameter choices (as shown in the study), the Lipschitz bound to be chosen for LMNs might be a problem in practice. We see no reason to prefer LMN over SMM.

Reviewer comment (Experiments):
How should the scaling factor $\beta$ be chosen in practice? My understanding is that tuning it to ensure that the output network is monotone is not trivial and requires retraining the entire network.

Author response:
**Question 1:** "My understanding is that tuning it to ensure that the output network is monotone is not trivial and requires retraining the entire network.": *The network is monotone for all choices of beta.*  

Tuning beta is not an issue in practice. Note that beta is adapted during training. All experiments (except those in reported in Table C.7) were conducted with the same "default" initialization ln(beta)=-1. This shows the robustness of our method. We suggest to simply use the default initialization (much too large or too small initial beta values may impact the performance, see also reply to reviewer RbCP).