Paper:
Regularization Learning Networks: Deep Learning for Tabular Datasets
Despite their impressive performance, Deep Neural Networks (DNNs) typically underperform Gradient Boosting Trees (GBTs) on many tabular-dataset learning tasks. We propose that applying a different regularization coefficient to each weight might boost the performance of DNNs by allowing them to make more use of the more relevant inputs. However, this will lead to an intractable number of hyperparameters. Here, we introduce Regularization Learning Networks (RLNs), which overcome this challenge by introducing an efficient hyperparameter tuning scheme which minimizes a new Counterfactual Loss. Our results show that RLNs significantly improve DNNs on tabular datasets, and achieve comparable results to GBTs, with the best performance achieved with an ensemble that combines GBTs and RLNs. RLNs produce extremely sparse networks, eliminating up to 99.8% of the network edges and 82% of the input features, thus providing more interpretable models and reveal the importance that the network assigns to different inputs. RLNs could efficiently learn a single network in datasets that comprise both tabular and unstructured data, such as in the setting of medical imaging accompanied by electronic health records. An open source implementation of RLN can be found at https://github.com/irashavitt/regularization_ learning_networks.

Review:
Summary This work develops Regularization Learning Networks (RLN), an approach to learn varying levels of regularization for different network weights. Regularization levels for (D)NN are usually  a global hyper-parameter, optimized via CV for the whole network. The author point out that figuring different levels or regularization for different features is appealing especially for data where features are of different “type” with some possibly highly important while others not and the transition between those is more striking than common in say structured images. Good examples are smaller tabular data of expression, EHR where DL commonly underperforms.  The CV approach without a direct target function to optimize does not scale for optimizing each weight separately. Instead, the authors develop an elegant solution in the paper, the counterfactual loss (L_cf). The SGD now has two contributions: the gradient for the loss and the gradient for the regularization, as a result of evaluating the delta from moving the weight i to a new point W_{t+1,i} on a new batch Z_{t+1} as a derivative of the hyperparameter for penalty \Lamba_i.   The authors evaluate the performance of their method on a dataset of microbiome from patients with several different target values to predict (Age, BMI, etc.). Their RLN greatly improves compared to DNN with standard “global” regularization (optimized by CV), but GBT is still better. They also try ensembles of learners. As expected these give improved results, with the best predictor composed of a combination of GBT and RLN. Finally, they evaluate the level or regularization on the features during learning (Fig7) and the features importance compared to a DNN (Fig8), showing learned models are extremely sparse.  Pros: The paper addresses an important and challenging problem. The solution seems novel, simple/elegant, yet efficient. Paper is generally clearly written. The solution appears to significantly improve performance compared to standard DNN regularization. It was nice to see the authors also make an effort to understand “what is going on” in terms of regularization (Fig7) and interpretation applying matrix product  (p. 8 top paragraph, Fig 8).  Cons:  Overall, we did not find any major flaws though several improvements can definitely be made.   First, we note that at the end of the day on all tasks tested GBT was still the better model. Only an ensemble of GBT+ RLN gave a very slight improvement. Thus, on all tasks tested, there is no real benefit in the proposed method.  The above is related to the fact that while the authors rightfully motivate their work with a general hypothesis about (un)structured/tabular data etc. they only use a single dataset with no synthetic testing to explore behavior. Why? We would like to see the final including a more elaborate evaluation of different datasets which the authors are already aware of (e.g. line 21). This could include different synthetic settings. For example: What happens if the features are informative but (highly) correlated? How does the amount of data affect the new framework?  Even on the dataset used the effect of the RLN specific hyperparameters are not explored (they are listed in the supp). Similarly, the analysis of feature importance and how to decide on those important ones was lacking. Why are 5 features selected? What is the criteria for significance? Do they make sense? There are many other ways to extract meaning from DNN (e.g. IG) which are at least worth discussing.   It is also important that the authors make code/analysis available - it could be highly useful *if* indeed it proves to give a boost (pun intended) in performance in other domains.  Other comments:  In the introduction and related work the authors keep interchanging the (undefined) terms (un)structured data, (non) distributed representation, and tabular data. Though the general intention is clear this adds unnecessary confusion.  The authors can do a better job introducing, deriving the main idea (p.3) and giving the intuition behind it. As is, it took a few back and forth to understand things.  Many of the figures are informative yet not so user-friendly and their caption limited (Fig1, 5, 7, 8). Consider plotting differently or adding info.