Paper:
LEARNING CONCISE REPRESENTATIONS FOR REGRES- SION BY EVOLVING NETWORKS OF TREES
We propose and study a method for learning interpretable representations for the task of regression. Features are represented as networks of multi-type expression trees comprised of activation functions common in neural networks in addition to other elementary functions. Differentiable features are trained via gradient descent, and the performance of features in a linear model is used to weight the rate of change among subcomponents of each representation. The search process maintains an archive of representations with accuracy-complexity trade-offs to assist in generalization and interpretation. We compare several stochastic optimization approaches within this framework. We benchmark these variants on 100 open-source regression problems in comparison to state-of-the-art machine learning approaches. Our main finding is that this approach produces the highest average test scores across problems while producing representations that are orders of magnitude smaller than the next best performing method (gradient boosting). We also report a negative result in which attempts to directly optimize the disentanglement of the representation result in more highly correlated features.

Review:
The paper proposes a method for learning regression models through evolutionary
algorithms that promise to be more interpretable than other models while
achieving similar or higher performance. The authors evaluate their approach on
99 datasets from OpenML, demonstrating very promising performance.
The authors take a very interesting approach to modeling regression problems by
constructing complex algebraic expressions from simple building blocks with
genetic programming. In particular, they aim to keep the constructed expression
as small as possible to be able to interpret it easier. The evaluation is
thorough and convincing, demonstrating very good results.
The presented results show that the new method beats the performance of existing
methods; however, as only very limited hyperparameter tuning for the other
methods was performed, it is unclear to what extent this will hold true in
general. As the main focus of the paper is on the increased interpretability of
the learned models, this is only a minor flaw though.
The interpretability of the final models is measured in terms of their size.
While this is a reasonable proxy that is easy to measure, the question remains
to what extent the models are really interpretable by humans. This is definitely
something that should be explored in future work, as a small-size model does not
necessarily imply that humans can understand it easily, especially as the
generated algebraic expressions can be complex even for small trees.
The description of the proposed method could be improved; in particular it was
unclear to this reviewer why the features needed to be differentiable and what
the benefit of this was (i.e. why was this the most appropriate way of adjusting
weights).
In summary, the paper should be accepted.