Paper:
Subset Selection by Pareto Optimization
Selecting the optimal subset from a large set of variables is a fundamental problem in various learning tasks such as feature selection, sparse regression, dictionary learning, etc. In this paper, we propose the POSS approach which employs evolutionary Pareto optimization to find a small-sized subset with good performance. We prove that for sparse regression, POSS is able to achieve the best-so-far theoretically guaranteed approximation performance efficiently. Particularly, for the Exponential Decay subclass, POSS is proven to achieve an optimal solution. Empirical study verifies the theoretical results, and exhibits the superior performance of POSS to greedy and convex relaxation methods.

Review:
The paper is concerned with a new algorithm (named POSS) for small-sized subset selection based on Pareto optimization. The proposed algorithm is extensively analyzed, compared and tested. A detailed comparison is presented between POSS and the FR algorithm (Forward Regression) for which a theoretical approximation bound is known. The authors also give such a theoretical approximation bound for their algorithm and conclude that POSS does at least as well as the FR algorithm on any sparse regression problem. A second result is given on the exponential decay subclass (a subclass of sparse regression: the observation variables can be ordered in a line such that their covariances are decreasing exponentially with the distance): On this subclass POSS has strictly better guarantees than FR. The authors also show that POSS finds (again on the exponential decay subclass) an optimal solution. This guarantee cannot be given for the FR algorithm as demonstrated be the authors. This concludes the theoretical part which is followed by an empirical study: 4 different algorithms (POSS, FR, OMP, FoBa) and the (exact) optimal solution (OPT), where it could be calculated in acceptable time, are compared on 12 different data sets. POSS outperforms all 4 algorithms when calculating the squared multiple correlation (R2) which is equivalent to the mean squared error. Although the authors claim a significantly better performance in all 4 cases, the reviewer is still not fully convinced about the attribute 'significantly'. The empirical study is extensive: calculating R2 on test and training data, performance vs running-time analysis and discussion of a possible pitfall (sparsity alone may not be a good sparsity measure and thus better optimization does not always lead to better generalization performance) and a solution to that pitfall (introduce l2 norm regularization into the objective function).In general, the paper is written in a clear and transparent way, and it is easy to follow the main arguments. Minor Comment: Line 126 - shouldn't it spell 'subset' instead of 'subsets'?