Paper:
Learning From Small Samples: An Analysis of Simple Decision Heuristics
Simple decision heuristics are models of human and animal behavior that use few pieces of information-perhaps only a single piece of information-and integrate the pieces in simple ways, for example, by considering them sequentially, one at a time, or by giving them equal weight. We focus on three families of heuristics: single-cue decision making, lexicographic decision making, and tallying. It is unknown how quickly these heuristics can be learned from experience. We show, analytically and empirically, that substantial progress in learning can be made with just a few training samples. When training samples are very few, tallying performs substantially better than the alternative methods tested. Our empirical analysis is the most extensive to date, employing 63 natural data sets on diverse subjects.

Review:
The authors evaluate a number of decision rules from the ABC research group. They first analyze, through simulations, that the "cue direction" and "learning to order cue" heuristics can provide accurate performance with even just a few samples. Next, they evaluate a number of heuristics on 63 data sets and demonstrate that some of them achieve performance close to an additive logistic regression and CART, with default parameter settings.This is a well-written paper containing two substantial results that I believe will be of interest to a broad community at NIPS. I have a few main criticisms regarding the submission (in order of appearance). The first result they have (few samples suffice for some heuristics) is very similar in nature to a previous result by Vul, Goodman, Griffiths and Tenenbaum (2009; 2014). Although the results are not identical, I did not find the simulations terribly surprising or impressive given this previous work. Also, I am confused why the simulations discuss cue direction and informative sampling when neither are discussed nor used later in the paper. The other main criticism is concerned with the models used as comparisons to evaluate the performance of the heuristics on the 63 data sets. A simple logistic regression where each cue is an independent input is not that powerful of a statistical model. What if pairs and triplets are included as well? What about support vector machines/kernel methods on these data sets? I know this increases the computational complexity of the comparison models, but if the argument is that heuristics are close to the best models, than the best models should really be powerful. I was more impressed with CART, but the fact that the default settings were used with no exploration made me concerned that CART is underperforming because there is no tuning whatsoever to each data set. If the best parameters are chosen for CART on each data set, how well would it do? What are the best parameters for CART that do the best across all data sets?Minor concerns: Line 131: it might be worth mentioning that e1 is simply the CDF of a binomial distribution. Figure 1: The text is too small in many places and I found the subplots difficult to interpret. Also, the scale of the colormap for the bottom left subplot is different than the other subplots. Line 201-202: "..., but the in the majority" -> "..., but in the majority" Line 219: Wouldn't it be more appropriate to use ordinal logistic regression, (polyr in the MASS library of R), rather than turn an ordinal factor into a nominal factor? Line 258-259: "with upto 3000 repetitions." -> "with up to 3000 repetitions."Vul, E., Goodman, N., Griffiths, T., & Tenenbaum, J. (2014). One and Done? Optimal Decisions from Very Few Samples. Cognitive Science, 38, 599-637.Vul, E., Goodman, N., Griffiths, T., & Tenenbaum, J. (2009). One and done? Optimal decisions from few samples. In N. Taatgen, H. van Rijn, L. Schomaker, & J. Nerbonne (Eds.), Proceedings of the 31st annual conference of the cognitive science society. Amsterdam, the Netherlands.Edits based on author feedback: I'm glad to hear the initial review was helpful. The paper is a really great contribution and your feedback increases my confidence in that assessment. I apologize regarding the confusion with "pairs and triplets". What I mean is including interaction terms between your predictors in the logistic regression. Perhaps you did that and I misunderstood what you did in the manuscript? If so, it's probably worth revising the section on logistic regression a bit to make this clear.