Paper:
Adaptive optimal training of animal behavior
Neuroscience experiments often require training animals to perform tasks designed to elicit various sensory, cognitive, and motor behaviors. Training typically involves a series of gradual adjustments of stimulus conditions and rewards in order to bring about learning. However, training protocols are usually hand-designed, relying on a combination of intuition, guesswork, and trial-and-error, and often require weeks or months to achieve a desired level of task performance. Here we combine ideas from reinforcement learning and adaptive optimal experimental design to formulate methods for adaptive optimal training of animal behavior. Our work addresses two intriguing problems at once: first, it seeks to infer the learning rules underlying an animal's behavioral changes during training; second, it seeks to exploit these rules to select stimuli that will maximize the rate of learning toward a desired objective. We develop and test these methods using data collected from rats during training on a two-interval sensory discrimination task. We show that we can accurately infer the parameters of a policy-gradient-based learning algorithm that describes how the animal's internal model of the task evolves over the course of training. We then formulate a theory for optimal training, which involves selecting sequences of stimuli that will drive the animal's internal policy toward a desired location in the parameter space. Simulations show that our method can in theory provide a substantial speedup over standard training methods. We feel these results will hold considerable theoretical and practical implications both for researchers in reinforcement learning and for experimentalists seeking to train animals.

Review:
The authors have tried to accomplish two goals in this paper. First, they develop a technique to dynamically track the (time-varying) psychometric function of an animal while it learns a new task. Then they propose an algorithm to choose an optimal stimulus sequence that would most quickly drive the animal towards a desired (target) psychometric function. For simplicity, the focus of the paper is restricted to binary choice tasks in which the psychometric model is assumed to be described by a logistic function. The model is somewhat enriched by incorporating additional parameters (“stickiness coefficients”) to allow for possible history-dependence in choice behavior typically observed in the laboratory. The bulk of the results are presented in sections 3-5.  In section 3, the authors develop a general framework to estimate time-varying parameters of the logistic model when the changes in parameter values are purely stochastic (follow a gaussian random walk). This is achieved by first using an empirical Bayes approach to determine the hyperparameter that characterises the variability of the random walk, followed by standard MAP estimation of the model parameters for the given dataset. The approach is validated through simulations and applied to data collected from training rodents on an auditory task.  This exercise is repeated in section 4, but now after adding a non-random component to the temporal dynamics of the model parameters to mimic the effect of learning. This deterministic component is assumed to push the model parameters along the gradient of the expected reward with a learning rate which enters the model as a second hyperparameter. They call this the “RewardMax” model of learning. This section ends with a comparison of the two hyperpameters fit to behavioural data from rodents, and the authors make a point about how these results imply that the rodents are indeed behaving in a regime where the effect of learning trumps that of random noise. In section 5, the authors present an algorithm to select a stimulus sequence that would drive the psychometric function of the trainee towards a desired shape as quickly as possible. This is achieved using an “AlignMax” algorithm in which an upcoming stimulus is chosen as the one that maximally aligns the expected change in model parameters along the direction of the target model parameters. A comparison of the evolution of model-parameters using optimal and random stimulus sequences revealed that the former is indeed faster albeit only marginally. The more remarkable observation (although strangely this is not emphasised) was the fact that employing optimal stimulus sequence completely eliminated bias and history-dependence whereas random stimuli failed to achieve this even after training for several thousand trials. The authors then highlight key features of the resulting optimal stimulus sequence, and explain how their algorithm exploits a large stimulus space to optimise training. 
This is an innovative study that establishes a nice model-based method to optimize animal learning of a task. It addresses a simple task and simple model but this is challenging enough and a good foundation from which to build. The authors don’t yet applying this method to real experiments, but I’m sure that’s coming soon, and it’s not needed here to demonstrate their technical advance. - There should be some discussion of the impact of the restriction to a memory-less model of reward (tau = 0). - Figure 4 – define success rate. Better explain why success is so low (because you’re giving essentially adversarial examples?). Why is expected reward worse than for the random stimulus presentation schedule? - Figure 4 – The performance measures are interesting, but the how well does the model learn the desired weights (which is the expressed goal of the model)? - L is likelihood of data or likelihood of weights? - Figure 2 - dw of different realisations are not identical but there is a strong correlation. - Define “lazy fluke”. - It would be appropriate to address/cite Christian Machens’ work on optimal stimulus ensembles (http://www.ncbi.nlm.nih.gov/pubmed/16055067) and, especially, Liam Paninski’s work on optimizing stimuli to maximize information about an underlying model (http://www.stat.columbia.edu/~liam/research/pubs/pbr.pdf).