This article introduces a class of incremental learning procedures specialized for prediction that is, for using past experience with an incompletely known system to predict its future behavior. Whereas conventional prediction-learning methods assign credit by means of the difference between predicted and actual outcomes, tile new methods assign credit by means of the difference between temporally successive predictions. Although such temporal-difference method~ have been used in Samuel's checker player, Holland's bucket brigade, and the author's Adaptive Heuristic Critic, they have remained poorly understood. Here we prove their convergence and optimality for special cases and relate them to supervised-learning methods. For most real-world prediction problems, telnporal-differenee methods require less memory and less peak computation than conventional methods and they produce more accurate predictions. We argue that most problems to which supervised learning is currently applied are really prediction problems of the sort to which temporaldifference methods can be applied to advantage. Given the current interest in learning procedures for multi-layer connectionist networks (e.g., note that here we are c(meerned with a different, set of issues. The work with multi-layer networks focuses on learning input-output mat)pings of more comt)lex flm('tional forms. Most of that work remains within the supervised-learning paradigm, whereas here we are interested in extending and going beyond it. We consider mostly mappings of very simple flmctional forms, because the differences between supervised learning methods and TD methods are clearest in these cases. Nevertheless. the TD methods presented here can [)e directly extende(t to multi-layer networks (see Seetiou 6.2). The next section introduces a specific ('lass of temporal-difference t)roeedures by contrasting them with convent ional, supervised-h,arning at)proaches, focusing on ('omputational issues. Section 3 deveh)ps an extended ~'xample that illustrates the i)otential I)erfl)rmance a.dvantage~ of TD methods. Section 4 contains the convergeilee and optimality theorems and discusses TD methods as gradient descent. Section 5 discusses how to extend TD t)rocedures, and Se('th)n (i relates them to other research.
Sutton et al. REF assigns credit by means of the difference between temporally successive predictions.
Learning to predict by the methods of temporal differences
Machine Learning
Machine Learning
Computer Science