Abstract-As living organisms, one of our primary character istics is the ability to rapidly process and react to unknown and unexpected events. To this end, we are able to recognize an event or a sequence of events and learn to respond properly. Despite advances in machine learning, current cognitive robotic systems are not able to rapidly and efficiently respond in the real world: the challenge is to learn to recognize both what is important, and also when to act. Reinforcement Learning (RL) is typically used to solve complex tasks: to learn the how. To respond quickly -to learn when -the environment has to be sampled often enough. For "enough", a programmer has to decide on the step-size as a time representation, choosing between a fine-grained representation of time (many state-transitions; difficult to learn with RL) or to a coarse temporal resolution (easier to learn with RL but lacking precise timing). Here, we derive a continuous-time version of on-policy SARSA-Iearning in a working-memory neural network model, AuGMEnT. Using a neural working memory network resolves the what problem, our when solution is built on the notion that in the real world, instantaneous actions of duration dt are actually impossible. We demonstrate how we can decouple action duration from the internal time-steps in the neural RL model using an action selection system. The resultant CT-AuGMEn T successfully learns to react to the events of a continuous-time task, without any pre-imposed specifications about the duration of the events or the delays between them. A self-driving car travels along the way when suddenly a man crosses the street. The car has to stop immediately to avoid the impact. This is an example of the complexity of the environment that we live in, where many unknown and unexpected events have to be recognized and processed rapidly and continuously. . Another common approach for continuous-time RL is based on actor-critic models, e.g. [7] . We find however that for the latter, there is no current work that includes methods for learning compact working memory representations, and our own efforts had great difficulty with reaching convergence when using an actor-critic method.
REF also presented a continuous-time on-policy RL algorithm to learn task-specific working memory in order to decouple action duration from the internal time-steps of the RL model using a Winner-Take-All action selection mechanism.
Continuous-time on-policy neural Reinforcement Learning of working memory tasks
2015 International Joint Conference on Neural Networks (IJCNN)
2015 International Joint Conference on Neural Networks (IJCNN)
Computer Science