Labeling Q-learning in POMDP environments
This paper presents a new Reinforcement Learning (RL) method, called "Labeling Q-learning (LQ-learning)," to solve the partially obervable Markov Decision Process (POMDP) problems. Recently, hierarchical RL methods are widely studied. However, they have the drawback that the learning time and memory are exhausted only for keeping the hierarchical structure, though they wouldn't be necessary. On the other hand, our LQ-Iearning has no hierarchical structure, but adopts a new type of internal memory mechanism. Namely, in the LQ-Iearning, the agent percepts the current state by pair of observation and its label, and then, the agent can distinguish states, which look as same, but obviously different, more exactly. So to speak, at each step t, we define a new type of perception of its environment (o) over tilde (t) = (o(t), theta(t)), where o(t) is conventional observation, and theta(t) is the label attached to the observation ot. Then the classical RL-algorithm is used as if the pair (o(t), theta(t)) serves as a Markov state. This labeling is carried out by a Boolean variable, called "CHANGE," and a hash-like or mod function, called Labeling Function (LF). In order to demonstrate the efficiency of LQ-Iearning, we will apply it to "maze problems" in Grid-Worlds, used in many literatures as POMDP simulated environments. By using the LQ-Iearning, we can solve the maze problems without initial knowledge of environments.
labeling q-learning pomdp reinforcement learning grid-world non-hierarchinal structure single agent