In this paper, we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains. We begin by introducing the theory of Markov decision processes (mdps) and partially observable mdps (pomdps). We then outline a novel algorithm for solving pomdps o line and show how, in some cases, a nite-memory controller can be extracted from the solution to a pomdp. We conclude with a discussion of the complexity of nding exact solutions to pomdps and of some possibilities for nding approximate solutions. Consider the problem of a robot navigating in a large o ce building. The robot can move from hallway intersection to intersection and can make local observations of its world. Its actions are not completely reliable, however. Sometimes, when it intends to move, it stays where it is or goes too far; sometimes, when it intends to turn, it overshoots. It has similar problems with observation. Sometimes a corridor looks like a corner; sometimes a T-junction looks like an L-junction. How can such an error-plagued robot navigate, even given a map of the corridors? In general, the robot will have to remember something about its history of actions and observations and use this information, together with its knowledge of the underlying dynamics of the world (the map and other information), to maintain an estimate of its location. Many engineering applications follow this approach, using methods like the Kalman lter 10] to maintain a running estimate of the robot's spatial uncertainty, expressed as an ellipsoid or normal distribution in Cartesian space. This approach will not do for our robot, though. Its uncertainty may be discrete: it might be almost certain that it is in the north-east corner of either the fourth or the seventh oors, though it admits a chance that it is on the fth oor, as well. Then, given an uncertain estimate of its location, the robot has to decide what actions to take. In some cases, it might be su cient to ignore its uncertainty and take actions that would be appropriate for the most likely location. In other cases, it might be better for the robot to take actions for the purpose of gathering information, such as searching for a landmark or reading signs on the wall. In general, it will take actions that ful ll both purposes simultaneously.
Our work also relates closely to a body of work on Partially Observable Markov Decision Process (POMDP) REF .
Planning and Acting in Partially Observable Stochastic Domains
Artif. Intell.
Mathematics Computer Science