Active inference is a normative framework for generating behaviour based upon the free energy principle, a theory of global brain function . This framework has been successfully used to solve reinforcement learning and stochastic control tasks , yet, the formal relation between active inference and reward maximisation has not been fully explicated. In this paper, we consider the relation between active inference and dynamic programming under the Bellman equation, which underlies many approaches to reinforcement learning and control. We show that, on finite-horizon partially observed Markov decision processes, dynamic programming is a limiting case of active inference. In a fully observed environment, active inferenceagents seek to sample a target distribution encoding  preferences. When these target states correspond to rewarding states, this maximises expected reward  as in reinforcement learning. When states are partially observed or ambiguous, active inference agents will choose the action that minimises both risk and ambiguity. This allows active inference agents to supplement goal-seeking with exploratory behaviour. This speaks to the unifying potential of active inference , as the objective optimised during action selection subsumes many important quantities used in decision-making in the physical, engineering, and life sciences .
Active inference is a normative framework for generating behaviour based upon the free energy principle, a theory of URLanisation . This framework has been successfully used to solve reinforcement learning and stochastic control problems , yet, the formal relation between active inference and reward maximisation has not been fully explicated. In this paper, we consider the relation between active inference and dynamic programming under the Bellman equation, which underlies many approaches to reinforcement learning and control. We show that, on partially observable Markov decision processes, dynamic programming is a limiting case of active inference. In active inference, agents select actions to minimise expected free energy. In the absence of ambiguity about states, this reduces to matching expected states with a target distribution encoding the agent's preferences. When target states correspond to rewarding states, this maximises expected reward , as in reinforcement learning. When states are ambiguous, active inference agents will choose actions that simultaneously minimise ambiguity. This allows active inference agents to supplement their reward maximising (or exploitative) behaviour with novelty-seeking (or exploratory) behaviour. This clarifies the connection between active inference and reinforcement learning, and how both frameworks may benefit from each other .
R
global brain function
URLanisation
R
tasks
problems
R
finite-horizon partially observed
partially observable
R
a fully observed environment, active inferenceagents seek to sample
active inference, agents select actions to minimise expected free energy. In the absence of ambiguity about states, this reduces to matching expected states with
A
the agent's
D
these
A
,
D
partially observed or
R
the action that minimises both risk and
actions that simultaneously minimise
R
goal-seeking with exploratory
their reward maximising (or exploitative) behaviour with novelty-seeking (or exploratory)
R
speaks to the unifying potential of active inference , as the objective optimised during action selection subsumes many important quantities used in decision-making in the physical, engineering, and life sciences
clarifies the connection between active inference and reinforcement learning, and how both frameworks may benefit from each other