Active inference is a first (Bayesian) principles account of how autonomous agents might operate in dynamic, non-stationary environments. The optimization of congruent formulations of the free energy functional (variational and expected) , in active inference , enables agents to make inferences about the environment and select optimal behaviors. The agent achieves this by evaluating (sensory) evidence in relation to its internal generative model that entails beliefs about future (hidden) states and sequence of actions that it can choose. In contrast to analogous frameworks - by operating in a pure belief-based setting (free energy functional of beliefs about states) - active inference agents can carry out epistemic exploration and naturally account for uncertainty about their environment  . Through this review, we disambiguate these properties , by providing a condensed overview of the theory underpinning active inference . A T-maze simulation is used to demonstrate how these behaviors emerge naturally, as the agent makes inferences about the observed outcomes and optimizes its generative model (via belief updating). Additionally, the discrete state-space and time formulation presented provides an accessible guide on how to derive the (variational and expected) free energy equations and belief updating rules. We conclude by noting that this formalism can be applied in other engineering applications ; e.g., robotic arm movement, playing Atari games, etc., if appropriate underlying probability distributions (i.e. generative model) can be formulated  .
Active inference is a first principle account of how autonomous agents operate in dynamic, non-stationary environments. This problem is also considered in reinforcement learning, but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in reinforcement learning; 2) to our knowledge, the first explicit discrete-state comparison between active inference and reinforcement learning on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of reinforcement learning. We show that by operating in a pure belief-based setting , active inference agents can carry out epistemic exploration - and account for uncertainty about their environment - in a Bayes optimal fashion. We make these properties explicit by showing that the active inference agent's ability to carry out online planning, in a pure-belief setting, enables it to act optimally, given the non-stationary dynamics of an environment when compared to both Q-learning and Bayesian model-based reinforcement learning agents . We conclude by noting that this formalism can be applied to more complex settings ; e.g., robotic arm movement, Atari games, etc., if appropriate generative models can be formulated . In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside reinforcement learning agents .
R
(Bayesian) principles
principle
D
might
R
The optimization of congruent formulations of the free energy functional (variational and expected) ,
This problem is also considered in reinforcement learning, but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors
R
, enables agents to make inferences about the environment and select optimal behaviors. The agent achieves this by evaluating (sensory) evidence in relation to its internal generative model that entails beliefs about future (hidden) states and sequence of actions that it can choose. In contrast to analogous frameworks -
that are generally engineered in reinforcement learning; 2) to our knowledge, the first explicit discrete-state comparison between active inference and reinforcement learning on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of reinforcement learning. We show that
R
(free energy functional of beliefs about states) -
,
R
and naturally
- and
A
- in a Bayes optimal fashion. We make these properties explicit by showing that the active inference agent's ability to carry out online planning, in a pure-belief setting, enables it to act optimally, given the non-stationary dynamics of an environment when compared to both Q-learning and Bayesian model-based reinforcement learning agents
D
Through this review, we disambiguate these properties , by providing a condensed overview of the theory underpinning active inference . A T-maze simulation is used to demonstrate how these behaviors emerge naturally, as the agent makes inferences about the observed outcomes and optimizes its generative model (via belief updating). Additionally, the discrete state-space and time formulation presented provides an accessible guide on how to derive the (variational and expected) free energy equations and belief updating rules.
R
in other engineering applications
to more complex settings
D
playing
R
underlying probability distributions (i.e. generative model)
generative models
A
. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside reinforcement learning agents