Active inference is a first principle account of how autonomous agents operate in dynamic, non-stationary environments. This problem is also considered in reinforcement learning  , but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in reinforcement learning ; 2) to our knowledge, the first explicit discrete-state comparison between active inference and reinforcement learning on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of reinforcement learning . We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration - and account for uncertainty about their environment - in a Bayes optimal fashion. We make these properties explicit by showing that the active inference agent's ability to carry out online planning, in a pure-belief setting, enables it to act optimally, given the non-stationary dynamics of an environment when compared to both Q-learning and Bayesian model-based reinforcement learning agents . We conclude by noting that this formalism can be applied to more complex settings ; e.g., robotic arm movement, Atari games, etc., if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside reinforcement learning agents.
Active inference is a first principle account of how autonomous agents operate in dynamic, non-stationary environments. This problem is also considered in reinforcement learning (RL) , but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in RL ; 2) an explicit discrete-state comparison between active inference and RL on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of RL . We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration , and account for uncertainty about their environment in a Bayes-optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in RL is removed in active inference, where reward can simply be treated as another observation; even in the total absence of rewards, agent behaviors are learned through preference learning. We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based RL agents; by placing zero prior preferences over rewards and by learning the prior preferences over the observations corresponding to reward . We conclude by noting that this formalism can be applied to more complex settings if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside RL agents.
A
(RL)
R
reinforcement learning
RL
R
to our knowledge, the first
an
R
reinforcement learning
RL
R
reinforcement learning
RL
R
-
,
R
- in a Bayes optimal fashion.
in a Bayes-optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in RL is removed in active inference, where reward can simply be treated as another observation; even in the total absence of rewards, agent behaviors are learned through preference learning.
R
that the active inference agent's ability to carry out online planning, in a pure-belief setting, enables it to act optimally, given the non-stationary dynamics of an environment when
two scenarios in which active inference agents can infer behaviors in reward-free environments
R
reinforcement learning agents
RL agents; by placing zero prior preferences over rewards and by learning the prior preferences over the observations corresponding to reward
D
; e.g., robotic arm movement, Atari games, etc.,
R
reinforcement learning
RL