Original: P RIOR P REFERENCE L EARNING FROM E XPERTS : D ESIGNING A R EWARD WITH A CTIVE I NFERENCE
Revised: P RIOR P REFERENCE L EARNING FROM E XPERTS : D ESIGNING A R EWARD WITH A CTIVE I NFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Active inference may be deﬁned as Bayesian modeling of a brain with a biologically plausible model of the agent.
Revised: Active inference may be deﬁned as Bayesian modeling of a brain with a biologically plausible model of the agent.

Original: Its primary idea relies on the free energy principle and the prior preference of the agent.
Revised: Its primary idea relies on the free energy principle and the prior preference of the agent.

Original: An agent will choose an action that leads to its prior preference for a future observation.
Revised: An agent will choose an action that leads to its prior preference for a future observation.

Original: In this paper, we claim that active inference can be interpreted using reinforcement learning (RL) algorithms and ﬁnd a theoretical connection between them.
Revised: In this paper , we claim that active inference can be interpreted using reinforcement learning (RL) algorithms and ﬁnd a theoretical connection between them.

Original: We extend the concept of expected free energy (EFE), which is a core quantity in active inference and claim that EFE can be treated as a negative value function.
Revised: We extend the concept of expected free energy (EFE), which is a core quantity in active inference, and claim that EFE can be treated as a negative value function.

Original: Motivated by the concept of prior preference and a theoretical connection, we propose a simple but novel method for learning a prior preference from experts.
Revised: Motivated by the concept of prior preference and a theoretical connection, we propose a simple but novel method for learning a prior preference from experts.

Original: This illustrates that the problem with  RL can be approached with a new perspective of active inference.
Revised: This illustrates that the problem with inverse RL can be approached with a new perspective of active inference.

Original: Experimental results of prior preference learning show the possibility of active inference with EFE-based rewards and its application to an inverse RL problem
Revised: Experimental results of prior preference learning show the possibility of active inference with EFE-based rewards and its application to an inverse RL problem

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Active inference (Friston et al., 2009) is a theory emerging from cognitive science using a Bayesian modeling of the brain function (Friston et al., 2006; Friston, 2010; Friston et al., 2015; 2013), predictive coding (Friston et al., 2011; Lopez-Persem et al., 2016), and the free energy principle (Friston, 2012; Parr & Friston, 2019; Friston, 2019).
Revised: Active inference (Friston et al., 2009) is a theory emerging from cognitive science using a Bayesian modeling of the brain function (Friston et al., 2006; Friston, 2010; Friston et al., 2015; 2013), predictive coding (Friston et al., 2011; Lopez-Persem et al., 2016), and the free energy principle (Friston, 2012; Parr & Friston, 2019; Friston, 2019).

Original: It states that the agents choose actions to minimize an expected future surprise (Friston et al., 2012; 2017a;b), which is a measurement of the difference between an agent’s prior preference and expected future.
Revised: It states that the agents choose actions to minimize an expected future surprise (Friston et al., 2012; 2017a;b), which is a measurement of the difference between an agent’s prior preference and expected future.

Original: Minimization of an expected future surprise can be achieved by minimizing the expected free energy (EFE), which is a core quantity of active inference.
Revised: Minimization of an expected future surprise can be achieved by minimizing the expected free energy (EFE), which is a core quantity of active inference.

Original: Although active inference and EFE have been inspired and derived from cognitive science using a biologically plausible brain function model, its usage in RL tasks is still limited owing to its computational issues and prior-preference design.
Revised: Although active inference and EFE have been inspired and derived from cognitive science using a biologically plausible brain function model, its usage in RL tasks is still limited owing to its computational issues and prior-preference design.

Original: (Millidge, 2020; Fountas et al., 2020)
Revised: (Millidge, 2020; Fountas et al., 2020)

Original: First, EFE requires heavy computational cost.
Revised: First, EFE requires heavy computational cost.

Original: A precise computation of an EFE theoretically averages all possible policies, which is clearly intractable as an action space A and a time horizon T increase in size.
Revised: A precise computation of an EFE theoretically averages all possible policies, which is clearly intractable as an action space A and a time horizon T increase in size.

Original: Several attempts have been made to calculate the EFE in a tractable manner, such as limiting the future time horizon from t to t
Revised: Several attempts have been made to calculate the EFE in a tractable manner, such as limiting the future time horizon from t to t

Original: + H (Tschantz et al., 2019), and applying Monte-Carlo based sampling methods (Fountas et al., 2020; C¸atal et al., 2020) for the search policies.
Revised: + H (Tschantz et al., 2019), and applying Monte-Carlo based sampling methods (Fountas et al., 2020; C¸atal et al., 2020) for the search policies.

Original: Second, it is unclear how the prior preferences should be set.
Revised: Second, it is unclear how the prior preferences should be set.

Original: This is the same question as how to design the rewards in the RL algorithm.
Revised: This is the same question as how to design the rewards in the RL algorithm.

Original: In recent studies (Fountas et al., 2020; C¸atal et al., 2020;
Revised: In recent studies (Fountas et al., 2020; C¸atal et al., 2020;

Original: Ueltzh¨offer, 2018)
Revised: Ueltzh¨offer, 2018)

Original: the agent’s prior preference is simply set as the ﬁnal goal of a given environment for every time step.
Revised: the agent’s prior preference is simply set as the ﬁnal goal of a given environment for every time step.

Original: There are some environments in which the prior preference can be set as time independent.
Revised: There are some environments in which the prior preference can be set as time independent.

Original: However, most prior preferences in RL problems are neither simple nor easy to design because prior preferences of short and long-sighted futures should generally be treated in different ways.
Revised: However, most prior preferences in RL problems are neither simple nor easy to design because prior preferences of short and long-sighted futures should generally be treated in different ways.

Original: In this paper, we ﬁrst claim that there is a theoretical connection between active inference and RL algorithms.
Revised: In this paper, we ﬁrst claim that there is a theoretical connection between active inference and RL algorithms.

Original: We then propose prior preference learning (PPL), a simple and novel method for learning a prior-preference of an active inference from an expert simulation.
Revised: We then propose prior preference learning (PPL), a simple and novel method for learning a prior-preference of an active inference from an expert simulation.

Original: In Section 2, we brieﬂy introduce the concept of an active inference.
Revised: In Section 2, we brieﬂy introduce the concept of an active inference.

Original: From the previous deﬁnition of the EFE of a deterministic policy, in Section 3, we extend the previous concepts of active inference and theoretically demonstrate that it can be analyzed in view of the RL algorithm.
Revised: From the previous deﬁnition of the EFE of a deterministic policy, in Section 3, we extend the previous concepts of active inference and theoretically demonstrate that it can be analyzed in view of the RL algorithm.

Original: We extend this quantity to a stochastic policy network and deﬁne an action-conditioned EFE for a given action and a given policy network.
Revised: We extend this quantity to a stochastic policy network and deﬁne an action-conditioned EFE for a given action and a given policy network.

Original: Following Millidge (2020) using a bootstrapping argument, we show that the optimal distribution over the ﬁrst-step action induced from active inference can be interpreted using Q-Learning.
Revised: Following Millidge (2020) using a bootstrapping argument, we show that the optimal distribution over the ﬁrst-step action induced from active inference can be interpreted using Q-Learning.

Original: Consequently, we show that EFE can be treated as a negative value function from an RL perspective.
Revised: Consequently, we show that EFE can be treated as a negative value function from an RL perspective.

Original: From this connection, in Section 4, we propose a novel method for designing EFE-based rewards, by learning a prior preference from expert demonstrations.
Revised: From this connection, in Section 4, we propose a novel inverse RL algorithm for designing EFEbased rewards, by learning a prior preference from expert demonstrations.

Original: Through such expert demonstrations, an agent learns its prior preference given the observation to achieve a ﬁnal goal, which can effectively handle the difference between local and global preferences.
Revised: Through such expert demonstrations, an agent learns its prior preference given the observation to achieve a ﬁnal goal, which can effectively handle the difference between local and global preferences.

Original:  Our experiments in Section 5 show the applicability of active inference based rewards using EFE to an inverse RL problem.
Revised: It will extend the scope of active inference to inverse RL problem. Our experiments in Section 6 show the applicability of active inference based rewards using EFE to an inverse RL problem.

Original:  ACTIVE INFERENCE
Revised:  ACTIVE INFERENCE

Original: The active inference environment rests on the partially observed Markov decision process settings with an observation that comes from sensory input o t and a hidden state s t which is encoded in the agent’s latent space.
Revised: The active inference environment rests on the partially observed Markov decision process settings with an observation that comes from sensory input o t and a hidden state s t which is encoded in the agent’s latent space.

Original: We will discuss a continuous observation/hidden state space, a discrete time step, and a discrete action space A :
Revised: We will discuss a continuous observation/hidden state space, a discrete time step, and a discrete action space A :

Original: At a current time t < T with a given time horizon T , the agent receives an observation o t .The
Revised: At a current time t < T with a given time horizon T , the agent receives an observation o t .

Original:  agent encodes this observation to a hidden state s t in its internal generative model, (i.e., a generative model for the given environment in an agent) and then searches for the action sequence that minimizes the expected future surprise based on the agent’s prior preference ˜ p
Revised: The agent encodes this observation to a hidden state s t in its internal generative model, (i.e., a generative model for the given environment in an agent) and then searches for the action sequence that minimizes the expected future surprise based on the agent’s prior preference ˜ p

Original: ( o τ ) of a future observation o τ with τ > t .
Revised: ( o τ ) of a future observation o τ with τ > t .

Revised: (i.e. The agent avoids an action which leads to unexpected and undesired future observations, which makes the agent surprised.)

Original: In detail, we can formally illustrate the active inference agent’s process as follows: s t and o t are a hidden state and an observation at time t , respectively.
Revised: In detail, we can formally illustrate the active inference agent’s process as follows: s t and o t are a hidden state and an observation at time t , respectively.

Original: In addition, π = ( a 1 , a 2 , ..., a T ) is a sequence of actions.
Revised: In addition, π = ( a 1 , a 2 , ..., a T ) is a sequence of actions.

Original: Let p ( o 1: T , s 1: T ) be a generative model of the agent with its transition model p ( s t +1 | s t , a t ) , and Q ( o 1: T , s 1: T , π ) be a variational density.
Revised: Let p ( o 1: T , s 1: T ) be a generative model of the agent with its transition model p ( s t +1 | s t , a t ) , and q ( o 1: T , s 1: T , π ) be a variational density.

Original: A distribution over policies Q ( π ) will be determined later.
Revised: A distribution over policies q ( π ) will be determined later.

Original: From here, we can simplify the parameterized densities as trainable neural networks with p ( o t | s t ) as a decoder, Q ( s t | o t ) as an encoder, and p ( s t +1 | s t , a t ) as a transition network in our generative model.
Revised: From here, we can simplify the parameterized densities as trainable neural networks with p ( o t | s t ) as a decoder, q ( s t | o t ) as an encoder, and p ( s t +1 | s t , a t ) as a transition network in our generative model.

Original: First, we minimize the current surprise of the agent, which is deﬁned as − log p ( o t ) .
Revised: First, we minimize the current surprise of the agent, which is deﬁned as − log p ( o t ) .

Original: Its upper bound can be interpreted as the well-known negative ELBO term, which is frequently referred to as the variational free energy F t at time t in studies on active inference.
Revised: Its upper bound can be interpreted as the well-known negative ELBO term, which is frequently referred to as the variational free energy F t at time t in studies on active inference.

Original: [Equation]
Revised: [Equation]

Original: Minimizing F t provides an upper bound on the current surprise, and makes our networks in the generative model well-ﬁtted with our known observations of the environment and its encoded states.
Revised: Minimizing F t provides an upper bound on the current surprise, and makes our networks in the generative model well-ﬁtted with our known observations of the environment and its encoded states.

Original: For the future action selection, the total EFE G ( s t ) over all possible policies at the current state s t at time t should be minimized.
Revised: For the future action selection, the total EFE G ( s t ) over all possible policies at the current state s t at time t should be minimized.

Original: [Equation]
Revised: [Equation]

Original: Focusing on the distribution Q ( π ) , it is known that the total EFE G ( s t ) is minimized when the distribution over policies Q ( π ) follows σ ( − G π ( s t )) , where σ ( · ) is a softmax over the policies and
Revised: Focusing on the distribution q ( π ) , it is known that the total EFE G ( s t ) is minimized when the distribution over policies q ( π ) follows σ ( − G π ( s t )) , where σ ( · ) is a softmax over the policies and

Original: G π ( s t ) is the EFE under a given state s t for a ﬁxed sequence of actions π at time t .
Revised: G π ( s t ) is the EFE under a given state s t for a ﬁxed sequence of actions π at time t .

Original: (Millidge et al., 2020)
Revised: (Millidge et al., 2020)

Original: [Equation]
Revised: [Equation]

Original: This means that a lower EFE is obtained for a particular action sequence π ; a lower future surprise will be expected and a desired behavior ˜ p ( o ) will be obtained.
Revised: This means that a lower EFE is obtained for a particular action sequence π ; a lower future surprise will be expected and a desired behavior ˜ p ( o ) will be obtained.

Original: Several active inference studies introduce a temperature parameter γ > 0 such that Q γ ( π ) =
Revised: Several active inference studies introduce a temperature parameter γ > 0 such that q γ ( π ) =

Original: σ ( − γG π ( s t )) to control the agent’s behavior between exploration and exploitation.
Revised: σ ( − γG π ( s t )) to control the agent’s behavior between exploration and exploitation.

Original: Because we know that the optimal distribution over the policies is Q ( π )
Revised: Because we know that the optimal distribution over the policies is q ( π )

Original: = σ ( − G π ( s t )) , the action selection problem boils down to a calculation of the expected free energy G π ( s t ) of a given action sequence π .
Revised: = σ ( − G π ( s t )) , the action selection problem boils down to a calculation of the expected free energy G π ( s t ) of a given action sequence π .

Original: The learning process of active inference contains two parts: (1) learning an agent’s generative model with its trainable neural networks p ( o t | s t ) , Q ( s t | o t ) , and p ( s t +1 | s t , a t ) that explains the current observations and (2) learning to select an action that minimizes a future expected surprise of the agent by calculating the EFE of a given action sequence.
Revised: The learning process of active inference contains two parts: (1) learning an agent’s generative model with its trainable neural networks p ( o t | s t ) , q ( s t | o t ) , and p ( s t +1 | s t , a t ) that explains the current observations and (2) learning to select an action that minimizes a future expected surprise of the agent by calculating the EFE of a given action sequence.

Original:  EFE AS A NEGATIVE VALUE : B ETWEEN RL AND ACTIVE INFERENCE
Revised:  EFE AS A NEGATIVE VALUE : B ETWEEN RL AND ACTIVE INFERENCE

Original: In this section, we ﬁrst extend the deﬁnition of an EFE to a stochastic policy.
Revised: In this section, we ﬁrst extend the deﬁnition of an EFE to a stochastic policy.

Original: We then, propose an action-conditioned EFE that has a similar role as a negative action-value function in RL.
Revised: We then, propose an action-conditioned EFE that has a similar role as a negative action-value function in RL.

Original: Based on these extensions, we will repeat the arguments in the active inference and claim that the RL algorithm with EFE as a negative value is equivalent to controlling the policy network toward an ideal distribution in the active inference.
Revised: Based on these extensions, we will repeat the arguments in the active inference and claim that the RL algorithm with EFE as a negative value is equivalent to controlling the policy network toward an ideal distribution in the active inference.

Original: Calculating the expected free energy G π ( s t ) for all possible deterministic policies π is intractable even in a toy-example task because the number of policies rapidly increases as the time horizon T and the number of actions |A| increases.
Revised: Calculating the expected free energy G π ( s t ) for all possible deterministic policies π is intractable even in a toy-example task because the number of policies rapidly increases as the time horizon T and the number of actions |A| increases.

Original: Instead of searching for a number of deterministic policies, we extend the concept of EFE to a stochastic policy based on a policy network φ = φ ( a t | s t ) .
Revised: Instead of searching for a number of deterministic policies, we extend the concept of EFE to a stochastic policy based on a policy network φ = φ ( a t | s t ) .

Original: In this case, we also extend Q ( s τ | π ) to Q ( s τ , a τ − 1 | φ ) :=
Revised: In this case, we also extend q ( s τ | π ) to q ( s τ , a τ − 1 | φ ) :

Original:  p ( s τ | s τ − 1 , a τ − 1 ) φ ( a τ − 1 | s τ − 1 ) , which is a distribution over the states and actions, where each action a τ − 1 is only dependent on state s τ − 1 with φ ( a τ − 1 | s τ − 1 ) .
Revised: = p ( s τ | s τ − 1 , a τ − 1 ) φ ( a τ − 1 | s τ − 1 ) , which is a distribution over the states and actions, where each action a τ − 1 is only dependent on state s τ − 1 with φ ( a τ − 1 | s τ − 1 ) .

Original: Plugging in a deterministic policy φ = π yields Q ( s τ , a τ − 1 | π ) =
Revised: Plugging in a deterministic policy φ = π yields q ( s τ , a τ − 1 | π ) =

Original: p ( s τ | s τ − 1 , a (cid:48) ) with π ( a τ − 1 )
Revised: p ( s τ | s τ − 1 , a (cid:48) ) with π ( s τ − 1 )

Original: = a (cid:48) , which is the same equation used in previous studies on active inference.
Revised: = a (cid:48) , which is the same equation used in previous studies on active inference.

Original: Suppose we choose an action a t based on the current state s t and a given action network φ , its corresponding expected free energy term can then be written as follows.
Revised: Suppose we choose an action a t based on the current state s t and a given action network φ , its corresponding expected free energy term can then be written as follows.

Original: This can be interpreted as an EFE of a sequence of policies ( φ, φ, ..., φ ) .
Revised: This can be interpreted as an EFE of a sequence of policies ( φ, φ, ..., φ ) by substituting our extensions for the probabilities in (3).

Original: [Equation]
Revised: [Equation]

Original: Note that the agent uses the same action network φ ( a τ | s τ ) for all τ , and we can therefore rewrite this equation in a recursive form.
Revised: Note that the agent uses the same action network φ ( a τ | s τ ) for all τ , and we can therefore rewrite this equation in a recursive form.

Original: [Equation]
Revised: [Equation]

Original: Replacing and ﬁxing the ﬁrst action a t = a , an action-conditioned EFE G φ ( s t | a ) of a given action a and a policy network φ is then deﬁned as follows:
Revised: Replacing and ﬁxing the ﬁrst action a t = a , an action-conditioned EFE G φ ( s t | a ) of a given action a and a policy network φ is then deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Taking an expectation over φ ( a | s t )
Revised: Taking an expectation over φ ( a | s t )

Original: [Equation]
Revised: [Equation]

Original: we obtain the relationship between G φ ( s t ) and G φ ( s t | a t ).
Revised: we obtain the relationship between G φ ( s t ) and G φ ( s t | a t )

Original: [Equation]
Revised: [Equation]

Original: We may consider separating the distribution over the ﬁrst-step action from φ to ﬁnd an alternative distribution that minimizes the EFE.
Revised: We may consider separating the distribution over the ﬁrst-step action from φ to ﬁnd an alternative distribution that minimizes the EFE.

Original: Substituting the distribution over the ﬁrst-step action as Q ( a t ) instead of φ ( a t ) [Equation] we obtain its one-step substituted EFE as indicated below.
Revised: Substituting the distribution over the ﬁrst-step action as q ( a t ) instead of φ ( a t ) , we obtain its one-step substituted EFE as indicated below.

Original: This can be interpreted as the EFE of a sequence of a policies ( Q ( a t ) , φ, ..., φ ) 
Revised: This can be interpreted as the EFE of a sequence of a policies ( q ( a t ) , φ, ..., φ ) [Equation]

Original: [Equation]
Revised: [Equation]

Original: Under a given φ , the value above depends only on the distribution Q ( a t ) .
Revised: Under a given φ , the value above depends only on the distribution q ( a t ) .

Original: Thus, minimizing the quantity above will naturally introduce the distribution Q ∗ ( a t ) =
Revised: Thus, minimizing the quantity above will naturally introduce the distribution q ∗ ( a t ) =

Original: σ a ( − G φ ( s t | a t )) , which is known to be similar in terms of active inference to the γ = 1 case.
Revised: σ a ( − G φ ( s t | a t )) , which is known to be similar in terms of active inference to the γ = 1 case.

Original: [Equation]
Revised: [Equation]

Original: Therefore, G φ 1 − step ( s t ) 
Revised: Therefore, G φ 1 − step ( s t ) ≥ − log

Original: [Equation]
Revised: [Equation]

Original: − log

Original: [Equation]

Original: a t ∈A exp( − G φ ( s t | a t )) [Equation] and the equality holds if and only if Q ( a t ) =
Revised: a t ∈A exp( − G φ ( s t | a t )) , and the equality holds if and only if q ( a t ) =

Original: σ a ( − G φ ( s t | a t )) .
Revised: σ a ( − G φ ( s t | a t )) .

Original: Let us further consider the temperature hyperparameter γ > 0 such that Q
Revised: Let us further consider the temperature hyperparameter γ > 0 such that q

Original: [Equation]
Revised: [Equation]

Original: γ ( [Equation] =
Revised: ( a t ) =

Original: σ a ( − γG ( s t | a t )) .
Revised: σ a ( − γG ( s t | a t )) .

Original: Similarly to the above, we obtain the following:
Revised: Similarly to the above, we obtain the following:

Original: [Equation]
Revised: [Equation]

Original: From the arguments above [Equation] we can conclude that (1) G φ 1 − step ( s t ) is minimized when Q ∗ ( a t )
Revised: From the arguments above, we can conclude that (1) G φ 1 − step ( s t ) is minimized when q ∗ ( a t )

Original: =
Revised: =

Original: σ a ( − G ( s t | a t )) , a ‘natural case’ in which γ = 1 , which was heuristically set in the experiments described in Millidge (2020).
Revised: σ a ( − G ( s t | a t )) , a ‘natural case’ in which γ = 1 , which was heuristically set in the experiments described in Millidge (2020).

Original: (2) Plugging Q ∗ γ ( a t ) =
Revised: (2) Plugging q ∗ γ ( a t ) =

Original: σ a ( − γG ( s t | a t )) to Q ( a ) in the equation above, we obtain
Revised: σ a ( − γG ( s t | a t )) to q ( a ) in the equation above, we obtain

Original: [Equation]
Revised: [Equation]

Original: where D γ = (cid:80) a t ∈A exp( − γG φ ( s t | a t )) , and the lastapproximation comes from the known property of  the log-sum-exp with
Revised: where D γ = (cid:80) a t ∈A exp( − γG φ ( s t | a t )) , and the last comes from the smooth approximation to the maximum function of log-sum-exp. This approximation becomes accurate when the maximum is much larger than the others.

Original: [Equation]
Revised: [Equation]

Original: When γ = 1 , the quantity G φ 1 − step ( s t ) is minimized with Q ∗ 1 ( a t ) =
Revised: When γ = 1 , the quantity G φ 1 − step ( s t ) is minimized with q ∗ 1 ( a t ) =

Original: σ a ( − G ( s t | a t )) , and its minimum value can be approximated as − log D 1 ≈ min a t ∈A { G φ ( s t | a t ) } .
Revised: σ a ( − G ( s t | a t )) , and its minimum value can be approximated as − log D 1 ≈ min a t ∈A { G φ ( s t | a t ) } .

Original: When γ → ∞ , Q ∗ γ ( a t ) can be considered as a deterministic policy seeking the smallest EFE, which leads to the following:
Revised: When γ → ∞ , q ∗ γ ( a t ) can be considered as a deterministic policy seeking the smallest EFE, which leads to the following:

Original: [Equation]
Revised: [Equation]

Original: Note that Q-learning with a negative EFE can be interpreted as Q ∗ γ with the case of γ = inf .
Revised: Note that Q-learning with a negative EFE can be interpreted as q ∗ γ with the case of γ = ∞ .

Original: When γ (cid:38) 0 , Q ∗ γ ( a t ) converges to a uniform distribution over the action space A , and the temperature hyperparameter γ motivates the agent to explore other actions with a greater EFE.
Revised: When γ (cid:38) 0 , q ∗ γ ( a t ) converges to a uniform distribution over the action space A , and the temperature hyperparameter γ motivates the agent to explore other actions with a greater EFE.

Original: Its weighted sum also converges to an average of the action-conditioned EFE.
Revised: Its weighted sum also converges to an average of the action-conditioned EFE.

Original: Considering the optimal policy φ ∗ that seeks an action with a minimum EFE, we obtain the following:
Revised: Considering the optimal policy φ ∗ that seeks an action with a minimum EFE, we obtain the following:

Original: [Equation]
Revised: [Equation]

Original: This equation is extremely similar to the Bellman optimality equation.
Revised: This equation is extremely similar to the Bellman optimality equation.

Original: Here, G φ ( s t | a ) and G φ ( s t +1 ) correspond to the action-value function and the state value function, respectively.
Revised: Here, G φ ( s t | a ) and G φ ( s t +1 ) correspond to the action-value function and the state value function, respectively.

Original: Based on this similarity, we can consider the ﬁrst term log
Revised: Based on this similarity, we can consider the ﬁrst term log

Original: [Equation]
Revised: [Equation]

Original: as a one-step negative reward (because active inference aims to minimize the expected free energy of the future) and EFE as a negative value function.
Revised: as a one-step negative reward (because active inference aims to minimize the expected free energy of the future) and EFE as a negative value function.

Original:  PPL: P RIOR P REFERENCE L EARNING FROM E XPERTS
Revised:  PPL: P RIOR P REFERENCE L EARNING FROM E XPERTS

Original: From the previous section, we veriﬁed that using log
Revised: From the previous section, we veriﬁed that using log

Original: [Equation]
Revised: [Equation]

Original: as a negative reward can handle EFE with traditional RL methods.
Revised: as a negative reward can handle EFE with traditional RL methods.

Original: Through a simple calculation, the given term can be decomposed as follows:
Revised: Through a simple calculation, the given term can be decomposed as follows:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term measures the similarity between a preferred future and a predicted future, which is an intuitive value.
Revised: The ﬁrst term measures the similarity between a preferred future and a predicted future, which is an intuitive value.

Original: The second term is called the epistemic value, which encourages the exploration of an agent.
Revised: The second term is called the epistemic value, which encourages the exploration of an agent.

Original: (Friston et al., 2015)
Revised: (Friston et al., 2015)

Original: The epistemic value can be computed either with prior knowledge on the environment of an agent or with various algorithms to learn a generative model.
Revised: The epistemic value can be computed either with prior knowledge on the environment of an agent or with various algorithms to learn a generative model.

Original: (Igl et al., 2018;
Revised: (Igl et al., 2018;

Original: Kaiser et al., 2019)
Revised: Kaiser et al., 2019)

Original: The core key to calculating the one-step reward and learning EFE is the prior preference ˜ p ( o t ) , which includes information about the agent’s preferred observation and goal.
Revised: The core key to calculating the one-step reward and learning EFE is the prior preference ˜ p ( o t ) , which includes information about the agent’s preferred observation and goal.

Original: Setting a prior preference for an agent is a challenging point for active inference.
Revised: Setting a prior preference for an agent is a challenging point for active inference.

Original: A simple method that has been used in recent studies (Ueltzh ¨ offer, 2018; C¸atal et al., 2020) is to set a prior preference as a Gaussian distribution  of themean as goal observation.
Revised: A simple method that has been used in recent studies (Ueltzh ¨ offer, 2018; C¸atal et al., 2020) is to set a prior preference as a Gaussian distribution with mean of the goal position.

Original: However, it would be inefﬁcient to use the same prior preference for all observations o .
Revised: However, it would be inefﬁcient to use the same prior preference for all observations o .

Original: Local preferences may differ from a global preference.
Revised: We called this type of preference global preference .

Original: Taking a mountaincar environment as an example, to reach the goal position, the car must move away from the goal position in an early time step.
Revised: Taking a mountaincar environment as an example, to reach the goal position, the car must move away from the goal position in an early time step.

Revised: Therefore ˜ p ( o t ) must contain information about local preference .

Original: It is also difﬁcult to directly design the prior preference for some observation space, such as in an image.
Revised: It is also difﬁcult to directly design the prior preference for some observation space, such as in an image.

Original: To solve the problems arising from active inference, we introduce prior preference learning from experts (PPL).
Revised: To solve the problems arising from active inference, we introduce prior preference learning from experts (PPL).

Original: Suppose that an agent can access expert simulations S
Revised: Suppose that an agent can access expert simulations S

Original: = {
Revised: = {

Original: ( o i, 1 , ..., o i,T } Ni =1 , where
Revised: ( o i, 1 , ..., o i,T } Ni =1 , where

Original: N is the number of simulations.
Revised: N is the number of simulations.

Original: Note that the agent does not need an access to the expert’s action.

Original: (Edwards et al., 2019)

Original: From the expert simulations S , an agent can learn an expert’s prior p ( o t +1 | o t ) based on the current observation o t .
Revised: From the expert simulations S , an agent can learn an expert’s prior p ( o t +1 | o t ) based on the current observation o t .

Original: Model p ( o t +1 | o t ) captures the expert’s local preference, which is more effective than the global preference.
Revised: Model p ( o t +1 | o t ) captures the expert’s local preference, which is more effective than the global preference.

Original: We can state that this method can be applied only with expert simulations, regardless of the knowledge on the forward dynamics of the environment.
Revised: We can state that this method can be applied only with expert simulations, regardless of the knowledge on the forward dynamics of the environment.

Original: E XPERIMENTS
Revised: Once learning the prior preference p ( o t +1 | o t ) , we can calculate R t given in (15).

Original: In this section, we discuss and compare the experimental results of the classical control environments.
Revised: We can use any

Original: First, we aim to compare the conventional global preference method to the PPL.
Revised: RL alogorithm to approximate EFE, G φ ∗ ( s t ) in (14). In Algorithm 1, we provide our pseudo-code of PPL with Q-learning.

Original:  We expect the PPL to be effective in environments where local and global preferences are different. We also observe how an epistemic value in a one-step reward term inﬂuences the learning process and its performance.
Revised:  R ELATED WORKS

Original: In addition, we use expert simulations for batch sampling when learning EFE and a forward dynamic model.
Revised: Active inference on RL. Our works are based on active inference and reinforcement learning.

Original: This allows an EFE network and a dynamic model to be trained even in states that do not reach the early stage of learning. Note that our dynamic model needs access to expert actions.
Revised: Active inference was ﬁrst introduced in Friston et al.

Original: We use the Q-learning algorithm to learn the EFE network.
Revised: (2006), inspired from neuroscience and free energy principle.

Original: Commonly used techniques in RL, such as replay memory and target network (Mnih et al., 2015) are used.
Revised: It explains how a biological system is maintained with a brain model. Furthermore, Friston et al. (2009) treated the relation between active inference and reinforcement learning.

Revised: Early studies (Friston, 2010; Friston et al., 2011; 2015; 2017a) dealt with a tabular or model based problem due to computational cost.

Revised: Recently, Ueltzh¨offer (2018) introduced deep active inference which utilizes deep neural network to approximate the observation and transition models on MountainCar environment.

Revised: This work used

Revised: EFE as an objective function and its gradient propagates through the environment dynamics.

Original: We evaluate our approach with a classic control environment implemented in Open AI Gym (Brockman et al., 2016).
Revised: To handle this problem, Ueltzh¨offer (2018) used stochastic weights to apply an evolutionary strategy (Salimans et al., 2017), which is a method for gradient approximation. For a stable gradient approximation, about 10 4 environments were used in parallel. C¸atal et al.

Original: Expert simulations were obtained from Open AI RL baseline zoo (Rafﬁn, 2018).
Revised: Millidge (2020) introduced end-to-end differentiable models by including environment transition models.

Original: Table 1 contains four experimental settings, where the setting 1 is our proposed method and the others are experimental groups to compare the effects of PPL, an expert batch, and the epistemic value. 5. C LASSIC CONTROL ENVIRONMENT We tested three classical control environments: Acrobot, Cartpole, and MountainCar. We used 5000 pairs of ( o t , o t +1 ) to train the prior preference ˜ p
Revised: Both require much less interaction with environments than before.

Original: ( o t +1 ) .
Revised: C¸atal et al.

Original: We used the same neural network architecture for all environments, except for the number of input and output dimensions.
Revised: (2020) used the Monte Carlo sampling to approximate EFE and used global prior preference.

Original: During the training process, we clip the epistemic value to prevent a gradient explosion while using the epistemic value in settings 1, 2, and 3.
Revised: On the other hand Millidge (2020) used bootstrapping methods and used common RL rewards with model driven values derived from EFE. Millidge (2020) veriﬁed that the model driven values induce faster and better results on MountainCar environment.

Original: Learning prior preference from expert simulations;
Revised: Learning prior preference from expert simulations;

Original: Input: Expert simultaions S
Revised: Input: Expert simultaions S

Original: =
Revised: =

Original: ( o i, 1 , ...
Revised: ( o i, 1 , ...

Original: o i,T ) Ni =
Revised: o i,T ) Ni =

Original: Initialize a prior preference network ˜ p θ ( o ) ; while not converge do
Revised: Initialize a prior preference network ˜ p θ ( o ) ; while not converge do

Original: Compute loss L ppl (˜ p ( o t ) , o t +1 ) ;
Revised: Compute loss L ppl (˜ p ( o t ) , o t +1 ) ;

Original: Update θ ← θ − α ∇ L ppl endOutput: ˜
Revised: Update θ ← θ − α ∇ L ppl endOutput: ˜

Original: p θ ( o )
Revised: p θ ( o )

Original: Learning EFE and forward dynamic of an environment;
Revised: Learning EFE and forward dynamic of an environment;

Original: Input: Prior preference ˜ p
Revised: Input: Prior preference ˜ p

Original: θ ( o )
Revised: θ ( o )

Original: Initialize the forward dynamic p η ( o | s ) , T η ( s t +1 | s t , a t ) , Q η ( s | o ) , and EFE network G ξ ( s t , a t ) .
Revised: Initialize the forward dynamic p η ( o | s ) , T η ( s t +1 | s t , a t ) , q η ( s | o ) , and EFE network G ξ ( s t , a t ) .

Original: while not converge do
Revised: while not converge do

Original: Reset environment; for t ← 0 do
Revised: Reset environment; for t ← 0 do

Original: Select action a t = arg max a G ( s t , a ) with (cid:15) -greedy;
Revised: Select action a t = arg max a G ( s t , a ) with (cid:15) -greedy;

Original: Observe new observation o t +1 ;
Revised: Observe new observation o t +1 ;

Original: Compute R t = log a t ) 
Revised: Compute R t = log T ( s t +1 | s t ,a t ) ˜

Original:  ( o  +1 )  +1 ) ;
Revised: p ( o t +1 ) q ( s t +1 | o t +1 ) ;

Original: Compute the environment model loss L model (( p ◦ T ◦ Q )( o t ) , o t +1 ) ;
Revised: Compute the environment model loss L model (( p ◦ T ◦ q )( o t ) , o t +1 ) ;

Original: Compute the EFE network loss L efe ( G ξ ( s t , a t ) , R t + max a G ξ ( s t +1 , a )) ;
Revised: Compute the EFE network loss L efe ( G ξ ( s t , a t ) , R t + max a G ξ ( s t +1 , a )) ;

Original: Update η ← η − α ∇ L d and ξ ← ξ − α ∇ L e fe .
Revised: Update η ← η − α ∇ L d and ξ ← ξ − α ∇ L e fe .

Original: end end
Revised: end end

Revised: Our work extends the scope of active inference to inverse RL by learning a preferred observation from experts.

Revised: A common limitation of previous studies on active inference is the ambiguity of prior preference distribution.

Revised: Previous works have done in environments where its prior preference can be naturally expressed, which is clearly not true in general.

Revised: PPL is an active inference based approach for the invser RL problem setting, which allows us to learn a prior preference from expert simulations.

Revised: This broadens the scope of active inference and provides a new perspective about a connection between active inference and RL.

Original: 5.1. A CROBOT
Revised: Control as inference. Levine (2018) proposed control as inference framework, which interprets a control problem as a probabilistic inference with an additional binary variable O t that indicates whether given action is optimal or not. Several studies on the formulation of RL as an inference problem (Todorov, 2008; Kappen et al., 2009) have been proposed. Control as inference measures a probability that a given action is optimal based on a given reward with p ( O t = 1 | s t , a t ) =

Revised: exp( r ( s t , a t )) .

Revised: That is, from the given reward and the chosen probability model, control as inference calculates the probability of the given action a t with the state s t to be optimal.

Revised: In contrast, active inference measures this probability as in (15) with a prior preference distribution ˜ p and constructs a reward, viewing EFE as a negative value function.

Revised: Although control as inference and active inference as a

Revised: RL in Section 3 have a theoretical similarity based on a duality of a control problem and an inference, our proposed PPL can interpret the reward r ( s t , a t ) and the message function β

Revised: ( s t ) in Levine (2018) as EFE-related quantities.

Revised: Therefore, learning the expert as a prior preference ˜ p immediately constructs its active inference based reward function and thereby applicable to a reward construction problem and several inverse RL problems.

Revised:  E XPERIMENTS

Revised: In this section, we discuss and compare the experimental results of our proposed algorithm PPL and other inverse RL algorithms on the several classical control environments.

Revised: We evaluate our approach with a classic control environment implemented in Open AI

Revised: Gym (Brockman et al., 2016).

Revised: First, we aim to compare the conventional global preference method to the PPL.

Revised: We expect the PPL to be effective in environments where the local and global preferences are different.

Revised: Second, we claim

Revised: [Table]

Revised: that our active inference based approach can achieve a compatible results with current inverse RL algorithms.

Revised: Expert simulations were obtained from Open AI RL baseline zoo (Rafﬁn, 2018).

Revised: 6. PPL AND GLOBAL PREFERENCE

Revised: First, we compare our proposed algorithm (setting 1 in Table 1) and its variants.

Revised: Table 1 contains four experimental settings, where the setting 1 is our proposed method and the others are experimental groups to compare the effects of an expert batch, the epistemic value R t,e , and PPL.

Revised: Expert Batch.

Revised: We use expert simulations for batch sampling when learning EFE and a forward dynamic model.

Revised: This allows an EFE network and a dynamic model to be trained even in states that do not reach the early stage of learning.

Revised: We use the Q-learning algorithm to learn the EFE network.

Revised: Commonly used techniques in RL, such as replay memory and target network (Mnih et al., 2015) are used.

Revised: Reward.

Revised: We observe how an epistemic value R t,e in (15) inﬂuences the learning process and its performance.

Revised: Global preference.

Revised: Global preference is a distribution over a state which can be naturally induced from the agent’s goal, whereas our PPL is a learned prior preference from the expert’s simulations.

Revised: Roughly, global preference can be understood as a hard-coded prior preference based on the prior knowledge of the environment, as a ’goal directed behavior’ in Ueltzh ¨ offer (2018).

Revised: Detailed hardcoded global preferences in our experiments can be found in the below sub-subsection.

Revised: 6.1. E NVIRONMENTS AND EXPERIMENT DETAILS

Revised: We tested three classical control environments: Acrobot, Cartpole, and MountainCar.

Revised: We used 5000 pairs of ( o t , o t +1 ) to train the prior preference ˜ p

Revised: ( o t +1 ) .

Revised: We used the same neural network architecture for all environments, except for the number of input and output dimensions.

Revised: During the training process, we clip the epistemic value to prevent a gradient explosion while using the epistemic value in settings 1, 2, and 4.

Original: Acrobot is an environment with a two-link pendulum, where the second link is actuated.
Revised: Acrobot is an environment with a two-link pendulum, where the second link is actuated.

Original: The goal of this environment is to swing the end effector up to the horizontal line.
Revised: The goal of this environment is to swing the end effector up to the horizontal line.

Original: The state space has six dimensions: (cos θ 1 , sin θ 1 , cos θ 2 , sin θ 2 , ˙ θ 1 , ˙ θ 2 ) , where θ 1 and θ 2 are the two rotational joint angles.
Revised: The state space has six dimensions: (cos θ 1 , sin θ 1 , cos θ 2 , sin θ 2 , ˙ θ 1 , ˙ θ 2 ) , where θ 1 and θ 2 are the two rotational joint angles.

Original: The action space consists of three actions: accelerating +1, 0, or -1 torque on the joint between the two links.
Revised: The action space consists of three actions: accelerating +1, 0, or -1 torque on the joint between the two links.

Original: We did not run setting 2 in this study, because Acrobot is ambiguous in deﬁning the global preference of the environment.5.1. C ARTPOLE
Revised: We did not run setting 4 in this study, because Acrobot is ambiguous in deﬁning the global preference of the environment.

Original: Cartpole is an environment with a cart and a pole on the cart.
Revised: Cartpole is an environment with a cart and a pole on the cart.

Original: The goal is to balance the pole by controlling the cart.
Revised: The goal is to balance the pole by controlling the cart.

Original: The state space is of 4 dimensions: ( x, ˙ x, θ, ˙ θ ) , where x is the position of the cart and θ is the angle of the pole from the vertical line.
Revised: The state space is of 4 dimensions: ( x, ˙ x, θ, ˙ θ ) , where x is the position of the cart and θ is the angle of the pole from the vertical line.

Original: The action space consists of two actions: accelerating the cart to the left or right.
Revised: The action space consists of two actions: accelerating the cart to the left or right.

Original: The global preference is given by a Gaussian distribution with mean (0, 0, 0, 0) with a ﬁxed standard deviation 0 .
Revised: The global preference is given by a Gaussian distribution with mean (0, 0, 0, 0) with a ﬁxed standard deviation 0 .

Original: 1 on each dimension.
Revised: 1 on each dimension.

Original: 5.1. M OUNTAIN C AR

Original: MountainCar is an environment with a car on the valley.
Revised: MountainCar is an environment with a car on the valley.

Original: The goal is to make the car reach the goal position, the right peak of the mountain.
Revised: The goal is to make the car reach the goal position, the right peak of the mountain.

Original: The state space is of two dimensions ( x, ˙ x ) where x is the position of the car.
Revised: The state space is of two dimensions ( x, ˙ x ) where x is the position of the car.

Original: The action space consists of three actions: accelerating the car to the left or right, or leaving it.
Revised: The action space consists of three actions: accelerating the car to the left or right, or leaving it.

Original: The global preference is given by a Gaussian distribution with mean (0.5, 0) with a ﬁxed standard deviation 0 .
Revised: The global preference is given by a Gaussian distribution with mean (0.5, 0) with a ﬁxed standard deviation 0 .

Original: 1 on each dimension.
Revised: 1 on each dimension.

Original: Note that (0 . 5 , 0) is the state of the goal position with zero velocity.
Revised: Note that (0 . 5 , 0) is the state of the goal position with zero velocity.

Original: 5. R ESULTS AND D ISCUSSIONS
Revised: 6.1. R ESULTS AND D ISCUSSIONS

Original: Figure 1 shows the experimental results on  Acrobot,Cartpole, and MountainCar.
Revised: Figure 1 shows the experimental results on MountainCar, Acrobot, and Cartpole.

Original: Their performances are compared and benchmarked with the default reward of the environments
Revised: Their performances are compared and benchmarked with the default reward of the environments.

Original: [Table] • PPL and global preference : Comparing setting 1 (blue line) and setting 2 (red line), it can be seen that PPL is more efﬁcient than the conventional global preference as expected.
Revised: Comparing setting 1 (blue line) and setting 2 (red line), it can be seen that PPL is more efﬁcient than the conventional global preference as expected.

Original: In particular, in MountainCar, we can see that little learning is achieved.
Revised: In particular, in MountainCar, we can see that little learning is achieved.

Original: This seems to be because the difference between global and expert preferences is greater in the MountainCar environment.
Revised: This seems to be because the difference between global and expert preferences is greater in the MountainCar environment.

Original: In the Cartpole, setting 2 learned more slowly than setting 1.
Revised: In the Cartpole, setting 2 learned more slowly than setting 1.

Original: • Expert Batch :
Revised: Expert Batch.

Original: Comparing setting 1 (blue line) and setting 3 (orange line), it can be seen that using the expert batch is helpful for certain tasks.
Revised: Comparing setting 1 (blue line) and setting 3 (orange line), it can be seen that using the expert batch is helpful for certain tasks.

Original: With Acrobot and MountainCar, the use of an expert batch performs better than the case without an expert batch.
Revised: With Acrobot and MountainCar, the use of an expert batch performs better than the case without an expert batch.

Original: However, the results without expert batch are marginally better than those of setting 1 for Cartpole.
Revised: However, the results without expert batch are marginally better than those of setting 1 for Cartpole.

Original: This is because an agent of Cartpole only moves near the initial position, and thus there is no need for an expert batch to discover the dynamics of the generative model.
Revised: This is because an agent of Cartpole only moves near the initial position, and thus there is no need for an expert batch to discover the dynamics of the generative model.

Original: • Epistemic Value : We found that the epistemic value in the EFE term does not signiﬁcantly impact the training process.
Revised: Epistemic Value. We found that the epistemic value in the EFE term does not signiﬁcantly impact the training process.

Original: Comparing setting 1 (blue line) and setting 4 (green line), the results were similar regardless of whether the epistemic value was used.
Revised: Comparing setting 1 (blue line) and setting 4 (green line), the results were similar regardless of whether the epistemic value was used.

Original: In Acrobot and MountainCar, standard deviations were marginally smaller, but there were no signiﬁcant differences between them.
Revised: In Acrobot and MountainCar, standard deviations were marginally smaller, but there were no signiﬁcant differences between them.

Original: In the result of CartPole-v1, we found that setting 4 with no epistemic value term learned faster than our proposed setting 1 at the beginning of the training process.
Revised: In the result of CartPole-v1, we found that setting 4 with no epistemic value term learned faster than our proposed setting 1 at the beginning of the training process.

Original: We deduce that this initial performance drop is due to the instability of the epistemic term.
Revised: We deduce that this initial performance drop is due to the instability of the epistemic term.

Original: At the beginning of the training process, the generative model is not learned, and thus the related epistemic term becomes unstable.
Revised: At the beginning of the training process, the generative model is not learned, and thus the related epistemic term becomes unstable.

Original: We leave this issue to a future study.
Revised: We leave this issue to a future study.

Revised: 6. PPL AND INVERSE RL ALGORITHMS

Revised: Second, we check that our proposed PPL is compatible with traditional inverse RL algorithms.

Revised: We compared PPL with behavioral cloning (BC, Pomerleau (1989)) and maximum entropy inverse RL (MaxEnt, Ziebart et al.

Revised: as benchmark models.

Original: C ONCLUSION
Revised: We use setting 1 in Table 1 as our proposed PPL here, and we test on MountainCar-v0 and CartPole-v1. Note that BC does not need to interact with the environment and the state space was discretized to use original MaxEnt algorithm. We also tried to run the experiment on Acrobot-v1 for PPL and other benchmarks, but we failed to make the agent learn with MaxEnt.

Revised: A discretized state space for MaxEnt becomes larger exponentially to its state dimension.

Revised: We think it is due to a larger dimension of its state space compared to the others.

Revised: Therefore, we only report that PPL and BC give similar results to Acrobot-v1.

Revised: We veriﬁed that our method PPL gives compatible results on the MountainCar-v0 and CartPolev1.

Revised: Compared to MaxEnt, PPL shows better results than MaxEnt on both environments.

Revised: Note that

Revised: MaxEnt needs much more episodes to converge.

Revised: Also, PPL obtained almost similar mean rewards to BC on MountainCar-v0, whereas BC gives better results than PPL on CartPole-v1.

Original: In this paper, we introduced the use of active inference from the perspective of RL.
Revised: In this paper we introduced the use of active inference from the perspective of RL.

Original: Although active inference emerged from the Bayesian model of cognitive process, we show that the concepts of active inference, especially for EFE, are highly related to RL using the bootstrapping method.
Revised: Although active inference emerged from the Bayesian model of cognitive process, we show that the concepts of active inference, especially for EFE, are highly related to RL using the bootstrapping method.

Original: The only difference is that, the value function of RL is based on a reward, while active inference is based on the prior preference.
Revised: The only difference is that, the value function of RL is based on a reward, while active inference is based on the prior preference.

Original: We also show that active inference can provide insights to solve the inverse RL problems.
Revised: We also show that active inference can provide insights to solve the inverse RL problems.

Original: Using expert simulations, an agent can learn a local prior preference, which is more effective than the global preference.
Revised: Using expert simulations, an agent can learn a local prior preference, which is more effective than the global preference.

Original: Furthermore, our proposed active inference based reward with a prior preference and a generative model makes the previous IRL problems free from an ill-posed state.
Revised: Furthermore, our proposed active inference based reward with a prior preference and a generative model makes the previous invser RL problems free from an ill-posed state.

Original: Our work on active inference is complementary to RL because it can be applied to modelbased RL for the design of reward and model-free RL for learning of generative models.
Revised: Our work on active inference is complementary to RL because it can be applied to model-based RL for the design of reward and model-free RL for learning of generative models.