Exploration and preference satisfaction trade-oﬀ in reward-free learning
Abstract
Biological agents have meaningful interactions with their environment despite the absence of immediate reward signals.
In such instances, the agent can learn preferred modes of behaviour that lead to predictable states – necessary for survival.
In this paper, we pursue the notion that this learnt behaviour can be a consequence of reward-free preference learning that ensures an appropriate trade-oﬀ between exploration and preference satisfaction.
For this, we introduce a model-based Bayesian agent equipped with a preference learning mechanism ( pepper ) using conjugate priors.
These conjugate priors are used to augment the expected free energy planner for learning preferences over states (or outcomes) across time.
Importantly, Pepper enables the agent to learn preferences that encourage adaptive behaviour at test time.
We illustrate this in the OpenAI Gym FrozenLake and the 3D mini-world environments – with and without volatility.
Given a constant environment, these agents learn conﬁdent (i.e., precise) preferences and act to satisfy them.
Conversely, in a volatile setting, perpetual preference uncertainty maintains exploratory behaviour.
Our experiments suggest that learnable (rewardfree) preferences entail a trade-oﬀ between exploration and preference satisfaction.
Pepper oﬀers a straightforward framework suitable for designing adaptive agents, when reward functions cannot be predeﬁned as in real environments.
1. Introduction
Extrinsic rewards are not necessary to characterise an agent’s interaction with its environment.
For example, humans have been shown to rely on intrinsic motivation [47, 43, 6, 72], that can adequately regulate behaviour 1 .
Consequently, in the absence of immediate rewards, there is a preferred exchange with the environment [2, 53, 15], that can be updated under changing circumstances.
Interestingly, this can result in accruing preferences – and habits – that may be at odds with objective goals, e.g., kleptomania.
In this paper, we demonstrate that this kind of behaviour can be a consequence of reward-free preference learning that encourages self-evidencing[29] and maintains an appropriate arbitration between exploration and preference satisfaction.
In brief, we will see that agents learn to explore or exploit, depending upon the predictability of environmental contingencies.
Preference satisfaction subsumes homeostatic (extrinsic) motivations that encourage individuals to maintain some ’preferred’ behaviour [43] and resist eﬀects of perturbations (external or otherwise).
Generally, these refer to base needs that can be satisﬁed, e.g., going to sleep or eating food.
Conversely, exploration involves heterostatic (intrinsic) motivations that distract the agent from its homeostatic imperatives, e.g., noveltyseeking behaviour.
Exploratory behaviours would include trying a new hobby or taking a diﬀerent route to work.
This kind of exploration is distinct from random behaviour because it depends upon what the agent does not know.
Interestingly, over time, exploration can become the primary mode of behaviour if exploration satisﬁes the agent’s (learnt) preferences when dealing with an uncertain environment.
Our work is based on the notion that an adaptive agent learns the preferences that best reﬂect its environment.
To this end, we present pepper ; a preference learning mechanism that can accumulate preferences over states (or outcomes) using conjugate priors – given a model- based Bayesian agent.
Here, we instantiate Pepper as a deep active inference agent [17, 69, 10, 68, 14], maximising the evidence lower bound (or minimising the free energy) during training, and optimising the expected free energy (or free energy of future trajectories) for planning [44, 40].
However, it would be straightforward to add Pepper to other Bayesian reinforcement learning (RL) agents instead e.g., MaxEnt[20], Max[57] or Dreamer[25], etc.
Active inference was chosen deliberately to leverage the expected free energy (EFE) as a planning objective that captures the imperative to maximise: a) intrinsic value – from interactions with the environment – about latent states, and b) extrinsic value, namely, realising prior preferences over outcomes.
Moreover, active inference’s Bayesian formulation provides a natural way to introduce conjugate priors necessary for amortised learning of preferences over states (or outcomes) – as previously shown in a simpliﬁed setting for outcome preference learning[49].
Brieﬂy, Pepper comprises a two-step procedure which occurs after the (generative) model of the agent is optimised for the environment (i.e., training time – see Fig. 1).
The ﬁrst step consists of short episodes of direct exchange with the environment, where a history of observations and latent state representations are retained.
Once each episode ﬁnishes, the second step involves updating prior preferences based on the history using simple update rules (see Section 4).
Importantly, this means that agent can learn (diﬀerent) preferences that encourage adaptive behaviour at test.
The key contributions of this work are:
• We present a simple, and ﬂexible, preference learning mechanism (Pepper) to augment the planning objective (i.e., EFE) for learning (state or outcome) preferences using deep learning.
• Pepper is reward-free at train and test time.
This is achieved by casting rewards as a random variable in our generative model; equivalent to any other observation.
• Adaptive behaviour is conceptualised as a trade-oﬀ between exploration and preference satisfaction.
In what follows, we review the related literature.
Next, we introduce the problem setting and pepper (the preference learning mechanism).
We then evaluate the diﬀerent types of preferences learnt during test time, and how they engender an appropriate trade-oﬀ between exploration and preference satisfaction.
Finally, we discuss the potential implications of this work.
2. Related work
RL is regarded as a suitable framework for building artiﬁcial agents.
However, by deﬁnition, it relies on a reward signal to reinforce agent behaviour [66].
In reality, agents do not operate in a problem-solving setting, where a “critic” may not be readily available to provide immediate rewards [5, 59, 60].
Without taskspeciﬁc reward signal (also called extrinsic reward), the agent is driven by intrinsic motivations that promote exploration, play and curiosity [47, 58, 59].
Over the years, a variety of intrinsic motivation methods have been proposed, largely focusing on exploration, based on information gain [30, 64], prediction error [1, 45, 61], novelty search [39, 56], curiosity [50, 51], entropy [21, 38], or empowerment [36, 42].
Lately, through the popularisation of self-supervised learning (SSL) methods [22, 41], the deep RL community has turned its attention to self-supervised reinforcement learning.
Auxiliary tasks or rewards [31] are used – in the absence of any extrinsic rewards during train time – to train intrinsically motivated agents for representation learning [70, 19, 35, 65] or generative model learning [57, 55, 3].
Recent work [32, 71] has focused on theoretical properties of reward-free Reinforcement Learning.
However, the ultimate goal of such methods is to yield easily transferable representations to be exploited upon introduction of a task.
Our approach diﬀers in several ways.
First, we formulate intrinsic motivation using the Expected Free
Energy [17, 33] and focus on investigating the behaviour of intrinsically motivated agents.
Now exploration is simply an emergent behaviour of the planning objective and not a mechanism for improving future task performance.
Second, Pepper can be used to learn preferences over both states and outcomes in a deep learning setting – extending previous formulations to high-dimensional spaces [49].
Conceptually, openended learning [52, 62, 63], where agents are responsible for never-ending learning opportunities, is closest to our formulation.
However, Pepper extends this scenario to show how agents can trade-oﬀ between actively looking for opportunities to learn but also enjoy moments of preference satisfaction.
3. Problem setting
We consider a world that can be represented as a discrete-time Markov decision process (MDP), formally deﬁned as a tuple of ﬁnite sets ( S , Π , Ω , P , R ), such that: s ∈ S is a particular latent state, o ∈ Ω is a particular image observation, r ∈ R is a particular reward observation, and P is a set of transition probabilities.
Notice, we cast the reward function as another random variable – no diﬀerent to an image observation.
Further, π ∈ Π where π = { a 1 , a 2 , ..., a T } is a policy (i.e., action trajectory) and Π a ﬁnite set of all possible policies up to a given time horizon T ∈ N + and
T = { 0 , .., t, .., τ, T } a ﬁnite set which stands for discrete time; t the current time and τ some future time.
In short, we do not assume an optimal state-action policy but consider sequential policy optimisation inherent in active inference.
Accordingly, the agent’s generative model is deﬁned as a probability density, P θ ( o, r, s, π ), parameterised by θ (Fig. 1).
The generative model is instantiated as a Recurrent
State-Space Model (RSSM) [24, 23, 25] 2 where a history of observations ( o 0 , o 1 , .., o t ) and actions ( a 0 , a 1 , .., a t ) are mapped to a sequence of deterministic states h t .
Using these, distributions over the latent states – both prior and posterior – can be attained.
Formally, this consists of the following: 1) a GRU [12] based deterministic recurrent model, h t = f
[Equation]
( h <t , s <t , π <t ); 2) latent state posterior, Q φ ( s t | h t , o t ) ∼ Cat , and prior,
P ( s ) ∼ Cat ( D ); 3) transition model, P θ ( s t | h t ) ∼ Cat ; 3) image predictor (or emission model), Q φ ( o t | h t , s t ) ∼
Bernoulli ; and 4) reward model, Q φ ( r t | h t , s t ), and prior, P ( r ) ∼ Cat ( C ); where Q φ ( · ) denotes the approximate distribution, parameterised by φ .
3.1. Learning the generative model
To learn the generative model the evidence lower bound ( ELBO ) of the likelihood p ( o 1: T , r 1: T | π ) [24] or equivalently, the variational free energy [16, 14, 49] was optimised:
[Equation]
where, D KL denotes the Kullback–Leibler divergence.
Practically, this entails using trajectories generated under a random policy.
See Appendix A for ELBO implementation details.
4. Pepper: preference learning mechanism
After learning the generative model, we substitute the planning objective with the expected free energy.
At time-step t and for a time horizon up to time T , the expected free energy (EFE) is [17, 14]:
[Equation]
where
[Equation]
= Q
[Equation]
[Equation]
and
[Equation]
P
[Equation]
o
[Equation]
) Q ( s τ | o τ ) P ( θ | s τ , o τ , π ).
This is an appropriate planning objective because it: 1) is analogous to the expectation of the ELBO (Eq.1) under the predictive posterior P θ ( o t | s t , π ) and 2) can be decomposed into extrinsic and intrinsic value without any additional terms [13, 40].
Accordingly, in the absence of learnt preferences – or whilst learning them – intrinsic motivation contextualises agent’s interactions with the environment environment in a way that depends upon its posterior beliefs about latent environmental states [4, 47].
Actions are selected by sampling from distribution P ( π ) = arg max( − G ( π )).
We Pepper this planning objective with conjugate priors to allow for preference learning over time.
4.1. Learning preferences using conjugate priors
A natural way to learn preferences is to extend the agent’s generative model with conjugate priors over prior beliefs (i.e., hyper-priors) for each appropriate probability distributions, that are learnt over time [17, 49, 13].
Generally, for closed-form updates any exponential family would be appropriate [46].
Since our distributions of interest, latent state and rewards are
Categorical, we used the Dirichlet distribution as the conjugate prior.
For the latent state, P ( s ) ∼ Cat ( D ), this is deﬁned as:
[Equation]
where, (cid:122) is the digamma function, d ∼ R + and same parameterisation holds for P ( r ) ∼ Cat ( C ).
The posteriors for the Dirichlet hyper–parameters are evaluated by updating the prior using the following rule d i,j + α ∗
s i,j where s i,j are the observations for that particular category and α the learning rate.
These estimates can be treated as pseudo–counts, and the ensuing learning procedure is reminiscent of Hebbian plasticity – see [17, 48] for discussion.
In eﬀect, this allows the agent to accumulate contingencies and learn about what it prefers – either via outcomes or states.
4.1.1. Augmented expected free energy
To incorporate preference over both states and reward outcomes, we use two distinct EFE decompositions.
The ﬁrst one instantiates preference learning over reward outcomes, as presented below for a single time instance τ [54, 14]:
[Equation]
where, P ( r | C ) is the probability of a particular reward outcome given (learnt) prior preferences ( C ).
The second decomposition incorporates preferences over latent states:
[Equation]
where, P ( s | D ) is the probability of a particular state given (learnt) prior preferences ( D ).
These prior distributions are read as ‘preferences’ in active inference [49] – in the sense they are the outcomes the agent expects its plans to secure.
For both formulations, we drop the conditioning on policy when learning of preferences and the requisite probability is calculated using Thompson sampling.
This entails sampling from the prior Dirichlet distribution and estimating the likelihood.
Additionally, two of the three terms that constitute the expected free energy cannot be easily computed as written in
Eq.5 & Eq.6.
To ﬁnesse their computation, we rearrange these expressions and use deep ensembles [37] to render these expressions tractable.
See Appendix A for implementation details.
Input :
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
Reward model
Initialise uniform Dirichlet prior over P ( r ) or P ( s ) /
* prior preference being learnt */ learning rate α for each episode e do reset environment and collect initial observations ( o 0 or r 0 ) for each time step t do compute s po ∼ Q φ ( s t | h t , o t ) or s pr ∼ Q φ ( s t | h t ) compute G (Eq.5 or Eq.6) using (learnt) priors, observed and predicted posteriors a t ← arg max( − G ( π )) execute a t and receive o or r o t +1 ← o and r t +1 ← r endif reward preference learning then
c i [Equation] ←
[Equation]
i,
[Equation]
/* Update rule for dir ( c ) */ else if state preference learning then d ij,
[Equation]
← d
[Equation]
[Equation]
+ [Equation] [Equation] /* Update rule for dir ( d ) */ end
4.1.2. Pepper
Pepper comprises a double loop during preference learning.
The ﬁrst loop is across time-steps when the agent interacts with the environment.
This loop stores information about what happened (including observations, rewards, posterior, prior, etc).
The second loop, evolving at a slower timescale, is across episodes and entails preference learning.
Speciﬁcally, once the interaction with the environment ends, the agent updates the prior
Dirichlet distribution (over preferences) using the data gathered during the episode.
In the subsequent time- steps the updated preferences are used to select the next action (via their inﬂuence on expected free energy).
The data gathered during this episode are used to update preferences.
In turn, these preferences are used to select actions during the next episode, and so on.
This demonstrates a bi-directional ﬂow between the two loops: information from environment interactions inﬂuences preference learning and the learnt preferences inﬂuence environment interaction in the subsequent episode.
The Pepper preference learning procedure is summarised in Algorithm.4.1.1.
5. Results
Here, we present two sets of numerical experiments that underwrite the face validity of pepper in two and three-dimensional environments, respectively.
5.1. FrozenLake
We used a variation of the OpenAI Gym [8] 3 FrozenLake environment to: 1) evaluate diﬀerent behaviours acquired (at test time) when either P ( s ) or P ( r ) was learnt, 2) qualify how preferences can evolve as a result of environmental volatility, and 3) quantify the trade-oﬀ between exploration and preference satisfaction.
The agent in the original FrozenLake formulation is tasked with navigating a grid world comprised of frozen, hole and goal tiles, using 4 actions (left, right, down or up).
The agent receives a reward of 10 upon reaching the goal and a penalty − 0 .
25 upon moving to the hole.
To test preference learning, we included a sub-goal tile and removed the reward signal (Fig.2A).
In other words, although the preference learning agent can diﬀerentiate between tile categories – given its generative model – it receives no extrinsic signal from the environment.
Here, we simulated a volatile environment by switching the FrozenLake tile conﬁguration every K steps and initialising the agent in a diﬀerent location at the start of each episode.
This furnished an appropriate test-bed to assess how much volatility was necessary to induce exploratory behaviour and shifts in learnt prior preference.
5.2. TileWorld
We extended the FrozenLake environment in the miniworld framework [11] 4 to three dimensions to test the generalisation and scalability of preference learning, when operating in a 3D visual world (Fig.2C).
In this task, the agent moves around in a small room with grey walls, frozen and goal tiles on the ﬂoor.
The agent spawns in a random location and receives pixel observations (32x32 pixels with RGB channels) and a scalar value (reward) containing some information regarding the tile its currently on (1 for red tiles, 2 for green tiles).
Additionally, we introduce environmental volatility by changing the ﬂoor tiles to a random map and back to the original map every K steps.
Alternating between the original and a random map every K step is important to promote exploratory, novelty-seeking behaviour.
In the experiments that follow, we test agent behaviour in the two environments, with and without volatility.
The Dirichlet distribution for either prior preference distribution, P ( s ) or P ( r ), was initialised as 1 (i.e., uniform preferences).
Trained network weights, optimised Eq.1 using ADAM [34], were frozen during these experiments.
Therefore, behavioural diﬀerences are a direct consequence of pepper that induces diﬀerences in estimation of the EFE.
See Appendix B for architecture and training details for each environment.
5.3. Learnt preferences
State preferences Unsurprisingly, preference learning over latent states in the FrozenLake environment revealed two types of behaviours: exploration and preference satisfaction (Fig.3A & 4A)).
Here, preference satisfaction entailed restricted movement within a small section of the grid with gradual accumulation of prior preferences (see examples in the Appendix C.1).
This speaks to the self-evidencing nature of Pepper.
That is as the Pepper agent sees similar observations across episodes it grows increasingly conﬁdent (via increased precision over the prior preference) that these are the outcomes it prefers.
Conversely, exploratory behaviour is evident in a volatile setting (Fig.3A & 4A)), as gradual preference accumulation entails encoding of previously unseen states (see examples in Appendix C.1).
Reward preferences Reward preference learning revealed subtle diﬀerences in preferences (Fig.3B), where certain agents preferred sub-goal tiles more than neutral tiles.
All pepper agents were able to immediately maximise their marginal likelihood 5 over the reward (Fig.3B).
However, we did not observe clear diﬀerences in preference learning as the environment context shifted from non-volatile (i.e., map change) to increasingly volatile (map changed every time step).
We consider this to be consequence of the sparse categories over the reward distribution, and the large percentage of map being taken up by the neutral tiles.
This meant preference accumulation was biased in favour of the neutral tile (see examples in Appendix C.1).
5.4. Exploration and preference satisfaction trade-oﬀ
We evaluated the exploration and preference satisfaction trade-oﬀ using Hausdorﬀ distance (Fig.4) [7].
This is an appropriate metric, which calculates the maximum distance of the agents position in a particular trajectory to the nearest position taken in another trajectory.
Accordingly, a high Hausdorﬀ distance denotes increased exploration, since trajectories observed across episodes diﬀer from one other.
Whereas, a low distance entails prior preference satisfaction as agents repeat trajectories across episodes.
Using this metric, an inverted u-shaped association (Fig.4A), between volatility in the environment and preference satisfaction, is observed for preference learning over the states.
Here, 50% volatility in the environment, shifts the pepper agents behaviour from satisfying preferences to becoming exploratory, when faced with an uncertain environment (and inability to predict the future).
Agents in this setting (with the highest Hausdorﬀ distance) tend to pursue long paths from the initial location.
(Fig.3A).
Interestingly, at 100% environmental volatility, the Pepper agents behaviour shifts back to satisfying its preferences.
These agents learn bi-modal preferences over the latent states.
Irregardless of how the map changes they move directly to either location given the initial position (see trajectories in Appendix C.2).
This ability to disregard random, noisy information with continuous map changes (analogous to the noisy-TV setting introduced in [9]) highlights a motivation beyond random exploration under state preference learning.
For reward learning, exploration is also instantiated with increased volatility in the environment.
Yet, complete volatility does not trigger a deﬁnitive shift back to preference satisfaction.
Additionally, the expected free energy agent, without preference learning capacity, also exhibits a shift in behaviour as the environment becomes volatile.
The exploratory behaviour here is driven exclusively by an imperative to resolve state uncertainty, (Eq.5b): i.e., the mutual information between the agent’s beliefs about its latent state representation of the world, before and after making a new observation.
5.5. Preference learning in the volatile TileWorld
Preference learning agents evinced a strong preference for looking at grey walls in the volatile TileWorld environment (Fig.5A).
This was consistently observed for both preference learning over latent states and rewards.
Importantly, when spawned in a location right next to the wall, these agents were happy to satisfy their preferences and not move (Fig.5B).
This is driven by three factors: fast preference accumulation over grey walls, a small number of state conﬁgurations and a generative model that is able to appropriately predict future trajectories (see example reconstructions and imagined roll-outs in the Appendix B.2).
However, volatility in the environment does inﬂuence the encoding of prior preferences – as evident from the observed state entropy ﬂuctuations across the episodes (Fig.5).
6. Concluding remarks
Summary: Pepper – the reward-free preference learning mechanism presented – provides a simple way to inﬂuence agent behaviour.
Although, unlike the RL formulation, we cast what is preferred to the agent instead of the environment ’designer’.
That is, the agent is responsible for interacting with the environment and over time developing preferences that it acts to satisfy without an extrinsic signal.
Our experiments revealed that rich category spaces are necessary for learning preferences that can establish distinct behavioural strategies – speciﬁcally, within a volatile setting.
Thus, future experiments looking to leverage pepper should provide a suitable category space to learn over.
Conjugate priors and Hebbian plasticity: We employed a simple learning strategy for accruing preferences, using conjugate priors.
This type of learning usually calls on associative or Hebbian plasticity [26], where synaptic eﬃcacy is reinforced by the simultaneous ﬁring of pre-and post-synaptic neurons.
For example, as more neutral tiles are observed, more evidence is accumulated in the synaptic connection to support the hypothesis that neutral tiles are preferentially observed.
Implicitly, pepper rests upon experiencedependent plasticity, i.e., strengthening of synaptic connections during inference.
These updates can have diﬀerent times scales, and experiential levels (please see Appendix D).
Additionally, our learning process is driven by synaptic plasticity that allows certain random variables ( s, r ) to expand in light of new experiences [18].
They are only updated after an exchange with the environment.
This separation of ’experiencing’ the world, and then ’updating’ model parameters is reminiscent of sleep; in which synaptic homeostasis resets model parameterisation, by encoding new synapses or removing redundant ones [27, 67, 28].
Removal of redundant model parameters is evaluated in experiments presented in Appendix D, where removal of accumulated Dirichlet parameters reveals consistently exploratory agents.
Limitations: Notably, having a reward-free formulation is both an advantage and limitation of our approach.
By not specifying an extrinsic reward function we forego control over the agent’s behaviour.
In other words, by removing the ability to manipulate agent preferences regarding what is considered rewarding leads to the removal of the only clear communication channel that can be used by a designer to control agent behaviour and/or deﬁne task goals.
Therefore, it would not be appropriate to use pepper in a setting where control of the agent’s behaviour is required e g., for autonomous vehicles.
And, if it were, precise preferences would have to be established under supervision.
A. Pepper implementation
Pepper was implemented as an extension to Dreamer
V2 [25] public implementation 6 .
Speciﬁcally, Dreamer’s generative model training loop was used, alongside a model predictive control (MPC) planner.
Therefore, the actor learning part of Dreamer was not incorporated, and the generative model was trained using
Plan2Explore [55].
Like Plan2Explore, an ensemble of image encoders were learnt and the “disagreement” of the encoders was used as an intrinsic reward during training.
This guides the agent to explore areas of the map that have high novelty and potentially high information gain, when acquiring a generative (i.e., world) model.
Unfortunately, replacing the amortised policy with this planner made the environment interaction (i.e., the acting loop) relatively slow.
We implemented the planner as described in Algorithm A.
Algorithm 2 Planner
Input : s t current state N Number of random action sequences to evaluate
Initialise for i=1. . .
N do π i ∼ U
* Random action */ score i ← 0 for τ = t . . .
H do score i ← score i − G
( π i , τ ) /*
Updated according Eq.5 or Eq.6 */ end end k ← arg max score Return π k
Upon training completion, we froze the generative model learnt weights and only allowed learning of prior preferences.
These (state or reward) preferences were updated after each episode, as described in Algorithm 4.1.1.
A.1. Evidence lower bound
The generative model was optimised using the ELBO formulation introduced in [23]:
L [Equation]
reconstruction (
[Equation]
[Equation]
s t | o t
[Equation]
s t −
[Equation]
π ))
k P θ ( s t | s t −
[Equation]
π )]
Q
[Equation]
dynamics (cid:3) .
A.2. Expected free energy for pepper
We implemented EFE using the parameterisations introduced in [14] and adapted for [25]:
• Term 5a was modelled as a categorical likelihood model using normalised Dirichlet counts
• Term 5b was computed as the KL divergence between the prior ( s ∼ Q ( s τ | π )) and the posterior ( s ∼ P ( s τ | o τ , π )) states.
This could be computed analytically because the prior and posterior state distributions were modelled as Categorical distributions.
Here, the dependency on the policy π was accounted for by using the RNN hidden state h t summarising past actions and roll-outs.
• Term 6a was computed as the entropy of the observation model P ( o τ | s τ , π ).
Happily, the factorisation of the observation model – as independent Gaussian distributions – allowed us to calculate the entropy term in closed form.
• Term 6b was computed as the diﬀerence between log Q ( s τ | π ) and log P ( s τ | D ), where log Q ( s τ | π ) was approximated using a single sample from the prior model Q ( s τ | θ, π ).
Again, the dependency on π was substituted by h t .
• Terms 5c and 6c were more challenging to compute.
Like [14], we rearranged the expression to H ( o τ | s τ , θ, π ) − H
( o τ | s τ , π ).
This translates to I ( o τ ; θ | s τ , π ), and can be approximated using Deep Ensembles [37, 55] and calculating their variance Var θ [ E Q ( o τ | s τ , θ, π )].
Here, each ensemble component can be seen as a sample from the posterior
Q ( θ | s τ , π ).
Our experiments showed that usingcomponents was suﬃcient.
B. Experiments
FrozenLake
For these experiments, we simulated the agent in ﬁve distinct situations ranging from a non-volatile, static environment to a highly volatile one i.e., a diﬀerent FrozenLake map every step.
For all episodes in the static setting, the agent was initialised at a ﬁxed location with no changes to the FrozenLake map throughout that particular episode.
Conversely, agents operating in the volatile setting were initialised at a diﬀerent location each time.
Moreover, the FrozenLake map was also changed every N steps – given the desired volatility level.
For 100% volatility the map changed every step, 75% volatility corresponded to map changes every 10 steps, 50% volatility corresponded to map changes every 20 steps and 25% volatility corresponded to map changes every 40 steps.
Additionally, the generative model used for these experiments was trained in a volatile setting where the map changed every 5 steps (Table 1).
TileWorld For these experiments, we simulated the agent under two conditions (non-volatile and volatile).
For the non-volatile setting, the agent was initialised at a ﬁxed location with no changes to the TileWorld map throughout training and testing.
In the volatile setting, for every K step, we toggle alternate between a randomly sampled map and the original map.
This allowed us to simulate uncertain states that trigger exploratory behaviour.
The generative model used for these experiments was trained in a volatile setting where the map changed every 10 steps (Table 1).
[Table] [Table]
Map Every 1 , 10 ,
[Table]
[Table]
B.1. Computational requirements
Overall, our experiments required 1344 GPU hours.
Each GPU was a GeForce RTX 3090.
B.2. Image reconstruction and imagined roll-outs
For apt learning of preferences, the agent’s generative model must be able to accurately infer the current (and future) states of aﬀairs.
To evaluate this for our learnt generative models, we illustrate representative examples of reconstructions encoded by the pepper agents for a particular episode.
Fig.6 shows the reconstructions for FrozenLake, and Fig.7 for TileWorld.
The imagined roll-outs for the TileWorld environment are shown in Fig.8. [Figure] [Figure]
C. Behaviour under long-term preference learning C.1. Learnt preferences
We expected diﬀerences in learnt preferences to induce shifts in agent behaviour.
For example, agents who repeatedly accrued Dirichlet pseudo-counts for the same category would exhibit preference satisfying behaviour.
This would be due to high precision (or conﬁdence) over that particular category.
In contrast, an agent who accrued Dirichlet pseudo-counts for diﬀerent categories would exhibit exploratory behaviour given an imprecise (or low conﬁdence) distribution over the categories.
To illustrate how diﬀerent environment settings shaped preference learning we looked at the static and volatile setting where the FrozenLake map changed every step.
Fig.9 shows a representative example of state preference learning under these conditions, and
Fig.10 an example of reward preference learning.
We observed that state preferences learnt under a static setting were precise – denoted by the repeated pseudocount accumulation over category 25.
Conversely, for the volatile setting an imprecise state preference distribution was learnt (Fig.9).
Separately, we observed that the learnt reward preferences were precise – regardless of the setting.
We posit that this is a consequence of diﬀerences in the reward and state category space.
In other words, having a large number of state categories allowed distinct preferences to be learnt under static and volatile settings.
This is reﬂected in the qualitative diﬀerences seen between the two (Fig.9).
C.2. Agent trajectories
Next, we evaluated how disparate the agent trajectories were given the observed diﬀerences in preference accumulation (Figure 9 and 10).
For the static setting, we observed agents satisfying their preferences by restricting movement to a small patch in the FrozenLake.
This behaviour was observed consistently across all agents (i.e., diﬀerent seeds) and episodes.
We present a representative example in Fig.11.
Separately, agents simulated in the volatile setting (where the map changed every step) learnt a bi-modal preference set (i.e., preferred to go to one of two locations in the FrozenLake).
Here, the location preference depended on the initial location i.e., if the agent was initialised in a tile close to the ﬁrst preferred location then it choose to go there.
However, the second location was preferred if the agent was initialised close to it.
We present a representative example in Fig.12.
Interestingly, this behaviour was observed in agents where the environment was 100% volatile, whereas agents operating in slightly less volatile settings continued exploring (Fig.4A).
Importantly, these agents were able to disregard, noisy information about the states from the environment.
To qualify this, we looked at how the variance between the posterior ( s ∼ Q φ ( s t | h t , o t )) and prior ( s ∼ Q φ ( s t | h t )) estimates diﬀered across the 50 episodes for these agents (Fig.13).
We observed that the posterior estimates had a greater variance across the 50 dimensions relative to the prior variance.
Given how these estimates are calculated, we postulate that diﬀerences in the variance were due to the change in the FrozenLake map that the agent ﬁnds itself in after it moved one step.
These high variances in the posterior estimate, under a highly volatile setting, induced a change in behaviour from exploratory to preference satisfaction.
D. Behaviour under short-term preference learning
We expected reduced preference learning timescales to inﬂuence the agent’s preferred behaviour.
To evaluate this, we consider a setting where the agent was equipped with a sliding preference window i e., after k steps the previous preferences were removed in favour of new ones.
To evaluate short-term preference learning, we considered state preferences with a sliding window ofepisodes (Table 3).
[Figure]
[Table]
, 20 , 40 , 50 steps
Reset Preference
Every 5 episodes
No. Agents 5 agents For this, we looked at the preferences learnt in the static and volatile (map changes every 10 steps) setting (Fig.14).
Predictably, we observed diﬀerences in the preference accumulation when the agents learnt short-term preferences regardless of the setting.
Explicitly, in the static setting the accrued preferences were ﬂexibly learnt and unlearnt over time e.g., categorywas slowly unlearnt in favour of 11 category.
Volatile conditions fostered perpetual preference uncertainty as accumulated Dirichlet pseudo-counts were repeatedly updated.
Therefore, we would expect these agents to exhibit exploratory behaviour compared to agents equipped with long-term preferences due to imprecise preference learning.
To quantify this behaviour, we projected the latent states onto the ﬁrst two components (ﬁtted using long-term state preferences simulation data).
The short-term simulation data only mapped onto a small space in Fig.3 latent space.
Furthermore, there was no clear separation between projected latent states across the the volatile and static settings.
This is reﬂected in the increased state entropy ( ∼ 0 .
5 nats), under both settings, as previously learnt preferences were removed (Fig.16).
Next, we considered how the exploration and preference satisfaction trade-oﬀ might vary when preferences were learnt over a short time horizon.
Using the Hausdorﬀ distance, we evaluated how the volatility in the environment changed the agent’s behaviour to either exploratory or satisfying preferences (Fig.16).
In contrast to the long-term state preference learning setting, we see a (slight) linear association between environment volatility and preference satisfaction.
This is consistent with our expectation that a slow removal of accumulated Dirichlet parameters engenders consistently exploratory agents.