M INIMAL V ALUE -E QUIVALENT P ARTIAL M ODELS FOR S CALABLE AND R OBUST P LANNING IN L IFELONG R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning models of the environment from pure interaction is often considered an essential component of building lifelong reinforcement learning agents.
However, the common practice in model-based reinforcement learning is to learn models that model every aspect of the agent’s environment, regardless of whether they are important in coming up with optimal decxisions or not.
In this paper, we argue that such models are not particularly well-suited for performing scalable and robust planning in lifelong reinforcement learning scenarios and we propose new kinds of models that only model the relevant aspects of the environment, which we call minimal value-equivalent partial models .
After providing the formal definitions of these models, we provide theoretical results demonstrating the scalability advantages of performing planning with such models and then perform experiments to empirically illustrate our theoretical results.
Finally, we provide some useful heuristics on how to learn these kinds of models with deep learning architectures and empirically demonstrate that models learned in such a way can allow for performing planning that is robust to distribution shifts and compounding model errors.
Overall, both our theoretical and empirical results suggest that minimal value-equivalent partial models can provide significant benefits to performing scalable and robust planning in lifelong reinforcement learning scenarios.
I NTRODUCTION
It has long been argued that in order for reinforcement learning (RL) agents to perform well in lifelong RL (LRL) scenarios, they should be able to learn a model of their environment, which allows for advanced computational abilities such as counterfactual reasoning and fast re-planning (Sutton & Barto, 2018;
Schaul et al., 2018; Sutton et al., 2022).
Even though this is a widely accepted view in the RL community, the question of what kinds of models would better suite for performing LRL still remains unanswered.
As LRL scenarios involve large environments with lots of irrelevant aspects and periodic or non-periodic distribution shifts, directly applying the ideas developed in the classical model-based RL literature (see e.g., Ch. 8 of Sutton & Barto, 2018) to these problems is likely to lead to catastrophic results in building scalable and robust lifelong learning agents.
Thus, there is a need to rethink some of the ideas developed in the classical model-based RL literature while developing new concepts and algorithms for performing model-based RL in LRL scenarios.
In this paper, we argue that one important idea to reconsider is whether if the agent’s model should model every aspect of its environment.
In classical model-based RL, the learned model is a model over every aspect of the environment.
However, due to the large state spaces of LRL environments, these types of models are likely to lead to serious problems in performing scalable model-based RL, i.e., in quickly learning a model and in quickly performing planning with the learned model to come up with an optimal policy.
Also, due to the inherent non-stationarity of LRL environments, these types of detailed models are likely to lead to models that overfit to the irrelevant aspects of the environment and cause serious problems in performing robust model-based RL, i.e., learning & planning with models that are robust to distributions shifts and compounding model errors.
To this end, we argue that models that only model the relevant aspects of the agent’s environment, which we call minimal value-equivalent partial models , would be better suited for performing model-based RL in LRL scenarios.
We first start by developing the theoretical underpinnings of how such models could be defined and studied in model-based RL.
Then, we provide theoretical results demonstrating the scalability advantages, i.e., the value and planning loss and computational and sample complexity advantages, of performing planning with minimal value-equivalent partial models and then perform several experiments to empirically illustrate these theoretical results.
Finally, we provide some useful heuristics on how to learn these kinds models with deep learning architectures and empirically demonstrate that models learned in such a way can allow for performing planning that is robust to distribution shifts and compounding model errors.
Overall, both our theoretical and empirical results suggest that minimal value-equivalent partial models can provide significant benefits to performing scalable and robust model-based RL in LRL scenarios.
We hope that our study will bring the community a step closer in building model-based RL agents that are able to perform well in LRL scenarios.
B ACKGROUND
Reinforcement Learning.
In RL (Sutton & Barto, 2018), an agent interacts with its environment through a sequence of actions to maximize its long-term cumulative reward.
Here, the environment is usually described as a Markov decision process (MDP) M ≡ ( S , A , P, R, γ ) , where S and A are the (finite) set of states and actions, P : S×A×S → [0 , 1] is the transition distribution, R : S×A → [0 , R max ] is the reward function, and γ ∈ [0 , 1) is the discount factor.
On the agent’s side, through the use of a perfect state encoder ϕ ∗ : S → F , every state s ∈ S can be represented, without any loss of information, as an n -dimensional feature vector f =
[ f 1 , f 2 , . .
, f n ] ⊤ ∈ F , which consists of n different features F = { f i } ni =
where f
i ∈ F i ∀ i ∈ { 1 , . . . , n } (also see Boutilier et al.
Note that as there is no loss of information, F contains all the possible features that are relevant in describing the states of the environment.
Thus, from the agent’s side, the MDP M can losslessly be represented as another MDP m ∗ =
( F , A , p ∗ , r ∗ , γ ) , where F and A are the (finite) set of feature vectors and actions, p ∗ : F × A × F → [0 , 1] and r ∗ :
F × A → [0 , R max ] are the transition distribution and reward function, and γ ∈ [0 , 1) is the discount factor.
For convenience, we take the agent’s view and refer to the environment as m ∗ throughout this study.
The goal of the agent is to learn a value estimator Q :
F × A → R that induces a policy π ∈ Π ≡ { π | π : F × A → [0 , 1] } , maximizing E π,p ∗ [ (cid:80) ∞ t =0 γ t r ∗ ( F t , A t ) | F 0 ] for all F 0 ∈ F .
Model-Based RL.
One of the prevalent ways of achieving this goal is through the use of modelbased RL methods in which there are two main phases: the learning and planning phases.
In the learning phase, the gathered experience is mainly used in learning an encoder ϕ : S → F and a model m ≡ ( p, r ) ∈ M ≡ { ( p, r ) | p : F × A × F → [0 , 1] , r : F × A → [0 , R max ] } , and optionally, the experience may also be used in improving the value estimator.
In the planning phase, the learned model m is then used either for solving for the fixed point of a system of Bellman equations (Bellman, 1957), or for simulating experience, either to be used alongside real experience in improving the value estimator, or just to be used in selecting actions at decision time (Alver &
Precup, 2022; Sutton & Barto, 2018).
Value-Equivalence.
One of the recent trends in model-based RL is to learn models that are specifically useful for value-based planning (see e.g., Silver et al., 2017; Schrittwieser et al., 2020), which has been recently formalized in several different ways through the studies of Grimm et al.
Inspired by these studies, we define a related form of value-equivalence as follows.
Let
V πm ∈ R |F| be the value vector of a policy π ∈ Π evaluated in model m , whose elements are defined ∀ f ∈ F {+as V πm ( f ) ≡ E π,p [ (cid:80) ∞ t =
+}
γ t r ( F t , A t ) | F 0 = f ] , and let V ∗ m ∈ R |F| be the optimal value vector in model m .
We say that a model m ∈ M is a value-equivalent (VE) model of the true environment m ∗ ∈ M if the following equality holds: [Equation]
[Equation]
where π ∗ m is an optimal policy obtained as a result of planning with model m .
M INIMAL V ALUE -E QUIVALENT P ARTIAL M ODELS
In classical model-based RL (Ch. 8 of Sutton & Barto, 2018), an agent learns a very detailed model of its environment that models every aspect of it, regardless of whether these aspects are relevant in the process of coming up with optimal decisions or not.
However, in LRL scenarios, where the agent is “small” and the environment is “vast” (Schaul et al., 2018), this approach is likely to be problematic as modeling every aspect of the environment becomes quite impractical.
Even if the agent overcomes its capacity limitations and manages to model every aspect, as we will demonstrate, these kinds of detailed models can lead to large planning losses and dramatically slowdown both the model-learning and planning processes.
And, as we will further demonstrate, detailed models can also be fragile to the distribution shifts in the environment and to the compounding model errors that happen during the unrollment of the learned model.
In order to overcome these challenges, we start by proposing new kinds of models that only model certain aspects, either relevant or irrelevant, of the agent’s environment.
For this, we first start by clarifying the notion of “aspect”: in this study, by “aspect”, we mean a feature of the environment f i ∈ F i that is learnable by the agent (see Sec. 2).
We are now ready to define partial models :
Definition 1 (Partial Models) .
Given a set of features F , let F P ⊂ F s.t.
| F P | < | F | .
Let F P be a space of feature vectors in which the feature vectors consist the features in F P .
We say that a model m P is a partial model of the true environment m ∗ ∈ M if it is defined over the feature vector space
F P , i.e., m P ∈ M P ≡ { ( p P , r P ) | p P : F P ×
A × F P → [0 , 1] , r P : F P × A → [0 , R max ] } .
According to Defn. 1, any model that only models certain features of the environment is a partial model of the environment m ∗ ∈ M .[-
-]
However, in order for a partial model to be useful, it should be able to model the relevant features of the environment that allow for achieving the task of interest.
In order to separate out the relevant features from the irrelevant ones, we define the relevant ones as:
Definition 2 (Relevant Features) .
Given a set of features F , let F R ⊂ F .
Let F R be a space of feature vectors in which the feature vectors consist of the features in F R .
We say that the features f i ∈ F R are relevant features of the task of interest if they are necessary and sufficient for defining a space of models M R ≡ { ( p R , r R ) | p R : F R ×
A × F R → [0 , 1] , r R : F R × A → [0 , R max ] } that contains value-equivalent models of the true environment m ∗ ∈ M .
Now that we have defined partial models and distinguished between the relevant and irrelevant features of the environment, we are ready to define an important class of partial models that at the very least model the relevant aspects of the environment:
Definition 3 (VE Partial Models) .
Given a set of features F , let F VEP ⊂ F s.t.
| F VEP | < | F | and F
R ⊆
F VEP .
Let F VEP be a space of feature vectors in which the feature vectors consist of the features in F VEP .
Let m VEP be a partial model that is defined over the feature vector space F VEP , i.e., m VEP
[Equation]
×A → [0 , R max ] } .
We say that m
VEP is a VE partial model of the true environment m ∗ ∈ M if it is a VE model of m ∗ , i.e.,
[Equation]
where π ∗ m VEP is an optimal policy obtained as a result of planning with model m VEP and Π VEP ≡ { π | π : F VEP ×
A → [0 , 1] } .
Although it is important to learn partial models that at the very least model the relevant aspects of the environment, as we will theoretically and empirically demonstrate, partial models are mostly beneficial when they only model the relevant aspects of the environment, i.e., when F VEP = F R .
We refer to these models as minimal VE partial models .
Note that minimal VE partial models are a special class of VE partial models, and VE partial models are a special class of partial models.
[Figure]
Illustrative Example.
As an illustration of the models defined above, let us start by considering the Squirrel’s World (SW) environment depicted in Fig.
1, in which the squirrel’s (the agent) job is to navigate from cell E1 to cell E16 to pickup the nut without getting caught by the hawk that flies back and forth horizontally along row
C. At each time step, the squirrel receives as input an 5 × 16 image of the current state of the environment and then, through the use of a predefined state encoder, transforms this image into a feature vector that contains information regarding all aspects of the current state of the environment, i.e., the feature vector contains information on the current position of the squirrel, hawk and the cloud, the current direction of the hawk, the current wind direction in rows A and B and the current weather condition.
Based on this, the squirrel selects an action that either moves it to the left or right cell, or keeps it position fixed.
If the squirrel gets caught by the hawk or if it is out of time, it receives a reward of 0 and the episode terminates, and if the squirrel successfully navigates to the nut, it gets a reward of +10 and the episode terminates.
In this environment, as the hawk moves 5x the speed of the squirrel, a straightforward policy of always moving to the right will not get the squirrel to the nut.
Thus, the squirrel has to come up with non-trivial policies that take into account both the cells with bushes (see e.g., cells E2, E3), which allow for sheltering, and the position and direction of the hawk.
In this environment, examples of partial models can be a model that only models the cloud position and the wind direction for rows A and B, or a model that only models the weather condition and the hawk’s direction.
However, for a partial model to be VE or minimal VE, it has to model the relevant features for the tasks of interest which is reaching the nut.
In the SW environment, there are three relevant features: (i) the squirrel’s position, (ii) the hawk’s position, and (iii) the hawk’s direction, as the squirrel would have to have access to all three of these features to come up with optimal policies.
Thus, an example of a VE partial model can be a model that models both the three relevant features and the weather condition, and an example of a minimal VE partial model can be a model that only models the three relevant features.
T HEORETICAL R ESULTS
In this section, we first analyze the value and planning losses (Sec. 4.1) of VE partial models and then derive formal results demonstrating the computational and sample complexity benefits (Sec.
4.2) of using such models.
We then discuss scenarios where the VE partial model is a minimal one.
4. V ALUE AND P LANNING L OSS A NALYSES
We start our formal analysis by studying the value loss incurred due to planning with a VE partial model m VEP in place of the true environment m ∗ .
To simplify the analysis, we assume that the agent already has access to this model and does not need to learn it.
Theorem 1.
Let m VEP ∈ M VEP be a VE partial model of the true environment m ∗ ∈ M .
Then, the value loss between an optimal policy in m ∗ , π ∗ , and an optimal policy in m VEP , π ∗ m VEP is given by:
[Equation]
Due to space constraints, we defer all the proofs to App. A. Theorem 1 says that by planning with a (non-minimal or minimal) VE partial model, an agent would incur no value loss compared to planning with the true environment itself.
Next, we study the planning loss (Jiang et al., 2015) incurred due to planning with an approximate
VE partial model ˜ m VEP ∈ M VEP in place of the actual VE partial model m VEP ∈ M VEP .
Similar to
Jiang et al.
, we also consider the certainty-equivalence control setting in which the agent acts according to a policy that is optimal with respect to its current approximate model.
Theorem 2.
Let m VEP ∈ M VEP be a VE partial model of the true environment m ∗ ∈ M , and let ˜ m VEP ∈ M VEP be model that comprises of the reward function of m VEP and a transition distribution that is estimated from n samples for each ( f, a ) pair.
Let Π r VEP ≡ { π | ∃ p VEP s.t π is optimal in ( p VEP , r VEP ) } .
Then, certainty-equivalence planning with ˜ m VEP has planning loss:
[Equation]
with probability at least 1 − δ .
Theorem 2 implies that given a fixed amount of data, the upper bound of the planning loss of a VE partial model depends on both the size of its feature vector space, |F VEP | , and the size of its policy class being searched over by planning, | Π r VEP | .
1 This in turn implies that, given a fixed amount of data, compared to a regular model, a VE partial model is likely to have less planning loss and this loss is likely to be minimized when the VE partial model is a minimal one.
4. C OMPUTATIONAL AND S AMPLE C OMPLEXITY B ENEFITS
We now study the computational and sample complexity benefits of performing model-based RL with VE partial models.
Due to the well-established theoretical results around it, we choose to study these benefits in the context of value iteration (Bertsekas & Tsitsiklis, 1996).
However, we note that the implications of our results would apply to a wide variety of planning algorithms.
Starting with the computational complexity benefits, it is well-known that the computational complexity of performing a single step of value iteration with an arbitrary model m ∈ M is O ( |F| 2 |A| )
(Agarwal et al., 2022).
Thus, the computational complexity of performing a single step of value iteration with a VE partial model m VEP ∈ M VEP would be O ( |F VEP | 2 |A| ) .
This implies that compared to planning with regular models, planning with VE partial models would provide a significant computational complexity benefit and this benefit would be maximized when the model used for planning is a minimal VE partial model.
Moving on to the sample complexity benefits, previous studies of Kearns & Singh (1998); Kakade (2003); Azar et al.
(2012) have shown that the sample complexity of obtaining an ε estimation of the optimal action value function through the use of Q-value iteration (see Alg. 1) given access only to a generative model is in the order of the magnitude of the model’s state and action space.
Building on top of this result, we now study the sample complexity benefits of planning with approximate VE partial models that are obtained as a result of sampling generative VE partial models.
Theorem 3.
Let m VEP ∈ M VEP be a VE partial model of the true environment m ∗ ∈ M .
Let ˜ m VEP ∈ M VEP be the corresponding approximate VE partial model that has the same reward function as m VEP , but whose transition distribution is estimated by m calls to the generative model m VEP , where
[Equation]
and let Q k ˜ m VEP be the value returned by Q-value iteration at the k th epoch.
Then, with probability greater than 1 − δ , the following holds for all f ∈ F VEP and a ∈ A :
[Equation]
where k = log( ε (1 − γ )) log γ and Q ∗
m VEP is the optimal action value function in m VEP .
Theorem 3 implies that compared to a regular model, a VE partial model is likely to require less samples in obtaining an ε estimation of the optimal action value function through the use of Q-value iteration with a generative model, and the number of samples required is likely to be minimized when the VE partial model is a minimal one.
E XPERIMENTAL R ESULTS
We start this section by performing experiments to demonstrate the scalability advantages of minimal
VE partial models, which are illustrations of the theoretical results derived in Sec. 4, and then we perform experiments to demonstrate the robustness advantages of these models.
The details of our experiments can be found in App.
C.
Environments.
We perform experiments on both the SW environment (see Fig. 1) and on variations of the Two Rooms Dynamic Obstacles (2RDO) environment that are built on top of Minigrid (Chevalier-Boisvert et al., 2018)
(see Fig. 2), as these environments allow for designing controlled experiments that are helpful in answering the questions of interest to this study.
Some of the details of the SW environment are already presented in Sec.
and we refer the reader to App.
C for more details.
In the 2RDO environments, the agent, depicted by the red triangle, spawns in top-left of the top room and has to navigate to the green goal cell located in the bottom-right of the same room, regardless of the gaseous motions of the obstacles in the bottom room.
At each time step, the agent receives an image of the current state of the grid and then, through the use of a learned state encoder, transforms this image into a feature vector.
Based on this, the agent selects an action that either turns [Figure] it left or right, or moves it forward.
If the agent successfully navigates to the goal cell, it receives a reward of +1 and the episode terminates.
More details on the 2DRO environments can be found in
App. C as well.
5. S CALABILITY E XPERIMENTS
For our scalability experiments, we perform experiments with several non-VE ( m 1 , m 2 , m 3 ) and
VE ( m 4 , m 5 , m 6 ) partial models of both the deterministic and stochastic versions of the SW environment, referred to as Det-SW and Stoch-SW, respectively.
The details of these models can be found in Table 1.
For all of our experiments, we use value iteration as our planning algorithm.
[Figure]
Question 1.
Do minimal VE partial models allow for planning with no value loss?
In Sec. 4.1, we argued that by planning with a (non-minimal or minimal) VE partial model, an agent would incur no value loss compared to planning with the true environment itself.
To empirically verify this, we present the agent with a set of non-VE partial models m 1 , m 2 , m 3 and a minimal VE partial model m 4 , and compare the value losses on both the Det-SW and Stoch-SW environments.
Results are shown in Fig. 3a.
We can indeed see that while the VE partial model incurs no value loss, the non-VE ones do incur serious value losses.
Question 2.
Do minimal VE partial models allow for planning with less planning loss?
In Sec. 4.1, we argued that given a fixed amount of data, compared to a regular model, a VE partial model is likely to incur less planning loss, and this loss is likely to be minimized when the VE partial model is a minimal one.
For empirical verification, we compare the planning losses of a minimal
VE partial model m 4 , two (non-minimal) VE partial models m 5 and m 6 , and a regular model m 7 , across dataset sizes of 3 , 5 , 10 and 20 , which corresponds to the number of samples for each ( f, a ) pair, on the Stoch-SW environment.
Results in Fig.
3b show that, as expected, VE partial models indeed incur less planning losses than regular models, and the minimal VE partial model incurs the least planning loss.
Question 3.
Do minimal VE partial models provide computational complexity benefits?
In Sec.
4.2, we argued that compared to regular models, planning with VE partial models would provide a significant computational complexity benefit and this benefit would be maximized when the model used for planning is a minimal VE partial model.
To empirically verify this, we present the agent with a minimal VE partial model m 4 , two VE partial models m 5 and m 6 , and a regular model m 7 of the Det-SW environment, and compare the average time it takes to perform a single step of value iteration for each of these models.
Results are shown in Fig.
3c.
As can be seen, planning with
VE partial models indeed provides significant computational complexity benefits, and this benefit is maximized when the VE partial model is a minimal one.
Question 4.
Do minimal VE partial models provide sample complexity benefits?
[Figure]
In Sec.
4.2, we argued that compared to regular models, planning with VE partial models is likely to provide a sample complexity benefit and this benefit is likely to be maximized when the model that is used for planning is a minimal VE partial model.
For empirical verification, we present the agent with a minimal
VE partial model m 4 and with a regular model m 7 as generative models, and compare the sample efficiencies, as a result of performing Qvalue iteration, on the Det-SW and Stoch-SW environments.
In these experiments, after every episodic interaction, the agent updates its model with the collected trajectory, and then performs Q-value iteration until convergence.
Results in Fig.
4 show that, as expected, planning with minimal VE partial models indeed provides significant sample efficiency benefits compared to planning with regular models.
5. R OBUSTNESS E XPERIMENTS
For our robustness experiments, we perform experiments on variations of the 2RDO environment with grid sizes of 8x8 and 16x16.
For convenience, we will refer to these environments with their grid size followed by their obstacle type.
For example.
we will refer to the 8x8 2DRO environment with red balls as 8x8 RedBalls (see Fig. 2).
For all of our experiments, we use the straightforward decision-time planning algorithm of Zhao et al.
(2021) (see Alg. 2) whose details can be found in App.
C.
As this algorithm makes use of neural networks, before moving on to the robustness experiments, we try to answer the following question.
Question 5.
How to learn minimal VE partial models with deep learning architectures?
So far, for illustration purposes, we have only performed experiments in which we had a direct control over the features of the agent’s model (see the models in Table 1).
However, in realistic scenarios, the agent would have to come up on its own with a set of features to build a model of the only relevant aspects of its environment.
A very popular way of letting the agent come up with its own features is to use neural networks in the representation of the agent’s encoder, value estimator and model, and then to train it end-to-end on the environment of interest.
However, in order for the agent to come up with only the relevant features, it has to be trained with the right inductive biases.
Even though finding the right inductive biases to train a model-free or model-based RL agent is still an open problem in the representation learning literature (Bengio et al., 2013), in this study, we propose two inductive biases that are likely to guide the agent in coming up with only the relevant features.
The first one is to only let the value estimator shape the encoder and prevent the model from doing so (see Fig. 7).
In this way, the agent can be guided in learning the features that are relevant for predicting the right values in the environment.
And, the second one is to train the agent across a variety of environments in which the irrelevant aspects keep changing and the relevant ones stay the same.
In this way, the agent can be guided in not learning the irrelevant aspects of the environment.
In order to test the usefulness of these two inductive biases in coming up with only the relevant features of the environment, we compare three different agents: (i) a regular agent, A REG , that was trained on the 8x8 BlueBalls environment and whose encoder was jointly shaped by its value estimator and model, (ii) an agent, A VES , that was again trained on the 8x8 BlueBalls environment, but whose encoder was only shaped by its value estimator, and (iii) an agent, A VES + ME , that was trained on the 8x8 BlueBalls, GreenBalls, PurpleBalls and YellowBalls environments and whose encoder was only shaped by its value estimator.
We compare these agents on the 8x8 BlueBalls and
NoObstacles environments.
If the agent is successful in coming up with only the relevant features of the environment, which are the positions of the agent and the goal, and not the positions and motions of the obstacles, we would expect it to perform similarly on the 8x8 BlueBalls and 8x
NoObstacles environments.
Results are shown in Fig.
5a & 5b.
As can be seen, even though all of the agents perform well on the 8x8 BlueBalls environment, the A REG agent completely fails on the 8x8 NoObstacles environment, demonstrating that without the necessary inductive biases an agent is [Figure] [Figure] not capable of coming up with only the relevant features itself.
We can also see that the A VES agent achieves a better performance than the A REG agent and that the A VES + ME agent achieves an even better performance than the A VES agent, demonstrating the usefulness of our proposed inductive biases in inducing models that display the behavior of minimal VE partial models.
In order to test the scalability of our results, we have also performed the same experiments with 16x16 versions of the environments.
As can be seen in Fig.
5c & 5d, we obtain similar results.
Question 6.
Can minimal VE partial models be useful for performing robust transfer?
As minimal VE partial models only model the relevant aspects of the environment, we would expect them to be robust to the distribution shifts happening in the irrelevant aspects of the environment.
In order to test this, we compare the performances of the A REG , A VES and A VES + ME agents on the 8x8 and 16x16 RedBalls, GreyBalls, RedBoxes and GreyBoxes environments.
Results are shown in
Fig.
5e-5l.
As can be seen, while the A REG agent fails and the A VES agent only shows signs of robust transfer, the A VES + ME agent is able to perform robust transfer without any problem.
These results illustrate the ability of minimal VE partial models in performing robust transfer.
Question 7.
Are minimal VE partial models more robust to compounding model errors?
As minimal VE partial models only model the relevant aspects of the environment, compared to regular models, we would expect them to be less susceptible to compounding model errors during planning.
In order to test this, we compare the performances of the A REG and A VES + ME agents with search budgets of 20, 40 and 80 on the 16x16 BlueBalls environment.
Note that this environment has been seen before by both of the agents.
Results in Fig.
6 show that while the performance of A REG agent drops significantly with the increase in the search budget, the performance of the
A VES + ME agent stays close to optimal, demonstrating the robustness of minimal VE partial models to compounding model errors.
R ELATED W ORK
[Figure]
Partial Models.
In the context of RL, the initial studies of partial models can be dated back to the seminal study of Talvitie & Singh (2008) which proposes to learn several models of an uncontrolled dynamical systems that are partial at the observation level.
In contrast, we propose to learn a single and useful partial model of a controlled dynamical system that is partial at the feature level, which provides several advantages such as eliminating the question of how to combine the learned models, using them for control purposes, and making them compatible with function approximation.
Our work also has a very close connection to the study of Zhao et al.
(2021) which proposes a transformer-based deep model-based agent that dynamically attends to relevant parts of its state representation during planning.
However, our work differs in that we propose the general concept of partial models for LRL that is independent of the agent’s implementation details.
Lastly, another related line of research is the studies of Khetarpal et al.
(2020; 2021) on affordances which focus on building models that partial in the action space.
Our study is complementary to these studies in that they can still leverage (non-minimal or minimal) VE partial models to reduce the size of the feature space and further increase the benefits of performing model-based RL with partial models.
Value-Equivalence.
A recent trend in model-based RL is to learn models that are specifically useful for value-based planning (see e.g. Silver et al., 2017;
Oh et al., 2017; Farquhar et al., 2017;
Schrittwieser et al., 2020), which has recently been formalized through the studies of Grimm et al.
Even though our work also advocates the idea that models should be useful in valuebased planning, our work differs in that we also argue that the explicit partiality of the models can provide significant scalability and robustness benefits when performing model-based RL in LRL scenarios.
Planning in Learned Feature Spaces.
Even though there has been recent studies that study the effect of the introduction of the irrelevant features in the agent’s learned representation (Efroni et al., 2022a;b), our study differs in that we are mainly interested in LRL environments in which environment mostly consists of irrelevant features.
Our work is also different from the studies that learn models through self-supervised learning (see e.g., Sekar et al., 2020) in that we explicitly focus on the structure of the learned representation.
C ONCLUSION AND D ISCUSSION
In conclusion, in this study, we have introduced special types of models, called minimal VE partial models, that only model the relevant aspects of the environment and are particularly useful in LRL scenarios.
Our theoretical results suggest that these models can provide significant advantages in the value and planning losses that are incurred during planning and in the computational and sample complexity of planning.
Our empirical results (i) validate our theoretical results and show that these models can scale to large environments, and (ii) show that these models can be robust to distribution shifts and compounding model errors.
Overall, our findings suggest that minimal VE partial models can provide significant advantages in performing model-based RL in LRL scenarios.
One limitation of our work is that, rather than providing a principled method, we have only provided several heuristics for training deep RL agents that can come up with only the relevant features of the environment.
However, we note that this is mainly due to the lack of principled approaches in the representation learning literature, and we believe that this limitation can be overcomed with more principled approaches being introduced in the literature.
We hope to tackle this limitation in future work.
Another important limitation is that, due to the need to perform illustrative and controlled experiments, we have only performed experiments in the SW and 2RDO environments where there is just a single task and there is no sequence of tasks, requiring a model the same relevant features, that unfold over time.
However, experiments in more environments that have this sequential nature can be helpful in further validating the advantages of minimal VE partial models in LRL scenarios, which we also hope to tackle in future work.
A P ROOFS
Theorem 1.
Let m VEP ∈ M VEP be a VE partial model of the true environment m ∗ ∈ M .
Then, the value loss between an optimal policy in m ∗ , π ∗ , and an optimal policy in m VEP , π ∗ m VEP is given by:
[Equation]
Proof.
This result directly follows from Defn.
Recall that, according to Defn.
3, we have:
[Equation]
which implies:
[Equation]
Theorem 2.
Let m VEP ∈ M VEP be a VE partial model of the true environment m ∗ ∈ M , and let ˜ m VEP ∈ M VEP be model that comprises of the reward function of m VEP and a transition distribution that is estimated from n samples for each ( f, a ) pair.
Let Π r VEP ≡ { π | ∃ p VEP s.t π is optimal in ( p VEP , r VEP ) } .
Then, certainty-equivalence planning with ˜ m VEP has planning loss:
[Equation]
with probability at least 1 − δ .
Proof.
Similar to Jiang et al.
, we prove Theorem 2 with two lemmas:
Lemma 1 translates planning loss to value error, and Lemma 2 relates value error to a Bellman-residual-like quantity that has a uniform deviation bound which depends on | Π r VEP | .
Lemma 1.
For any ˜ m VEP =
(˜ p VEP , ˜ r VEP ) with ˜ r VEP bounded by [0 , R max ] ,
[Equation]
In particular
[Equation]
if ˜ r VEP = r VEP
[Equation]
we have
[Equation]
Proof.
∀ f ∈ F VEP ,
[Equation]
Eqn. 11 follows from taking the max over all feature vectors on both sides of the inequality and noticing that the set of all policies is a trivial superset of (cid:8) π ∗ m VEP , π ∗ ˜ m VEP (cid:9) .
If ˜ r VEP = r VEP , the bound can be tightened since (cid:8) π ∗ m VEP , π ∗ ˜ m VEP (cid:9)
∈
Π r VEP , and Eqn. 12 follows.
Lemma 2.
For any ˜ m VEP =
(˜ p VEP , ˜ r VEP ) with ˜ r VEP bounded by [0 , R max ] , ∀ π : F VEP → A ,
[Equation]
Proof.
Given any policy π , define action value functions such that Q 0 , Q 1 , . .
. , Q n , . . .
such that
Q 0 =
Q πm VEP , and
[Equation]
17) where V n − 1 ( f ) =
Q n − 1 ( f, π ( f )) .
Notice that
[Equation]
so
[Equation] [Equation]
Taking the limit of n → ∞ , Q n → Q π ˜ m VEP , and we have,
[Equation]
This completes the proof, noticing that Q 0 = Q πm VEP , V
= V πm VEP , and Q 1 ( f, a )
=
˜ r VEP ( f, a
[Equation]
+ γ ⟨ ˜ p VEP ( f, a, · ) , V πm VEP ⟩ .
From Eqn. 12 in Lemma 1 and Lemma 2, we have (cid:13)(cid:13)(cid:13)
[Equation]
[Equation] [Equation]
For any particular f , a , π tuple, according to Hoeffding’s inequality, ∀ t > 0 , p (cid:0)(cid:12)
[Equation]
(29) as ˜ r VEP ( f, a )+ γ ⟨ ˜ p VEP ( f, a, · ) , V πm VEP ⟩ is the average of i.i.d.
samples bounded in [0 , R max / (1 − γ )] , with mean Q πm VEP ( f, a ) .
To obtain a uniform bound over all ( f, a, π ) tuples, we set the right-hand side of Eqn. 29 to δ/ |F VEP ||A|| Π r VEP | and solve for t , and the theorem follows.
Theorem 3.
Let m VEP ∈ M VEP be a VE partial model of the true environment m ∗ ∈ M .
Let ˜ m VEP ∈ M VEP be the corresponding approximate VE partial model that has the same reward function as m VEP , but whose transition distribution is estimated by m calls to the generative model m VEP , where
[Equation]
and let Q k ˜ m VEP be the value returned by Q-value iteration at the k th epoch.
Then, with probability greater than 1 − δ , the following holds for all f ∈ F VEP and a ∈ A :
[Equation]
where k = log( ε (1 − γ )) log γ and Q ∗
m VEP is the optimal action value function in m VEP .
Proof.
Before starting the proof, let us first define generative models.
A generative model , or a sampler , is a model that can provide us with samples f ′ ∼ p ( f, a, · ) for all f ∈ F VEP and a ∈ A .
Now that we have defined generative models, let us assume we have access to a generative model m VEP and suppose we call our this model N times at each ( f, a ) pair.
Let ˆ p be the transition distribution of our empirical model, defined as follows:
[Equation]
where f i ∼ p ( f, a, · ) , ∀ i ∈ { 1 , . .
. , N } , and count ( f, a, f ′ ) is the number of times the pair ( f, a ) transitions to f ′ .
Moving on the main proof, by adding and subtracting Q π ∗ ˜ m VEP ˜ m VEP , we can rewrite Q k ˜ m VEP − Q ∗ m VEP as follows:
[Equation]
Bounding Term ( i ) :
[Equation]
Unrolling the last inequality k times
[Equation]
we obtain:
[Equation]
Bounding Term ( ii ) :
[Equation]
Therefore,
[Equation]
Fix a ( f, a ) pair:
[Equation]
where S N = (cid:80) Ni =
1 X i and X i
= V ∗ m VEP ( f ′ i ) .
X i are random independent variables and | X i | ≤ 11 − γ .
Applying Hoeffding’s inequality
[Equation]
we obtain ∀ t > 0 :
[Equation]
Let the failure probability δ > 0 .
Solve for t ,
[Equation] [Equation]
With probability at least 1 − δ ,
[Equation]
We conclude
[Equation]
By choosing
[Equation]
and
[Equation]
=
[Equation]
we get (cid:13) (cid:13) Q k ˜ m VEP − Q ∗ m VEP (cid:13) (cid:13) ∞ ≤ ε/ 2 + ε/ 2 = ε .
Therefore, the total number of samples (calls to the generative model) to get an ε estimation of the optimal Q -value is:
[Equation]
B A LGORITHM P SEUDOCODES
1: Initialize the parameters V 0 = 0 and Q 0 =2: for episode k = 1 , . .
, K do 3: for ( f, a ) ∈
F × A do 4:
Q k ( f, a ) =
r ( f, a ) + γ ˜ pV k − 1 ( f, a ) 5:
V k ( f ) = max a ∈A Q k ( f, a ) 6: end for 7: end for 8: Return Q K
1: Initialize the parameters θ , η & ω of ϕ θ : S → F , Q η : F × A → R & m ω
= ( p ω , r ω ) 2: Initialize the replay buffer B ← {} 3: N ple ← number of episodes to perform planning and learning 4: N rbt ← number of samples that the replay buffer must hold to perform planning and learning 5: n s ← number of time steps to perform search 6: n bs ← number of samples to sample from the replay buffer 7: h ← search heuristic 8: T ← replay buffer sampling strategy 9: i ←10: while i < N ple do 11:
S ← reset environment 12: while not done do 13:
A ← ϵ -greedy ( tree search with bootstrapping ( ϕ θ ( S ) , m ω , Q η , n s , h )) 14:
R, S ′ , done ← environment( A ) 15:
B ← B + { ( S, A, R, S ′ , done ) } 16: if |B| ≥ N rbt then 17:
D ← sample batch ( B , n bs , T ) 18:
Update ϕ θ , Q η & m ω with D 19: end if 20:
S ← S ′ 21: end
while 22: i ← i +23: end while 24: Return ϕ θ , Q η & m ω
Note that Alg.
2 2 does not employ the “bottleneck mechanism” introduced in (Zhao et al., 2021).
C E XPERIMENTAL D ETAILS
In this section, we provide the implementation details of the environments that are used in Sec.together with the details of the models that are used in the scalability experiments of Sec. 5.1.
We also provide the implementation details of the straightforward decision-time planning algorithm of
Zhao et al.
(2021) that was used in Sec.
C. I MPLEMENTATION D ETAILS OF THE SW E NVIRONMENT
As stated in Sec. 3, in the Squirrel’s World (SW) environment the squirrel’s job is to navigate from cell E1 (its initial state) to cell E16 (the terminal state) to pickup the nut without getting caught by the hawk that flies back and forth horizontally along row C. At each time step, the squirrel receives as input an 5 × 16 image of the current state of the environment and then, through the use of a predefined state encoder, transforms this image into a feature vector that contains information regarding all aspects of the current state of the environment, i.e., the feature vector contains information on the current position of the squirrel and the cloud, the current wind direction in rows A and B, the current position and direction of the hawk and the current weather condition.
Based on this, the squirrel selects an action that either moves it to the left or right cell, or keeps it position fixed (except if the agent is trying to move out of the boundaries of the world in which case its position is kept constant).
If the squirrel gets caught by the hawk or if it is out of time, it receives a reward of 0 and the episode terminates, and if the squirrel successfully navigates to the nut within the given time limit, it gets a reward of +10 and the episode terminates.
The agent-environment interaction lasts for 100 time steps, after which the agent receives a done signal, marking the end of the episode.
C. I MPLEMENTATION D ETAILS OF THE 2RDO E NVIRONMENTS
In the 2RDO environments, the agent, depicted by the red triangle, spawns in top-left of the top room and has to navigate to the green goal cell located in the bottom-right of the same room, regardless of the gaseous motions of the obstacles in the bottom room.
Here, at each time step, the obstacles move to one of its neighboring cells (except if it is trying to move out of the boundaries of the world in which case its position is kept constant).
At each time step, the agent receives an image of the current state of the grid and then, through the use of a learned state encoder, transforms this image into a feature vector.
Based on this, the agent selects an action that either turns it left or right, or moves it forward (except if the agent is trying to move out of the boundaries of the world in which case its position is kept constant).
If the agent successfully navigates to the goal cell within the given time limit, it receives a reward of +1 and the episode terminates.
The agent-environment interaction lasts for 50 time steps for the 8x8 environments and 100 time steps for the 16x16 environments, after which the agent receives a done signal, marking the end of the episode.
C. D ETAILS OF THE H AND -E NGINEERED M ODELS
The details of what the models in Sec.
5.1 model can be found in Table
[Table] [Table]
C. D ETAILS AND H YPERPARAMETERS OF THE D ECISION -T IME P LANNING A LGORITHM
The details and hyperparameters of the straightforward decision-time of Zhao et al.
(2021) that we have used can be found in Table
[Table]
[Table]
For more details (such as the NN architectures, replay buffer sizes, learning rates, exact details of the tree search, ...), we refer the reader to the publicly available code and the supplementary material of Zhao et al.
C. D ETAILS OF THE E NCODER S HAPING P ROCEDURE D URING T RAINING
In Sec.
5.2, we argued that one of the important inductive biases that is likely to guide the agent in coming up with only the relevant features of the environment is to only let the value estimator shape the encoder and to prevent the model from doing so.
This is pictorially depicted in Fig.
[Figure]