Approximate Value Equivalence
Anonymous Author(s)
AffiliationAddress email
Abstract
Model-based reinforcement learning agents must make compromises about whichaspects of the environment their models should capture.
The value equivalence(VE) principle posits that these compromises should be made considering themodel’s eventual use in value-based planning.
Given sets of functions and policies,a model is said to be order- k VE to the environment if k applications of the Bellmanoperators induced by the policies produce the correct result when applied to thefunctions.
Prior work investigated the classes of models induced by VE when wevary k and the sets of policies and functions.
This gives rise to a rich collectionof topological relationships and conditions under which VE models are optimalfor planning.
Despite this effort, relatively little is known about the planningperformance of models that fail to satisfy these conditions.
This is due to therigidity of the VE formalism, as classes of VE models are defined with respect toexact constraints on their Bellman operators.
This limitation gets amplified by thefact that such constraints themselves may depend on functions that can only beapproximated in practice.
To address these problems we propose approximate valueequivalence (AVE), which extends the VE formalism by replacing equalities witherror tolerances.
This extension allows us to show that AVE models with respect toone set of functions are also AVE with respect to any other set of functions if wetolerate a high enough error.
We can then derive bounds on the performance of VEmodels with respect to arbitrary sets of functions by relating them to a particularmodel class with known performance guarantees.
We support these results byproviding intuitions and discussions of their implications.
Introduction
Reinforcement learning (RL) is a general framework in which an agent learns to maximize the rewardit receives from its environment by interacting with it [Sutton and Barto, 2018].
Though RL makesno prescriptions about their internal processes, agents are commonly endowed with the ability tolearn simulators of their environment, called models , which enable predicting the effects of sequencesof actions without executing them in the environment.
The subfield of RL which studies such agentsis referred to as model-based reinforcement learning (MBRL).
In classical MBRL, agents endeavor to learn models that accurately capture the dynamics of theirenvironments and subsequently use these models in their estimation of the values of different policies.
Though having an accurate model is sufficient for this task, they are frequently impractical to learn—as MBRL agents are deployed in more complex environments, the demands of learning modelswhich accurately capture these complexities become more severe.
Fortunately, such models are notnecessary for learning performant MBRL agents, as demonstrated by several recent works [Oh et al.
,2017, Silver et al., 2017, Farquhar et al., 2018, Schrittwieser et al., 2020].
The agents in these workslearn models that are accurate at what they are eventually used for: the estimation of policy values.
In particular, these models are learnt to be able to accurately predict the values of policies in theenvironment rather than to be able to accurately predict the environment’s dynamics.
Grimm et al.
[2020, 2021] provide a theoretical underpinning for models learned in this way byestablishing the value equivalence (VE) principle.
VE partitions the space of models an agent canlearn into different VE classes according to properties of their Bellman operators.
For a giveninteger k , an order- k VE class consists of models whose k -step Bellman operators of certain policiesmatch the environment’s when applied to certain functions.
The policies and functions used as VEconstraints are the identifiers of the equivalent classes.
Accordingly, much of the prior work on valueequivalence has concerned the selection of these sets such that the corresponding VE models areof use to an agent.
In particular, Grimm et al.
[2021] proved that any model in the infinite-order
VE—also known as proper value equivalent or PVE—class induced by all deterministic policies issufficient for optimal planning.
However, with the exception of this specific model class, the analysesby Grimm et al.
[2020, 2021] offer no bounds on the performance of more general VE classes.
In this work we extend the VE framework by developing an approximate theory of value equivalencewhich augments VE classes with an error tolerance parameter that controls the degree to which modelscan violate their class’ constraints (Definition 1).
We call the resulting framework approximate valueequivalence (AVE).
We show that topological relationships exist between arbitrary pairs of AVEclasses (Proposition 1).
By leveraging these topological relationships, we extend the performancebound on the aforementioned PVE class to arbitrary AVE classes (Proposition 2).
Initially, we presentthe bound in terms of generic topological relationship between an AVE class and the reference PVEclass.
We then conduct a quantitative analysis of how topological properties of an AVE class impacton the bound.
To aid our analysis, we study these properties for different aspects of AVE classes inisolation: first we study the impact of the model class order (Section 4.1) and then we look at theimpact of specific choices of function sets (Section 4.2).
We then combine these results to obtain ageneral topological relationship between AVE classes as order and function sets vary (Section 4.3),which, due to the connection we showed between AVE topology and performance (Proposition 2),directly translates into a bound on the performance of models in AVE classes with arbitrary ordersand function sets.
This result extends the set of VE classes whose performance can be analyzed fromthe PVE class mentioned above to AVE classes with arbitrary orders and function sets.
Background
An agent’s interaction with its environment is modeled using a Markov Decision Process (MDP)[Puterman, 1994] denoted by ⟨S , A , r, p, γ ⟩ , where S is a state-space, A is a set of actions that can betaken from each state, r ( s, a ) is the expected reward of taking action a from state s , p ( s ′ | s, a ) is theprobability of transitioning to state s ′ when action a is taken from state s and γ ∈ [0 , 1] is a discountfactor.
The behavior of an agent is specified by a policy, π : S (cid:55)→ P ( A ) , where P ( A ) is the set ofprobability distributions over A .
We denote the set of all policies as Π and the set of all functions ofsignature S (cid:55)→ R as V .
The performance of a policy in an MDP is measured by its value function:
[Equation]
where E π,p [ · ] denotes expectation over trajectories generated by policy π in an MDP with transitiondynamics given by p , and S t and A t are random variables for the state occupied and action taken by theagent at timestep t .
We denote the set of all value functions in an environment as V Π = { v π : π ∈ Π } .
An RL agent’s goal is to find a policy with maximal value across all states [Sutton and Barto, 2018].
This process is typically carried out by alternating between computing the value of the agent’s policyand improving it.
Value functions can be computed iteratively using the policy’s Bellman operator :
[Equation]
which accepts a function v ∈ V and returns an updated function in also in V
Repeated applicationsof a policy’s Bellman operator are denoted by T k π and referred to as the policy’s k -step Bellmanoperator.
It is known that, in the limit, k -step Bellman operators converge to their corresponding valuefunctions: lim k →∞ T kπ
[ v ] = v π ∀ v ∈ V .
In RL it is assumed that the agent does not have access to rand p , and thus T π cannot be computed directly.
MBRL addresses this problem by learning a model,˜ m ≡ (˜ r, ˜ p ) , that is used in place of the environment’s reward and transition dynamics to constructmodel Bellman operators, denoted ˜ T π .
Traditional MBRL methods attempt to learn models such that ˜
r ≈ r and ˜ p ≈ p
[Sutton, 1991].
Incontrast, there are alternative paradigms in which a model is learned with its eventual use in mindsuch as value-aware model learning (VAML; [Farahmand et al., 2017]) and the value equivalence(VE) principle [Grimm et al., 2020].
This work concerns the VE principle, under which modelsare learned to support their eventual use in estimating value functions.
Accordingly, VE delineatesmodels according to VE constraints , each of which require, for a specific k , policy π ∈ Π and functionv ∈ V , that ˜ T kπ v =
T kπ v .
We denote the class of order- k VE models with respect to a set of policiesΠ ⊆
Π and a set of functions V ⊆ V as
[Equation]
In the limit of k → ∞ , Grimm et al.
[2021] showed that order- k VE classes converge to specialmodel classes whose value functions ˜ v π = lim k →∞ ˜ T kπ v match those of the environment:
[Equation]
These limiting model classes are referred to as proper value equivalent (PVE) due to their dependenceon model and environment value functions .
The PVE class with respect to all deterministic policies,denoted M ∞ ( Π det ) , contains only models which can plan optimally.
Notably, this is the only modelclass within the VE framework with such a performance guarantee [Grimm et al., 2021].
Approximate value equivalence
We begin our treatment of the approximate theory of value equivalence by extending the definitionsof order- k VE and PVE classes given in Eqs.
3 and 4 to the approximate setting.
Recall that the exactversions of these model classes are characterized by sets of policies and functions which induce a setof VE constraints on the model’s Bellman operators.
In this work we will consider model classeswhose models can violate these constraints up to some limited degree, ϵ ≥ 0 .
We say that theseapproximate value equivalent (AVE) classes tolerate ϵ error.
Definition 1.
(Approximate Value Equivalence)
Given Π ⊆ Π , V ⊆ V and ϵ ∈ R , we denote the setof order- k AVE models which tolerate ϵ error as:
[Equation]
where ∥ · ∥ denotes the ℓ ∞ -norm.
Moreover, we can analogously define approximate proper value equivalent (APVE) classes as follows:
[Equation]
3.1 Topological properties
In what follows we investigate the topological properties of AVE classes, ultimately showing that,by introducing approximation to VE, we can relate arbitrary model classes.
An order- k VE class isspecified in terms of four quantities: a model class M , a set of policies Π , a set of functions V , andan order k .
Grimm et al.
[2020] illustrated a variety of topological properties of order- 1 VE classesby studying the effect of varying M , Π and V .
These results were later extended by Grimm et al.[2021], who also investigated the effect of varying the order k .
We now conduct a similar analysis of
AVE classes, which are extensions of VE classes that depend on a fifth quantity: an error toleranceϵ .
We start by showing that AVE models are a strict generalization of VE models, which become aspecial case when ϵ = 0 :
Property 1.
For any Π ⊆ Π , V ⊆ V and M ⊆ M , it follows that M k (Π , V ; 0) =
M k (Π , V ) and
M ∞ (Π; 0) =
M ∞ (Π) .
Proofs of these properties will be deferred to Appendix A.1.
Since Property 1 tells us that AVE is ageneralization of exact VE, it is natural to inspect which other topological properties carry over.
Property 2.
For any ϵ ≥ 0 , M ⊆ ¯ M ⊆ M , Π ⊆ Π ′ ⊆
Π and V ⊆ V ′ ⊆
V , it follows that
[Equation] [Equation]
Property 2 generalizes two of the topological properties described by Grimm et al.
[2020] to AVE.
Inparticular, the first subset relation in Eq.
7 shows that, like with VE, as the sizes of the sets of policiesand functions increase, the corresponding AVE class shrinks.
The second relation shows that as theset of models M increases in size, so too do AVE classes.
We now investigate the effect of the new parameter, the error tolerance ϵ , on the topology of theresulting AVE classes:
Property 3.
For any Π ⊆ Π , V ⊆ V and ϵ ′ ≥ ϵ ≥ 0 , it follows that
[Equation]
Property 3 shows that increasing the tolerated error of an AVE class causes it to include more models.
While this is intuitive from our parlance (e.g., “a model class that tolerates a higher error also toleratesa lower error”), it may be less obvious that this property can be used to relate AVE classes withrespect to arbitrary orders and sets of functions and policies.
Proposition 1.
For any ϵ ≥ 0 , Π , Π ′ ⊆
Π , V , V ′ ⊆
V and k, K ∈ Z
+ there exists some ϵ ′ ≥
suchthat
[Equation]
M K (Π ′ , V ′ ; ϵ ′ ) grows— eventually to the point of containing M k (Π , V ; ϵ ) .
Proposition 1 shows that it is possible to relate arbitary
AVE classes.
This is significantly broader than previoustopological results in which VE classes could be relatedonly when their corresponding sets of policies and functions were subsets of each other (when Π ⊆ Π ′ or V ⊆ V ′ )
[Grimm et al., 2020], or when the orders being comparedwere multiples of each other (when k divided K )
[Grimmet al., 2021].
The proposition follows directly from takingthe limit of ϵ ′ → ∞ in Property 3: by tolerating a highenough error any AVE class can be made to contain anyother (as illustrated in Figure 1).
Why is it important to ensure that one AVE class is a subclass of another?
We know that, if a property applies to amodel class, it must also apply to any sub-class.
Grimmet al.
[2021] have shown that all models in M ∞ ( Π det ; 0)have the same optimal policy as the environment.
Theyalso showed that, without further assumptions, this is the“largest” model class with this property.
Thus, up untilnow, the best performance guarantees one could provideregarding a model class were: “if the model class is asub-class of M ∞ ( Π det ; 0) , any model in it yields an optimal policy; otherwise, there is nothing onecan say without making further assumptions.”
Proposition 1 allows us to go much further and provide performance guarantees for any AVE class.
This is possible because we know we can relate arbitrary AVE classes of the form
M k
(Π , V ; ϵ )
to
M ∞ ( Π det ; 0) by tolerating a sufficiently large amount of error ϵ ′ .
Obviously, the tolerated error ϵ ′will play a role in the resulting performance guarantees.
In general, the larger this error the weakerthe guarantees.
So, we are interested in finding the minimum ϵ ′ that makes Proposition 1 true.
The minimum ϵ ′ for which Proposition 1 holds between any two AVE classes can also be thought ofas a function, which we will refer to as the minimum tolerated error .
Definition 2.
(Minimum Tolerated Error)
For any AVE classes M k (Π , V ; ϵ ) and M K (Π ′ , V ′ ; ϵ ′ )
we denote the minimum ϵ ′ ≥
such that M k (Π , V ; ϵ ) ⊆
M K (Π , V ′ ; ϵ ′ )
as E ϵ (Π , V , k | Π ′ , V ′ , K ) .
Note that the minimum tolerated error is well defined for any two AVE classes.
Equipped with thisconcept, we can now present the following result:
Proposition 2.
For any ˜ m ∈ M k (Π , V ; ϵ ) it follows that
[Figure]
[Equation]
where V Π is the set of all value functions of the environment m and ˜ π ∗ is any optimal policy of ˜ m .
Proposition 2 assures us that we can bound the performance of any AVE class M k (Π , V ; ϵ ) in termsof the minimum error tolerance E ϵ (Π , V , k | Π , V , ∞ ) .
Accordingly, understanding how this functionvaries with respect to its inputs is crucial to determining the performance of models in M k
(Π , V ; ϵ ) .
It is worth emphasizing that E ϵ ( ·|· ) is fundamentally a function concerned with the topology of
AVE classes and that Proposition 2 provides a link between such topological properties and theperformance of the corresponding models.
In what follows we will quantify E ϵ ( ·|· ) by providingupper-bounds on it that shed light on how it behaves as its inputs are varied.
Bounding the minimum tolerated error
We are interested in studying how the function E ϵ (Π , V , k | Π ′ , V ′ , K ) changes as we vary its parameters.
To facilitate our analysis, we will begin by varying these inputs in pairs, focusing first on orders( k, K ) and then on function sets ( V , V ′ ) .
1 Later, we combine these results to obtain a general upperbound for E ϵ ( ·|· ) , which will in turn yield a generic performance bound for any AVE class.
4.1 Model class order and error tolerance
In this section we consider pairs of model classes that share a common set of policies Π and functions
V , but have different orders k and K .
In particular, we explore this dependence in the special case inwhich k divides K and that V is closed under environment Bellman updates with respect to any ofthe policies in Π .
Proposition 3.
For any ϵ ∈ R , Π ⊆ Π , V ⊆ V such that v ∈ V =
⇒ T π v ∈ V ∀
π ∈ Π andk, K ∈ Z + such that k divides K , we have that
[Equation]
Proposition 3 shows that, in our special case, our upper-bound on the minimum tolerated errorincreases according to a multiplicative factor that depends on the orders k and K .
For a fixed k , as Kgrows our upper-bound will also grow, however the rate of growth is decreasing since the numeratorof this factor, (1 − γ K ) , approaches 1 as K → ∞ .
The fact that our upper-bound does not growwithout bound suggests that there is a similar relationship between order- k AVE and APVE classes.
Indeed, by sending K → ∞ we obtain such a relationship.
Corollary 1.
For any set of policies Π ⊆ Π , set of functions V ∈ V such that { v π : π ∈ Π } ⊆ V andk ∈
Z + , it follows that
[Equation]
Notice in Corollary 1 how the constraint on V has been relaxed, since now it is no longer requiredthat V be closed under arbitrary compositions of the Bellman operators of the policies in Π .
Instead,
V must contain the value functions associated with Π .
This looser constraint implies that V can befinite when Π is finite, which in turn reflects the fact that PVE and APVE classes do not depend on aset of functions V .
4.2 Function sets and error tolerance
In this section we consider two AVE classes with the same order k and set of policies Π but differentfunction sets, that is, we compare M k
(Π , V ; ϵ ) and M k
(Π , V ′ ; ϵ ′ ) .
Though we are able to providean upper-bound on the minimum tolerated error for arbitrary function sets, it can be shown to scaledifferently when there is special relation between V and V ′ .
We begin by examining this special case.
4.2.1 Related function sets
We begin by examining the case in which V ′ consists of constrained linear combinations of theelements of V .
In particular, we consider a set V ′ of the following form:
[Equation]
which we refer to as the constrained vspan of V with parameter c .
The coefficients of the linearcombinations in c - vspan( V ) are constrained to sum to 1 and to have magnitudes which sum to lessthan c .
Grimm et al.
[2020] studied a special case of Eq.
12 without the coefficient magnitude constraint (bytaking c → ∞ ) and showed that M 1 (Π , V ; 0) =
M 1 (Π , V ′ ; 0) (where V ′ = ∞ - vspan
( V ) ).
Noticehow, in the language of Proposition 1, we have just related two VE classes with different sets offunctions V and V ′ without tolerating any additional error.
However, this result is relatively narrow,only applying to exact order- 1 VE classes.
In what follows we extend this relationship to order- k
AVE classes:
Proposition 4.
For any set of policies Π ⊆ Π , set of functions V ∈ V , c > 1 and error ϵ ∈ R , wehave
[Equation]
Proposition 4 shows how M k (Π , V ; ϵ ) is “squeezed” between two other model classes whose functionsets are replaced by c - vspan( V ) .
In particular, by considering the rightmost subset relationship in
Eq. 13, we see that the error term scales multiplicatively with c .
It is interesting to note that, forparticular choices of ϵ and c , the left- and right-most terms in Eq.
13 can be made identical, reducingthe expression to an equality.
Corollary 2.
When either c = 1 or ϵ = 0 , for any Π ∈ Π , V ∈ V it follows that
[Equation]
Note that, when ϵ = 0 , Corollary 2 recovers Grimm et al.
[2020]’s result by sending c → ∞ .
On the other hand, when c = 1 , c - vspan( V ) reduces to the convex hull of V .
In the context of
Proposition 1, this means that, regardless of ϵ and k , we do not need to tolerate any additional errorfor M k
(Π , 1 - vspan( V ); ϵ ) to contain M k (Π , V ; ϵ ) .
4.2.2 Unrelated function sets
In the previous section we studied the case in which the set V ′ coincided with c - vspan( V ) ; we nowconsider a more general scenario where V and V ′ are not assumed to have any relationship to eachother.
In this setting, we will measure the similarity of sets of functions using a distance functionβ : 2 V × 2 V (cid:55)→ R defined as
[Equation]
In words, β measures the distance between the farthest function in V ′ from any function in V .
Beforewe continue, it is worth pointing out some interesting properties of β .
Proposition 5.1.
(Asymmetry)
For any V ⊆ V ′ ⊆ V ′′ ⊆
V it follows that
[Equation]
(Convex, Compact V )
When V is convex and compact it follows that
[Equation]
Proposition 5 illustrates several interesting features of β .
Firstly, it measures the similarity betweensets V and V ′ asymmetrically: if V ′ grows then β also increases, since there are more elements tomaximize over; however, if V grows, β decreases instead, since there are more elements to minimizeover.
Secondly, when V is a convex compact set, V ′ can be replaced with its convex-hull (equivalently,1 - vspan( · ) ) without changing β .
Using the similarity measure β , we can then determine the amount of error an AVE class with respectto V ′ must tolerate to contain a counterpart defined with respect to V .
Proposition 6.
For any Π ∈ Π , V , V ′ ∈ V and ϵ ≥ 0 , it follows that
[Equation]
Moreover, if V is convex and compact, we obtain:
[Equation]
Proposition 6 shows that, for any two sets of functions V and V ′ , the our upper-bound on the minimumtolerated error is proportional to β ( V||V ′ ) , which we analyzed in Proposition 5.
In particular, weknow that our upper-bound increases as V ′ grows and decreases as V grows.
In addition, in thespecial case in which V is a convex compact set, V ′ can be replaced by its convex hull without anyincrease in our upper-bound.
4.3 Unifying results
In the previous sections we studied upper-bounds on the minimum tolerated error such that modelsin particular kinds of AVE classes would be contained in other kinds of AVE classes.
Each sectionconcerned a different type of upper-bound constructed by varying certain aspects of the AVE classes:
Section 4.1 held function sets fixed and varied model-class orders whereas Section 4.2 held modelclass orders fixed and varied function sets.
Now, we combine these results to obtain a more generaltheorem.
Theorem 1.
For any Π ⊆ Π , V ⊆ V , k ∈ Z + , and ϵ ≥
it follows that
[Equation]
Proof.
The proof follows by a sequential application of our previous results.
[Equation]
[Equation] [Equation]
Since Eq.
16 holds for all c ≥ 1 , it follows that:
[Equation]
as needed.
Theorem 1 reveals an interesting tension.
Notice how the quantity inside the minimum is determinedby two terms, c · ϵ and 2 γ k β ( c - vspan( V ) ||V ′ ) .
As c grows, the first term increases; however, due tothe properties of β described in Proposition 5, the second term will shrink.
Bounds on model performance
Propositions 1 and 2 form a powerful connection between the topology of AVE classes and modelperformance, which, in principle, allows us to quantify the performance of models in any AVEclass M k (Π , V ; ϵ ) with respect to the minimum tolerated error
E ϵ
(Π , V , k | Π , V Π , ∞ )
(Definition 2).
Although we know that the aforementioned minimum tolerated error exists , Proposition 1 does notspecify how to quantify it.
Accordingly, in Section 4 we produced topological results about AVEmodels when policy sets Π and either order or function sets are held fixed.
We ultimately combinedthese results into Theorem 1, which we can use to obtain a general bound on the minimum toleratederror: E ϵ (Π , V , k | Π , V Π , ∞ ) .
By combining Theorem 1 with Proposition 2 we can obtain a performance bound on AVE classeswith respect to arbitrary orders and function sets:
Theorem 2.
For any ˜ m ∈ M k ( Π , V ; ϵ ) it follows that [Equation]
[Equation]
where ˜ π ∗ is an optimal policy of ˜ m .
Theorem 2 suggests that ideal AVE classes have low error ϵ and function sets such that c - vspan( V ) ⊇
V Π .
However, in practice an agent does not directly have control over the error ϵ with which it learnsa model.
Rather, this error is dependent on the choice of V : some functions may be harder to learn
VE models with respect to than others.
Since finding a model that is VE with respect to a smallnumber of functions is generally easier, this dependence between ϵ and V introduces a tension in theselection of the set V .
The smaller the number of functions in V , the smaller the resulting ϵ tends tobe, but on the other hand the distance β ( c - vspan( V ) || V Π ) generally increases.
One option for choosing V , which was explored in both Grimm et al.
[2020] and Grimm et al.
[2021],is to use some manageable set of value functions: { v π i } ni
=1 .
This is sensible as it is known that valuefunctions form a polytope V Π which could be covered by c - vspan( · )
[Dadashi et al., 2019].
However,value functions are quantities of the environment which are not assumed to be known by the agent.
At best, an agent can produce estimates of such quantities.
Interestingly, our AVE formalism can address not only the error in the satisfaction of VE constraintsbut also in the estimation of functions that the constraints depend upon.
If an agent can estimate eachv π i by ˆ v π i with bounded approximation error ∥ v π i − ˆ v π i ∥ ≤ ϵ approx , we have:
[Equation]
Eq.
18 shows that we can exchange the estimation error of functions in VE function sets for errorin the satisfaction of VE constraints.
This ultimately means that we can bound the performance ofmodels which are approximately VE with respect to functions that deviate from V Π :
Corollary 3.
Let ˆ V Π
= { ˆ v π : π ∈ Π } be a set of approximate value functions satisfying ∥ v π − ˆ v π ∥ ≤ϵ approx for all π ∈ Π .
Then for any ˜ m ∈ M k ( Π , ˆ V Π ; ϵ ) it follows that:
[Equation]
where ˜ π ∗ is any optimal policy in ˜ m .
Trade-offs between model capacity and function set size
In this section we show that our approximate formulation of value equivalence can guide the choiceof functions to learn a VE model with respect to when model capacity is limited. In particular, weshow that, depending on the situation, an agent may prefer to either learn a VE model with respect toa larger class of functions (and tolerate a higher approximation error ϵ ) or learn a VE model withrespect to a smaller class of functions (and tolerate a lower ϵ ). These situations are interesting because,as we showed in Theorem 2, higher approximation error results in worse bounds on the planningperformance of the associated models—which raises the possibility that in certain situations a limitedmodel might plan better when it is VE with respect to fewer functions.
To illustrate situations where this might occur, we consider the tabular Four Rooms domain and learntabular VE models whose per-action transition matrices are constrained to have rank at most K .
Welearn these models to be in the VE class M ( Π , V ) , where V is a set of D functions generated bysampling v ( s ) ∼ Uniform ( − 10 , 10) for each v ∈ V and each s ∈ S .
Due to the rank constraint onthe model, we will only be able to learn approximate VE models with varying error tolerances fordifferent settings of K and D .
Figure 2 shows a histogram of the planning performance of such models.
Each cell in the figurecorresponds to a model associated with specific values of K and D , and the cells’ color denotesthe value of the model’s optimal policy averaged over states and over 10 independent executions.
Interestingly, near the bottom left portion of the histogram there is a region in which low-capacity VEmodels with respect to small sets of functions perform better than their counterparts generated usingmore functions.
[Figure] Our work directly extends previoustheoretical results on VE to the approximate setting.
This naturally situates us closest to the works thatoriginally explicated value equivalence and its higher order analogues[Grimm et al., 2020, 2021].
Our focuson exploring models whose Bellmanoperators behave approximately likethose of the environment is also similar to works which studied approximate forms of policy and value iteration [Bertsekas and Tsitsiklis, 1995, Munos, 2005].
A closely related line of work is valueaware model learning (VAML, IterVAML, VaGraM)
[Farahmand et al. ,2017, Farahmand, 2018, Voelckeret al., 2021]).
In VAML models arelearned in order to minimize the discrepancy between their Bellman optimality operators and those of the environment.
In practice, a family offunctions is constructed and a model is learned to minimize the maximum discrepancy across thisfamily.
In IterVAML, instead of minimizing the worst case discrepency across a family of functions,the functions are generated by the model during a value iteration procedure.
In VaGraM, the authorsconsider a similar discrepancy-based loss as in VAML, but use a Taylor series approximation toavoid having loss terms involving the values of predicted future states.
This ultimately gives rise toa squared-error loss between predicted and actual next states weighted by the gradients of a valuefunction.
VE and VAML can be seen as complementary: both concern models that are learned withtheir eventual use in mind, but VAML focuses on the associated optimization problems of findingsuch models whereas VE focuses on characterizing these models into classes.
There have also been several recent empirical works which explored concepts adjacent to VE.
Farquhar et al.
[2021] explored adding auxilliary terms to different model learning procedureswhich encouraged the learned models’ Bellman operators and value estimates to satisfy the Bellmanequations (i.e., to be self-consistent).
They show that there are circumstances in which this leads tobetter data-efficiency in practice.
Another example is Nikishin et al.
[2021], who study the problemof learning models whose Bellman optimality operators induce optimal policies in the environment.
In the service of this, they propose using implicit differentiation [Christianson, 1994] to computegradients from the model’s optimality operator all the way to its optimal policy.
Conclusion and future work
We provided an approximate theory of value equivalence that extends the existing VE formalism[Grimm et al., 2020, 2021].
The main concept introduced was a family of AVE classes whose modelscan violate VE constraints up to a limited degree.
We showed that, as a product of introducingapproximation to VE, arbitrary AVE classes can be related to each other topologically.
Moreover, weshowed that, by relating them to specific VE classes for which there are performance guarantees, wecan derive bounds on the performance of arbitrary AVE classes in terms of their topological properties.
Motivated by this link between topology and performance bounds, we proceed to derive isolatedtopological relationships between AVE classes whose function sets, orders and error tolerances vary,which we then can combine to produce bounds which hold over AVE classes with respect to arbitraryorders, functions sets and error tolerances.
This addresses a fundamental open question in the VEtheory: how to provide performance guarantees for models that do not satisfy the VE constraintsexactly.
It may be possible to derive topological relationships considering all constraints of a VEclass jointly and, in so doing, provide tighter performance bounds, but we leave this for future work.
We also believe that the performance bounds we have introduced might give rise to new algorithmsthat attempt to minimize them, either directly or indirectly. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
In the conclusion / future work
section.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Most are in theappendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]