Reproducibility study of the Value Equivalence principle for Model Based Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
Grimm et al.
[2020] introduces and studies the concept of equivalence for Reinforcement Learning models with respectto a set of policies and value functions.
It further shows that this principle can be leveraged to ﬁnd models constrainedby representational capacity, which are better than their maximum likelihood counterparts.
Methodology
The code for this project is closed sourced. Therefore, we re-implemented the three sets of experiments (including thebaseline) and the authors’ custom environments.
All experiments were performed using Google Colab and required atotal of 160 hours of Google Colab GPU.
Results
Since all the results in the original paper are presented in graphical form, we cannot provide precise numbers.
Forexperiments with span( V ) ≈ ¨ V , our results match the reported results.
For experiments with span( V ) ≈ ˜ V and linearfunction approximation, our results for both the baseline and the author’s method diverge from the reported graphs.
For experiments with span( V ) ≈ ˜ V and neural networks, our results follow the reported trend, but not always with thesame values.
What was easy
Even though we had to re-implement everything from scratch, the general pipeline for all experiments was straightforward and well described in the original paper.
The environments used for all three experiments were reasonablystraightforward.
What was difﬁcult
Grimm et al.
[2020] is a combination of theory and experiments.
It is crucial to understand the theorems presented inthe paper, requiring a solid knowledge of linear algebra.
For experiments with linear function approximation, featureselection was made using k-means which depends heavily on centroids’ initialisations.
It takes a signiﬁcant amount oftime to repeatedly apply k-means on a large set of data to ﬁnd the best ﬁt (more than 10 hours for 10,000 initialisationson a dataset of 1,000,000).
For these experiments, we also found that the given learning rate did not learn a good model(see ﬁgure 1).
Communication with original authors
We contacted the author for multiple queries related to custom environments, hyper-parameters, feature selections andother minute experimental details via email.
The author replied to all of them thoroughly and in a reasonable time.
Introduction
Grimm et al.
[2020] introduces the principle of Value Equivalent models with respect to a set of policies and valuefunctions.
Consider, Π to be a set of policies and V to be a set of functions from S → R .
Any two models m and ˜ m arevalue equivalent with respect to Π and V if and only if, for all π ∈ Π and all ν ∈ V , the following is true.
[Equation]
T π is the bellman operator of the MDP with respect to policy π .
The space of models which are value equivalent to the true model of the MDP, with respect to Π and V , is called
M (Π , V ) .
The main argument of the paper is that we can aptly choose V and Π and exploit the Value Equivalence
Principle to ﬁnd models in M (Π , V ) which are simpler and save computation and memory.
Scope of reproducibility
Grimm et al.
[2020] shows that it is possible to exploit some structure and deﬁne the sets V and Π for training modelsusing the Value Equivalence principle.
And that these models perform better than Maximum Likelihood models underrank constraints.
Exploiting structure in the problem: The set of true value functions of an MDP forms a polytype ¨ V (Dadashi et al.[2019]).
To span as much of the polytype as possible, we deﬁne V to contain true value functions ν π associated withrandom deterministic policies π .
Exploiting structure in the solution:
In most applications of model based RL, value functions or policies arerepresented using function approximation.
Suppose the agent can only represent policies π ∈ ˜Π and value
functionsν ∈
˜ V , then any model belonging to the set M (˜Π , ˜ V ) is as good as the true model of the MDP.
We aim to verify these claims by using rank constrained models to compare the performance of Value Equivalence and
Maximum Likelihood Estimation.
Methodology
We re-implement the training, evaluation, and custom environments for all the authors’ experiments using the experimental pipeline and environment description provided in the appendix of Grimm et al.
3.1 Experimental setup and code
The total reproducibility can be divided into three type of experiments:
• span( V ) ≈ ¨ V and ﬁnite state space (4.1.1). • span( V ) ≈ ˜ V and ﬁnite state space, using linear function approximation (4.1.2). • span( V ) ≈ ˜ V and inﬁnite state space, using neural networks (4.1.3).
We used the Catch environment for the ﬁrst two experiments and the CartPole environment for the last one.
The authorshave used a custom Four-Rooms environment as well, which has been uploaded to our Github repository and can beeasily used by future practitioners.
Our entire code is available at https://github.com/RajGhugare19/VE-principle-formodel-based-RL
3.2 Hyperparameters
The authors have provided all the hyper-parameters in the appendix.
We found that for experiments with span( V )
≈ ˜
Vand linear function approximation, the speciﬁed learning rate ( 5 · 10 − 5 ) did not work.
After doing a learning rate search,we found that 5 · 10 − 2 works best for these experiments (see ﬁgure 1).
3.3 Computational requirements
We carried out all the experiments on the free GPUs from Google Colab.
The GPU memory requirements and the total
GPU hours required by every experiment are provided below.
[Table]
Results
In experiments with span( V ) ≈ ¨ V and span( V ) ≈ ˜ V with neural networks, we found that the Value Equivalence modelsconsistently performed better than the baseline, under rank constraints.
For experiments with span( V ) ≈ ˜ V with linearfunction approximation we couldn’t reproduce the results for both Value Equivalence training as well as the baseline.
Although using a different evaluation technique for linear value function approximation, we show new results thatconvey the superiority of Value Equivalent models under constraints.
4.1 Results reproducing original paper4.1.1 Result 1
The results from table 2 support the claim that Value Equivalence training converges towards a value equivalent modelin a rank constrained space, i.e. a space with lesser span, when enforcing value equivalence with the value functionpolytype ( span( V ) ≈ ¨ V ).
[Table]
of policies)
=
[Table]
Results
[Table]
The results are average state values corresponding to the policies formed using respective models.
4.1.2 Result 2
We followed the given pipeline and performed several experiments, the code for which can be found in our repository.
However, the values corresponding to the resulting policies for both VE and MLE models were highly stochastic anddid not reﬂect the authors’ results.
To further evaluate the experiments with span( V ) ≈ ˜ V and linear function approximation we use Double DQN wherethe action value function is a linear function approximator (see table 5).
The results clearly show that the Value Equivalence Principle can exploit the fact that the agent can only represent linearvalue functions. They also support the claim that under rank constraints, agents trained using the VE principle performsubstantially better. [Table] [Table]
4.1.3 Result 3
The results from table 8 indicate that even when the set ˜ V consists of neural networks [Table] the Value Equivalent modelswith rank constraints tend to perform better than the baseline.
However, as we reduce the constraints, the performanceof the VE models seems to deteriorate. [Table] [Table]
These results are average returns for 100 consecutive episodes corresponding to the best policies formed using Double
DQN.
4.2 Results beyond original paper
We used a different evaluation technique for the results of table 5.
Instead of using approximate policy iteration with
LSTD [Table] we used Double DQN with linear action-value function approximators.
The hyper-parameters used for this canbe found in table 11.
[Table]
Discussion
Our results support the existence of Value Equivalent models in lower-dimensional spaces and the ability of Value
Equivalence training to converge to these models.
Even though we were not able to reproduce the results for experiments with span( V ) ≈ ˜ V with linear functionapproximation, we have a few likely reasons behind this rather than the failure of VE models.
• The performance of both the VE and baselines models depend highly on the feature formation, which wasdone using k-means.
The results of k-means are heavily dependent on the initialisation of centroids.
• The evaluation method not only diverged for the VE models but also for MLE models. • Table 5 suggests that the problem could lie in the authors’ evaluation technique, i
e. LSTD with approximatepolicy iteration.
5.1 What was easy
The paper’s experiments were easy to follow and code, as the authors have divided the entire pipeline into differentstages with clear descriptions for every stage.
5.2 What was difﬁcult
Some points were left out or not evident in the original paper, and we had to ask the authors to get them cleared.
• Feature formation for experiments with span( V ) ≈ ˜ V and linear function approximation was done by applyingk-means on a set of 10 6 states collected using a random policy.•
The size of experience replay used in the paper was 10 6 . • Rank constrained matrices where optimised by applying gradient descent to the parameters F D and F K . 5.3 Future work Grimm et al.
[2020] uses rank constraints for Neural Networks in experiments with span( V )
≈ ˜ V .
For experimentswith ranks greater than 8, constraints on the ﬁrst and last layer are effectively removed.
This is because these layerseach have one dimension, which is smaller than 8.
It would be interesting to use other constraints like positive-semideﬁniteness (Lezcano-Casado [2019]) along with rank constraints to compare the performance of VE and MLE models.