Reproducibility study of the Value Equivalence principle for Model Based Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
Grimm et al.
[2020] introduces and studies the concept of equivalence for Reinforcement Learning models with respectto a set of polices and value functions.
It further shows that this principle can be leveraged to ﬁnd models, constrainedby representational capacity, which are better than their maximum likelihood counterparts.
Methodology
The code for this project was closed sourced, therefore we re-implemented the three sets of experiments (including thebaseline) and the custom environments used by the authors.
All experiments were performed using Google Colab andrequired a total of 160 hours of Google Colab GPU.
Results
Since all the results in the original paper are presented in graphical form, we cannot provide precise numbers.
Forexperiments with span( V ) ≈ ¨ V , our results match the reported results.
For experiments with span( V ) ≈ ˜ V and linearfunction approximation, our results for both the baseline and the author’s method, diverge from the reported graphs.
Forexperiments with span( V ) ≈ ˜ V and neural networks our results follow the reported trend, but not always with the samevalues.
What was easy
Even though we had to re-implement everything from scratch, the general pipeline for all experiments were straightforward and well described in the original paper.
The environments used for all three experiments were fairly simple.
What was difﬁcult
Grimm et al.
[2020] is a combination of theory and experiments.
It is crucial to understand the theorems presented in thepaper which require a strong knowledge of linear algebra.
For experiments with linear function approximation, featureselection was done using k-means which depends heavily on initialisations of centroids. It takes a large amount of timeto repeatedly apply k-means on a large set of data to ﬁnd the best ﬁt (more than 10 hours for 10,000 initialisations on adataset of 1,000,000).
Communication with original authors
We contacted the author for multiple queries related to custom environments, hyper-parameters, feature selections andother minute experimental details via email.
The author replied to all of them thoroughly and in reasonable time
Introduction
Grimm et al.
[2020] introduces the principle of Value Equivalent models with respect to a set of policies and a set ofvalue functions.
Consider, Π to be a set of policies and V to be a set of functions from S → R .
Any two models m and˜ m are value equivalent with respect to Π and V if and only if, for all π ∈ Π and all ν ∈ V , the following is true.
[Equation]
where T π is the bellman operator of the MDP with respect to policy π .
The space of models which are value equivalent to the true model of the MDP, with respect to Π and V , is called
M (Π , V ) .
The main argument of the paper is that we can aptly choose V and Π and exploit the Value Equivalence Pinciple to ﬁnd models in M (Π , V ) which are simpler and save computation and memory.
Scope of reproducibility
Grimm et al.
[2020] shows that it is possible to exploit some structure and deﬁne the sets V and Π for training modelsusing the Value Equivalence principle.
And that these models perform better than Maximum Likelihood models, underrank constraints.
Exploiting structure in the problem: The set of true value functions of an MDP forms a polytype ¨ V (Dadashi et al.[2019]).
To span as much of the polytype as possible, we deﬁne V to contain true value functions ν π associated withrandom deterministic policies π .
Exploiting structure in the solution:
In most applications of model based RL, value functions or policies arerepresented using function approximation.
Suppose the agent can only represent policies π ∈ ˜Π and value
functionsν ∈
˜ V , then any model belonging to the set M (˜Π , ˜ V ) is as good as the true model of the MDP.
We aim to verify these claims by using rank constrained models to compare the performance of Value Equivalence and
Maximum Likelihood Estimation.
Methodology
We re-implement the training, evaluation and the custom environments for all the experiments performed by the authorsusing the experimental pipeline and environment description provided in the appendix of Grimm et al.
3.1 Experimental setup and code
The total reproducibility can be divided into three type of experiments:
• span( V ) ≈ ¨ V and ﬁnite state space.• span( V ) ≈ ˜ V and ﬁnite state space, using linear function approximation.• span( V ) ≈ ˜ V and inﬁnite state space, using neural network.
We used the Catch environment for the ﬁrst two experiments and the CartPole environment for the last one.
The authorshave used a custom Four-Rooms environment as well which has been uploaded to our Github repository and can beeasily used by future practitioners.
Our entire code can be accessed at https://github.com/RajGhugare19/VE-principlefor-model-based-RL
3.2 Hyperparameters
All the hyper-parameters are provided by the authors in the appendix.
We found that for experiments with span( V ) ≈ ˜ Vand linear approximation, the speciﬁed learning rate ( 5 · 10 − 5 ) did not work.
After doing a learning rate search wefound that, 5 · 10 − 3 works best for these experiments.
3.3 Computational requirements
All experiments were carried out on the free GPUs from Google Colab.
The GPU memory requirements and the total
GPU hours required by every experiment are provided below.
[Table]
Results
[Table]
In experiments with span( V ) ≈ ¨ V and span( V ) ≈ ˜ V with neural networks
[Table]
we found that the Value Equivalence modelsconsistently performed better than the baseline, under rank constraints.
For experiments with span( V ) ≈ ˜ V with linearfunction approximation we couldn’t reproduce the results for both Value Equivalence training as well as the baseline.
4.1 Results reproducing
original paper
[Table]
4.1.1 Result
[Table] The results from table 2 support the claim that Value Equivalence training converges towards a value equivalent modelin a rank constrained space,
i
[Table]
e [Table] a space with lesser span, when enforcing value equivalence with the value functionpolytype ( span( V ) ≈ ¨ V ).
[Table] [Table]
The results are average state values corresponding to the policies formed using respective models.
4.1.2 Result 2
We followed the given pipeline and performed 10 experiments, the code for which can be found in our repository.
Butthe values corresponding to the resulting policies for both VE and MLE models were highly stochastic and did notreﬂect the results reported by the authors.
To further evaluate the experiments with span( V ) ≈ ˜ V and linear function approximation we used Double DQN toform a policy where the action value functions were linear function approximators (see table 5).
The number of experiments performed were relatively small, but their results clearly show that value equivalencemodels perform better than the baseline, under rank constrained.
4.1.3 Result 3
The results from table 6 indicate that even when the set ˜ V consists of neural networks, the Value Equivalent modelswith rank constraints tend to perform better than the baselines.
But as the constraints are reduced the performance ofthe VE models seems to deteriorate.
These results are average returns for 100 consecutive episodes corresponding to the best policies formed using Double
DQN. [Table]
[Table]
[Table]
4.2 Results beyond original paper
We used a different evaluation technique for the results of table 5.
Instead of using approximate policy iteration with
LSTD, we used Double DQN with linear action-value function approximators.
The hyper-parameters used for this canbe found in table 9.
[Table]
Discussion
Our results support the existence of Value Equivalent models in lower dimensional spaces and the ability of Value
Equivalence training to converge to these models.
Even though we weren’t able to reproduce the results for experiments with span( V ) ≈ ˜ V with linear function approximation, we have a few likely reasons behind this.
• The performance of both the VE and baselines models depend highly on the feature formation, which wasdone using k-means.
The results of k-means are heavily dependent on the initialisation of centroids.• LSTD with policy evaluation was used to evaluate the model.
We are not sure if this method is guaranteed toconverge to the best policy.
We found that for most models this was not the case.• The evaluation method not only diverged for the VE models but also for MLE models.
The results of table 5 also solidify the above reasons.
5.1 What was easy
The experiments in the paper were easy to follow and code, as the authors have divided the entire pipeline into differentstages with clear descriptions for every stage.
5.2 What was difﬁcult
Some points were left out or not clear in the original paper and we had to ask the authors to get them cleared
• Feature formation for experiments with span( V ) ≈ ˜ V and linear function approximation was done by applyingk-means on a set of 1000000 states collected using a random policy.•
The size of experience replay used in the paper was 1000000.• Rank constrained matrices where optimised by applying gradient descent to the parameters F D and F K .