Dual Generator Offline Reinforcement Learning
Abstract
In offline RL, constraining the learned policy to remain close to the data is essential to prevent the policy from outputting out-of-distribution (OOD) actions with erroneously overestimated values.
In principle, generative adversarial networks (GAN) can provide an elegant solution to do so, with the discriminator directly providing a probability that quantifies distributional shift.
However, in practice, GAN-based offline RL methods have not performed as well as alternative approaches, perhaps because the generator is trained to both fool the discriminator and maximize return – two objectives that can be at odds with each other.
In this paper, we show that the issue of conflicting objectives can be resolved by training two generators: one that maximizes return, with the other capturing the “remainder” of the data distribution in the offline dataset, such that the mixture of the two is close to the behavior policy.
We show that not only does having two generators enable an effective GAN-based offline RL method, but also approximates a support constraint, where the policy does not need to match the entire data distribution, but only the slice of the data that leads to high long term performance.
We name our method DASCO, for
D ual-Generator
A dversarial S upport C onstrained O ffline RL.
On benchmark tasks that require learning from sub-optimal data, DASCO significantly outperforms prior methods that enforce distribution constraint.
Introduction
Offline reinforcement learning (RL) algorithms aim to extract policies from datasets of previously logged experience.
The promise of offline RL is to extract decision making engines from existing data [32].
Such promise is especially appealing in domains where data collection is expensive or dangerous, but large amounts of data may already exists (e.g., robotics, autonomous driving, taskoriented dialog systems).
Real-world datasets often consist of both expert and sub-optimal behaviors for the task of interest and also include potentially unrelated behavior corresponding to other tasks.
While not all behaviors in the dataset are relevant for solving the task of interest, even sub-optimal trajectories can provide an RL algorithm with some useful information.
In principle, if offline RL algorithms can combine segments of useful behavior spread across multiple sub-optimal trajectories together, the combined segments can then perform better than any behavior observed in the dataset.
Effective offline RL requires estimating the value of actions other than those that were taken in the dataset, so as to pick actions that are better than the actions selected by the behavior policy.
However, this requirement introduces a fundamental tension: the offline RL method must generalize to new actions, but it should not attempt to use actions in the Bellman backup for which the value simply cannot be estimated using the provided data.
These are often referred to in the literature as out-ofdistribution (OOD) actions [28].
While a wide variety of methods have been proposed to constrain offline RL to avoid OOD actions [26, 14, 1], the formulation and enforcement of such constraints can be challenging, and might introduce considerable complexity, such as the need to explicitly estimate the behavior policy [49] or evaluate high-dimensional integrals [30].
Generative adversarial networks (GANs) in principle offer an appealing and simple solution: use the discriminator to estimate whether an action is in-distribution, and train the policy as the “generator” in the GAN formulation to fool this discriminator.
Although some prior works have proposed variants on this approach [49], it has been proven difficult in practice as GANs can already suffer from instability when the discriminator is too powerful.
Forcing the generator (i.e., the policy) to simultaneously both maximize reward and fool the discriminator only exacerbates the issue of an overpowered discriminator.
We propose a novel solution that enables the effective use of GANs in offline RL, in the process not only mitigating the above challenge but also providing a more appealing form of support constraint that leads to improved performance.
Our key observation is that the generative distribution in
GANs can be split into two separate distributions, one that represents the “good parts” of the data distribution and becomes the final learned policy, and an auxiliary generator that becomes the policy’s complement, such that their mixture is equal to the data distribution.
This formulation removes the tension between maximizing rewards and matching the data distribution perfectly: as long as the learned policy is within the support of the data distribution, the complement will pick up the slack and model the “remainder" of the distribution, allowing the two generators together to perfectly fool the discriminator.
If however the policy ventures outside of the data support, the second generator cannot compensate for this mistake, and the discriminator will push the policy back inside the support.
We name our method DASCO, for D
ual-Generator
A dversarial S upport C onstrained O ffline RL.
Experimentally, we demonstrate the benefits of our approach, DASCO, on standard benchmark tasks.
For offline datasets that consist of a combination of expert, sub-optimal and noisy data, our method outperforms distribution-constrained offline RL methods by a large margin.
Related Work
Combining sub-optimal trajectories to obtain high-performing policies is a central promise of offline
RL.
During offline training, querying the value function on unseen actions often leads to value over-estimation and collapse in learning progress.
To avoid querying the value functions on outof-distribution actions, existing methods encourage the learned policies to match the distribution of the behavior policies.
This principle has been realized with a variety of practical algorithms [22, 49, 41, 42, 49, 29, 26, 25, 47, 12, 9, 15, 21, 37, 10, 34].
For example, by optimizing the policies with respect to a conservative lower bound of the value function estimate [30], only optimizing the policies on actions contained in the dataset [26], or jointly optimizing the policy on the long-term return and a behavior cloning objective [12].
While explicitly enforcing distribution constraint by adding the behavior cloning objective allows for good performance on near-optimal data, this approach fails to produce good trajectories on sub-optimal datasets [26].
Methods that implicitly enforce distribution constraints, such as CQL and IQL, have seen more successes on such datasets.
However, they still struggle to produce near-optimal trajectories when the actions of the dataset generation policies are corrupted with noise or systematic biases (a result we demonstrate in Section 5).
However, enforcing distribution constraints to avoid value over-estimation may not be necessary.
It is sufficient to ensure the learned policies do not produce actions that are too unlikely under the dataset generation policy.
That is, it is not necessary for the learned policy to fully cover the data distribution, only to remain in-support [29, 27, 32, 49, 51, 8].
Unfortunately, previous methods that attempt to instantiate this principle into algorithms have not seen as much empirical success as algorithms that penalize the policies for not matching the action distribution of the behavior policies.
In this paper, we propose a new GAN-based offline RL algorithm whose use of dual generators naturally induce support constraint and has competitive performance with recent offline RL methods.
In a number of prior works, GANs have been used in the context of imitation learning to learn from expert data [20, 33, 19, 35].
In this work, we show that dual-generator GANs can be used to learn from sub-optimal data in the context of offline RL.
To enforce the support constraint, we use the discriminator score to measure how likely an action is under the behavior policy.
To measure uncertainty, one could also use an ensemble of value functions, as is done in [2, 3].
We include [2] as a baseline in Table 1 and demonstrate the our method has significantly higher performance.
Nevertheless, it would be interesting as future work to combine the discriminator score and an ensemble of value functions to devise a higher performing algorithm.
Background
Let M =
( S , A , P, R, γ ) define a Markov decision process (MDP), where S and A are state and action spaces, P : S ×
A × S → R + is a state-transition probability function, R : S ×
A → R is a reward function and γ is a discount factor.
Reinforcement learning methods aim at finding a policy π ( a | s ) that maximizes the expected discounted reward R ( τ ) =
(cid:80) Tt =
γ t R ( s t , a t ) over trajectories τ = ( s 0 , a 0 , . . .
, s T , a T ) with time horizon T induced by the policy π .
In this work, we concentrate on the offline or off-policy RL setting, i.e. finding an optimal policy given a dataset D of previously collected experience τ ∼ D by a behavior policy π β .
A particularly popular family of methods for offline learning are based on training a Q-function through dynamic programming using temporal-difference (TD) learning [48, 44].
Such methods train a Q-function to satisfy the Bellman equation:
[Equation]
In Q-learning, the policy is replaced with a maximization, such that π ( a | s ) =
[Equation]
max a Q θ ( s, a ) , while actor-critic methods optimize a separate parametric policy π ϕ ( a | s ) that maximizes the Qfunction.
In this work, we extend the Soft Actor-Critic (SAC) method [18] for learning from diverse offline datasets.
Generative Adversarial Networks (GANs) [17] enable modeling a data distribution p D through an adversarial game between a generator G and a discriminator D :
[Equation]
For this two player zero-sum game, [17] shows that for a fixed generator G , the optimal discriminator is D ∗ G ( x ) =
p D ( x ) p D ( x ) + p G ( x ) and the optimal generator matches the data distribution p ∗ g ( x ) = p D .
GAN has been extended to the offline RL setting by interpreting the discriminator function as a measure of how likely an action is under the behavior policy, and jointly optimizing the policy to maximize an estimate of the long-term return and the discriminator function [49]:
[Equation]
where Q ( s, a ) is trained via the Bellman operator to approximate the value function of the policy π ( a | s ) .
This leads to iterative policy evaluation and policy improvement rules for the actor and the policy [49].
During the k th update step, given the most recent values for the policy π k , the value function Q k , and the discriminator D k , we perform the following updates to obtain the next values for the value function and the policy:
[Equation]
where the log D ( a | s ) term in the policy objective aims at regularizing the learnt policy to prevent it from outputting OOD actions.
In practice, training the policy to maximize both the value function and discriminator might lead to conflicting objectives for the policy and thus poor performance on either objective.
This can happen when the data contains a mixture of good and bad actions.
Maximizing the value function would mean avoiding low-reward behaviors.
On the other hand, maximizing the discriminator would require outputting all in-distribution actions, including sub-optimal ones.
Our approach alleviates this conflict and enables in support maximization of the value function when learning from mixed-quality datasets.
Dual-Generator Adversarial Support Constraint Offline RL
We now present our algorithm, which uses a novel dual-generator GAN in combination with a weighting method to enable GAN-based offline RL that constrains the learned policy to remain within the support of the data distribution.
We call our method Dual-generator Adversarial Support
Constraint Offline RL (DASCO) .
We will first introduce the dual-generator training method generically, for arbitrary generators that must optimize a user-specified function f ( x ) within the support of the data distribution in Section 4.1.
We will then show this method can be incorporated into a complete offline RL algorithm in Section 4.2 in combination with our proposed weighting scheme, and then summarize the full resulting actor-critic method in Section 4.3.
4.1 Dual generator in-support optimization
The mixed distribution also perfectly matches the distribution g
Note that in these two figures, the primary generator aims to maximize f ( x ) (instead of minimize) to allow for more intuitive interpretation.
In
Appendix B, we provide a numerical example to further illustrate the benefit of the dual generator technique on a 1D example.
In this section, we will develop an approach performing a joint optimization of adversarial and secondary objectives of the generator in a
GAN framework, which we will then apply to offline RL.
This is a necessary component for performing the joint optimization in Eq.
2 without introducing a conflict of these objectives.
All proofs for theorems presented in this section are in Appendix A.
Let’s consider a general objective that requires training a generator G to fool the discriminator
D while also optimizing the expected value of some other function f :
[Equation]
where the first two terms are the same as the objective of the GAN formulation.
We have also added an additional term
E z ∼ p
( z ) [ f ( G ( z ))] , where f is a mapping from the generator output to a scalar value.
The third term represents a secondary objective that the generator should optimize.
Theorem 4.1 The optimal generator of
Eq. induces a distribution
[Equation]
, where ν > 0 is the Lagrange multiplier that ensures that p ∗ g ( x ) is normalized to 1.
We can see that by adding a secondary objective function for the generator, in general, the optimal generator does not attempt to match the data distribution p D ( x ) anymore, but instead tries to match the data distribution weighted by
[Equation]
We expect that in such case, the discriminator clearly has an advantage in the two player zero-sum game and will be able to distinguish between real samples and sample generated by the generator.
To allow the generator to specialize in optimizing the secondary objective function, we propose to introduce a second auxiliary generator that matches the portion of the data distribution that is not well captured by the primary generator.
Let p mix =
p g + p aux, consider the min-max problem:
[Equation]
(5) where we mix samples from the primary generator G and the auxiliary generator G aux to generate samples that can fool the discriminator.
The mixing is indicated by the distribution p mix in the second term of Eq.
The first and third term of Eq.
5 are the same as the objective in Eq.
We next theoretically demonstrate the benefit of adding the auxiliary generator to the GAN formulation with the following Theorem.
Theorem 4.2 (Informal)
The primary generator p G performs in-support optimization of f ( x ) .
We first note that the optimal solution of the mixed distribution from Eq.
5 is the real data distribution:
[Equation]
Accordingly, the optimal auxiliary generator distribution can be expressed as
[Equation]
Let x 0 to be the element inside the support of the data distribution p D that minimizes f .
That is:
[Equation]
When optimizing the secondary objective f ( x ) , the primary generator will maximize the probability mass of in-support samples that maximize f ( x ) .
However, Eq.
7 introduces a constraint that enforces 2 p D ( x ) − p ∗ g ( x ) ≥ 0 for p ∗
aux ( x ) ≥ 0 to remain a valid distribution.
This leads us to conclude that the optimal primary generator p ∗ g assigns the following probability to x 0 :
[Equation]
otherwise [Equation] Interestingly, if the global optimum x 0 is not taking the full probability mass, the rest of the probability mass is redistributed to the next best in-support optimum, which we can define recursively:
For x i ∈ arg min
[Equation]
We provide more explanation for the solution in Eq. 9.
In the first case, p ∗ g ( x i ) =
2 p D ( x i ) if (cid:80) ij =
p ∗ g ( x j ) < 1 .
That is, if the optimal solution for the primary generator p ∗ g can assign the probability 2 p D ( x i ) to the i th in support minima of f ( x ) without the total sum of probability assigned (cid:80) ij =
p ∗ g ( x j ) going over 1 , then the primary generator p ∗ g will assign the probability 2 p D ( x i ) to x i .
In the second case, p ∗ g ( x i ) = 1 − (cid:80) i − 1 j =
p ∗ g ( x j ) if (cid:80) ij =
p ∗ g ( x j ) > 1 .
That is, if by assigning the probability 2 p D ( x i ) to the i th in support minima of f ( x ) , the total sum of probability assigned (cid:80) ij =
p ∗ g ( x j ) goes over 1 , then the primary generator p ∗ g will assign the remaining probability 1 − (cid:80) i − 1 j =
p ∗ g ( x j ) to x i .
In the third case, the generator assigns probability 0 to x i because all the probability has already been assigned.
To summarize the benefit of dual generator, we note that by introducing an auxiliary generator and mixing it with the primary generator, not only does the optimal solution for the mixed distribution match the real data distribution, but also the primary generator can better optimize the secondary objective f on the part of the domain of f that is within the support of the data distribution p D .
To better illustrate the benefit, we provide a visual explanation of the benefit in Figure 1.
4.2 Update rules for offline reinforcement learning
We will now incorporate the dual-generator method to train policies for offline RL, based on optimizing the joint objective from Eq. 5.
The updates for the actor and the critic are generally similar to
Eq. 3.
However, simply combining Eq.
5 and Eq.
3 can still allow the policy to exploit errors in the value function during the policy improvement step.
We therefore augment the policy improvement step with an adaptive weight on the Q-value.
More concretely, as the policy improvement step samples actions from the current policy iterate π k to optimize the policy objective, ’, as we observe in our experiments.
To alleviate this issue, we use the probability assigned to the sampled actions by the discriminator to weight the value function estimates in the policy objective, leading to the following updates:
[Equation]
where a D ( s ) is the action from the offline dataset.
The term D k ( s, a ) down-weights the contribution of the gradient of the value function to the policy update if the discriminator deems the sampled action too unlikely.
We further calibrate the probability
D k ( s, a ) by dividing it with the probability
D k ( s, a D ( s )) that the discriminator assigns to the dataset action a D ( s ) .
It should be noted that during optimization the gradients are not propagated into these weights.
Next, we define the update rules for the auxiliary generator and the discriminator.
We mix the samples from the k th iterate of the policy π k and the distribution p aux induced by the k th iterate of the auxiliary generator G kaux , that is, let p mix =
π k + p aux.
At every iteration k , we update the k th iterate of the auxiliary generator G kaux and discriminator D k using the objectives:
[Equation]
4.3 Algorithm summary
Algorithm 1 provides a step-by-step description of our algorithm.
At every training step, we sample a batch of transitions from the offline dataset and proceed to update the parameters of the value function, the policy, the auxiliary generator and the discriminator in that order. Algorithm 1 DASCO algorithm summary 1: Initialize , auxiliary D do 3:
Bellman auxiliary generator k + Update discriminator D mixed samples G aux, in Eq.  Experiments
Our experiments aim at answering the following questions:
When learning from offline datasets that require combining actions from sub-optimal trajectories, does DASCO outperform existing methods that are based on distribution constraints?
On standard benchmarks such as D4RL [11], how does DASCO compare against recent methods? 3.
Are both the dual generator and the probability ratio weight important for the performance of DASCO?
5.1 Comparisons on standard benchmarks and new datasets
For our first set of experiments, we introduce four new datasets to simulate the challenges one might encounter when using offline RL algorithms on real world data.
These datasets introduce additional learning challenges and require the algorithm to combine actions in different trajectories to obtain good performance.
We use the existing AntMaze environments from the D4RL suite [11]: antmaze-medium and antmaze-large.
In these two environments, the algorithm controls an 8-DoF “Ant" quadruped robot to navigate a 2D maze to reach desired goal locations.
The D4RL benchmark generates the offline datasets for these two environments using two policies: 1.
a low-level goal reaching policy that outputs torque commands to move the Ant to a nearby goal location and 2.
a high-level waypoint generator to provide sub-goals that guide the low-level goal-reaching policy to the desired location.
We use the same two policies to generate two new classes of datasets.
For the noisy dataset, we add Gaussian noise to the action computed by the low-level goal-reaching policy.
The noise variance depends on the 2D location of the Ant in the maze – larger in some 2D regions than others.
We intend this dataset to be representative of situations where the data generation
[Table] [Table] [Table] policies are more deterministic in some states than others [31] – a robot picking up an object has many good options to approach the object, but when the robot grasps the object, its behavior should be more deterministic to ensure successful grasp without damaging or dropping the object [38].
For a biased dataset, in addition to adding Gaussian noise to the actions as it is done in the noisy dataset, we also add bias to the actions computed by the low-level policy.
The values of the bias also depend on the current 2D location of the Ant in the maze.
This setting is meant to simulate learning from relabelled data, where the dataset was generated when the data generation policies were performing a different task than the tasks we are using the dataset to learn to perform.
Relabelling offline data is a popular method for improving the performance of offline RL algorithms [46, 43], especially when we have much more data for some tasks than others [23].
In the AntMaze environment, offline
RL algorithms must combine data from sub-optimal trajectories to learn behaviors with high returns.
In addition, noisy and biased datasets present a more challenging learning scenarios due to the added noise and systematic bias which vary non-uniformly based on the 2D location of the Ant.
Table 1 illustrates the performance comparison of our method and representative methods that enforce distribution constraints, either through optimizing a conservative lower bound of the value estimates (CQL) or only optimizing the policy on actions in the dataset using Advantage Weighted Regression [41] (IQL).
Our method outperforms both CQL and IQL.
In these tasks, to ensure a fair comparison between different methods, we perform oracle offline policy selection to obtain the performance estimates for CQL, IQL, and our method.
We describe how we tune hyper-parameters for the baselines in
Appendix E. We also include results of recent methods, such as EDAC [2] and BEAR [28].
We also compare the performance on standard AntMaze tasks when learning from the datasets in the D4RL benchmark without modifications in Table 2.
In these tasks, our method outperforms
IQL by a large margin on two diverse datasets.
By comparing the results in Table 1 (learning from noisy and biased datasets) and Table 2 (learning from existing offline datasets in D4RL), we also note that our proposed algorithm outperforms distribution-constraint offline RL algorithms (CQL, IQL) more consistently when tested on the noisy and biased datasets.
For the results in these two tables, the definition of the antmaze-medium and antmaze-large environments are the same.
The only axis of variation in the learning setup is the noise and systematic bias added to the actions of the dataset generation policies.
We therefore conclude that our algorithm is more robust to the noise and systematic bias added to the actions than distribution-constrained offline RL algorithms.
Next, we evaluate our approach on Gym locomotion tasks from the standard D4RL suite.
The performance results on these tasks are illustrated in Table 3.
Our method is competitive with BC, [Table] [Table] one-step offline RL methods [5], and multi-step distribution-constraint RL methods [26, 30].
This is not surprising because in these tasks, the offline dataset contains a large number of trajectories with high returns.
In Table 3, 10%BC refers to performing BC using the top 10% trajectories with the highest episode return.
5.2 Ablations
We conduct three different sets of experiments to gain more insights into our algorithm.
The first experiment measures the importance of having an auxiliary generator.
We recall that there are two benefits to having the auxiliary generator.
Firstly, without the auxiliary generator, the generator does not in general match the data distribution (Theorem 4.1).
As such, the discriminator has an unfair advantage in learning how to distinguish between real and generated examples.
Secondly, the auxiliary generator plays the role of a support player and learns to output actions that are assigned non-zero probability by the data distribution, but have low Q values.
The support player allows the policy to concentrate on in-support maximization of the Q-function (Theorem 4.2).
Tabledemonstrates that having an auxiliary generator clearly leads to a performance improvement across different task families, from Gym locomotion tasks to AntMaze navigation tasks.
The second experiment compares the performance of the policy and the auxiliary generator on a subset of the Gym locomotion and AntMaze tasks (Table 5).
The difference in the performance of the policy and auxiliary generator illustrates their specialization of responsibility: the policy learns to output actions that lead to good performance, while the auxiliary generator learns to model the “remainder" of the data distribution.
If this “remainder" also contains good action, then the auxiliary generator will have non-trivial performance.
Otherwise, the auxiliary generator will have poor performance.
In the Gym locomotion tasks, the auxiliary generator has non-trivial performance, but it is still worse than the policy.
This demonstrates that: 1.
By optimizing the policy to maximize the long-term return and the discriminator function, the policy can outperform the auxiliary generator, which only maximizes the discriminator function, 2.
The dataset contains a large fraction of medium performance level actions contained in continuous trajectories, which the auxiliary generator has learnt to output.
In contrast, in the bias and noisy AntMaze tasks, the auxiliary generator fails to obtain non-zero performance while the policy has strong performance.
This reflects the necessity of carefully picking a subset of the in-support actions to obtain good performance.
The third set of experiments illustrates the importance of weighing the value function in the policy objective by the probability computed by the discriminator, as described in Eq.
Doing so provides a second layer of protection against exploitation of errors in the value function by the policy.
Tableillustrates that this is very important for the AntMaze tasks, which require combining optimal and sub-optimal trajectories to obtain good performance.
Perhaps this is because learning from such trajectories necessitates many rounds of offline policy evaluation and improvement steps, with each round creating an opportunity for the policy to exploit the errors in the value estimates. On the other
[Table] [Table]
hand
[Table] the dynamic weight is less important in the Gym locomotion tasks, presumably because a significant fraction of the corresponding offline datasets has high returns and therefore incorporating sub-optimal data is less criticial to obtain high performance.
Conclusions
In this paper
[Table] we introduced DASCO [Table] a GAN-based offline RL method that addresses the challenges of training policies as generators with a discriminator to minimize deviation from the behavior policy by means of two modifications: an auxiliary generator to turn the GAN loss into a support constraint, and a value function weight in the policy objective.
The auxiliary generator makes it possible for the policy to focus on maximizing the value function without needing to match the entirety of the data distribution, only that part of it that has high value, effectively turning the standard distributional constraint that would be enforced by a conventional GAN into a kind of support constraint.
This technique may in fact be of interest in other settings where there is a need to maximize some objective in addition to fooling a discriminator, and applications of this approach outside of reinforcement learning are an exciting direction for future work.
Further, since our method enables GAN-based strategies to attain good results on a range of offline RL benchmark tasks, it would also be interesting in future work to consider other types of GAN losses that induce different divergence measures.
We also plan to explore robust methods for offline policy and hyper-parameter selection in the future.
Potential negative societal impact Because our focus is on developing a generic offline RL algorithm and not its application in any particular domain, our method inherits the potential negative societal impacts that any offline RL algorithm might have, such as data bias and lack of robustness to distributional shift.
We also note that how to perform effective offline policy selection and evaluation in the general case remain challenging.
Therefore, the offline learned RL policies might not be easily deployed to solve real world problems where online policy evaluation is expensive
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A] 1.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Please see the experimental results in Section 5 and avenues for future work in the Conclusion (c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discussed the potential negative impacts in the Conclusion of the paper (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We will include the proofs for all theorems in the Appendix (b) Did you include complete proofs of all theoretical results?
[Yes] We will include the proofs for all theorems in the Appendix
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will include these results in the Appendix (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We will include this information in the Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We will include this information in the Appendix (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix D 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendices A Proofs for theorems in Section 4. A.1 Proof for Theorem 4.
In the following proof, we use p data to refer to the real data distribution, instead of p D as in Section 4.1, to avoid confusion with the discriminator distribution.
We recall Theorem 4.1:
Theorem 4.1 The optimal generator of Eq.
4 induces
[Equation]
where ν > 0 is the Lagrange multiplier that ensures that p ∗ g ( x ) is normalized to 1.
The optimization problem in Eq. 4 is:
[Equation]
The proof proceeds as follows: We first simplify the objective function into two terms.
The first term is the Jensen–Shannon divergence between the data distribution and the distribution induced by the generator [16].
The second term is the expected value of the secondary objective function f .
We then show that the problem is convex, where strong duality holds.
We then use the KKT conditions to find the functional form of the optimal solution, which gives us Theorem 4.1.
We only prove the statement for discrete sample space, and we let n be the size of the sample space – the random variable x can take on n different values.
Proof .
Since the third term in the objective function is not a function of the discriminator D , for G fixed, the optimal discriminator of Eq.
4 is D ∗ G ( x ) =
p data ( x ) p data ( x ) + p g ( x ) where p g is the distribution induced by the generator G .
(similar to Prop 1 in [16] ).
Similarly to how [16] shows that the GAN objective in Eq.
1 minimizes the JS divergence between the data distribution and the distribution induced by the generator, we can now rewrite the objective in Eq.
4 as:
[Equation]
For conciseness, let g ( i ) =
p g ( x i ) be the probability that p g assigns to x i and g =
[ g (1) , . .
. , g ( n ) ] T be a column vector containing the probabilities that p g assigns to each possible values of x , from xto x n .
Similarly, let f ( i ) =
f ( x i ) be the value that the secondary objective f assigns to x i .
We also overload the notation to let f =
[ f (1) , . . .
, f ( n ) ] T be a column vector containing the values that the secondary objective f assigns to each possible value of the random variable x , from x 1 to x n .
Also let p ( i ) data = p data ( x i ) be the probability that the data distribution assigns to x i .
We can then rewrite the problem in Eq.
4 in a standard form [4] as:
[Equation]
[Equation]
where 1 is a column vector of 1 , which has the same number of entries as the vector g .
The constraint 18 ensures that the probability that p g assigns to any x is non-negative and the constraint 19 ensures the probabilities sum up to 1 .
The problem is convex because the objective function is a nonnegative weighted sum of two convex functions (JSD is convex because JSD is itself a nonnegative weighted sum of KL, which is a convex function).
Strong duality also holds because Slater’s condition holds.
A strictly feasible point for Slater’s condition to hold is the uniform distribution, i.e. g ( i ) = 1 n, ∀ i .
The Lagrangian is:
[Equation]
where λ ( i ) and ν are the Lagrangian multipliers.
For any i ∈ [1 , n ] , the partial derivative of the Lagrangian with respect to g ( i ) is:
[Equation]
Let g ∗ and ( λ ∗ , ν ∗ ) be the primal and dual optimal solutions of the optimization problem.
As the strong duality holds, the variables g ∗ and ( λ ∗ , ν ∗ ) must satisfy the KKT conditions.
For any i ∈ [1 , n ] , the following holds:
[Equation]
From Equation 26, we have λ ( i ) ∗ = log (cid:32)
[Equation]
+ f ( i ) + ν ∗ , and substitute into Equation 25:
[Equation]
We consider what happens when g ∗ i > 0 , due to complementary slackness, we have:
[Equation]
We can then pick an appropriate value for the Lagrange multiplier ν such that the probabilities p ∗ g ( x i ) normalize to 1.
QED.
A.2 Proof for Theorem 4.
In the following proof, we use p data to refer to the real data distribution, instead of p D as in Section 4.1, to avoid confusion with the discriminator distribution.
Recall that we define p mix as p mix =
p g + p aux.
Theorem 4.2 is stated in reference to the optimization problem in Eq.
5, which we restate here:
[Equation]
where the first two terms in the objective function are the GAN objective and the last term is the secondary objective function.
Similar to the proof for Theorem 4.1, we can rewrite the objective function in Eq.
31 as [16]:
[Equation]
We are only interested in optimizing for the secondary objective function f in the space of optimal
GAN solutions.
We therefore enforce that p mix =
p g + p aux= p data , which makes the JSD term vanish in Eq.
33 and allows us to solve the following optimization problem.
[Equation]
We claim that the solution to the optimization problem above is as follows.
We define x 0 to be the element inside the support of the data distribution p data that minimizes f , i.e. x 0 = arg min x ∈ Supp ( p data ) f ( x ) .
The optimal primary generator p ∗ g assigns the following probability to x 0 :
[Equation]
otherwise (37)
If the global maximum x 0 is not taking the full probability mass, the rest of the probability mass is redistributed to the next best in-support maxima, which we can define recursively:
For x i ∈ arg min
[Equation]
Proof.
We show the proof by contradiction.
That is, assume that there exists another distribution p ag with the following properties:
• There exists x where p ag ( x ) ̸
= p ∗ g ( x ) • p ag satisfies the constraint (35)-(36) •
The value of the objective function achieved by p ag is better than the value achieved by p ∗ g .
That is, E x ∼ p ag [ f ( x )] <
E x ∼ p ∗ g [ f ( x )] .
We will show that the existence of such a distribution p ag will lead to contradiction,
We separate the analyses into three different cases, depending on the property of p ∗ g :
• Case 1: p ∗ g assigns all probability mass to x• Case 2: If p ∗ g assigns non-zero probability to x , then p ∗ g = {+
+}
p data ( x ) [-> 
-]Case
We will walk through the three cases independently and show the contradiction in each case.
Case 1: p ∗ g assigns the full probability mass to x 0 , that is p ∗ g ( x 0 ) = 1 , and assigns zero probability to every x not equal to x 0 .
Without loss of generality, we consider p g that assigns non-zero probability to a x k ̸ = x 0 , assigns the remaining probability mass to x 0 , and assigns zero probability to all x that is not equal to either x 0 or x k .
That is, assume there exists p ag such that:
[Equation]
where x k ∈ Supp ( p data ) follows from constraint 35 ( p g ≤ 2 p data , and thus p ag can only assign non-zero probability to x within the support of p data ).
We can then show that:
[Equation]
where the last inequity follows from these two facts:
[Equation]
Case 2:
[Equation]
Let { x 0 , . . .
, x i } be the set of x where p ∗ g ( x ) > 0 , then we also require that (cid:80) ij =
p ∗ g ( x ) = 1 .
Without loss of generality [Equation] we assume a distribution p ag exists with the following properties.
There exists x m , x n such that:
[Equation]
and
[Equation]
and
[Equation]
We note that f ( x m ) ≤ f ( x n ) since p ∗ g assigns non-zero probability to x m and assigns zero probability to x n . We can show that:
[Equation]
[Equation]
[Equation]
[Equation]
where the last inequality is true because f ( x m ) ≤ f ( x n ) as we noted above, and p ∗ g ( x m )
=
2 p data ( x m ) > p ag ( x m ) .
Case 3:
There exists x i such that 2 p data ( x i ) > p ∗ g ( x i ) > 0 .
For all x ̸ = x i : p [Equation] g
[Equation] (cid:26) [Equation] if
[Equation]
otherwise (62)
Let { x 0 , . . .
, x i } be the set of x where p ∗ g ( x ) > 0 , we also require (cid:80) ij =
p ∗ g ( x ) = 1 .
Without loss of generality, there are three cases we need to consider for the distribution p ag , each yielding a contradiction:
• p ag ( x i ) =
p ∗ g ( x i ) , but there exists x such that p ag ( x ) ̸
= p ∗ g ( x ) .
• p ag ( x i ) > p ∗ g ( x i ) .
• p ag ( x i ) <
p ∗ g ( x i ) .
In each case, the proof by contradiction is similar to the proof in Case 2 above, where we pick a pair of x m , x n and shows that p ag can not achieve a lower value of the objective function than p ∗ g .
We thus do not repeat the argument here.
QED
B Benefits
on 1D example
In this section, we provide a simple one-dimensional numerical example with discrete action space to illustrate the benefit of the dual generator technique.
In this example, the action space only consists two actions a 0 , a 1 .
The probability of action a 0 under the data distribution is 0 .
The probability of action a 1 under the data distribution is 0 .
We would like to maximize a secondary objective function f .
The function f assigns value 1 .
3 to a 0 and value 0 .
7 to a 1 .
We will next show in a self-contained Jupyter notebook that when using only a single generator, the expected value of the secondary objective function f under the optimal generator is 1 . 15 .
We also show that when using the dual generator technique, the expected value of the secondary objective function f under the optimal primary generator is 1 . 3 .
Since 1 . 3 is higher than 1 . 15 , we can see that using the dual generator technique allows us to better maximize the objective function f .
Fig 2 visually illustrates the benefits of the dual generator technique over using only a single generator in this example.
We can also observe from Fig 2 (Left) that the optimal generator is clearly different from the data distribution when only using a single generator.
As such, in the GAN framework, the discriminator has an advantage in learning how to distinguish between samples from the real data distribution and samples from the generator.
In contrast, when using the dual generator technique, the optimal primary generator assigns probability 1 to action a 0 and the optimal auxiliary generator assigns probability 1 to action a 1 (not shown in Fig 2).
Their mixed distribution therefore assigns probability 0 .
5 to either actions, matching the data distribution.
The Jupyter notebook in the next page illustrates the computations necessary to obtain the optimal solution for the generator.
In this toy example, the action space consists of two actions, denoted by and .
The probability of action under the data distribution is . The probability of action under the data distribution is .
The data distribution is represented by the variable in the code cell below.
The secondary objective function that we would like to maximize is represented by , where and .
When using only a single generator and not employing the dual generator technique, we can write the closed form solution of the optimal generator as a function of the data distribution , the secondary objective , and a Lagrange variable as follows.
The optimal solution of the generator is represented by the variable in the code cell below:
Since the distribution induced by the optimal generator must be a valid distribution, we solve for the
Lagrange variable such that the probabilities that the optimal generator assigns to the two actions sum to .
nu_star is [1. aaa0. a0.
p data
In [2]: p_data = np . array([ 0.5, 0.
]) f f ( a 0 ) = 1.
[Equation]
In [3]: f = np . array([ 1.3, 0.
]) p data f nu
[Equation]
In [4]: def get_p_star_single_generator(p_data, f, nu): # We use f instead of -f because # we are maximizing f in this example.
exp = np .
exp( f - nu ) p_g_star = p_data
* ( exp / ( 2 - exp ) ) return p_g_star nu a 0 , a
In [5]: def func(nu): # p_g_star is a array consisting of two elements.
# The first element is the density that the optimal # generator assigns to the action a_0.
# The second element is the density that the optimal # generator assigns to the action a_ nu_star is the value of nu that makes the distribution # induced by the optimal generator a valid distribution
) print('nu_star is', nu_star)
The probabilities that the optimal generator assigns to the two actions are: [0.75378698 0.24621302]
Sum of probabilities 1. Expected value of f under the optimal generator when using a single generator is: 1.
We next derive the optimal solution of the generator when using the dual generator technique and show that using the dual generator technique allows us to better maximize the secondary objective function .
When using the dual generator technique, we maintain two generators, a primary generator denoted by and an auxiliary generator denoted by .
The in-support global maxima of is , since .
As such, the optimal solution of the primary generator, denoted by , would like to assign as much probability to as possible.
Since we have the constraint , we have that .
That is, the optimal primary generator assigns probability to action .
Since there are only two actions, the optimal primary generator assigns probability to the remaining action .
The optimal primary generator is represented by the variable in the code cell below:
Expected value of f under the optimal generator when using dual generator is:
Since we want to maximize the secondary objective function , and is higher than , we can see that using the dual generator technique allows us to better maximize the secondary objective function .
print('The probabilities that the optimal generator to the two actions are:') print(p_g_star) print('Sum of probabilities', p_g_star .
sum() .
round(2))
In [7]: E_f = (p_g_star * f) .
sum() print( 'Expected value of f under the optimal generator when using a single generator is:' ) print(E_f .
round(2))
f
[Equation]
g ( a 0 ) = 1. a0.
adual _ p _ g _ star
In [8]: dual_p_g_star = np .
array([ 1.0, 0.
]) dual_E_f = (dual_p_g_star * f) .
sum() print( 'Expected value of f under the optimal generator when using dual generator is:')
print(dual_E_f .
round(2))
f 1. f
C Description of the offline dataset generation procedure for the noisy and biased AntMaze datasets
In the experiments section, we introduce the bias and noisy datasets for the AntMaze tasks.
In this section, we provide more details on how the datasets were generated in the form of Python syntax in
Code Listing 1.
We plan to open-source both the datasets and the code to generate the datasets upon acceptance.
Code Listing 1: Illustration of the dataset generation procedure for the bias and noisy datasets.
Given an action computed by the behavior_policy , we add noise and bias to the action.
The magnitudes of the noise and bias depend on the x-values of the position of the Ant in the 2D maze.
NOISES =
[0.1, 0.0, 0.2, 0.05, 0.3, 0.1, 0.4, 0.2] BIASES = [0.1, -0.1, 0.2, 0.0, 0.2, -0.3, 0.2, 0.0]
POSITION =
[-20.0, 0.0, 4.0, 8.0, 12.0, 16.0, 20.0, 24.0] action = behavior_policy.get_action(obs) x_position
= get_x_position(obs) pos =
[idx for idx in range(len(POSITION))
if POSITION[idx] <= x_position] pos = max(pos) noise = NOISES[pos] bias = BIASES[pos] action = action + np.random.normal(size=action.shape)
* noise - bias * np.ones_like(action) action = np.clip(action , -1.0, 1.0)
D Additional experimental details
For all tasks, we average mean returns overs 20 evaluation trajectories.
Similar to the pre-processing steps in previous works [25], we standardize MuJoCo locomotion task rewards by dividing by the difference of returns of the best and worst trajectories in each dataset.
For the AntMaze datasets, we subtract 1 from rewards for all transitions.
We use Adam optimizer [24] with a learning rate of 0 . 0003 .
For the value functions, we use an MLP with 3 hidden layers of size 256 .
For both the
GAN discriminator and auxiliary generator, we use an MLP with 1 hidden layer of size 750 .
The auxiliary generator takes a state as an input, and a noise vector and output actions deterministically as a function of the input state and noise vector.
For the policy, which is also the primary generator, we use an MLP with 4 hidden layers of size 256 .
The policy takes a state as an input and outputs the parameters of a diagonal Gaussian, from which we sample an action.
We update the target network with soft updates with parameter 0 .
For the discriminator loss function, we use the mean-squared error loss, inspired by LSGAN [36].
For the auxiliary generator, we use the standard vanilla GAN loss.
The loss functions and how they are used are further illustrated in Section F.
We also use instance noise [45] where we sample the instance noise from a Gaussian distribution for each action dimension independently.
The Gaussian is zero-center and has an initial standard deviation of 0 .
3 at the beginning of training.
We anneal the magnitude of the noise over time and also clamp the instance noise to have a maximum magnitude of 0 .
We also train the discriminator for more steps than the generators in each algorithm step ( training steps for the discriminator for every step of the generator).
In the policy objective (Eq. 11), we also use a hyper-parameter w to weight the contribution of the value function and the discriminator probability to the policy update.
That is, we use Eq.
63 to update the policy.
We fix the value of w throughout training.
For the AntMaze tasks, we set w = 0 .
For the Mujoco locomotation task, we set w = 1 . 0 .
[Equation]
In terms of total amount of compute and type of resources used, we use an internal cluster that allows for access up to 64 preemptive Nvidia RTX 2080 Ti GPUs.
For each experiment of learning from an offline dataset, we use half a GPU and 3 CPU cores.
Each experiment generally takes half a day to finish.
We implemented our algorithms in Pytorch [40].
E Additional details on baselines
IQL To obtain the result of IQL when learning from the noisy and biased AntMaze datasets presented in Table
[Table]
we tune the expectile hyper-parameter τ in IQL.
Table 7 illustrates the different values of τ we ran IQL for and the performance of each value
[Table]
[Table]
CQL We also tune the Lagrange threshold in CQL when learning from the noisy and biased
AntMaze datasets.
We refer to this hyper-parameter also as tau (but this is different from the hyperparameter tau in IQL).
The results for different values of the Lagrange threshold tau for CQL are illustrated in Table 8.
[Table]
EDAC To obtain the results for EDAC presented in Table 1, we performed a sweep over the number of value function used in the ensemble and the weight η
(used in step 5 in Algorithm 1 in EDAC paper to weight the gradient penalty term ES ).
For the number of value function, we ran the sweep over 20 , 50 , 100 .
For the weight τ , we ran the sweep over the values 1 , 5 , 10 , 50 , 100 , 1000 .
In other words, we ran 3 × 6 = 18 different combinations of hyper-parameter values for EDAC.
We use the implementation of EDAC released by the original authors.
Since we discovered that EDAC does not perform well in the medium size maze, we did not obtain the results of EDAC for the large maze.
BEAR For BEAR, we used the laplacian kernel and performed a sweep over the hyper-parameter mmd _ sigma using the values 1 , 10 , 20 , 50 .
We use the implementation available publicly at https: //github.com/rail-berkeley/d4rl_evaluations .
Similarly to EDAC, because BEAR does not perform well when learning from the medium size maze datasets, we did not obtain the results of
BEAR for the large maze.
F Detailed algorithm description
Algorithm 1 provides a summary of the training step given a batch of transitions from the offline dataset.
In this section, we provide the description of how the different networks in our algorithms are trained using Python syntax.
We include four Code Listings below, each illustrating the details of an update step in Algorithm 1.
[Table]
[Table]
[Table] F
[Table]
[Table] [Table] [Table] [Table]
policy [Table] optimizer.
[Table]
[Table]
[Table]
obs = batch[’observations’] actions = batch[’actions’] b_size = obs.size(0)
# Calculate loss on real action D_real_logits
= discriminator( obs , actions + get_instance_noise(actions), return_logit= True ) real_label =
torch.full( (b_size ,), 1) errD_real
= F.mse_loss( F.sigmoid(D_real_logits), real_label ) / 2.
# Calculate loss on fake action def loss_fake_action(fake_action): fake_label =torch.full((b_size,), 0,)
D_fake_logits = discriminator( obs , fake_action.detach() + get_instance_noise(fake_action), return_logit=
True ) errD_fake = F.mse_loss
( F.sigmoid(D_fake_logits), fake_label ) / 2.
return errD_fake fake_action_aux = auxiliary_generator(obs) fake_action_policy = policy(obs) err_D_fake
= loss_fake_action(fake_action_aux) \ + loss_fake_action(fake_action_policy) # Compute gradient and update the discriminator discriminator_optimizer.zero_grad() (errD_real+err_D_fake).backward()discriminator_optimizer.update()