Original: S TATE -A CTION J OINT R EGULARIZED I MPLICIT P OLICY FOR O FFLINE R EINFORCEMENT L EARNING
Revised: S TATE -A CTION J OINT R EGULARIZED I MPLICIT P OLICY FOR O FFLINE R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Ofﬂine reinforcement learning enables learning from a ﬁxed dataset, without further interactions with the environment.
Revised: Offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment.

Original: The lack of environmental interactions makes the policy training vulnerable to state-action pairs far from the training dataset and prone to missing rewarding actions.
Revised: The lack of environmental interactions makes the policy training vulnerable to state-action pairs far from the training dataset and prone to missing rewarding actions.

Original: For training more effective agents, we propose a framework that supports learning a ﬂexible and well-regularized policy, which consists of a fully implicit policy and a regularization through the stateaction visitation frequency induced by the current policy and that induced by the data-collecting behavior policy.
Revised: For training more effective agents, we propose a framework that supports learning a flexible and well-regularized policy, which consists of a fully implicit policy and a regularization through the stateaction visitation frequency induced by the current policy and that induced by the data-collecting behavior policy.

Original: We theoretically show the equivalence between policy-matching and state-action-visitation matching and thus the compatibility of many prior work with our framework.
Revised: We theoretically show the equivalence between policy-matching and state-action-visitation matching, and thus the compatibility of many prior work with our framework.

Original: An effective instantiation of our framework through the GAN structure is provided, together with some techniques to explicitly smooth the state-action mapping for robust generalization beyond the static dataset.
Revised: An effective instantiation of our framework through the GAN structure is provided, together with some techniques to explicitly smooth the state-action mapping for robust generalization beyond the static dataset.

Original: Extensive experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs
Revised: Extensive experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Ofﬂine reinforcement learning (RL), also known as batch RL, aims at training agents from previously-collected ﬁxed datasets that are typically large and heterogeneous, with a special emphasis on no interactions with the environment during the learning process (Ernst et al., 2005; Lange et al., 2012;
Revised: Offline reinforcement learning (RL), also known as batch RL, aims at training agents from previously-collected fixed datasets that are typically large and heterogeneous, with a special emphasis on no interactions with the environment during the learning process (Ernst et al., 2005; Lange et al., 2012;

Original: Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Agarwal et al., 2020; Siegel et al., 2020; Wang et al., 2020).
Revised: Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Agarwal et al., 2020; Siegel et al., 2020; Wang et al., 2020).

Original: This paradigm extends the applicability of RL to where the environmental interactions are costly or even potentially dangerous, such as healthcare (Tseng et al., 2017;
Revised: This paradigm extends the applicability of RL to where the environmental interactions are costly or even potentially dangerous, such as healthcare (Tseng et al., 2017;

Original: Gottesman et al., 2018; Nie et al., 2019), autonomous driving (Yurtsever et al., 2020), and recommendation systems (Swaminathan et al., 2017; Gilotte et al., 2018).
Revised: Gottesman et al., 2018; Nie et al., 2019), autonomous driving (Yurtsever et al., 2020), and recommendation systems (Swaminathan et al., 2017; Gilotte et al., 2018).

Original: While (online) off-policy RL algorithms, such as DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018a) could be directly adopted in an ofﬂine setting, their application can be unsuccessful (Fujimoto et al., 2019; Kumar et al., 2019), especially in high-dimensional continuous control tasks, where function approximations are inevitable and data samples are non-exhaustive.
Revised: While (online) off-policy RL algorithms, such as DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018a) could be directly adopted in an offline setting, their application can be unsuccessful (Fujimoto et al., 2019; Kumar et al., 2019), especially in high-dimensional continuous control tasks, where function approximations are inevitable and data samples are non-exhaustive.

Original: Such failures may be attributed to the shift between the state-action visitation frequency induced by the current policy and that by the data-collecting behavior policy, where unseen state-action pairs are presented to the action-value estimator, resulting in possibly uncontrollable extrapolation errors (Fujimoto et al., 2019; Kumar et al., 2019).
Revised: Such failures may be attributed to the shift between the state-action visitation frequency induced by the current policy and that by the data-collecting behavior policy, where unseen state-action pairs are presented to the action-value estimator, resulting in possibly uncontrollable extrapolation errors (Fujimoto et al., 2019; Kumar et al., 2019).

Original: In this regard, one approach to ofﬂine RL is to control the difference between the observed and policy-induced state-action visitation frequencies, so that the current policy mostly generates state-action pairs that have reliable action-value estimate.
Revised: In this regard, one approach to offline RL is to control the difference between the observed and policy-induced state-action visitation frequencies, so that the current policy mostly generates state-action pairs that have reliable action-value estimate.

Original: Previous work in this line of research typically (1) regularizes the current policy to be close to the behavior policy during the training process, i.e. , policy or state-conditional action distribution matching; (2) uses a Gaussian policy class with a learnable mean and diagonal covariance matrix (Kumar et al., 2019; Wu et al., 2019).
Revised: Previous work in this line of research typically (1) regularizes the current policy to be close to the behavior policy during the training process, i.e. , policy or state-conditional action distribution matching; (2) uses a Gaussian policy class with a learnable mean and diagonal covariance matrix (Kumar et al., 2019; Wu et al., 2019).

Original: See Appendix A for a detailed review.
Revised: See Appendix A for a detailed review.

Original: However, at any given state s , the underlying action-value function over the action space may possess multiple local maxima.
Revised: However, at any given state s , the underlying action-value function over the action space may possess multiple local maxima.

Original: A deterministic or uni-modal stochastic policy may only capture one of the local optima and neglect lots of rewarding actions.
Revised: A deterministic or uni-modal stochastic policy may only capture one of the local optima and neglect lots of rewarding actions.

Original: An even worse situation occurs when such policies exhibit a strong mode-covering behavior, artiﬁcially inﬂating the density around the average of multiple rewarding actions that itself may be inferior.
Revised: An even worse situation occurs when such policies exhibit a strong mode-covering behavior, artificially inflating the density around the average of multiple rewarding actions that itself may be inferior.

Original: Previous work under the policy-matching theme mainly takes two approaches.
Revised: Previous work under the policy-matching theme mainly takes two approaches.

Original: One approach directly estimates the divergence between the state-conditional distributions over actions (Wu et al., 2019).
Revised: One approach directly estimates the divergence between the state-conditional distributions over actions (Wu et al., 2019).

Original: However, on tasks with continuous state space, with probability one, no state can appear in the dataset more than once.
Revised: However, on tasks with continuous state space, with probability one, no state can appear in the dataset more than once.

Original: In other words, for each observed state s i , the ofﬂine dataset has only one corresponding action a i from the behavior policy.
Revised: In other words, for each observed state s i , the offline dataset has only one corresponding action a i from the behavior policy.

Original: Thus, one is only able to use a single point to assess whether the current policy is close to the data-collecting behavior policy at any particular state, which may not well reﬂect the true divergence between the two conditional distributions.
Revised: Thus, one is only able to use a single point to assess whether the current policy is close to the data-collecting behavior policy at any particular state, which may not well reflect the true divergence between the two conditional distributions.

Original: The other approach, e.g. , Kumar et al.
Revised: The other approach, e.g. , Kumar et al.

Original: (2019), resorts to a two-step strategy:
Revised: (2019), resorts to a two-step strategy:

Original: First, ﬁt a generative model π ( a | s ) to clone the behavior policy; Second, estimate the distance between the ﬁtted behavior policy π ( a | s ) and the current policy, and minimize that distance as a way to regularize.
Revised: First, fit a generative model π ( a | s ) to clone the behavior policy; Second, estimate the distance between the fitted behavior policy π ( a | s ) and the current policy, and minimize that distance as a way to regularize.

Original: While this approach, which can acquire multiple samples from the cloned behavior policy, is able to accurately estimate the distance between the current policy and cloned behavior, its success relies heavily on how well the inferred behavior-cloning generative model mimics the true behavior policy.
Revised: While this approach, which can acquire multiple samples from the cloned behavior policy, is able to accurately estimate the distance between the current policy and cloned behavior, its success relies heavily on how well the inferred behavior-cloning generative model mimics the true behavior policy.

Original: On tasks with large or continuous state space, however, the same problem arises that each state-conditional action distribution is ﬁtted on only one data point.
Revised: On tasks with large or continuous state space, however, the same problem arises that each state-conditional action distribution is fitted on only one data point.

Original: Moreover, some prior work use conditional VAE (CVAE, Sohn et al.
Revised: Moreover, some prior work use conditional VAE (CVAE, Sohn et al.

Original: as the generative model to clone the possibly-multimodal behavior policy, which further suffers to the problem that CVAE may exhibit a strong mode-covering behavior that allocates large probability density to low data-density regions.
Revised: as the generative model to clone the possibly-multimodal behavior policy, which further suffers to the problem that CVAE may exhibit a strong mode-covering behavior that allocates large probability density to low data-density regions.

Original: Inasmuch as these weaknesses, one may naturally question on how good the samples from such a cloned policy resemble the truth, and further, on the quality of the constraint imposed by sample-based calculation of the distance between such a cloned behavior policy and the current policy.
Revised: Inasmuch as these weaknesses, one may naturally question on how good the samples from such a cloned policy resemble the truth, and further, on the quality of the constraint imposed by sample-based calculation of the distance between such a cloned behavior policy and the current policy.

Original: To address these concerns, we are motivated to develop a new framework that not only supports an expressive policy, which can be as ﬂexible as needed, but also well regularizes this ﬂexible policy towards the data-collecting behavior policy.
Revised: To address these concerns, we are motivated to develop a new framework that not only supports an expressive policy, which can be as flexible as needed, but also well regularizes this flexible policy towards the data-collecting behavior policy.

Original: Speciﬁcally, (1) instead of using the classical deterministic or uni-modal Gaussian policy, we train a fully implicit policy for its ﬂexibility to capture multiple modes in the action-value function; (2) to avoid the aforementioned potential problems in the policy-matching regularization, we directly control the distance between the state-action visitation frequency induced by the current policy and that induced by the behavior policy, as an auxiliary training target.
Revised: Specifically, (1) instead of using the classical deterministic or uni-modal Gaussian policy, we train a fully implicit policy for its flexibility to capture multiple modes in the action-value function; (2) to avoid the aforementioned potential problems in the policy-matching regularization, we directly control the distance between the state-action visitation frequency induced by the current policy and that induced by the behavior policy, as an auxiliary training target.

Original: Hence, our approach does not need to build a generative model to clone the behavior policy.
Revised: Hence, our approach does not need to build a generative model to clone the behavior policy.

Original: On the theoretical side, we prove in Section 3 that the approach of matching the behavior and current policies is equivalent to matching their corresponding state-action visitation frequencies, which reveals the compatibility of many prior work with our framework.
Revised: On the theoretical side, we prove in Section 3 that the approach of matching the behavior and current policies is equivalent to matching their corresponding state-action visitation frequencies, which reveals the compatibility of many prior work with our framework.

Original: Similar notion of matching the state-action visitations in ofﬂine RL is taken by the DICE family (Nachum et al., 2019; Lee et al., 2021a), but they either use a Gaussian policy or a mixture of Gaussian policy with a per-dataset tuned number of mixtures.
Revised: Similar notion of matching the state-action visitations in offline RL is taken by the DICE family (Nachum et al., 2019; Lee et al., 2021a), but they either use a Gaussian policy or a mixture of Gaussian policy with a per-dataset tuned number of mixtures.

Original: Besides, these algorithms have high computational complexity, which, together with inﬂexible policies and intensive hyperparameter tuning, limit their practical applicability.
Revised: Besides, these algorithms have high computational complexity, which, together with inflexible policies and intensive hyperparameter tuning, limit their practical applicability.

Original: We instantiate our framework with a GAN structure that approximately minimizes the Jenson–Shannon divergence between the visitation frequencies.
Revised: We instantiate our framework with a GAN structure that approximately minimizes the Jenson–Shannon divergence between the visitation frequencies.

Original: Furthermore, we design techniques to explicitly encourage robust behavior of our policy at states not covered in the static dataset.
Revised: Furthermore, we design techniques to explicitly encourage robust behavior of our policy at states not covered in the static dataset.

Original: We conduct ablation study on several components of our algorithm and analyze their contributions.
Revised: We conduct ablation study on several components of our algorithm and analyze their contributions.

Original: With all these considerations, our full algorithm achieves state-of-the-art performance on various tasks from the D4RL dataset (Fu et al., 2021), validating the effectiveness of our framework and implementation.
Revised: With all these considerations, our full algorithm achieves state-of-the-art performance on various tasks from the D4RL dataset (Fu et al., 2021), validating the effectiveness of our framework and implementation.

Original:  B ACKGROUND AND M OTIVATION
Revised:  B ACKGROUND AND M OTIVATION

Original: We ﬁrst present some background information and then introduce a toy example to illustrate the motivations of the proposed framework for ofﬂine RL.
Revised: We first present some background information and then introduce a toy example to illustrate the motivations of the proposed framework for offline RL.

Original: Ofﬂine RL.
Revised: Offline RL.

Original: Following the classic RL setting (Sutton & Barto, 2018), the interaction between the agent and environment is modeled as a Markov decision process (MDP), speciﬁed by the tuple M =
Revised: Following the classic RL setting (Sutton & Barto, 2018), the interaction between the agent and environment is modeled as a Markov decision process (MDP), specified by the tuple M =

Original: ( S , A , P , r, γ ) , where S denotes the state space,
Revised: ( S , A , P , r, γ ) , where S denotes the state space,

Original: A the action space, γ ∈ (0 , 1] the discount factor,
Revised: A the action space, γ ∈ (0 , 1] the discount factor,

Original: P ( s (cid:48) | s , a ) : S × S ×
Revised: P ( s ′ | s , a ) : S × S ×

Original: A → [0 , 1] the environmental dynamics, and r ( s , a ) : S ×
Revised: A → [0 , 1] the environmental dynamics, and r ( s , a ) : S ×

Original: A → [ R min , R max ] the reward function.
Revised: A → [ R min , R max ] the reward function.

Original: The goal of RL is to learn a policy π φ ( a t | s t ) , parametrized by φ , that maximizes the expected cumulative discounted reward R t (cid:44) R ( s t , a t ) =
Revised: The goal of RL is to learn a policy π ϕ ( a t | s t ) , parametrized by ϕ , that maximizes the expected cumulative discounted reward R t ≜ R ( s t , a t ) =

Original: E (cid:2)(cid:80) ∞ k =0 γ k r t + k +1 (cid:3) .
Revised: E (cid:2)(cid:80) ∞ k =0 γ k r t + k +1 (cid:3) .

Original: In ofﬂine RL (Fujimoto et al., 2019; Kumar et al., 2019; 2020; Levine et al., 2020), the agent only has access to a ﬁxed dataset D (cid:44) { ( s , a , r, s (cid:48) ) } , consisting of transition tuples from rollouts by some behavior policies π b ( a | s ) .
Revised: In offline RL (Fujimoto et al., 2019; Kumar et al., 2019; 2020; Levine et al., 2020), the agent only has access to a fixed dataset D ≜ { ( s , a , r, s ′ ) } , consisting of transition tuples from rollouts by some behavior policies π b ( a | s ) .

Original: We denote the state-action visitation frequency induced by the behavior policy π b as d b ( s , a ) and its state-marginal, the state visitation frequency, as d b ( s ) .
Revised: We denote the state-action visitation frequency induced by the behavior policy π b as d b ( s , a ) and its state-marginal, the state visitation frequency, as d b ( s ) .

Original: Similarly, d φ ( s , a ) and d φ ( s ) are the counterparts for the current policy π φ .
Revised: Similarly, d ϕ ( s , a ) and d ϕ ( s ) are the counterparts for the current policy π ϕ .

Original: Here, d b ( s , a ) =
Revised: Here, d b ( s , a ) =

Original: d b ( s ) π b ( a | s ) and we assume D ∼ d b ( s , a ) .
Revised: d b ( s ) π b ( a | s ) and we assume D ∼ d b ( s , a ) .

Original: The visitation frequencies in the dataset are denoted as d D ( s , a ) and d D ( s ) , which are discrete approximations to d b ( s , a ) and d b ( s ) , respectively.
Revised: The visitation frequencies in the dataset are denoted as d D ( s , a ) and d D ( s ) , which are discrete approximations to d b ( s , a ) and d b ( s ) , respectively.

Original: Actor-Critic Algorithm.
Revised: Actor-Critic Algorithm.

Original: Denote Q π ( s , a ) =
Revised: Denote Q π ( s , a ) =

Original: E [ (cid:80) [Equation] r ( s t , a {+t ) | s 
+}
Revised: E [ (cid:80) ∞ t =0 γ t r ( s t , a {+t ) | s 
+}

Original: [Equation]
Revised: =

Original: ) | s 0 = s , a 0 = a ] as the action-value function.
Revised: s , a 0 = a ] as the action-value function.

Original: In the actor-critic scheme (Sutton & Barto, 2018), the critic Q π ( s , a ) is often approximated by a neural network Q θ ( s , a ) , parametrized by θ and trained by applying the Bellman operator (Lillicrap et al., 2016; Haarnoja et al., 2018a; Fujimoto et al., 2019) as
Revised: In the actor-critic scheme (Sutton & Barto, 2018), the critic Q π ( s , a ) is often approximated by a neural network Q θ ( s , a ) , parametrized by θ and trained by applying the Bellman operator (Lillicrap et al., 2016; Haarnoja et al., 2018a; Fujimoto et al., 2019) as

Original: [Equation]
Revised: [Equation]

Original: The actor π φ aims at maximizing the expected value of Q θ , with the training objective expressed as
Revised: The actor π ϕ aims at maximizing the expected value of Q θ , with the training objective expressed as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where d φ ( s ) is the state visitation frequency under policy π φ .
Revised: where d ϕ ( s ) is the state visitation frequency under policy π ϕ .

Original: In ofﬂine RL, sampling from d φ ( s ) is infeasible as no interactions with the environment are allowed.
Revised: In offline RL, sampling from d ϕ ( s ) is infeasible as no interactions with the environment are allowed.

Original: A common and practically effective approximation (Fu et al., 2019; Levine et al., 2020) to Equation 1 is
Revised: A common and practically effective approximation (Fu et al., 2019; Levine et al., 2020) to Equation 1 is

Original: [Equation]
Revised: [Equation]

Original: where sampling from d b ( s ) can be implemented easily as sampling from the ofﬂine dataset D .
Revised: where sampling from d b ( s ) can be implemented easily as sampling from the offline dataset D .

Original: Generative Adversarial Nets.
Revised: Generative Adversarial Nets.

Original: GAN (Goodfellow et al., 2014) provides a framework to train deep generative models, with two neural networks jointly trained in an adversarial manner: a generator
Revised: GAN (Goodfellow et al., 2014) provides a framework to train deep generative models, with two neural networks jointly trained in an adversarial manner: a generator

Original: G φ , parametrized by φ , that ﬁts the data distribution and a discriminator D w , parametrized by w , that outputs the probability of a sample coming from the training data rather than G φ .
Revised: G ϕ , parametrized by ϕ , that fits the data distribution and a discriminator D w , parametrized by w , that outputs the probability of a sample coming from the training data rather than G ϕ .

Original: Samples x ’s from the generator’s distribution d φ ( x ) are drawn via z ∼ p z ( z ) , x = G φ ( z ) , where p z ( z ) is some noise distribution.
Revised: Samples x ’s from the generator’s distribution d ϕ ( x ) are drawn via z ∼ p z ( z ) , x = G ϕ ( z ) , where p z ( z ) is some noise distribution.

Original: Both G φ and D w are trained via a two-player min-max game as
Revised: Both G ϕ and D w are trained via a two-player min-max game as

Original: [Equation]
Revised: [Equation]

Original: where d D ( · ) is the data distribution.
Revised: where d D ( · ) is the data distribution.

Original: Given the optimal discriminator D ∗ G at G φ , the training objective of G φ is determined by the Jensen–Shannon divergence (JSD) (Lin, 1991) between d D and d φ as V ( D ∗ G , G φ )
Revised: Given the optimal discriminator D ∗ G at G ϕ , the training objective of G ϕ is determined by the Jensen–Shannon divergence (JSD) (Lin, 1991) between d D and d ϕ as V ( D ∗ G , G ϕ )

Original: = − log 4 + 2 · JSD ( d D (cid:107) d φ ) , with the global minimum achieved if and only if d φ = d D .
Revised: = − log 4 + 2 · JSD ( d D ∥ d ϕ ) , with the global minimum achieved if and only if d ϕ = d D .

Original: Therefore, one may view GAN as a distributional matching framework that approximately minimizes the Jensen–Shannon divergence between the generator distribution and data distribution.
Revised: Therefore, one may view GAN as a distributional matching framework that approximately minimizes the Jensen–Shannon divergence between the generator distribution and data distribution.

Original: Motivations.
Revised: Motivations.

Original: To illustrate our motivations of training an expressive policy under an appropriate regularization, we conduct a toy experiment of behavior cloning, as shown in Figure 1, where we use the x - and y -axis values to represent the state and action, respectively.
Revised: To illustrate our motivations of training an expressive policy under an appropriate regularization, we conduct a toy experiment of behavior cloning, as shown in Figure 1, where we use the x - and y -axis values to represent the state and action, respectively.

Original: Figure 1a illustrates the state-action joint distribution of the data-collecting behavior policy that we try to mimic.
Revised: Figure 1a illustrates the state-action joint distribution of the data-collecting behavior policy that we try to mimic.

Original: For
Revised: For

Original: Figures 1b-1e, we use the same test-time state marginal distribution, which consists of an equal mixture of the behavior policy’s state distribution and a uniform state distribution between − 1 . 5 and 1 . 5 .
Revised: Figures 1b-1e, we use the same test-time state marginal distribution, which consists of an equal mixture of the behavior policy’s state distribution and a uniform state distribution between − 1 . 5 and 1 . 5 .

Original: If the inferred policy well approaches the behavior policy, we expect (1) clear concentration on the eight centers and (2) smooth interpolation between centers, which implies a good and smooth ﬁt to the behavior policy.
Revised: If the inferred policy well approaches the behavior policy, we expect (1) clear concentration on the eight centers and (2) smooth interpolation between centers, which implies a good and smooth fit to the behavior policy.

Original: We start the toy experiment with ﬁtting a CVAE model, a representative behavior-cloning method, to the dataset.
Revised: We start the toy experiment with fitting a CVAE model, a representative behavior-cloning method, to the dataset.

Original: As shown in Figure 1b, CVAE exhibits a mode-covering behavior that covers the data density modes at the expense of overestimating unwanted low data density regions.
Revised: As shown in Figure 1b, CVAE exhibits a mode-covering behavior that covers the data density modes at the expense of overestimating unwanted low data density regions.

Original: Hence, the regularization ability is questionable of using CVAE as a proxy for the behavior policy in some prior work.
Revised: Hence, the regularization ability is questionable of using CVAE as a proxy for the behavior policy in some prior work.

Original: Replacing CVAE with the conditional GAN (CGAN, Mirza & Osindero (2014)), i.e. , replacing the KL loss with JSD loss, but adopting the Gaussian policy popular in prior ofﬂine RL work partially alleviates the issues but drops necessary modes, as shown in Figure 1c.
Revised: Replacing CVAE with the conditional GAN (CGAN, Mirza & Osindero (2014)), i.e. , replacing the KL loss with JSD loss, but adopting the Gaussian policy popular in prior offline RL work partially alleviates the issues but drops necessary modes, as shown in Figure 1c.

Original: This shows the inﬂexibility of Gaussian policies.
Revised: This shows the inflexibility of Gaussian policies.

Original: Replacing the Gaussian policy in CGAN with an implicit policy, while still training CGAN via sampled-based policy-matching, improves the capability of capturing multiple modes, as shown in Figure 1d.
Revised: Replacing the Gaussian policy in CGAN with an implicit policy, while still training CGAN via sampled-based policy-matching, improves the capability of capturing multiple modes, as shown in Figure 1d.

Original: Nevertheless, it is still prone to mode collapse and interpolates less smoothly between the seen states.
Revised: Nevertheless, it is still prone to mode collapse and interpolates less smoothly between the seen states.

Original: Finally, training the implicit-policy CGAN via direct state-action-visitation joint matching leads to the best performance.
Revised: Finally, training the implicit-policy CGAN via direct state-action-visitation joint matching leads to the best performance.

Original: As shown in Figure 1e, it concentrates clearly on the eight centers and interpolates smoothly between the seen states.
Revised: As shown in Figure 1e, it concentrates clearly on the eight centers and interpolates smoothly between the seen states.

Original: Thus, constraining the state-action visitations can be an effective way to regularize implicit policies in ofﬂine RL.
Revised: Thus, constraining the state-action visitations can be an effective way to regularize implicit policies in offline RL.

Original: These observations motivate us to train implicit policies in ofﬂine
Revised: These observations motivate us to train implicit policies in offline

Original: RL, with sample-based regularization on the state-action visitations.
Revised: RL, with sample-based regularization on the state-action visitations.

Original: (a) Truth 1.5 1.0 0.5 0.0 0.5 1.0 1.5    (b) CVAE 1.5 1.0 0.5 0.0 0.5 1.0 1.5 0.
Revised: (a) Truth 1.5 1.0 0.5 0.0 0.5 1.0 1.5    (b) CVAE 1.5 1.0 0.5 0.0 0.5 1.0 1.5 0.

Original: (c) G-CGAN 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1.
Revised: (c) G-CGAN 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1.

Original: (d) CGAN 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1.
Revised: (d) CGAN 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1.

Original: (e) GAN
Revised: (e) GAN

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: As discussed in Section 1 and detailed in Appendix A, one common theme in prior work in ofﬂine
Revised: As discussed in Section 1 and detailed in Appendix A, one common theme in prior work in offline

Original: RL is controlling the distance between the behavior policy and current policy during the training process.
Revised: RL is controlling the distance between the behavior policy and current policy during the training process.

Original: In this section, we prove that this is equivalent to controlling the corresponding stateaction visitation frequencies.
Revised: In this section, we prove that this approach, in essence, controls the corresponding state-action visitations.

Original: Consequently, this line of prior  workis compatible in theory with our proposed framework.
Revised: This analysis theoretically links a line of prior offline-RL work with our proposed framework, manifesting the generality of our algorithmic idea.

Original: Note that d φ ( s , a ) =
Revised: Note that d ϕ ( s , a ) =

Original: d φ ( s ) π φ ( a | s ) and similarly for d b ( s , a ) .
Revised: d ϕ ( s ) π ϕ a | s ) and similarly for d b ( s , a ) .

Original: Hence, it is sufﬁcient to show the closeness between d φ ( s ) and d b ( s ) when π φ ( a | s ) is close to π b ( a | s ) .
Revised: Hence, it is sufficient to show the closeness between d ϕ ( s ) and d b ( s ) when π ϕ ( a | s ) is close to π b ( a | s ) .

Original: Below we give our analysis for the matrix (ﬁnite state space) case.
Revised: Below we give our analysis for the matrix (finite state space) case.

Original: Continuous state-space cases may be analyzed similarly and are left for future work.
Revised: Continuous state-space cases may be analyzed similarly and are left for future work.

Original: The Proofs are deferred to Appendix C.
Revised: The Proofs are deferred to Appendix C.

Original: Notation.
Revised: Notation.

Original: Denote A − i ∗ as matrix A with its i -th row removed; κ ( A ) as the 2-norm condition number of A ; 1 as a row vector of all ones and I as an identity matrix, both with an appropriate dimension.
Revised: Denote A − i ∗ as matrix A with its i -th row removed; κ ( A ) as the 2-norm condition number of A ; 1 as a row vector of all ones and I as an identity matrix, both with an appropriate dimension.

Original: Assume that the state space S is ﬁnite with cardinality N , i.e. , S = (cid:0) s 1 , . . .
Revised: Assume that the state space S is finite with cardinality N , i.e. , S = (cid:0) s 1 , . . .

Original: , s N (cid:1) .
Revised: , s N (cid:1) .

Original: The transition probabilities associated with policy π φ over S is then an N × N matrix T φ , whose ( i, j ) entry is T
Revised: The transition probabilities associated with policy π ϕ over S is then an N × N matrix T ϕ , whose ( i, j ) entry is T

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: | s

Original: [Equation]
Revised: [Equation]

Original:  , and similarly for T b , the transition matrix associated with π b .
Revised: d a t , and similarly for T b , the transition matrix associated with π b .

Original: Note that in this case, d φ ( s ) , d b ( s ) are vectors and we denote d φ (cid:44) d φ ( s ) , d b (cid:44) d b ( s ) ∈ R N and d φ = d b + ∆ d .
Revised: Note that in this case, d ϕ ( s ) , d b ( s ) are vectors and we denote d ϕ ≜ d ϕ ( s ) , d b ≜ d b ( s ) ∈ R N and d ϕ = d b + ∆ d .

Original: Theorem 1.
Revised: Theorem 1.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: If
Revised: If

Original: [Equation]

Original: <

Original: [Equation]
Revised: [Equation]

Original:  and T φ i,j , T bi,j > 0 , ∀ i, j ∈ { 1 , 2 , . . . , N } , then
Revised: max and T ϕ i,j , T bi,j > 0 , ∀ i, j ∈ { 1 , 2 , . . . , N } , then

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: (1) We note that κ max is a constant for ﬁxed T b and can be calculated by iteratively removing columns of T b and computing the SVD of the referred matrix.
Revised: (1) We note that κ max is a constant for fixed T b and can be calculated by iteratively removing columns of T b and computing the SVD of the referred matrix.

Original: The assumption that
Revised: The assumption that

Original: T φ i,j , T bi,j > 0 , ∀ i, j ∈ { 1 , . . 
Revised: T ϕ i,j , T bi,j > 0 , ∀ i, j ∈ { 1 , . . .

Original: , N } can be satisﬁed by substituting the zero entries in the original transition matrix with a small number and re-normalized each row of the resulting matrix, as in the
Revised: , N } can be satisfied by substituting the zero entries in the original transition matrix with a small number and re-normalized each row of the resulting matrix, as in the

Original: PageRank algorithm (Page et al., 1998; Langville & Meyer, 2004).
Revised: PageRank algorithm (Page et al., 1998; Langville & Meyer, 2004).

Original: In practice, the ofﬂine dataset D often consists of samples collected by several policies.
Revised: In practice, the offline dataset D often consists of samples collected by several policies.

Original: Equivalently, the behavior policy π b ( · | s ) is a mixture of single policies.
Revised: Equivalently, the behavior policy π b ( · | s ) is a mixture of single policies.

Original: Assume that there are K such policies { π b k ( · | s ) }
Revised: Assume that there are K such policies { π b k ( · | s ) }

Original: Kk =1 with mixture probabilities { w k } Kk =1 , i.e. , π b ( · | s ) =
Revised: Kk =1 with mixture probabilities { w k } Kk =1 , i.e. , π b ( · | s ) =

Original: (cid:80) K k
Revised: (cid:80) K k

Original: =1 w k π b k ( · | s ) , (cid:80) K k
Revised: =1 w k π b k ( · | s ) , (cid:80) K k

Original: =

Revised: =

Original: w k = 1 .
Revised: w k = 1 .

Original: Since we collect D by running each π b k ( · | s ) a proportion of w k of total time, we may decompose D as D =
Revised: Since we collect D by running each π b k ( · | s ) a proportion of w k of total time, we may decompose D as D =

Original: (cid:83) Kk =1 D k , where D k consists of w k proportion of data in D .
Revised: (cid:83) Kk =1 D k , where D k consists of w k proportion of data in D .

Original: Thus, d D ( s ) =
Revised: Thus, d D ( s ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk
Revised: Kk

Original: =1 w k d D k ( s ) and the approximation d φ ( s ) ≈
Revised: =1 w k d D k ( s ) and the approximation d ϕ ( s ) ≈

Original: d D ( s ) has population version d φ ( s )
Revised: d D ( s ) has population version d ϕ ( s )

Original: ≈
Revised: ≈

Original: (cid:80)
Revised: (cid:80)

Original: Kk
Revised: Kk

Original: =1 w k d b k 
Revised: =1 w k d b k ( s ) ≜

Original: ( s ) (cid:44) d b ( s ) .
Revised: d b ( s ) .

Original: As before, denote d b k ( s ) ∈ R N as the limiting stateoccupancy measure induced by π b k on M ; T b k as the transition matrix induced by π b k over S ; and d φ = d b k + ∆ d k .
Revised: As before, denote d b k ( s ) ∈ R N as the limiting stateoccupancy measure induced by π b k on M ; T b k as the transition matrix induced by π b k over S ; and d ϕ = d b k + ∆ d k .

Original: We extend Theorem 1 into this mixture of policies case as follows.
Revised: We extend Theorem 1 into this mixture of policies case as follows.

Original: Theorem 2.
Revised: Theorem 2.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: and T φ i,j , T b k i,j > 0 , ∀ i, j ∈ { 1 , . . . , N } , k ∈ { 1 , . . .
Revised: and T ϕ i,j , T b k i,j > 0 , ∀ i, j ∈ { 1 , . . . , N } , k ∈ { 1 , . . .

Original: , K } , then
Revised: , K } , then

Original: [Equation]
Revised: [Equation]

Original: In particular, if w k = 1 /K, ∀ k ∈ { 1 , . .
Revised: In particular, if w k = 1 /K, ∀ k ∈ { 1 , . .

Original: , K } , then
Revised: , K } , then

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: We note that κ max,k is a constant for ﬁxed T b k and κ max is a constant for ﬁxed { π b k } Kk =1 .
Revised: We note that κ max,k is a constant for fixed T b k and κ max is a constant for fixed { π b k } Kk =1 .

Revised: We notice that similar analysis has been given in the prior work of bounding D KL ( d ϕ ( s )

Revised: ∥

Revised: d b ( s )) by

Revised: O (cid:0) ϵ/ (1 − γ ) 2 (cid:1) (Schulman et al., 2015; Levine et al., 2020).

Revised: However, this bound depends on 1 − γ on the denominator and hence cannot be applied to the undiscounted MDP (discount factor γ = 1 ), and can be loose when γ is close to 1, which is typically the case in the default hyperparameter of prior offline-RL algorithms.

Original:   S TATE -A CTION J OINT R EGULARIZED I MPLICIT P OLICY
Revised: By contrast, our bound avoids the dependency on γ and hence can apply equally well on both discounted and undiscounted MDP. In this sense, our theoretical result is more general. Besides, instead of the KL divergence, we bound the total variation distance between the state visitation frequencies, which is a well-defined metric.  S TATE -A CTION J OINT R EGULARIZED I MPLICIT P OLICY

Original: In this section we discuss an instance of our framework that will be used in our empirical study in Section 5.
Revised: In this section we discuss an instance of our framework that will be used in our empirical study in Section 5.

Original: Concretely, we train a fully implicit policy via a GAN structure to approximately minimizes the JSD between the state-action visitation frequency induced by the current policy and that induced by the behavior policy.
Revised: Concretely, we train a fully implicit policy via a GAN structure to approximately minimizes the JSD between the state-action visitation frequency induced by the current policy and that induced by the behavior policy.

Original: Our basic algorithm is discussed in Section 4.1, followed by three enhancing components presented in Section 4.2 to build up our full algorithm.
Revised: Our basic algorithm is discussed in Section 4.1, followed by three enhancing components presented in Section 4.2 to build up our full algorithm.

Original: This instantiation manifests three facets we consider important in ofﬂine RL: (1) the ﬂexibility of the policy class, (2) an effective sample-based regularization without explicitly modelling the behavior policy, and (3) smoothness of the learned policy.
Revised: This instantiation manifests three facets we consider important in offline RL: (1) the flexibility of the policy class, (2) an effective sample-based regularization without explicitly modelling the behavior policy, and (3) smoothness of the learned policy.

Original: 4. B ASIC A LGORITHM
Revised: 4. B ASIC A LGORITHM

Original: Motivated by the standard actor-critic and GAN frameworks described in Section 2, our basic algorithm consists of a critic Q θ , an actor π φ , and a discriminator D w .
Revised: Motivated by the standard actor-critic and GAN frameworks described in Section 2, our basic algorithm consists of a critic Q θ , an actor π ϕ , and a discriminator D w .

Original: For training stability, we adopt the double Q-learning (Hasselt, 2010) and target network formulation to train a pair of critics
Revised: For training stability, we adopt the double Q-learning (Hasselt, 2010) and target network formulation to train a pair of critics

Original: Q θ 1 , Q θ 2 and the target networks Q θ (cid:48) 1 , Q θ (cid:48) 2 , π φ (cid:48) .
Revised: Q θ 1 , Q θ 2 and the target networks Q θ ′ 1 , Q θ ′ 2 , π ϕ ′ .

Original: To penalize uncertainty in the action-value estimates at future states, while controlling conservatism, we follow Fujimoto et al.
Revised: To penalize uncertainty in the action-value estimates at future states, while controlling conservatism, we follow Fujimoto et al.

Original: and Kumar et al.
Revised: and Kumar et al.

Original: (2019) to use the following critic-training target:
Revised: (2019) to use the following critic-training target:

Original: [Equation]
Revised: [Equation]

Original: with some hyperparameter λ ∈ [0 , 1] , where one a (cid:48) is sampled at each s (cid:48) in the mini-batch.
Revised: with some hyperparameter λ ∈ [0 , 1] , where one a ′ is sampled at each s ′ in the mini-batch.

Original: The training objective for both critic networks is minimizing the mean-squared-error between their respective action-value estimates Q θ j ( s , a ) and (cid:101) Q ( s , a ) over state-action pairs in the mini-batch.
Revised: The training objective for both critic networks is minimizing the mean-squared-error between their respective action-value estimates Q θ j ( s , a ) and (cid:101) Q ( s , a ) over state-action pairs in the mini-batch.

Original: Our actor consists of three components: an implicit policy, state-action-visitation joint regularization, and a conservative training target.
Revised: Our actor consists of three components: an implicit policy, state-action-visitation joint regularization, and a conservative training target.

Original: Implicit Policy.
Revised: Implicit Policy.

Original: As discussed in Sections 1 and 2, a deterministic or Gaussian policy may miss important rewarding actions, or even concentrate on inferior average actions.
Revised: As discussed in Sections 1 and 2, a deterministic or Gaussian policy may miss important rewarding actions, or even concentrate on inferior average actions.

Original: For online off-policy
Revised: For online off-policy

Original: RL, Yue et al.
Revised: RL, Yue et al.

Original: (2020) shows the beneﬁt of introducing an implicit distribution mixed Gaussian policy.
Revised: (2020) shows the benefit of introducing an implicit distribution mixed Gaussian policy.

Original: Generalizing that idea to ofﬂine RL, we train a fully implicit policy, which transforms a given noise distribution into the state-conditional action distribution via a neural network, in reminiscent of the generator in CGAN.
Revised: Generalizing that idea to offline RL, we train a fully implicit policy, which transforms a given noise distribution into the state-conditional action distribution via a neural network, in reminiscent of the generator in CGAN.

Original: Speciﬁcally, given s ,
Revised: Specifically, given s ,

Original: [Equation]
Revised: [Equation]

Original: where π φ is a deterministic function and p z ( z ) is some noise distribution.
Revised: where π ϕ is a deterministic function and p z ( z ) is some noise distribution.

Original: As shown in Figures 1d and 1e, an implicit policy has stronger capability of learning a multi-modal policy, if needed.
Revised: As shown in Figures 1d and 1e, an implicit policy has stronger capability of learning a multi-modal policy, if needed.

Original: State-action Joint Regularization.
Revised: State-action Joint Regularization.

Original: As discussed in Section 2, directly sampling from d φ ( s ) is infeasible in ofﬂine RL.
Revised: As discussed in Section 2, directly sampling from d ϕ ( s ) is infeasible in offline RL.

Original: Motivated by Equation 2, we approximate d φ ( s ) by d b ( s ) in our implementation, which allows an easy sampling scheme that simply samples s from D , without needing an importance sampling correction that may possess high variance (Liu et al., 2018).
Revised: Motivated by Equation 2, we approximate d ϕ ( s ) by d b ( s ) in our implementation, which allows an easy sampling scheme that simply samples s from D , without needing an importance sampling correction that may possess high variance (Liu et al., 2018).

Original: Such an approximation is classical in the off-policy and ofﬂine RL literature (Degris et al., 2012; Levine et al., 2020), with demonstrated empirical effectiveness in ofﬂine RL (Fu et al., 2019), besides its usage in off-policy RL (Silver et al., 2014; Schulman et al., 2015; Lillicrap et al., 2016).
Revised: Such an approximation is classical in the off-policy and offline RL literature (Degris et al., 2012; Levine et al., 2020), with demonstrated empirical effectiveness in offline RL (Fu et al., 2019), besides its usage in off-policy RL (Silver et al., 2014; Schulman et al., 2015; Lillicrap et al., 2016).

Original: With this simpliﬁcation , we can efﬁciently maximize the similarity between d b ( s , a ) and d φ ( s , a ) with respect to sample-based estimate of some statistical divergence, such as the Jensen–Shannon divergence.
Revised: With this simplification , we can efficiently maximize the similarity between d b ( s , a ) and d ϕ ( s , a ) with respect to sample-based estimate of some statistical divergence, such as the Jensen–Shannon divergence.

Original: Using notations in GAN, the generator sample x and the data sample y are deﬁned as
Revised: Using notations in GAN, the generator sample x and the data sample y are defined as

Original: [Equation]
Revised: [Equation]

Original: We then constraint this statistical divergence value, named generator loss L g ( φ ) , in the training of actor.
Revised: We then constraint this statistical divergence value, named generator loss L g ( ϕ ) , in the training of actor.

Original: In our instantiation of approximately minimizing JSD via GAN, L g = E x [log (1 − D w ( x ))] .
Revised: In our instantiation of approximately minimizing JSD via GAN, L g = E x [log (1 − D w ( x ))] .

Original: Our choice of directly matching state-action joint visitations mitigates the issue of uncontrollable extrapolation errors in the action-value function estimate, since proximity of visitation frequencies leads to reduced chances of estimating the action-values of state-action pairs far from the ofﬂine dataset.
Revised: Our choice of directly matching state-action joint visitations mitigates the issue of uncontrollable extrapolation errors in the action-value function estimate, since proximity of visitation frequencies leads to reduced chances of estimating the action-values of state-action pairs far from the offline dataset.

Original: Furthermore, the problem in the policy-matching approach of ﬁtting each state-conditional action distributions on only one data point can be circumvented, since state-action pairs ( s i , a i ) in the ofﬂine dataset are all viewed as samples from the joint visitation frequency, instead of each pair being separately viewed as one sample from the state-conditional distribution, i.e. , a i ∼ π ( · | s i ) .
Revised: Furthermore, the problem in the policy-matching approach of fitting each state-conditional action distributions on only one data point can be circumvented, since state-action pairs ( s i , a i ) in the offline dataset are all viewed as samples from the joint visitation frequency, instead of each pair being separately viewed as one sample from the state-conditional distribution, i.e. , a i ∼ π ( · | s i ) .

Original: Besides, the state-action-visitation joint matching approach implicitly encourages the smoothness of the state-action mapping, namely, similar states should have similar actions.
Revised: Besides, the state-action-visitation joint matching approach implicitly encourages the smoothness of the state-action mapping, namely, similar states should have similar actions.

Original: This is because, for example, the discriminator in GAN can easily discriminate as “fake” a generator sample x should it has state similar to a data sample but action very different from.
Revised: This is because, for example, the discriminator in GAN can easily discriminate as “fake” a generator sample x should it has state similar to a data sample but action very different from.

Original: This smoothness feature helps ensure a reliable generalization of our policy to unseen states.
Revised: This smoothness feature helps ensure a reliable generalization of our policy to unseen states.

Original: Actor-Training Target.
Revised: Actor-Training Target.

Original: To prevent the accumulation of accidental errors in the training process and in the approximation of action-value function, we adopt the strategy in Kumar et al.
Revised: To prevent the accumulation of accidental errors in the training process and in the approximation of action-value function, we adopt the strategy in Kumar et al.

Original: (2019) to train the policy with respect to a conservative estimate of the action-values.
Revised: (2019) to train the policy with respect to a conservative estimate of the action-values.

Original: Speciﬁcally, we exploit the double-Q structure and use the minimum of the two action-value estimates for policy improvement.
Revised: Specifically, we exploit the double-Q structure and use the minimum of the two action-value estimates for policy improvement.

Original: For the ease of optimization, we use the Lagrange form of the constraint optimization problem and penalize the generator loss L g ( φ ) while improving the policy.
Revised: For the ease of optimization, we use the Lagrange form of the constraint optimization problem and penalize the generator loss L g ( ϕ ) while improving the policy.

Original: Our policy-training target is
Revised: Our policy-training target is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where α is a ﬁxed Lagrange multiplier.
Revised: (7) where α is a fixed Lagrange multiplier.

Original: At the test time, we follow Fujimoto et al.
Revised: At the test time, we follow Fujimoto et al.

Original: and Kumar et al.
Revised: and Kumar et al.

Original: (2019) to ﬁrst sample 10 actions from π φ and then execute the action that maximizes Q θ 1 .
Revised: (2019) to first sample 10 actions from π ϕ and then execute the action that maximizes Q θ 1 .

Original: The discriminator is trained to better distinguish samples from d φ ( s , a ) and from d b ( s , a ) .
Revised: The discriminator is trained to better distinguish samples from d ϕ ( s , a ) and from d b ( s , a ) .

Original: It aids the matching of the state-action-visitation frequencies through outputting L g ( φ ) .
Revised: It aids the matching of the state-action-visitation frequencies through outputting L g ( ϕ ) .

Original: As an example, for approximately minimizing JSD via GAN, the discriminator outputs the probability that the input, either the x or y in Equation 6, comes from d b ( s , a ) .
Revised: As an example, for approximately minimizing JSD via GAN, the discriminator outputs the probability that the input, either the x or y in Equation 6, comes from d b ( s , a ) .

Original: In this case, the discriminator is trained to minimize the error in assigning x as “fake” and y as “true,” the inner maximization of Equation 3.
Revised: In this case, the discriminator is trained to minimize the error in assigning x as “fake” and y as “true,” the inner maximization of Equation 3.

Original: 4. E NHANCING C OMPONENTS
Revised: 4. E NHANCING C OMPONENTS

Original: In this section we present three enhancing components for further improving our basic algorithm.
Revised: In this section we present three enhancing components for further improving our basic algorithm.

Original: Multiple Action-samples at Bellman Backup.
Revised: Multiple Action-samples at Bellman Backup.

Original: The expectation part of the critic learning target in
Revised: The expectation part of the critic learning target in

Original: Equation 4 can be better estimated, in terms of a smaller sample variance, by averaging over N a actions a (cid:48) at each s (cid:48) in the mini-batch, rather than just one a (cid:48) as in Section 4.1.
Revised: Equation 4 can be better estimated, in terms of a smaller sample variance, by averaging over N a actions a ′ at each s ′ in the mini-batch, rather than just one a ′ as in Section 4.1.

Original: State-smoothing at Bellman Backup.
Revised: State-smoothing at Bellman Backup.

Original: Due to the stochastic nature of environmental transition, multiple next states s (cid:48) are possible after taking action a at state s , while the ofﬂine dataset D only contains one such s (cid:48) .
Revised: Due to the stochastic nature of environmental transition, multiple next states s ′ are possible after taking action a at state s , while the offline dataset D only contains one such s ′ .

Original: Since the agent is unable to interact with environment to collect more data in ofﬂine RL, local exploration (Sinha et al., 2021) in the state-space appears as an effective strategy to regularize the Bellman backup by taking consideration of states close to the records in the ofﬂine dataset.
Revised: Since the agent is unable to interact with environment to collect more data in offline RL, local exploration (Sinha et al., 2021) in the state-space appears as an effective strategy to regularize the Bellman backup by taking consideration of states close to the records in the offline dataset.

Original: We assume that: (1) a small transformation to a state results in states physically plausible in the underlying environment (as in Sinha et al.
Revised: We assume that: (1) a small transformation to a state results in states physically plausible in the underlying environment (as in Sinha et al.

Original: ; (2) when the state space is continuous, the transition kernel P ( · | s , a ) is locally continuous and centers at the recorded s (cid:48) in the dataset.
Revised: ; (2) when the state space is continuous, the transition kernel P ( · | s , a ) is locally continuous and centers at the recorded s ′ in the dataset.

Original: With these assumptions, we propose to ﬁt Q θ ( s , a ) on the value of a small region around the recorded next state s (cid:48) .
Revised: With these assumptions, we propose to fit Q θ ( s , a ) on the value of a small region around the recorded next state s ′ .

Original: Speciﬁcally, with a pre-speciﬁed standard deviation σ B , we sample around s (cid:48) as ˆ s = s (cid:48)
Revised: Specifically, with a pre-specified standard deviation σ B , we sample around s ′ as ˆ s = s ′

Original: + (cid:15) , (cid:15) ∼ N ( 0 , σ 2 B I ) , and modify Equation 4 as
Revised: + ϵ , ϵ ∼ N ( 0 , σ 2 B I ) , and modify Equation 4 as

Original: [Equation]
Revised: [Equation]

Original: where N B ˆ s are sampled to estimate the expectation.
Revised: where N B ˆ s are sampled to estimate the expectation.

Original: This strategy is equivalent to using a Gaussian distribution centered at s (cid:48) to approximate the otherwise non-smooth δ s (cid:48) transition kernel manifested in the ofﬂine dataset.
Revised: This strategy is equivalent to using a Gaussian distribution centered at s ′ to approximate the otherwise non-smooth δ s ′ transition kernel manifested in the offline dataset.

Original: Similar technique is also considered as the target policy smoothing regularization in Fujimoto et al.
Revised: Similar technique is also considered as the target policy smoothing regularization in Fujimoto et al.

Original: (2018), though smoothing therein is applied on the target action.
Revised: (2018), though smoothing therein is applied on the target action.

Original: State-smoothing at Joint-matching.
Revised: State-smoothing at Joint-matching.

Original: As described in Section 4.1, we approximate d φ ( s ) by d D ( s ) in the sample-based estimate of the chosen statistical distance.
Revised: As described in Section 4.1, we approximate d ϕ ( s ) by d D ( s ) in the sample-based estimate of the chosen statistical distance.

Original: However, d D ( s ) is in essence discrete and the idea of smoothing the discrete state-distribution can be applied again to provide a better coverage of the state space.
Revised: However, d D ( s ) is in essence discrete and the idea of smoothing the discrete state-distribution can be applied again to provide a better coverage of the state space.

Original: This design explicitly encourages a predictable and smooth behavior at states unseen in the ofﬂine dataset.
Revised: This design explicitly encourages a predictable and smooth behavior at states unseen in the offline dataset.

Original: Speciﬁcally, with some pre-speciﬁed σ 2 J , we modify the sampling scheme of ˜ s in Equation 6 as
Revised: Specifically, with some pre-specified σ 2 J , we modify the sampling scheme of ˜ s in Equation 6 as

Original: [Equation]
Revised: [Equation]

Original: Our strategy is akin to sampling from a kernel density approximation (Wasserman, 2006) of d φ ( s ) with data points s ∈ D and with radial basis kernel of bandwidth σ J .
Revised: Our strategy is akin to sampling from a kernel density approximation (Wasserman, 2006) of d ϕ ( s ) with data points s ∈ D and with radial basis kernel of bandwidth σ J .

Original: Algorithm 1 shows the main steps of our algorithm, instantiated by approximately minimizing JSD via GAN, and a detailed version is in Appendix B. Implementation details are in Appendix D.2.[Table] [Table] [Table]
Revised: Algorithm 1 shows the main steps of our algorithm, instantiated by approximately minimizing JSD via GAN, and a detailed version is in Appendix B. Implementation details are in Appendix D.2.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: As discussed in Section 1, we consider it important the ﬂexibility of the policy class, the wellregularization and the smoothness of the learned policy.
Revised: As discussed in Section 1, we consider it important the flexibility of the policy class, the wellregularization and the smoothness of the learned policy.

Original: To this end, we develop our state-action joint regularized implicit policy, which is a framework that supports training an expressive implicit policy via an effective sample-based regularization without an explicit modelling of the behavior policy, and with techniques encouraging its smoothness.
Revised: To this end, we develop our state-action joint regularized implicit policy, which is a framework that supports training an expressive implicit policy via an effective sample-based regularization without an explicit modelling of the behavior policy, and with techniques encouraging its smoothness.

Original: In this section we will test an instantiation of our framework on the continuous-control RL tasks.
Revised: In this section we will test an instantiation of our framework on the continuous-control RL tasks.

Original: Speciﬁcally, we will ﬁrst show the effectiveness of implicit policy, state-action-visitation matching, and our full algorithm (Section 5.1).
Revised: Specifically, we will first show the effectiveness of implicit policy, state-action-visitation matching, and our full algorithm (Section 5.1).

Original: We then show in ablation study (Section 5.2) the contributions of several building blocks.
Revised: We then show in ablation study (Section 5.2) the contributions of several building blocks.

Original: Datasets.

Original: We use the continuous control tasks provided by the D4RL dataset (Fu et al., 2021) to conduct algorithmic evaluations.

Original: Due to limited computational resources, we select therein the “medium-expert,” “medium-replay,” and “medium” datasets for the Hopper, HalfCheetah, Walker2d tasks in the Gym-MuJoCo domain, which are commonly used benchmarks in prior work (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Kumar et al., 2020).

Original: We do not test on the “random” and “expert” datasets as they are less practical.

Original: We also consider the Maze2D domain 1 of tasks for the non-Markovian data-collecting policy and the Adroit tasks 2 (Rajeswaran et al., 2018) for their sparse reward-signal and high dimensionality.

Original: Implementation.
Revised: Implementation.

Original: We use GAN to approximately control the Jensen–Shannon divergence between the state-action visitation frequencies.
Revised: We use GAN to approximately control the Jensen–Shannon divergence between the state-action visitation frequencies.

Original: Our source code builds on the ofﬁcial BCQ repository and largely follows its network architectures.
Revised: Our source code builds on the official BCQ repository and largely follows its network architectures.

Original: Our implementation of the GAN structure and the hyperparameter choices follow the literature (Goodfellow et al., 2014; Mirza & Osindero, 2014; Radford et al., 2016; Salimans et al., 2016; White, 2016; Goodfellow, 2017).
Revised: Our implementation of the GAN structure and the hyperparameter choices follow the literature (Goodfellow et al., 2014; Mirza & Osindero, 2014; Radford et al., 2016; Salimans et al., 2016; White, 2016; Goodfellow, 2017).

Original: To mimic a hyperparameteragnostic setting, we minimize hyperparameter tuning across datasets.
Revised: To mimic a hyperparameteragnostic setting, we minimize hyperparameter tuning across datasets.

Original: Implementation details and hyperparameter setting of our full algorithm with GAN joint-matching is in Appendix D.2.1.
Revised: Implementation details and hyperparameter setting of our full algorithm with GAN joint-matching is in Appendix D.2.1.

Original: 5. M AIN R ESULTS
Revised: M AIN R ESULTS

Original: To validate the effectiveness of our framework, we test three implementations of the GAN instantiation: (1) basic algorithm (Section 4.1) regularized by the classical policy-matching 3 (“GANCond:Basic”), (2) basic algorithm regularized by the state-action-visitation joint matching (“GANJoint:Basic”), (3) full algorithm, which adds state-smoothing techniques onto our basic algorithm (“GAN-Joint”).
Revised: To validate the effectiveness of our framework, we test three implementations of the GAN instantiation: (1) basic algorithm (Section 4.1) regularized by the classical policy-matching 1 (“GANCond:Basic”), (2) basic algorithm regularized by the state-action-visitation joint matching (“GANJoint:Basic”), (3) full algorithm, which adds state-smoothing techniques onto our basic algorithm (“GAN-Joint”).

Original: We compare them with the state-of-the-art (SOTA) ofﬂine-RL algorithms: BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019) (BRAC-v: value penalty), BCQ (Fujimoto et al., 2019), and CQL (Kumar et al., 2020); together with ofﬂine soft actor-critic (SAC) (Haarnoja et al., 2018a).
Revised: We compare them with the state-of-the-art (SOTA) offline-RL algorithms: BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019) (BRAC-v: value penalty), BCQ (Fujimoto et al., 2019), and CQL (Kumar et al., 2020); together with offline soft actor-critic (SAC) (Haarnoja et al., 2018a).

Original: We re-run CQL using the ofﬁcial source code (details in Appendix D.2.2).
Revised: We re-run CQL using the official source code (details in Appendix D.2.2).

Original: Results for other algorithms are from Fu et al.
Revised: Results for other algorithms are from Fu et al.

Original: Table 1 validates the effectiveness of our full algorithm, together with the efﬁcacy of the implicit policy, state-action-visitation matching, and state-smoothing.
Revised: Table 1 validates the effectiveness of our full algorithm, together with the efficacy of the implicit policy, state-action-visitation matching, and state-smoothing.

Original: Our full algorithm on average outperforms the baseline algorithms, and its performance is relatively stable across tasks and datasets that possess diverse nature.
Revised: Our full algorithm on average outperforms the baseline algorithms, and its performance is relatively stable across tasks and datasets that possess diverse nature.

Original: Our full algorithm especially shows robust and comparatively-good performance on the high-dimensional Adroit tasks and the Maze2D tasks that are collected by non-Markovian policies, which may traditionally be considered as hard in ofﬂine RL.
Revised: Our full algorithm especially shows robust and comparatively-good performance on the high-dimensional Adroit tasks and the Maze2D tasks that are collected by non-Markovian policies, which may traditionally be considered as hard in offline RL.

Original: On the Gym-MuJoCo domain, our full algorithm shows its ability to learn from datasets collected by a mixture of behavior policies, and from medium-quality examples, which are likely the datasets to encounter in real-world application.
Revised: On the Gym-MuJoCo domain, our full algorithm shows its ability to learn from datasets collected by a mixture of behavior policies, and from medium-quality examples, which are likely the datasets to encounter in real-world application.

Original: These results support our design of implicit policy, matching the state-action visitation frequencies, and explicit state-smoothing.
Revised: These results support our design of implicit policy, matching the state-action visitation frequencies, and explicit state-smoothing.

Original: Comparing “GAN-Cond:Basic” with the baseline algorithms, especially BEAR and CQL that use
Revised: Comparing “GAN-Cond:Basic” with the baseline algorithms, especially BEAR and CQL that use

Original: Gaussian policies, we see that an implicit policy does in general help the performance.
Revised: Gaussian policies, we see that an implicit policy does in general help the performance.

Original: This aligns with our intuition in Sections 1 and 2 of the incapability of the uni-modal Gaussian policy in capturing multiple action-modes.
Revised: This aligns with our intuition in Sections 1 and 2 of the incapability of the uni-modal Gaussian policy in capturing multiple action-modes.

Original: To verify the gain of our joint-visitation-matching approach over the classical policy matching, apart from the comparison between our full algorithm with BEAR and BRAC, the SOTA policy-matching algorithms, we further compare “GAN-Joint:Basic” with “GAN-Cond:Basic.”
Revised: To verify the gain of our joint-visitation-matching approach over the classical policy matching, apart from the comparison between our full algorithm with BEAR and BRAC, the SOTA policy-matching algorithms, we further compare “GAN-Joint:Basic” with “GAN-Cond:Basic.”

Original: We see that on 11 out of 16 datasets, “GAN-Joint:Basic” wins against “GAN-Cond:Basic,” while results on other datasets are close.
Revised: We see that on 11 out of 16 datasets, “GAN-Joint:Basic” wins against “GAN-Cond:Basic,” while results on other datasets are close.

Original: This empirical result may be attributed to the advantage of state-action-visitation matching, e.g. , good regularization and smoothness in the state-action mapping (Section 4.1).
Revised: This empirical result may be attributed to the advantage of state-action-visitation matching, e.g. , good regularization and smoothness in the state-action mapping (Section 4.1).

Original: Comparing “GAN-Joint” with “GAN-Joint:Basic,” we see that our state-smoothing techniques do in general help the performance.
Revised: Comparing “GAN-Joint” with “GAN-Joint:Basic,” we see that our state-smoothing techniques do in general help the performance.

Original: This performance gain may be related to a smoother action-choice at states not covered by the ofﬂine dataset, and a more regularized Bellman backup.
Revised: This performance gain may be related to a smoother action-choice at states not covered by the offline dataset, and a more regularized Bellman backup.

Original: Note that different datasets may require different smoothing strength, and thus this comparison can potentially be more signiﬁcant should one is allowed to per-dataset tune the smoothing hyperparameter.
Revised: Note that different datasets may require different smoothing strength, and thus this comparison can potentially be more significant should one is allowed to per-dataset tune the smoothing hyperparameter.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: The ablation study serves to answer the following questions: (a): Is implicit policy better than
Revised: The ablation study serves to answer the following questions: (a): Is implicit policy better than

Original: Gaussian policy in our framework?
Revised: Gaussian policy in our framework?

Original: (b): Does state-smoothing at joint-matching help?
Revised: (b): Does state-smoothing at joint-matching help?

Original: (c): Does state-smoothing at Bellman backup matter?
Revised: (c): Does state-smoothing at Bellman backup matter?

Original: (d): How important is the standard deviation of the
Revised: (d): How important is the standard deviation of the

Original: Gaussian noise injected in state-smoothing?
Revised: Gaussian noise injected in state-smoothing?

Original: Unless stated otherwise, hyperparameters for all algorithmic variants in all datasets are in Table 2.
Revised: Unless stated otherwise, hyperparameters for all algorithmic variants in all datasets are in Table 2.

Original: (a): We compare the performance of our basic algorithm with its variant where the implicit policy therein is replaced by a Gaussian policy.
Revised: (a): We compare the performance of our basic algorithm with its variant where the implicit policy therein is replaced by a Gaussian policy.

Original: To make a fair comparison, the critics, discriminator, and hyperparameter setting remain the same.
Revised: To make a fair comparison, the critics, discriminator, and hyperparameter setting remain the same.

Original: Technical details are in Appendix D.2.3.
Revised: Technical details are in Appendix D.2.3.

Original: Table 4 presents the results.
Revised: Table 4 presents the results.

Original: On 11 out of 16 datasets, our basic GAN-joint-matching algorithm has higher average return than the Gaussian policy variant.
Revised: On 11 out of 16 datasets, our basic GAN-joint-matching algorithm has higher average return than the Gaussian policy variant.

Original: This empirical result coincides with our 
Revised: This empirical result coincides with our intuition in Section 4.1 and results in Section 5.1 that a Gaussian policy is less flexible to capture all

Original: [Table]
Revised: [Table]

Original: intuition in Section 4.1 and results in Section 5.1 that a Gaussian policy is less ﬂexible to capture all the rewarding actions, of which an implicit policy is likely to be capable.
Revised: the rewarding actions, of which an implicit policy is likely to be capable.

Original: Appendix F visualizes this comparison and shows in plots that Gaussian policies do leave out action modes in ofﬂine RL.
Revised: Appendix F visualizes this comparison and shows in plots that Gaussian policies do leave out action modes in offline RL.

Original: (b): We compare our full algorithm with its variant of no state-smoothing in the matching of stateaction visitations.
Revised: (b): We compare our full algorithm with its variant of no state-smoothing in the matching of stateaction visitations.

Original: The results are shown in Table 5, where our full algorithm overall performs better than the no state-smoothing variant.
Revised: The results are shown in Table 5, where our full algorithm overall performs better than the no state-smoothing variant.

Original: This performance gain may be attributed to a better coverage of the state-space by the smoothed state-distribution (Section 4.2), which is related to a more predictable and smoother action choice at unseen states.
Revised: This performance gain may be attributed to a better coverage of the state-space by the smoothed state-distribution (Section 4.2), which is related to a more predictable and smoother action choice at unseen states.

Original: As stated previously, this comparison can potentially be more signiﬁcant should one is allowed to per-dataset tune the σ J parameter.
Revised: As stated previously, this comparison can potentially be more significant should one is allowed to per-dataset tune the σ J parameter.

Original: (c): We compare our full algorithm with its variant of no state-smoothing in the Bellman backup.
Revised: (c): We compare our full algorithm with its variant of no state-smoothing in the Bellman backup.

Original: Table 6 shows the results.
Revised: Table 6 shows the results.

Original: Again, overall our full algorithm performs better than the no state-smoothing version, showing the beneﬁt of smoothing the empirical transition kernel δ s (cid:48) (Section 4.2), e.g. , taking the stochasticity of state-transitions into account.
Revised: Again, overall our full algorithm performs better than the no state-smoothing version, showing the benefit of smoothing the empirical transition kernel δ s ′ (Section 4.2), e.g. , taking the stochasticity of state-transitions into account.

Original: As before, we use the same smoothing strength across all datasets, while a per-dataset tuning of the σ B parameter may improve the distinction.
Revised: As before, we use the same smoothing strength across all datasets, while a per-dataset tuning of the σ B parameter may improve the distinction.

Original: (d): To simplify hyperparameter tuning, in actual implementation we ﬁx σ B = σ J (cid:44) σ (see Appendix D.2.1).
Revised: (d): To simplify hyperparameter tuning, in actual implementation we fix σ B = σ J ≜ σ (see Appendix D.2.1).

Original: To test the robustness to the σ hyperparameter, we run our full algorithm under σ ∈ (cid:8) 1 × 10 − 2 , 3 × 10 − 3 , 1 × 10 − 3 , 3 × 10 − 4 , 1 × 10 − 4 , 0 (cid:9) .
Revised: To test the robustness to the σ hyperparameter, we run our full algorithm under σ ∈ (cid:8) 1 × 10 − 2 , 3 × 10 − 3 , 1 × 10 − 3 , 3 × 10 − 4 , 1 × 10 − 4 , 0 (cid:9) .

Original: Table 7 shows the normalized returns.
Revised: Table 7 shows the normalized returns.

Original: We see that our algorithm is relatively insensitive to the setting of σ , especially in the range σ ∈ (cid:2) 1 × 10 − 4 , 1 × 10 − 3 (cid:1) where the performance varies little with σ .
Revised: We see that our algorithm is relatively insensitive to the setting of σ , especially in the range σ ∈ (cid:2) 1 × 10 − 4 , 1 × 10 − 3 (cid:1) where the performance varies little with σ .

Original: A too-small σ cannot provide enough smoothing to the state distributions.
Revised: A too-small σ cannot provide enough smoothing to the state distributions.

Original: On the contrary, a too-large σ may highly distort the information contained in the ofﬂine dataset, such as the state-transition kernel.
Revised: On the contrary, a too-large σ may highly distort the information contained in the offline dataset, such as the state-transition kernel.

Original: In both cases, a degradation in the overall performance is expected.
Revised: In both cases, a degradation in the overall performance is expected.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we develop a framework that supports learning a ﬂexible while well-regularized policy in ofﬂine RL.
Revised: In this paper, we develop a framework that supports learning a flexible while well-regularized policy in offline RL.

Original: Speciﬁcally, we train a fully implicit policy via regularization on the difference between the state-action visitation frequency induced by the current policy and that induced by the data-collecting behavior policy.
Revised: Specifically, we train a fully implicit policy via regularization on the difference between the state-action visitation frequency induced by the current policy and that induced by the data-collecting behavior policy.

Original: An effective instantiation of our framework through the GAN structure is provided for approximately minimizing the JSD between the visitation frequencies.
Revised: An effective instantiation of our framework through the GAN structure is provided for approximately minimizing the JSD between the visitation frequencies.

Original: Other divergence metrics, such as the MMD, may also be applied and are left for future work.
Revised: Other divergence metrics, such as the MMD, may also be applied and are left for future work.

Original: We further augment our algorithm with explicit state-smoothing techniques to enhance its generalizability on states beyond the ofﬂine dataset.
Revised: We further augment our algorithm with explicit state-smoothing techniques to enhance its generalizability on states beyond the offline dataset.

Original: On the theoretical side, we show the equivalence between policymatching and state-action-visitation matching, and hence the compatibility of many prior algorithms with our framework.
Revised: On the theoretical side, we show the equivalence between policymatching and state-action-visitation matching, and hence the compatibility of many prior algorithms with our framework.

Original: We note that our implementation-wise simpliﬁcation of approximating current policy’s state visitation frequency by the behavior’s can be improved by a more tactful approximation of current policy’s state occupancy measure, which is left for future work.
Revised: We note that our implementation-wise simplification of approximating current policy’s state visitation frequency by the behavior’s can be improved by a more tactful approximation of current policy’s state occupancy measure, which is left for future work.

Original: Nevertheless, the effectiveness of our framework and implementations is validated through extensive experiment and ablation study on the D4RL dataset.
Revised: Nevertheless, the effectiveness of our framework and implementations is validated through extensive experiment and ablation study on the D4RL dataset.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To help reproduce our empirical work, we provide detail algorithmic description in Appendix B and implementation details, including the hyperparameter choice and model architecture, in Appendix D.
Revised: To help reproduce our empirical work, we provide detail algorithmic description in Appendix B and implementation details, including the hyperparameter choice and model architecture, in Appendix D.

Original: For reproducing our theoretical results, a step-by-step proof is provided in Appendix C
Revised: For reproducing our theoretical results, a step-by-step proof is provided in Appendix C

Original: A R ELATED W ORK
Revised: A R ELATED W ORK

Original: Ofﬂine Reinforcement Learning.
Revised: Offline Reinforcement Learning.

Original: Three major themes currently exist in ofﬂine-RL research.
Revised: Three major themes currently exist in offline-RL research.

Original: The ﬁrst focuses on more robustly estimate the action-value function (Agarwal et al., 2020; Gulcehre et al., 2021) or provide a conservative estimate of the Q-values (Kumar et al., 2020;
Revised: The first focuses on more robustly estimate the action-value function (Agarwal et al., 2020; Gulcehre et al., 2021) or provide a conservative estimate of the Q-values (Kumar et al., 2020;

Original: Yu et al., 2021;
Revised: Yu et al., 2021;

Original: Sinha et al., 2021), which may better guide the policy optimization process.
Revised: Sinha et al., 2021), which may better guide the policy optimization process.

Original: The second research theme aims at designing a tactful behavior-cloning scheme so as to learn only from “good” actions in the ofﬂine dataset (Wang et al., 2020; Chen et al., 2021).
Revised: The second research theme aims at designing a tactful behavior-cloning scheme so as to learn only from “good” actions in the offline dataset (Wang et al., 2020; Chen et al., 2021).

Original: Most similar to our approach, the third line of research tries to constraint the current policy to be close to the behavior policy during the training process, under the notion that Q-value estimates at unfamiliar state-action pairs can be pathologically worse due to a lack of supervised training.
Revised: Most similar to our approach, the third line of research tries to constraint the current policy to be close to the behavior policy during the training process, under the notion that Q-value estimates at unfamiliar state-action pairs can be pathologically worse due to a lack of supervised training.

Original: Speciﬁcally, Kumar et al.
Revised: Specifically, Kumar et al.

Original: and Wu et al.
Revised: and Wu et al.

Original: (2021) use conditional variational autoencoder (CVAE) (Kingma & Welling, 2013; Sohn et al., 2015) to train a behavior cloning policy to sample multiple actions at each state for calculating the MMD constraint.
Revised: (2021) use conditional variational autoencoder (CVAE) (Kingma & Welling, 2013; Sohn et al., 2015) to train a behavior cloning policy to sample multiple actions at each state for calculating the MMD constraint.

Original: Wu et al.
Revised: Wu et al.

Original: (2019), Siegel et al.
Revised: (2019), Siegel et al.

Original: (2020), and Cang et al.
Revised: (2020), and Cang et al.

Original: ﬁt a (Gaussian) behavioral prior to the ofﬂine dataset trained by (weighted) maximum likelihood objective.
Revised: (2021) fit a (Gaussian) behavioral prior to the offline dataset trained by (weighted) maximum likelihood objective.

Original: Jaques et al.
Revised: Jaques et al.

Original: (2019) consider a pre-trained generative prior of human dialog data before applying KL-control to the current policy.
Revised: (2019) consider a pre-trained generative prior of human dialog data before applying KL-control to the current policy.

Original: Note that these work essentially constraint the distance between the current policy and the cloned behavior policy.
Revised: Note that these work essentially constraint the distance between the current policy and the cloned behavior policy.

Original: Laroche & Trichelair (2019) assume a known stochastic data-collecting behavior policy.
Revised: Laroche & Trichelair (2019) assume a known stochastic data-collecting behavior policy.

Original: Fujimoto et al.
Revised: Fujimoto et al.

Original: and Urp ´ ı et al.
Revised: and Urp ´ ı et al.

Original: (2021) implicitly control the state-conditional action distribution by decomposing action into a behavior cloning component, trained by ﬁtting a CVAE onto the ofﬂine data, and a perturbation component, trained to optimize the (risk-averse) returns.
Revised: (2021) implicitly control the state-conditional action distribution by decomposing action into a behavior cloning component, trained by fitting a CVAE onto the offline data, and a perturbation component, trained to optimize the (risk-averse) returns.

Original: Besides, some work, such as Wu et al.
Revised: Besides, some work, such as Wu et al.

Original: (2019), directly estimates and regularizes the divergence between the state-conditional action distributions.
Revised: (2019), directly estimates and regularizes the divergence between the state-conditional action distributions.

Original: In the paper, we instead take on the perspective of matching the state-action joint visitation frequencies, which avoids using a single point to estimate the divergence between distribution and removes the need for a good approximator to the behavior policy, while implicitly smooths the mapping from state to actions (Section 1).
Revised: In the paper, we instead take on the perspective of matching the state-action joint visitation frequencies, which avoids using a single point to estimate the divergence between distribution and removes the need for a good approximator to the behavior policy, while implicitly smooths the mapping from state to actions (Section 1).

Original: Note that if we use the same states in constructing the generator samples and the data samples, i.e. , ˜ s
Revised: Note that if we use the same states in constructing the generator samples and the data samples, i.e. , ˜ s

Original: = s in Equation 6, our implementation of state-action-visitation matching is fundamentally equivalent to matching the state-conditional action distributions, as in some prior work.
Revised: = s in Equation 6, our implementation of state-action-visitation matching is fundamentally equivalent to matching the state-conditional action distributions, as in some prior work.

Original: Online Off-policy Reinforcement Learning.
Revised: Online Off-policy Reinforcement Learning.

Original: A large class of modern online off-policy deep reinforcement learning algorithms trains the policy using experience replay buffer (Lin, 1992), which is a storage of the rollouts of past policies encountered in the training process (Mnih et al., 2013;
Revised: A large class of modern online off-policy deep reinforcement learning algorithms trains the policy using experience replay buffer (Lin, 1992), which is a storage of the rollouts of past policies encountered in the training process (Mnih et al., 2013;

Original: Lillicrap et al., 2016; Haarnoja et al., 2017; Fujimoto et al., 2018; Haarnoja et al., 2018b; Kuznetsov et al., 2020; Yue et al., 2020; Lee et al., 2021b).
Revised: Lillicrap et al., 2016; Haarnoja et al., 2017; Fujimoto et al., 2018; Haarnoja et al., 2018b; Kuznetsov et al., 2020; Yue et al., 2020; Lee et al., 2021b).

Original: This approach essentially use the state-visitation frequency of past policies to approximate that of the current policy (Equation 2).
Revised: This approach essentially use the state-visitation frequency of past policies to approximate that of the current policy (Equation 2).

Original: We adopt this notion in both the policy improvement step and in the implementation of state-action-visitation joint matching, with additional smoothing techniques to encourage the smoothness of the empirical transition kernel and the approximated state-visitation frequency of current policy (Section 4.2).
Revised: We adopt this notion in both the policy improvement step and in the implementation of state-action-visitation joint matching, with additional smoothing techniques to encourage the smoothness of the empirical transition kernel and the approximated state-visitation frequency of current policy (Section 4.2).

Original: Computational Distribution Matching.
Revised: Computational Distribution Matching.

Original: Many computationally efﬁcient algorithms exist to match two probability distributions with respective to some statistical divergence.
Revised: Many computationally efficient algorithms exist to match two probability distributions with respective to some statistical divergence.

Original: GAN (Goodfellow et al., 2014) approximately minimizes the Jensen-Shannon divergence between the the model’s distribution and the data-generating distribution.
Revised: GAN (Goodfellow et al., 2014) approximately minimizes the Jensen-Shannon divergence between the the model’s distribution and the data-generating distribution.

Original: Similar adversarial training strategy is further applied to estimate a class of statistical divergence, termed the integral probability metrics (M¨uller, 1997), in a sample-based manner.
Revised: Similar adversarial training strategy is further applied to estimate a class of statistical divergence, termed the integral probability metrics (M¨uller, 1997), in a sample-based manner.

Original: For example, Arjovsky et al.
Revised: For example, Arjovsky et al.

Original: Gulrajani et al.
Revised: Gulrajani et al.

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: (2018) estimate the Wasserstein-1 distance by enforcing the Lipschitz norm of the witness function to be bounded by 1 .
Revised: (2018) estimate the Wasserstein-1 distance by enforcing the Lipschitz norm of the witness function to be bounded by 1 .

Original: Li et al.
Revised: Li et al.

Original: Binkowski et al.
Revised: Binkowski et al.

Original: (2018) consider Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) with learnable kernels.
Revised: (2018) consider Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) with learnable kernels.

Original: Bellemare et al.
Revised: Bellemare et al.

Original: (2017) studies the energy distance, an instance of the MMD (Sejdinovic et al., 2013).
Revised: (2017) studies the energy distance, an instance of the MMD (Sejdinovic et al., 2013).

Original: Another important class of statistical divergence, the optimal transport distance, is also studied under a computational lens.
Revised: Another important class of statistical divergence, the optimal transport distance, is also studied under a computational lens.

Original: Peyr´e &
Revised: Peyr´e &

Original: Cuturi (2019) summarizes recent developments in computational optimal transport.
Revised: Cuturi (2019) summarizes recent developments in computational optimal transport.

Original: Cuturi (2013);
Revised: Cuturi (2013);

Original: Feydy et al.
Revised: Feydy et al.

Original: (2019) study sinkhorn-based divergence, which solve an entropic regularized form of the optimal transport problem (Schr ¨ odinger, 1932) and may result in a physically more feasible distance measure in some real-world application.
Revised: (2019) study sinkhorn-based divergence, which solve an entropic regularized form of the optimal transport problem (Schr ¨ odinger, 1932) and may result in a physically more feasible distance measure in some real-world application.

Original: Zheng & Zhou (2021) contributes to this line of research by developing conditional transport algorithm, which possesses a better control over the mode-covering and mode-seeking behaviors in distributional matching.
Revised: Zheng & Zhou (2021) contributes to this line of research by developing conditional transport algorithm, which possesses a better control over the mode-covering and mode-seeking behaviors in distributional matching.

Original: In this paper, we consider the classical GAN structure to approximately control the JSD between the state-action visitation frequencies, since the GAN structure is simple, effective and well-studied.
Revised: In this paper, we consider the classical GAN structure to approximately control the JSD between the state-action visitation frequencies, since the GAN structure is simple, effective and well-studied.

Original: Other divergence metrics may also be applicable in our framework and are left for future work.
Revised: Other divergence metrics may also be applicable in our framework and are left for future work.

Original: B
Revised: B

Original: F ULL A LGORITHM
Revised: F ULL A LGORITHM

Original: Calculate
Revised: Calculate

Original: (cid:101)Q ( s a ) (cid:44) r ( s + γ [Table]
Revised: (cid:101)

Original: [Equation]
Revised: [Equation]

Original: [Table] Minimize the critic loss with respect to θ j , j = 1 , 2 , over ( s , a ) ∈ B , with learning rate η θ , 
Revised: Minimize the critic loss with respect to θ j , j = 1 , 2 , over ( s , a ) ∈ B , with learning rate η θ , arg [Equation] s , a )

Original: [Equation]
Revised: [Equation]

Revised: , a )

Revised: [Equation]

Original: { // Policy Improvement }
Revised: { // Policy Improvement }

Original: Resample |B| new states ˜ s ∼ D independent of s .
Revised: Resample |B| new states ˜ s ∼ D independent of s .

Original: Add state-smoothing to ˜ s with noise standard deviation σ J using (cid:15) ∼ N (cid:0) 0 , σ 2 J I (cid:1) , ˜ s ← ˜ s + (cid:15) .
Revised: Add state-smoothing to ˜ s with noise standard deviation σ J using ϵ ∼ N (cid:0) 0 , σ 2 J I (cid:1) , ˜ s ← ˜ s + ϵ .

Original: Form the generator sample x and data sample y using x (cid:44) (˜ s , ˜ a ) , ˜ a ∼ π φ ( · | ˜ s ); y (cid:44) ( s , a ) ∈ B .
Revised: Form the generator sample x and data sample y using x ≜ (˜ s , ˜ a ) , ˜ a ∼ π ϕ ( · | ˜ s ); y ≜ ( s , a ) ∈ B .

Original: Calculate the generator loss L g ( φ ) = 1 |B| (cid:80) x
Revised: Calculate the generator loss L g ( ϕ ) = 1 |B| (cid:80) x

Original: [log (1 − D w ( x ))] using x and discriminator D w .
Revised: [log (1 − D w ( x ))] using x and discriminator D w .

Original: if iteration count % k
Revised: if iteration count % k

Original: == 

Revised: ==

Original: then if epoch count < N warm then
Revised: then if epoch count < N warm then

Original: Optimize policy with respect to L g ( φ ) only, with learning rate η φ , i.e. , [Equation] min φ α · L [Equation]
Revised: Optimize policy with respect to L g ( ϕ ) only, with learning rate η ϕ , i.e. , arg min ϕ α · L g ( ϕ ) .

Original: ( φ ) .

Original: else
Revised: else

Original: Optimize policy with learning rate η φ for the target 
Revised: Optimize policy with learning rate η ϕ for the target arg min ϕ − 1 |B| (cid:80) s ∼B , a ∼ π ϕ ( · | s ) (cid:2)

Original: [Equation] φ ) .
Revised: min j =1 , 2 Q θ j ( s , a ) (cid:3) + α · L g ( ϕ ) .

Original: end if else
Revised: end if else

Original: Skip the policy improving step.
Revised: Skip the policy improving step.

Original: end if { // Training the Discriminator }
Revised: end if { // Training the Discriminator }

Original: Optimize the discriminator D w to maximize 1 |B| (cid:80) y
Revised: Optimize the discriminator D w to maximize 1 |B| (cid:80) y

Original: [log D w ( y )]
Revised: [log D w ( y )]

Original: + 1 |B| (cid:80) x
Revised: + 1 |B| (cid:80) x

Original: [log (1 − D w ( x ))] with respect to w with learning rate η w . { // Soft Update the Target Networks } φ (cid:48) ← β φ + (1 −β ) φ (cid:48) ; θ (cid:48)
Revised: [log (1 − D w ( x ))] with respect to w with learning rate η w . { // Soft Update the Target Networks } ϕ ′ ← β ϕ + ( [Equation] −

Original:  j ← β θ j + (1 − β ) θ (cid:48)
Revised: [Equation] ) ϕ ′ ; θ ′ j ← β θ j + (1 − β ) θ [Equation]

Original: j for j = 1 , 2 .
Revised: j for j = 1 , 2 .

Original: end for end for
Revised: end for end for

Original: C P ROOFS
Revised: C P ROOFS

Original: We follow the ofﬂine RL literature (Liu et al., 2018; Nachum et al., 2019; Kallus & Zhou, 2020;
Revised: We follow the offline RL literature (Liu et al., 2018; Nachum et al., 2019; Kallus & Zhou, 2020;

Original: Mousavi et al., 2020; Zhang et al., 2020) to assume the following regularity condition on the MDP structure, which ensures ergodicity and that the limiting state occupancy measure exists and equals to the stationary distribution of the chain.
Revised: Mousavi et al., 2020; Zhang et al., 2020) to assume the following regularity condition on the MDP structure, which ensures ergodicity and that the limiting state occupancy measure exists and equals to the stationary distribution of the chain.

Original: Assumption 1 (Ergodicity of MDP) .
Revised: Assumption 1 (Ergodicity of MDP) .

Original: The MDP M is ergodic, i.e., the Markov chains associated with any π b and any π φ under consideration is positive Harris recurrent (Baxendale, 2011).
Revised: The MDP M is ergodic, i.e., the Markov chains associated with any π b and any π ϕ under consideration is positive Harris recurrent (Baxendale, 2011).

Original: Lemma 3.
Revised: Lemma 3.

Original: Let A ∈ R N × N be nonsingular and let 0 (cid:54) = b ∈ R N , x =
Revised: Let A ∈ R N × N be nonsingular and let 0 ̸ = b ∈ R N , x =

Original: A − 1 b ∈ R N .
Revised: A − 1 b ∈ R N .

Original: Let ∆ A ∈ R N × N be an arbitrary perturbation on A .
Revised: Let ∆ A ∈ R N × N be an arbitrary perturbation on A .

Original: Assume that the norm on R N × N satisﬁes (cid:107) Ax (cid:107) ≤ [Equation]
Revised: Assume that the norm on R N × N satisfies ∥

Original: A [Equation]
Revised: Ax ∥ ≤ ∥

Original:  x (cid:107) for all A and x .
Revised: A ∥∥ x ∥ for all A and x .

Original: If
Revised: If

Revised: [Equation]

Revised: b and

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let (cid:98) x
Revised: Let (cid:98) x

Original: = x + ∆ x , A ( x + ∆ x ) + ∆ A (cid:98) x
Revised: = x + ∆ x , A ( x + ∆ x ) + ∆ A (cid:98) x

Original: = b = ⇒
Revised: = b = ⇒

Original: A ∆ x + ∆ A (cid:98) x = 0 = ⇒ ∆ x = − A − 1 ∆ A (cid:98) x .
Revised: A ∆ x + ∆ A (cid:98) x = 0 = ⇒ ∆ x = − A − 1 ∆ A (cid:98) x .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: since κ ( A ) (cid:107) ∆ A (cid:107) / (cid:107) A (cid:107) < 1 by assumption.
Revised: since κ ( A ) ∥ ∆ A ∥ / ∥ A ∥ < 1 by assumption.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: By ergodicity, d φ ( s ) , d b ( s ) ∈
Revised: By ergodicity, d ϕ ( s ) , d b ( s ) ∈

Original: R N uniquely exists.
Revised: R N uniquely exists.

Original: For d ∈ { d φ , d b } , T ∈ (cid:8)
Revised: For d ∈ { d ϕ , d b } , T ∈ (cid:8)

Original: T φ , T b (cid:9) , stationarity implies that d is an eigenvector of T (cid:62) associated with eigenvalue 1 , and furthermore
Revised: T ϕ , T b (cid:9) , stationarity implies that d is an eigenvector of T ⊤ associated with eigenvalue 1 , and furthermore

Original: [Equation]
Revised: [Equation]

Original:  ...   (10) Since T is a positive matrix, by the Perron-Frobenius theorem (Meyer, 2000), 1 is an eigenvalue of
Revised: Since T is a positive matrix, by the Perron-Frobenius theorem (Meyer, 2000), 1 is an eigenvalue of

Original: T (cid:62) with algebraic multiplicity, and hence geometry multiplicity, 1 .
Revised: T ⊤ with algebraic multiplicity, and hence geometry multiplicity, 1 .

Original: The eigen-equation T (cid:62) v = 1 v has unique solution v up to a constant multiplier.
Revised: The eigen-equation T ⊤ v = 1 v has unique solution v up to a constant multiplier.

Original: Since T (cid:62) d = d , the eigenspace of T (cid:62) associated with eigenvalue 1 is span ( { d } ) .
Revised: Since T ⊤ d = d , the eigenspace of T ⊤ associated with eigenvalue 1 is span ( { d } ) .

Original: Hence dim ker (cid:0) I − T (cid:62) (cid:1) = 1 = ⇒ rank (cid:0) I − T (cid:62) (cid:1) =
Revised: Hence dim ker (cid:0) I − T ⊤ (cid:1) = 1 = ⇒ rank (cid:0) I − T ⊤ (cid:1) =

Original: N − 1 = ⇒ rank (cid:16) (cid:98) T (cid:17) =
Revised: N − 1 = ⇒ rank (cid:16) (cid:98) T (cid:17) =

Original: N .
Revised: N .

Original: The reason is that if ∃ v (cid:54) = [-
-] s.t. (cid:98) T v = 0 , then (cid:98) T 
Revised: The reason is that if ∃ v ̸ = [-
-]0 s.t. (cid:98) T v = 0 , then (cid:98) T v =

Original: [Equation]
Revised: [Equation]

Original:  =
Revised: ⊤ (cid:1) v (cid:19) =

Original: [Equation] (cid:19)
Revised: (cid:18) (cid:19)

Original: = ⇒ v =
Revised: = ⇒ v =

Original: c d for scalar c ∈ R and 1 v = c 1 d = 0 =
Revised: c d for scalar c ∈ R and 1 v = c 1 d = 0 =

Original: ⇒ c
Revised: ⇒ c

Original: = 0 , and hence v =
Revised: = 0 , and hence v =

Original: c d = 0 which contradicts to v (cid:54) = 0 .
Revised: c d = 0 which contradicts to v ̸ = 0 .

Original: Since rank (cid:16) (cid:98) T (cid:17)
Revised: Since rank (cid:16) (cid:98) T (cid:17)

Original: =
Revised: =

Original: N , dim (cid:16)(cid:110) v ∈ R N +1 : v (cid:62) (cid:98) T = 0 (cid:111)(cid:17) = 1 .
Revised: N , dim (cid:16)(cid:110) v ∈ R N +1 : v ⊤ (cid:98) T = 0 (cid:111)(cid:17) = 1 .

Original: For such a v (cid:54) = 0 , ∃ i ∈ { 2 , . . .
Revised: For such a v ̸ = 0 , ∃ i ∈ { 2 , . . .

Original: , N + 1 } s.t.
Revised: , N + 1 } s.t.

Original: vi (cid:54)
Revised: v

Original:  = 0 .
Revised: i ̸ = 0 .

Original: WLOG, assume v N [-+
 (cid:54)-] = 0 .
Revised: WLOG, assume v N [-+
 (cid:54)-]+1 ̸ = 0 .

Original: Let A ∈ R N ×
Revised: Let A ∈ R N ×

Original: N be the ﬁrst N rows of (cid:98) T , then rank( A )
Revised: N be the first N rows of (cid:98) T , then rank( A )

Original: = N .
Revised: = N .

Original: The reason is that if rank( A ) < N , ∃ w (cid:54) = [-
-]
Revised: The reason is that if rank( A ) < N , ∃ w ̸ = [-
-]0 s.t.

Original: s.t. w (cid:62) A
Revised: w ⊤ A

Original: = 0 , then (cid:18) w (cid:19) (cid:62) (cid:98) T = w (cid:62)
Revised: = 0 , then (cid:18) w (cid:19) ⊤ (cid:98) T = w ⊤ A

Original: A =and v N [-+
-]
Revised: =and v N [-+
-]+1 ̸ = 0 =

Original: [-(cid:54) = 
 =-] ⇒ (cid:0) w (cid:62) , 0 (cid:1) (cid:62) is not a constant multiple of v = ⇒ dim ker (cid:16) (cid:98) T (cid:62) (cid:17) ≥ 2 , which contradicts to the fact that rank (cid:16) (cid:98) T (cid:17) =
Revised: [-(cid:54) = 
 =-] ⇒ (cid:0) w ⊤ , 0 (cid:1) ⊤ is not a constant multiple of v = ⇒ dim ker (cid:16) (cid:98) T ⊤ (cid:17) ≥ 2 , which contradicts to the fact that rank (cid:16) (cid:98) T (cid:17) =

Original: N .
Revised: N .

Original: Thus, we conclude that rank ( A ) =
Revised: Thus, we conclude that rank ( A ) =

Original: N = ⇒
Revised: N = ⇒

Original: A is invertible.
Revised: A is invertible.

Original: Let e (1) =
Revised: Let e (1) =

Original: , 0) (cid:62) ∈ R N , then Equation 10 implies A d = e (1) .
Revised: ⊤ ∈ R N , then Equation 10 implies A d = e (1) .

Original: Plug in d φ , d b , T φ , T b and deﬁne A φ , A b similarly, we have A φ d φ = e (1) , A b d b = e (1) .
Revised: Plug in d ϕ , d b , T ϕ , T b and define A ϕ , A b similarly, we have A ϕ d ϕ = e (1) , A b d b = e (1) .

Original: For T b and T φ , we notice that by
Revised: For T b and T ϕ , we notice that by

Original: Jensen’s inequality,
Revised: Jensen’s inequality,

Original: [Equation]
Revised: [Equation]

Original: Therefore, by assumption,
Revised: Therefore, by assumption

Revised: [Equation]

Revised: ϵ

Original: [Equation]
Revised: [Equation]

Original: Let A φ =
Revised: Let A ϕ =

Original: A b + ∆ A .
Revised: A b + ∆ A .

Original: For (cid:107) ∆ A (cid:107) 1 , we notice that
Revised: For ∥ ∆ A ∥ 1 , we notice that

Original: [Equation]
Revised: [Equation]

Original: Notice that matrix 1-norm satisﬁes (cid:107) Mv (cid:107) 1 ≤ [Equation] M
Revised: Notice that matrix 1-norm satisfies ∥

Original: [Equation] 1 (cid:107) v [Equation] 1 for all matrix M and vector v , that (cid:107) A b (cid:107) 1 ≥ 1 and that (cid:107) d b (cid:107) 1 = 1 .
Revised: Mv ∥ 1 ≤ ∥ M ∥ 1 ∥ v ∥ 1 for all matrix M and vector v , that ∥ A b ∥ 1 ≥ 1 and that ∥ d b ∥ 1 = 1 .

Original: Lemma 3 implies that
Revised: Lemma 3 implies that

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: By ergodicity, d φ ( s ) , d b k ( s ) ∈ R N uniquely exists, ∀ k .
Revised: By ergodicity, d ϕ ( s ) , d b k ( s ) ∈ R N uniquely exists, ∀ k .

Original: For d ∈ { d φ , d b 1 , . . .
Revised: For d ∈ { d ϕ , d b 1 , . . .

Original: , d b k } and T ∈ (cid:8) T φ , T b 1 , . . .
Revised: , d b k } and T ∈ (cid:8) T ϕ , T b 1 , . . .

Original: , T b K (cid:9) , we follow the steps and notations in the proof of Theorem 1 to conclude that rank ( A ) =
Revised: , T b K (cid:9) , we follow the steps and notations in the proof of Theorem 1 to conclude that rank ( A ) =

Original: N = ⇒
Revised: N = ⇒

Original: A is invertible and that Ad =
Revised: A is invertible and that Ad =

Original: e (1) ∈ R N .
Revised: e (1) ∈ R N .

Original: Plug in d φ , d b k , T φ , T b k and deﬁne A φ , A b k similarly, we have A φ d φ = e (1) , A b k d b k = e (1) .
Revised: Plug in d ϕ , d b k , T ϕ , T b k and define A ϕ , A b k similarly, we have A ϕ d ϕ = e (1) , A b k d b k = e (1) .

Original: For the transition matrix T b induced by the mixture of policies π b , we have
Revised: For the transition matrix T b induced by the mixture of policies π b , we have

Original: [Equation]
Revised: [Equation]

Original: and therefore T b = (cid:80)
Revised: and therefore T b = (cid:80)

Original: Kk =1 w k T b k .
Revised: Kk =1 w k T b k .

Original: For T b k and T φ , as in the proof of Theorem 1, by Jensen’s inequality,
Revised: For T b k and T ϕ , as in the proof of Theorem 1, by Jensen’s inequality,

Revised: N (cid:88) j =

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Therefore, by assumption,
Revised: Therefore, by assumption,

Original: [Equation]
Revised: [Equation]

Original: Let A φ
Revised: Let A ϕ

Original: =
Revised: =

Original: A b k + ∆
Revised: A b k + ∆

Original: A b k .
Revised: A b k .

Original: For (cid:107) ∆ A b k (cid:107) 1 , we have
Revised: For ∥ ∆ A b k ∥ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Note that ∀ k, 
Revised: Note that ∀ k, ∥

Original: [Equation] A b k [Equation] ≥ 
Revised: A b k ∥ 1 ≥ 1 + 1 − (cid:80) Ni

Original: [Equation] + 1 − [Equation] = 1 , 
Revised: =1 T b k 1 i = 1 , ∥ d b k ∥ 1 = 1 .

Original: [Equation]

Original: d

Original: [Equation]

Original: 1 = 1 .

Original: Lemma 3 implies that
Revised: Lemma 3 implies that

Original: [Equation]
Revised: [Equation]

Original: Thus for the relative distance between d φ and d b [Equation] we have
Revised: Thus for the relative distance between d ϕ and d b , we have

Original: [Equation]

Original: i =

Original: d

Original: [Equation]

Original: =

Original: N (cid:88) i =

Original: [Equation]

Original: (cid:88) k

Original: [Equation]
Revised: [Equation]

Original: w [Equation] = [Equation]
Revised: d ϕ − d b ∥∥ d b ∥= ∥

Original: (cid:88)

Original: [Equation]

Original: w

Original: [Equation]

Original: (cid:88)

Original: [Equation] = d [Equation] (cid:0)
Revised: d ϕ − d b ∥ 1 = (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: (cid:1) =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = w

Original: [Equation]

Original: = 1 , (cid:107) d φ − d b (cid:107)(cid:107) d

Original: [Equation]

Original: (cid:107)= (cid:107)

Original: [Equation]

Original: = (cid:13)

Original: [Equation] k = ( [Equation] ) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)≤
Revised: = ( w k d ϕ − w k d b k ) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)≤

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = w [Equation] d [Equation]
Revised: = w k ∥ d ϕ − d b k ∥ 1 ≤

Original: (cid:107)

Original: [Equation]

Original: ≤

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = w k (cid:15) κ max.k 1 − (cid:15) κ max,k ≤ (cid:15) κ max 1 − (cid:15) κ max → 0 , as (cid:15) → 0 .
Revised: = w k ϵ κ max.k 1 − ϵ κ max,k ≤ ϵ κ max 1 − ϵ κ max → 0 , as ϵ → 0 .

Original: Plug w k = 1 /K, ∀ k ∈ { 1 , . . .
Revised: Plug w k = 1 /K, ∀ k ∈ { 1 , . . .

Original: , K } into the second to last equation, we get (cid:107) d φ − d b (cid:107) 1 ≤ 1 K
Revised: , K } into the second to last equation, we get ∥ d ϕ − d b ∥ 1 ≤ 1 K

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = (cid:15) κ max,k 1 − (cid:15) κ max,k → 0 , as (cid:15) → 0 .
Revised: = ϵ κ max,k 1 − ϵ κ max,k → 0 , as ϵ → 0 .

Original: Remark.
Revised: Remark.

Original: In general, d (cid:62) b T b = (cid:80) Kk =1 w 2 k d (cid:62) b k + (cid:80) i 
Revised: In general, d ⊤ b T b = (cid:80) Kk =1 w 2 k d ⊤ b k + (cid:80) i ̸

Original: (cid:54) =
Revised: =

Original: j w i w j d (cid:62) b i
Revised: j w i w j d ⊤ b i

Original: T b j (cid:54) = d (cid:62) b .
Revised: T b j ̸ = d ⊤ b .

Original: One sufﬁcient condition is d (cid:62) b i
Revised: One sufficient condition is d ⊤ b i

Original: T b j = d (cid:62) b i , ∀ i, j = ⇒ d b i =
Revised: T b j = d ⊤ b i , ∀ i, j = ⇒ d b i =

Original: d b j , ∀ i, j = ⇒
Revised: d b j , ∀ i, j = ⇒

Original: π b i = π b j , ∀ i, j , similar to Ho & Ermon (2016).
Revised: π b i = π b j , ∀ i, j , similar to Ho & Ermon (2016).

Original: In such case, π b reduces to a single policy, not a mixture, and Theorem 1 applies.
Revised: In such case, π b reduces to a single policy, not a mixture, and Theorem 1 applies.

Original: D T ECHNICAL D ETAILS D. T OY E XPERIMENT
Revised: D T ECHNICAL D ETAILS D. T OY E XPERIMENT

Original: Denote the total sample size as N total , we follow the convention to construct the eight-Gaussian dataset as in Algorithm 3.
Revised: Denote the total sample size as N total , we follow the convention to construct the eight-Gaussian dataset as in Algorithm 3.

Original: In our experiment we use N total = 2000 .
Revised: In our experiment we use N total = 2000 .

Original: Algorithm 

Revised: Algorithm

Original: Constructing the Eight-Gaussian Dataset
Revised: Constructing the Eight-Gaussian Dataset

Original: Input: Total sample size N total .
Revised: Input: Total sample size N total .

Original: Output: Generated dataset D Gaussian .
Revised: Output: Generated dataset D Gaussian .

Original: while Dataset size < N total do
Revised: while Dataset size < N total do

Original: Draw a random center c uniformly 
Revised: Draw a random center c uniformly c

Original: [Equation]
Revised: [Equation]

Original: Sample datapoint x = ( x, y ) ∼ N (cid:0) c ,  2 × 10 − 4 · I
Revised: √ 2 , 0 (cid:1) , (cid:0) −√ 2 , 0 (cid:1) , (cid:0) 0 , √ 2 (cid:1) , (cid:0) 0 , −√ 2 (cid:1) , (1 ,

Original: [Equation]
Revised: [Table]

Revised: , (1 , − 1) , ( − 1 , 1) , ( − 1 , − 1) (cid:9) .

Revised: Sample datapoint x = ( x, y ) ∼ N (cid:0) c , 2 × 10 − 4 · I 2 (cid:1) .

Original: Store x in the dataset.
Revised: Store x in the dataset.

Original: end while
Revised: end while

Original: We are interested in the 2-D eight-Gaussian dataset because (a) the conditional distribution of p ( y | x ) is multi-modal in many x ; and (b) interpolation is needed to ﬁll-in the blanks between
Revised: We are interested in the 2-D eight-Gaussian dataset because (a) the conditional distribution of p ( y | x ) is multi-modal in many x ; and (b) interpolation is needed to fill-in the blanks between

Original: Gaussian-centers, where a smooth-interpolation into a circle is naturally expected.
Revised: Gaussian-centers, where a smooth-interpolation into a circle is naturally expected.

Original: To rephrase this dataset into ofﬂine reinforcement learning setting, we deﬁne x as state and the corresponding y as action.
Revised: To rephrase this dataset into offline reinforcement learning setting, we define x as state and the corresponding y as action.

Original: Note that in the behavior cloning algorithm, the information of reward, next state, and the episodic termination is not required.
Revised: Note that in the behavior cloning algorithm, the information of reward, next state, and the episodic termination is not required.

Original: Hence, the generated dataset D Gaussian can serve as an ofﬂine RL dataset readily applicable to train behavior cloning policies.
Revised: Hence, the generated dataset D Gaussian can serve as an offline RL dataset readily applicable to train behavior cloning policies.

Original: In order to compare the ability to approximate the behavior policy by the KL loss and the JSD loss, the Gaussian policy and the implicit policy, the conditional-distribution matching and the jointvisitation matching, we ﬁt a conditional VAE (“CVAE”), a Gaussian generator conditional GAN (“G-CGAN”) and a conditional GAN (“CGAN”) using the conditional-distribution matching approach similar to Kumar et al.
Revised: In order to compare the ability to approximate the behavior policy by the KL loss and the JSD loss, the Gaussian policy and the implicit policy, the conditional-distribution matching and the jointvisitation matching, we fit a conditional VAE (“CVAE”), a Gaussian generator conditional GAN (“G-CGAN”) and a conditional GAN (“CGAN”) using the conditional-distribution matching approach similar to Kumar et al.

Original: We ﬁt, using basic joint-visitation matching strategy, a conditional GAN (“GAN”).
Revised: We fit, using basic joint-visitation matching strategy, a conditional GAN (“GAN”).

Original: As discussed in Appendix A, the major distinction between “CGAN” and “GAN” is that the former uses the same states in constructing the generator samples and the data samples while the later resamples states.
Revised: As discussed in Appendix A, the major distinction between “CGAN” and “GAN” is that the former uses the same states in constructing the generator samples and the data samples while the later resamples states.

Original: The network architecture of our conditional VAE is as follows.
Revised: The network architecture of our conditional VAE is as follows.

Original: Conditional Variational Auto-encoder (CVAE) in Toy Experiment
Revised: Conditional Variational Auto-encoder (CVAE) in Toy Experiment

Original: Encoder
Revised: Encoder

Original: Linear(state_dim+action_dim, H)
Revised: Linear(state_dim+action_dim, H)

Original: BatchNorm1d(H)
Revised: BatchNorm1d(H)

Original: ReLULinear(H,H//2)
Revised: ReLULinear(H,H//2)

Original: BatchNorm1d(H//2)
Revised: BatchNorm1d(H//2)

Original: ReLUmean=Linear(H//2, latent_dim) log_std = Linear(H//2, latent_dim)
Revised: ReLUmean=Linear(H//2, latent_dim) log_std = Linear(H//2, latent_dim)

Original: Decoder
Revised: Decoder

Original: Linear(state_dim+latent_dim, H)
Revised: Linear(state_dim+latent_dim, H)

Original: BatchNorm1d(H)
Revised: BatchNorm1d(H)

Original: ReLULinear(H,H//2)
Revised: ReLULinear(H,H//2)

Original: BatchNorm1d(H//2)
Revised: BatchNorm1d(H//2)

Original: ReLULinear(H//2,action_dim)
Revised: ReLULinear(H//2,action_dim)

Original: with hidden dimension H = 100 and latent dimension latent dim = 50 .
Revised: with hidden dimension H = 100 and latent dimension latent dim = 50 .

Original: CVAE is trained for 1200 epochs with a mini-batch size of 100 and random seed 0 , using the mean-squared-error as the reconstruction loss, and the Gaussian-case closed-form formula in Kingma & Welling (2013) for the
Revised: CVAE is trained for 1200 epochs with a mini-batch size of 100 and random seed 0 , using the mean-squared-error as the reconstruction loss, and the Gaussian-case closed-form formula in Kingma & Welling (2013) for the

Original: KL term.
Revised: KL term.

Original: The network architecture of our conditional GAN, used in “CGAN” and “GAN,” is as follows.
Revised: The network architecture of our conditional GAN, used in “CGAN” and “GAN,” is as follows.

Original: Conditional Generative Adversarial Nets (CGAN) in Toy Experiment Generator
Revised: Conditional Generative Adversarial Nets (CGAN) in Toy Experiment Generator

Original: Linear [Equation]
Revised: Linear(state_dim+z_dim, H)

Original: BatchNorm1d(H)
Revised: BatchNorm1d(H)

Original: ReLULinear(H,H//2)
Revised: ReLULinear(H,H//2)

Original: BatchNorm1d(H//2)
Revised: BatchNorm1d(H//2)

Original: ReLU
Revised: ReLU

Original: Linear(H//2, action_dim)
Revised: Linear(H//2, action_dim)

Original: Discriminator
Revised: Discriminator

Original: Linear(state

Original: [Equation]

Original: dim

Original: [Equation] action [Equation] dim, H)
Revised: Linear(state_dim+action_dim, H)

Original: LeakyReLU(0.1)
Revised: LeakyReLU(0.1)

Original: Linear [Equation] [Equation] [Equation]
Revised: Linear(H,H//2)

Original: LeakyReLU(0.1)
Revised: LeakyReLU(0.1)

Original: Linear(H//2,1) where the structure of BatchNorm1d, LeakyReLU follows Radford et al.
Revised: Linear(H//2,1) where the structure of BatchNorm1d, LeakyReLU follows Radford et al.

Original: Here we again use H = 100 , z dim = 50 .
Revised: Here we again use H = 100 , z dim = 50 .

Original: Conditional GAN is trained for 2000 epochs with a mini-batch size of 100 and random seed 0 .
Revised: Conditional GAN is trained for 2000 epochs with a mini-batch size of 100 and random seed 0 .

Original: We follow Radford et al.
Revised: We follow Radford et al.

Original: (2016) to train CGAN using Adam optimizer with β 1 = 0 .
Revised: (2016) to train CGAN using Adam optimizer with β 1 = 0 .

Original: The network architecture of our Gaussian-generator version of conditional GAN, used in the experiments of Appendix F, is as follows.
Revised: The network architecture of our Gaussian-generator version of conditional GAN, used in the experiments of Appendix F, is as follows.

Original: GeneratorLinear(state_dim, H)
Revised: GeneratorLinear(state_dim, H)

Original: BatchNorm1d(H)
Revised: BatchNorm1d(H)

Original: ReLULinear(H,H//2)
Revised: ReLULinear(H,H//2)

Original: BatchNorm1d(H//2)
Revised: BatchNorm1d(H//2)

Original: ReLUmean=Linear(H//2, action_dim), log_std = Linear(H//2, action_dim) with Discriminator and other technical details the same as CGAN.
Revised: ReLUmean=Linear(H//2, action_dim), log_std = Linear(H//2, action_dim) with Discriminator and other technical details the same as CGAN.

Original: This Gaussian-generator version of CGAN is again trained for 2000 epochs with a mini-batch size of 100 , random seed 0 , and β 1 = 0 .
Revised: This Gaussian-generator version of CGAN is again trained for 2000 epochs with a mini-batch size of 100 , random seed 0 , and β 1 = 0 .

Original: 5 in the Adam optimizer.
Revised: 5 in the Adam optimizer.

Original: Our test set is formed by a random sample of 2000 new states ( x ) from [ − 1 . 5 , 1 . 5] together with the states in the training set.
Revised: Our test set is formed by a random sample of 2000 new states ( x ) from [ − 1 . 5 , 1 . 5] together with the states in the training set.

Original: The performance on the test set thus shows both the concentration on the eight centers and the smooth interpolation between centers, which translates into a good and smooth ﬁt to the behavior policy.
Revised: The performance on the test set thus shows both the concentration on the eight centers and the smooth interpolation between centers, which translates into a good and smooth fit to the behavior policy.

Original: Figure 1 shows the training set (“Truth”) and the kernel-density-estimate plot of each methods.
Revised: Figure 1 shows the training set (“Truth”) and the kernel-density-estimate plot of each methods.

Original: D. R EINFORCEMENT L EARNING E XPERIMENTS
Revised: D. R EINFORCEMENT L EARNING E XPERIMENTS

Original: In practice, the rollouts contained in the ofﬂine dataset have ﬁnite horizon, and thus special treatment is needed per appearance of the terminal states in calculating the Bellman update target.
Revised: In practice, the rollouts contained in the offline dataset have finite horizon, and thus special treatment is needed per appearance of the terminal states in calculating the Bellman update target.

Original: We follow the standard treatment (Mnih et al., 2013; Sutton & Barto, 2018) to deﬁne the update target y as
Revised: We follow the standard treatment (Mnih et al., 2013; Sutton & Barto, 2018) to define the update target y as

Original: [Equation]
Revised: [Equation]

Original: if s (cid:48) is a non-terminal state r ( s , a ) if s (cid:48) is a terminal state , where (cid:101) Q (cid:48) ( s (cid:48) , a (cid:48) ) refers to the expectation term in Equation 4 for basic algorithm (Section 4.1) or the expectation term in Equation 8 for the enhanced versions with state-smoothing at the Bellman
Revised: if s ′ is a non-terminal state r ( s , a ) if s ′ is a terminal state , where (cid:101) Q ′ ( s ′ , a ′ ) refers to the expectation term in Equation 4 for basic algorithm (Section 4.1) or the expectation term in Equation 8 for the enhanced versions with state-smoothing at the Bellman

Original: Backup (Section 4.2).
Revised: Backup (Section 4.2).

Original: For simplicity, we follow White (2016) to choose the noise distribution
Revised: For simplicity, we follow White (2016) to choose the noise distribution

Original: p z ( z ) as the multivariate standard normal distribution, where the dimension of z is conveniently chosen as dim ( z ) =
Revised: p z ( z ) as the multivariate standard normal distribution, where the dimension of z is conveniently chosen as dim ( z ) =

Original: min ( 10 , state dim // 2 ) .
Revised: min ( 10 , state dim // 2 ) .

Original: To sample from the implicit policy, for each state s , we ﬁrst sample independently z ∼ N ( 0 , I ) .
Revised: To sample from the implicit policy, for each state s , we first sample independently z ∼ N ( 0 , I ) .

Original: We then concatenate s with z and feed the resulting [ s , z ] into the deterministic policy network to generate stochastic actions.
Revised: We then concatenate s with z and feed the resulting [ s , z ] into the deterministic policy network to generate stochastic actions.

Original: To sample from a small region around the next state s (cid:48) (Section 4.2), we keep the original s (cid:48) and repeat it additionally N B times.
Revised: To sample from a small region around the next state s ′ (Section 4.2), we keep the original s ′ and repeat it additionally N B times.

Original: For each of the N B replications, we add an independent Gaussian noise (cid:15) ∼ N ( 0 , σ 2 B I ) .
Revised: For each of the N B replications, we add an independent Gaussian noise ϵ ∼ N ( 0 , σ 2 B I ) .

Original: The original s (cid:48) and its N B noisy replications are then fed into the implicit policy to sample the corresponding action.
Revised: The original s ′ and its N B noisy replications are then fed into the implicit policy to sample the corresponding action.

Original: For fair comparison, we use the same network architecture as the ofﬁcial implementation of the
Revised: For fair comparison, we use the same network architecture as the official implementation of the

Original: BCQ algorithm, which will be stated in detail in Sections D.2.1.
Revised: BCQ algorithm, which will be stated in detail in Sections D.2.1.

Original: Due to limited computational resources, we leave a ﬁne-tuning of the noise distribution p z ( z ) , the network architectures, and the optimization hyperparameters for future work, which also leaves room for further improving our results.
Revised: Due to limited computational resources, we leave a fine-tuning of the noise distribution p z ( z ) , the network architectures, and the optimization hyperparameters for future work, which also leaves room for further improving our results.

Original: For a more stable training of the policy, we adopt the warm start strategy (Kumar et al., 2020;
Revised: For a more stable training of the policy, we adopt the warm start strategy (Kumar et al., 2020;

Original: Yue et al., 2020).
Revised: Yue et al., 2020).

Original: Speciﬁcally, in the ﬁrst N warm epochs, the policy is trained to minimize L g only, with the same learning rate η φ as the rest epochs that also try to maximize the expected Q-values. 
Revised: Specifically, in the first N warm epochs, the policy is trained to minimize L g only, with the same learning rate η ϕ as the rest epochs that also try to maximize the expected Q-values. Datasets.

Revised: We use the continuous control tasks provided by the D4RL dataset (Fu et al., 2021) to conduct algorithmic evaluations.

Revised: Due to limited computational resources, we select therein the “medium-expert,” “medium-replay,” and “medium” datasets for the Hopper, HalfCheetah, Walker2d tasks in the Gym-MuJoCo domain, which are commonly used benchmarks in prior work (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Kumar et al., 2020).

Revised: We follow the literature (Cang et al., 2021; Chen et al., 2021; Kostrikov et al., 2021) to not test on the “random” and “expert” datasets as they are less practical (Matsushima et al., 2021) and can be respectively solved by directly using standard off-policy RL algorithms (Agarwal et al., 2020) and the behavior cloning algorithms.

Original:  Evaluation Protocol.
Revised: We note that a comprehensive benchmarking of prior offline-RL algorithms on the “expert” datasets is currently unavailable in the literature, which is out of the scope of this paper. Apart from the Gym-MuJoCo domain, we also consider the Maze2D domain 2 of tasks for the non-Markovian datacollecting policy and the Adroit tasks 3 (Rajeswaran et al., 2018) for their sparse reward-signal and high dimensionality. Evaluation Protocol.

Original: In all the experiments, we follow Fu et al.
Revised: In all the experiments, we follow Fu et al.

Original: (2021) to use the “v0” version of the datasets in the Gym-MuJoCo and Adroit domains.
Revised: (2021) to use the “v0” version of the datasets in the Gym-MuJoCo and Adroit domains.

Original: In our preliminary study, we ﬁnd that the rollout results of some existing algorithms can be unstable across epochs in some datasets, even towards the end of training.
Revised: In our preliminary study, we find that the rollout results of some existing algorithms can be unstable across epochs in some datasets, even towards the end of training.

Original: To reduce the randomness in evaluation, we report, for our algorithm, the mean and standard deviation of the last ﬁve rollouts across three random seeds { 0 , 1 , 2 } .
Revised: To reduce the randomness in evaluation, we report, for our algorithm, the mean and standard deviation of the last five rollouts across three random seeds { 0 , 1 , 2 } .

Original: We train all agents for 1000 epochs, where each epoch consists of 1000 mini-batch stochastic gradient descent steps.
Revised: We train all agents for 1000 epochs, where each epoch consists of 1000 mini-batch stochastic gradient descent steps.

Original: We rollout each agent for 10 episodes after each epoch of training.
Revised: We rollout each agent for 10 episodes after each epoch of training.

Original: D.2. GAN J OINT M ATCHING
Revised: D.2. GAN J OINT M ATCHING

Original: In approximately matching the Jensen-Shannon divergence between the state-action visitation frequencies via Generative Adversarial Nets, a crucial step is to stably and effectively train the GAN structure.
Revised: In approximately matching the Jensen-Shannon divergence between the state-action visitation frequencies via Generative Adversarial Nets, a crucial step is to stably and effectively train the GAN structure.

Original: To this end, we adopt the following tricks from the literature.
Revised: To this end, we adopt the following tricks from the literature.

Original: • To provide stronger gradients early in training, rather than training the policy π φ to minimize
Revised: • To provide stronger gradients early in training, rather than training the policy π ϕ to minimize

Original: [Equation]
Revised: [Equation]

Original: we follow (Goodfellow et al., 2014) to train π φ to maximize
Revised: we follow (Goodfellow et al., 2014) to train π ϕ to maximize

Original: E x [log ( D w ( x ))]
Revised: [Equation]

Original: • Motivated by (Radford et al., 2016), we use LeakyReLU activation in both the generator and discriminator, with default negative slope=0.01 .
Revised: • Motivated by (Radford et al., 2016), we use LeakyReLU activation in both the generator and discriminator, with default negative slope=0.01 .

Original: • To stabilize the training, we follow (Radford et al., 2016) to use a reduced momentum term β 1 = 0 .
Revised: • To stabilize the training, we follow (Radford et al., 2016) to use a reduced momentum term β 1 = 0 .

Original: 4 in the Adam optimizer (Kingma & Ba, 2014).
Revised: 4 in the Adam optimizer (Kingma & Ba, 2014).

Original: • We follow (Radford et al., 2016) to use actor and discriminator learning rate η φ = η w = 2 × 10 − 4 .
Revised: • We follow (Radford et al., 2016) to use actor and discriminator learning rate η ϕ = η w = 2 × 10 − 4 .

Original: • To avoid overﬁtting of the discriminator, we are motivated by Salimans et al.
Revised: • To avoid overfitting of the discriminator, we are motivated by Salimans et al.

Original: (2016) and Goodfellow (2017) to use one-sided label smoothing with soft and noisy labels.
Revised: (2016) and Goodfellow (2017) to use one-sided label smoothing with soft and noisy labels.

Original: Speciﬁcally, the labels for the data sample y is replaced with a random number between 0 . 8 and 1 . 0 , instead of the original 1 .
Revised: Specifically, the labels for the data sample y is replaced with a random number between 0 . 8 and 1 . 0 , instead of the original 1 .

Original: No label smoothing is applied for the generator sample x , therefore their labels are all 0 .
Revised: No label smoothing is applied for the generator sample x , therefore their labels are all 0 .

Original: The loss function for training the discriminator in GAN is the Binary Cross Entropy between the labels and the outputs from the discriminator
Revised: The loss function for training the discriminator in GAN is the Binary Cross Entropy between the labels and the outputs from the discriminator.

Original: Furthermore, motivated by TD3 (Fujimoto et al., 2018) and GAN (Section 2), we update π φ ( · | s ) once per k updates of the critics and discriminator.
Revised: Furthermore, motivated by TD3 (Fujimoto et al., 2018) and GAN (Section 2), we update π ϕ ( · | s ) once per k updates of the critics and discriminator.

Original: Table 2 shows the hyperparameters for our GAN joint-matching framework.
Revised: Table 2 shows the hyperparameters for our GAN joint-matching framework.

Original: Note that several simpliﬁcations are made to minimize hyperparameter tuning, such as ﬁxing η φ = η w as in (Radford et al., 2016) and σ B = σ J .
Revised: Note that several simplifications are made to minimize hyperparameter tuning, such as fixing η ϕ = η w as in (Radford et al., 2016) and σ B = σ J .

Original: We also ﬁx N a = 1 to ease computation.
Revised: We also fix N a = 1 to ease computation.

Original: We comment that many of these hyperparameters can be set based on literature, for example, we use η φ = η w = 2 × 10 − 4 as in Radford et al.
Revised: We comment that many of these hyperparameters can be set based on literature, for example, we use η ϕ = η w = 2 × 10 − 4 as in Radford et al.

Original: (2016), η θ = 3 × 10 − 4 and N warm =
Revised: (2016), η θ = 3 × 10 − 4 and N warm =

Original: 40 as in Kumar et al.
Revised: 40 as in Kumar et al.

Original: (2020), λ = 0 .
Revised: (2020), λ = 0 .

Original: 75 as in Fujimoto et al.
Revised: 75 as in Fujimoto et al.

Original: and policy frequency k = 2 as in Fujimoto et al.
Revised: and policy frequency k = 2 as in Fujimoto et al.

Original: Unless speciﬁed, the same hyperparameter is used across all datasets.
Revised: Unless specified, the same hyperparameter is used across all datasets.

Original: [Table] Below we state the network architectures of the actor, critic [Table] and the discriminator in GAN.
Revised: Below we state the network architectures of the actor, critic, and the discriminator in GAN.

Original: Note that we use a pair of critic networks with the same architecture to perform clipped double Q-learning.
Revised: Note that we use a pair of critic networks with the same architecture to perform clipped double Q-learning.

Original: Actor
Revised: Actor

Original: Linear(state_dim+noise_dim, 400)
Revised: Linear(state_dim+noise_dim, 400)

Original: LeakyReLU Linear(400, 300)
Revised: LeakyReLULinear(400, 300)

Original: LeakyReLULinear(300, action_dim) max_action * tanh
Revised: LeakyReLULinear(300, action_dim) max_action * tanh

Original: [Table]
Revised: Critic

Original: Linear(state_dim+action_dim, 400)
Revised: Linear(state_dim+action_dim, 400)

Original: LeakyReLU Linear(400, 300)
Revised: LeakyReLULinear(400, 300)

Original: LeakyReLULinear(300, 1)
Revised: LeakyReLULinear(300, 1)

Original: Discriminator in GAN
Revised: Discriminator in GAN

Original: Linear( [Equation]
Revised: Linear(state_dim+action_dim, 400)

Original: LeakyReLULinear(400, 300)
Revised: LeakyReLULinear(400, 300)

Original: LeakyReLULinear(300, 1)
Revised: LeakyReLULinear(300, 1)

Original: Sigmoid
Revised: Sigmoid

Original: Note that the network architectures follow from Fujimoto et al.
Revised: Note that the network architectures follow from Fujimoto et al.

Original: and that all the LeakyReLU activation uses the default negative slope=0.01 .
Revised: and that all the LeakyReLU activation uses the default negative slope=0.01 .

Revised: [Table]

Original: D.2. R ESULTS OF CQL
Revised: D.2. R ESULTS OF CQL

Original: We note that the ofﬁcial CQL GitHub repository does not provide hyperparameter settings for the
Revised: We note that the official CQL GitHub repository does not provide hyperparameter settings for the

Original: Maze2D and Adroit domain of tasks.
Revised: Maze2D and Adroit domain of tasks.

Original: For datasets in these two domains, we train an CQL agent using ﬁve hyperparameter settings: four recommended Gym-MuJoCo settings and one recommended
Revised: For datasets in these two domains, we train an CQL agent using five hyperparameter settings: four recommended Gym-MuJoCo settings and one recommended

Original: Ant-Maze setting.
Revised: Ant-Maze setting.

Original: We then calculate the average (normalized) return of the last ﬁve rollouts over random seeds { 0 , 1 , 2 } and report the per-dataset best results across those ﬁve hyperparameter settings.
Revised: We then calculate the average (normalized) return of the last five rollouts over random seeds { 0 , 1 , 2 } and report the per-dataset best results across those five hyperparameter settings.

Revised: We comment that this approach is non-conventional

Original: We comment that this approach is non-conventional, but rather a compensation for the missing of recommended hyperparameter settings, and may give CQL some advantage on the Maze2D and
Revised: [Table] but rather a compensation for the missing of recommended hyperparameter settings [Table] and may give CQL some advantage on the Maze2D and

Original: Adroit domains.
Revised: Adroit domains.

Original: For the Gym-MuJoCo domain, we follow Kumar et al.
Revised: For the Gym-MuJoCo domain, we follow Kumar et al.

Original: (2020) to use the recommended hyperparameter setting across all datasets.
Revised: (2020) to use the recommended hyperparameter setting across all datasets.

Original: D.2. A BLATION S TUDY ON G AUSSIAN P OLICY
Revised: D.2. A BLATION S TUDY ON G AUSSIAN P OLICY

Original: The network architecture of the Gaussian policy we used in the ablation study (Section 5.2) follows common practice ((Haarnoja et al., 2018a; Kumar et al., 2020)).
Revised: The network architecture of the Gaussian policy we used in the ablation study (Section 5.2) follows common practice ((Haarnoja et al., 2018a; Kumar et al., 2020)).

Original: Gaussian Policy
Revised: Gaussian Policy

Original: Linear(state
Revised: Linear(state_dim, 400) LeakyReLULinear(400, 300) LeakyReLUmean= [Equation]

Original: [Equation]
Revised: [Equation]

Original: dim, 400)
Revised: log_

Original: LeakyReLULinear(400, 300)
Revised: [Equation]

Original: LeakyReLUmean=Linear(300, action_dim) log_std = Linear(300, action_dim)
Revised: Linear [Equation] [Equation] action_

Revised: [Equation]

Original: Critics and discriminator are the same as in the implicit policy case (Appendix D.2.1).
Revised: Critics and discriminator are the same as in the implicit policy case (Appendix D.2.1).

Original: For action selection from the Gaussian policy, a given state s is ﬁrst mapped to the mean µ ( s ) and standard deviation vector σ ( s ) .
Revised: For action selection from the Gaussian policy, a given state s is first mapped to the mean µ ( s ) and standard deviation vector σ ( s ) .

Original: A raw action is sampled as a raw ∼ N (cid:0) µ ( s ) , diag (cid:0) σ 2 ( s ) (cid:1)(cid:1) .
Revised: A raw action is sampled as a raw ∼ N (cid:0) µ ( s ) , diag (cid:0) σ 2 ( s ) (cid:1)(cid:1) .

Original: Finally, a raw is mapped into the action space as max action × tanh( a raw ) .
Revised: Finally, a raw is mapped into the action space as max action × tanh( a raw ) .

Original: For fair comparison, other technical details, including the training procedure and hyperparameter setting, are exactly the same as the implicit policy case (Appendix D.2.1).
Revised: For fair comparison, other technical details, including the training procedure and hyperparameter setting, are exactly the same as the implicit policy case (Appendix D.2.1).

Original: E A DDITIONAL T ABLES E. R AW S CORES OF THE M AIN T ABLE
Revised: E A DDITIONAL T ABLES E. R AW S CORES OF THE M AIN T ABLE

Original: In Table 3 we report the un-normalized results corresponding to Table 1 for reference.
Revised: In Table 3 we report the un-normalized results corresponding to Table 1 for reference

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: T ABLES FOR THE A BLATION S TUDY
Revised: T ABLES FOR THE A BLATION S TUDY

Original: Table 4 - 7 correspond to the results for our ablation study in Section 5.2.
Revised: Table 4 - 7 correspond to the results for our ablation study in Section 5.2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F F URTHER C OMPARISON B ETWEEN I MPLICIT AND G AUSSIAN P OLICY
Revised: F F URTHER C OMPARISON B ETWEEN I MPLICIT AND G AUSSIAN P OLICY

Original: In Section 5.2 we note that a uni-modal stochastic policy, such as the Gaussian policy, is less ﬂexible to capture all the rewarding actions, on which an implicit policy may ﬁt well.
Revised: In Section 5.2 we note that a uni-modal stochastic policy, such as the Gaussian policy, is less flexible to capture all the rewarding actions, on which an implicit policy may fit well.

Original: In this section we visualize such a difference.
Revised: In this section we visualize such a difference.

Original: Figure 2 compares the ﬁtting of the eight-Gaussian toy dataset by implicit policy and Gaussian policy.
Revised: Figure 2 compares the fitting of the eight-Gaussian toy dataset by implicit policy and Gaussian policy.

Original: Speciﬁcally, Figure 2a plots the dataset; Figure 2b plots CGAN with the default implicit generator (implicit policy) ﬁtted by the conditional-distribution matching approach; Figure 2c plots
Revised: Specifically, Figure 2a plots the dataset; Figure 2b plots CGAN with the default implicit generator (implicit policy) fitted by the conditional-distribution matching approach; Figure 2c plots

Original: CGAN with Gaussian generator (Gaussian policy) ﬁtted by the conditional-distribution matching approach; Figure 2d plots CGAN with implicit policy ﬁtted by the basic joint-visitation matching strategy (Section 4.1); Figure 2e plots CGAN with Gaussian policy ﬁtted by the basic joint-visitation matching strategy.
Revised: CGAN with Gaussian generator (Gaussian policy) fitted by the conditional-distribution matching approach; Figure 2d plots CGAN with implicit policy fitted by the basic joint-visitation matching strategy (Section 4.1); Figure 2e plots CGAN with Gaussian policy fitted by the basic joint-visitation matching strategy.

Original: Experimental details are on Appendix D.1.
Revised: Experimental details are on Appendix D.1.

Original: (a) Truth 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1.

Original: (b) CGAN 1.5 1.0 0.5 0.0 0.5 1.0 1.5 0. (c) G-CGAN 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1. (d) GAN 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1. (e) G-GAN
Revised: [Figure]

Original: We see that whatever training strategies, Gaussian policies fail to learn multi-modal state-conditional action distributions, even if needed.
Revised: We see that whatever training strategies, Gaussian policies fail to learn multi-modal state-conditional action distributions, even if needed.

Original: Even though the Gaussian policy version of CGAN may still correctly capture some modes in the action distributions, an improvement over the mode-covering
Revised: Even though the Gaussian policy version of CGAN may still correctly capture some modes in the action distributions, an improvement over the mode-covering

Original: CVAE, they miss other modes.
Revised: CVAE, they miss other modes.

Original: Besides, these Gaussian policy versions interpolate less-smoothly between the centers.
Revised: Besides, these Gaussian policy versions interpolate less-smoothly between the centers.

Original: In ofﬂine RL, these weaknesses is related the missing of some rewarding actions and less-predictable action-choices at unseen states.
Revised: In offline RL, these weaknesses is related the missing of some rewarding actions and less-predictable action-choices at unseen states.

Original: To visualize the differences between the implicit and the Gaussian policy in the ofﬂine RL setting, we plot the kernel density estimates of the distributions of the ﬁrst two action-dimensions in the “maze2d-umaze-v1” dataset, where a performance difference is shown in Table 4.
Revised: To visualize the differences between the implicit and the Gaussian policy in the offline RL setting, we plot the kernel density estimates of the distributions of the first two action-dimensions in the “maze2d-umaze-v1” dataset, where a performance difference is shown in Table 4.

Original: Speciﬁcally,
Revised: Specifically,

Original: Figure 3a plots the distribution of actions in the ofﬂine dataset.
Revised: Figure 3a plots the distribution of actions in the offline dataset.

Original: Figure 3b and 3c respectively plot action distributions produced by the ﬁnal Gaussian policy and the ﬁnal implicit policy generating
Revised: Figure 3b and 3c respectively plot action distributions produced by the final Gaussian policy and the final implicit policy generating

Original: Table 4.
Revised: Table 4.

Revised: (a) Truth 1.

Original: [Figure]
Revised: [Figure]

Revised: ) Gaussian Policy 1.

Original: [Figure]
Revised: [Figure]

Original: We see from Figure 3a and Figure 3b that Gaussian policy leaves out two action modes, namely, modes on the upper-left and upper-right corners.
Revised: We see from Figure 3a and Figure 3b that Gaussian policy leaves out two action modes, namely, modes on the upper-left and upper-right corners.

Original: Figure 3c shows that our implicit policy does capture all the modes shown in Figure 3a.
Revised: Figure 3c shows that our implicit policy does capture all the modes shown in Figure 3a.

Original: Note that “maze2d-umaze” is a navigation task requiring agents to reach a goal location (Fu et al., 2021).
Revised: Note that “maze2d-umaze” is a navigation task requiring agents to reach a goal location (Fu et al., 2021).

Original: Gaussian policy thus may miss out some directions in the ofﬂine dataset pertaining to short paths to the goal state, which may explain its inferior performance on this dataset in Table 4.
Revised: Gaussian policy thus may miss out some directions in the offline dataset pertaining to short paths to the goal state, which may explain its inferior performance on this dataset in Table 4.