O FFLINE R EINFORCEMENT L EARNING WITH D IFFERENTIAL P RIVACY
Anonymous authors
Paper under double-blind review
A BSTRACT
The ofﬂine reinforcement learning (RL) problem is often motivated by the need to learn data-driven decision policies in ﬁnancial, legal and healthcare applications.
However, the learned policy could retain sensitive information of individuals in the training data (e.g., treatment and outcome of patients), thus susceptible to various privacy risks.
We design ofﬂine RL algorithms with differential privacy guarantees which provably prevent such risks.
These algorithms also enjoy strong instancedependent learning bounds under both tabular and linear Markov Decision Process (MDP) settings.
Our theory and simulation suggest that the privacy guarantee comes at (almost) no drop in utility comparing to the non-private counterpart for a medium-size dataset.
I NTRODUCTION
Ofﬂine Reinforcement Learning (or batch RL) aims to learn a near-optimal policy in an unknown environment 1 through a static dataset gathered from some behavior policy µ .
Since ofﬂine RL does not require access to the environment, it can be applied to problems where interaction with environment is infeasible, e.g. , when collecting new data is costly (trade or ﬁnance (Zhang et al., 2020)), risky (autonomous driving (Sallab et al., 2017)) or illegal / unethical (healthcare (Raghu et al., 2017)).
In such practical applications, the data used by an RL agent usually contains sensitive information.
Take medical history for instance, for each patient, at each time step, the patient reports her health condition (age, disease, etc.), then the doctor decides the treatment (which medicine to use, the dosage of medicine, etc.), ﬁnally there is treatment outcome (whether the patient feels good, etc.) and the patient transitions to another health condition.
Here, (health condition, treatment, treatment outcome) corresponds to (state, action, reward) and the dataset can be considered as n (number of patients) trajectories sampled from a MDP with horizon H (number of treatment steps).
However, learning agents are known to implicitly memorize details of individual training data points verbatim (Carlini et al., 2019), even if they are irrelevant for learning (Brown et al., 2021), which makes ofﬂine
RL models vulnerable to various privacy attacks.
Differential privacy (DP) (Dwork et al., 2006) is a well-established deﬁnition of privacy with many desirable properties.
A differentially private ofﬂine RL algorithm will return a decision policy that is indistinguishable from a policy trained in an alternative universe any individual user is replaced, thereby preventing the aforementioned privacy risks.
There is a surge of recent interest in developing
RL algorithms with DP guarantees, but they focus mostly on the online setting (Vietri et al., 2020;
Garcelon et al., 2021; Liao et al., 2021; Chowdhury & Zhou, 2021;
Luyo et al., 2021).
Ofﬂine RL is arguably more practically relevant than online RL in the applications with sensitive data.
For example, in the healthcare domain, online RL requires actively running new exploratory policies (clinical trials) with every new patient, which often involves complex ethical / legal clearances, whereas ofﬂine RL uses only historical patient records that are often accessible for research purposes.
Clear communication of the adopted privacy enhancing techniques (e.g., DP) to patients was reported to further improve data access (Kim et al., 2017).
Our contributions.
In this paper, we present the ﬁrst provably efﬁcient algorithms for ofﬂine RL with differential privacy.
Our contributions are twofold.
• We design two new pessimism-based algorithms DP-APVI (Algorithm 1) and DP-VAPVI (Algorithm 2), one for the tabular setting (ﬁnite states and actions), the other for the case with linear function approximation (under linear MDP assumption).
Both algorithms enjoy DP guarantees (pure DP or zCDP) and instance-dependent learning bounds where the cost of privacy appears as lower order terms.
• We perform numerical simulations to evaluate and compare the performance of our algorithm DP-VAPVI (Algorithm 2) with its non-private counterpart VAPVI (Yin et al., 2022) as well as a popular baseline PEVI (Jin et al., 2021).
The results complement the theoretical ﬁndings by demonstrating the practicality of DP-VAPVI under strong privacy parameters.
Related work.
To our knowledge, differential privacy in ofﬂine RL tasks has not been studied before, except for much simpler cases where the agent only evaluates a single policy (Balle et al., 2016; Xie et al., 2019).
Balle et al.
(2016) privatized ﬁrst-visit Monte Carlo-Ridge Regression estimator by an output perturbation mechanism and Xie et al.
(2019) used DP-SGD.
Neither paper considered ofﬂine learning (or policy optimization), which is our focus.
There is a larger body of work on private RL in the online setting, where the goal is to minimize regret while satisfying either joint differential privacy (Vietri et al., 2020; Chowdhury & Zhou, 2021;
Ngo et al., 2022; Luyo et al., 2021) or local differential privacy (Garcelon et al., 2021; Liao et al., 2021;
Luyo et al., 2021; Chowdhury & Zhou, 2021).
The ofﬂine setting introduces new challenges in DP as we cannot algorithmically enforce good “exploration”, but have to work with a static dataset and privately estimate the uncertainty in addition to the value functions.
A private online RL algorithm can sometimes be adapted for private ofﬂine RL too, but those from existing work yield suboptimal and non-adaptive bounds.
We give a more detailed technical comparison in Appendix B.
Among non-private ofﬂine RL works, we build directly upon non-private ofﬂine RL methods known as Adaptive Pessimistic Value Iteration (APVI, for tabular MDPs)
(Yin & Wang, 2021b) and VarianceAware Pessimistic Value Iteration (VAPVI, for linear MDPs)
(Yin et al., 2022), as they give the strongest theoretical guarantees to date.
We refer readers to Appendix B for a more extensive review of the ofﬂine RL literature.
Introducing DP to APVI and VAPVI while retaining the same sample complexity (modulo lower order terms) require nontrivial modiﬁcations to the algorithms.
A remark on technical novelty.
Our algorithms involve substantial technical innovation over previous works on online DP-RL with joint DP guarantee 2 .
Different from previous works, our
DP-APVI (Algorithm 1) operates on Bernstein type pessimism, which requires our algorithm to deal with conditional variance using private statistics.
Besides, our DP-VAPVI (Algorithm 2) replaces the
LSVI technique with variance-aware LSVI (also known as weighted ridge regression, ﬁrst appears in (Zhou et al., 2021)).
Our DP-VAPVI releases conditional variance privately, and further applies weighted ridge regression privately.
Both approaches ensure tighter instance-dependent bounds on the suboptimality of the learned policy.
P ROBLEM S ETUP
Markov Decision Process.
A ﬁnite-horizon Markov Decision Process (MDP) is denoted by a tuple
M = ( S , A , P, r, H, d 1 ) (Sutton & Barto, 2018), where S is state space and A is action space.
A nonstationary transition kernel P h : S ⇥
A ⇥ S 7!
[0 , 1] maps each state action ( s h , a h ) to a probability distribution P h ( ·| s h , a h ) and P h can be different across time.
Besides, r h : S ⇥
A 7!
R is the expected immediate reward satisfying r h  1 , d 1 is the initial state distribution and H is the horizon.
A policy ⇡ = ( ⇡ 1 , · · · , ⇡ H ) assigns each state s h 2 S a probability distribution over actions according to the map s h 7!
⇡ h ( ·| s h ) , 8 h 2 [ H ] .
A random trajectory s 1 , a 1 , r 1 , · · · , s H , a H , r H , s H +1 is generated according to s
[Equation]
For tabular MDP, we have S ⇥
A is the discrete state-action space and S := |S| , A :=
|A| are ﬁnite.
In this work, we assume that r is known 3 .
In addition, we denote the per-step marginal state-action occupancy d ⇡ h ( s, a ) as: d ⇡ h ( s, a ) :=
P [ s h
= s | s 1 ⇠ d 1 , ⇡ ] · ⇡ h ( a | s ) , which is the marginal state-action probability at time h .
Value function, Bellman (optimality) equations.
The value function V ⇡ h ( · ) and Q-value function Q ⇡ h ( · , · ) for any policy ⇡ is deﬁned as: V ⇡ h ( s
[Equation]
=
E ⇡ [
[Equation] [Equation]
E
[Equation] [Equation]
[Equation]
The performance is deﬁned as v ⇡ :=
E d
[Equation] [Equation]
⇡ 1 ] = E ⇡ ,dhP
Ht =1 r t i .
The Bellman (optimality) equations follow 8 h 2 [ H ] : Q ⇡ h = r
[Equation]
+
[Equation] [Equation]
[Equation]
[Equation] [Equation]
, V ?
[Equation]
= max a Q ?
[Equation] [Equation]
a ) .
Linear MDP (Jin et al., 2020b).
An episodic MDP ( S , A , H, P, r ) is called a linear MDP with known feature map  : S ⇥ A !
R d if there exist H unknown signed measures ⌫ h 2 R d over S and H unknown reward vectors ✓ h 2 R d such that
[Equation]
Without loss of generality, we assume k  ( s, a ) k 2  1 and max( k ⌫ h ( S ) k 2 , k ✓ h k 2 )  p d for all h, s, a 2 [ H ] ⇥ S ⇥ A .
An important property of linear MDP is that the value functions are linear in the feature map, which is summarized in Lemma E.14.
Ofﬂine setting and the goal.
The ofﬂine RL requires the agent to ﬁnd a policy ⇡ in order to maximize the performance v ⇡ , given only the episodic data D
= { ( s ⌧ h , a ⌧ h , r ⌧ h , s ⌧ h +1 ) }
h 2 [ H ] ⌧ 2 [ n ] 4 rolled out from some ﬁxed and possibly unknown behavior policy µ , which means we cannot change µ and in particular we do not assume the functional knowledge of µ .
In conclusion, based on the batch data D and a targeted accuracy ✏ > 0 , the agent seeks to ﬁnd a policy ⇡ alg such that v ?
v ⇡ alg
 ✏ .
2. A SSUMPTIONS IN OFFLINE RL
In order to show that our privacy-preserving algorithms can generate near optimal policy, certain coverage assumptions are needed.
In this section, we will list the assumptions we use in this paper.
Assumptions for tabular setting.
Assumption 2.1 ((Liu et al., 2019)) .
There exists one optimal policy ⇡ ? , such that ⇡ ? is fully covered by µ , i.e. 8 s h , a h 2 S ⇥ A , d ⇡ ?
h ( s h , a h ) > 0 only if d µh ( s h , a h ) > 0 .
Furthermore, we denote the trackable set as C h :=
{ ( s h , a h ) :
d µh ( s h , a h ) > 0 } .
Assumption 2.1 is the weakest assumption needed for accurately learning the optimal value v ? by requiring µ to trace the state-action space of one optimal policy ( µ can be agnostic at other locations).
Similar to (Yin & Wang, 2021b), we will use Assumption 2.1 for the tabular part of this paper, which enables comparison between our sample complexity to the conclusion in (Yin & Wang, 2021b), whose algorithm serves as a non-private baseline.
Assumptions for linear setting.
First, we deﬁne the expectation of covariance matrix under the behavior policy µ for all time step h 2 [ H ] as below:
[Equation]
As have been shown in (Wang et al., 2021;
Yin et al., 2022), learning a near-optimal policy from ofﬂine data requires coverage assumptions.
Here in linear setting, such coverage is characterized by the minimum eigenvalue of ⌃ ph .
Similar to (Yin et al., 2022), we apply the following assumption for the sake of comparison.
Assumption 2.2 (Feature Coverage, Assumption 2 in (Wang et al., 2021)) .
The data distributions µ satisfy the minimum eigenvalue condition: 8 h 2 [ H ] ,  h :=
min ( ⌃ ph ) > 0 .
Furthermore, we denote  =
min
h  h .
2. D IFFERENTIAL P RIVACY IN OFFLINE RL
In this work, we aim to design privacy-preserving algorithms for ofﬂine RL.
We apply differential privacy as the formal notion of privacy.
Below we revisit the deﬁnition of differential privacy.
Deﬁnition 2.3 (Differential Privacy (Dwork et al., 2006)) .
A randomized mechanism M satisﬁes ( ✏ ,  ) -differential privacy ( ( ✏ ,  ) -DP) if for all neighboring datasets U, U 0 that differ by one data point and for all possible event E in the output range, it holds that
[Equation]
When  = 0 , we say pure DP, while for  > 0 , we say approximate DP.
In the problem of ofﬂine RL, the dataset consists of several trajectories, therefore one data point in
Deﬁnition 2.3 refers to one single trajectory.
Hence the deﬁnition of Differential Privacy means that the difference in the distribution of the output policy resulting from replacing one trajectory in the dataset will be small.
In other words, an adversary can not infer much information about any single trajectory in the dataset from the output policy of the algorithm.
During the whole paper, we will use zCDP (deﬁned below) as a surrogate for DP, since it enables cleaner analysis for privacy composition and Gaussian mechanism.
The properties of zCDP
(e.g., composition, conversion formula to DP) are deferred to Appendix E.3.
Deﬁnition 2.4 (zCDP (Dwork & Rothblum, 2016; Bun & Steinke, 2016)) .
A randomized mechanism
M satisﬁes ⇢ -Zero-Concentrated Differential Privacy ( ⇢ -zCDP), if for all neighboring datasets U, Uand all ↵ 2 (1 , 1 ) ,
[Equation]
where D ↵ is the Renyi-divergence (Van Erven & Harremos, 2014).
Finally, we go over the deﬁnition and privacy guarantee of Gaussian mechanism.
Deﬁnition 2.5 (Gaussian Mechanism (Dwork et al., 2014)) .
Deﬁne the ` 2 sensitivity of a function f : N X 7!
R d as
[Equation]
The Gaussian mechanism M with noise level  is then given by
[Equation]
Lemma 2.6 (Privacy guarantee of Gaussian mechanism (Dwork et al., 2014; Bun & Steinke, 2016)) .
Let f : N X 7!
R d be an arbitrary d-dimensional function with ` 2 sensitivity  2 .
Then for any ⇢ > 0 ,
Gaussian Mechanism with parameter  2 =  22 2 ⇢ satisﬁes ⇢ -zCDP.
In addition, for all 0 <  , ✏ < 1 ,
Gaussian Mechanism with parameter  =  2 ✏
[Equation]
We emphasize that the privacy guarantee covers any input data.
It does not require any distributional assumptions on the data.
The RL-speciﬁc assumptions (e.g., linear MDP and coverage assumptions) are only used for establishing provable utility guarantees.
R ESULTS UNDER TABULAR MDP: DP-APVI (A LGORITHM 1)
For reinforcement learning, the tabular MDP setting is the most well-studied setting and our ﬁrst result applies to this regime.
We begin with the construction of private counts.
Private Model-based Components.
Given data D
= { ( s ⌧ h , a ⌧ h , r ⌧ h , s ⌧ h +1 ) }
h 2 [ H ] ⌧ 2 [ n ] , we denote n s h ,a h :=
P n ⌧ =1 1 [ s ⌧ h , a ⌧ h =
s h , a h ] be the total counts that visit ( s h , a h ) pair at time h and n s h ,a h ,s h +1 :=
P n ⌧ =1 1 [ s ⌧ h , a ⌧ h , s ⌧ h
+1 =
s h , a h , s h
+1 ] be the total counts that visit ( s h , a h , s h +1 ) pair at time h , then given the budget ⇢ for zCDP, we add independent Gaussian noises to all the counts:
[Equation]
However, after adding noise, the noisy counts n 0 may not satisfy n 0 s h ,a h =
[Equation]
To address this problem
[Equation]
we choose the private counts of visiting numbers as the solution to the following optimization problem (here E ⇢ =
[Equation]
such that
[Equation]
Remark 3.1.
The optimization problem (3) can be reformulated as:
[Equation]
Note that (4) is a Linear Programming problem with S + 1 variables and 2 S
+ 2 (one constraint on absolute value is equivalent to two linear constraints) linear constraints, which can be solved efﬁciently by the simplex method (Ficken, 2015) or other provably efﬁcient algorithms (Nemhauser & Wolsey, 1988).
In addition, if we do not solve this optimization problem and directly take e
[Equation]
1 , we can only derive | e n s h ,a
[Equation]
n s h ,a h |  e
O ( p SE ⇢ ) through concentration on summation of S i.i.d.
Gaussian noises.
In contrast, solving (3) ensures that | e
[Equation]
with high probability 5 .
The private estimation of the transition kernel is deﬁned as:
[Equation]
if e n s h ,a h > E ⇢ and e
[Equation]
Remark 3.2.
Different from the transition kernel estimate in previous works (Vietri et al., 2020;
Chowdhury & Zhou, 2021) that may not be a distribution, we have to ensure that ours is a probability distribution, because our Bernstein type pessimism (line 5 in Algorithm 1) needs to take variance over this transition kernel estimate.
The intuition behind the construction of our private transition kernel is that, for those state-action pairs with e n s h ,a h  E ⇢ , we can not distinguish whether the non-zero private count comes from noise or actual visitation.
Therefore we only take the empirical estimate of the state-action pairs with sufﬁciently large e n s h ,a h .
e n s h ,a h , e n s h ,a h ,s h +1 as
3), e
P h ( s h
+
| s h , a h ) as ( e
V H +1 ( · ) 0 .
E ⇢
[Equation] [Equation]
⇢
Var e
Psh,ah e
V h +1 ) · ◆ e n sh,ah  E ⇢ + C 2 SHE ⇢ · ◆ e n sh,ah s h ,a h >
⇢ ph ( · e
Q h ( ·
· )   h ( · , · )
⇡ h ( ·| s h ) argmax ⇡ h h Q h ( s h , · ) , ⇡ h ( ·| s h ) i and e
V h ( s h ) h Q h ( s h , · ) , b ⇡ h ( ·| s h ) i .
9: end for 10: Output: { b ⇡ h } .
Algorithmic design.
Our algorithmic design originates from the idea of pessimism, which holds conservative view towards the locations with high uncertainty and prefers the locations we have more conﬁdence about.
Based on the Bernstein type pessimism in APVI (Yin & Wang, 2021b), we design a similar pessimistic algorithm with private counts to ensure differential privacy.
If we replace e n and e
P with n and b
P 6 , then our DP-APVI (Algorithm 1) will degenerate to APVI.
Compared to the pessimism deﬁned in APVI, our pessimistic penalty has an additional term e
O [Equation]
SHE ⇢ e
[Equation]
⌘ , which accounts for the additional pessimism due to our application of private statistics.
We state our main theorem about DP-APVI below, the proof sketch is deferred to Appendix C.1 and detailed proof is deferred to Appendix C due to space limit.
Theorem 3.3.
DP-APVI (Algorithm 1) satisﬁes ⇢
-zCDP.
Furthermore, under Assumption 2.1, denote ¯ d m := min
h 2 [ H ] { d µh ( s h , a h ) :
d µh ( s h , a h ) > 0 } .
For any 0 <  < 1 , there exists constant c 1 > 0 , such that when n > c 1 · max { H 2 , E ⇢ } / ¯ d m · ◆ ( ◆ = log( HSA/  ) ), with probability 1   , the output policy b ⇡ of DP-APVI satisﬁes ( e
O hides constants and Polylog terms
[Equation] [Equation]
Comparison to non-private counterpart APVI (Yin & Wang, 2021b).
According to Theorem 4.1 in (Yin & Wang, 2021b), the sub-optimality bound of APVI is for large enough n , with high probability, the output b
[Equation] [Equation]
Compared to our Theorem 3.3, the additional sub-optimality bound due to differential privacy is e
[Equation]
E ⇢ n
[Equation]
◆ .
In the most popular regime where the privacy budget ⇢ or ✏ is a constant, the additional term due to differential privacy appears as a lower order term, hence becomes negligible as the sample complexity n becomes large.
Comparison to Hoeffding type pessimism.
We can simply revise our algorithm by using Hoeffding type pessimism, which replaces the pessimism in line 5 with C 1 H ·
[Equation]
ah .
Then with a similar proof schedule, we can arrive at a sub-optimality bound that with high probability,
[Equation]
Compared to our Theorem 3.3, our bound is tighter because we express the dominate term by the system quantities instead of explicit dependence on H (and
Var P h ( ·| s h ,a h ) ( V ?
h +1 ( · ))  H 2 ).
In addition, we highlight that according to Theorem G.1 in (Yin & Wang, 2021b), our main term nearly matches the non-private minimax lower bound.
For more detailed discussions about our main term and how it subsumes other optimal learning bounds, we refer readers to (Yin & Wang, 2021b).
Apply Laplace Mechanism to achieve pure DP.
To achieve Pure DP instead of ⇢ -zCDP, we can simply replace Gaussian Mechanism with Laplace Mechanism (deﬁned as Deﬁnition E.19).
Given privacy budget for Pure DP ✏ , since the ` 1 sensitivity of { n s h ,a h } [ { n s h ,a h ,s h +1 } is  1 = 4 H , we can add independent Laplace noises Lap( 4 H ✏ ) to each count to achieve ✏ -DP due to Lemma E.20.
Then by using E ✏ = e
O
H ✏  instead of E ⇢ and keeping everything else ((3), (5) and Algorithm 1) the same, we can reach a similar result to Theorem 3.3 with the same proof schedule.
The only difference is that here the additional learning bound is
[Equation]
⌘ , which still appears as a lower order term.
R ESULTS UNDER LINEAR MDP: DP-VAPVI(A LGORITHM 2)
In large MDPs, to address the computational issues, the technique of function approximation is widely applied, and linear MDP is a concrete model to study linear function approximations.
Our second result applies to the linear MDP setting.
Generally speaking, function approximation reduces the dimensionality of private releases comparing to the tabular MDPs.
We begin with private counts.
Private Model-based Components.
Given the two datasets D and D 0 (both from µ ) as in Algorithm 2, we can apply variance-aware pessimistic value iteration to learn a near optimal policy as in
VAPVI (Yin et al., 2022).
To ensure differential privacy, we add independent Gaussian noises to the 5 H statistics as in DP-VAPVI (Algorithm 2) below.
Since there are 5 H statistics, by the adaptive composition of zCDP (Lemma E.17), it sufﬁces to keep each count ⇢
-zCDP, where ⇢ 0 = ⇢ 5 H .
In
DP-VAPVI, we use  1 ,  2 ,  3 , K 1 , K 2 8 to denote the noises we add.
For all  i , we directly apply
Gaussian Mechanism.
For K i , in addition to the noise matrix 1 p 2 ( Z + Z > ) , we also add E 2 I d to ensure that all K i are positive deﬁnite with high probability (The detailed deﬁnition of E, L can be found in Appendix A).
Budget
[Table]
, e
V
Sample  1 ⇠ N ⇣ 0 , 2 H 4 ⇢ 0 I d ⌘ ,  2 ,  3 ⇠ N ⇣ 0 , 2 H 2 ⇢ 0 I d ⌘ ,
I
+
⇣ 0 , 14 ⇢⌘ ( i.i.d. ) , E = e
O ⇣q
Hd ⇢ e
O ⇣
H 2 L  + H 4 E p d  3 / 2 + H 3 p d ⌘ .
. , 1 do 4:
Set e ⌃ h
P
K ⌧ =1  (¯ s ⌧ h , ¯ a ⌧ h )  (¯ s ⌧ h , ¯ a ⌧ h ) >
+  I + K5:
Set e  h e ⌃  1 h [
P
K ⌧ =1  (¯ s ⌧ h , ¯ a ⌧ h ) · e
V h +1 (¯ s ⌧ h +1 ) 2 +  1 ] 6:
Set e ✓ h e ⌃  1 h
P
K ⌧ =1  (¯ s ⌧ h , ¯ a ⌧ h ) · e
V h +1 (¯ s ⌧ h
+
+  2 ] 7:
Set ⇥g
Var h e
V h
+ ⇤ ( · , · ) ⌦  ( · , · ) ,
e  h ↵ [ 0 , ( H  h +1) 2 ]  ⇥⌦  ( · , · ) , e ✓ h ↵ [0 ,H  h
+
⇤ 8:
Set e  h ( · , · )max { 1 , g
Var h e
V h +1 ( · , · ) } 9:
Set
e
⇤ h
P
K ⌧ =1  ( s ⌧ h , a ⌧ h )  ( s ⌧ h , a ⌧ h ) > / e  2 h ( s ⌧ h , a ⌧ h )
+  I + K10:
Set e w h
e
⇤  1 h ⇣P
K ⌧ =1  ( s ⌧ h , a ⌧ h ) · ⇣ r ⌧ h + e
V h
+
( s ⌧ h +1 )
⌘ / e  2 h ( s ⌧ h , a ⌧ h ) + ⌘ 11:
Set  h ( · , · )
C p d · ⇣  ( · , · ) >
e ⇤  1 h  ( · , · ) ⌘ 1 / 2 + DK 12:
Set ¯ Q h ( · , · )  ( · , · ) > e w h   h ( · , · ) 13:
Set b
Q h ( · , · ) min  ¯ Q h ( · , · ) , H  h + 1 + 14:
Set b ⇡ h ( · | · ) argmax ⇡ h ⌦ b
Q h ( · , · ) , ⇡ h ( · | · ) ↵ A , e
V h ( · ) max ⇡ h ⌦ b
Q h ( · , · ) , ⇡ h ( · | · ) ↵ A 15: end for 16: Output: { b ⇡ h } Hh =1 .
Below we will show the algorithmic design of DP-VAPVI (Algorithm 2).
For the ofﬂine dataset, we divide it into two independent parts with equal length:
D = { ( s ⌧ h , a ⌧ h , r ⌧ h , s ⌧ h +1 ) }
h 2 [ H ] ⌧
[ K ] and
D
= { (¯ s ⌧ h , ¯ a ⌧ h , ¯ r ⌧ h , ¯ s ⌧ h +1 ) }
h 2 [ H ] ⌧ 2 [ K ] .
One for estimating variance and the other for calculating Q -values.
Estimating conditional variance.
The ﬁrst part (line 4 to line 8) aims to estimate the conditional variance of e
V h +1 via the deﬁnition of variance:
[Var h
[Equation]
([ P h e
V h
+1 ]
( s, a )) 2 .
For the ﬁrst term, by the deﬁnition of linear MDP, it holds that h
P
[Equation]
e
[Equation]
h
[Equation]
i
[Equation] [Equation]
) =
[Equation]
s,
[Equation] [Equation]
) i .
We can estimate  h =
R S e
V 2 h +1 ( s 0 )
d ⌫ h ( s 0 ) by applying ridge regression.
Below is the output of ridge regression with raw statistics without noise:
[Equation]
where deﬁnition of ¯ ⌃ h can be found in Appendix A. Instead of using the raw statistics, we replace them with private ones with Gaussian noises as in line 5.
The second term is estimated similarly in line 6.
The ﬁnal estimator is deﬁned as in line 8:
[Equation]
[Equation]
Variance-weighted LSVI.
Instead of directly applying LSVI (Jin et al., 2021), we can solve the variance-weighted LSVI (line 10).
The result of variance-weighted LSVI with non-private statistics is shown below:
[Equation]
=
[Equation]
⇤
[Equation]
K
[Equation]
k =
[Equation]
[Equation]
kh
[Equation]
a kh  · h r kh + e
V h +  s kh + i e
2 h ( s kh , a kh ) , where deﬁnition of b
⇤ h can be found in Appendix A.
For the sake of differential privacy, we use private statistics instead and derive the e w h as in line 10.
Our private pessimism.
Notice that if we remove all the Gaussian noises we add, our DP-VAPVI (Algorithm 2) will degenerate to VAPVI (Yin et al., 2022).
We design a similar pessimistic penalty using private statistics (line 11), with additional DK accounting for the extra pessimism due to DP.
Main theorem.
We state our main theorem about DP-VAPVI below, the proof sketch is deferred to Appendix D.1 and detailed proof is deferred to Appendix D due to space limit.
Note that quantities M i , L, E can be found in Appendix A and brieﬂy, L = e
O ( p
H 3 d/ ⇢ ) , E = e
O ( p
Hd/ ⇢ ) .
For the sample complexity lower bound, within the practical regime where the privacy budget is not very small, max {M i } is dominated by max { e
O ( H 12 d 3 /  5 ) , e
O ( H 14 d/  5 ) } , which also appears in the sample complexity lower bound of VAPVI (Yin et al., 2022).
The  2 V ( s, a ) in Theorem 4.1 is deﬁned as max { 1 , Var P h ( V )( s, a ) } for any V .
Theorem 4.1.
DP-VAPVI (Algorithm 2) satisﬁes ⇢
-zCDP.
Furthermore, let K be the number of episodes.
Under the condition that K > max {M 1 , M 2 , M 3 , M 4 } and p d > ⇠ , where ⇠ := sup V 2 [0 ,H ] , s
⇠ P h ( s,a ) , h 2 [ H ]  [Equation] + [Equation]
s [Equation]
)  ( T h V )( s,a )  V ( s,a )  , for any 0 <  <  , with probability 1   , for all policy ⇡ simultaneously, the output b ⇡ of DP-VAPVI satisﬁes ( e
O hides constants and Polylog terms)
[Equation]
(9) where ⇤ h =
P
Kk
[Equation]
[Equation]
kh ) ·  (
[Equation]
kh , [Equation] kh
[Equation]
e
Vh +1( skh,akh )
+  I d and D = e
O ⇣
H 2 L  + H 4 E p d  3 /+ H 3 p d ⌘ .
In particular, deﬁne ⇤ ?
h =
P
[Equation]
[Equation]
[Equation]
skh,akh ) +  I d , we have with probability 1   ,
[Equation]
Comparison to non-private counterpart VAPVI (Yin et al., 2022).
Plugging in the deﬁnition of L, E (Appendix A), under the meaningful case that the privacy budget is not very large, DH is dominated by
[Equation]
According to Theorem 3.2 in (Yin et al., 2022), the sub-optimality bound of VAPVI is for sufﬁciently large K , with high probability, the output [Equation] ⇡ satisﬁes:
[Equation]
Compared to our Theorem 4.1, the additional sub-optimality bound due to differential privacy is e
[Equation]
In the most popular regime where the privacy budget ⇢ or ✏ is a constant, the additional term due to differential privacy also appears as a lower order term.
Instance-dependent sub-optimality bound.
Similar to DP-APVI (Algorithm 1), our DP-VAPVI (Algorithm 2) also enjoys instance-dependent sub-optimality bound.
First, the main term in (10) improves PEVI (Jin et al., 2021) over O ( p d ) on feature dependence.
Also, our main term admits no explicit dependence on H , thus improves the sub-optimality bound of PEVI on horizon dependence.
For more detailed discussions about our main term, we refer readers to (Yin et al., 2022).
S IMULATIONS
In this section, we carry out simulations to evaluate the performance of our DP-VAPVI (Algorithm 2), and compare it with its non-private counterpart VAPVI (Yin et al., 2022) and
another pessimism-based algorithm PEVI (Jin et al., 2021) which does not have privacy guarantee.
Experimental setting.
We evaluate DP-VAPVI (Algorithm 2) on a synthetic linear MDP example that originates from the linear MDP in (Min et al., 2021;
Yin et al., 2022) but with some modiﬁcations.
For details of the linear MDP setting, please refer to
Appendix F.
The two MDP instances we use both have horizon H = 20 .
We compare different algorithms in ﬁgure 1(a), while in ﬁgure 1(b), we compare our DP-VAPVI with different privacy budgets.
When doing empirical evaluation, we do not split the data for DP-VAPVI or VAPVI and for DP-VAPVI, we run the simulation for 5 times and take the average performance.
[Figure]
Results and discussions.
From Figure 1, we can observe that DP-VAPVI (Algorithm 2) performs slightly worse than its non-private version VAPVI (Yin et al., 2022).
This is due to the fact that we add Gaussian noise to each count.
However, as the size of dataset goes larger, the performance of DP-VAPVI will converge to that of VAPVI, which supports our theoretical conclusion that the cost of privacy only appears as lower order terms.
For DP-VAPVI with larger privacy budget, the scale of noise will be smaller, thus the performance will be closer to VAPVI, as shown in ﬁgure 1(b).
Furthermore, in most cases, DP-VAPVI still outperforms PEVI, which does not have privacy guarantee.
This arises from our privitization of variance-aware LSVI instead of LSVI.
C ONCLUSION AND FUTURE WORKS
In this work, we take the ﬁrst steps towards the well-motivated task of designing private ofﬂine RL algorithms.
We propose algorithms for both tabular MDPs and linear MDPs, and show that they enjoy instance-dependent sub-optimality bounds while guaranteeing differential privacy (either zCDP or pure DP).
Our results highlight that the cost of privacy only appears as lower order terms, thus become negligible as the number of samples goes large.
Future extensions are numerous.
We believe the technique in our algorithms (privitization of
Bernstein-type pessimism and variance-aware LSVI) and the corresponding analysis can be used in online settings too to obtain tighter regret bounds for private algorithms.
For the ofﬂine RL problems, we plan to consider more general function approximations and differentially private (deep) ofﬂine RL which will bridge the gap between theory and practice in ofﬂine RL applications.
Many techniques we developed could be adapted to these more general settings.