Private Isotonic Regression
Abstract
In this paper, we consider the problem of differentially private (DP) algorithms for isotonic regression.
For the most general problem of isotonic regression over a partially ordered set (poset) X and for any Lipschitz loss function, we obtain a pure-DP algorithm that, given n input points, has an expected excess empirical risk of roughly width( X ) · log |X| /n , where width( X ) is the width of the poset.
In contrast, we also obtain a near-matching lower bound of roughly (width( X ) + log |X| )
/n , that holds even for approximate-DP algorithms.
Moreover, we show that the above bounds are essentially the best that can be obtained without utilizing any further structure of the poset.
In the special case of a totally ordered set and for ℓ 1 and ℓ 22 losses, our algorithm can be implemented in near-linear running time; we also provide extensions of this algorithm to the problem of private isotonic regression with additional structural constraints on the output function.
Introduction
Isotonic regression is a basic primitive in statistics and machine learning, which has been studied at least since the 1950 s [4, 9]; see also the textbooks on the topic [5, 38].
It has seen applications in numerous fields including medicine [31, 39] where the expression of an antigen is modeled as a monotone function of the DNA index and WBC count and education [19], where isotonic regression was used to predict college GPA using high school GPA and standardized test scores.
Isotonic regression is also arguably the most common non-parametric method for calibrating machine learning models [51], including modern neural networks [23]
In this paper we study isotonic regression with a differential privacy (DP) constraint on the output model.
DP [17, 16] is a highly popular notion of privacy for algorithms and machine learning primitives and has seen increased adoption due to its powerful guarantees and properties [37, 43].
Despite the plethora of work on DP statistics and machine learning (see Section 5 for related work), ours is, to the best of our knowledge, the first to study DP isotonic regression.
In fact, we consider the most general version of the isotonic regression problem.
We first set up some notation to describe our results.
Let ( X , ≤ ) be any partially ordered set ( poset ).
A function f : X → [0 , 1] is monotone
if and only if f ( x ) ≤ f ( x ′ ) for all x ≤ x ′ .
For brevity, we use F ( X , Y ) to denote the set of all monotone functions from X to Y ; throughout, we consider Y ⊆
Let [ n ] denote { 1 , . . . , n } .
Given an input dataset D = { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } ∈ ( X × [0 , 1]) n , let the empirical risk of a function f : X → [0 , 1] be L ( f ; D ) :
= 1 n (cid:80) i ∈ [ n ] ℓ ( f ( x i ) , y i ) , where ℓ : [0 , 1] × [0 , 1] → R is a loss function .
We study private isotonic regression in the basic machine learning framework of empirical risk minimization.
Specifically, the goal of the isotonic regression problem, given dataset D
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } ∈ ( X × [0 , 1]) n , is to find a monotone function f : X → [0 , 1] that minimizes L ( f ; D ) .
The excess empirical risk of a function f is defined as L ( f ; D ) − L ( f ∗ ; D ) where f ∗ := argmin g ∈F ( X , Y ) L ( g ; D ) .
1.1 Our Results
General Posets.
Our first contribution is to give nearly tight upper and lower bounds for any poset, based on its width , as stated below (see Section 4 for a formal definition.)
Theorem 1 (Upper Bound for General Poset) .
Let X be any finite poset and let ℓ be an L -Lipschitz loss function.
For any ε ∈ (0 , 1] , there is an ε -DP algorithm for isotonic regression for ℓ with expected excess empirical risk at most O (cid:16) L · width( X ) · log |X|· (1+log 2 ( εn )) εn (cid:17) .
Theorem 2 (Lower Bound for General Poset; Informal) .
For any ε ∈ (0 , 1] and any δ < 0 .
01 · ε/ |X| , any ( ε, δ ) -DP algorithm for isotonic regression for a “nice” loss function ℓ must have expected excess empirical risk
[Equation]
While our upper and lower bounds do not exactly match because of the multiplication-vs-addition of log |X| , we show in Section 4.3 that there are posets for which each bound in tight.
In other words, this gap cannot be closed for generic posets.
Totally Ordered Sets.
The above upper and lower bounds immediately translate to the case of totally ordered sets, by plugging in width( X )
= 1 .
More importantly, we give efficient algorithms in this case, which runs in time ˜ O
( n 2 + n log |X| ) for general loss function ℓ , and in nearly linear ˜ O
( n · log |X| ) time for the widely-studied ℓ 22 - and ℓ 1 -losses 1 .
Theorem 3.
For all finite totally ordered sets X , L -Lipschitz loss functions ℓ , and ε ∈ (0 , 1] , there is an ε -DP algorithm for isotonic regression for ℓ with expected excess empirical risk
O (cid:16) L · (log |X| ) · (1+log 2 ( εn )) εn (cid:17) .
The running time of this algorithm is ˜ O ( n 2 + n log |X| ) in general and can be improved to ˜ O ( n log |X| ) for ℓ 1 and ℓ 22 losses.
We are not aware of any prior work on private isotonic regression.
A simple baseline algorithm for this problem would be to use the exponential mechanism over the set of all monotone functions taking values in a discretized set, to choose one with small loss.
We show in Appendix A that this achieves an excess empirical risk of O ( L · (cid:112) width( X ) · log |X| /εn ) , which is quadratically worse than the bound in Theorem 1.
Moreover, even in the case of a totally ordered set, it is unclear how to implement such a mechanism efficiently.
We demonstrate the flexibility of our techniques by showing that it can be extended to variants of isotonic regression where, in addition to monotonicity, we also require f to satisfy additional properties.
For example, we may want f to be L f -Lipchitz for some specified L f > 0 .
Other constraints we can handle include k -piecewise constant, k -piecewise linear, convexity, and concavity.
For each of these constraints, we devise an algorithm that yields essentially the same error compared to the unconstrained case and still runs in polynomial time.
Theorem 4.
For all finite totally ordered sets X , L -Lipschitz loss functions ℓ , and ε ∈ (0 , 1] , there is an ε -DP algorithm for k -piecewise constant, k -piecewise linear, Lipchitz, convex, or concave isotonic regression for ℓ with expected excess empirical risk ˜ O (cid:16) L · (log |X| ) εn (cid:17) .
The running time of this algorithm is ( n |X| )
O (1) .
Organization.
We next provide necessary background on DP.
In Section 3, we prove our results for totally ordered sets (including Theorem 3).
We then move on to discuss general posets in Section 4.
Section 5 contains additional related work.
Finally, we conclude with some discussion in
Section 6.
Due to space constraints, we omit some proofs from the main body; these can be found in the Appendix.
Background on Differential Privacy
Two datasets D = { (( x 1 , y 1 ) , . . .
, ( x n , y n ) } and D ′ =
{ ( x ′ 1 , y ′ 1 ) , . .
, ( x ′ n , y ′ n ) } are said to be neighboring , denoted D ∼ D ′ , if there is an index i ∈ [ n ] such that ( x j , y j ) =
( x ′ j , y ′ j ) for all j ∈ [ n ] \ { i } .
We recall the formal definition of differential privacy [18, 16]:
Definition 5 (Differential Privacy (DP) [18, 16]) .
Let ε > 0 and δ ∈
A randomized algorithm
M : X n → Y is ( ε, δ ) -differentially private ( ( ε, δ ) -DP ) if, for all D ∼ D ′ and all (measurable) outcomes S ⊆
Y , we have that Pr[ M ( D ) ∈ S ] ≤ e ε · Pr[ M ( D ′ ) ∈ S ] + δ .
We denote ( ε, 0) -DP as ε -DP (aka pure -DP).
The case when δ > 0 is referred to as approximate -DP.
We will use the following composition theorems throughout our proofs.
Lemma 6.
( ε, δ )
-DP satisfies the following:
• Basic Composition: If mechanisms M 1 , . . .
, M t are such that M i satisfies ( ε i , δ i )
-DP, then the composed mechanism ( M 1 ( D ) , . .
, M t ( D ))
satisfies ( (cid:80) i ε i , (cid:80) i δ i ) -DP.
This holds even under adaptive composition, where each M i can depend on the outputs of M 1 , . . .
, M i − 1 .
• Parallel Composition [33]: If a mechanism M satisfies ( ε, δ ) -DP, then for any partition of D
= D 1 ⊔ · · · ⊔ D t , the composed mechanism given as ( M ( D 1 ) , . .
, M ( D t ))
satisfies ( ε, δ ) - DP.
Exponential Mechanism.
The
exponential mechanism solves the basic task of selection in data analysis: given a dataset D ∈ Z n and a set A of options, it outputs the (approximately) best option, where “best” is defined by a scoring function s : A × Z n → R .
The ε -DP exponential mechanism [34] is the randomized mechanism M : Z n → A given by
[Equation]
where ∆ s := sup D ∼ D ′ max a ∈A | s ( a, D ) − s ( a, D ′ ) | is the sensitivity of the scoring function.
Lemma 7 ([34]) .
For M being the ε -DP exponential mechanism, it holds for all D ∈ Z n that
[Equation]
Lower Bound for Privatizing Vectors.
Lower bounds for DP algorithms that can output a binary vector that is close (say, in the Hamming distance) to the input are well-known.
Lemma 8 (e.g., [32]) .
Let ε, δ > 0 , m ∈ N , let the input domain be { 0 , 1 } m and let two vectors z , z ′ ∈ { 0 , 1 } m be neighbors if and only if ∥ z − z ′ ∥ 0 ≤ 1 .
Then, for any ( ε, δ ) -DP algorithm
M : { 0 , 1 } m → { 0 , 1 } m , we have E z ∼{ 0 , 1 } m
[ ∥M ( z ) − z ∥ 0 ] ≥ e − ε · m · 0 . 5 · (1 − δ ) .
It is also simple to extend the lower bound for the case where the vector is not binary, as stated below.
We defer the full proof to Appendix B.
Lemma 9.
Let D, m be any positive integer such that D ≥ 2 , let the input domain be [ D ] m and let two vectors z ,
z ′ ∈
[ D ] m be neighbors if and only if ∥ z − z ′ ∥ 0 ≤ 1 .
Then, for any (ln( D/ 2) , 0 . 25) -
DP algorithm M :
[ D ] m → [ D ] m , we have that E z ∼
[ D ] m [ ∥M ( z ) − z ∥ 0 ] ≥ Ω( m ) .
Group Differential Privacy.
For any neighboring relation ∼ , we write ∼ k as a neighboring relation where D ∼ k D
iff there is a sequence D
= D 0 , . .
, D k ′ =
D ′ for some k ′ ≤ k such that
D i − 1 ∼ D i for all i ∈
[ k ′ ] .
Fact 10 (e.g., [41]) .
Let ε > 0 , δ ≥ 0 and k ∈ N .
Suppose that M is an ( ε, δ ) -DP algorithm for the neighboring relation ∼ .
Then M is (cid:16) kε, e kε − 1 e ε − 1 · δ (cid:17) -DP for the neighboring relation ∼ k .
DP Isotonic Regression over Total Orders
We first focus on the “one-dimensional” case where X is totally ordered; for convenience, we assume that X
= [ m ] where the order is the natural order on integers.
We first present an efficient algorithm for the this case and then a matching lower bound.
3.1 An Efficient Algorithm
To describe our algorithm, it will be more convenient to use the unnormalized version of the empirical risk, which we define as L abs ( f ; D ) :
= (cid:80) ( x
,y ) ∈ D ℓ ( f ( x ) , y ) .
We now provide a high-level overview of our algorithm.
Any monotone function f : [ m ] → [0 , 1] contains a (not necessarily unique) threshold α ∈ { 0 } ∪ [ m ] such that f ( a ) ≥ 1 / 2 for all a > α and f ( a ) ≤ 1 / 2 for all a ≤ α .
Our algorithm works by first choosing this threshold α in a private manner using the exponential mechanism.
The choice of α partitions [ m ] into [ m ] >α := { a ∈ [ m ] | a > α } and [ m ] ≤ α :=
{ a ∈ [ m ] | a ≤ α } .
The algorithm recurses on these two parts to find functions f > : [ m ] >α
→
[1 / 2 , 1] and f ≤ : [ m ] ≤ α → [0 , 1 / 2] , which are then glued to obtain the final function.
In particular, the algorithm proceeds in T stages, where in stage t , the algorithm starts with a partition of [ m ] into 2 t intervals { P i,t | i ∈ { 0 , . .
, 2 t − 1 }} , and the algorithm eventually outputs a monotone function f such that f ( x ) ∈
[ i/ 2 t , ( i + 1) / 2 t ] for all x ∈ P i,t .
This partition is further refined for stage t + 1 by choosing a threshold α i,t in P i,t and partitioning P i,t into P >α i,t i,t and
P ≤ α i,t i,t .
In the final stage, the function f is chosen to be the constant i/ 2 T − 1 over P i,T − 1 .
Note that the algorithm may stop at T
=
Θ ε (log n ) because the Lipschitzness of ℓ ensures that assigning each partition to the constant i/ 2 T − 1 cannot increase the error by more than L/ 2 T ≤ O ε ( L/n ) .
We already have mentioned above that each α i,t has to be chosen in a private manner.
However, if we let the scoring function directly be the unnormalized empirical risk, then its sensitivity remains as large as L even at a large stage t .
This is undesirable because the error from each run of the exponential mechanism can be as large as O ( L · log m ) but there are as many as 2 t runs in stage t .
Adding these error terms up would result in a far larger total error than desired.
To circumvent this, we observe that while the sensitivity can still be large, they are mostly “ineffective” because the function range is now restricted to only an interval of length 1 / 2 t .
Indeed, we may use the following “clipped” version of the loss function which has low sensitivity of L/ 2 t instead.
Definition 11 (Clipped Loss Function) .
For a range [ τ, θ ] ⊆
[0 , 1] , let ℓ
[ τ,θ ] : [ τ, θ ] × [0 , 1] → R be given as ℓ [ τ,θ ] (ˆ y, y ) :
=
ℓ (ˆ y, y ) − min y ′ ∈
[ τ,θ ] ℓ ( y ′ , y ) .
Similar to above, we also define
L
[Equation]
[Equation]
) , y i ) .
Observe that range( ℓ [ τ,θ ] ) ⊆
[0 , L · ( θ − τ )] , since ℓ is L -Lipschitz.
In other words, the sensitivity of L abs[ τ,θ ] ( f ; D ) is only L · ( θ − τ ) .
Algorithm 1 contains a full description.
Proof of Theorem 3.
Before proceeding to prove the algorithm’s privacy and utility guarantees, we note that the output f is indeed monotone since for every x ′ < x that gets separated when we partition P i,t into P 2 i,t +1 , P
i +1 ,t +1 , we must have x ′ ∈
P 2 i,t +1 and x ∈ P 2 i +1 ,t +1 .
Privacy Analysis.
Since the exponential mechanism is ε ′
-DP
and the dataset is partitioned with the exponential mechanism being applied only to each partition once, the parallel composition property (Lemma 6) implies that the entire subroutine for each t is ε
′ -DP.
Thus, by basic composition (Lemma 6), it follows that Algorithm 1 is ε -DP (since ε = ε ′ T ).
Utility Analysis.
Since the sensitivity of score i,t ( · ) is at most L/ 2 t , we have from Lemma 7, that for all t ∈ { 0 , . .
, T − 1 } and i ∈ { 0 , 1 , . . .
, 2 t } ,
[Equation]
Let h i,t denote argmin h ∈F ( P i,t , [ i/ 2 t , ( i +1) / 2 t ])
L abs ( h ; D i,t ) (with ties broken arbitrarily).
Then, let ˜ α
i,t denote the largest element in P i,t such that h i,t (˜ α i,t ) ≤ ( i +1 / 2) / 2 t ; namely, ˜ α
i,t is the optimal threshold when restricted to D i,t .
Under this notation, we have that
[Equation]
Algorithm
Input: X = [ m ] , dataset D
=
{ ( x 1 , y 1 ) , . . .
, ( x n , y n ) } , DP parameter ε .
Output: Monotone function f : [ m ] → [0 , 1] .
T ← ⌈ log( εn ) ⌉ ε ′ ← ε/T
P 0 , 0 ← [ m ] for t = 0 , . .
, T − 1 do for i = 0 , . . .
, 2 t − 1 do ▷ D
i,t ← { ( x j , y j ) | j ∈ [ n ] , x j ∈ P i,t } { Set of all input points whose x belongs to P i,t } { Notation: Define D ≤ α i,t := { ( x, y ) ∈
D i,t | x ≤ α } and D >αi,t similarly } { Notation: Define P ≤ α i,t := { x ∈ P i,t | x ≤ α } and P >αi,t similarly } ▷ Choose threshold α i,t ∈ { 0 } ∪ P i,t , using ε ′ -DP exponential mechanism with scoring function score i,t ( α ) :=
[Equation]
{ Note: score i,t ( α ) has sensitivity at most L/ 2 t .
▷ P 2 i,t +1 ← P ≤ α i,t i,t and P 2 i +1 ,t +1 ← P >α
i,t i,t .
Let f : [ m ] → [0 , 1] be given as f ( x ) =
i/ 2 T − 1 for all x ∈ P i,T − 1 and all i ∈ [2 T ] .
return f
[Equation]
Finally
[Equation]
notice that
[Equation]
With all the ingredients ready
[Equation]
we may now bound the expected (unnormalized) excess risk:
[Equation]
Taking the expectation on both sides and using (1) and (2) yields
[Equation]
Dividing both sides by n yields the desired claim.
Running Time.
To obtain a bound on the running time for general loss functions, we need to make a slight modification to the algorithm: we will additionally only restrict the range of f 1 , f 2 to multiples of 1 / 2 T − 1 .
We remark that this does not affect the utility since anyway we always take the final output whose values are multiples of 1 / 2 T − 1 .
Given any dataset D = { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } where x 1 < · · · < x n , the prefix isotonic regression algorithm is to compute, for each i ∈ [ n ] , the optimal loss in isotonic regression on ( x 1 , y 1 ) , . . .
, ( x i , y i ) .
Straightforward dynamic programming solves this in O ( n · v ) time, where v denote the number of possible values allowed in the function.
Now, for each i, t , we may run the above algorithm with D = D i,t and the allowed values are all multiples of 1 / 2 T − 1 in [ i 2 t , i +0 .
5 2 t ] ; this gives us min f 1 ∈F ( P ≤ α i,t , [ i 2 t , i +0 .
5 2 t ])
L abs[ i 2 t , i +12 t ] ( f 1 ; D ≤ α i,t ) for all α ∈ P i,t in time O ( | D i,t | · 2 T − t + | P i,t | ) .
Analogously, we can also compute min f 2 ∈F ( P >αi,t , [ ( i +0 . 5) 2 t , ( i +1)2 t ])
L abs[ i 2 t , i +12 t ] ( f 2 ; D >αi,t ) for all α ∈ P i,t in a similar time.
Thus, we can compute (score i,t ( α ))
α ∈ P i,t in time O
( | D i,t | · 2 T − t + | P i,t | ) , and then sample accordingly.
We can further speed up the algorithm by observing that the score remains constant for all α ∈ [ x i , x i +1 ) .
Hence, we may first sample an interval among [0 , x 1 ) , [ x 1 , x 2 ) , . . .
, [ x n − 1 , x n ) , [ x n , m ) and then sample α i,t uniformly from that interval.
This entire process can be done in O ( | D i,t | · 2 T − t + log m ) time.
In total, the running time of the algorithm is thus
T − 1 (cid:88) t = 2 t − 1 (cid:88) i =
O ( | D i,t | · 2 T − t + log m ) ≤
T − 1 (cid:88) t =
O ( n 2 T + 2 t · log m ) ≤ O ( n 2 log n + n log m ) .
Near-Linear Time Algorithms for ℓ 1 -, ℓ 22 -Losses.
We now describe faster algorithms for the ℓ 1and ℓ 22 -loss functions, thereby proving the last part of Theorem 3.
The key observation is that for convex loss functions, the restricted optimal is simple: we just have to “clip” the optimal function to be in the range [ τ, θ ] .
Below clip [ τ,θ ] denotes the function y
(cid:55)→ min { θ, max { τ, y }} .
Observation 12.
Let ℓ be any convex loss function, D any dataset, f ∗ ∈ argmin f ∈F ( X , Y ) L ( f ; D ) and τ ≤ θ any real numbers such that τ, θ ∈ Y .
Define f ∗ clipped ( x ) :
=
clip [ τ,θ ] ( f ∗ ( x )) .
Then, we must have f ∗ clipped ( x ) ∈ argmin f ∈F ( X , Y∩ [ τ,θ ]) L ( f ; D ) .
Proof.
Consider any f ∈ F ( X , Y ∩ [ τ, θ ]) .
Let X > (resp.
X < ) denote the set of all x ∈ X such that f ∗ ( x ) > θ (resp.
f ∗ ( x ) < τ ).
Consider the following operations:
• For each x ∈ X > , change f ( x ) to θ . • For each x ∈ X < , change f ( x ) to τ .
• Let f ( x ) =
f ∗ ( x )
for all x ∈ X \ ( X > ∪ X < ) .
At the end, we end up with f ( x ) =
f ∗ clipped ( x ) .
Each of the first two changes does not increase the loss L ( f ; D ) ; otherwise, due to convexity, changing f ∗ ( x ) to f ( x ) would have decrease the objective function.
Finally, the last operation does not decrease the loss; otherwise, we may replace this section of f ∗ with the values in f instead.
Thus, we can conclude that f ∗ clipped ( x ) ∈ argmin g ∈F ( X , Y∩ [ τ,δ ])
L ( f ; D ) .
We will now show how to compute the scores in Algorithm 1 simultaneously for all α (for fixed i, t ) in nearly linear time.
To do this, recall the prefix isotonic regression problem from earlier.
For this problem, Stout [42] gave an O ( n ) -time algorithm for ℓ 2 -loss and an O ( n log n ) -time algorithm for ℓ 1 -loss (both the unrestricted value case).
Furthermore, after the i th iteration, the algorithm also keeps a succinct representation S opt i of the optimal solution in the form of an array ( i 1 , v 1 , ℓ 1 ) , . . .
, ( i k , v k , ℓ k ) , which denotes f ( x ) = v j for all x ∈ [ x i j , x i j +1 ) , and ℓ j indicates the loss L abs up until x i j +1 , not including.
We can extend the above algorithm to prefix clipped isotonic regression problem, which we define in the same manner as above except that we restrict the function range to be [ τ, θ ] for some given τ < θ .
Using Observation 12, it is not hard to extend the above algorithm to work in this case.
Lemma 13.
There is an O ( n log n ) -time algorithm for ℓ 22 - and ℓ 1 - prefix clipped isotonic regression .
Proof.
We first precompute c τ ( i ) =
(cid:80) j ≤ i ℓ ( τ, x j ) and c θ ( i ) =
(cid:80) j ≤ i ℓ ( θ, x j ) for all i ∈ [ n ] .
We then run the aforementioned algorithm from [42].
At each iteration i , use binary search to find the largest index j τ such that v j τ < τ and the largest index j θ such that v j θ < θ .
Observation 12 implies that the optimal solution of the clipped version is simply the same as the unrestricted version except that we need to change the function values before x j τ to τ and after x j θ to θ .
The loss of this clipped optimal can be written as ℓ j θ − ℓ
j τ + c τ ( j τ )+( c θ ( i ) − c θ ( j θ )) , which can be computed in O (1) time given that we have already precomputed c τ , c θ .
The running time of the entire algorithm is thus the same as that of [42] together with the binary search time; the latter totals to O ( n log n ) .
Our fast algorithm for computing (score i,t ( α ))
α ∈ P i,t first runs the above algorithm with τ = i 2 t , θ = i +0 .
5 2 t and D
= D i,t ; this gives us min f 1 ∈F
[Equation]
P
[Equation]
) for all α ∈ P i,t in time O ( | D i,t | log | D i,t | + | P i,t | ) .
Analogously, we can also compute min f 2 ∈F ( P >αi,t , [ ( i +0 . 5) 2 t , ( i +1)2 t ])
L abs[ i 2 t , i +12 t ] ( f 2 ; D >αi,t ) for all α ∈ P i,t in a similar time.
Thus, we can compute (score i,t ( α ))
α ∈ P i,t in time O
( | D i,t | log | D i,t | + | P i,t | ) , and sample accordingly.
Using the same observation as the general loss function case, this can be sped up further to
O ( | D i,t | log | D i,t | + log m ) time.
In total, the running time of the algorithm is thus
[Equation] log [Equation] ( n
[Equation]
log
[Equation]
n
[Equation]
log m )) .
3.2 A Nearly Matching Lower Bound
We show that the excess empirical risk guarantee in Theorem 3 is tight, even for approximate-DP algorithms with a sufficiently small δ , under a mild assumption about the loss function stated below.
Definition 14 (Distance-Based Loss Function) .
For R ≥ 0 , a loss function ℓ is said to be R -distancebased if there exist g : [0 , 1] → R + such that ℓ ( y, y ′ ) =
g ( | y − y ′
| ) where g is a non-decreasing function with g (0)
=
and g (1 / 2) ≥ R .
We remark that standard loss functions, including ℓ 1 - or ℓ 22 -loss, are all Ω(1) -distance-based.
Our lower bound is stated below.
It is proved via a packing argument [25] in a similar manner as a lower bound for properly PAC learning threshold functions [10].
This is not a coincidence: indeed, when we restrict the range of our function to { 0 , 1 } , the problem becomes exactly (the empirical version of) properly learning threshold functions.
As a result, the same technique can be used to prove a lower bound in our setting as well.
Theorem 15.
For all 0 ≤ δ < 0 .
1 · ( e ε − 1) /m , any ( ε, δ ) -DP algorithm for isotonic regression over [ m ] for any R -distance-based loss function ℓ must have expected excess empirical risk Ω (cid:16) R · min (cid:110) 1 , log m εn (cid:111)(cid:17) .
Proof.
Suppose for the sake of contradiction that there exists an ( ε, δ ) -DP algorithm M for isotonic regression with an R -distance-based loss function ℓ with expected excess empirical risk 0 .
01 · (cid:16) R · min (cid:110) 1 , log(0 .
1 m ) εn (cid:111)(cid:17) .
Let k :=
1 log(0 .
1 m ) /ε ⌋ .
We may assume that n ≥ 2 k , as the Ω( R ) lower bound for the case n = 2 k can easily be adapted for an Ω( R ) lower bound for the case n < 2 k as well.
We will use the standard packing argument [25].
For each j ∈ [ m − 1] , we create a dataset D j that contains k copies of ( j, 0) , k copies of ( j + 1 , 1) and n − 2 k copies of (1 , 0) .
Finally, let D m denote the dataset that contains k copies of ( m, 0) and n − k copies of (1 , 0) .
Let V j denote the set of all f ∈ F ([ m ] , [0 , 1]) such that L ( f ; D ) <
Rk/n .
The utility guarantee of M implies that
Pr[ M ( D j ) ∈
V j ] ≥ 0 .
Furthermore, it is not hard to see that V 1 , . .
, V m are disjoint.
In particular, for any function f ∈
F ([ m ] , [0 , 1]) , let x f be the largest element x ∈ [ m ] for which f ( x ) ≤ 1 / 2 ; if no such x exists (i.e., f (0) > 1 / 2 ), let x f = 0 .
For any j < x f , we have L ( f ; D j ) ≥ kn ℓ ( f ( j + 1) , 1) ≥ kn · g (1 / 2) ≥
Rk/n .
Similarly, for any j > x f , we have L ( f ; D j ) ≥ kn ℓ ( f ( j ) , 0) kn · g (1 / 2) ≥ Rk/n This implies that f can only belong to V j , as claimed.
Therefore, we have that
[Equation]
a contradiction.
3.3 Extensions
We now discuss several variants of the isotonic regression problem that places certain additional constraints on the function f that we seek, as listed below.
• k -Piecewise
Constant: f must be a step function that consists of at most k pieces.
• k -Piecewise
Linear: f must be a piecewise linear function with at most k pieces.
• Lipschitz Regression: f must be L f -Lipschitz for some specified L f > 0 .
• Convex/Concave: f must be convex/concave.
We devise a general meta algorithm that, with a small tweak in each case, works for all of these constraints to yield Theorem 4.
At a high-level, our algorithm is similar to Algorithm 1, except that, in addition to using exponential mechanism to pick the threshold α i,t , we also pick certain auxiliary information that is then passed onto the next stage.
For example, in the k -piecewise constant setting, the algorithm in fact picks also the number of pieces to the left of α i,t and that to the right of it.
These are then passed on to the next stage.
The algorithm stops when the number of pieces become one, and then simply use the exponential mechanism to find the constant value on this subdomain.
The full description of the algorithm and the corresponding proof are deferred to Appendix C.
DP Isotonic Regression over General Posets
We now provide an algorithm and lower bounds for the case of general discrete posets.
We first recall basic quantities about posets.
An anti-chain of a poset ( X , ≤ ) is a set of elements such that no two distinct elements are comparable, whereas a chain is a set of elements such that every pair of elements is comparable.
The width of a poset ( X , ≤ ) , denoted by width( X ) , is defined as the maximum size among all anti-chains in the poset.
The height of ( X , ≤ ) , denoted by height( X ) , is defined as the maximum size among all chains in the poset.
Dilworth’s theorem and Mirsky’s theorem give the following relation between chains an anti-chains:
Lemma 16 (Dilworth’s and Mirsky’s theorems [12, 36]) .
A poset with width w can be partitioned into w chains.
A poset with height h can be partitioned into h anti-chains.
4.1 An Algorithm
Our algorithm for general posets is similar to that of totally ordered set presented in the previous section.
The only difference is that, instead of attempting to pick a single maximal point α such that f ( α ) ≤ τ as in the previous case, there could now be many such maximal α ’s.
Indeed, we need to use the exponential mechanism to pick all such α ’s. Since these are all maximal, they must be incomparable; therefore, they form an anti-chain.
Since there can be as many as |X| width( X ) anti-chains in total, this means that the error from the exponential mechanism is O (cid:0) log |X| width( X ) /ε
′ (cid:1)
=
O (width( X ) log |X| /ε
′ ) , leading to the multiplicative increase of width( X ) in the total error.
This completes our proof sketch for Theorem 1.
4.2 Lower Bounds
To prove a lower bound of Ω(width( X ) /εn ) , we observe that the values of the function in any antichain can be arbitrary.
Therefore, we may use each element in a maximum anti-chain to encode X as a binary vector.
The lower bound from Lemma 8 then gives us an Ω(width( X ) /n )
lower bound for ε = 1 , as formalized below.
Lemma 17.
For any δ > 0 , any (1 , δ ) -DP algorithm for isotonic regression for any R -distancebased loss function ℓ must have expected excess empirical risk Ω (cid:16) R (1 − δ ) · min (cid:110) 1 , width( X ) n (cid:111)(cid:17) .
Proof.
Consider any (1 , δ ) -DP isotonic regression algorithm M ′ for loss ℓ .
Let A be any maximum anti-chain (of size width( A ) ) in X .
We use this algorithm to build a (1 , δ ) -DP algorithm M for privatizing a binary vector of m = min { n, | A | − 1 } dimensions as follows:
• Let x 0 , x 1 , . . .
, x m be distinct elements of A .
• On input z ∈ { 0 , 1 } m , create a dataset D
= { ( x 1 , z 1 ) , . . .
, ( x m , z m ) , ( x 0 , 0) , . . .
, ( x 0 , 0) } where ( x 0 , 0) is repeated n − m times.
• Run M ′ on the instance D to get f , and output a vector z ′ where z ′ i =
[ f ( x i ) ≥ 1 / 2] .
It is obvious that this algorithm is (1 , δ ) -DP.
Observe also that L ( f ∗ ; D ) = 0 and thus M ′ ’s expected excess empirical risk is E f ∼M
( D )
[ L ( f ; D )] ≥ R · E z ′ ∼M ( z ) [ ∥ z − z ′ ∥ 0 ] /n , which, from Lemma 8, must be at least Ω( Rm (1 − δ ) /n )
=
Ω (cid:16) R (1 − δ ) · min (cid:110) 1 , width( X ) n (cid:111)(cid:17) .
By using group privacy (Fact 10) and repeating each element Θ(1 /ε ) times, we arrive at a lower bound of Ω (cid:16) R · min (cid:110) 1 , width( X ) εn (cid:111)(cid:17) .
Furthermore, since X contains height( X ) elements that form a totally ordered set, Theorem 15 gives a lower bound of Ω( R · log(height( X ))
/εn )
as long as δ < 0 .
01 · ε/ height( X ) .
Finally, due to Lemma 16, we have height( X ) ≥ |X| / width( X ) , which means that max { width( X ) , log(height( X )) } ≥ Ω(log |X| ) .
Thus, we arrive at:
Theorem 18.
For any ε ∈ (0 , 1] and any δ < 0 .
01 · ε/ |X| , any ( ε, δ ) -DP algorithm for isotonic regression for R -distance-based loss function ℓ must have expected excess empirical risk Ω (cid:16) R · min (cid:110) 1 , width( X )+log |X| εn (cid:111)(cid:17) .
4.3 Tight Examples for Upper and Lower Bounds
Recall that our upper bound is ˜ O (cid:16) width( X ) · log |X| εn (cid:17) while our lower bound is Ω (cid:16) width( X )+log |X| εn (cid:17) .
One might wonder whether this gap can be closed.
Below we show that, unfortunately, this is impossible in general: there are posets for which each bound is tight.
Tight Lower Bound Example.
Let X disj( w,h ) denote the poset that consists of w disjoint chains,
C 1 , . .
C w where | C 1 | = h and | C 2 | = · · · = | C w | = 1 .
(Every pair of elements on different chains are incomparable.)
In this case, we can solve the isotonic regression problem directly on each chain and piece the solutions together into the final output f .
Note that |X disj( w,h ) | = w + h − 1 and width( X ) = w, height( X )
= h .
According to Theorem 1, the unnormalized excess empirical risk in C i is ˜ O
(log( | C i | ) /ε ) .
Therefore, the total (normalized) empirical risk for the entire domain X is ˜ O (cid:16) log h
+( w − 1) εn (cid:17) .
This is at most ˜ O (cid:0) wεn (cid:1) as long as h ≤ exp( O ( w )) ; this matches the lower bound.
Tight Upper Bound Example.
Consider the grid poset X grid( w,h ) :
=
[ w ] ×
[ h ] where ( x, y ) ≤ ( x ′ , y ′ )
if and only if x ≤ x ′ and y ≤ y ′ .
We assume throughout that w ≤ h .
Observe that width( X grid( w,h ) )
=
w and height( X grid( w,h ) )
=
w + h .
We will show the following lower bound, which matches the ˜ O (cid:16) width( X )log |X| εn (cid:17) upper bound in the case where h ≥ w 1+Ω(1) , up to O (log 2 ( εn )) factor.
We prove it by a reduction from Lemma 9.
Note that this reduction is in some sense a “combination” of the proofs of Theorem 15 and Lemma 17, as the coordinate-wise encoding aspect of Lemma 17 is still present (across the rows) whereas the packing-style lower bound is present in how we embed elements of [ D ] (in blocks of columns).
Lemma 19.
For any ε ∈ (0 , 1] and δ < O ε (1 /h ) , any ( ε, δ ) -DP algorithm for isotonic regression for any R -distance-based loss function ℓ must have expected excess empirical risk Ω (cid:16) R · min (cid:110) 1 , w · log( h/w ) εn (cid:111)(cid:17) .
Proof.
Let D :=
⌊ h/w − 1 ⌋ , m = w and r :=
min {⌊ 0 .
5 n/m ⌋ , ⌊ 0 .
5 ln( D/ 2) /ε ⌋} .
Consider any ( ε, δ ) -DP algorithm M ′ for isotonic regression for ℓ on X grid( w,h ) where δ ≤ 0 .
01 ε/D .
We use this algorithm to build a (ln( D/ 2) , 0 . 25) -DP algorithm M for privatizing a vector z ∈
[ D ] m as follows:
• Create a dataset D that contains : – For all i ∈ [ m ] , r copies of (( i, ( w − i )( D + 1) + z i ) , 0) and r copies of (( i, ( w − i )( D + 1) + z i + 1) , 1) .
– n − 2 rm copies of ((1 , 1) , 0) .
• Run M ′ on instance D to get
f . •
Output a vector z ′ where z ′ i
=
max { j ∈
[ D ] | f
(( i, ( w − i )
( D + 1) + j )) ≤ 1 / 2 } .
(For simplicity, when such j does not exist let z ′ i = 0 .)
By group privacy, M is (ln( D/ 2) , 0 . 25) -DP.
Furthermore, L ( f ∗ ; D ) = 0 and the expected empirical excess risk of M ′ is
[Equation]
which must be at least Ω( Rrm/n )
=
Ω (cid:16) R · min (cid:110) 1 , w · log( h/w ) εn (cid:111)(cid:17) by Lemma 9.
Additional Related Work
(Non-private) isotonic regression is well-studied in statistics and machine learning.
The onedimensional (aka univariate) case has long history [9, 46, 5, 44, 45, 13, 8, 35, 14, 15, 49]; for a general introduction, see [22].
Moreover, isotonic regression has been studied in higher dimensions [24, 27, 26], including the sparse setting [21], as well as in online learning [29].
A related line of work studies learning neural networks under (partial) monotonicity constraints [3, 50, 30, 40].
There has been a rich body of work on DP machine learning, including DP empirical risk minimization (ERM), e.g., [11, 6, 48, 47], and DP linear regression, e.g., [1]; however, to the best of our knowledge none of these can be applied to isotonic regression to obtain non-trivial guarantees.
Another line of work related to our setting is around privately learning threshold functions [7, 20, 10, 2, 28].
We leveraged this relation to prove our lower bound for totally ordered case (Section 3.2).
Conclusions
In this paper we obtained new private algorithms for isotonic regression on posets and proved nearly matching lower bounds in terms of the expected empirical excess risk.
Although our algorithms for totally ordered sets are efficient, our algorithm for general posets is not.
Specifically, a trivial implementation of the algorithm would run in time exp( ˜ O (width( X ))) .
It remains an interesting open question whether this can be sped up.
To the best of our knowledge, this question does not seem to be well understood even for the non-private setting, as previous algorithmic works have focused primarily on the totally ordered case.
Similarly, while our algorithm is efficient for the totally ordered sets, it remains interesting to understand whether nearly linear time algorithms for ℓ 1 - and ℓ 22 -losses can be extended to a larger class of loss functions.
[-+ Dδ ′ (cid:17) = (cid:16) 
-]
[-P 2 i,t +1 , P 2 i +1 ,t +1 , we must have x ′ ∈ P 2 i,t +1 and x ∈ P 
-]