R EGRESSION WITH L ABEL D IFFERENTIAL P RIVACY
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the task of training regression models with the guarantee of label differential privacy (DP).
Based on a global prior distribution on label values, which could be obtained privately, we derive a label DP randomization mechanism that is optimal under a given regression loss function.
We prove that the optimal mechanism takes the form of a “randomized response on bins”, and propose an efficient algorithm for finding the optimal bin values.
We carry out a thorough experimental evaluation on several datasets demonstrating the efficacy of our algorithm.
I NTRODUCTION
In recent years, differential privacy (DP, Dwork et al., 2006a;b) has emerged as a popular notion of user privacy in machine learning (ML).
On a high level, it guarantees that the output model weights remain statistically indistinguishable when any single training example is arbitrarily modified.
Numerous DP training algorithms have been proposed, with open-source libraries tightly integrated in popular ML frameworks such as TensorFlow Privacy (Radebaugh & Erlingsson, 2019) and PyTorch
Opacus (Yousefpour et al., 2021).
In the context of supervised ML, a training example consists of input features and a target label.
While many existing research works focus on protecting both features and labels (e.g., Abadi et al.
, there are also some important scenarios where the input features are already known to the adversary, and thus protecting the privacy of the features is not needed.
A canonical example arises from computational advertising where the features are known to one website (a publisher), whereas the conversion events, i.e., the labels, are known to another website (the advertiser).
Thus, from the first website’s perspective, only the labels can be treated as unknown and private.
This motivates the study of label DP algorithms, where the statistical indistinguishability is required only when the label of a single example is modified.
The study of this model goes back at least to the work of
Chaudhuri & Hsu (2011).
Recently, several works including (Ghazi et al., 2021a; Malek Esmaeili et al., 2021) studied label DP deep learning algorithms for classification objectives.
Our Contributions.
In this work, we study label DP for regression tasks.
We provide a new algorithm that, given a global prior distribution (which, if unknown, could be estimated privately), derives a label DP mechanism that is optimal under a given objective loss function.
We provide an explicit characterization of the optimal mechanism for a broad family of objective functions including the most commonly used regression losses such as the Poisson log loss, the mean square error, and the mean absolute error.
More specifically, we show that the optimal mechanism belongs to a class of randomized response on bins (Algorithm 1).
We show this by writing the optimization problem as a linear program (LP), and characterizing its optimum.
With this characterization in mind, it suffices for us to compute the optimal mechanism among the class of randomized response on bins.
We then provide an efficient algorithm for this task, based on dynamic programming (Algorithm 2).
In practice, a prior distribution on the labels is not always available.
This leads to our two-step algorithm (Algorithm 3) where we first use a portion of the privacy budget to build an approximate histogram of the labels, and then feed this approximate histogram as a prior into the optimization
[Figure]
algorithm in the second step; this step would use the remaining privacy budget.
We show that as the number of samples grows, this two-step algorithm yields an expected loss (between the privatized label and the raw label) which is arbitrarily close to the expected loss of the optimal local DP mechanism.
(We give a quantitative bound on the convergence rate.)
Our two-step algorithm can be naturally deployed in the two-party learning setting where each example is vertically partitioned with one party holding the features and the other party holding the (sensitive) labels.
The algorithm is in fact one-way, requiring a single message to be communicated from the labels party to the features party, and we require that this one-way communication satisfies (label) DP.
We refer to this setting, which is depicted in Figure 1, as feature-oblivious label DP .
We evaluate our algorithm on three datasets: the 1940 US Census IPUMS dataset, the Criteo Sponsored Search Conversion dataset, and a proprietary app install ads dataset from a commercial mobile app store.
We compare our algorithm to several baselines, and demonstrate that it achieves higher utility across all test privacy budgets, with significantly lower test errors for the high privacy regime.
For example, for privacy budget ε = 0 . 5 , comparing to the best baseline methods, the test MSE for our algorithm is ∼ 1 .
5 × smaller on the Criteo and US Census datasets, and the relative test error is ∼ 5 × smaller on the App Ads dataset.
Organization.
In Section 2, we recall some basics of DP and learning theory, and define the feature-oblivious label DP setting in which our algorithm can be implemented.
Our label DP algorithm for regression objectives is presented in Section 3.
Our experimental evaluation and results are described in Section 4.
A brief overview of related work appears in Section 5.
We conclude with some interesting future directions in Section 6.
Most proofs are deferred to the Appendix (along with additional experimental details and background material).
P RELIMINARIES
We consider the standard setting of supervised learning, where we have a set of examples of the form ( x, y ) ∈
X × Y , drawn from some unknown distribution D
and we wish to learn a predictor f θ (parameterized by θ ) to minimize L ( f θ ) :
=
E ( x,y ) ∼D ℓ ( f θ ( x ) , y ) , for some loss function ℓ : R × Y → R ≥ 0 ; we will consider the case where Y ⊆ R .
Some common loss functions include the zero-one loss ℓ 0-1 (˜ y, y ) :
=
[˜ y ̸
= y ] , the logistic loss ℓ log (˜ y, y ) :
= log(1 + e − ˜ yy ) for binary classification, and the squared loss ℓ sq (˜ y, y ) :
=
(˜ y − y ) 2 , absolute-value loss ℓ abs (˜ y, y ) :
= | ˜
y − y | ,
Poisson log loss ℓ Poi (˜ y, y ) :
=
˜
y − y · log(˜ y ) for regression.
This paper focuses on the regression setting.
However, we wish to perform this learning with differential privacy (DP).
We start by recalling the definition of DP, which can be applied to any notion of adjacent pairs of datasets.
For an overview of DP, we refer the reader to the book of Dwork & Roth (2014).
Definition 1 (DP; Dwork et al. (2006b)) .
Let ε be a positive real number.
A randomized algorithm A taking as input a dataset is said to be ε -differentially private (denoted ε -DP ) if for any two adjacent datasets X and X ′ , and any subset S of outputs of A , we have Pr[ A ( X ) ∈ S ] ≤ e ε · Pr[ A ( X ′ ) ∈ S ] .
In supervised learning applications, the input to a DP algorithm is the training dataset (i.e., a set of labeled examples) and the output is the description of a predictor (e.g., the weights of the trained model).
Typically, two datasets are considered adjacent if they differ on a single training example; this notion protects both the features and the label of any single example.
As discussed in Section 1, in certain situations, protecting the features is either unnecessary or impossible, which motivates the study of label DP that we define next.
Definition 2 (Label DP; Chaudhuri & Hsu (2011)) .
An algorithm A taking as input a training dataset is said to be ε -label differentially private (denoted as ε - LabelDP ) if it is ε -DP when two input datasets are considered adjacent if they differ on the label of a single training example.
We next define an important special case of training with label DP, which commonly arises in practice.
The setting could be viewed as a special case of vertical federated learning , where each training example is divided across different parties at the beginning of the training process; see, e.g., the survey of Kairouz et al.
(2021) and the references therein.
We consider the special case where we only allow a single round of communication from the first party who holds all the labels, to the second party, who holds all the features, and who then trains the ML model and outputs it.
We refer to this setting (depicted in Figure 1) as “feature-oblivious label DP” since the label DP property is guaranteed without having to look at the features (but with the ability to look at the labels from all users); we next give the formal definition.
Definition 3 (Feature-Oblivious Label DP) .
Consider the 2 -party setting where the “features party” holds as input a sequence ( x i ) ni
=1 of all feature vectors (across all the n users), and the “labels party” holds as input a sequence ( y i ) ni
=1 of the corresponding labels.
The labels party sends a single message M ( y 1 , . .
y n ) to the features party; this message can be randomized using the internal randomness of the labels party.
Based on its input and on this incoming message, the features party then trains an ML model that it outputs.
We say that this output is feature-oblivious ε - LabelDP if the message M ( y 1 , . . .
, y n ) is ε -DP with respect to the adjacency relation where a single y i can differ across the two datasets.
We stress the practical applicability of the setting described in Definition 3.
The labels party could be an advertiser who observes the purchase data of the users, and wants to enable a publisher (the features party) to train a model predicting the likelihood of a purchase (on the advertiser) being driven by the showing of a certain type of ad (on the publisher).
The advertiser would also want to limit the leakage of sensitive information to the publisher.
From a practical standpoint, the simplest option for the advertiser is to send a single privatized message to the publisher who can then train a model using the features at its disposal.
This is exactly the feature-oblivious label DP setting.
L ABEL DP L EARNING A LGORITHM
A common template for learning with label DP is to: (i) compute noisy labels using a local DP mechanism M , (ii) use a learning algorithm on the dataset with noisy labels.
All of the baseline algorithms that we consider follow this template, through different ways of generating noisy labels, such as, (a) randomized response (for categorical labels), (b) (continuous/discrete) Laplace mechanism, (c) (continuous/discrete) staircase mechanism, etc.
(for the formal definitions of these mechanisms, see Appendix D).
Intuitively, such a learning algorithm will be most effective when the noisy label mostly agrees with the true label.
Suppose the loss function ℓ satisfies the triangle inequality, namely that ℓ (˜ y, y ) ≤ ℓ (ˆ y, y ) + ℓ (˜ y, ˆ y ) for all y, ˆ y, ˜ y .
Then we have that
[Equation]
where, for ease of notation, we use P to denote the marginal on y for ( x, y ) ∼ D .
The learning algorithm, in part (ii) of the template above, aims to minimize the second term in the RHS of (1).
Thus, it is natural to choose a mechanism M , in part (i) of the template, to minimize the first term in the RHS of (1).
Ghazi et al.
(2021a) studied this question for the case of 0-1 loss and showed that a randomized response on top k labels with highest prior masses (for some k ) is optimal.
Their characterization was limited to this classification loss.
In this work, we develop a characterization for a large class of regression loss functions.
3. R ANDOMIZED R ESPONSE ON B INS : A N O PTIMAL M ECHANISM
We propose a new mechanism for generating noisy labels that minimizes the first term in the RHS of (1).
Namely, we define randomized response on bins ( RR - on - Bins Φ ε ), which is a randomized algorithm parameterized by a scalar ε > 0 and a non-decreasing function Φ :
Y → ˆ Y that maps
the
Parameters: Φ : Y → ˆ Y (for label set
Y and output set ˆ Y ), privacy parameter ε ≥ 0 .
Input: A label y ∈ Y .
Output: ˆ y ∈
ˆ Y .
return a sample ˆ y ∼ ˆ Y , where the random variable ˆ Y is distributed as
Pr[ ˆ Y =
ˆ y ]
= (cid:40)
e ε e ε + | ˆ
Y|−if ˆ y =
Φ( y ) 1 e ε + | ˆ Y|−otherwise, for each ˆ y ∈
ˆ Y .
Input: Distribution P over Y ⊆ R , privacy param.
ε ≥ 0 , loss function ℓ : R 2 → R ≥ 0 .
Output: Output set ˆ Y ⊆ R and Φ :
Y → ˆ Y .
y 1 , . .
, y k ← elements of Y in increasing order
Initialize A [ i ][ j ] ← ∞ for all i, j ∈ { 0 , . . .
, k } .
for r, i ∈
[ k ] do
L [ r ][ i ] ← min ˆ y ∈ R (cid:80) y ∈Y p y · e 1 [ y ∈
[ y r ,y i ]] · ε · ℓ (ˆ y, y )
A [0][0] ←for i ∈ [ k ] do for j ∈ [ i ] do
A [ i ][ j ] ← min 0 ≤ r<i
A [ r ][ j − 1] + L [ r + 1][ i ] return Φ , ˆ Y correspond.
to min d ∈ [ k ] 1 d − 1+ e ε
A
[ k ][ d ] . label set Y ⊆ R to an output set ˆ Y ⊆ R .
This algorithm is simple: perform ε -randomized response on Φ( y ) , randomizing over ˆ Y ; see Algorithm 1 for a formal definition.
Any Φ we consider will be non-decreasing unless otherwise stated, and so we often omit mentioning it explicitly.
An important parameter in RR - on - Bins is the mapping Φ and the output set ˆ Y .
We choose Φ with the goal of minimizing E ℓ (ˆ y, y ) .
First we show, under some basic assumptions about ℓ , that for any given distribution P over labels Y , there exists a non-decreasing map Φ such that
M = RR - on - Bins Φ ε minimizes E y ∼ P, ˆ y ∼M ( y ) ℓ (ˆ y, y ) .
Since Φ is non-decreasing, it follows that Φ − 1 (ˆ y ) is an interval 3 of Y , for all ˆ y ∈
ˆ Y .
Exploiting this property, we give an efficient algorithm for computing the optimal Φ .
To state our results formally, we use L ( M ; P ) to denote E y ∼ P, ˆ y ∼M ( y ) ℓ (ˆ y, y ) , the first term in the
RHS of (1).
Our results hold under the following natural assumption; note that all the standard loss functions such as squared loss, absolute-value loss, and Poisson loss satisfy Assumption 4.
Assumption 4.
Loss function ℓ : R × R → R ≥ 0 is such that
• For all y ∈ R , ℓ ( · , y ) is continuous.
• For all y ∈ R , ℓ (ˆ y, y ) is decreasing in ˆ y when ˆ y ≤ y and increasing in ˆ y when ˆ y ≥
y .
• For all ˆ y ∈ R , ℓ (ˆ y, y ) is decreasing in y when y ≤ ˆ y and increasing in y when y ≥ ˆ y .
We can now state our main result:
Theorem 5.
For any loss function ℓ : R × R → R ≥ 0 satisfying Assumption 4, all finitely supported distributions P over Y ⊆ R , there is an output set ˆ Y ⊆ R and a non-decreasing map Φ : Y → ˆ Y such that
[Equation]
where the infimum is over all ε -DP mechanisms M .
Proof Sketch.
This proof is done in two stages.
First, we handle the case where ˆ Y is restricted to be a subset of O , where O is a finite subset of R .
Under this restriction, the optimal mechanism M that minimizes L ( M ; P ) can be computed as the solution to an LP.
Since an optimal solution to an
LP can always be found at the vertices of the constraint polytope, we study properties of the vertices of the said polytope and show that the minimizer amongst its vertices necessarily takes the form of
RR - on - Bins Φ ε .
To handle the case of general ˆ Y , we first note that due to Assumption 4, it suffices to consider ˆ Y ⊆
[ y min , y max ] (where y min =
min
y ∈Y y and y max = max y ∈Y y ).
Next, we consider a sequence of increasingly finer discretizations of [ y min , y max ] , and show that RR - on - Bins Φ ε can come arbitrarily close to the optimal L ( M ; P ) when restricting ˆ Y to be a subset of the discretized set.
But observe that any Φ induces a partition of Y into intervals.
Since there are only finitely many partitions of Y into intervals, it follows that in fact RR - on - Bins Φ ε can exactly achieve the optimal
L ( M ; P ) .
The full proof is deferred to Appendix A.
Given Theorem 5, it suffices to focus on RR - on - Bins Φ ε mechanisms and optimize over the choice of Φ .
We give an efficient dynamic programming-based algorithm for the latter in Algorithm 2.
The main idea is as follows.
We can breakdown the representation of Φ into two parts (i) a partition
P Φ = { S 1 , S 2 , . . .
} of Y into intervals 4 , such that Φ is constant over each S i , and (ii) the values
y i that Φ( · ) takes over interval S i .
Let y 1 , . . .
, y k be elements of Y in increasing order.
Our dynamic program has a state
A [ i ][ j ] , which is (proportional to) the optimal mechanism if we restrict the input to only { y 1 , . . .
, y i } and consider partitioning into j intervals S 1 , . . .
, S j .
To compute A [ i ][ j ] , we try all possibilities for S j .
Recall that S j must be an interval, i.e., S j = { y
r
+1 , . .
, y i } for some r < j .
This breaks the problem into two parts: computing optimal RR - on - Bins for { y 1 , . . .
, y r } for j − 1 partitions and solving for the optimal output label ˆ y for S j .
The answer to the first part is simply A
[ r ][ j − 1] , whereas the second part can be written as a (single-variate) optimization problem (denoted by L [ r ][ i ] in Algorithm 2).
When ℓ is convex (in the first argument), this optimization problem is convex and can thus be solved efficiently.
Furthermore, for squared loss, absolute-value loss, and Poisson loss, this problem can be solved in amortized O (1) time, resulting in a total running time of just O ( k 2 ) for the entire dynamic programming algorithm.
The full description is presented in Algorithm 2 5 ; the complete analysis of its correctness and running time is deferred to Appendix B.
3. E STIMATING THE P RIOR P RIVATELY
Algorithm
Labels
Party’s
Randomizer
LabelRandomizer ε 1 ,ε
Parameters: Privacy parameters ε 1 , ε 2 ≥ 0 .
Input: Labels y 1 , . .
. , y n ∈ Y .
Output: ˆ y 1 , . . .
, ˆ y n ∈
ˆ Y .
P ′ ← M Lap ε 1 ( y 1 , . . .
, y n ) Φ ′ ,
ˆ Y ′ ← Result of running Algorithm 2 with distribution P ′ and privacy parameter ε 2 .
for i ∈ [ n ] do ˆ y i ← RR - on - Bins Φ ′
ε 2 ( y i ) .
return (ˆ y 1 , . . .
, ˆ y n )
In the previous subsection, we assumed that the distribution P of labels is known beforehand.
This might not be the case in all practical scenarios.
Below we present an algorithm that first privately approximates the distribution P and work with this approximation instead.
We then analyze how using such an approximate prior affects the performance of the algorithm.
To formalize this, let us denote by M Lap ε the εDP Laplace mechanism for approximating the prior.
Given n samples drawn from P , M Lap ε constructs a histogram over Y and adds Laplace noise with scale 2 /ε to each entry, followed by clipping (to ensure that entries are nonnegative) and normalization.
A formal description of M Lap ε can be found in Algorithm 4 in the Appendix.
Our mechanism for the unknown prior case—described in Algorithm 3—is now simple: We split the privacy budget into ε 1 , ε 2 , run the ε 1 -DP Laplace mechanism to get an approximate prior distribution
P ′ and run the optimal RR - on - Bins for privacy parameter ε 2 to randomize the labels.
It is not hard to see that the algorithm is ( ε 1 + ε 2 ) -DP.
(Full proof deferred to the Appendix.)
Theorem 6.
LabelRandomizer ε 1 ,ε 2 is ( ε 1 + ε 2 ) -DP.
Let us now discuss the performance of the above algorithm in comparison to the case where the prior P is known, under the assumption that y 1 , . . .
, y n are sampled i.i.d.
from P .
We show in the following theorem that the difference between the expected population loss in the two cases converges to zero as the number of samples n → ∞ , provided that Y is finite:
[Figure]
Theorem 7.
Let ℓ : R × R → R ≥ 0 be any loss function satisfying Assumption 4.
Furthermore, assume that ℓ (ˆ y, y ) ≤ B for some parameter B for all y, ˆ y ∈ Y .
For any distribution P on Y , ε > 0 , and any sufficiently large n ∈ N , there is a choice of ε 1 , ε 2 > 0 such that ε 1 + ε 2 = ε
and
[Equation]
where P ′ , Φ ′ , ˆ Y ′ are as in LabelRandomizer
ε 1 ,ε 2 and the infimum is over all ε -DP mechanisms M .
We remark that the above bound is achieved by setting ε 1 = (cid:112) |Y| /n ; indeed, we need to assume that n is sufficiently large so that ε 1 < ε .
The high-level idea of the proof is to first use a known utility bound for Laplace mechanism Diakonikolas et al.
(2015) to show that P, P ′ are close.
Then, we argue that the optimal RR - on - Bins is robust, in the sense that small changes in the prior (from P to P ′ ) and privacy parameter (from ε to ε 2 ) do not affect the resulting population loss too much.
E XPERIMENTAL E VALUATION
We evaluate the RR - on - Bins mechanism on three datasets, and compare with the Laplace mechanism (Dwork et al., 2006b), the staircase mechanism (Geng & Viswanath, 2014) and the exponential mechanism (McSherry & Talwar, 2007).
All of these algorithms can be implemented in the featureoblivious label DP setting of Figure 1.
Detailed model and training configurations can be found in
Appendix E.
4. C RITEO S PONSORED S EARCH C ONVERSION
The Criteo Sponsored Search Conversion Log Dataset (Tallis & Yadav, 2018) contains logs obtained from Criteo Predictive Search (CPS).
Each data point describes an action performed by a user (click on a product related advertisement), with additional information consisting of a conversion (product was bought) within a 30 -day window and that could be attributed to the action.
We formulate a (feature-oblivious) label DP problem to predict the revenue (in C) obtained when a conversion takes place (the SalesAmountInEuro attribute).
This dataset represents a sample of 90 days of Criteo live traffic data, with a total of 15 , 995 , 634 examples.
We remove examples where no conversion happened ( SalesAmountInEuro is − 1 ), resulting in a dataset of 1 , 732 , 721 examples.
The conversion
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
value goes up to 62 , 458 .
773 C. We clip the conversion value at 400 C, which corresponds to the 95 th percentile of the value distribution.
In Figure 2, we visualize an example of how RR - on - Bins randomizes those labels, and compare it with the Laplace mechanism, which is a canonical DP mechanism for scalars.
In this case (and for ε = 3 ), RR - on - Bins chooses 3 bins at around 50 , 100 , and 250 and maps the sensitive labels to those bins with Algorithm 1.
The joint distribution of the sensitive labels and randomized labels maintains an overall concentration along the diagonal.
On the other hand, the joint distribution for the Laplace mechanism is generally spread out.
Table 1 quantitatively compares the two mechanisms across different privacy budgets.
The first block (Mechanism) shows the MSE between the sensitive training labels and the private labels generated by the two mechanisms, respectively.
We observe that RR - on - Bins leads to significantly smaller MSE than the Laplace mechanism for the same label
DP guarantee.
Furthermore, as shown in the second block (Generalization), the reduced noise in the training labels leads to lower test errors.
[Figure]
Figure 3 compares with two additional baselines: the exponential mechanism, and the staircase mechanism.
For both the “Mechanism” errors and “Generalization” errors, RR - on - Bins consistently outperforms other methods for both low- and high- ε regimes.
4. US C ENSUS
The 1940 US Census dataset has been made publicly available for research since 2012 , and has 131 , 903 , 909 rows.
This data is commonly used in the DP literature (e.g., Wang et al.
; Cao et al.
; Ghazi et al.
(2021b)).
In this paper, we set up a label DP problem by learning to predict the duration for which the respondent worked during the previous year (the WKSWORK1 field, measured in number of weeks).
Figure 4a shows that RR - on - Bins outperforms the baseline mechanisms across a wide range of privacy budgets.
4. A PP A DS C ONVERSION C OUNT P REDICTION
Finally, we evaluate our algorithm on an app install ads prediction dataset from a commercial mobile app store.
The examples in this dataset are ad clicks and each label counts post-click events (aka conversions) occurring in the app after the user installs it.
For example, if a user installs a ride share app after clicking the corresponding ad, the label could be the total number of rides that the user purchases in a given time window after the installation.
We note that ad prediction tasks/datasets of a similar nature have been previously used for evaluation in previous publications in the literature
Badanidiyuru et al.
Figure 4b shows the performance on this dataset.
Consistently with the results observed on the other datasets, RR - on - Bins outperforms other mechanisms across all the privacy budget values.
In summary, by first estimating a prior distribution (privately), RR - on - Bins significantly reduces label noise compared to other label DP mechanisms, especially in the high-privacy (i.e., intermediate to small ε ) regime.
This leads to significantly better test performance for the regression networks trained on these less noisy labels.
For example, for ε = 0 . 5 , comparing to the best baseline methods, the test MSE for RR - on - Bins is ∼ 1 .
5 × smaller on the Criteo and US Census datasets, and the relative test error is ∼ 5 × smaller on the App Ads dataset.
R ELATED W ORK
There has been a large body of work on learning with DP.
Various theoretical and practical settings have been studied including empirical risk minimization (Chaudhuri et al., 2011), optimization (Song et al., 2013), regression analysis (Zhang et al., 2012), and deep learning (Abadi et al., 2016).
The vast majority of the prior works studied the setting where both the features and the labels are deemed sensitive and hence ought to be protected.
Chaudhuri & Hsu (2011) and Beimel et al.
(2016) studied the sample complexity of learning with label DP for classification tasks.
Wang & Xu (2019) studied label DP for sparse linear regression in the local DP setting.
Ghazi et al.
(2021a) provided a procedure for training deep neural classifiers with label DP.
For the classification loss, they formulate a linear program and derive an explicit solution, which they refer to as RRTop- k .
Our work could be viewed as an analog of theirs for regression tasks.
Malek Esmaeili et al.
(2021) used the PATE framework of Papernot et al.
(2016; 2018) to propose a new label DP training method.
We note that this, as well as related work on using unsupervised and semi-supervised learning to improve label DP algorithms (Esfandiari et al., 2022;
Tang et al., 2022), do not apply to the feature-oblivious label DP setting.
A variant of two-party learning (with a “features party” and a “labels party”) was recently studied by Li et al.
(2021), who considered the interactive setting where the two parties can engage in a protocol with an arbitrary number of rounds (with the same application to computational advertising described in Section 1 as a motivation).
By contrast, we considered in this paper the one-way communication setting (from Figure 1), which is arguably more practical and easier to deploy.
Kairouz et al.
(2016) study optimal local DP algorithms under a given utility function and prior.
For a number of utility functions, they show that optimal mechanisms belong to a class of staircase mechanisms .
Staircase mechanisms are much more general than randomized response on bins. In that regard, our work shows that a particular subset of staircase mechanisms are optimal for regression tasks.
In particular, while we give an efficient dynamic programming algorithm for optimizing over randomized response on bins, we are not aware of any efficient algorithm that can compute an optimal staircase mechanism.
(In particular, a straightforward algorithm takes 2 O ( k ) time.)
C ONCLUSIONS AND F UTURE D IRECTIONS
In this work we propose a new label DP mechanism for regression.
The resulting training algorithm can be implemented in the feature-oblivious label DP setting.
We provide theoretical results shedding light on the operation and guarantees of the algorithm.
We also evaluate it on three datasets, demonstrating that it achieves higher accuracy compared to several non-trivial, natural approaches.
Our work raises many questions for future exploration.
First, in our evaluation, we set the objective function optimized using the LP to be the same as the loss function used during training; different ways of setting the LP objective in relation to the training loss are worth exploring.
Second, our noising mechanism typically introduces a bias; mitigating it, say by adding unbiasedness constraints into the LP, is an interesting direction.
Third, it might also be useful to investigate de-noising techniques that can be applied as a post-processing step on top of our label DP mechanism; e.g., the method proposed in Tang et al.
(2022) for computer vision tasks, and the ALIBI method (Malek Esmaeili et al., 2021) for classification tasks, which is based on Bayesian inference.
We believe that due to its simplicity and practical appeal, the setting of learning with featureoblivious label DP merits further study, from both a theoretical and an empirical standpoint.
Finally, we leave the question of obtaining better “feature-aware” label DP regression algorithms for a future investigation.
A O PTIMALITY OF RR - on - Bins Φ ε
In this section, we prove Theorem 5.
We prove this in two stages.
First we consider the case where
P has finite support and ˆ Y is constrained to be a subset of a pre-specified finite set O ⊆ R .
Next, we consider the case where ˆ Y is allowed to be an arbitrary subset of R , and thus, the output distribution of the mechanism on any input y can be an arbitrary probability measure over R .
Stage I: Finitely supported P , ˆ Y ⊆ O , ℓ strictly increasing/decreasing.
Theorem 8. Let P be a distribution over R with finite support, O a finite subset of R , and ℓ : R × R → R ≥ 0 is such that • For all y ∈ R , ℓ (ˆ y, y ) is strictly decreasing in ˆ y when ˆ y ≤ y and strictly increasing in ˆ y when ˆ y ≥
y . • For all ˆ y ∈ R , ℓ (ˆ y, y ) is strictly decreasing in y when y ≤ ˆ y and strictly increasing in y when y ≥ ˆ y . Then there exists Φ : Y → O such that [Equation]
The proof of Theorem 8 is inspired by the proof of Theorem 3.1 of Ghosh et al.
Namely, we describe the problem of minimizing L ( M ; P ) as a linear program (LP).
We arrange the variables of the linear program into a matrix and associate any solution to the LP a signature matrix, which represents when constraints of the LP are met with equality.
Then we make several observations of the signature matrix for any optimal solution, which eventually leads to the proof of the theorem.
Proof of Theorem 8. Without loss of generality, we may assume that Y = supp( P ) and therefore is finite. When ˆ Y is restricted to be a subset of O , the optimal mechanism M which minimizes L ( M , P ) =
E y ∼ P, ˆ y ∼M ( y ) ℓ (ˆ y, y ) , is encoded in the solution of the following linear program (LP), with |Y| · |O| variables M y → ˆ y =
Pr[ M ( y ) =
ˆ y ] indexed by ( y, ˆ y ) ∈
Y × O .
Namely, the first two constraints enforce that ( M y → ˆ y )
y is a valid probability distribution and the third constraint enforces the ε -DP constraint.
[Equation]
subject to ∀ y
[Equation]
O
[Equation]
Corresponding to any feasible solution M :
=
( M y → ˆ y )
y, ˆ y , we associate a |Y|×|O| signature matrix S M .
First, let p minˆ y =
min y M y → ˆ y and let p maxˆ y = max y M y
→ ˆ y .
Note that, from the constraints it follows that p maxˆ y ≤ e ε · p minˆ y .
Definition 9 (Signature matrix) .
For any feasible M for the LP in (2), the signature entry S M ( y, ˆ y ) for all y ∈ Y and ˆ y ∈ O is defined as
• 0 if M y → ˆ y = 0 , else • U if M y → ˆ y = p maxˆ y =
e ε · p minˆ y , else, • L if M y → ˆ y = p minˆ y
=
e − ε · p maxˆ y , else, • S otherwise.
We visualize S M as a matrix with rows corresponding to y ’s and columns corresponding to ˆ y ’s, both ordered in increasing order (see Figure 5). M 1 .2 .3 .4 . U S
L U S L
L
S
U
S
S
S
S
S
S S M If M is an RR - on - Bins Φ ε mechanism for some Φ : Y → O , then it is easy to see that the corresponding signature matrix satisfies some simple properties (given in Claim 10 below). Interestingly, we establish a converse, thereby characterizing the signature of matrices M that correspond to RR - on - Bins Φ ε mechanism for some Φ : Y → O .
Claim 10.
M corresponds to an RR - on - Bins Φ ε mechanism for some non-decreasing Φ :
Y → ˆ Y (for ˆ Y ⊆
O )
if and only if either
(1) One column consists entirely of S , while all other columns are entirely 0 ; let Ψ(
y )
= ˆ y , where ˆ y corresponds to the unique S column
, or all of the following hold:
(2a)
Each column in S M is either entirely 0 , or entirely consisting of U ’s and L ’s, with at least one U and one L .
(2b)
Each row contains a single U entry, with all other entries being either L or 0 ; for each y , we denote this unique column index of U by Ψ( y ) .
(2c)
For all y <
y ′
it holds that Ψ( y ) ≤ Ψ( y ′ ) .
In each case, it holds that Φ = Ψ .
Proof of Claim 10.
Suppose M corresponds to RR - on - Bins Φ ε for some non-decreasing Φ .
If Φ is constant, then condition (1) holds, since all columns corresponding to ˆ y / ∈ range(Φ) in S M are all 0 , whereas, the only remaining column consists of all S .
If Φ is not constant, then M y → ˆ y = e ε 1 [Φ( y )
=
ˆ y ] / ( e ε + | ˆ Y| − 1) , and hence its signature S M is such that S M ( y, ˆ y ) is U
if Φ( y ) = ˆ y , and
L if ˆ y ∈ range(Φ) ∖ { Φ( y ) } , and 0 otherwise.
It is easy to verify that all three conditions hold: (2a) a column corresponding to ˆ y is entirely
if and only if ˆ y / ∈ range (Φ) and
entirely consisting of
U ’s and L ’s otherwise with at least one U corresponding
to y such that Φ( y )
=
ˆ y and at least one L corresponding to y
such that Φ( y ) ̸
= ˆ y (since Φ is non-constant), (2b)
Each row corresponding to y has a unique U corresponding to ˆ y
= Ψ( y ) =
Φ( y ) , and (2c) For y < y ′ , it holds that Ψ( y ) ≤ Ψ( y ′ ) since Φ is non-decreasing.
To establish the converse, suppose we have that S M satisfies condition (1).
Then we immediately get that M corresponds to RR - on - Bins for the constant map Φ =
Ψ .
Next, suppose S M satisfies all three conditions (2a)–(2c).
Immediately, we have that M is given as
[Equation]
Let ˆ Y correspond to the set of non-zero columns of S M .
Since each row of M corresponds to a probability distribution, we have for each ˆ y ∈
ˆ Y that (cid:80) ˆ
y ′ ∈
ˆ Y p minˆ y ′
+( e ε − 1) p minˆ y = 1 by considering the row corresponding to any y ∈ Ψ − 1 (ˆ y ) .
Thus, we get that p min ˆ y is the same for all values in ˆ Y , and in particular, p minˆ y =
[Equation]
e ε + | ˆ Y|− 1 , and thus, we get that the corresponding M is uniquely given by
[Equation]
which clearly corresponds to RR - on - Bins Φ ε for Φ( y )
=
Ψ( y ) .
We have that Ψ , and hence Φ , is non-decreasing from condition (2c).
This completes the proof of Claim 10.
Thus, to show optimality of RR - on - Bins Φ ε , it suffices to show that there exists a minimizer M of the
LP (2), such that S M satisfies the conditions in Claim 10.
It is well known that an optimal solution to any LP can be found at the vertices of the constraint polytope and that for a LP with n variables, vertices must meet n linearly independent constraints. We use this to find a submatrix of S M which will determine M in its entirety.
If M is a feasible point, then each column of S M is either entirely 0 , or entirely consisting of S , U and L .
This is necessary since M y → ˆ y = 0 , then 0 ≤ M y ′
→ ˆ y ≤ e ε M y
→
ˆ y = 0 .
Let k denote the number of non-zero columns of S M .
[-( y → ˆ y 1 ) and M ′ ( y → ˆ y ) = 
 for ˆ y / ∈ { ˆ y 1 , ˆ y 2 } .-]Lemma 11.
Suppose M is a vertex of the LP (2) with k non-zero columns of S M .
If k ≥ 2 , then[-it is possible to write M as a convex combination of two other feasible points given as M + ηM ′ and M − ηM ′ for small enough η , where M ′ ( y → ˆ y 2 ) = − M ′ ( y → ˆ y 1 ) = 1 and M ′ ( y ′ → ˆ y ′ ) = 
-]
M has a k × k submatrix with all distinct rows, consisting of only U ’s and L ’
s.
Proof.
M is a vertex if and only if there are |Y|·|O| many linearly independent constraints which are tight.
We count the number of tight constraints just from S M and note some instances of dependence to give a lower bound on the number of linearly independent constraints.
|Y| constraints, given by (cid:80) ˆ
y M y
→ ˆ y = 1 are always tight.
For a zero column of S M corresponding to ˆ y , each M y
→ ˆ y ≥ 0 is tight corresponding to each y ∈ Y .
These correspond to |Y| · ( |O| − k ) constraints.
For a non-zero column of S M corresponding to ˆ y , let C ˆ y ( U ) denote the number of U entries in column ˆ y and similarly, define C ˆ y ( L ) and C ˆ y ( S ) analogously.
For each pair y 1 , y 2 such that S M ( y 1 , ˆ y )
=
L and S M ( y 2 , ˆ y )
=
U , we have a tight constraint M y 1 → ˆ y ≤ e ε · M y 2 → ˆ y .
However, these constraints are not linearly independent.
In fact, there are only C ˆ y ( U ) + C
ˆ y
( L ) − 1 = |Y|− C
ˆ y ( S ) − many linearly independent constraints among these.
Another instance where the constraints might be dependent is if two rows of S M , corresponding to say y 1 and y 2 , are identical and do not contain any S ’s.
In this instance, the two equations of (cid:80) ˆ y M y
i
→ ˆ y = 1 and the inequalities given by the 0 s, U ’s, and L ’s are not independent.
The differential privacy inequality conditions imply that all the coordinates are equal between the two rows, which imply a dependence relation between those and the two conditions.
Counting these all up, we have a lower bound on the number of linearly independent constraints. This must be at least |Y| · |O| .
Let (# of duplicate rows not containing S ) be the difference between the number of all rows not containing S and the size of the set of all rows not containing S .
Thus, we get
[Equation]
duplicate rows [Equation] ≥ [Equation]
Rearranging,
[Equation]
) ≥ k and because (cid:88) [Equation] containing
[Equation] we conclude [Equation] and hence, there are at least k rows, which are all distinct and contain only 0 ’s,
U ’s and L ’s. Narrowing our scope to just the k non-zero columns, we get a k × k sub-matrix which contains only U ’s and L ’s. This concludes the proof of Lemma 11.
So far, we did not use any information about the objective.
Next we use the properties of the loss function ℓ to show that if M is an optimal solution to the LP, any submatrix provided by Lemmacan only be of one form, the matrix with U ’s along the diagonal and L ’s everywhere else: Lemma 12. Suppose M is an optimal vertex solution to the LP 2 with k ≥ 2 non-zero columns. Then any k × k submatrix N given by Lemma 11 has U ’s along the diagonal and L ’s otherwise.
Proof. Firstly, in each row of N , the U ’s are consecutive. Suppose for any y and ˆ y 1 <
ˆ y 2 <
ˆ y 3 , it holds that S M ( y, ˆ y 1 ) =
S M ( y, ˆ y 3 ) =
U , whereas S M ( y, ˆ y 2 ) is either L or S .
This implies that ℓ (ˆ y 1 , y ) , ℓ (ˆ y 3 , y ) <
ℓ (ˆ y 2 , y ) , since otherwise, it is possible to reduce M y
→ ˆ y 1 (resp.
M ( y → ˆ y 3 ) ) and increase M y →
ˆ y 2 thereby further reducing the objective, without violating any constraints.
But this is a contradiction to the assumption of ℓ in Theorem 8, since ℓ ( · , y ) cannot increase from ˆ y 1 to ˆ y 2 and then decrease from ˆ y 2 to ˆ y 3 .
Secondly, S M cannot contain the following 2 × 2 matrix, where y 1 < y 2 and ˆ y 1 < ˆ yˆ y
[Equation]
y [Equation] L Since M is optimal, we get that ℓ (ˆ y 1 , y 1 ) >
ℓ (ˆ y 2 , y 1 ) , and from the assumption on ℓ it follows that ˆ y 1 < y 1 (since otherwise, y 1 ≤ ˆ y 1 <
ˆ y 2 which would imply ℓ (ˆ y 1 , y 1 ) <
ℓ (ˆ y 2 , y 1 ) ).
Similarly, we have that ℓ (ˆ y 2 , y 2 ) >
ℓ (ˆ y 1 , y 2 ) and y 2 <
ˆ y 2 .
However, since ˆ y 1 < y 1 < y 2 < ˆ y 2 , we have that ℓ (ˆ y 2 , y 1 ) >
ℓ (ˆ y 2 , y 2 ) and ℓ (ˆ y 1 , y 2 ) > ℓ (ˆ y 1 , y 1 ) , which gives rise to a contradiction: [Equation]
In particular, this claim of not containing the above 2 × 2 matrix applies to N . Lastly, every row of N has at least one U . Suppose for contradiction that the row y in S M does not contain a single U . Then it has all L ’s and 0 ’s. Any column that contains an L must contain a U in
S M .
So necessarily there is a row y ′ in S M containing a U . y containing only L ’s and 0 ’s implies that (cid:80) ˆ y M y
→ ˆ y < (cid:80) ˆ y M y ′
→ ˆ y which is a contradiction of the constraints of the linear program.
Since the rows of this k × k sub-matrix are all distinct, and it does not contain any 2 × 2 submatrix as above, the only possible signature is where all the diagonal signature entries are U and the rest are L .
Let s i ∈
[ k ] be the index of the first index U for the i th row and similarly e i ∈
[ k ] be the last index of U .
Because the U ’s are continuous, s i and e i determine the signature of the row.
If s i
=
s j for i ̸
= j , then e i =
e j .
Else if e i < e j , then (cid:80) ˆ y M y
→ ˆ y < (cid:80) ˆ y M y ′
→ ˆ y where y corresponds to row i and y ′ corresponds to j which is a contradiction.
Similarly for e i > e j .
The same argument can be made if e i =
e j , then s i =
s j .
The rows of N are distinct, so this implies that s i ̸
= s j , e i ̸ =
e j for i ̸ =
j. The observation about the 2 × 2 matrix shows that if i < j , s i ≤ s j and e i ≤ e j .
So the s i and e i are both k distinct ordered sequences of [ k ] .
The only such sequence is s i
=
e i = i, implying
N is a diagonal matrix of this form.
This completes the proof of Lemma 12.
We now have the necessary observations to prove Theorem 8.
Let M be an optimal vertex solution to the LP 2.
That is L ( M ; P ) = inf M , M→O L ( M ; P ) . If M has one non-zero column, then S M has one column entirely of S and the rest 0.
By Claim 10, M corresponds to a RR - on - Bins Φ ε mechanism.
If M has k ≥ 2 non-zero columns, then by Lemma 12, S M has a k × k submatrix with U along the diagonal and L otherwise.
We show this completely determines S M in the form described by Claim 10 [2a-c].
Note that S M for any vertex M has at most one S per row.
If S M ( y, ˆ y 1 ) and S M ( y, ˆ y 2 ) are both equal to S for y 1 ̸ = y 2 , then it is possible to write M as a convex combination of two other feasible points given as M + ηM ′ and M − ηM ′ for small enough η , where M ′ ( y → ˆ y 2 )
= − M ′
( y → ˆ y 1 ) = and M ′ ( y ′ → ˆ y ′ ) {+= 
 for all other y ′ , ˆ y ′+} .
Intuitively this corresponds to moving mass M y → ˆ y 1 to / from M y
→ ˆ y 2 , in a way that does not violate any of the constraints.
But M is a vertex so it cannot be the convex combination of two feasible points.
But moreover, S M for an optimal vertex solution has exactly one U per row and the rest L ’s and 0’s.
Suppose the row corresponding to y has S M ( y, ˆ y ) either L or S and the rest of the entries either L or 0.
From the observations above of the submatrix N , there exists a row y ′
such that S M ( y ′ , ˆ y )
=
U .
Then, we have (cid:80) ˆ y M y
→ ˆ y < (cid:80) ˆ y M y ′
→ ˆ y = 1 , which contradicts feasibility.
Similarly, if a row of
S M contains a U and an S , then we would have (cid:80) ˆ y M y
→ ˆ y > (cid:80) ˆ y M y ′
→ ˆ y .
This allows us to define Ψ :
Y → O to be the unique ˆ y such that S M ( y, ˆ y )
=
U .
Note that Ψ is non-decreasing from the observation earlier that S M cannot contain the 2 × 2 signature above.
This completely characterizes S M as the form described by Claim 10 [2a-c], completing the proof of Theorem 8.
We use a continuity argument to show that Theorem 8 holds in the case where the loss function ℓ (ˆ y, y ) is decreasing / increasing instead of strictly decreasing / increasing.
Corollary 13. Let P be a distribution over R with finite support, O a finite subset of R , and ℓ : R × R → R ≥ 0 is such that • For all y ∈ R , ℓ (ˆ y, y ) is decreasing in ˆ y when ˆ y ≤ y and increasing in ˆ y when ˆ y ≥
y .
• For all ˆ y ∈ R , ℓ (ˆ y, y ) is decreasing in y when y ≤ ˆ y and increasing in y when y ≥ ˆ y .
Then there exists Φ : Y → O such that
[Equation]
Proof.
Let V correspond to the set of solutions M that are vertices of the LP.
Let W ⊆
V correspond to the set of vertex solutions which are RR - on - Bins .
A rephrasing of Theorem 8 is that the minimum loss over mechanisms in V is equal to the minimum loss over mechanisms in W .
For η > 0 , define ℓ η (ˆ y, y ) =
ℓ (ˆ y, y )+ η ·| y − ˆ y | .
Note that ℓ η satisfies the conditions of Theorem 8.
For any fixed mechanism M , the loss in continuous at η = 0 :
[Equation]
The minimum of finitely many continuous functions is continuous.
In particular
[Equation]
and similarly
[Equation]
but the LHS of both equations are equal by Theorem
[Equation]
so
[Equation]
completing our proof
[Equation] Stage II: Finitely supported P and ˆ Y ⊆ R .
We now set out to prove Theorem 5.
Let y min and y max denote the minimum and maximum values in Y , respectively.
We will first show that
[Equation]
where the infimum on the RHS is over all ε -DP mechanisms M .
Since RR - on - Bins Φ ε is an ε -DP mechanism, we have
[Equation]
To show the converse, let γ > 0 be any parameter.
There must exist an ε -DP mechanism M ′ :
Y → R such that
[Equation]
Let O ⊆
[ y min , y max ] be defined as follows:
• For each y ∈ Y , let a y
=
min ˆ y ∈
[ y min ,y max ] ℓ (ˆ y, y ) and b y = max ˆ y ∈
[ y min ,y max ] ℓ (ˆ y, y ) .
Let T :=
⌈ 4( b y − a y ) /γ
• Let o y,t be the finite set containing the maximal and minimal element of { ˆ y | ℓ (ˆ y, y )
= a y + tT ( b y − a y ) } if the set is non-empty.
Otherwise, let it be the empty set.
• Let O y :
= (cid:83) Tt =
o y,t .
• Finally, let O =
(cid:83) y ∈Y O y .
Naturally, O is finite.
Finally, let M ′′ be the mechanism that first runs M ′ to get ˆ y and then outputs the element in O closest to ˆ y .
By post-processing of DP, M ′′ remains ε -DP.
Furthermore, it is not hard to see that by the construction of O and by Assumption 4, we have
[Equation]
Finally, since range( M ′′ ) = O is finite, the proof in Stage I implies that
[Equation]
Combining (5), (6) and (7), we can conclude that
[Equation]
Since (8) holds for any γ > 0 , combining with (4), we can conclude that (3) holds.
Next, we will show that there exists ˆ Y ∗ and Φ ∗ :
Y → ˆ Y ∗ such that
[Equation]
Combining this with (3) completes the proof.
For any Φ : Y → R , let P Φ denote the partition on Y induced by Φ − 1 .
Note that the RHS of (9) can be written as
[Equation]
where the minimum is over all partitions of P .
For a fixed partition P = P Φ of Y , L ( RR - on - Bins Φ ε ; P ) can simply be written as
[Equation]
where ˆ y i is the output corresponding to the part S i in the partition.
Notice that the function ˆ y i → (cid:16)(cid:80)
y ∈Y p
y e ε 1 [ y ∈ Si ] e ε + | ˆ Y|− 1 ℓ (ˆ y i , y ) (cid:17) is continuous.
Furthermore, it is obvious that it increases once it becomes further from [ y min , y max ] .
Thus, the minimum must be achieved at some point ˆ y ∗ i ∈
[ y min , y max ] .
As a result, by defining Φ ∗P such that Φ ∗P ( S i ) =
ˆ y ∗ i , this also achieves the minimum for inf Φ:
Y→ ˆ Y ˆ Y⊆ R
P Φ
= P
L ( RR - on - Bins Φ ε ; P ) .
Therefore, plugging this back into (10), we can conclude that the minimum of inf ˆ Y⊆ R inf Φ: Y→ ˆ Y L ( RR - on - Bins Φ ε ; P ) must be achieved by some ˆ Y ∗ , Φ ∗ .
This completes our proof of Theorem 5.
A. E XTENSION TO ARBITRARY ( INFINITELY - SUPPORTED ) P AND ˆ Y ⊆ R .
We show that Theorem 5 can be extended to hold even for infinitely-supported distributions P over a bounded interval, but under an additional assumption that the loss ℓ is Lipschitz over the said interval.
We remark that the assumption of P supported over a bounded interval is reasonable in practice because, intuitively speaking if this were to not be the case, then the differential privacy guarantee would require that the output of the mechanism on y 1 and y 2 be “close” (in the DP sense), but | y 1 − y 2 | can be arbitrarily far from each other making the loss L ( M ; P ) to be arbitrarily large for any mechanism.
Assumption 14.
For a specified bounded interval [ y min , y max ] , loss function ℓ : [ y min , y max ] × [ y min , y max ] → R ≥ 0 is such that both ℓ ( · , y ) and ℓ (ˆ y, · ) is L -Lipschitz. Theorem 15. For all y min , y max ∈ R , loss functions ℓ : R × R → R ≥ 0 satisfying Assumptionsand 14, and all distributions P over Y ⊆ [ y min , y max ] , there is a finite output set ˆ Y ⊆ R and a non-decreasing map Φ : Y → ˆ Y such that
[Equation]
where the infimum is over all ε -DP mechanisms M .
We break down the proof into a series of lemmas.
First we show that the infimum is reached by the infimum of RR - on - Bins mechanisms:
Lemma 16.
Under the assumptions of Theorem
[Equation]
Proof.
It suffices to show that for any γ > 0 , there exists a choice of ˆ Y ⊆ R and Φ :
Y → ˆ Y such that L ( RR - on - Bins Φ ε ; P ) ≤ inf M L ( M ; P ) + γ .
Without loss of generality, we can assume that Y =
[ y min , y max ] .
Consider a discretization of Y given as
[Equation]
where we assume for simplicity that L · y max − y min γ is an integer .
Let ρ : Y → ˜ Y be the map that rounds any element of Y to the closes element of ˜ Y .
Note that | y − ρ ( y ) | ≤ γ/ (2 L ) for all y ∈ Y .
Consider the distribution ˜ P over ˜ Y given by the “rounding process”, which samples y ∼ P and returns ρ ( y ) .
Note that there is a natural coupling between P and ˜ P such that | y − ˆ y | ≤ γ/ (2 L ) holds with probability 1 .
From Theorem 5, we have that there exists a finite ˆ Y ⊆ R and nondecreasing map ˜Φ :
˜ Y → ˆ Y such that
[Equation]
(11) where ˜ M is an ε -DP mechanism mapping ˜ Y to ˆ Y .
We can extend ˜Φ to Φ :
Y →
ˆ Y given as Φ( y ) :
= ˜Φ( ρ ( y )) .
It is easy to see that since ˜Φ is non-decreasing, Φ is also non-decreasing.
From
Assumption 14, it follows that
[Equation]
from Assumption 14)
[Equation]
Similarly, for any ε -DP mechanism M mapping Y to ˆ Y , we can construct an ε -DP mechanism ˜ M mapping ˜ Y to ˆ Y where ˜ M (˜ y ) is sampled as M ( y ) for y ∼ P | ρ ( y )=
˜
y .
Note that sampling ˜ y ∼ ˜ P and returning y ∼ P | ρ ( y )=
˜
y is equivalent to sampling y ∼ P .
Thus, we have
[Equation]
Assumption 14)
[Equation]
Thus, combining Equations (11) to (13), we get
[Equation]
Next we show that the infimum of RR - on - Bins is reached by considering the RR - on - Bins with finitely many bins.
Towards this end, let RR - on - Bins nε denote the set of all RR - on - Bins Φ ε mechanisms where Φ : Y → ˆ Y with | ˆ Y| = n .
Lemma 17.
Suppose ℓ (ˆ y, y ) is integrable over y for any ˆ y .
Then for all ε > 0 , it holds that
[Equation]
Remark: Interestingly, Lemma 17 does not require any assumption about the distribution P or ℓ aside from integrability.
For example, P can be an unbounded distribution.
Proof.
Let
[Equation]
This is precisely inf M∈ RR - on - Bins 1 ε L ( M ; P ) , where the ˆ y is selecting the single bin to output to.
Note that α does not depend on ε .
For any n > 1 , consider RR - on - Bins Φ ε , where Φ : Y → ˆ Y for ˆ Y =
{ ˆ y 1 , . . .
, ˆ y n } .
Then we can bound the loss from below:
[Equation]
In particular, inf M∈ RR - on - Bins n ε L ( M ; P ) ≥
[Equation]
which implies that
[Equation]
Corollary 18.
Suppose ℓ (ˆ y, y ) is integrable over y for any ˆ y .
Then for all ε > 0 , there exists n ≥such that
[Equation]
Proof.
For any n , let α n :=
inf M∈ RR - on - Bins nε L ( M ; P ) .
From Lemma 17, we have that lim n →∞ α n ≥ α 1 .
If inf n α n = α 1 , then the corollary is true for n = 1 .
Else, if inf n α n = α ′ < α 1 , then there exists n 0 > 0 such that for all n > n 0 , it holds that α n > ( α 1 + α ′ ) /
Thus, inf n α n = min n ≤ n 0 α n which implies that the infimum is realized for some finite n .
Finally, we show that the infimum over RR - on - Bins nε is also achievable, completing the proof of
Theorem 15.
Lemma 19.
Suppose P is bounded within the interval Y =
[ y min , y max ] .
Suppose ℓ : R × R → R ≥satisfies Assumptions 4 and 14, then for any n > 0 , there is an output set ˆ Y ⊆ R with | ˆ Y| = n and a non-decreasing map Φ : Y → ˆ Y such that
[Equation]
Proof.
Because of the increasing / decreasing nature of ℓ , we can restrict the output of Φ to be in the range [ y min , y max ] .
Given an output range { ˆ y 1 , . . .
, ˆ y n } , the RR - on - Bins Φ ε mechanism that minimizes L ( RR - on - Bins Φ ε ; P ) satisfies
[Equation]
So we can consider L ( RR - on - Bins Φ ε ; P ) as a function of (ˆ y 1 , . .
, ˆ y n ) ∈
[ y min , y max ] n .
We claim that L ( RR - on - Bins Φ ε ; P ) is continuous with respect to these inputs { ˆ y 1 , . .
, ˆ y n } .
Indeed by the Lipschitz continuity, a change in [ˆ y 1 , . . .
, ˆ y n ] to [ˆ y ′ 1 , . .
, ˆ y ′ n ] where | ˆ y i −
ˆ y ′ i
| < δ for some δ > 0 results in a change in L ( RR - on - Bins Φ ε ; P ) bounded by nLδ .
Therefore L ( RR - on - Bins Φ ε ; P ) is a continuous function of (ˆ y 1 , . .
, ˆ y n ) over a compact set [ y min , y max ] n and hence it attains its infimum.
This infimum defines Φ that satisfies the lemma.
B A NALYSIS OF D YNAMIC P ROGRAMMING A LGORITHM FOR F INDING O PTIMAL RR - on - Bins
Below we give correctness and running time analysis of Algorithm 2.
Correctness Proof.
We will prove the following statement by strong induction on i :
[Equation]
where the minimum is across all Φ : { y 1 , .
, y i } → R such that, for each ˆ y ∈ range(Φ) , Φ − 1 (ˆ y ) is an interval and that |P
Φ | =
j where the j intervals are S 1 , . .
, S j (in increasing order).
Before we prove the above statement, note that it implies that A [ n ][ d ] = ( d − 1 + e ε ) · min [Equation] Y [Equation]
| =
d L ( RR - on - Bins Φ ε ; P ) .
Thus, the last line of the algorithm ensures that we output the optimal RR - on - Bins as desired.
We will now prove (14) via strong induction on i .
• Base Case.
For i =
(and thus j = 0 ), the statement is obviously true.
• Inductive Step.
Now, suppose that the statement is true for all i = 0 , . .
, t − 1 for some t ∈ N .
We will show that it is also true for i = t .
To see this, we may rewrite the RHS term (for i = t ) as
[Equation]
where the third inequality follows from the inductive hypothesis and the definition of L [ r + 1]
[ t ] .
The last expression is exactly how A [ t ][ j ] is computed in our algorithm.
Thus, (14) holds for i
= t .
Running Time Analysis.
It is clear that, apart from the computation of L [ r + 1][ i ] , the remainder of the algorithm runs in time O ( k 2 ) .
Therefore, the total running time is O ( k 2 · T ) where T denotes the running time for solving the problem min
ˆ y ∈ R (cid:80)
y ∈Y p y · e 1 [ y ∈
[ y r ,y i ]] · ε · ℓ (ˆ y, y ) .
When the loss function ℓ is convex, this problem is a (univariate) convex optimization problem and can be solved in polynomial time.
We can even speed this up further.
In fact, for the three main losses we consider (squared loss, Poisson loss, and absolute-value loss) this problem can be solved in amortized constant time, as detailed below.
Therefore, for these three losses, the total running time of the dynamic programming algorithm is only O ( k 2 ) .
• Squared Loss.
In this case, the minimizer is simply
[Equation]
Therefore, to compute ˆ y ∗ r,i , it suffices to keep the values (cid:80)
y ∈Y p y · e 1 [ y ∈
[ y r ,y i ]] · ε y and (cid:80)
y ∈Y p y · e
[ y ∈
[ y r ,y i ]] · ε .
We can start with r =
i and as we increase i , these quantities can be updated in constant time.
Note also that
[Equation]
Therefore, to compute the minimum value, we additionally keep (cid:80) y ∈Y p y · e 1 [ y ∈
[ y r ,y i ]] · ε y 2 , which again can be updated in constant time for each i .
• Poisson Log Loss.
The minimizer is exactly the same as in the squared loss, i [Equation] e.,
[Equation]
Therefore, using the same method as above [Equation] we can compute ˆ y ∗ in amortized constant time.
The minimum is the simply
[Equation]
so this can also be computed in constant time with the quantities that we have recorded.
• Absolute-Value Loss.
To describe the minimizer, we define a weighted version of median.
Let { ( w 1 , a 1 ) , . . .
, ( w t , a t ) } be a set of t tuples such that w 1 , . . .
, w t ∈ R ≥ 0 and a 1 , . .
. , a t ∈ R with a 1 ≤ · · · ≤ a t .
The weighted median of { ( w 1 , a 1 ) , . .
, ( w t , a t ) } , denoted by wmed( { ( w 1 , a 1 ) , . . .
, ( w t , a t ) } ) is equal to the minimum value a ∗ such that (cid:80) j ∈ [ t ] a j ≤ a ∗ w j ≥ ( (cid:80) j ∈ [ t ] w j ) / 2 .
It is not hard to see that
[Equation]
Notice also that we must have ˆ y ∗ r,i ∈ Y .
For a fixed r (and varying i ), the algorithm is now as follows: first compute ˆ y ∗ r,r , and also compute
[Equation]
and
[Equation]
For i = r + 1 , . . .
, k , initialize ˆ y ∗ r,i =
ˆ y ∗ r,i − 1 and update w lo or w hi (corresponding to the weight change from p y
i to e ε p y i of y i ).
We then perform a “walk” to reach the correct value of ˆ y ∗ r,i That is, if w lo < w hi , then move to the next larger value in Y ; if w lo − p ˆ y ∗ r,i · e 1 [ˆ y ∗ r,i ∈
[ y r ,y i ]] · ε ≥ w hi , then move to the next smaller value in Y .
Otherwise, stop and keep the current ˆ y ∗ r,i .
To understand the running time of this process, note that the initial running time for computing ˆ y ∗ r,r and w lo , w hi
is O ( k ) .
Furthermore, each “walk step” for ˆ y ∗ r,i takes O (1) time.
Finally, observe that if we ever move ˆ y ∗ r,i to a larger value, it must be that y i > ˆ y ∗ r,i .
After this i , ˆ y ∗ r,i will never decrease again.
As a result, in total across all i = r +1 , . . .
, k , the total number of steps can be at most 2 k .
Thus, the total running time for a fixed r is O ( k ) .
Summing up across r ∈ [ k ] , we can conclude that the total running time of the entire dynamic programming algorithm is O ( k 2 ) . Privacy parameter ≥ 0 . for y ∈ Y do
C RR - on - Bins WITH A PPROXIMATE P RIOR
Recall from Section 3.2 that, when the prior is unknown, we split the budget into ε 1 , ε 2 , use the ε 1DP Laplace mechanism to approximate the prior and then run the RR - on - Bins with privacy parameter ε 2 .
The remainder of this section gives omitted details and proofs from Section 3.2.
Throughout this section, we use k to denote |Y| (the size of the input label set).
C. L APLACE M ECHANISM AND I TS G UARANTEES
We start by recalling the Laplace mechanism for estimating distribution.
Recall that the Laplace distribution with scale parameter b , denoted by Lap( b ) , is the distribution supported on R whose probability density function is 12 b exp( −| x | /b ) .
The Laplace mechanism is presented in Algorithm 4.
It is well-known (e.g., Dwork et al.
(2006b))
that this mechanism satisfies ε -DP.
Its utility guarantee, which we will use in the analysis of LabelRandomizer , is also well-known:
Theorem 20 (e.g., Diakonikolas et al.
For any distribution P on Y , n ∈ N , and ε > 0 , we have
[Equation]
P ROOF OF T HEOREM
Theorem 6.
LabelRandomizer ε 1 ,ε 2 is ( ε 1 + ε 2 ) -DP.
Proof of Theorem 6.
The Laplace mechanism is ε 1 -DP; by post-processing property of DP, Φ ′ is also ε 1 -DP.
For fixed Φ ′ , since RR - on - Bins Φ ′ ε 2 is ε 2 -DP
and it is applied only once on each label, the parallel composition theorem ensures that (ˆ y 1 , . . .
, ˆ y n ) is ε 2 -DP.
Finally, applying the basic composition theorem, we can conclude that the entire algorithm is ( ε 1 + ε 2 ) -DP.
C. P ROOF OF T HEOREM
Theorem 7.
Let ℓ : R × R → R ≥ 0 be any loss function satisfying Assumption 4.
Furthermore, assume that ℓ (ˆ y, y ) ≤ B for some parameter B for all y, ˆ y ∈ Y .
For any distribution P on Y , ε > 0 , and any sufficiently large n ∈ N , there is a choice of ε 1 , ε 2 > 0 such that ε 1 + ε 2 = ε
and
[Equation]
where P ′ , Φ ′ , ˆ Y ′ are as in LabelRandomizer
ε 1 ,ε 2 and the infimum is over all ε -DP mechanisms M .
Proof of Theorem 7.
From Theorem 20, we have
[Equation]
Recall from Theorem 5 that there exists Φ : Y → ˆ Y such that L ( RR - on - Bins Φ ε ; P ) =
inf M L ( M ; P ) .
Consider instead the RR - on - Bins Φ ε 2 mechanism.
We have | inf M L [Equation] M
[Equation] P [Equation] RR - on - Bins Φ ε
[Equation]
P [Equation] RR [Equation]
on
[Equation]
Bins
[Equation]
RR
[Equation]
≤ [Equation] B ( [Equation]
= 2 Bε 1 . Recall that RR - on - Bins Φ ′ ε 2 is an ε 2 -DP optimal mechanism for prior P ′ .
That is, we have
L ( RR - on -
[Equation]
Finally [Equation] we also have
[Equation]
Combining the above five inequalities [Equation] we arrive at
[Equation]
Setting ε 1 = (cid:112) k/n then yields the desired bound 6 .
D DP M ECHANISMS D EFINITIONS
In this section [Equation] we recall the definition of various DP mechanisms that we use throughout the paper.
Definition 21.
Let f be a function taking as input a dataset and returning as output a vector in R d .
Then, the global sensitivity ∆( f ) of f is defined as the maximum, over all pairs ( X, X ′ ) of adjacent datasets, of || f ( X ) − f ( X ′ )
|| 1 .
The (discrete) Laplace distribution with scale parameter b > 0 is denoted by DLap( b ) .
Its probability mass function is given by p ( y ) ∝ exp( −| y | /b ) for any y ∈ Z .
Definition 22.
Let f be a function taking as input a dataset X and returning as output a vector in Z d .
The Discrete Laplace mechanism applied to f on input X returns f ( X ) + ( Y 1 , . . .
, Y d ) where each Y i is sampled i.i.d.
from DLap(∆( f ) /ε ) .
The output of the mechanism is ε -DP.
Next, recall that the (continuous) Laplace distribution Lap( b ) with scale parameter b > 0 has probability density function given by h ( y )
∝ exp( −| y | /b ) for any y ∈ R .
Definition 23 (Dwork et al.
(2006b)) .
Let f be a function taking as input a dataset X and returning as output a vector in R d .
The Continuous Laplace mechanism applied to f on input X returns f ( X ) + ( Y 1 , . . .
, Y d ) where each Y i is sampled i.i.d.
from Lap(∆( f ) /ε ) .
The output of the mechanism is ε -DP.
We next define the discrete and continuous versions of the staircase mechanism (Geng & Viswanath, 2014).
Definition 24.
Fix ∆ ≥ 2 .
The Discrete Staircase distribution is parameterized by an integer 1 ≤ r ≤ ∆ and has probability mass function given by:
[Equation]
where
[Equation]
Let f be a function taking as input a dataset X and returning as output a scalar in Z .
The Discrete
Staircase mechanism applied to f on input X returns f ( X ) + Y where Y is sampled from the
Discrete Staircase distribution given in (15).
Definition 25.
The Continuous Staircase distribution is parameterized by γ ∈ (0 , 1) and has probability density function given by:
[Equation]
where
[Equation]
Let f be a function taking as input a dataset X and returning as output a scalar in R .
The Continuous
Staircase mechanism applied to f on input X returns f ( X ) + Y where Y is sampled from the
Continuous Staircase distribution given in (16).
Definition 26 (McSherry & Talwar (2007)) .
Let q ( · , · ) be a scoring function such that q ( X, r ) is a real number equal to the score to be assigned to output r when the input dataset is X .
The
Exponential mechanism returns a sample from the distribution that puts mass ∝ exp( εq ( X, r )) on each possible output r .
It is (2 ε ∆( q )) -DP, where ∆( q ) is defined as the maximum global sensitivity of q ( · , r ) over all possible values of r .
Definition 27 (Warner (1965)) .
Let ε ≥ 0 , and q be a positive integer.
The Randomized Response mechanism with parameters ε and K (denoted by RR ε,q ) takes as input y ∈ { 1 , . .
, K } and returns a random sample ˜
y drawn from the following probability distribution:
[Equation]
The output of RR ε,q is ε -DP.
E A DDITIONAL E XPERIMENT D ETAILS
We evaluate the proposed RR - on - Bins mechanism on three datasets, and compare with the Laplace
Mechanism (Dwork et al., 2006b), the Staircase Mechanism (Geng & Viswanath, 2014) and the
Exponential Mechanism (McSherry & Talwar, 2007).
For real valued labels (the Criteo Sponsored
Search Conversion dataset), we use the Continuous Laplace Mechanism (Definition 23) and the Continuous Staircase Mechanism (Definition 25), and for integer valued labels (the US Census dataset and the App Ads Conversion Count dataset), we use the Discrete Laplace Mechanism (Definition 22) and the Discrete Staircase Mechanism (Definition 24).
E. C RITEO S PONSORED S EARCH C ONVERSION
The Criteo Sponsored Search Conversion dataset is publicly available from https://ailab.
criteo.com/criteo-sponsored-search-conversion-log-dataset/ .
To predict the conversion value ( SalesAmountInEuro ), we use the following attributes as inputs:
• Numerical attributes: Time_delay_for_conversion , nb_clicks_1week , product_price .
• Categorical attributes: product_age_group , device_type , audience_id , product_gender , product_brand , product_category_1 ∼ product_category_7 , product_country , product_id , product_title , partner_id , user_id .
All categorical features in this dataset have been hashed.
We build a vocabulary for each feature by counting all the unique values.
All the values with less than 5 occurrences are mapped to a single out-of-vocabulary item.
We randomly partition the dataset into 80%/20% train/test splits.
For each evaluation configuration, we report the mean and std over 10 random runs.
In each run, the dataset is also partitioned with a different random seed.
Our deep neural network consists of a feature extraction module and 3 fully connected layers.
Specifically, each categorical attribute is mapped to a 8-dimensional feature vector using the prebuilt vocabulary for each attribute.
The mapped feature vectors are concatenated together with the numerical attributes to form a feature vector.
Then 3 fully connected layers with the output dimension 128, 64 and 1 are used to map the feature vector to the output prediction.
The ReLU activation is applied after each fully connected layer, except for the output layer.
We train the model by minimizing the mean squared error (MSE) with L2 regularization 10 − 4 , using the RMSProp optimizer.
We use learning rate 0.001 with cosine decay (Loshchilov & Hutter, 2016), batch size 8192, and train for 50 epochs.
Those hyperparameters are chosen based on a setup with minor label noise (generated via Laplace Mechanism for ε = 6 ), and then fixed throughout all runs.
For the RR - on - Bins mechanism, we use the recommended ε 1 = (cid:112) |Y| /n in Theorem 7 to query a private prior, and with the remaining privacy budget, optimize the mean squared loss via dynamic programming.
When running Algorithm 2, it would be quite expensive to use Y as the set of all unique (real valued) training labels.
So we simply discretize the labels by rounding them down to integer values, and use Y = { 0 , 1 , . .
The integer labels are then mapped via Algorithm 1.
E. US C ENSUS
The 1940 US Census can be downloaded from https://www.archives.gov/research/census/ 1940 .
We predict the time that the respondent worked during the previous year (the WKSWORK1 field, measured in number of weeks), based on the following input fields: the gender ( SEX ), the age ( AGE ), the marital status ( MARST ), the number of children ever born to each woman ( CHBORN ), the school attendance ( SCHOOL ), the employment status ( EMPSTAT ), the primary occupation ( OCC ), and the type of industry in which the person performed an occupation ( IND ).
We use only 50,582,693 examples with non-zero WKSWORK1 field, and randomly partition the dataset into 80%/20% train/test splits.
For each evaluation configuration, we report the mean and std over 10 random runs.
In each run, the dataset is also partitioned with a different random seed.
Our deep neural network consists of a feature extraction module and 3 fully connected layers.
The feature extraction module can be described via the following pseudocode, where the vocabulary size is chosen according to the value range of each field in the 1940 US Census documentation, and the embedding dimension for all categorical features are fixed at 8.
Features = Concat([ Embedding{vocab_size=2}(SEX - 1), AGE / 30.0, Embedding{vocab_size=6}(MARST - 1), Embedding{vocab_size=100}(CHBORN), Embedding{vocab_size=2}(SCHOOL-1)
Embedding{
vocab _size=1000}(OCC)
Embedding{vocab_size=1000}(IND)
The features vectors are mapped to the output via 3 fully connected layers with the output dimension 128, 64, and 1.
The ReLU activation is applied after each fully connected layer, except for the output layer.
We train the model by minimizing the MSE with L2 regularization 10 − 4 , using the RMSProp optimizer.
We use learning rate 0.001 with cosine decay (Loshchilov & Hutter, 2016), batch size 8192, and train for 200 epochs.
For the RR - on - Bins mechanism, we use the recommended ε 1 = (cid:112) |Y| /n in Theorem 7 to query a private prior, and with the remaining privacy budget, optimize the mean squared loss via dynamic programming.
E. A PP A DS C ONVERSION C OUNT P REDICTION
The app install ads prediction dataset is collected from a commercial mobile app store.
The examples in this dataset are ad clicks, and each label counts post-click events (aka conversions) happening in the app after the user installs the app.
The neural network models used here is similar to the models used in other experiments: embedding layers are used to map categorical input attributes to dense feature vectors, and then the contatenated feature vectors are passed through several fully connected layers to generate the prediction.
We use the the Adam optimizer (Kingma & Ba, 2015) and the Poisson regression loss (Cameron & Trivedi, 2013) for training.
For the RR - on - Bins mechanism, we use the recommended ε 1 = (cid:112) |Y| /n in
Theorem 7 to query a private prior, and with the remaining privacy budget, optimize the mean squared error via dynamic programming.
Following the convention in event count prediction in ads prediction problems, we train the loss with the Poisson Log Loss.
We report the relative Poisson
Log Loss on the test set with respect to the non-private baseline.
F L ABEL C LIPPING
The Laplace Mechanism and Staircase Mechanism we compared in this paper could both push the randomized labels out of the original label value ranges.
This issue is especially severe for small ε ’s, as the range of randomized labels could be orders of magnitude wider than the original label value range.
Since the original label value range is known (required for deciding the noise scale of each
DP mechanism), we could post process the randomized labels by clipping to this range.
We compare the effects of clipping for both Laplace Mechanism and Staircase Mechanism on the
Criteo dataset in Figure 6.
In both case, this simple mitigation helps reduce the error significantly, especially for smaller epsilons.
So in all the experiments of this paper, we always apply clipping to the randomized labels.
Figure 6 also shows the Exponential Mechanism (McSherry & Talwar, 2007), which is equivalent to restricting the Laplace noises to the values that would not push the randomized labels out of the original value range.
In our case, we implement it via rejection sampling on the standard Laplace
Mechanism.
This algorithm avoid the boundary artifacts of clipping, but to guarantee the same level of privacy, the noise needs to scale with 2 /ε , instead of 1 /ε as in the vanilla Laplace Mechanism.
As a result, while it could be helpful in the low epsilon regime, in moderate to high ε regime, it is noticeably worse than all other methods, including the vanilla Laplace Mechanism due to the 2 × noise scaling with 1 /ε . G C OMPARISON WITH RRWithPrior In this paper, we extended the formulation of Ghazi et al. (2021a)
from classification to regression losses.
Here we provide a brief comparison between our algorithm ( RR - on - Bins ) and the
RRWithPrior algorithm from Ghazi et al.
(2021a)
on the US Census dataset.
Instead of using multistage training, we fit RRWithPrior in our feature-oblivious label DP framework, and use the same privately queried global prior as the input to both RR - on - Bins and RRWithPrior .
For RRWithPrior ,
[Figure]
PrivacyBudget
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
∞
[Table]
[Table]
[Table]
we simply treat each of the integer label values in US Census as an independent category during the randomization stage.
Once the private labels are obtained, the training stage is identical for both algorithms.
The results are shown in Table 2.
Note the results are for reference purposes, as this is not a fair comparison.
Because RRWithPrior was designed for classification problems (Ghazi et al., 2021a), which ignore the similarity metrics between different labels, while RR - on - Bins takes that into account.