T ASK - AWARE P RIVACY P RESERVATION FOR M ULTI - DIMENSIONAL D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Local differential privacy (LDP), a state-of-the-art technique for privacy preservation, has been successfully deployed in a few real-world applications.
In the future, LDP can be adopted to anonymize richer user data attributes that will be input to more sophisticated machine learning (ML) tasks.
However, today’s LDP approaches are largely task-agnostic and often lead to sub-optimal performance – they will simply inject noise to all data attributes according to a given privacy budget, regardless of what features are most relevant for an ultimate task.
In this paper, we address how to signiﬁcantly improve the ultimate task performance for multidimensional user data by considering a task-aware privacy preservation problem.
The key idea is to use an encoder-decoder framework to learn (and anonymize) a task-relevant latent representation of user data, which gives an analytical nearoptimal solution for a linear setting with mean-squared error (MSE) task loss.
We also provide an approximate solution through a learning algorithm for general nonlinear cases.
Extensive experiments demonstrate that our task-aware approach signiﬁcantly improves ultimate task accuracy compared to a standard benchmark
LDP approach while guaranteeing the same level of privacy.
I NTRODUCTION
In recent years, there has been a tremendous growth in the volume of available data for machine learning (ML) tasks, leading to increasing emphasis on protecting user privacy.
Differential privacy (DP) (Dwork et al., 2006; 2014) is a state-of-the-art technique for data privacy, and its local variant – local differential privacy (LDP) (Kasiviswanathan et al., 2011) – provides stronger privacy guarantees for individual users without dependence on any trusted third party.
In practice, LDP has been successfully deployed in products of companies like Google (Erlingsson et al., 2014), Apple (Differential Privacy Team), and Microsoft (Ding et al., 2017) for some basic frequency or histogram estimation tasks where raw user data is restricted to an n -bit discrete variable.
In the future, LDP has the promising potential to be adopted in more complex scenarios (Hassan et al., 2019; Dankar & El Emam, 2013; Zhao et al., 2014; Cort´es et al., 2016)
(e.g., health care, power grids, Internet of Things) that feature richer user data attributes that feed into more sophisticated downstream ML tasks.
In such cases, today’s standard task-agnostic LDP approaches may not be ideal.
For example, consider complex user data that must be anonymized before passing it into a
ML task function, such as a neural network classiﬁer for credit scores.
A standard approach would be to simply perturb the data by adding artiﬁcial noise whose scale depends on the sensitivity of user data (i.e. worst-case variation among a user population) and a given privacy budget, regardless of what ultimate task the anonymized data will be used for.
However, as the dimension and variability of user data inevitably grows, today’s methods would generally have to increase the scale of noise to provide the same LDP guarantee, even though many data attributes might be highly variable across a user population, but minimally relevant for a task.
As a consequence, one often adds excessive noise to all data attributes, which can severely degrade an ultimate task’s performance.
To address these challenges, this paper introduces a fundamentally different task-aware LDP approach.
Our method improves the performance of ML tasks that operate on multi-dimensional user data while still guaranteeing the same levels of privacy.
Our key technical insight is to characterize the dependence of task performance on various user data attributes, which guides how we learn a concise, task-relevant encoding (i.e. latent representation) of user data.
Then, for the same privacy budget, we can directly expose and perturb only the task-relevant encoding rather than raw user data, which often allows us to add less noise and thereby improve task accuracy (see Section 2 for a concrete example).
Crucially, user privacy is guaranteed under the same privacy budget according to the post-processing immunity of DP (Dwork et al., 2014) (i.e., one cannot make the output of a privacy algorithm less differentially private without additional knowledge).
As such, an adversary cannot decode the anonymized latent representation to reduce the level of privacy.
Our method allows us to learn and expose only high-valued data attributes and ﬂexibly adjust their signal-to-noise ratio based on their importance to a task.
Moreover, when different data attributes are inter-dependent, task-aware LDP preservation is even more promising in that we can consider the utilities of the underlying orthogonal bases through principal component analysis (PCA) (Dunteman, 1989), instead of the raw data attributes.
Contributions.
In light of prior work, our contributions are three-fold.
First, we propose a taskaware privacy preservation problem in which the effect of noise perturbation to preserve LDP is effectively considered, based on an encoder-decoder framework (Section 3).
Second, in terms of task-aware privacy preservation, we obtain an analytical near-optimal solution for a linear setting and MSE task loss, and provide a heuristic learning algorithm for more general settings (Section 4).
Third, we validate the effectiveness of our task-aware approach through three real-world experiments, which show our task-aware approach outperforms the benchmark approaches on overall task loss under various LDP budgets by as much as 73 .
(Section 5).
All the proofs are given in the
Appendix.
Related Work.
1) Utility maximization in DP/LDP.
Most theoretical DP/LDP research either provides a utility upper bound for a given privacy budget under some weak assumptions (Alvim et al., 2011; Kenthapadi et al., 2012; Duchi et al., 2013; Makhdoumi & Fawaz, 2013; Hardt & Talwar, 2010;
Wang et al., 2019b; Acharya et al., 2020), or designs optimal privacy preservation mechanisms in some speciﬁc use cases (McSherry & Talwar, 2007; Wasserman & Zhou, 2010; Friedman & Schuster, 2010; Xiao et al., 2010; Thakurta & Smith, 2013; Kairouz et al., 2014; Geng & Viswanath, 2015; Liu et al., 2016; Yiwen et al., 2018; Joseph et al., 2019; Gondara & Wang, 2020;
Wang et al., 2020).
To the best of our knowledge, with respect to multi-dimensional user data , Murakami & Kawamoto (2019), Wang et al.
(2019a)
and
Chen et al.
(2021) are the only three similar works to ours.
Murakami & Kawamoto (2019) develops a utility-maximizing LDP framework under the assumption that some data attributes may not be privacy-sensitive, and hence the utility improvement is mainly achieved by providing privacy guarantees for only a subset of attributes.
Wang et al. (2019a) mainly focuses on developing novel LDP mechanisms for multi-dimensional data with an objective of minimizing the worst-case noise variance, which naturally improves utility.
Chen et al.
(2021) preserves DP for images by adding noise to a latent representation learned through back-propagation of task loss, where the effect of noise perturbation is not considered.
The key differences of our work are: i) we don’t make additional assumptions on the sensitivity of user data, ii) our task-aware approach achieves a better task performance than standard LDP benchmarks by directly studying the dependencies between the task objective and different attributes of user data, and iii) we effectively capture the effect of noise perturbation resulting from privacy requirements.
2) End-to-end (E2E) learning .
There has been a wide variety of works training a cascade of deep neural networks (DNNs) through E2E learning, where a task-speciﬁc output is directly predicted from the raw inputs (Muller et al., 2006; Wang et al., 2012; Donti et al., 2017; Zhou & Tuzel, 2018; Amos et al., 2018).
Our work also follows such a practice, but introduces an LDP guarantee while improving the task performance.
3) DP in deep learning.
The popularity of deep learning also draws great attention to DP preservation therein (Song et al., 2013; Shokri & Shmatikov, 2015;
Abadi et al., 2016; Phan et al., 2016; Papernot et al., 2016; Phan et al., 2017; McMahan et al., 2018;
Wang et al., 2018; Phan et al., 2019; Arachchige et al., 2019; Liu et al., 2020; Mireshghallah et al., 2020; Bu et al., 2020).
Our work is fundamentally different.
Instead of preserving LDP during the learning process , we use learning as a tool to ﬁnd the salient representation that improves the task performance under a given privacy budget.
In other words, we don’t perturb the gradient for backpropagation but perturb the representation to guarantee LDP.
Furthermore, we don’t speciﬁcally deal with privacy preservation during the ofﬂine training process, which requires some ground truth user data (e.g., from a small set of consenting volunteers).
However, LDP of user data is guaranteed after a trained model is deployed online .
B ACKGROUND AND M OTIVATING E XAMPLE
(cid:15) -LDP (Kasiviswanathan et al., 2011).
Let x ∈ R n be an individual data sample, and X be the domain of x , which is assumed to be a compact subset of R n .
A randomized algorithm M : X (cid:55)→ R d is said to satisfy (cid:15) -LDP with privacy-budget (cid:15) > 0 , if
[Equation]
Essentially, when (cid:15) is small, one cannot readily differentiate whether the input of M is an individual user x or x (cid:48) based on M ’s outcome.
Laplace mechanism (Dwork et al., 2014).
To release a sensitive function g : X (cid:55)→ R d under (cid:15) -LDP, ∀ (cid:15) > 0 , the Laplace mechanism is a widely used mechanism which adds Laplace noise to function g :
[Equation]
where Lap d ( µ, b ) is a d -dimensional vector whose elements are i.i.d.
Laplace random variables with mean µ and scale b , which leads its variance to be 2 b 2 .
For concreteness, the analysis and evaluation of this paper focus on the Laplace mechanism although the central idea of learning taskrelevant data representations is applicable to other random privacy mechanisms as well.
∆ 1 g = max x,x (cid:48) ∈
X (cid:107) g
( x ) − g ( x (cid:48) )
(cid:107) 1 measures the sensitivity of g under the (cid:96) 1 norm.
[Figure]
the perturbs the a smaller while
A motivating example.
Consider an example shown in Fig.
For simplicity, we only consider an example with two people, Alice and Bob, and two data attributes, age and wage.
Suppose we need to preserve (cid:15) -LDP for each person with (cid:15)
= 1 and our task is to estimate the mean wage as closely as possible.
A straightforward taskagnostic approach will directly expose both the two data attributes, and add Laplace noise with scale b = 20 to each attribute.
However, a task-aware approach will expose only the wage attribute and add Laplace noise with scale b = 10 .
Both the approaches guarantee LDP under the same budget ( (cid:15) = 1 ), but the wage attribute given by the task-aware approach is less noisy, and we can expect that the corresponding estimated mean wage (i.e. the ultimate task objective) is close to the real value with a higher probability.
In more complex scenarios, such as when each data attribute is not redundant but is valued differently in terms of the considered task or data attributes are dependent but not perfectly correlated, etc., the optimal solution will not be as straightforward as the given example and will be explored in the following sections.
P ROBLEM F ORMULATION
We now introduce the task-aware privacy preservation problem depicted in Fig.
Let y =
f ( x ) ∈ R m denote the task output associated with each ground truth data sample x , where f represents the task function.
To guarantee (cid:15) -LDP for each data sample x , its true value should never be exposed to the task function.
Instead, an estimate of x , denoted by ˆ x , is used as the input to the task function with the corresponding task output ˆ y = f
(ˆ x ) .
The objective is to minimize the overall task loss
L = E
[ l (ˆ y, y )] due to the difference between ˆ x and x , where x follows distribution D x , and l is a task loss function that captures the discrepancy between task output ˆ y and y , such as the common (cid:96)and cross-entropy loss.
More concretely, x is ﬁrst mapped to a latent representation φ ∈ R Z through an encoder function φ = g e ( x ; θ e ) , where θ e are a set of encoder parameters.
φ is then perturbed by a Laplace noise vector w ∈ R Z .
That is, g e is treated as the sensitive function g in Eq.2.
Next, ˆ x is reconstructed from φ + w using a decoder function ˆ x = g d ( φ + w ; θ d ) where θ d are a set of decoder parameters.
Note that in reality the encoder is deployed at the end of each individual user and in general has to be lightweight (e.g., linear or one hidden-layer neural network).
[Figure]
The optimal task-aware ˆ x minimizes L while preserving (cid:15) -LDP.
In other words, the task-aware privacy preservation problem aims to co-design encoder and decoder, i.e., ﬁnd proper values for Z , θ e and θ d , such that L is minimized and (cid:15) -LDP is preserved.
Formally,
[Equation]
The difﬁculty of our task-aware LDP problem mainly comes from the discrepancy of the measurement of overall task loss L , which depends on D x and captures the average performance, and the mechanism of preserving LDP, which depends on X and focuses only on the worst-case privacy guarantee.
Benchmarks.
We now describe two natural approaches to preserve (cid:15) -LDP.
First, a task-agnostic approach adds noise directly to the normalized x 1 .
For convenience we assume x is already normalized, and we have Z = n and g e ( x ) = x .
Second, a privacy-agnostic approach adds noise to φ obtained by considering the problem deﬁned in Eq.3-6 with a pre-determined Z ≤ n and w being a zero vector instead.
That is, the privacy-preservation part is neglected when designing the encoder, and hence a proper Z needs to be pre-determined or one would always conclude a larger Z (under which more information can be delivered when noise is absent) is better.
Both two benchmark approaches still need to determine the optimal decoder parameters θ d for input φ + w .
Note that for the task-agnostic approach even though g e is an identity function, the corresponding optimal g d is usually not an identity function, as exempliﬁed in Section A.7.1.
A NALYSIS
In this section, we solve the task-aware privacy preservation problem.
Assuming a linear model and mean-squared error (MSE) task loss, we are able to ﬁnd near-optimal analytical results, which shed clear insight on how to co-design an encoder and decoder.
We then move to general settings and present a heuristic learning algorithm which demonstrates strong empirical performance.
4. L INEAR M ODEL WITH MSE T ASK L OSS
In this subsection, we consider a linear model with MSE task loss.
More speciﬁcally, encoder function g e , decoder function g d , and task function f are assumed to be linear functions in their corresponding inputs, and the loss function is l = (cid:107) (ˆ y − y )
(cid:107) 22 .
The task function f can then be expressed as f ( x ) =
Kx , where K ∈ R m × n is the task matrix.
Practicality of the setting.
Linear transformation is a common encoding and decoding approach for many dimensionality-reduction techniques, such as PCA.
And (cid:96) 2 task loss is widely used in many application scenarios.
For example, given N samples of x , suppose we want to estimate the mean value of these samples in a few directions, given by task matrix K .
Then the sum of the variance of these estimates by using ˆ x instead of x will be 1 N E x,w [ (cid:107) K
(ˆ x − x ) (cid:107) 2 2 ] , so
L = E x,w [ (cid:107) K (ˆ x − x ) (cid:107) 2 2 ] =
E x,w [ (cid:107) ( ˆ y −
y ) (cid:107) 22 ] is a natural objective function.
Contents.
We ﬁrst summarize key results.
For our task-aware approach, the optimal decoder is ﬁrst determined in Proposition 1, and then the optimal encoder and the corresponding optimal loss is formulated in Proposition 2-4 under Assumption 1.
We then relax Assumption 1 and provide lower and upper bounds for the optimal loss in Theorem 1 and an approximate solution.
All proofs are in the Appendix.
Detailed analysis.
We start our analysis with a few deﬁnitions.
First, without loss of generality, we assume the covariance matrix of x − µ x , i.e., E [( x − µ x )( x − µ x ) (cid:62) ] , is positive deﬁnite, where µ x ∈ R n is the mean vector of x .
This assumption guarantees x cannot be linear transformed to a low-dimensional representation without losing information.
We then factorize E [( x − µ x )( x − µ x ) (cid:62) ] into LL (cid:62) through Cholesky decomposition, where L ∈ R n × n is a lower triangular matrix with positive diagonal entries.
For analytical convenience, we let h =
L − 1 ( x − µ x ) , which can be viewed as another representation of x , with mean
µ h = 0 and covariance matrix Σ hh = I .
Let D h denote the distribution of h , and H =
{ L − 1 ( x − µ x ) | x ∈ X } denote the compact set that contains all the possible values of h ∼ D h .
Since K (ˆ x − x ) =
P (ˆ h − h ) , where P = KL , working with data representation h with task matrix P is equivalent to using x and
K .
Considering zero-centered h instead of original x saves us from considering the constant terms in the linear encoder and decoder functions.
Let E ∈ R Z × n and D ∈ R n × Z denote the encoder and decoder matrix associated with h , i.e., φ =
Eh and ˆ h
= D ( Eh + w ) .
Without loss of generality, we let Z ≥ n and allow some rows of E to be zero.
Equivalently, based on the relationship between x and h , we have φ =
EL − 1 ( x − µ x ) and ˆ x −
µ x = LD
( EL − 1 ( x − µ x ) + w ) .
We denote the covariance matrix of w by Σ ww , and it can be expressed as Σ ww =
σ 2 w I , where σ 2 w is the variance of the noise added to each dimension of φ .
We ﬁrst determine the optimal decoder D that minimizes L for a given encoder E , which is given by the following proposition.
Proposition 1 (Optimal decoder D that minimizes L ) .
An optimal decoder D that minimizes L for a given encoder E and σ 2 w can be expressed as D = E (cid:62) ( EE (cid:62) + σ 2 w I ) − 1 , and corresponding L is
[Equation]
where Tr ( · ) denotes the trace of a matrix.
The next main step is to ﬁnd an encoder E that minimizes Eq.7.
Since ∆ 1 g e = max v,v [Equation] ∈ E ( H ) (cid:107) v − v (cid:48) (cid:107) 1 , where E ( H ) =
{ Eh | h ∈ H } is the image of H under linear transformation E , the design of encoder E will affect ∆ 1 g e and therefore σ 2 w , and for different H ’s the effect of E is also different in general.
So we need to carefully consider the relationship between E and H .
When computing ∆ 1 g e we can actually use H ’s convex hull S instead of H itself, according to the following lemma.
This brings some analytical convenience in the further analysis.
Lemma 1 (Convex hull preserves ∆ 1 g e ) .
[Equation]
For encoder E , consider its singular value decomposition (SVD) instead: U Σ V (cid:62) ,
where U ∈ R
Z × Z and V ∈ R n × n are orthogonal matrices and Σ ∈ R Z × n is a rectangular diagonal matrix.
And the singular values are denoted by σ 1 , · · · , σ n with | σ 1 | ≥ · · · ≥ | σ n | .
Then designing E is equivalent to designing matrix U , V and Σ .
The geometric interpretation of applying transform E = U Σ V (cid:62) to set S consists of three sub-transforms: 1) rotate S by applying rotation matrix V (cid:62) ; 2) scale V (cid:62) ( S ) by applying scaling matrix Σ ; 3) rotate Σ V (cid:62) ( S ) by applying rotation matrix U .
In general, the choice of any of U , Σ , and V will affect ∆ 1 g e and hence σ 2 w .
Our overall strategy is to ﬁrst minimize the loss L and the sensitivity value ∆ 1 g e over U and V .
The resulting U and V only depend on Σ under Assumption 1 (Propositions 2, 3).
We then ﬁnd the optimal Σ that minimizes the L within a given privacy budget (cid:15) (Proposition 4).
By the end, we relax Assumption 1 to discuss the quality of our solution (Theorem 1).
We start by noting that for two points within a compact set, they must lay on the boundary to have the maximum (cid:96) 1 distance.
We then make the following assumption in terms of ∂S , which is the boundary of S .
It decouples the relationship between the choice of V and the value of ∆ 1 g e .
Assumption 1 (Boundary ∂S is a centered hypersphere) .
The boundary ∂S is a centered hypersphere of radius r ≥ 0 , which is expressed as { h ∈ R n |(cid:107) h (cid:107) 2 2 = r 2 } .
This is a strong assumption, but at the end of this subsection we will give a lower and upper bound of
L for any possible ∂S based on the results under this assumption.
Since ∂V ( S ) = { h ∈ R n |(cid:107) h (cid:107) 22 = r 2 } = ∂S for any orthogonal V , this assumption gives us a nice property: the choice of V doesn’t affect ∆ 1 g e and σ 2 w .
Based on the above assumption, we can safely consider the optimal design of
V that minimizes L when Σ and σ 2 w are given, which leads to the following proposition.
Proposition 2 (Optimal rotation matrix V that minimizes L under Assumption 1) .
Suppose the eigen-decomposition of the Gram matrix P (cid:62) P is expressed as P (cid:62) PQ = Q Λ , where Λ = diag
( λ 1 , · · · , λ n ) ∈ R n × n is a diagonal matrix whose diagonal elements are eigenvalues with λ 1 ≥ · · · ≥ λ n ≥ 0 , and Q ∈ R n × n is an orthogonal matrix whose columns are corresponding normalized eigenvectors.
Then, when Σ and σ 2 w are given, L is minimized for V = Q , any Z ≥ n , and any orthogonal U .
And the corresponding L can be expressed as:
[Equation]
It is clear that choosing a Z > n brings no additional beneﬁt.
Hence we can only consider Z = n for simplicity.
After the ﬁrst two sub-transforms Σ and V (cid:62) , the boundary ∂S = { h ∈ R n |(cid:107) h (cid:107) 22 = r 2 } becomes ∂ Σ V (cid:62) ( S )
= { v ∈ R n | (cid:80) ni
=1 v 2 i /σ
2 i =
r 2 } , which is a hyperellipsoid.
We then have the following proposition that gives the optimal U which minimizes ∆ 1 g e .
Proposition 3 (Optimal rotation matrix U that minimizes ∆ 1 g e under Assumption 1) .
For a given Σ , U = I minimizes ∆ 1 g e , and the corresponding minimum value is
[Equation]
Next, we need to consider how to design the scaling matrix Σ , or equivalently, the values of σ 21 , · · · , σ 2 n , to minimize Eq.9, which is the only remaining piece.
Clearly, for any given σ 21 , · · · , σ 2 n if we increase them proportionally, σ 2 w also needs to be increased proportionally to preserve the same (cid:15) -LDP.
So without loss of generality, we impose an additional constraint (cid:80) ni
=1 σ 2 i = M , where M is a positive constant.
And the following proposition gives the optimal choice of Σ that minimizes
L and preserves (cid:15) -LDP with the Laplace mechanism.
Proposition 4 (Optimal scaling matrix Σ that minimizes L and preserves (cid:15) -LDP with Laplace mechanism under Assumption 1) .
The optimal choice of σ 21 , · · · , σ 2 n that minimize L and preserve (cid:15) -LDP with Laplace mechanism under constraint (cid:80) ni =1 σ 2 i
= M is given by
[Equation]
where Z (cid:48) ≤ n is the largest integer such that:
[Equation]
and the corresponding L is
[Equation]
For our task-aware approach, Proposition 1-4 complete the optimal encoder and decoder design that preserves (cid:15) -LDP with the Laplace mechanism under Assumption 1.
Validation of Task Loss from Theoretical Results (Fig. 3).
We now compare the performance of our task-aware approach and the benchmark approaches when L
= I and Assumption 1 holds.
The derivations of the benchmark approaches are in Section A.7.
We consider three different settings which have n = 4 and λ 1 = 4 in common.
And in setting 1, 2, and 3 we let λ
=
λ {+
+}
= λbe 0, 1, 2 respectively.
For the privacy-agnostic approach 2 , we use a pre-determined Z = 2 .
Our observations are: 1) Compared to the task-agnostic approach, our task-aware approach achieves the largest improvement in setting 1, because λ 2:4 = 0 implies that x 2:4 are purely redundant.
We can even expect higher gain than setting 1 when we have larger n and zero λ 2: n ’s, and the gain will be zero if all the λ i ’s are equal.
The privacy-agnostic approach completely missed the information carried by x 3:4 , which explains the improvement of our task-aware approach for small r/(cid:15) in setting 2 and 3.
We can expect higher gain than setting 3 when we have larger n and larger λ 2: n ’s, and the gain will be zero if all the missed x i ’s correspond to
zero λ
i ’s and all the other λ
i ’s are equal.
[Figure]
Transition to general boundary ∂S .
Based on the results under Assumption 1, we can give a lower and upper bound of L ∗ for general ∂S , which is not necessarily a centered hypersphere.
Theorem 1 (Lower and upper bound of L ∗ for general boundary ∂S when (cid:15) -LDP is preserved with Laplace mechanism) .
Suppose ∂S ⊂ { h ∈ R n : r 2 min ≤ (cid:107) h (cid:107) 2 2 ≤ r 2 max } .
Then when (cid:15) -
LDP is preserved with the Laplace mechanism, the optimal L ∗ is bounded by:
[Equation]
where L ( r ; λ 1: n , (cid:15) ) is the value of L determined by Eq.12 and 13 for given radius r , eigenvalues λ 1: n and privacy budget (cid:15) .
Therefore, to preserve (cid:15) -LDP with the Laplace mechanism, our task-aware solution for general ∂S is: 1) First, ﬁnd the smallest r max and the largest r min that bound ∂S ; 2) Then assume ∂S is { h ∈ R n |(cid:107) h (cid:107) 22 =
r 2 max } , and choose the encoder E and decoder D based on Proposition 2-4.
We don’t use the corresponding σ 2 w however, because it may guarantee a higher LDP than needed.
Next, compute σ 2 w for real ∂S under decoder D and privacy budget (cid:15) .
The associated loss for the task-aware approach is at most L ( r max ; λ 1: n , (cid:15) ) .
Though in general not optimal, it differs from L ∗ by at most L ( r max ; λ 1: n , (cid:15) )
− L ( r min ; λ 1: n , (cid:15) ) .
The difference is small when ∂S is “nearly” a hypersphere, i.e., r max − r min ≈ 0 .
4. G ENERAL S ETTINGS
For more complex scenarios, it is challenging to give an analytical solution to the task-aware privacy preservation problem, especially when the encoder function g e , decoder function g d , and task function f correspond to neural networks.
Thus, we present a heuristic learning algorithm.
(The benchmark algorithms are given in Section A.7.4 due to space limit.)
Algorithm 1 is our proposed task-aware algorithm for general settings.
First, the privacy budget (cid:15) and the latent dimension Z are required inputs for the algorithm.
In general, Z should be proper, i.e., it is neither too small (we can ﬁnd a better solution by choosing a larger Z ) nor too big (which introduces unnecessary complexity).
In practice a practitioner may need to determine a proper Z on a case-by-case basis (See Section A.9 of the Appendix for more details).
Next, the algorithm adopts an alternating iteration approach, where in each epoch, we ﬁrst update parameters θ e , θ d by their corresponding negative gradients in line 3, and then recompute ∆ 1 g e and re-sample w from
Lap Z (0 , ∆ 1 g e /(cid:15) ) in line 4.
Note that, in terms of encoder parameter θ e , instead of considering the gradient of L , we add an (cid:96) 2 regularization term η (cid:107) θ e (cid:107) 2 where η is a positive constant.
Therefore, we update θ e with the negative gradient − ( ∇ θ e L + 2 ηθ e ) .
Without regularization, the (cid:107) θ e (cid:107) 2 will grow to inﬁnity since we can always achieve a smaller L by increasing the scale of φ proportionally.
But it is not a direction we are looking for, since σ 2 w will also increase proportionally to guarantee (cid:15) -LDP. [Figure]
[Figure]
E VALUATION
Our evaluation compares the performance of the proposed task-aware approach and the benchmark approaches.
Three applications and corresponding datasets from the standard UCI Machine Learning Repository (Dua & Graff, 2017) are considered: mean estimation of hourly household power consumption, real estate valuation [Figure] and breast cancer detection.
Conﬁguration and training details are provided in the appendix due to space limitations.
Mean Estimation of Hourly Household Power Consumption.
We ﬁrst consider a mean estimation problem, based on measurements of individual household electric power consumption over four years (Hebrail & Berard, 2012).
Each data sample x ∈ R 24 is a time-series that contains the hourly household power consumption for one single day, and our objective is to estimate the mean of the hourly household power consumption for N days.
As discussed in Section 4.1, we can deﬁne the overall task loss in the following way:
[Equation]
where K = diag ( k 1 , k 2 , · · · , k 24 ) factors the importance of the mean estimation for each hour.
In our experiment we set k i = 2 for i ∈ { 9 , 10 , · · · , 20 } (i.e., day-time hours) and k i = 1 for other i ’s (i.e., night-time hours).
And we adopt a linear encoder and decoder model.
As the considered problem is based on a linear model with MSE task loss, we adopt the solutions developed in Section 4.1 for the three approaches (we choose Z = 3 for the privacy-agnostic approach 2 ).
Fig.
4 shows our experimental results.
First, on the left, we compare the task loss l (ˆ y, y ) for the three approaches under different LDP budgets.
For each approach, the overall task loss L decreases when a larger LDP budget (cid:15) is given.
Besides, for a given LDP budget, our task-aware approach always outperforms the benchmark approaches on overall task loss L , and the maximum improvements against the task-agnostic and privacy-agnostic approach are 22 .
9% ( (cid:15) = 10 ) and 11 .
7% ( (cid:15) = 5 ), respectively.
Second, on the right, we select (cid:15) =
5 and compare the MSE of power consumption for each hour.
We see that our task-aware approach achieves a lower MSE for all the day-time hours, and a similar MSE for the night-time hours.
This observation can be explained by three reasons:
1) We select a higher k i for the day-time hours, so our task-aware approach gives higher priority to
[Figure]
minimizing the loss for those dimensions in x ; 2) Although x has 24 dimensions, the variance in each dimension can be mostly explained by several common latent dimensions, so our task-aware approach still achieves a similar MSE for the night-time hours; 3) Our task-aware approach is able to adopt different scales to different latent dimensions according to their task relevance while the privacy-agnostic approach cannot.
Next, we consider a real estate valuation problem and a breast cancer detection problem.
Both the problems are not based on the linear model with MSE task loss, so we use Algorithm 1 developed in
Section 4.2 to solve them (we use Z = 3 for both our task-aware approach and the privacy-agnostic approach for fair comparison; and the performance of the task-aware approach under different Z ’s can be found in Section A.9 of the Appendix).
Real Estate Valuation.
For this problem, we use historical real estate valuation data collected from
Taiwan (Yeh & Hsu, 2018), which contains 400+ instances.
Here, x ∈ R 6 contains 6 attributes that are highly related to the value of a house, including transaction date, house age, geographic coordinates, etc.
And y ∈ R represents the valuation of a house.
We ﬁrst train a one-hidden-layer feedforward neural network regression model using the ground truth x and y , to serve as our task function f .
Then, we minimize the (cid:96) 2 loss of ˆ y and y , based on a linear encoder and decoder model.
Breast Cancer Detection.
For this problem, we use a well-known breast cancer diagnostic dataset (Street et al., 1993) from Wisconsin, which contains 500+ instances.
Here, x ∈ R 30 containsattributes that measure 10 features of a cell nucleus.
And y is a binary variable that represents a diagnosis result (malignant or benign).
We ﬁrst train a one-hidden-layer feedforward neural network classiﬁcation model using the ground truth x and y , to serve as our task function f .
Then we aim to minimize the cross-entropy loss of ˆ y and y , with encoder and decoder both being one-hidden-layer feedforward neural networks.
Fig.
5 shows the evaluation results.
For both problems, we can see our task-aware approach nearly always outperforms the benchmark approaches on overall task loss
L under different LDP budgets, which demonstrates the effectiveness of our proposed solution.
The maximum improvements against the task-agnostic and privacy-agnostic approach are 21 .
9% ( (cid:15) = 5 ) and 13 .
5% ( (cid:15) = 5 ) for real estate valuation, and are 73 .
0% ( (cid:15) = 20 ) and 45 .
6% ( (cid:15) = 20 ) for breast cancer detection.
C ONCLUSION AND F UTURE W ORK
This paper provides a principled task-aware privacy preservation method to improve the privacyutility trade-off for ML tasks that increasingly operate on rich, multi-dimensional user data.
We gave an analytical near-optimal solution for a general linear encoder-decoder model and MSE task loss, and developed a heuristic learning algorithm for more general nonlinear settings.
Our evaluation showed that our task-aware approach outperforms the benchmark approaches on overall task loss under various LDP budgets.
Our future work will focus on the analysis of the task-aware privacy preservation problem for approximate LDP and other LDP mechanisms as well as multi-task learning.
A A PPENDIX A. P ROOF OF P ROPOSITION
Proof.
We have
[Equation]
And we can verify that D
= E (cid:62) ( EE (cid:62) + σ 2 w I ) − 1 is a zero point of
[Equation]
[Equation]
Then we plug the expression of D into Eq.18 and get Eq.7.
A. P ROOF OF L EMMA
In this subsection we give the proof of Lemma 1.
Here we treat E as a mapping instead of a matrix, and its inverse mapping, which is also linear, is denoted by E − 1 .
We ﬁrst prove the following lemma.
Lemma 2 (Convex hull after linear transformation) .
E ( S ) is the convex hull of E ( H ) .
Proof.
Since H ⊆ S , we have E ( H ) ⊆
E ( S ) .
And for any v, v (cid:48)
∈ E ( S ) and δ ∈ [0 , 1] , we have δh + (1 − δ ) h
(cid:48) ∈ S , where h ∈ E − 1 ( v ) ∩ S and h (cid:48)
∈
E − 1 ( v (cid:48) )
∩ S .
Thus δv + (1 − δ ) v (cid:48)
=
E ( δh + (1 − δ ) h (cid:48) )
∈ E ( S ) .
Therefore, E ( S ) is a convex set containing E ( H ) .
If E ( S ) is not the convex hull, then we can ﬁnd a convex set B such that E ( H ) ⊆ B ⊂ E ( S ) .
Then we have H ⊆
E − 1 ( B ) ∩ S
⊂ S
(If E − 1 ( B ) ∩ S = S , we will have B = E
( E − 1 ( B ) ∩ S )
=
E ( S ) , which is not true).
Besides, for any h, h (cid:48) ∈
E − 1 ( B ) ∩ S and δ ∈ [0 , 1] , we have δv +(1 − δ ) v (cid:48) ∈ B , where v = E ( h ) and v (cid:48)
=
E ( h (cid:48) ) .
Thus δh + (1 − δ ) h (cid:48)
∈
E − 1 ( δv + (1 − δ ) v (cid:48) )
∩ S ∈ E − 1 ( B ) ∩ S .
Therefore, E − 1 ( B ) ∩ S is a convex set containing H .
This is contradictory to the fact that S is the convex hull of H .
So E ( S ) must be the convex hull of E ( H ) .
Now we can proceed to the proof of Lemma 1.
Proof.
Notice that ∆ 1 g e = max v,
[Equation]
∈ E
[Equation]
H ) (cid:107) v − v
[Equation]
(cid:107) 1 , so our target is to prove
[Equation]
First, since E ( H ) ⊆
E ( S ) , we have max v,v
[Equation]
−
[Equation]
1 ≥ max v,v (cid:48)
∈ E ( H ) (cid:107) v − v (cid:48) (cid:107) 1 .
Next, suppose v 1 , v 2 are the two points in E ( S ) such that the (cid:96) 1 distance between them is the largest.
Since E ( S ) is the convex hull of E ( H ) , we can express v i as ( ∀ i ∈ { 1 , 2 } ): v i = p ( i ) (cid:88)
j = t i,j ˜ v i,j , (21) s.t.
˜ v i,j ∈ E ( H ) , ∀ j ∈ { 1 , 2 , · · · , p ( i ) } , (22) p ( i ) (cid:88)
j = t i,j = 1 , and t i,j ≥ 0 , ∀ j ∈ { 1 , 2 , · · · , p ( i ) } , (23) where p ( i ) is an positive integer, ∀ i ∈ { 1 , 2 } .
For convenience, we let A = max v,v (cid:48) ∈{
˜ v i,j |∀ i,j } (cid:107) v − v (cid:48) (cid:107) 1 .
Clearly, A ≤ max v,v (cid:48)
∈ E ( H ) (cid:107) v − v (cid:48) (cid:107) 1 .
Notice that v 1 − v 2 can be expressed as the linear combination of ˜ v 1 ,j − ˜ v 2 ,q , ∀ j ∈ { 1 , 2 , · · · , p (1) } , q ∈ { 1 , 2 , · · · , p (2) } .
That is, ∃ γ j,q ≥ 0 such that:
[Equation] [Equation]
Then we have
[Equation] [Equation]
[Equation]
So we also have max
[Equation]
v
[Equation]
v (cid:48)
∈ E ( H ) (cid:107) v − v (cid:48) (cid:107) 1 .
Thus ∆
[Equation]
A. P ROOF OF P ROPOSITION
Proof.
We have Tr ( P (cid:62) P ) =
(cid:80) ni
=
λ i , which is a ﬁxed number.
Therefore we focus on maximizing the second trace term in Eq
[Equation]
For any Z ≥ n we have
[Equation]
Thus we have for any orthogonal U :
[Equation]
So E (cid:62) ( EE (cid:62) + σ 2 w I ) − 1 E is a positive semi-deﬁnite matrix with eigen-values
[Equation]
σ
[Equation]
σ
[Equation]
+ σ 2 w ≥ 0 .
Then by Ruhe’s trace inequality (Ruhe, 1970) (a corollary of Von Neumanns trace inequality (Von Neumann, 1937)):
[Equation]
and when V = Q the equality holds.
So we have Eq.9 for V = Q , any Z ≥ n and any orthogonal
U .
A. P ROOF OF P ROPOSITION
In this subsection we give the proof of Proposition 3.
For convenience we let a i
=
r
| σ i | , ∀ i ∈ { 1 , 2 , · · · , n } , and then the hyperellipsoid { v ∈ R n | (cid:80) ni
=1 v 2 i /σ
2 i =
r 2 } can be written as { v ∈
R n | (cid:80) ni
=1 v 2 i /a
i = 1 } , which is the standard expression.
And we also have a 1 ≥ a 2 ≥ · · · ≥ a n .
Before proving Proposition 3, we ﬁrst give two lemmas related to the properties of a hyperellipsoid.
Lemma 3 (Tangent hyperplane of hyperellipsoid) .
Any tangent hyperplane of hyperellipsoid { v ∈ R n | (cid:80) ni
=1 v 2 i /a
i = 1 } can be expressed as:
[Equation]
where u 1 , u 2 , · · · , u n are the coefﬁcients.
Proof.
It can be easily veriﬁed that point ˜ v ∈ R n such that
[Equation]
is located on the hyperellipsoid.
And by adjusting the values of u 1 , u 2 , · · · , u n we can express any point on the hyperellipsoid with Eq.35.
Moreover, the tangent hyperplane for point of tangency ˜ v can be expressed as
[Equation]
Plugging Eq.35 into Eq.36 we get Eq.34.
Lemma 4 (Locus of the vertices of circumscribed orthotope) .
The vertices of any orthotope that circumscribes hyperellipsoid { v ∈ R n | (cid:80) ni
=1 v 2 i /a
i = 1 } is on hypersphere { v ∈ R n | (cid:80) ni
=1 v 2 i = (cid:80) ni
=1 a 2 i } .
Proof.
When n = 2 , the hypersphere reduces to a circle that is well-known as the orthoptic circle of an ellipse (Casey, 1893).
We hereby generalize this result to any n .
A vertex of a circumscribed orthotope can be viewed as the intersection of n tangent hyperplanes.
According to Lemma 3, we can express them as:
[Equation]
where i is the index of the i -th hyperplane.
Here we let (cid:80) nj =1 u 2 i,j = 1 , ∀ j ∈ { 1 , 2 , · · · , n } .
Besides, these hyperplanes are perpendicular to each other, so the coefﬁcients also satisfy (cid:80) ni
=1 u i,j u i,k = 0 , ∀ k (cid:54) =
j .
So if we let Ω ∈ R n × n be a matrix with u i,j on the i -th row and j -th column, ∀ i, j , then Ω is an orthogonal matrix, and we further have (cid:80) ni
=
u 2 i,j = 1 , ∀ i ∈ { 1 , 2 , · · · , n } .
Thus the considered vertex satisﬁes Eq.37, ∀ i ∈ { 1 , 2 , · · · , n } , which implies it also satisﬁes:
[Equation]
For the left hand side, we have
[Equation]
( n (cid:88) i
[Equation]
[Equation]
(cid:88) j (cid:54) =
p ( n (cid:88) i =
u
i
[Equation]
j
[Equation]
k )
[Equation]
= n (cid:88)
j = v 2 j .
And for the right hand side,
[Equation]
Thus the vertex is on hypersphere { v ∈ R n | (cid:80) ni
=1 v 2 i = (cid:80) ni
=1 a 2 i } .
Since the equation of a centered hypersphere after rotation remains unchanged, we have the following corollary.
Corollary 1 (Locus of the vertices of circumscribed orthotope after rotation) .
The vertices of any orthotope that circumscribes hyperellipsoid { v ∈ R n | (cid:80) ni
=1 v 2 i /a
i = 1 } after rotation is on hypersphere { v ∈ R n | (cid:80) ni
=1 v 2 i = (cid:80) ni
=1 a 2 i } .
Now we can proceed to the proof of Proposition 3.
Proof.
We ﬁrst consider the case when n is a power of 2 .
For the rotated hyperellipsoid, we consider whether there’s any point ˜ v ∈ R n s.t.
(cid:107) ˜
v (cid:107) 1 ≥
[Equation]
nj =1 a 2 j .
Suppose we couldn’t ﬁnd such a point.
Then consider any tangent hyperplane whose normal vector has the following form: ( u 1 , u 2 , · · · , u n ) , s.t.
u
i = ± 1 , ∀ i ∈ { 1 , 2 , · · · , n } .
It can be expressed as:
[Equation]
where W : ( u 1 , u 2 , · · · , u n ) (cid:55)→ R maps ( u 1 , u 2 , · · · , u n ) to a corresponding constant.
And we have W ( u 1 , u 2 , · · · , u n ) <
(cid:113)(cid:80) nj =1 a 2 j .
Since n is a power of 2 , we can ﬁnd a Hadamard matrix Ω in R n × n whose elements are either 1 or -1, such that ΩΩ (cid:62)
=
nI .
We let u i,j be the element of R on the i -th row and j -th column, ∀ i, j , and consider the intersection of the following n tangent hyperplanes:
[Equation]
whose intersection point must satisfy:
[Equation]
Similar to the proof of Lemma 4, we can easily prove the left hand side equals n (cid:80) nj =1 v 2 j , and the right hand side is strictly less than n (cid:80) nj =1 a 2 j .
Thus the intersection point doesn’t locate on the hypersphere { v ∈ R n | (cid:80) ni
=1 v 2 i = (cid:80) ni
=1 a 2 i } .
But since the considered n hyperplanes are also the surfaces of a circumscribed orthotope, the intersection point is hence a vertex and must be on the hypersphere { v ∈ R n | (cid:80) ni
=1 v 2 i = (cid:80) ni
=1 a 2 i } , according to Corollary 1.
This contradiction means that, the assumption that we cannot ﬁnd a point ˜ v
∈ R n s.t.
(cid:107) ˜
v (cid:107) 1 ≥ (cid:113)(cid:80) nj =1 a 2 j is false.
Thus the point ˜ v ∈ R n s.t.
(cid:107) ˜
v (cid:107) 1 ≥ (cid:113) (cid:80) n j =1 a 2 j exists, and the (cid:96) 1 distance between ˜ v and − ˜ v (which both lie on the hyperepllisoid) is 2 (cid:113)(cid:80) nj
=1 a 2 j .
Thus we have ∆ 1 g e ≥ 2 (cid:113)(cid:80) nj =1 a 2 j .
For U
=
I , which means we don’t actually rotate the ellipsoid { v ∈ R n | (cid:80) ni
=1 v 2 i /a
i = 1 } , we have for any point on the ellipsoid
[Equation]
according to the Cauchy-Schwartz inequality.
This implies any point v on the hyperellipsoid has (cid:107) v (cid:107) 1 ≤ (cid:113)(cid:80) nj =1 a 2 j .
So ∆ 1 g e ≤ 2 (cid:113)(cid:80) nj =1 a 2 j .
Combined with the result in the above paragraph we know ∆ 1 g e =
[Equation]
for U
= I .
Thus the proposition is proved for n being a power of 2 .
For other n ’s, we can treat the considered hyperellipsoid as a degenerated hyperellipsoid in space R ˜ n , where ˜ n is the smallest power of 2 such that ˜ n > n .
This implies the proposition still holds.
Therefore, the proposition is true for any n .
A. P ROOF OF P ROPOSITION
Proof.
First, to preserve (cid:15) -LDP with Laplace mechanism, the minimum σ 2 w required is:
[Equation]
based on Proposition 3 and constraint (cid:80) ni =1 σ 2 i = M .
Next we need to determine σ 21 , · · · , σ 2 n .
The considered problem is an optimization problem which aims at minimizing L in Eq.9 under constraint (cid:80) ni
=1 σ 2 i
=
M and σ 2 i ≥
(here we view σ 2 i instead of σ i as the decision variable).
Note that though in Eq.9 we also have σ 21 ≥ σ 2 n , we don’t need to explicitly consider this constraint, because minimizing L will implicitly guarantee that, according to the rearrangement inequality.
This problem can be solved by KarushKuhnTucker (KKT) approach, with the following Lagrangian function:
[Equation]
where α 1 , · · · , α n and β are Lagrangian multipliers.
We know that the solution will automatically guarantee σ 21 ≥ · · · ≥ σ 2 n , so we can safely assume there exists Z (cid:48) ≤ n such that α i = 0 for i ≤ Z (cid:48) , and α i > 0 for i > Z (cid:48) .
Then for i > Z (cid:48)
we have σ 2 i = 0 , and for i ≤ Z (cid:48)
we have
[Equation]
Combining with (cid:80) Z (cid:48) i =1 σ 2 i
=
M and Eq.46 we eventually get Eq.11.
Enforcing σ 2 Z (cid:48) >
we get
Eq.12.
And plugging Eq.11 into Eq.9 we get Eq.13.
A. P ROOF OF T HEOREM
In this subsection we give the proof of Theorem 1.
We start with two deﬁnitions:
L ∗ ( ∂S ; P, (cid:15) ) denotes the optimal loss for any boundary ∂S , task matrix P that preserves (cid:15) -LDP with Laplace mechanism; R ( r )
= { h ∈ R n |(cid:107) h (cid:107)
=
r 2 } is the centered hypersphere with radius r .
Next we give the following lemma.
Lemma 5 (Invariance of the optimal loss after scaling) .
[Equation]
where ρ > 0 is a scalar and ρ ( S )
= { ρh | h ∈ S } .
Proof.
We only need to consider ﬁxed task matrix P .
For any encoder E , decoder D and noise variance σ 2 w , if they preserve (cid:15) -LDP with Laplace mechanism for boundary ∂S , then we have
[Equation]
50) where
[Equation]
Thus we know encoder E , decoder D and noise variance σ 2 w also preserve ρ(cid:15) -LDP with Laplace mechanism for boundary ∂ρ ( S ) .
And the reverse also holds true.
So we must have L ∗ ( ∂S ; P, (cid:15) )
=
L ∗ ( ∂ρ ( S ); P, ρ(cid:15) ) .
Now we can proceed to the proof of Theorem 1.
Proof.
We ﬁrst construct a distribution D h (cid:48) , s.t. points drawn from D h (cid:48) are uniformly distributed on
R (2 n − 1 ) .
Then h (cid:48) ∼ D h (cid:48) satisﬁes Σ h (cid:48) h (cid:48)
=
I .
And we also have ∂S (cid:48)
=
R (2 n − 1 ) , where S (cid:48) is the convex hull of H (cid:48)
( H (cid:48) is the domain of h (cid:48) ∼ D h (cid:48) ).
According to Proposition 4, for h (cid:48) ∼ D h (cid:48) we have
[Equation]
We next consider scalar ρ = 2 n − 1 /r min and the optimal loss L ∗ ( ∂ρ ( S ); P, ρ(cid:15) ) .
For any encoder E , decoder D and noise variance σ 2 w , if they preserve ρ(cid:15) -LDP to boundary ∂ρ ( S ) , then they preserve at least ρ(cid:15) -LDP to boundary R (2 n − 1 ) , since R (2 n − 1 ) ⊂ ρ ( S ) .
Thus we have L ∗ ( ∂ρ ( S ); P, ρ(cid:15) ) ≥
L ∗ ( R (2 n − 1 ); P, ρ(cid:15) ) .
This further implies:
[Equation]
So the lower bound of Theorem 1 is proved.
The upper bound can be proved in the same way.
A. B ENCHMARKS A.7. T ASK LOSS FOR TASK - AGNOSTIC APPROACH
One can obtain the resultant optimal L for the task-agnostic approach by letting E =
L and using
Eq.7 as stated in Proposition 1.
It is worth noting that the associated decoder D = L (cid:62) ( LL (cid:62) + σ 2 w I ) − 1 is not an identity matrix in general.
Corollary 2 (Optimal L for the task-agnostic approach that preserves (cid:15) -LDP) .
For the task-agnostic approach, the optimal L that preserves (cid:15) -LDP is
[Equation]
where σ 2 w = 2(∆ 1 g e ) 2 /(cid:15) 2 with g e ( x ) = x .
A.7. T ASK LOSS FOR PRIVACY - AGNOSTIC APPROACH
Through similar analysis as Proposition 2, one can obtain the resultant optimal L for the privacyagnostic approach, which has a pre-determined Z ≤ n .
Corollary 3 (Optimal L for the privacy-agnostic approach that preserves (cid:15) -LDP) .
For the privacyagnostic approach with a pre-determined Z ≤ n , the optimal L that preserves (cid:15) -LDP is
[Equation]
(55) where σ 2 w = 2(∆ 1 g e ) 2 /(cid:15) 2 .
Proof.
For privacy-agnostic approach, we have Z ≤ n .
Since for encoder we need to select the top- Z principal components, we have V = Q .
Similar to the proof of Proposition 2, for a given
Z ≤ n , we have
[Equation]
and through similar derivations we eventually get
[Equation]
Combined with Eq.
7 we get Eq.
A.7. T ASK LOSS FOR BENCHMARK APPROACHES WHEN L = I AND A SSUMPTION 1 HOLDS
When L = I and Assumption 1 holds, for the task-agnostic approach, according to Eq.
54, we have
[Equation]
For the privacy-agnostic approach, according to Eq.
55 and Eq. 10, and assuming we have equal σ i ’s and minimum δ 1 g e , we get
[Equation]
where the value of Z is pre-determined.
A.7. B ENCHMARK ALGORITHMS UNDER GENERAL SETTINGS
For the privacy-agnostic approach, we ﬁrst train the encoder and decoder without considering privacy preservation by updating θ e and θ d with −∇ θ e L and −∇ θ d L , respectively.
Next, we ﬁx encoder parameters θ e and train the decoder with input φ + w (a modiﬁcation of Algorithm 1 line 3-4).
The task-agnostic approach trains the decoder in the same way, but ﬁxes g e to an identity mapping function.
A. C ONFIGURATION AND T RAINING D ETAILS OF THE E VALUATION
[Table]
< 2 HR S
Our evaluation runs on a personal laptop with 2.7 GHz Intel Core I5 processor and 8-GB
MHz DDR3 memory.
Our code is based on Pytorch.
We use the Adam optimizer and learning rate 10 − 3 for all the applications.
The number of samples, train/test split, training epochs, and resulting runtime are summarized in Table 1.
(Note that the evaluation for hourly household power consumption is based on the theoretical solutions, so “training epochs”, which is associated with the gradient-based method, doesn’t apply.)
We also provide all our documented code as supplementary material and will make it publicly-available after the review process .
All three datasets cited in the evaluation are publicly-available from the standard UCI Machine Learning Repository (Dua & Graff, 2017) and anonymized using standard practices.
The individual dataset licenses were not available.
For task function f , we use a one-hidden-layer feedforward neural network with input size n , hidden size 1 .
5 n and output size 1 in both the real estate valuation and breast cancer detection experiments.
The activation function used by the hidden layer and output layer is a Rectiﬁed Linear Unit (ReLU).
In our experiments, we ﬁnd that the chosen network architecture is good enough to yield near-zero loss with ground truth x and y , and to avoid overﬁtting we don’t choose a deep neural network.
For example, we did not see any task improvement using a two layer network etc.
For the encoder/decoder, we use a one-layer neural network (linear model) with input and output size n in the real estate valuation experiment.
For this experiment, a linear encoder/decoder model is already enough to provide good performance (see the next subsection for further details).
We use one-hidden-layer feedforward neural network with input size n , hidden size n and output size n in the breast cancer detection experiment.
The activation functions used by the hidden layer and output layer are a logistic and identity function, respectively.
For the heuristic learning algorithm developed in Section 4.2, we set η = 0 .
2 and η = 0 .
001 in real estate valuation and breast cancer detection experiments,
[Figure] [Figure]
A. A P ROPER Z SHOULD BE D ETERMINED ON A C ASE - BY - CASE B ASIS
As mentioned in Section 4.
[Figure]
a practitioner may need to determine a proper Z on a case-by-case basis for our task-aware approach.
Fig.
6 illustrates the performance of our task-aware approach under different Z ’s for real estate valuation and breast cancer detection experiments.
We can observe that in both two experiments, we obtained the best performance on average when Z = 3 , i.e., n/for real estate valuation and n/ 10 for breast cancer detection
[Figure]
A. N ON -L INEAR E NCODER AND D ECODER M ODEL IN R EAL E STATE V ALUATION E XPERIMENT
Theoretically, increasing the model complexity of the encoder and decoder can potentially provide a better performance.
However, in our real estate valuation experiment we ﬁnd that linear encoder and decoder model is already good enough.
To illustrate this, we adopt one-hidden-layer feedforward neural network architecture for both encoder and decoder, which has input size n , hidden size n and output size n , and the activation functions used by the hidden layer and output layer are ReLU and identity function, respectively.
Then we set η = 0 .
001 and repeat the real estate valuation experiment (with other conﬁgurations being the same).
We observe the performance of our task-aware approach is worse in Fig.
7 than in
Fig.
The explanation is, due to the unnecessary complexity of the encoder and decoder, it is harder to learn good encoder and decoder parameters heuristically.
Thus, we conclude that using a simple encoder and decoder, as also observed for the other datasets, is sufﬁcient for high task accuracy.
A. E XPERIMENT WITH H IGH - DIMENSIONAL D ATA
[Figure]
To illustrate our task-aware approach in Section 4.2 also works well for high-dimensional data, such as image data, we consider a handwritten digit recognition problem with well known MNIST dataset LeCun et al.
Here, x ∈ R 784 represents a 28 × 28 image of handwritten digit.
And y ∈ { 0 , 1 , · · · , 9 } is a discrete variable represents the digit in the image.
We ﬁrst train a linear classiﬁcation model using the ground truth x and y , to serve as our task function f .
Then we aim to minimize the cross-entropy loss of ˆ y and y , with linear encoder and decoder.
We use Z = 10 for both our task-aware approach and the privacy-agnostic approach.
Fig.
5 shows the evaluation result.
Since the task loss L of the task-agnostic approach is much larger than the other two approaches, we put it in a separate sub-ﬁgure on the left.
On the right, we can see our task-aware approach always outperforms the privacy-agnostic approach on overall task loss
L under different LDP budgets, which demonstrates the effectiveness of our proposed solution.
The maximum improvement against the privacy-agnostic approach is 23 .
8% ( (cid:15) = 20 ).