I NFORMATION L AUNDERING FOR M ODEL P RIVACY
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we propose information laundering, a novel framework for enhancing model privacy.
Unlike data privacy that concerns the protection of raw data information, model privacy aims to protect an already-learned model that is to be deployed for public use.
The private model can be obtained from general learning methods, and its deployment means that it will return a deterministic or random response for a given input query.
An information-laundered model consists of probabilistic components that deliberately maneuver the intended input and output for queries to the model, so the model’s adversarial acquisition is less likely.
Under the proposed framework, we develop an information-theoretic principle to quantify the fundamental tradeoffs between model utility and privacy leakage and derive the optimal design.
I NTRODUCTION
An emerging number of applications involve the following user-scenario.
Alice developed a model that takes a speciﬁc query as input and calculates a response as output.
The model is a stochastic black-box that may represent a novel type of ensemble models, a known deep neural network architecture with sophisticated parameter tuning, or a physical law described by stochastic differential equations.
Bob is a user that sends a query to Alice and obtains the corresponding response for his speciﬁc purposes, whether benign or adversarial.
Examples of the above scenario include many recent Machine-Learning-as-a-Service (MLaaS) services (Alabbadi, 2011;
Ribeiro et al., 2015; Xian et al., 2020) and artiﬁcial intelligence chips, where Alice represents a learning service provider, and
Bob represents users.
Suppose that Bob obtains sufﬁcient paired input-output data as generated from Alice’s black-box model, it is conceivable that Bob could treat it as supervised data and reconstruct Alice’s model to some extent.
From the view of Alice, her model may be treated as valuable and private.
As Bob that queries the model may be benign or adversarial, Alice may intend to offer limited utility for the return of enhanced privacy.
The above concern naturally motivates the following problem.
(Q1)
How to enhance the privacy for an already-learned model?
Note that the above problem is not about data privacy, where the typical goal is to prevent adversarial inference of the data information during data transmission or model training.
In contrast, model privacy concerns an alreadyestablished model.
We propose to study a general approach to jointly maneuver the original query’s input and output so that Bob ﬁnds it challenging to guess Alice’s core model.
As illustrated in
Figure 1a, Alice’s model is treated as a transition kernel (or communication channel) that produces ˜
Y conditional on any given ˜ X .
Compared with an honest service Alice would have provided (Figure 1b), the input ˜ X is a maneuvered version of Bob’s original input X ; Moreover, Alice may choose to return a perturbed outcome Y instead of ˜ Y to Bob.
Consequently, the apparent kernel from Bob’s input query X to the output response Y is a cascade of three kernels, denoted by K in Figure 1a.
The above perspective provides a natural and general framework to study model privacy.
Admittedly, if
Alice produces a (nearly) random response, adversaries will ﬁnd it difﬁcult to steal the model, while benign users will ﬁnd it useless.
Consequently, we raise another problem.
(Q2)
How to formulate the model privacy-utility tradeoff, and what is the optimal way of imposing privacy?
To address this question, we formulate a model privacy framework from an informationtheoretic perspective, named information laundering.
We brieﬂy describe the idea below.
The general goal is to jointly design the input and output kernels ( K 1 and K 2 in Figure 1a) that deliberately maneuver the intended input and output for queries to the model so
that 1) the effective kernel ( K
[Figure]
in Figure 1a) for Bob is not too far away from the original kernel ( K ∗ in Figure 1a), and 2) adversarial acquisition of the model becomes difﬁcult.
Alternatively, Alice ‘launders’ the input-output information maximally given a ﬁxed utility loss.
To ﬁnd the optimal way of information laundering, we propose an objective function that involves two components: the ﬁrst being the information shared between X, ˜ X and between ˜ Y , Y , and the second being the average Kullback-Leibler (KL) divergence between the conditional distribution describing K and K ∗ .
Intuitively, the ﬁrst component controls the difﬁculty of guessing K ∗ sandwiched between two artiﬁcial kernels K 1 and K 2 , while the second component ensures that overall utility is maximized under the same privacy constraints.
By optimizing the objective for varying weights between the components, we can quantify the fundamental tradeoffs between model utility and privacy.
1. R ELATED W ORK
We introduce some closely related literature below.
Section 3.3 will incorporate more technical discussions on some related but different frameworks, including information bottleneck, local data privacy, information privacy, and adversarial model attack.
A closely related subject of study is data privacy, which has received extensive attention in recent years due to societal concerns (Voigt & Von dem Bussche, 2017; Evans et al., 2015; Cross & Cavallaro, 2020; Google, 2019; Facebook, 2020).
Data privacy concerns the protection of (usually personal) data information from different perspectives, including lossless cryptography (Yao, 1982; Chaum et al., 1988), randomized data collection (Evﬁmievski et al., 2003; Kasiviswanathan et al., 2011; Ding & Ding, 2020), statistical database query (Dwork & Nissim, 2004; Dwork, 2011), membership inference (Shokri et al., 2017), and Federated learning (Shokri & Shmatikov, 2015; Konevcny et al., 2016; McMahan et al., 2017; Yang et al., 2019; Diao et al., 2020).
A common goal in data privacy is to obfuscate individual-level data values while still enabling population-wide learning.
In contrast, the subject of model privacy focuses on protecting a single learned model ready to deploy.
For example, we want to privatize a classiﬁer to deploy on the cloud for public use, whether the model is previously trained from raw image data or a data-private procedure.
Another closely related subject is the model extraction in (Tram`er et al., 2016; Papernot et al., 2016b), where Bob’s goal is to reconstruct Alice’s model from several queries’ inputs and outputs, knowing what speciﬁc model Alice uses.
For example, suppose that Alice’s model is a generalized linear regression with p features.
In that case, it is likely to be reconstructed using p queries of the expected mean (a known function of Xβ ) by solving equations (Tram`er et al., 2016).
In the supervised classiﬁcation scenario, when only labels are returned to any given input, model extraction could be cast as an active learning problem where the goal is to query most efﬁciently (Chandrasekaran et al., 2018).
Model extraction was also studied in contexts beyond the prediction API, e.g., when an adversary utilizes the gradient information (Milli et al., 2019).
From Alice’s perspective, there exist several solutions to safeguard against model leakage. A warning-based method was developed in (Kesarwani et al., 2018), where Alice continuously monitors the information gain and raise alarms when they become unusual. Another warning method was developed in (Juuti et al., 2019), where the detection of an adversary is based on testing whether the pairwise distances among queried data approximately follow the Gaussian distribution. The work in (Lee et al., 2018) developed a defense strategy for the setting where the target is a classiﬁer, and the adversary queries each class’s probability.
The probabilities are maximally perturbed under the constraint that the most-likely class label) remains the same.
The work in (Orekondy et al., 2019) studied a similar setting but from a different perspective.
The main idea is to perturb the probabilities within an (cid:96) 2 -distance constraint to poison the adversary’s gradient signals.
1. C ONTRIBUTIONS AND O UTLINE
The main contributions of this work are three folds.
First, we develop a novel concept, theory, and method, generally referred to as information laundering , to study model privacy.
Unlike data privacy that concerns the protection of raw data information, model privacy aims to privatize an already-learned model for public use.
To the best of the authors’ knowledge, this work is the ﬁrst framework to study model privacy in a principled manner.
Second, under the developed informationtheoretic framework, we cast the tradeoffs between model privacy and utility as a general optimization problem.
We derive the optimal solution using the calculus of variations and provide extensive discussions on the solution’s insights from different angles.
Third, we develop a concrete algorithm, prove its convergence, and elaborate on some speciﬁc cases.
We also provide some experimental studies to illustrate the concepts, and discuss several future problems at the end.
The paper is organized as follows.
In Section 2, we describe the problem formulation and a general approach to protect the model.
In Section 3, we propose the information laundering method that casts the model privacy-utility tradeoff as an optimization problem and derives a general solution.
In
Section 3.3, we provide some additional discussions of the related frameworks, including information bottleneck, local data privacy, information privacy, and adversarial model attack.
In Section 5, we conclude the paper with some potential future work.
In the Appendix, we provide the proofs of the main results and experimental studies.
F ORMULATION 2. B ACKGROUND
The private model can be obtained from general learning methods, and its deployment means that it will return a response for a given input query.
Suppose that X and Y are the input and output alphabets (data space), respectively.
Deﬁnition 1 (Learned model)
A learned model is a kernel p : X × Y → [0 , 1] , which induces a class of conditional distributions { p ( · | x ) : x ∈ X} .
A model in the above deﬁnition is also referred to as a communication channel in information theory.
A model can be regarded as the input-output (or Alice’s application programming interface, API) offered to Bob.
Examples include a regression/classiﬁcation model that outputs predicted labels, a clustering model that outputs the probabilities of belonging to speciﬁc groups, and a stochastic differential equation system that outputs the likely paths for various inputs variables.
It does not matter where the model comes from since we are only concerned about the privacy of a ﬁxed given model.
The (authentic) model of Alice is denoted by p K ∗ .
An adversary Bob is a user that can access the above model’s API, providing an arbitrary input, X , and obtaining an output, Y .
Bob aims to use as few queries as possible to construct a model that closely matches Alice’s model p K ∗ .
We will formalize the ‘closeness’ using the KL divergence.
What is model privacy?
Our perspective is that privacy is not an intrinsic quantity associated with a model; instead, it is a measure of information that arises from interactions between the model and its queries.
In our context, the interactions are through X (offered by Bob) and Y (offered by Alice).
The key idea of enhancing Alice’s model privacy is to let Alice output noisy predictions ˜ Y for any input X so that Bob cannot easily infer Alice’s original model.
Similarly, Alice may choose to manipulate X as well before passing it through K ∗ .
Alternatively, Alice intends to 1) impose some ambiguity between X, ˜ X , and between Y, ˜ Y , which conceivably will produce response deviating from the original one, and 2) seek the K closest to K ∗ under the same amount of ambiguity imposed.
Motivated by the above concepts, we introduce the following notion.
Deﬁnition 2 (Information-laundered model)
A information-laundered model with respect to a given model K ∗ is a model K that consists of three internal kernels K = K 1 ◦ K ∗ ◦
K 2 (illustrated in Figure 1).
Naturally, the information-laundered model of Alice is denoted by p K .
2. N OTATION
We let p K ∗ ( · | · ) , p K 1 ( · | · ) , p K 2 ( · | · ) , p K ( · | · ) denote the kernels that represent the authentic model, input kernel, output kernel, and the information-laundered model, respectively.
We let p X ( · ) denote the marginal distribution of X .
Similar notation is for p ˜ X ( · ) ,
p ˜ Y ( · ) , and p Y ( · ) .
Note that the p Y implicitly depends on the above conditional distributions.
We use p K 1 ◦ K ∗ ( · | · ) and p K ∗ ◦ K 2 ( · | · ) to denote cascade conditional distributions of ˜ Y | X and Y | ˜ X , respectively.
Throughout the paper, random variables are denoted by capital letters.
Suppose that X ∈ X , ˜ X ∈ ˜ X , ˜
Y ∈ ˜
Y , and Y ∈ Y .
For technical convenience, we will assume that X , ˜ X , ˜ Y , Y are ﬁnite alphabets unless otherwise stated.
We will discuss some special cases when some of them are the same.
Our theoretical results apply to continuous alphabets as well under suitable conditions.
For notational convenience, we write the sum (cid:80) x ∈X u ( x ) as (cid:80) x u ( x ) for any function u .
With a slight abuse of notation, we will use p to denote a distribution, density function, or transition kernel, depending on the context.
I NFORMATION L AUNDERING 3. T HE I NFORMATION L AUNDERING P RINCIPLE
The information laundering method is an optimization problem formulated from the concept of
KL-divergence between the (designed) effective kernel and the original kernel, with constraints of the privacy leakage during the model-data interaction.
In particular, we propose to minimize the following objective function over ( p K 1 , p K 2 ) ,
[Equation]
In the above, K 1 and K 2 are implicitly involved in each additive term of L , and β 1 ≥ 0 , β 2 ≥are constants that determine the utility-privacy tradeoffs.
Small values of β 1 and β 2 (e.g., zeros) pushes the K to be the same as K ∗ , while large values of β 1 pushes ˜ X to be nearly-independent with X (similarly for β 2 ).
It is worth mentioning that the principle presumes a given alphabet (or representation) for ˜ X and ˜ Y .
The variables to optimize over is the transition laws X → ˜ X and ˜
Y → Y .
The objective in (1) may be interpreted in the following way.
On the one hand, Alice aims to develop an effective system of K that resembles the authentic one K ∗ for the utility of benign users.
This goal is realized through the ﬁrst term in (1), which is the average divergence between two system dynamics.
On the other hand, Alice’s model privacy leakage is through interactions with Bob, which in turn is through the input X (publicly offered by Bob) and output Y (publicly offered by Alice).
Thus, we control the information propagated through both the input-interfaces and out-interfaces, leading to the second and third terms in (1).
We note that the above objective function may also be formulated in alternative ways from different perspectives.
For example, we may change the third term to be β
I ( Y ; ˜ Y | X, ˜ X ) , interpreted in the way that Alice will design K 1 ﬁrst, and then design K 2 conditional on K 1 .
Likewise, we may change the second term to be β 1 I ( X ; ˜ X | ˜
Y , Y ) , meaning that K 2 is designed ﬁrst.
From
Bob’s perspective, we may also change the third term to β 2 I ( Y ; ˜ Y | X ) , interpreted for the scenario where Bob conditions on the input information X during model extraction.
Additionally, from the perspective of adaptive interactions between Alice and Bob, we may consider p X as part of the optimization and solve the max-min problem max p X min p K 1 ,p K 2 L ( p K 1 , p K 2 ) .
We leave these alternative views for future work. [Table] [Table] [Table]
[Table] [Equation] where p ( t +1) K 1 ◦ K ∗ , p ( t ) K ∗ ◦ K 2 , and p ( t +1 ,t ) K denote the kernels cascaded from ( p ( t +1) K 1 , p K ∗ ) , ( p K ∗ , p ( t ) K 2 ) , and 3. T HE OPTIMAL SOLUTION
We derive the solution that corresponds to the optimal tradeoffs and point out some nice interpretations of the results.
The derivation is nontrivial as the functional involves several nonlinear terms of the variables to optimize over.
Note that for the notation deﬁned in Subsection 2.2, only p X and p K ∗ are known and others are (implicitly) determined by p K 1 , p K 2 .
Theorem 1 The optimal solution of (1) satisﬁes the following equations.
[Equation]
where κ x and τ ˜ y are normalizing constants implicitly deﬁned so that the conditional density function integrates to one.
Note that the distributions of ˜ X , ˜ Y , Y , and ˜ Y , Y | ˜ X , implicitly depend on p K 1 and p K 2 .
The above theorem naturally leads to an iterative algorithm to estimate the unknown conditional distributions p K 1 and p K 2 .
In particular, we may alternate Equations (2) and (3) to obtain p ( (cid:96) )
K 1 (˜ x | x ) , p ( (cid:96) )
K 2 ( y | ˜ y ) from p ( (cid:96) − 1) K(˜ x | x ) , p ( (cid:96) − 1) K( y | ˜ y ) at step (cid:96) = 1 , 2 , . . .
with random initial values at (cid:96) = 0 .
The pseudocode is summarized in Algorithm 1.
In the next theorem, we show that the convergence of the algorithm.
The sketch of the proof is described below.
First, we treat the original objective L as another functional J of four independent variables, p K 1 , p K 2 , h 1 , h 2 , evaluated at h 1 = p ˜ X and h 2 = p Y .
Using a technique historically used to prove the convergence of the Blahut-Arimoto algorithm for calculating rate-distortion functions in information theory, we show that J ≥ L .
We also show that L is convex in each variable so that the objective function is non-increasing in each alternation between four equations.
Since L ≥ 0 , the convergence is implied by the monotone convergence theorem.
Theorem 2 Algorithm 1 converges to a minimum that satisﬁes equations (2) and (3).
Note that the minimum is possibly a local minimum.
We will later show the convergence to a global minimum in a particular case.
Next, we provide interpretations of the parameters and how they affect the ﬁnal solution.
A large β 1 in the optimization of (1) indicates a higher weight on the term
I ( X ; ˜ X ) .
In the extreme case when β 1 = ∞ , minimizing I ( X ; ˜ X ) is attained when ˜ X is independent with X .
Consequently, the effective model of Alice produces a ﬁxed distribution of responses for whatever Bob queries.
The above observation is in line with the derived equation (2), which will become p K 1 (˜ x | x ) ≈ κ x p ˜ X
(˜ x ) (and thus κ x ≈ 1 ) for a large β 1 > 0 .
Similar to the effect of β 1 , a larger β 2 imposes more independence between ˜ Y and Y .
In the case β 2 = ∞ , Alice may pass the input to her internal model K ∗ but output random results.
This can be seen from either the Formulation (1) or Equation (3).
For the ﬁrst expectation in equation (2), the term may be interpreted as the average likelihood ratio of y conditional on ˜ x against x .
From Equation (2), it is more likely to transit from x to ˜ x in the presence of a larger likelihood ratio.
This result is intuitively appealing because a large likelihood ratio indicates that x may be replaced with ˜ x without harming the overall likelihood of observing Y .
3. F URTHER D ISCUSSIONS ON R ELATED W ORK
Information Bottleneck: extracting instead of privatizing information .
The information bottleneck method (Tishby et al., 2000) is an information-theoretic approach that aims to ﬁnd a parsimonious representation of raw data X , denoted by ˜ X , that contains the maximal information of a variable Y of interest.
The method has been applied to various learning problems such as clustering, dimension reduction, and theoretical interpretations for deep neural networks (Tishby & Zaslavsky, 2015).
Formally, the information bottleneck method assumes the Markov chain
[Equation]
and seeks the the optimal transition law from X to ˜ X by minimizing the functional
[Equation]
with β being is a tuning parameter that controls the tradeoffs between compression rate (the ﬁrst term) and amount of meaningful information (second term).
The alphabet of the above ˜ X needs to be pre-selected and often much smaller in size compared to the alphabet of X to meet the purpose of compression.
In other words, the information that X provides about Y is passed through a ‘bottleneck’ formed by the parsimonious alphabet of ˜ X .
A similarity between the information bottleneck method and the particular case of information laundering in Subsection A.2 is that they both optimize a functional of the transition law of X → ˜ X .
Nevertheless, their objective and formulation are fundamentally different.
First, the objective of information bottleneck is to compress the representation while preserving meaningful information, under the assumption of (4); Our goal is to distort X while minimizing the gap between the (random) functionality of X → Y , under a different Markov chain X
→ ˜ X → Y .
Data Privacy and Information Privacy: protecting data instead of a model .
The tradeoffs between individual-level data privacy and population-level learning utility have motivated active research on what is generally referred to as ‘local data privacy’ across multiple ﬁelds such as data mining (Evﬁmievski et al., 2003), security (Kasiviswanathan et al., 2011), statistics (Duchi et al., 2018), and information theory (du Pin Calmon & Fawaz, 2012;
Sun et al., 2016).
For example, a popular framework is the local differential privacy (Evﬁmievski et al., 2003; Dwork et al., 2006;
Kasiviswanathan et al., 2011), where raw data X is suitably randomized (often by adding Laplace noises) into Y so that the ratio of conditional densities
[Equation]
for any y, x 1 , x 2 ∈ X , where α > 0 is a pre-determined value that quantities the level of privacy.
In the above, X and Y represent the private data and the processed data to be collected or publicly [Figure] distributed.
The requirement (5) guarantees that the KL-divergence between p Y | x 1 and p Y | x 2 is universally upper-bounded by a known function of α (see, e.g., Duchi et al., 2018), meaning that x 1 and x 2 are barely distinguishable from the observed y .
Note that the above comparison is made between two conditional distributions, while the comparison in information laundering (recall the ﬁrst term in (1)) is made between two transition kernels.
The local differential privacy framework does not need to specify a probability space for X , since the notion of data privacy is only built on conditional distributions.
Another related framework is the information privacy (du Pin Calmon & Fawaz, 2012), which assumes a probabilistic structure on X and a Markov chain X
→ ˜ Y → Y .
In the above chain, X is the private raw data, ˜ Y is a set of measurement points to transmit or publicize, and Y is a distortion of ˜ Y that is eventually collected or publicized.
We deliberately chose the above notation of X, ˜ Y , Y , so that the Markov chain appears similar to the special case of information laundering in Subsection 4.
Nevertheless, the objective of information privacy is to minimize I ( X ; Y ) over p Y | ˜
Y subject to utility constraints, assuming that the joint distribution of X, ˜ Y is known.
In other words, the goal is to maximally hide the information of X .
In the context of information laundering, the system input X is provided by users and is known.
Adversarial Model Attack: rendering harm instead of utility to a model .
The adversarial model attack literature concerns the adversarial use of specially crafted input data to cause a machine learning model, often a deep neural network, to malfunction (Papernot et al., 2016a; Narodytska & Kasiviswanathan, 2017; Papernot et al., 2017).
For example, an adversarial may inject noise into an image so that a well-trained classiﬁer produces an unexpected output, even if the noise is perceptually close to the original one.
A standard attack is the so-called (Adaptive) Black-Box
Attack against classiﬁers hosted by a model owner, e.g., Amazon and Google (Rosenberg et al., 2017; Chakraborty et al., 2018).
For a target model K ∗ , a black-box adversary has no information about the training process of K ∗ but can access the target model through query-response interfaces.
The adversary issues (adaptive) queries and record the returned labels to train a local surrogate model.
The surrogate model is then used to craft adversarial samples to maximize the target model’s prediction error.
If we let X, ˜ X, Y denote the model input, adversarially perturbed input, and output, respectively, then we may draw a similarity between adversarial model attack and the particular case of information laundering in Subsection A.2 since they both look for the law X
→ ˜ X .
The main difference is in the objective.
While the model attack aims to ﬁnd an input domain that maximally distorts the model, information laundering aims to maintain a small model discrepancy.
Under our notation, a possible formulation for the model attack is to seek max p ˜ X
| X E X ∼ p
X D KL ( p K ∗ ( · | X ) , p K ∗ ( · | ˜ X ))
under a constraint on p ˜ X | X .
S PECIAL C ASE : I NFORMATION LAUNDERING OF THE OUTPUT ( Y ) ONLY
Two special cases of an information-laundered system are illustrated in Figure 2.
Here, we elaborate on one case and include the other special case in the Appendix.
Suppose that K 1 is an identity map and let β 1 = 0 .
In other words, we alter the output data only (Figure 2b).
Furthermore, suppose that for each given ˜ x , the conditional distribution p K ( · | ˜ x ) assigns all the mass at ˜ y .
In other words,
K ∗ reduces to a deterministic function mapping from each ˜ x ∈ X to a unique ˜ y ∈ Y , which is denoted by ˜ y
=
f
(˜ x ) .
For example, Alice’s model is a classiﬁer that takes input features and returns hard-thresholded classiﬁcation labels.
Then the optimization problem (1) reduces to minimizing
[Equation]
Corollary 1 The solution to the optimization problem (6) satisﬁes
[Equation]
where τ ˜ y is a normalizing constant.
In particular, if K ∗ is deterministic, equation (7) becomes
[Equation]
To exemplify the proposed methodology, we study a speciﬁc case with the following conditions.
1) X may be large or continuously-valued, ˜ Y
=
Y is a moderately-large alphabet, 2) ˜
Y
=
Y so that ˜ Y and Y are in the same space, 3) K ∗ is deterministic.
Under the above scenario, we can apply Algorithm 1 and Corollary 1 to obtain a simpliﬁed procedure below (denoted by OIL-Y).
At each time step t = 1 , 2 , . . . , , for each ˜ y, y ∈ Y , we calculate
[Equation]
y β 2 p ( t ) K 2 ( y | y ) (cid:27) , where τ − 1 ˜ y = (cid:88) y p ( t ) Y ( y ) exp (cid:26) 1 y =˜ y β 2 p ( t ) K 2 ( y | y ) (cid:27) , p ( t +1)
Y ( y ) = r ˜ y p ( t +1) K 2 ( y | ˜ y ) , where r ˜ y = (cid:88)
x : f ( x )=˜
y
p X ( x ) .
Note that the above r ˜ y is the probability that Alice observes ˜ y as an output of K ∗ ∗ if Bob inputs
X ∈ p X .
Therefore, r ˜ y can be easily estimated to be the empirical frequency of observing ˜
y
at the end of Alice.
Note that since Y is a ﬁnite alphabet, we can use a matrix representation for easy implementation.
In particular, we represent the elements of Y by 1 , . .
, a , where a = card ( Y ) .
We then represent p K 2 by P ∈ R a × a , and p Y by q ∈ R a , where P y, ˜ y
= p K 2 ( y | ˜ y ) .
Such a representation will lead to a matrix form of the above procedure, summarized in Algorithm 2. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
← diag ( P ( t +1) ) · exp { 1 / ( β 2 diag ( P ( t ) ) } , where the operations are element-wise 6:
Scale each column (conditional distribution) of P ( t +1) so that it sums to one 7:
Calculate q ( t +
) =
P ( t +1) × r 8: end for 9: Return p ( T ) K 2 that is represented by P ( T ) .
Moreover, we proved the convergence to the global minimum for the alternating equations in the above scenario.
The same technique can be emulated to show a similar result when we employ K(instead of K 2 ) only.
The result is summarized in Theorem 3.
Theorem 3 Suppose that K ∗ is deterministic.
The alternating equation (9), or its matrix form in
Algorithm 1, converges to a global minimum of the problem (6).
C ONCLUSION AND F URTHER R EMARKS
Despite extensive studies on data privacy, little has been studied for enhancing model privacy.
Motivated by the emerging concern of model privacy from the perspective of machine learning service providers, we develop a plug-and-play methodology “information laundering” to enhance the privacy of any given model of interest.
The information laundering is model-agnostic as it applies to general API models, including classiﬁcation and regression models that output labels/probabilities, and black-box models generating probabilistic outputs.
We believe that the developed principles, theories, and insights can lead to new resilient machine learning algorithms and services.
An interesting problem is to integrate information laundering with various application scenarios on a case-by-case basis. Another problem is to adapt the developed principle to speciﬁc API models to incorporate side information (also mentioned in Section 3.1). Taking into account the potential side information available to an adversary can be essential in some pathetic situations. Consider an example where the adversary knows that the output associated with extremely large input is a ﬁxed constant.
Then, the adversary may strategically send the same input with extreme values to accurately identify that constant.
An information-laundered model may be vulnerable in the above scenario since the current information laundering concerns the average over data distributions.
Theoretically, there are three open problems left from the work that deserves further research.
First, how does the imposed constraint of mutual information affect the rate of convergence from the adversary perspective for speciﬁc models (e.g., generalized linear models, decision trees, neural networks)?
Second, we focused on ﬁnite alphabets for technical convenience.
How to emulate the current methods for continuously-valued alphabets (especially with large dimensions)?
Third, what would be the relative importance of laundering X versus Y , and will this depend on speciﬁc learning problems?
Appendix .
In Appendices A.1 and A.2, we ﬁrst include two particular cases of information laundering that were not included in the main part of the paper.
We then include the proofs of the theorems in Appendix A.3.
Experimental results are included in Appendices A.4, A.5, A.6, and A. to demonstrate the algorithm convergence, model privacy-utility tradeoffs, how tradeoff parameters and unbalanced samples may inﬂuence the optimized information laundering, and how the information laundering effectively mitigates adversarial attacks.
A A PPENDIX A. S PECIAL CASES : DETERMINISTIC MODEL K ∗
In this case, Theorem 1 implies the following corollary.
We will use this result in later sections.
Corollary
The optimal solution of (1) satisﬁes the following equations.
[Equation]
where κ x and τ ˜ y are normalizing constants implicitly deﬁned so that the conditional density function integrates to one.
A. I NFORMATION LAUNDERING OF THE INPUT ( X ) ONLY
Suppose that K 2 is an identity map and let β 2 =
so that we only maneuver the input data (Figure 2a).
Then the optimization problem (1) reduces to minimizing
[Equation]
Corollary
The optimal solution of (10) satisﬁes the following equations.
[Equation]
where κ x is an implicitly deﬁned normalizing constant.
In particular, if K ∗ is deterministic, equation (11) becomes
[Equation]
As we can see from Corollaries 1 and 3, for a deterministic K ∗ (represented by f ), the simpliﬁed equation of (8) is similar to that of (12).
The subtle difference that one has a sum while the other does not is because f may not be a one-to-one mapping.
A. P ROOFS
Proof 1 (Proof of Theorem 1) Introducing Lagrange multipliers, λ 1 ( x ) for the normalization of the conditional distributions p K 1 ( · | x ) at each x , λ 2 (˜ y ) for the normalization of the conditional distributions p K 2 ( · | ˜ y ) at each ˜ y .
The Lagrangian of (1) can be written as
L
[Equation]
(13) up to an additive constant c that is determined by the known p X and p K ∗ .
It can be veriﬁed that
[Equation]
Using (14)-(17)
[Equation]
for a given x and ˜ x
[Equation]
we calculate the derivatives of each term in (13) with respect to p K 1 (˜ x | x ) to be
[Equation]
Taking equations (18)-(22) into (13), we obtain the ﬁrst-order equation
[Equation]
23) where ˜ λ ( x ) =
λ 1 ( x ) /p
X ( x ) − β 2 .
Rearranging the terms in Equation (23)
[Equation]
we obtain log p K 1 (˜ x | x )
[Equation]
which implies Equation (2)
[Equation]
Similarly, taking derivatives with respect to p K 2 ( y | ˜ y ) for given ˜ y and y , it can be veriﬁed that
[Equation]
Letting Equation (24) be zero and rearranging it, we obtain Equation (3).
Proof 2 (Proof of Theorem 2)
We deﬁne the following functional of four variables:
[Equation]
We will use the following known result (Cover, 1999, Lemma 10.8.1).
Suppose that X and Y have a joint distribution with density p XY , and the marginal densities are p X , p Y , respectively.
Then a density function r Y of y that minimizes the KL-divergence D ( p XY , p X r Y ) is the marginal distribution p Y .
This result implies that minimizing the objective function in (1) can be written as a quadruple minimization
[Equation]
It can be veriﬁed from (23) and its preceding identities that
[Equation]
Thus
[Equation]
J ( p K 1 , p K 2 , h 1 , h 2 ) is convex in each of the variables.
We begin with a choice of initial p K 2 , h 1 , h
[Equation]
and calculate the p K 1 that minimizes the objective.
Using the method of Lagrange multipliers for this minimization (in a way similar to (13)), we obtain the solution of p K 1 shown in the ﬁrst equation of Line 3, Algorithm 1.
Similarly, we obtain the second equation in Algorithm 1.
For the conditional distributions p K 1 and p K 2 , we then calculate the marginal distributions
h 1 (of ˜ x ) that minimizes (25).
Note that the terms of (25) involving hmay be rewritten as
[Equation]
which, by the aforementioned lemma
[Equation]
is minimized by the third equation of Line 3, Algorithm 1.
Similar arguments apply for h 2 .
Consequently, each iteration step in Algorithm 1 reduces J .
By the non-negativeness of KL-divergence, J + c ≥ L ≥ 0 , where L is in (1) and c is introduced in (13).
Therefore, J has a lower bound, and the algorithm will converge to a minimum.
Note that J ( p K 1 , p K 2 , h 1 , h 2 ) is convex in each of the variables independently but not in the variables’ product space.
The current proof does not imply the convergence to a global minimum.
Proof 3 (Proof of Theorem 3) Similar to the technique used in the above proof of Theorem 2, we cast the optimization problem in (6) as a double minimization with respect to ( p K 2 , h 2 ) , J
[Equation]
We only need to check that J is strongly convex in its arguments.
Direct calculations show that
[Equation]
The above equations indicate that the determinant of the Hessian satisﬁes
[Equation]
which further implies the convexity of J in the product space of p K 2 and
h
[Equation]
A. V ISUALIZATION OF A LGORITHM
We provide a toy example to visualize Algorithm 2.
In the simulation, we choose an alphabet of size 100 , and p ˜ Y as described by r ∈ [0 , 1] a is uniform-randomly generated from the probability simplex.
We independently replicate the experiment 50 times, each time running Algorithm 2 foriterations, and calculate the average of the following results.
First, we record (cid:107) P ( t +1) − P ( t ) (cid:107) 1 /a at each iteration t , which traces the convergence of the estimated transition probabilities.
Second, we record the ﬁnal transition probability matrix into a heat-map where P y, ˜
y means the estimated p K 2 ( y | ˜ y ) .
The experiments are performed for β = 100 , 10 , 1 , corresponding to columns 1-3.
The plots indicate the convergence of the algorithm, though the rate of convergence depends on β .
They also imply the expected result that a small β induces an identity transition while a large β induces ˜
Y that is nearly independent with Y .
A. D ATA STUDY : N EWS T EXT CLASSIFICATION
In this experimental study, we use the ‘20-newsgroups’ dataset provided by scikit-learn opensource library (Scikit-learn, 2020d), which comprises news texts on various topics.
The experiment is intended to illustrate the utility-privacy tradeoff and the optimality of our proposed solution compared with other methods.
For better visualization we pick up the ﬁrst four topics (in alphabetic order), which are ‘alt.atheism’, ‘comp.graphics’, ‘comp.os.ms-windows.misc’, ‘comp.sys.ibm.pc.hardware’.
Suppose that the service Alice provides is to perform text-based clustering, which takes text data as input and returns one of the four categories (denoted by 0 , 1 , 2 , 3 ) as output.
The texts are transformed into vectors of numerical values using the technique of term frequency-inverse document frequency (TF-IDF) (Rajaraman & Ullman, 2011).
In the transformation, metadata such as headers, signature blocks, and quotation blocks are removed.
To evaluate the out-sample utility, we split the data into two parts using the default option provided in (Scikit-learn, 2020d), which results in a training part ( 2245 samples, 49914 features) and a testing part (samples, 49914 features).
The above split between the training and testing is based upon messages posted before and after a speciﬁc date.
Alice trains a classiﬁer using the Naive Bayes method and records the frequency of observing each category [0 . 220 . 270 . 210 . 30] ( r in Algorithm 2).
Then, Alice runs the OIL-Y Algorithm (under a given β 2 ) to obtain the transition probability matrix P ∈ [0 , 1] 4 × 4 .
In other words, the effective system provided by Alice is the cascade of the learned classiﬁer, and P determines the Markov
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
rr
[Figure]
r ( i n l o g s c a l e ) (a1)
[Figure]
Y (a2)
[Figure] [Figure]
(b1)
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
rr
[Figure]
( i n l o g
[Figure]
c a l e ) (c1)
[Figure]
(c2)
[Figure]
transition.
Alice’s resulting out-sample performance from the testing data is recorded in Figure 4a, where we considered different β ’s summarized in Table 1.
As we expected, a larger value of β 2 cuts off more information propagated from ˜ Y to Y , resulting in a degraded out-sample performance of
Alice’s effective system.
We also visualize the model privacy-utility tradeoff by the following procedure.
First, we approximate the utility that quantiﬁes the useful information conveyed by Alice.
With Alice’s trained model and the optimally laundered Y (from training data), we retrain another Naive Bayes classiﬁer and generate predictions on the testing data, denoted by y pred K .
Meanwhile, we apply Alice’s authentic model to generate predictions on the testing data, denoted by y pred K ∗ .
We approximate the model utility as the accuracy measure between y pred K and y pred K ∗ .
The model utility can be approximated by other measures.
We also considered retraining methods such as tree-based classiﬁers and average F1score in computing the model utility, and the results are consistent in the data experiments.
Second, we approximate the privacy leakage as Alice’s prediction accuracy on the testing data.
Intuitively speaking, for a given utility, larger out-sample prediction accuracy indicates less information laundered, indicating a higher privacy leakage of Alice’s internal model.
We plot the model leakage against utility obtained from our proposed solution in Figure 4b.
For comparison, we considered a benchmark method described below.
The conditional probability mass function p K 2 ( · | ˜ y ) given each ˜ y is independently drawn from a Dirichlet distribution with parameters [ b, . . .
, b, a, b, . . .
, b ] , where a is the ˜ y th entry.
An interpretation of the parameter is that a larger a/b favors a larger probability mass at y = ˜
y
(and thus less noise).
We consider different pairs of ( a, b ) so that the tradeoff curve matches the counterpart curve from our proposed method.
The curve is averaged over 50 independent replications.
As shown in Figure 4b, the results indicate that our proposed solution produces less leakage (and thus better privacy) for a given utility.
We also plot heatmaps illustrating the transition laws p K 2 ( y | ˜ y ) obtained from the proposed information laundering in Figure 5.
We considered two cases, where there are 20% class-0 labels, and where there are 1% class-0 labels (by removing related samples from the original dataset).
Intuitively, once we reduce the size of class-0 data in (b), the transition probabilities p K 2 (0 | ˜ y ) for each ˜ y should be smaller compared with those in (a) as class-0 is no longer ‘important’.
Our expectation
[Table]
[Figure]
[Table]
[Figure]
is aligned with Figure 5, where the ﬁrst row in (b) are indicated by darker colors compared with that in (a), meaning that the class-0 is less likely to be observed.
A.
D ATA STUDY : L IFE E XPECTANCY REGRESSION
In this experimental study, we use the ‘life expectancy’ dataset provided by kaggle open-source data (Kaggle, 2020), originally collected from the World Health Organization (WHO).
The data was collected from 193 countries from 2000 to 2015, and Alice’s model is a linear regression that predicts life expectancy using potential factors such as demographic variables, immunization factors, and mortality rates.
This experiment is intended to illustrate the utility-privacy tradeoff and our proposed solution in regression contexts.
In the regression model, we quantize the output alphabet Y by 30 points equally-spaced in between µ ± 3 σ , where µ, σ represent the mean and the standard deviation of Y in the training data.
We then applied a similar procedure as in Subsection A.6, except that we use the empirical R 2 score as the underlying measure of utility and leakage.
The empirical R 2 score has been commonly used for evaluating regression performance, and it can be negative, meaning that the predictive performance is worse than sample mean-based prediction (Scikit-learn, 2020a).
In particular, we obtain tradeoff curves in Figure 6, where we compared the information laundering results based on the proposed technique and Dirichlet-based technique (similar to that in Subsection A.6).
The different β ’s and Dirichlet parameters are summarized in Table 2.
The detailed performance values are also summarized in Table 2.
[Figure]
Y
Y (b) 1% class0.
[Table]
To illustrate the impact of tradeoffs
[Table]
we considered two cases corresponding to β 2 = 1 and β 2 = 20 .
We compute the transition laws p K 2 ( y | ˜ y ) obtained from Algorithm 2 and illustrate them in the ﬁrst row of Figure 5.
We also take the snapshot at the year ˜ Y = 69 and plot the conditional density function p K 2 ( · | ˜ Y = 69) (as approximated by the quantizers) in the second row of Figure 5.
The visualized results are aligned with our expectation that a larger penalty of model leakage will cause a more dispersed transition law.
A. D ATA STUDY : MITIGATION OF ADVERSARIAL ATTACKS
In this experimental study, we demonstrate the use of information laundering in mitigating two model extraction attacks studied in (Tram`er et al., 2016).
We consider settings where the target model is a classiﬁer.
The ﬁrst attack is the Retraining attack.
The adversary Bob sends random queries and receives class labels to train a local model.
Bob does not need to know Alice’s model architecture. The second attack is the equation-solving attack. Alice’s model is assumed to be a Logistic classiﬁer, and Bob knows about it.
Bob sends random queries in this attack and receives class probabilities (a vector on a simplex).
Bob then solves a linear equation to obtain the coefﬁcients of Alice’s Logistic model.
We note that there exist more types of attacks suitable for various speciﬁc settings (see., e.g., Tram`er et al., 2016; Juuti et al., 2019).
The experiments in this section indicate the following two points.
First, an adversary needs more samples to achieve the same utility under the information laundering (Figure 8).
Second, the effects of attack and laundering depend on the particular models speciﬁed by the adversary and target (Figure 9).
[Figure]
[Figure]
[Figure]
The experimental details are given below.
In Figure 8(a), Alice uses half of the Breast Cancer dataset (Scikit-learn, 2020b) (standardized) to train a Logistic classiﬁcation model.
Bob queries the class labels with standard Gaussian random input and locally trains another Logistic classiﬁer.
In responding to Bob, Alice employs different levels of laundering.
The utility is deﬁned as the
[Figure] [Figure]
percentage of agreement of Alice’s and Bob’s models tested on the other half of the dataset.
In
Figure 8(b), Bob sends the same random queries and solves a linear equation to estimate Alice’s
Logistic coefﬁcients.
From the results, we can see that Bob’s performance is better than that in
Figure 8(a).
It is mainly due to the substantial side information and the least-squares estimate of
Bob.
In Figure 9, Alice used half of the simulated Moons dataset (Scikit-learn, 2020c) (with 1000 samples, 0 .
1 standard deviation for the noise) to train a Random Forest model.
The other half of the data are reserved to evaluate Bob’s utility.
Suppose that Bob uses the above Retraining attack, but with two different models.
In Figure 9(a), Bob uses the Random Forest classiﬁer, which has the expressive power to extract Alice’s model.
In Figure 9(b), Bob uses the Logistic classiﬁer, which has a linear decision boundary.
Bob’s model in (b) is inadequate because the Moons data is not linearly separable.
Figure 9 shows that the inﬂuence of information laundering on the adversarial query size needed to maintain the same utility largely depends on Bob’s model choices.
Also, Bob’s inadequate model may show more robustness against laundered information even though it does not perform satisfactorily on cleaned data.