Theoretical Limits of Provable Security Against Model Extraction by Efficient Observational Defenses
Abstract —Can we hope to provide provable security against model extraction attacks?
As a step towards a theoretical study of this question, we unify and abstract a wide range of “observa- tional” model extraction defense mechanisms — roughly, those that attempt to detect model extraction using a statistical analysis conducted on the distribution over the adversary’s queries.
To accompany the abstract observational model extraction defense, which we call OMED for short, we define the notion of complete defenses – the notion that benign clients can freely interact with the model – and sound defenses – the notion that adversarial clients are caught and prevented from reverse engineering the model.
We then propose a system for obtaining provable security against model extraction by complete and sound OMEDs, using (average-case) hardness assumptions for PAC-learning.
Our main result nullifies our proposal for provable security, by establishing a computational incompleteness theorem for the
OMED: any efficient OMED for a machine learning model computable by a polynomial size decision tree that satisfies a basic form of completeness cannot satisfy soundness, unless the subexponential Learning Parity with Noise (LPN) assumption does not hold.
To prove the incompleteness theorem, we introduce a class of model extraction attacks called natural Covert Learning attacks based on a connection to the Covert Learning model of Canetti and Karchmer (TCC ’21), and show that such attacks circumvent any defense within our abstract mechanism in a black-box, nonadaptive way.
Finally, we further expose the tension between Covert Learning and OMEDs by proving that the existence of Covert Learning algorithms requires the nonexistence of provable security via efficient OMEDs.
Therefore, we observe a “win-win” result, by obtaining a characterization of the existence of provable security via efficient OMEDs by the nonexistence of natural Covert
Learning algorithms.
I. I NTRODUCTION
In a model extraction attack , an adversary maliciously probes an interface to a machine learning model in an attempt to extract the machine learning model itself.
In many cases, preventing model extraction helps increase security and privacy, especially with respect to model inversion and adversarial example attacks (see e.g. [1] and references therein).
Additionally, in Machine Learning as a Service (MLaaS), the model is considered confidential as the server usually operates with a pay-per-query scheme.
Therefore, maintaining the secrecy of ML models and finding effective model extraction defense mechanisms is paramount.
Indeed, the problem of how to defend against model extraction has been considered from a practical perspective previously (e.g. [2]–[6]).
Most proposed model extraction defenses (MEDs) in the literature belong to two types (except a few notable exceptions, see e.g. [7]).
The first type aims to limit the amount of information revealed by each client query.
One intuitive proposal for this type of defense is to add independent noise (i.e. respond an incorrect prediction independently with some probability) or even deliberately modify the underlying model.
This type of solution is not a focus of this work, because it necessarily sacrifices predictive accuracy of the ML model, and is therefore not an option for many ML systems where accuracy is critical such as autonomous driving, medical diagnosis, or malware detection.
The second type of MED that has been proposed aims to separate “benign” clients — those that want to obtain predictions but will not attempt to extract the model — and “adverse” clients — clients that aim to extract the model.
This type of “observational” defense is the focus of the present work.
A common implementation of the observational defense involves so-called “monitors” that, receive as input a batch of queries submitted by the client, and compute some statistic meant to measure the likelihood of adversarial behavior, with the goal of rejecting a client’s requests when the queries pass a certain threshold on the statistic (e.g. [2], [5], [6]).
Essentially, observational defenses aim to control the distribution of the client’s queries, by classifying any clients that fail to conform to the appropriate distributions as adverse, and then prohibiting them from accessing the model.
To date, the choice of such appropriate distributions have been made heuristically, for instance, in [5], an appropriate distribution is one with the property that the distribution over hamming distances between independent samples is normally distributed.
However, no formal definitions of security against model extraction have been suggested, and there has also not been much formal work done in an effort to understand the theoretical underpinnings of the proposed observational defenses, in particular.
This fact is highlighted by Vaikuntanathan as an open problem in [8].
As a result, a “cat-and-mouse” progression of attacks and defenses has developed, while no satisfying guarantees have been discovered (for neither cat nor mouse).
In contrast, a lofty goal behind developing a theory of security against model extraction would be to ultimately obtain provable security guarantees.
For example, an initial attempt could attempt to leverage zero-knowledge style ideas, to obtain a guarantee that a client learns nothing about the ML model than they could have learned prior to the interaction with the model.
However, this is likely too strong of a goal, because at the very least the client will learn some queried examples.
What kind of guarantees could we feasibly hope to obtain, then?
One possible revised goal, could be to guarantee that a client learns only as much as possible from some random examples from the model (perhaps using some simulationbased security definition).
In fact, we observe that this notion of security appears to be implicitly behind existing observational defenses.
The literature on practical observational defenses tends to cite the goal of detecting model extraction, but the downstream effect is that the observational defenses seek to exactly confine the examples obtained by the client to some specific distributions (by enforcing a particular benign behavior).
Hence, the idea of only serving clients confined to these benign example distributions undoubtedly assumes that whatever the benign client can learn about the model is indeed “secure.”
To unpack this, let us focus on the case of observational defenses for binary classifiers.
At first glance, the beautiful learning theory of Vapnik and Chervonenkis — which tells us that a number of samples proportional to the VC dimension of the hypothesis class suffices for PAC learning — seems to dash the hopes of using this model of security to obtain any meaningful protection.
Indeed, an adversary could simply query the model a sufficient number of times according to one of the appropriate distributions of random examples, and then apply a PAC-learning algorithm.
The output of the algorithm would be a function which would be a strong approximation to the underlying ML model with high confidence.
However, this view does not account for the complexity of such attacks.
Indeed, for many important families of classifiers (e.g. boolean decision trees), no efficient (i.e., polynomial time)
PAC-learning algorithms are known despite intense effort from the learning theory community.
In fact, no efficient algorithms are known even when the examples are restricted to being uniformly distributed, and the classifier itself is drawn from some kinds of distributions (i.e., in an average-case way, see e.g. [11]).
Hence, this lends credence to the idea that the model of security implicitly considered by observational defenses might actually effective in preventing unwanted model extraction by computationally bounded clients , for instance by forcing the client to interact with the query interface in a way that mimics uniformly random examples, or some other hard example distribution.
In this way, security against model extraction could be provable in a complexity-theoretic way: one could give a reduction from PAC-learning to model extraction in the presence of observational defenses.
In other words, one could hope to prove a theorem that says “any efficient algorithm to learn an approximation of a proprietary
ML model when constrained by an observational defense yields a distribution-specific PAC-learning algorithm (that is currently beyond all known techniques).”
On the other hand, for classes of ML models that are efficiently PAC-learnable, the hope of any OMED is lost because of the VC theory argument described in the previous paragraph.
Thus, this paper focuses on defending ML models with at least some notion of hardness to learn (also, see Footnote 2).
One potential pitfall of the preceding discussion of provable security is that due to the worst-case guarantees for PAClearning, the described reduction would not rule out the useless case that a single model is hard to extract in the presence of observational defenses, but all others are easy.
However, even in an average-case or heuristic PAC-learning setting, where the concept itself is drawn from a distribution (see [11], [12]), there is still a conjectured cryptographic hardness of learning for sufficiently complex classes of concepts and concept distributions.
Therefore, we can continue to envision a reduction from average-case learning to model extraction for most underlying ML models (provided they are sufficiently complex to begin with).
Such provable security against efficient model extraction adversaries would be a significant development in finding the theory behind observational defenses.
B. Our Contributions
Since we have established some faith behind the idea that observational defenses might be effective against computationally bounded adversaries (even in a provable way), a natural follow-up question to consider is if and when observational defenses can be efficiently implemented against those efficient adversaries.
More broadly, we seek an answer to the following three-part question:
Can we provide provable security against model extraction attacks, for any ML model, using an observational defense?
If so, can it be efficiently implemented?
In fact, how do we even define provable security?
We propose a broad method for obtaining cryptographicstrength provable security via an observational defense.
The method can be viewed as an abstraction of the prevailing heuristics used in the literature on practical observational defenses.
Then, we provide a negative answer to the second part of the question, via the following program:
• We formally define a class of abstract MEDs by unifying the common observational defense technique seen in the literature.
• We formalize the concepts of complete and sound observational defenses, namely, the provable guarantees that
benign clients are accepted and may interact with the machine learning interface, and adverse clients are rejected.
We show how our formalisms initiate a method for obtaining a theory of provable security against efficient model extraction attacks by relying on the computational hardness of PAC-learning.
Furthermore, throughout the paper we argue that the proposed method is a useful abstraction of the methods that are (implicitly) employed by the literature on the construction of observational defenses.
• Then, via a connection to the Covert Learning model of [13], we give a method for generating provably good and efficient attacks on the abstract defense, granted that the defense is efficient and it satisfies a basic form of completeness.
We then obtain an attack on decision tree models protected by the abstract class of MEDs by an existing algorithm of [13].
The attack relies on the subexponential Learning Parity with Noise (LPN) assumption, and to the best of our knowledge, constitutes the first provable and efficient attack on any large class of MEDs, for a large class of ML models.
• Using the existence of the attack, we prove our main result: informally, every efficient defense mechanism (within the abstract class of MEDs) for decision tree models which satisfies a basic notion of completeness does not satisfy soundness, even for efficient attackers.
This result essentially prevents instantiating the described method for provable security, assuming the basic notion of completeness is satisfied.
Additionally, we further develop the relationship between
Covert Learning and
OMED
defenses
As a consolation to the negative result
we prove that “either way science wins”: provable security by
OMEDs
exist if and only if Covert Learning algorithms
do
not exist.
In the next two sections, we discuss related work, starting with a discussion on the nature of Covert Learning and the connection to model extraction, and an overview of the techniques used by [13] to obtain Covert Learning algorithms.
We then highlight more related work including other approaches to security against model extraction and existing Covert Learning algorithms.
C. Covert Learning, and the Relationship to Model Extraction in the Presence of Observational Defenses
The Covert Learning model — a variant of Valiant’s PAC model in the agnostic learning with membership queries setting — formalizes a new type of privacy in learning theory.
Specifically, the Covert Learning algorithms provide the guarantee that the membership queries leak very little information about the concept, with respect to a computationally bounded passive adversary.
In other words, the learner can PAC-learn the concept in question (using knowledge of some internal randomness only known to itself, i.e., not even shared with the concept), while the adversary remains nearly completely “in the dark” with respect to the concept, even given a view of the entire transcript of membership queries and oracle responses (but not to the secret randomness of the learning algorithm).
At its heart, the Covert Learning model uses the foundational simulation paradigm of cryptography to achieve these goals.
Roughly, any membership query learning algorithm is a Covert
Learning algorithm if it has an accompanying simulator that emulates the membership queries (the “real” learner) in a computational indistinguishable way — using nothing but random examples (the “ideal” learner).
To elaborate a further, the simulator functions without access to the underlying concept (or knowledge of the hypothesis class, in the case of agnostic learning).
Instead, the simulator accesses, for instance uniformly random examples, which for many interesting hypothesis classes makes the learning requirement hard.
Hence,
Covert Learning algorithms are reserved for concept classes that are may be not efficiently learnable in the original PACmodel with respect to a certain “hard distribution”.
In this work, we focus on a special case of Covert Learning, which we term natural Covert Learning.
In natural Covert
Learning, rather than making necessary a simulator for the interaction, we strengthen the requirement by asking simply for the distribution over the membership queries to be computationally indistinguishable from some pre-defined “hard example distribution.”
For concreteness, let us consider the uniform distribution as the pre-defined “hard distribution.”
Also, we will consider a version that lives in the realizable setting (rather than agnostic).
For this concrete case, Figure 1 provides a graphical depiction.
To explain a bit more formally, fix a concept f : { 0 , 1 } n → {− 1 , 1 } contained in a concept class C , and consider (for concreteness) the uniform distribution U n over { 0 , 1 } n .
A learning algorithm under the Covert Learning model is tasked with finding an hypothesis h : { 0 , 1 } → {− 1 , 1 } that best approximates the concept D on unobserved examples ( x, f ( x )) , for x ∼ U n .
This notion is captured by a loss function, such as L ( h ) =
Pr x ∼ U
[ h ( x ) ̸
= f ( x )] .
A natural Covert Learning algorithm should then satisfy the PAC- learning guarantee: output h such that Pr x ∼ U
[ h ( x ) ̸
= f ( x )] ≤ ϵ with high probability (such an h is called ϵ -good).
In order to achieve this goal, the learner is given access to a membership query oracle that labels a queried input x ∈ { 0 , 1 } n with a corresponding label f
( x ) .
However, the important part of natural Covert Learning is that, essentially, the joint distribution over the membership queries made by the learner [ x 1 , · · · x m ] must be indistinguishable from [ x 1 , · · · x m ] ∼ ( U n ) m by any computationally bounded adversary.
Definition I.1 (Natural Covert Learning — informal version of
Definition IV.1) .
A natural covert learning algorithm — for a class of concepts C and a distribution D over examples — is an algorithm that, for any f ∈ C and accuracy parameters ϵ, δ , interacts with an oracle that labels queries to the concept f such that the following are true:
• Learning.
The learning algorithm outputs an ϵ -good hypothesis for the concept with probability 1 − δ .
• Privacy.
The
joint distribution over all queries and
responses to and from the oracle
is
computationally indistinguishable from the distribution ( x 1 , f ( x )) , · · · ( x m , f ( x m )) for x 1 , · · · x m ∼ ( D n ) m .
The conceptual crux of this work .
We formalize a connection between the adversary in Covert Learning — a distinguisher that attempts to distinguish the membership queries made by the learner algorithm from random examples, — and the “adversary” in a model extraction attack — an
OMED.
At first glance this may be a confusing connection, but consider that the job of the OMED is to somehow distinguish benign clients from adverse clients.
Thus, thinking from the perspective of the actual adversary, i.e., the model extractor, the OMED is adversarial in nature and has essentially the same job as the natural Covert Learning adversary: to detect whether the client is benign or a model extractor.
In this way, we show that “natural” Covert Learning algorithms form a recipe for an attack that can fool the OMED in the same way that they fool the Covert Learning adversary.
The Learning guarantee of natural Covert Learning then allows us to still argue that the client can extract the underlying model.
Hence, we can show that a single Covert Learning attack can achieve extraction while “fooling” any OMED.
That is, any OMED will output “accept” with high probability even though an extraction attack is being performed.
Since the attack is completely black-box with respect to the implementation of the OMED (it only requires that the OMED is efficient and satisfies the basic uniform completeness condition), the existence of this attack can be used to demonstrate the incompleteness of the OMED.
Overview of the natural Covert Learning algorithm for decision trees of [13] .
Since we use the Covert Learning algorithm for decision trees from [13] as a black box, let us use the remainder of this section to provide a useful overview of the algorithm.
The covert learning algorithm for decision trees begins by using a “masked” Goldreich-Levin algorithm in order to obtain large Fourier coefficients.
The Goldreich-Levin algorithm [9] is an algorithm that selects correlated queries in an ingenious way to decode a noisy parity function.
The algorithm can also be used learn Fourier coefficients large (i.e., 1 / poly( n ) magnitude) of any function f : { 0 , 1 } n → {− 1 , 1 } , in time polynomial in n , and has been said to inspire the similar
Kushilevitz-Mansour algorithm
The “masked query” technique of [13] works as follows: for any given query x ∈ { 0 , 1 } n , generate a pseudo-random string by taking n samples from a carefully crafted Learning
Parity with Noise (LPN) distribution (see [14] for a useful introduction on LPN).
Letting the n samples be concatenated into string y ∈ { 0 , 1 } n , the masked query is taken as the bitwise xor m
= x ⊕ y .
It is easy to see that the LPN distribution is pseudo-random, then so is the masked query m .
Mathematically, the string y can be written as y =
As ⊕ e , where A is a random (but not uniformly random) n × n binary matrix, e ∈ { 0 , 1 } n is a noise vector sampled by taking an n -wise direct product of a biased Bernoulli random variable with small, constant mean, and s ∈ { 0 , 1 } n is secret sampled according to a “chopped tail” n -wise direct product Bernoulli distribution with minimum entropy Θ(log 2 ( n ) /n ) .
This LPN distribution is due to [15], who show its pseudo-randomness assuming the more standard subexponential LPN assumption (defined formally as Definition IV.4).
Then, [13] show that, by augmenting an instance of the
Goldreich-Levin algorithm for learning large Fourier coefficients by masking each query x
i with the mask y i
=
As i ⊕ e i , then the returned labels f
( x i ⊕ y i ) can be post-processed by re-incorporating the dependecy on the secret s , and using the approximate linearity of the LPN distribution and large Fourier coefficients, to access a noisy version of f (denoted ˜ f ) which crucially has the property that any large Fourier coefficient of f is also large for ˜ f .
However, the technique fails at extracting large Fourier coefficients of any degree, and instead obtaining only those with degree O (log n ) .
The reason for this is that the noise “overflows” for the higher degree coefficients, while it remains manageable for low-degree coefficients due to the low minimum entropy nature of the LPN secret.
Unfortunately, attempting to increase the degree bound of O (log n ) by further decreasing the minimum entropy of the secret does not work without breaking the pseudo-randomness of the masks.
Because each query is masked independently by a pseudorandom string, it is straightforward to see how this Covert
Learning algorithm is also a natural Covert Learning algorithm with the uniform distribution as the pre-defined “hard distribution.”
Once the set of large O (log n ) -degree
Fourier coefficients is found, one can use standard techniques to produce a hypothesis which is the sign of a low-degree polynomial.
Under further analysis, the hypothesis obtains agnostic learning guarantees for the hypothesis class of poly( n ) size decision trees.
D. More Related Work
a) Existing OMEDs: We point the reader to Section III-B for a detailed discussion on some of the practical OMEDs
proposed in the literature.
b) Secure inference for MLaaS: A somewhat related approach to improving the privacy of Machine Learning as a Service (MLaaS) termed “secure inference” has been proposed.
This approach borrows from ideas in the field of Secure
Function Evaluation (where parties can securely compute a function without revealing their inputs), and makes use of garbled circuits [16] or fully homomorphic encryption [17].
However, the principle guarantee of the “secure inference” approach only provides hiding of information about the model beyond what can be deduced from the query and the model’s output.
Hence, a secure inference approach to security against model extraction would implicitly assume (incorrectly) that total leakage from the predictions is little, and that recovering the model from its predictions would be infeasible or impossible.
Therefore, the “secure inference” approach does not properly prevent model extraction when considering clients who repeatedly interact with the service.
c) More natural Covert Learning attacks: The works of [18] and [19] introduce a method for sampling pairs of matrices ( A, T ) with entries in Z q , such that A is statistically close to a uniformly random matrix, while M 2 is a low-norm, full-rank trapdoor matrix such that A · T is the all zero matrix.
In [8], Vaikuntanathan notes that this sampling algorithm gives an easy, yet somewhat contrived model extraction attack.
In fact, it is a natural Covert Learning attack as well.
More specifically, for any ML model that is essentially a linear function over Z q with added Gaussian noise e ∈ Z mq , 3 the linear function (denoted s ∈ Z nq ) can be extracted by querying sA + e , and then taking ( sA + e ) T = eT , which can then be used to extract e via Gaussian elimination ( T is full rank).
Then, given e , s is easily recoverable.
However, A is statistically close to uniformly random, so the queries are impossible to distinguish from uniformly random queries with any significant advantage.
This statistical Covert Learning attack could rule out even unbounded OMEDs, however it only works for the very narrow class of noisy linear models, which do not appear frequently in practice.
The work of [13] discusses how to similarly sample trapdoors for the lownoise LPN problem of Alekhnovich [20].
The techniques gives rise to another natural Covert Learning attack for an
LPN variant of the above setting, however the queries are only computationally close to uniform.
Elaborating on this technique in the present paper would be quite time-consuming, hence we refer the reader to [13].
d) Related formalisms: We note that the formalisms in this work are inspired by the field of Interactive Proofs [21].
Also, the work of [22], who work on protocols for verifying forecasting algorithms, inspired the drive to prove computational incompleteness theorems in this setting.
II. T
We will use the following standard of computational indistinguishability.
Definition
II.
(Computational
Indistinguishability) .
Let { X n } , { Y n } be sequences of distributions with X n , Y n ranging over { 0 , 1 } m ( n ) for some m ( n ) =
n O (1) .
{ X n } and { Y n } are computationally indistinguishable if for every polynomial time algorithm A and sufficiently large n , | Pr[ A (1 n , X n ) = 1]
− Pr
[ A (1 n , Y n ) = 1] | ≤ negl( n )
Often, n is clear from the context, so the subscript is omitted.
We consider two notions of learnability of boolean concepts.
A boolean concept class C = {C n } n ∈ N is a sequence of sets where C n is a set of boolean functions each taking x ∈ { 0 , 1 } n as input, and outputting a label y ∈ {− 1 , 1 } .
Similarly, a distribution class D = {D n } n ∈ N is a sequence of sets of distributions, where each D n is a set of example distributions over { 0 , 1 } n .
We define the following learning models that are considered in this work.
The oracle E X ( f, D ) samples x ∼ D (for a distribution D over { 0 , 1 } n ) and returns ( x, f ( x )) .
Definition II.2 (Efficient PAC Learning) .
A boolean concept class C is PAC-learnable with respect to a distribution class
D if there exists an algorithm A such that for any n ∈ N , distribution D ∈ D n , concept f ∈ C n , when A is given as input n and ϵ, δ > 0 , plus access to E X ( f, D ) , it outputs a function h such that
[Equation]
We say that C is efficiently PAC-learnable with respect to
D if A runs in time polynomial in n, ϵ − 1 , δ − 1 , and the number of accesses to E X ( f, D ) is bounded by a polynomial in n, ϵ − 1 , δ − 1 .
The following definition of heuristic PAC learning due to
Nanashima [12] can be seen as a variant of many existing average-case learning models, where the distribution over concepts is fixed to be uniform.
In his original work, Nanashima defines a distribution over representation strings over concepts, but in this work it suffices to consider a distribution over actual concepts.
The definition can also be interpreted as requiring
PAC-learning for all but some fraction of concepts in the class, where this fraction is given as an input to the learner.
Definition II.3 (Efficient Heuristic PAC-learning — adapted from [12]) .
Let C =
{C n } n ∈ N be a boolean concept class, and let U n be the uniform distribution over C n .
We say that C is heuristically PAC-learnable with respect to the distribution class D if there exists an algorithm
A such that for any n ∈
N , D ∈ D n , when A is given as input n and ϵ, δ, η > 0 , plus access to E X ( f, D ) for some f sampeld uniformly at random from C n , it outputs a function h such that
[Equation]
We say that C is efficiently heuristically PAC-learnable with respect to D if A runs in time polynomial in n, η − 1 , ϵ − 1 , δ − 1 , and the number of accesses to E X ( f, D ) is bounded by a polynomial in n, η − 1 , ϵ − 1 , δ − 1 .
Additionally, we say that C is (efficiently) ( ϵ ′ , η ′ ) -heuristically
PAC-learnable if we have fixed the parameters
ϵ
=
ϵ ′ , η = η ′ .
III. T HE A BSTRACT O BSERVATIONAL M ODEL E XTRACTION D EFENSE
In this section, we formally introduce the Observational
Model Extraction Defense (OMED) as a unifying abstraction for the current state of the art MEDs.
Before we introduce
OMEDs formally, let us describe the setting of model extraction in the presence of an OMED in more detail.
At a very high level, the setting begins by an ML model f : { 0 , 1 } n → {− 1 , 1 } being adversarially chosen (potentially from some restricted class of functions).
Then, we consider the case that a probabilistic polynomial time algorithm C (the client), can interact via an oracle to the ML model f .
We denote this oracle by O f .
The goal of the benign client is to obtain some predictions f ( x ) for queries x ∈ { 0 , 1 } n .
On the other hand, the goal of the adversarial client is to output a function ˆ f , that approximately minimizes a loss function with respect to the model f .
We will refine the behavior of a client and the way it interacts with the ML model after we define the OMED.
The important part is that the adverse client must be able to perform the extraction in the presence of the OMED.
In particular, the OMED is able to view all the queries made by
C and the labels that would be returned (see Figure 2).
The
OMED then (after performing arbitrary computations) outputs a decision “accept” or “reject.”
In the case of “accept,” the
OMED also forwards the labels back to the client.
Otherwise, no labels are returned back to the client.
With this in mind, let us now formally define the OMED.
As noted in the introduction and in [23], defense mechanisms for model extraction have mostly split into two tribes: reducing the information gained per client query, and differentiating malicious extraction adversaries from benign users.
The OMED mechanism abstracts the latter approach; the implementation is left unspecified.
Hence, we define an OMED abstractly as follows.
Let F = {F n } n ∈ N be a class of ML models, where each F n in the sequence is a set of functions taking as input an element of the set { 0 , 1 } n and giving output in the set {− 1 , 1 } .
Definition III.1 (OMED) .
A probabilistic algorithm M is a T ( n ) -OMED for a class of ML models F if for every n ∈ N , f ∈ F n , M runs in time T ( n ) and takes as input a list of examples S =
[( x 1 , f ( x 1 )) , · · · ( x m f ( x m ))] ⊆ ( { 0 , 1 } n × {− 1 , 1 } ) m and outputs a decision σ ∈ { accept , reject } .
Discussion:
On the restrictive nature of the syntax of
OMED.
The definition of the OMED is defined as generally as possible from the perspective of the defense, but makes one important restriction on the client: the examples are requested in large batches, rather than as an adaptive sequence.
This nonadaptive setting (with respect to the query selection) can be viewed as unnecessarily restrictive on the client.
However, since we prove negative results on the possibility of MEDs via OMEDs, the restriction on the client actually strengthens our results.
Furthermore, the defense could be expanded to a multi-client setting, where all clients must submit their batches simultaneously.
This is again a restriction on the power of the client(s), and thus strengthens our negative results.
Finally, we mention that with respect to Sybil attacks, the underlying assumption is that the MED will not evaluate the queries requested by each identity jointly (perhaps because the defense is assumed not to know the identities of the participating attackers).
However, we prove our negative result on an OMED that always has access to all queries made by the attacker(s).
Hence, our attack works in a more restrictive setting than Sybil attacks.
A. How to Obtain Provable Security
The above definition of an OMED makes no claims about desirable properties given by the OMED.
Thus, what properties should we expect from the OMED?
As mentioned in
Section I-A, the goal of the OMED is not just to classify the behavior of the clients, but to actually confine the clients to certain predefined benign behaviors.
However, it is not enough to simply define security as the event that the client behavior is benign, because this actually needs to be detected and then enforced.
Therefore, it should be that a good OMED guarantees that (with high probability) any benign client is accepted, while any adverse client is rejected (and thus prevented from reverse engineering the underlying model).
Naturally, the former requirement resembles completeness in an interactive proof system while the latter requirement resembles soundness.
Through this lens of Interactive Proofs, we will formalize a notion of completeness and soundness.
First, let us explain how we model a client’s behavior in the context of the model extraction setting depicted in Figure 2 in more detail.
It has been noted in the model extraction literature (e.g. [5]) that defenses should consider how a client’s queries relate to each other, rather than how they look individually.
This idea is implemented by assuming that a client’s queries follow a distribution 5 .
We adopt a similar idea in this work: we assume that a client’s queries follow a distribution P over the domain of the ML model.
To this end, we now identify a client C as a tuple ( C, L ) consisting of a polynomial time samplable distribution over sets of m examples C , as well as a probabilistic polynomial time algorithm L that takes as input a list of m examples drawn according to C and labelled by the ML model f , and outputs an approximate model
ˆ f .
Thus, informally, the our idea is that a benign client request examples according to distributions that share some abstract property.
We may formalize this as follows.
Let ∆(( { 0 , 1 } n ×{− 1 , 1 } ) m ) be the convex polytope of all distributions over ( { 0 , 1 } n × {− 1 , 1 } ) m .
Definition III.2 .
P n,m is a property of distributions, where P n,m ⊆ ∆(( { 0 , 1 } n × {− 1 , 1 } ) m ) .
We then define a benign client:
Definition III.3 .
Fix any ML model f .
We say that a client
C =
( C, L ) is P n,m -benign if the distribution C over sets of m examples ( { 0 , 1 } n ) m is such that when S ∼ C , the distribution D over S = (( x 1 , f ( x 1 )) , · · · ( x m , f ( x m ))) makes it so that D ∈ P n,m .
and similarly an adverse client:
Definition III.4 .
Fix any ML model f .
We say that a client
C =
( C, L ) is P n,m -adverse if the distribution C over sets of m examples ( { 0 , 1 } n ) m is such that when x 1 , · · · , x m ∼ C , the distribution D over S = (( x 1 , f ( x 1 )) , · · · ( x m , f ( x m ))) makes it so that D / ∈ P n,m .
1) Completeness: Towards obtaining a theory of provable security against model extraction, we now argue that, intuitively, a good OMED (and any MED) should satisfy, informally,
MED Completeness:
For any benign client distribution, the defense mechanism does not reject and allows the client to continue to interact with the model, with high probability.
A completeness requirement on a MED can be interpreted as formalizing the usefulness of the defense.
In other words, the defense at the very least provides the opportunity for benign clients to interact with the model.
Formally, we define completness as follows:
Definition III.5 (MED completeness) .
We say that an OMED
M is δ -complete with respect to the property P n,m if for any
[Equation]
Provable security against benign clients: How should we choose the benign property?
The definition of completeness implicitly assumes that, in the case that the client is classified as benign, they are essentially free to interact with the model.
Thus, the underlying assumption is that by virtue of the client being benign, the ML model is not considered at risk for being extracted by the server.
Therefore, the choice of the benign property is of utmost importance.
As discussed in the introduction (see Section I-A), this leaves room for a theory of provable security.
For example, under our framework, a solid choice for a property would be the P n,m = ( U n ) m (i.e., the uniform distribution over m examples).
This is because many interesting classes of models are thought to be hard to learn from uniformly random examples, even for most models in the class (that is, in the heuristic PAC-learning case, rather than just in the worst-case).
This means that we can give a reduction from extracting the
ML model (in the average-case over the uniform distribution) using ( U n ) m -benign queries to heuristic PAC-learning with respect to the uniform distribution.
To formalize this intuition, we prove the following lemma, which essentially says that if PAC learning is impossible for a large fraction of the class, then, given that the OMED accepted a client that used ( U n ) m -benign queries, most models can not be extracted (with arbitrarily high fidelity) except with negligible probability.
After giving the lemma we will provide more color by further discussing the reasons behind defining security this way.
We will later use the lemma to show how to obtain a formal notion of security against model extraction by all clients, whether adverse or benign.
We first need to establish what constitutes a successful model extraction.
Similarly to [4], we define the following extraction experiment.
Again, let F n be a class of boolean ML models, and let C =
( C, L ) be a client.
Fix a loss function
L D,f : F n → [0 , 1] parameterized by an ML model f ∈ F n , and a distribution D over { 0 , 1 } n .
Definition III.6 (Extraction experiment) .
Let Exp C ,f,m,ϵ be defined as the output of the following process.
1) Sample x 1 · · · x m ∼ C .
These examples are paired with the labels to produce the set S =
[ x i , f ( x i )] i ∈ [ m ] .
Run L ( S ) to obtain ˆ f . 3)
If L C,f ( ˆ f ) < ϵ , output ( S, extracted ), else output ( S, unextracted) .
Lemma III.7 (Provable security against clients behaving benignly) .
Let M be any OMED satisfying δ -completeness for any δ ≥ 1 − 1 / poly( n ) , with respect to a property of distributions P n,m .
Then, if a class of ML models F n is not efficiently ( η, ϵ ) -heuristic PAC-learnable with respect to any of the singleton distribution classes in the set {{ D } : D ∈ P n,m } , then there exists F ⊆ F n of size at least η ·|F n | , such that for any client C =
( D, L ) that is P n,m -benign, and for all f ∈ F ,
[Equation]
Proof.
We show the contrapositive.
Let E, A be the events (over the randomness of Exp C ,f,m,ϵ ) that τ = extracted and
M ( S ) = accept , respectively.
Thus, suppose that for some
D ∈ P n,m , we have a client C
= ( D, L ) such that for ϵ > 0 , and a δ -complete M ,
[Equation]
This implies that
[Equation]
which is equivalent to
Pr
[Equation]
∼F n
[Equation]
since M is δ -complete (the inner probabilities are all over the randomness of M , ( S, τ ) ∼ Exp C ,f,m,ϵ ; this is omitted due to space constraints).
We now may observe that latest equation is the guarantee that there is an algorithm that ( η, ϵ )
-heuristically
PAC learns F n in time poly( n ) with accuracy ϵ and confidence (1 − δ ) / poly( n )) , for some singleton distribution class { D } (where D ∈ P n,m ).
The learning algorithm works by sampling a set of labelled examples T according to D , and the applying
L ( T ) to obtain a hypothesis.
Then, it follows that F is ( η, ϵ ) -heuristically PAC-learnable with respect to some singleton distribution class { D } (where
D ∈ P n,m ) by running the learning algorithm poly( n ) times to produce many hypotheses, testing each by random sampling, and outputting the most accurate hypothesis.
This works as long as δ ≥ 1 − 1 / poly( n ) .
This completes the proof by contrapostive, which only requires we show learnability with respect to at least 1 singleton distribution class { D } for
D ∈ P n,m .
Discussion: We reiterate that the preceding lemma shows that, under an assumption that a class of ML models is hard to learn in the heuristic sense (i.e., most of models in the class are hard concepts) with respect to some property of example distributions, then clients that are benign on that property of distributions cannot extract most of the model in the class.
This provides a theoretical foundation for instantiating the OMED paradigm, which assumes that some distributions are indeed benign, based off of hardness of heuristic PAClearning assumptions.
On the other hand, we remark that it is unfortunately not realistic to require hard extraction for all models, as many natural classes of models in the class (even for classes that are not even known to be efficiently
PAC-learnable, e.g. decision trees), because natural classes still contain very simple models that can be learned super efficiently (e.g. a decision tree that always outputs the label 1).
However, the model owner can make the assumption that their (fixed) model is inside the large set of hard-to-learn models over the benign distributions.
Indeed, it can be argued that if the model owner is not willing to make this assumption, then the model is not suitable for the OMED paradigm in the first place, since complete OMEDs (with respect to benign property
P n,m ̸ = Ø ) allow the client to obtain some random examples by definition.
2) Soundness:
The notion of provable security from
Lemma III.7 only deals with clients that request sets of examples that are P n,m -benign.
Thus, we need to provide some guarantees when this is not the case.
To this end, we also formalize soundness using the above extraction experiment.
Informally,
MED Soundness : For any adverse client that has attempted extraction, the defense mechanism rejects the client, with high probability.
Definition III.8 (MED soundness) .
We say that an OMED M for F n is δ -sound with respect to the property P n,m if for any
P n,m -adverse client C ∗ =
( C ∗ , L ) , and for any f ∈ F n , ϵ > 0 , it holds that
[Equation]
The soundness requirement can be interpreted as formalizing the security of the defense.
That is, attackers will not be able to deceive the OMED into granting interaction with the
ML model in such a way that allows extraction.
The definition also mirrors soundness from an Interactive
Proof, where for any client that is adverse with respect to the benign property, and given that the adversary would have extracted the model, the probability that the OMED M errs by accepting the queries is low.
In a cryptographic setting, the
OMED would set δ to a negligible function of n .
3) Cryptographically-Hard Model Extraction Against All
Clients: To tie it all together, we argue that combining completeness, soundness, and hardness assumptions for heuristic
PAC-learning light the way to providing full security against model extraction by an OMED.
Our notion of security against model extraction (defined below) constitutes bounding the probability that a client wins the extraction game by a negligible function, in this case of n (the size of the learning problem).
Definition III.9 (Security against Model Extraction by all clients) .
We say that an OMED M for F is ( η, ϵ ) -secure against model extraction if for sufficiently large n , there exists
F ⊆ F n of size at least η · |F n | , such that for any p.p.t.
client
C =
( C, L ) and for all f ∈ F, D n ∈ P n,m ,
[Equation]
We prove that, essentially, for any complete and sound,
OMED M for a class of ML models F n , if F n has no ηheuristic PAC-learning algorithm, then at least η fraction of models in F n can not be extracted by any client, no matter how it behaves, except with negligible probability.
Theorem III.10 (Provable Security from Complete and
Sound OMEDs) .
Let M be any p.p.t.
OMED satisfying δcompleteness and γ -soundness for any δ ≥ 1 − 1 / poly( n ) , γ ≤ negl( n ) , with respect to a property of distributions P n,m .
Then, if a class of ML models F n has no p.p.t.
( η, ϵ ) -heuristic PAC-learning algorithm with respect to any distribution D n ∈ P n,m , then M is ( η, ϵ ) -secure against model extraction.
Proof.
By definition, any client C =
( C, L ) is either P n,mbenign or P n,m -adverse.
In the former case, Lemma III. implies that
[Equation]
In the latter case, we have the guarantee from Definition III.
that
[Equation]
where γ is a quantity bounded above by a negligible function of n .
Therefore the statement follows.
Discussion: is this security enough?
We remark that the level of security is essentially determined by the strength of the hardness assumption on heuristic PAC-learning.
To be specific, an assumption of hardness of ( η, ϵ ) -heuristic PAClearning maps to an ( η, ϵ )
-secure
OMED.
This is meant to be a feature, rather than a bug, as this dynamic mirrors (but formalizes) the efforts of practical OMEDs in the literature, which implicitly make an underlying assumption that certain query distributions make extraction hard for a specific model owner’s model.
In other words, we have shown how to phrase the implicit assumptions as heuristic PAC-learning hardness assumptions.
B. Defense Proposals As Special Cases of the OMED
Our proposed OMED technique for provable security against model extraction is purposely defined as generally as possible.
However, we view it beneficial to discuss the relation to some concrete MEDs which have been proposed.
In this section, we will review three MEDs, [2], [5] and [6], demonstrating that each are special cases of an OMED (with unproved completeness and soundness guarantees).
We start each example with a direct quote from the original paper so as to directly demonstrate the relevance to our OMED framework.
1) Extraction Monitors: Information Gain and Feature
Space Coverage:
The work of [2] proposes two different strategies for detecting model extraction attacks.
Both strategies “[quantify] the extraction status of models by continually observing the API query and response streams of users” and provide a warning when a certain extraction status is reached.
This is indeed the paradigm outlined by the OMED.
The first proposal of [2] seeks to continuously train a “proxy model” for each client, where the client queries are used to train the model.
The function of the proxy model is to estimate the information/knowledge gained by a client with respect to a validation set which is given by the server (and when this information reaches some threshold the client is flagged).
The distribution of this validation set mimics the training set of the underlying model.
It is noted that it may require significant computational resources to train and update the proxy model (for every user and each incoming query), and thus [2] propose to use a lightweight decision tree proxy model.
In the second proposal, the observational keeps a short description of client queries, and estimates the client’s learning rate (of the extraction attack) by analyzing the feature space covered by these queries (as they relate to to the class boundaries of the underlying model.
It is noted that a drawback of this proposal is that the class boundaries of certain complex models (e.g. neural networks) are not easily found.
Thus, it is proposed that the owner of the underlying model uploads a “surrogate” decision tree which has high fidelity with respect to the complex model (class boundaries of decision trees are easily interpreted by their leaf nodes).
2) PRADA:
The MED known as PRADA [5] “analyzes the distribution of consecutive API queries and raises an alarm when this distribution deviates from benign behavior” [5].
Immediately, it is clear that the PRADA method is a candidate for being identified as a special case of the OMED.
The defense works under the observation that queries requested by an adversarial client are likely to have a distribution that differs from the characteristic distribution of queries from a benign client.
In PRADA, this benign characteristic was chosen to be the property that the distribution over hamming distances between each query in the requested batch should be normally distributed.
This choice is backed by observational evidence that certain popular attacks such as the attack of [24] do not satisfy this condition.
Hence, PRADA tries to satisfy completeness and soundness with respect to the property of all distributions that have a pairwise hamming distance normally distributed (e.g. the uniform distribution over { 0 , 1 } n ).
3) VarDetect: The work of [6] proposes a MED called
VarDetect which is designed “to continuously observational the distribution of queries to [the model] from each user” [6].
Specifically, VarDetect trains a Variational Autoencoder (VAE) to map the “problem domain” (PD) dataset distribution (the
PD distribution mimics the distribution of data that was used to train the underlying model) and the adversarial “outlier” (O) data distribution (the distribution of attacker queries) to distinct regions in latent space.
Benign clients are assumed to query from the PD distribution while adverse clients are assumed to query from an O distribution.
VarDetect purports to separate these two by computing the maximum mean discrepancy (MMD) between the latent mapping of the client’s queries and that of the PD distribution (the MMD test flags the client if the result is above a certain threshold).
IV. A TTACKS ON E FFICIENT OMED S FROM N ATURAL C OVERT L EARNING
In this section, we will consider the question:
Can we efficiently realize the provable security guarantees outlined in the previous section?
Towards a negative answer, we will introduce an attack on the OMED technique for provable security, via a connection to
Covert Learning [13].
Our attack will generate a distribution of examples which is computationally indistinguishable from a distribution in the property that is accepted by the OMED.
In other words, the attacker operates (computationally) indistinguishably from a benign client, in the eyes of the OMED.
Still, the labelled queries allow the algorithm to extract a model with high accuracy.
A. Natural Covert Learning
We will focus on a special case of Covert Learning, which we call natural Covert Learning .
As described in
Section I-C, a natural Covert Learning algorithm, essentially, is a membership query learning algorithm that satisfies the normal PAC-learning guarantees with respect to an example distribution D , with the added property that distribution over the membership queries and labels requested by the algorithms is computationally indistinguishable from examples sampled according to D .
More formally, let C =
{C n } n ∈ N be a boolean concept class, and let D = { D n } n ∈ N be a sequence of distributions where each D n is a distribution over { 0 , 1 } n .
Definition IV.1 (Natural Covert Learning) .
We say that A is an natural Covert Learning algorithm for C with respect to D if for every n ∈ N , f ∈ C n , ϵ, δ > 0 , A satisfies the following when given membership query access to f :
• Learning.
For the random variable h =
A O f ( n, ϵ, δ ) we have
[Equation]
• Privacy.
Let the query complexity of A be m .
For every p.p.t.
adversary Adv , and S ∼ E X ( f, D n , m ) ,
[Equation]
where T ( A O f ) denotes the distribution over the queries made by A and the responses by the oracle.
We say that A is efficient if it runs in time polynomial in n, ϵ − 1 , δ − 1 , and the number of queries m is bounded by polynomial in n, ϵ − 1 , δ − 1 .
We note how natural Covert Learning is defined here with respect to a sequence of example distributions, rather than a sequence of sets of example distributions.
This keeps the definition simpler.
One can consider for example natural covert learning algorithms for C with respect to the sequence of uniform distributions {U n } n ∈ N , each over { 0 , 1 } n .
B. Natural Covert Learning Attack
In this section, our goal is to show that the existence of a natural Covert Learning algorithm for a particular concept class implies inadequacy of a polynomial time OMED for a class of models equal to the concept class.
More specifically, we prove that satisfying soundness for an OMED is impossible, for any reasonable completeness parameter.
This suffices to rule out obtaining provable security against model extraction by an OMED by instantiating Theorem III.10.
Theorem IV.2 .
Suppose that there exists an efficient natural
Covert Learning algorithm A (with query complexity m ) for a concept class C , with respect to the sequence of distributions D = { D n } n ∈ N .
Then there exists a client C such that for any n ∈ N , any poly( n )
-OMED M for C that is δ -complete with respect to a property P n,m that contains E X ( f, D n , m ) , and any ϵ, δ A > 0 , and f ∈ C n ,
[Equation]
where negl( n ) denotes a negligible function of n and δ A is the failure probability of A .
Proof.
Let D A be the distribution over the set of m examples which is queried by A .
Define the client C ∗ =
( D A , A ) .
Let
S ∼ E X ( f, D n , m ) and S A ∼ E X ( f, D A , m ) .
Using the fact that by our Covert Learning assumption we have that for every p.p.t.
adversary Adv ,
[Equation]
and M , f are polynomial time computable, we get that
[Equation]
Then, we can conclude by independence of events
[Equation]
The last equation follows by the δ -completeness property of
M , and the fact that the covert learning algorithm succeeds to extract under the experiment with probability at least 1 − δ A .
Therefore, C ∗ is a client that satisfies the implication in the theorem statement.
Now from this attack, a generic incompleteness theorem follows, essentially because the attack works against any efficient OMED.
Corollary IV.3 .
Suppose that there exists an efficient natural
Covert Learning algorithm A (with query complexity m ) for a hypothesis class C with respect to the sequence of distributions D = { D n } n ∈ N .
Then if for any n ∈ N , M is a poly( n ) - OMED for C n and is δ -complete with respect to a property { E X ( f, D n , m ) } , then M is not (1 − δ − negl( n )) -sound with respect to P n,m .
Proof.
By Theorem IV.2 there exists a { D n } -adverse client
C =
( D A , A ) such that for any δ -complete OMED M (with respect to { D n } ) for C n , it holds that for any f ∈ C n , ϵ > 0 ,
[Equation]
where negl( n ) is a negligible function of n and δ C is the probability C fails to extract.
δ C =
δ A is derived directly from δ A , the failure probability of the natural covert learning algorithm.
Then, we can deduce that
[Equation]
Thus by taking an appropriately large n , M cannot be (1 − δ − negl( n )) -sound for a reasonable choice of δ A such as 1 / 100 , when creating the client C =
( D A , A ) based on A .
C. Concrete Attack and Incompleteness Theorem
In this section, we show that under the subexponential hardness assumption on the standard LPN problem, there exists an attack of the type outlined in the previous section.
Let us first formally introduce our assumption.
Definition IV.4 (Search LPN assumption) .
For µ (0 , 0 . 5) , n ∈ N , the ( m ( n ) , T ( n )) − SLPN µ,n search assumption states that for every inverter I running in time T ( n ) ,
[Equation]
where s $ ← Z n 2 , A $ ← Z m ( n ) × n, e $ ← β m ( n ) µ .
Thus, the assumption we adopt is the
[Equation]
assumption
[Equation]
The following theorem is implicit in [13].
Let DT [poly( n )] be the set of all decision trees of size poly( n ) .
Theorem IV.5 .
(Agnostic Covert Learning of decision trees from [13])
Given query access to a function f : { 0 , 1 } n → {− 1 , 1 } , there exists an algorithm A running in time poly( s, 1 /ϵ, log(1 /δ )) and making q ( n ) = poly( n, 1 /ϵ, log(1 /δ ))
query accesses such that, unless the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption does not hold, [Equation]
such that
[Equation]
with probability 1 − δ . 2) (Privacy) The distribution over examples requested by A is computationally indistinguishable, but statistically distinguishable, from E X ( f, U n , q ( n )) .
We may now proceed to combine Theorem IV.2 and Theorem IV.5 to obtain a concrete natural Covert Learning attack on models implemented by decision tree classifiers of polynomial size.
Let U =
{ ( U n ) m } n,m ∈ N , and let L D,f ( h ) =
Pr x ∼ D [ h ( x ) ̸ = f ( x )] .
Theorem IV.6 .
Under the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption, there exists, for any n ∈ N , there exists a ( U n ) m -adverse client C (for some m = poly( n ) ) such that for any δ -complete
OMED M (with respect to { ( U n ) m } ) for DT [poly( n )] , it holds that for any f ∈ DT [poly( n )] ,
[Equation]
Proof.
Observe that the algorithm described in Theorem IV.
constitutes a natural Covert Learning algorithm for C with respect to U , and loss function L D,f ( h ) =
Pr x [ h ( x ) ̸ = f ( x )] .
We can set the failure probability of the underlying natural covert learning algorithm for the client to 0.99.
Thus, the statement follows directly from Theorem IV.5 and Theorem IV.2.
1) Incompleteness Theorem: The previous theorem can be interpreted as the existence of an attack against any OMED for decision tree classifiers with δ -completeness with respect to the uniform property (up to SLPN assumptions).
Hence, we get the following corollary.
Corollary IV.7 .
Under the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption, if M is a poly( n ) -OMED for DT [poly( n )] and
U , then if M is δ -complete
it is not (1 − δ − negl( n )) -sound.
Proof.
By Theorem IV.6, there exists, for any n ∈ N , a U nadverse client C such that for any δ -complete OMED M (with respect to {U n } ) for DT [poly( n )] , it holds that for any f ∈
DT [poly( n )] ,
[Equation]
Then, we can deduce that
[Equation]
which completes the proof.
As an example, our result shows that for any reasonable choice of δ for uniform completeness (e.g. 0.1, meaning that at most 0.1 honest clients are incorrectly rejected), then it is impossible to obtain any even remotely useful soundness guarantee.
Using δ = 0 . 1 , our result shows that not even 0 .fraction of extraction attempts will be detected by the OMED.
More generally, let us recall the provable security theorem, restated below.
Theorem IV.8 .
(Restated Theorem III.10) Let M be any p.p.t.
OMED satisfying δ -completeness and γ -soundness for any δ ≥ 1 − 1 / poly( n ) , γ ≤ negl( n ) , with respect to a property of distributions P n,m .
Then, if a class of ML models F n has no p.p.t.
( η, ϵ ) -heuristic PAC-learning algorithm with respect to any distribution D n ∈ P n,m , then M is ( η, ϵ ) -secure against model extraction.
The theorem, to be instantiated, requires that the completeness parameter and soundness parameter are noticeably apart.
Hence, it cannot be instantiated under the (2 ω ( n 12 ) , 2 ω ( n 12 ) )
− SLPN µ,n assumption.
In this work, we have (from a theoretical perspective) explored the problem of defending against model extraction attacks using an observational defense.
Our main contributions are the formalization and abstraction of the theory of observational defenses, and in particular the abstraction of the methods implicitly used by the existing literature for how to choose the right benign client query distributions.
We do this with the goal for obtain provable security via assumptions on the hardness of PAC-learning.
After this, we have used existing natural Covert Learning algorithms to prove that, under cryptographic assumptions, this entire abstract technique for obtaining probable security cannot work for some basic (but natural) special cases of the technique.
We suggest the following two directions for future work.
First, it would be interesting to expand the natural Covert Learning attacks to more distributions than just uniform. Doing so would strengthen our incompleteness result by generalizing the requirement of completeness of the OMED with respect to the uniform distribution. Second, it would be very interesting to expand natural Covert Learning attacks to other real world classes of ML models, such as neural networks, as many other model extraction attacks target neural networks.
As it currently stands, our attacks only work for ML models computable by decision trees.
However, we observe that since the natural Covert Learning algorithm of [13] is actually agnostic , meaning that it finds a hypothesis (nearly) as good as the best decision tree, then if the ML model is a neural network that can at least be (weakly) approximated by a decision tree, then some level of extraction would still be possible (if not with arbitrary accuracy).
To this end, it would also be interesting to explore further to what extent the attack of this paper can be used to (weakly) extract some classes of neural networks.
A PPENDIX A P ROVABLE S ECURITY FROM THE I MPOSSIBILITY OF C OVERT L EARNING
In this section, we show the inverse of Corollary IV.3.
That is, if natural Covert Learning algorithms do not exist, then we can obtain good OMEDs needed for security via
Theorem III.10.
Theorem A.1 .
If there does not exist an efficient natural Covert
Learning algorithm for a concept class C with respect to D = { D n } n ∈ N , then for any n ∈ N , there exists a poly( n ) -OMED
M for a concept class C n that is δ -complete and η -sound with respect to the property { D n } for η ≤ negl( n ) .
Proof.
Since there does not exist an efficient natural Covert
Learning algorithm for the concept class C with respect to
D , then it means that for every learning algorithm
A such that for every f ∈ C n , ϵ, γ > 0 we have that with probability
[Equation]
, then distribution D
A over queries made by A is efficiently distinguishable from
D .
In other words, there exists p.p.t.
Adv such that for S ∼
[Equation] [Equation]
[Equation]
where T ( A O f ) denotes the distribution over the queries made by A and the responses by the oracle.
This implies that we can produce the desired OMED M by using Adv as follows:
1) Estimate Pr Adv ,S
[ Adv ( S )
= 1] within an additive factor of µ/ 4 with probability at least 1 − η/ 2 . 2) Using segments of T ( A O f ) ,
estimate
Pr Adv , T ( A O f ) (cid:2)
Adv ( T ( A O f ))
= 1 (cid:3) within an additive factor of µ/ 4 , with probability at least 1 − η/ 2 . 3) Output “reject” if the two estimates are not within µ/ 2 , and “accept” otherwise.
We analyze this process to prove the desired statement, starting with completeness and soundness.
Without loss of generality, assume that Pr Adv ,S
[ Adv ( S )
= 1] ≥ 1 − δ .
Then, it follows that M is δ -complete with respect to P n,m .
Now, by (1), it must be the case that
[Equation]
Hence, the estimation procedure is guaranteed to distinguish between the two cases with probability least 1 − η (using a union bound).
Using this fact and inspecting step 3 gives ηsoundness.
To see that M is an poly( n ) -OMED specifically, observe that Adv is efficient (and therefore runs in time poly( n ) .
We remark that in the above proof, it is important that
the
OMED has access to poly( n ) · O (log( η − 1 )) examples from the client (in order to complete the estimation procedures).
Ideally, we can reduce this number, as clients may be able to perform model extraction using less queries.