Theoretical Limits of Provable Security Against Model Extraction by Efficient Observational Defenses
Abstract —Can we hope to provide provable security against model extraction attacks?
As a step towards a theoretical study of this question, we unify and abstract a wide range of “observational” model extraction defenses (OMEDs) — roughly, those that attempt to detect model extraction by analyzing the distribution over the adversary’s queries.
To accompany the abstract OMED, we define the notion of complete OMEDs — when benign clients can freely interact with the model — and sound OMEDs — when adversarial clients are caught and prevented from reverse engineering the model.
Our formalism facilitates a simple argument for obtaining provable security against model extraction by complete and sound OMEDs, using (average-case) hardness assumptions for PAC-learning, in a way that abstracts current techniques in the prior literature.
The main result of this work establishes a partial compu- tational incompleteness theorem for the OMED: any efficient
OMED for a machine learning model computable by a polyno- mial size decision tree that satisfies a basic form of completeness cannot satisfy soundness, unless the subexponential Learning Parity with Noise (LPN) assumption does not hold.
To prove the incompleteness theorem, we introduce a class of model extraction attacks called natural Covert Learning attacks based on a connection to the Covert Learning model of Canetti and Karchmer (TCC ’21), and show that such attacks circumvent any defense within our abstract mechanism in a black-box, nonadaptive way. As a further technical contribution, we extend the Covert Learning algorithm of Canetti and Karchmer to work over any “concise” product distribution, by showing that the technique of learning with a distributional inverter of Binnendyk et al.
(ALT ’22) remains viable in the Covert Learning setting. Finally, we further expose the tension between Covert Learning and OMEDs by proving that the existence of Covert Learning algorithms requires the nonexistence of provable security via efficient OMEDs.
Therefore, we observe a “win-win” result, by obtaining a characterization of the existence of provable security via efficient OMEDs by the nonexistence of natural Covert
Learning algorithms.
I. I NTRODUCTION
In a model extraction attack , an adversary maliciously probes an interface to a machine learning model in an attempt to extract the machine learning model itself.
In many cases, preventing model extraction helps increase security and privacy, especially with respect to model inversion and adversarial example attacks (see e.g. [1] and references therein).
Additionally, in Machine Learning as a Service (MLaaS), the model is considered confidential as the server usually operates with a pay-per-query scheme.
Therefore, maintaining the secrecy of ML models by finding effective model extraction defense mechanisms is paramount.
Indeed, the problem of how to defend against model extraction has been considered from a practical perspective previously (e.g. [2]–[6]).
Most proposed model extraction defenses (MEDs) in the literature belong to two types (except a few notable exceptions, see e.g. [7]).
The first type aims to limit the amount of information revealed by each client query.
One intuitive proposal for this type of defense is to add independent noise (i.e. respond an incorrect prediction independently with some probability) or even deliberately modify the underlying model.
This type of solution is not a focus of this work, because it necessarily sacrifices predictive accuracy of the ML model, and is therefore not an option for many ML systems where accuracy is critical such as autonomous driving, medical diagnosis, or malware detection.
The second type of MED that has been proposed aims to separate “benign” clients — those that want to obtain predictions but will not attempt to extract the model — and “adverse” clients — clients that aim to extract the model.
This type of “observational” defense is the focus of the present work (see Figure 1).
A common implementation of the observational defense involves so-called “monitors” that, receive as input a batch of queries submitted by the client, and compute some statistic meant to measure the likelihood of adversarial behavior, with the goal of rejecting a client’s requests when the queries pass a certain threshold on the statistic (e.g. [2], [5], [6]).
Essentially, observational defenses aim to control the distribution of the client’s queries, by classifying any clients that fail to conform to the appropriate distributions as adverse, and then prohibiting them from accessing the model.
To date, the choice of such appropriate distributions has been made heuristically, for instance, in [5], an appropriate distribution is one with the property that the distribution over hamming distances between independent queries is normally distributed.
However, no formal definitions of security against model extraction have been suggested, and there has not been much formal work done in an effort to understand the theoretical underpinnings of the proposed observational defenses.
This is highlighted by Vaikuntanathan as an open problem in [8].
As a result, a “cat-and-mouse” progression of attacks and defenses has developed, while no satisfying guarantees have been discovered (for neither cat nor mouse).
In contrast, a lofty goal inspired by the theory of Cryptography would be to ultimately obtain provable security guar- antees.
For example, an initial attempt could try to leverage zero-knowledge style simulation-based security, to obtain the guarantee that a client learns nothing about the ML model that they could not have already learned prior to the interaction with the model.
However, this is too strong of a goal, because at the very least the client will learn some queried examples.
What kind of guarantees could we feasibly hope to obtain, then?
One possible revised goal, could be to guarantee that a client learns only as much as they could learn from a set of random queries on the model. This privilege constitutes a middle ground between the too restrictive full zero-knowledge guarantee, and allowing a client total query access to the model.
We observe that this notion of security appears to be implicitly behind existing observational defenses.
The literature on practical observational defenses tends to cite the goal of detecting model extraction, but the downstream effect is that the observational defenses seek to exactly confine the queries obtained by the client to some specific distributions (by enforcing a particular benign behavior). The benign behavior is enforced because the OMED will reject the client’s queries if their distribution fails some chosen statistical test.
Hence, the idea of only serving clients confined to these benign query distributions undoubtedly assumes that whatever can be deduced by a benign client about the model is indeed “secure.”
To think about this deeper, let us focus on the case of observational defenses for binary classifiers.
At first glance, the beautiful learning theory of Vapnik and Chervonenkis — which tells us that a number of samples proportional to the VC dimension of the hypothesis class suffices for PAC learning — seems to dash the hopes of using this model of security to obtain any meaningful protection.
Indeed, an adversary with unbounded computational power could simply query the model according to one of the appropriate distributions of random examples for a sufficient number of times, and then apply a PAC-learning algorithm.
The output of the algorithm would be a function which would be a strong approximation to the underlying ML model with high confidence.
However, this view does not account for the complexity of the implied model extraction attack: depending on the complexity of the model, this type of attack may have superpolynomial query and computational complexity.
For instance, for many important families of classifiers (e.g. boolean decision trees), no efficient (i.e., polynomial time) PAC-learning algorithms are known despite intense effort from the learning theory community (though they exist given superpolynomial computational resources).
In fact, no efficient algorithms are known even when the queries are restricted to being uniformly distributed, and the classifier itself is drawn from some kinds of distributions (i.e., in an average-case way, see e.g. [11]).
Hence, this lends credence to the idea that the model of security implicitly considered by observational defenses might actually be effective in preventing unwanted model extraction by computationally bounded adversaries , by forcing the adversary to interact with the query interface in a way that mimics uniformly random examples, or some other hard example distribution.
In this way, security against model extraction by computationally bounded adversaries could be provable in a complexity-theoretic way: one could hope to give a reduction from PAC-learning to model extraction in the presence of observational defenses.
In other words, one could hope to prove a theorem that says “any efficient algorithm to learn an approximation of a proprietary ML model when constrained by an observational defense yields a distributionspecific PAC-learning algorithm (that is currently beyond all known techniques).”
Yet, for classes of ML models that are efficiently PAClearnable, the hope of any OMED is lost because of the VC theory argument described in the earlier in the section.
Thus, for the reasons outlined, this paper focuses on polynomial time adversaries who attempt to extract machine learning models with evidence of (polynomial time) hardness of learning .
A concrete setting within focus that is handled in this paper is a polynomial time adversary trying to steal a polynomial size decision tree.
Since we have established some faith behind the idea that observational defenses might be effective against computationally bounded adversaries (even in a provable way), a natural follow-up question to consider is if and when observational defenses can be efficiently implemented against those efficient adversaries.
Thus, we seek an answer to the following two-part question:
Can we provide provable security against model extraction attacks, for any ML model, using an observational defense?
If so, can it be efficiently implemented?
In this work, we outline a framework for obtaining cryptographic-strength provable security via an observational defense.
The framework is an abstraction of the prevailing heuristics used in the literature on practical observational defenses.
Then, we provide a negative answer to the second part of the question. We do this via the following program:
• We formally define a class of abstract MEDs by unifying the common observational defense technique seen in the literature.
• We formalize the concepts of complete and sound observational defenses, namely, the provable guarantees that benign clients are accepted and may interact with the machine learning interface, and adverse clients are rejected.
We show how our formalisms give a route to obtaining a (very) basic form of provable security against efficient model extraction attacks by relying on the computational hardness of PAC-learning.
Throughout the paper, we argue that the proposed method is a useful abstraction of the methods that are (implicitly) employed by the literature on the construction of observational defenses.
• Via a connection to the Covert Learning model of [13], we give a method for generating provably good and efficient attacks on the abstract defense, granted that the defense is efficient and it satisfies a basic form of completeness.
We then obtain an attack on decision tree models protected by the abstract class of MEDs by an existing algorithm of [13].
The attack relies on the subexponential Learning Parity with Noise (LPN) assumption, and to the best of our knowledge, constitutes the first provable and efficient attack on any large class of MEDs, for a large class of ML models.
• Using the existence of the attack, we prove our main result: informally, every efficient defense mechanism (within the abstract class of MEDs) for decision tree models which satisfies a basic notion of completeness does not satisfy soundness, even for efficient attackers.
This result essentially prevents instantiating the described method for provable security, assuming the basic notion of completeness is satisfied.
Finally, we extend the algorithm of [13] to work in the more general setting of learning with respect to “concise” product distributions, which gives a stronger impossibility result on the viability of efficient OMEDs.
Additionally, we further develop the relationship between Covert Learning and OMED defenses.
As a consolation to the negative result, we prove that “either way science wins”: provable security by OMEDs exist if and only if Covert Learning algorithms do not exist. In the next two sections we discuss related work
, starting
with a discussion on the nature of Covert Learning and the connection
to
model extraction
, and an
overview of the techniques used by [13] to obtain Covert Learning algorithms. We then highlight more related
work including other approaches to security against model extraction and existing
Covert Learning algorithms C. Covert Learning, and the Relationship to Model Extraction in the Presence of Observational Defenses
The Covert Learning model — a variant of Valiant’s PAC model in the agnostic learning with membership queries setting — formalizes a new type of privacy in learning theory.
Specifically, Covert Learning algorithms provide the guarantee that the membership queries leak very little information about the concept, with respect to a computationally bounded passive adversary.
In other words, the learner can PAC-learn the concept in question (using knowledge of some internal randomness only known to itself), 4 while the adversary remains nearly completely “in the dark” with respect to the concept, even given a view of the entire transcript of membership queries and oracle responses. Crucially, the adversary is not privy to the secret randomness of the learning algorithm.
At its heart, the Covert Learning model uses the foundational simulation paradigm of cryptography to achieve these goals.
Roughly, any membership query learning algorithm is a
Covert Learning algorithm if it has an accompanying simulator which, when given access only to random examples (an “ideal” learner), emulates the distribution over the membership queries (the “real” learner) in a computational indistinguishable way.
In other words, the simulator is able to produce a “believable” transcript of the interaction between the learner and the concept oracle, without query access to the underlying concept, but only (for instance) uniformly random examples.
For a concept class where access only to uniformly random examples makes learning computationally hard, this proves that the learning transcript reveals very little information to a computationally bounded adversary.
Hence, Covert Learning algorithms are reserved for concept classes that are not (known to be) efficiently learnable in the original PAC-model with respect to a certain “hard example distribution.”
In this work, we focus on a special case of Covert Learning, which we term natural Covert Learning.
In natural Covert
Learning, rather than making necessary a simulator for the interaction, we strengthen the requirement by asking simply for the distribution over the membership queries to be computationally indistinguishable from some pre-defined “hard example distribution.”
For concreteness, let us consider the uniform distribution as the pre-defined “hard distribution.”
Also, we will consider a version that lives in the realizable learning setting (rather than agnostic).
For this concrete case, Figure 2 provides a graphical depiction.
To explain a bit more formally, fix a concept f : { 0 , 1 } n → {− 1 , 1 } contained in a concept class C , and consider (for concreteness) the uniform distribution U n over { 0 , 1 } n .
A learning algorithm under the Covert Learning model is tasked with finding an hypothesis h : { 0 , 1 } → {− 1 , 1 } that best approximates the concept D on unobserved examples ( x, f ( x )) , for x ∼ U n .
This notion is captured by a loss function, such as L ( h ) =
Pr x ∼ U
[ h ( x ) ̸
= f ( x )] .
A natural Covert Learning algorithm should then satisfy the PAClearning guarantee: output h such that Pr x ∼ U
[ h ( x ) ̸
= f ( x )] ≤ ϵ with high probability (such an h is called ϵ -good).
In order to achieve this goal, the learner is given access to a membership query oracle that labels a queried input x ∈ { 0 , 1 } n with a corresponding label f
( x ) .
However, the important part of natural Covert Learning is that, essentially, the joint distribution over the membership queries made by the learner [ x 1 , · · · x m ] must be indistinguishable from [ x 1 , · · · x m ] ∼ ( U n ) m by any computationally bounded adversary. The following is an informal definition:
Definition I.1 (Natural Covert Learning — informal version of
Definition IV.1) .
A natural covert learning algorithm — for a class of concepts C and a distribution D over examples — is an algorithm that, for any f ∈ C and accuracy parameters ϵ, δ , interacts with an oracle that labels queries to the concept f such that the following are true:
• Learning.
The learning algorithm outputs an ϵ -good hypothesis for the concept with probability 1 − δ .
• Privacy.
The joint distribution over all queries and responses to and from the oracle is computationally indistinguishable from the distribution ( x 1 , f ( x )) , · · · ( x m , f ( x m )) for x 1 , · · · x m ∼ ( D n ) m .
The conceptual crux of this work.
There is a connection between the adversary in Covert Learning — a distinguisher that attempts to distinguish the membership queries made by the learner algorithm from random examples, — and the “adversary” in a model extraction attack — an OMED.
At first glance this sounds confusing, because normally the adversary in model extraction is the malicious client that is trying to reverse engineer the model. However, consider that the job of the OMED is to somehow distinguish benign clients from adverse clients.
Thus, thinking from the perspective of the actual adversary, i.e., the model extractor, the OMED is adversarial in nature and has essentially the same job as the natural Covert Learning adversary: to detect some property about the client’s distribution of queries. This is even more apparent by considering the similarities between Figure 1 and
Figure 2. By exploiting this connection, we can show that natural Covert Learning algorithms form a recipe for an attack that can fool the OMED in the same way that they fool the Covert Learning adversary.
Plus, the Learning guarantee of natural Covert Learning then allows us to still argue that the client can extract the underlying model.
Hence, we can show that a single Covert Learning attack can achieve extraction while “fooling” any OMED.
That is, any OMED will output “accept” with high probability even though an extraction attack is being performed.
Since the attack is completely black-box with respect to the implementation of the OMED (it only requires that the OMED is efficient and satisfies the completeness condition for some basic distributions over queries), the existence of this attack can be used to demonstrate the incompleteness of the OMED. On the need for Covert Learning instead of PAClearning. It is important to understand exactly how Covert
Learning algorithms add to this paper.
In Section I-A, we touched on the fact that all models are at risk of extraction by the very fact that they are exposed by some interface — to repeat, the VC argument outlined in Section I-A shows that, given enough random access to the model, and enough computational power, an adversary has a PAC-learning algorithm to obtain an approximation to the underlying model.
Additionally, since the PAC-learning algorithm only uses random examples, then the adversary can choose to make queries according to a distribution that “passes” the OMED. However, the VC argument in general relies on unbounded computational power (only very simple concept classes are known to be PAC-learnable in polynomial time from random examples, e.g. linear models). Hence, the natural question is whether there exists effective model extraction defenses against polynomial time adversaries, as is customary in Mod- ern Cryptography. Again, this is because many important models (e.g. decision trees, low-depth circuits) are not known to be PAC-learnable in polynomial time.
On the other hand, a class like decision trees is learnable in polynomial time when using carefully synthesized membership queries .
However, classic membership query algorithms like the KushilevitzMansour algorithm [10] (which suffices to learn decision trees in polynomial time) make queries that can be tested and identified as problematic by the OMED.
Therefore, Covert Learning algorithms bridge the gap between these two settings; the Covert Learning algorithms constitute attacks that are benign-looking , but are actually adverse .
In other words, they are polynomial time membership query PAC-learning algorithms that synthesize queries in a special way that is provably hard to distinguish from the classic
PAC-learning with random examples setting.
This means that
Covert Learning attacks are both undetectable by an (efficient)
OMED, and run in polynomial time.
The VC argument is undetectable by the OMED, but is not efficient, while the membership query attack is efficient, but is detectable by the
OMED. Overview of the natural Covert Learning algorithm for decision trees of [13] . Since we use the Covert Learning algorithm for decision trees from [13] as a black box, let us use the remainder of this section to provide a useful overview of the algorithm.
The covert learning algorithm for decision trees begins by using a “masked” Goldreich-Levin algorithm in order to obtain large Fourier coefficients.
The Goldreich-Levin algorithm [9] is an algorithm that selects correlated queries in an ingenious way to decode a noisy parity function.
The algorithm can also be used learn Fourier coefficients large (i.e., 1 / poly( n ) magnitude) of any function f : { 0 , 1 } n → {− 1 , 1 } , in time polynomial in n , and has been said to inspire the similar
Kushilevitz-Mansour algorithm
The “masked query” technique of [13] works as follows: for any given query x ∈ { 0 , 1 } n , generate a pseudo-random string by taking n samples from a carefully crafted Learning
Parity with Noise (LPN) distribution (see [14] for a useful introduction on LPN).
Letting the n samples be concatenated into string y ∈ { 0 , 1 } n , the masked query is taken as the bitwise xor m
= x ⊕ y .
It is easy to see that the LPN distribution is pseudo-random, then so is the masked query m .
Mathematically, the string y can be written as y =
As ⊕ e , where A is a random (but not uniformly random) n × n binary matrix, e ∈ { 0 , 1 } n is a noise vector sampled by taking an n -wise direct product of a biased Bernoulli random variable with small, constant mean, and s ∈ { 0 , 1 } n is secret sampled according to a “chopped tail” n -wise direct product Bernoulli distribution with minimum entropy Θ(log 2 ( n ) /n ) .
This LPN distribution is due to [15], who show its pseudo-randomness assuming the more standard subexponential LPN assumption (defined formally as Definition IV.4).
Then, [13] show that, by augmenting an instance of the
Goldreich-Levin algorithm for learning large Fourier coefficients by masking each query x
i with the mask y i
=
As i ⊕ e i , then the returned labels f
( x i ⊕ y i ) can be post-processed by reincorporating (to the i th query) the dependency on the secret s i , and using the approximate linearity of the LPN distribution and large Fourier coefficients, to access a noisy version of f (denoted ˜ f ) which crucially has the property that any large low-degree Fourier coefficient of f is also large for ˜ f .
The technique fails at extracting large Fourier coefficients of any degree, and instead obtaining only those with degree O (log n ) .
The reason for this is that the noise “overflows” for the higher degree coefficients, while it remains manageable for lowdegree coefficients due to the low minimum entropy nature of the LPN secret.
Unfortunately, attempting to increase the degree bound of O (log n ) by further decreasing the minimum entropy of the secret does not work without breaking the pseudo-randomness of the masks.
Because each query is masked independently by a pseudorandom string, it is straightforward to see how this Covert
Learning algorithm is also a natural Covert Learning algorithm with the uniform distribution as the pre-defined “hard distribution.”
Once the set of large O (log n ) -degree
Fourier coefficients is found, one can estimate the magnitudes (using random examples) of each Fourier coefficient, and them produce a hypothesis which is the sign of linear combination of parity functions.
Under further analysis, the hypothesis obtains agnostic learning guarantees for the hypothesis class of poly( n ) size decision trees.
D. More Related Work
Existing OMEDs . We point the reader to Section III-B for a detailed discussion on some of the practical OMEDs proposed in the literature.
Secure inference for MLaaS . A somewhat related approach to improving the privacy of Machine Learning as a Service (MLaaS) termed “secure inference” has been proposed.
This approach borrows from ideas in the field of Secure
Function Evaluation (where parties can securely compute a function without revealing their inputs), and makes use of garbled circuits [16] or fully homomorphic encryption [17].
However, the principle guarantee of the “secure inference” approach only provides hiding of information about the model beyond what can be deduced from the query and the model’s output.
Hence, a secure inference approach to security against model extraction would implicitly assume (incorrectly) that total leakage from the predictions is little, and that recovering the model from its predictions would be infeasible or impossible.
Therefore, the “secure inference” approach does not properly prevent model extraction when considering clients who repeatedly interact with the service.
More natural Covert Learning attacks . The works of [18] and [19] introduce a method for sampling pairs of matrices ( A, T ) with entries in Z q , such that A is statistically close to a uniformly random matrix, while M 2 is a low-norm, full-rank trapdoor matrix such that A · T is the all zero matrix.
In [8], Vaikuntanathan notes that this sampling algorithm gives an easy, yet somewhat contrived model extraction attack.
In fact, it is a natural Covert Learning attack as well.
More specifically, for any ML model that is essentially a linear function over Z q with added Gaussian noise e ∈ Z mq , 5 the linear function (denoted s ∈ Z nq ) can be extracted by querying sA + e , and then taking ( sA + e ) T = eT , which can then be used to extract e via Gaussian elimination ( T is full rank).
Then, given e , s is easily recoverable.
However, A is statistically close to uniformly random, so the queries are impossible to distinguish from uniformly random queries with any significant advantage.
This statistical Covert Learning attack could rule out even unbounded OMEDs, however it only works for the very narrow class of noisy linear models, which do not appear frequently in practice.
The work of [13] discusses how to similarly sample trapdoors for the low-noise LPN problem of Alekhnovich [20].
The techniques gives rise to another natural Covert Learning attack for an LPN variant of the above setting, however the queries are only computationally close to uniform.
Elaborating on this technique in the present paper would be quite time-consuming, hence we refer the reader to [13].
Related formalisms . We note that the formalisms in this work are inspired by the field of Interactive Proofs [21].
Also, the work of [22], who work on protocols for verifying forecasting algorithms, inspired the drive to prove computational incompleteness theorems in this setting.
E. Organization
We define the abstract OMED and consider provable security in Section III. In Section IV we first show how to obtain a generic attack on the OMED using a generic natural
Covert Learning, and then instantiate the generic implication using the algorithm for Covert Learning of decision trees.
In Appendix B, we extend the Covert Learning algorithm to DNFs and product distributions over queries, and obtain an extension of the incompleteness result of OMEDs to this setting.
Finally, in Appendix C we show that ruling out Covert
Learning suffices to construct an OMED.
II. T ECHNICAL P RELIMINARIES
Due to space constraints, technical preliminaries for this work (mainly consisting of definitions of learning models, computational indistinguishability, and basic Fourier analysis) are deferred to the appendix in Section A.
Other less wellknown definitions are introduced in the body of the paper as needed.
III. T HE A BSTRACT O BSERVATIONAL M ODEL E XTRACTION D EFENSE
In this section, we formally introduce the Observational
Model Extraction Defense (OMED) as a unifying abstraction for the current state of the art MEDs.
The main purpose of this section is to establish a formal framework for considering a model extraction attack by an efficient client in the presence of an OMED, and to explain a simple way to argue for provable security in this framework. The argument for provable security is meant to reflect the ideas that appear implicitly in the literature for practical OMEDs (see e.g. Section III-B). However, as we show later in the paper, it is not likely to be a route to security that can be backed by a provable theory. Furthermore, we do not claim that it is necessarily a very well-modelled way of obtaining security in the first place.
Before we introduce OMEDs formally, let us describe the setting of model extraction in the presence of an OMED in more detail.
At a very high level, the setting begins by fixing an ML model f : { 0 , 1 } n → {− 1 , 1 } (potentially from some restricted class of functions).
Then, we consider the case that an efficient client, can interact via an oracle to the ML model f .
We denote this oracle by O f , and the client by C .
The goal of the benign client is to obtain some predictions f ( x ) for queries x ∈ { 0 , 1 } n .
On the other hand, the goal of the adversarial client is to output a function ˆ f , that approximately minimizes a loss function with respect to the model f .
We will refine the behavior of a client and the way it interacts with the ML model after we define the OMED.
The important part is that the adverse client must be able to perform the extraction in the presence of the OMED.
In particular, the OMED is able to view all the queries made by
C and the labels that would be returned (see Figure 3).
The
OMED then (after performing arbitrary computations) outputs a decision “accept” or “reject.”
In the case of “accept,” the
OMED also forwards the labels back to the client.
Otherwise, no labels are returned back to the client.
With this in mind, let us now formally define the OMED.
As noted in the introduction and in [23], defense mechanisms for model extraction have mostly split into two tribes: reducing the information gained per client query, and differentiating malicious extraction adversaries from benign users.
The OMED mechanism abstracts the latter approach; the implementation is left unspecified.
Hence, we define an OMED abstractly as follows.
Let F = {F n } n ∈ N be a class of ML models, where each F n in the sequence is a set of functions taking as input an element of the set { 0 , 1 } n and giving output in the set {− 1 , 1 } .
Definition III.1 (OMED) .
A probabilistic algorithm M is a
T ( n ) -OMED for a class of ML models F if for every n ∈
N , f ∈ F n , M runs in time T ( n ) and takes as input a list of examples S =
[( x 1 , f ( x 1 )) , · · · ( x m f ( x m ))] ⊆ ( { 0 , 1 } n × {− 1 , 1 } ) m and outputs a decision σ ∈ { accept , reject } .
Discussion: sybil attacks, and other restrictions on the nature of the syntax of OMED .
The definition of the OMED is defined as generally as possible from the perspective of the defense, but makes a restriction on the client: the queries are requested in large batches, rather than as an adaptive sequence.
This nonadaptive setting (with respect to the query selection) can be viewed as unnecessarily restrictive on the client.
However, since we prove negative results on the possibility of MEDs via OMEDs, the restriction on the client actually strengthens our results.
We note that, as we will see in the next section, a benign client can arguably be considered one who essentially requests queries selected independently from the same distribution.
In this case, adaptive query power inherently does not add strength to the client.
Furthermore, the defense could be expanded to a multiclient setting, where all clients must submit their batches simultaneously.
This is again a restriction on the power of the client(s) (as opposed to multi-clients who do adaptively make batch requests), and thus strengthens our negative results.
Sybil attacks . Finally, we mention that with respect to sybil attacks, the underlying assumption for a sybil attack is that the MED will not evaluate the queries requested by each identity jointly (perhaps because the defense is assumed not to know the identities of the participating attackers).
However, we prove our negative result on an OMED that always has access to all queries made by the attacker(s).
Hence, our attack works in a more restrictive setting than sybil attacks.
Alternatively, one can view our attack as a 1-sybil attack, or a sybil attack that works even when the defense knows the identity of every sybil participating in the attack.
A. A Potential Route to Provable Security via OMED
Definition III.1 for the OMED makes no claims about desirable properties given by the OMED.
Thus, what properties should we expect from the OMED?
As mentioned in
Section I-A, the goal of the OMED is not just to classify the behavior of the clients, but to actually confine the clients to certain predefined benign behaviors.
However, it is not enough to simply define security as the event that the client behavior is benign, because this actually needs to be detected and then enforced.
Therefore, it should be that a good OMED guarantees that (with high probability) any benign client is accepted, while any adverse client is rejected (and thus prevented from reverse engineering the underlying model).
Naturally, the former requirement resembles completeness in an interactive proof system while the latter requirement resembles soundness.
Through this lens of Interactive Proofs, we will formalize a notion of completeness and soundness.
First, let us explain how we model a client’s behavior in the context of the model extraction setting depicted in Figure 3 in more detail.
It has been noted in the model extraction literature (e.g. [5]) that defenses should consider how a client’s queries relate to each other, rather than how they look individually.
This idea is implemented by assuming that a client’s queries follow a distribution 7 .
We adopt a similar idea in this work: we assume that a client’s queries follow a distribution P over the domain of the ML model.
To this end, we now identify a client C as a tuple ( C, L ) consisting of a polynomial time samplable distribution over sets of m examples C , as well as a probabilistic polynomial time algorithm L that takes as input a list of m examples drawn according to C and labelled by the ML model f , and outputs an approximate model
ˆ f .
Thus, informally, the our idea is that a benign client request examples according to distributions that share some abstract property.
We may formalize this as follows.
Let ∆( { 0 , 1 } n × {− 1 , 1 } ) be the convex polytope of all distributions over { 0 , 1 } n × {− 1 , 1 } .
Definition III.2 .
P n is a property of distributions, where P n ⊆ ∆( { 0 , 1 } n × {− 1 , 1 } ) .
We then define a benign client as one which samples its queries in an independently and identically distributed fashion from some distribution satisfying the property:
Definition III.3 .
Fix any ML model f .
We say that a client
C =
( C, L ) is P n -benign if the distribution C is an m -wise direct product of samples drawn i.i.d. from a distribution D such that D ∈ P n . The i.i.d.
restriction on the benign client may be considered harsh, but is rooted in the reality that often a real-world benign client requires labels from some “natural” set of examples.
For instance, for an image recognition model, the benign client may simply forward requests for labels on images appearing in i.i.d.
fashion according to some organically occurring distribution in Nature.
We will also use the terminology P n -adverse client to describe a client that is not P n -benign. Note that, a P nadverse client need not sample queries i.i.d. from the same distribution; the set of m queries can be sampled using correlated randomness.
Completeness .
Towards obtaining a theory of provable security against model extraction, we now argue that, intuitively, a good OMED (and any MED) should satisfy, informally,
MED C OMPLETENESS :
For any benign client, the defense mechanism does not reject and allows the client to continue to interact with the model, with high probability.
A completeness requirement on a MED can be interpreted as formalizing the usefulness of the defense.
In other words, the defense at the very least provides the opportunity for benign clients to interact with the model.
Formally, we define completness as follows:
Definition III.4 (MED completeness) .
We say that an OMED
M is δ -complete with respect to the property P n if for any P n -benign client C = ( C, L ) ,
[Equation]
Provable security against benign clients: How should one choose the benign property?
The definition of completeness implicitly assumes that, in the case that the client is classified as benign, they are essentially free to interact with the model.
Thus, the underlying assumption is that by virtue of the client being benign, the ML model is not considered at risk for being extracted by the server.
Therefore, the choice of the benign property is of utmost importance.
As discussed in the introduction (see Section I-A), this leaves room for a theory of provable security.
For example, under our framework, a solid choice for a property would be the P n = U n (i.e., the uniform distribution over examples).
P n = U n makes a solid choice because many interesting classes of models are thought to be hard to learn from uniformly random examples, even for most models in the class (that is, in the heuristic PAC-learning case, rather than just in the worst-case).
Using this idea, we can give a reduction from extracting the
ML model (in the average-case over the uniform distribution) using U n -benign queries to heuristic PAC-learning with respect to the uniform distribution.
To formalize this intuition, we prove the following lemma, which essentially says that if PAC learning is impossible for a large fraction of the class, then, given that the OMED accepted a client that used U n -benign queries, most models can not be extracted (with arbitrarily high fidelity) except with negligible probability.
After giving the lemma we will provide more color by further discussing the reasons behind defining security this way.
We will later use the lemma to show how to obtain a formal notion of security against model extraction by all clients, whether adverse or benign.
We first need to establish what constitutes a successful model extraction.
Similarly to [4], we define the following extraction experiment.
Again, let F n be a class of boolean ML models, and let C =
( C, L ) be a client.
Fix a loss function
L D,f : F n → [0 , 1] parameterized by an ML model f ∈ F n , and a distribution D over { 0 , 1 } n .
Definition III.5 (Extraction experiment) .
Let Exp C ,f,m,ϵ be defined as the output of the following process.
1) Sample x 1 · · · x m ∼ C for C = ( C, L )
These examples are paired with the labels to produce the set S =
[ x i , f ( x i )] i ∈ [ m ] .
Run L ( S ) to obtain ˆ f . 3)
If L C,f ( ˆ f ) < ϵ , output ( S, extracted) , else output ( S, unextracted) .
Lemma III.6 (Provable security against clients behaving benignly) .
Let M be any OMED satisfying δ -completeness for any δ ≥ 1 − 1 / poly( n ) , with respect to a property of distributions P n .
Then, if a class of ML models F n is not efficiently ( η, ϵ ) -heuristic PAC-learnable with respect to any of the singleton distribution classes in the set {{ D } : D ∈ P n } , then there exists F ⊆ F n of size at least η ·|F n | , such that for any client C
=
( D, L ) that is P n -benign, and for all f ∈ F ,
[Equation]
Proof.
We show the contrapositive.
Let E, A be the events (over the randomness of Exp C ,f,m,ϵ ) that τ = extracted and
M ( S ) = accept , respectively.
Thus, suppose that for some
D ∈ P n , we have a client C
= ( D, L ) such that for ϵ > 0 , and a δ -complete M ,
[Equation]
This implies that
[Equation]
which is equivalent to
[Equation]
since M is δ -complete (the inner probabilities are all over the randomness of M , ( S, τ ) ∼ Exp C ,f,m,ϵ ; this is omitted due to space constraints).
We now may observe that latest equation is the guarantee that there is an algorithm that ( η, ϵ )
-heuristically
PAC learns F n in time poly( n ) with accuracy ϵ and confidence (1 − δ ) / poly( n )) , for some singleton distribution class { D } (where D ∈ P n ).
The learning algorithm works by sampling a set of labelled examples T according to D , and the applying
L ( T ) to obtain a hypothesis.
Then, it follows that F is ( η, ϵ ) -heuristically PAC-learnable with respect to some singleton distribution class { D } (where
D ∈ P n ) by running the learning algorithm poly( n ) times to produce many hypotheses, testing each by random sampling, and outputting the most accurate hypothesis.
This works as long as δ ≥ 1 − 1 / poly( n ) .
This completes the proof by contrapostive, which only requires we show learnability with respect to at least 1 singleton distribution class { D } for
D ∈ P n .
Discussion: We reiterate that the preceding lemma shows that, under an assumption that a class of ML models is hard to learn in the heuristic sense (i.e., most of models in the class are hard concepts) with respect to some property of example distributions, then clients that are benign on that property of distributions cannot extract most of the model in the class.
This provides a theoretical foundation for instantiating the OMED paradigm, which assumes that some distributions are indeed benign, based off of hardness of heuristic PAClearning assumptions.
On the other hand, we remark that it is unfortunately not realistic to require hard extraction for all models, as many natural classes of models in the class (even for classes that are not even known to be efficiently
PAC-learnable, e.g. decision trees), because natural classes still contain very simple models that can be learned super efficiently (e.g. a decision tree that always outputs the label 1).
However, the model owner can make the assumption that their (fixed) model is inside the large set of hard-to-learn models over the benign distributions.
Indeed, it can be argued that if the model owner is not willing to make this assumption, then the model is not suitable for the OMED paradigm in the first place, since complete OMEDs (with respect to benign property
P n ̸ = Ø ) allow the client to obtain some random examples by definition.
Soundness .
The notion of provable security from
Lemma III.6 only deals with clients that request sets of examples that are P n -benign.
Thus, we need to provide some guarantees when this is not the case.
To this end, we also formalize soundness using the above extraction experiment.
Informally,
MEDS OUNDNESS : For any adverse client that has attempted extraction, the defense mechanism rejects the client, with high probability.
Definition III.7 (MED soundness) .
We say that an OMED M for F n is δ -sound with respect to the property P n if for any
P n -adverse client C ∗ =
( C ∗ , L ) , and for any f ∈ F n , ϵ > 0 , it holds that
[Equation]
The soundness requirement can be interpreted as formalizing the security of the defense.
That is, attackers will not be able to deceive the OMED into granting interaction with the
ML model in such a way that allows extraction.
The definition also mirrors soundness from an Interactive
Proof, where for any client that is not P n -benign, and given that the adversary would have extracted the model, the probability that the OMED M errs by accepting the queries is low.
In a cryptographic setting, the OMED would set δ to a negligible function of n .
Cryptographically-Hard Model Extraction Against All
Clients. To tie it all together, we now argue how combining completeness, soundness, and hardness assumptions for heuristic PAC-learning give a method for obtaining provable security via OMED. We stress that this argument is meant as a unifying abstraction to back up specific and practical implementations of OMEDs — not as a silver bullet for the model extraction problem.
Our notion of security against model extraction (defined below) constitutes bounding the probability that a client wins the extraction game by a negligible function, in this case of n (the size of the learning problem).
Definition III.8 (Security against Model Extraction by all clients) .
We say that an OMED M for F is ( η, ϵ ) -secure against model extraction if for sufficiently large n , there exists
F ⊆ F n of size at least η · |F n | , such that for any p.p.t.
client
C =
( C, L ) and for all f ∈ F, D n ∈ P n ,
[Equation]
We prove that, essentially, for any complete and sound,
OMED M for a class of ML models F n , if F n has no ηheuristic PAC-learning algorithm, then at least η fraction of models in F n can not be extracted by any client, no matter how it behaves, except with negligible probability. The point of this is that the therefore the strength of the hardness assumption on heuristic PAC-learning, i.e. the size of η , directly translates to a better guarantee of hardness of extraction.
Theorem III.9 (Provable Security from Complete and
Sound OMEDs) .
Let M be any p.p.t.
OMED satisfying δcompleteness and γ -soundness for any δ ≥ 1 − 1 / poly( n ) , γ ≤ negl( n ) , with respect to a property of distributions P n .
Then, if a class of ML models F n has no p.p.t.
( η, ϵ ) -heuristic PAClearning algorithm with respect to any distribution D n ∈ P n , then M is ( η, ϵ ) -secure against model extraction.
Proof.
By definition, any client C =
( C, L ) is either P nbenign or P n -adverse.
In the former case, Lemma III.6 implies that
[Equation]
In the latter case, we have the guarantee from Definition III.
that
[Equation]
where γ is a quantity bounded above by a negligible function of n .
Therefore the statement follows.
B. Defense Proposals As Special Cases of the OMED
Our proposed OMED technique for provable security against model extraction is purposely defined as generally as possible.
However, we view it beneficial to discuss the relation to some concrete MEDs which have been proposed.
In this section, we will review three MEDs, [2], [5] and [6], demonstrating that each are special cases of an OMED (with unproved completeness and soundness guarantees).
We start each example with a direct quote from the original paper so as to directly demonstrate the relevance to our OMED framework.
1) Extraction Monitors: Information Gain and Feature
Space Coverage:
The work of [2] proposes two different strategies for detecting model extraction attacks.
Both strategies “[quantify] the extraction status of models by continually observing the API query and response streams of users” and provide a warning when a certain extraction status is reached.
This is indeed the paradigm outlined by the OMED.
The first proposal of [2] seeks to continuously train a “proxy model” for each client, where the client queries are used to train the model.
The function of the proxy model is to estimate the information/knowledge gained by a client with respect to a validation set which is given by the server (and when this information reaches some threshold the client is flagged).
The distribution of this validation set mimics the training set of the underlying model.
It is noted that it may require significant computational resources to train and update the proxy model (for every user and each incoming query), and thus [2] propose to use a lightweight decision tree proxy model.
In the second proposal, the observational keeps a short description of client queries, and estimates the client’s learning rate (of the extraction attack) by analyzing the feature space covered by these queries (as they relate to to the class boundaries of the underlying model.
It is noted that a drawback of this proposal is that the class boundaries of certain complex models (e.g. neural networks) are not easily found.
Thus, it is proposed that the owner of the underlying model uploads a “surrogate” decision tree which has high fidelity with respect to the complex model (class boundaries of decision trees are easily interpreted by their leaf nodes).
2) PRADA:
The MED known as PRADA [5] “analyzes the distribution of consecutive API queries and raises an alarm when this distribution deviates from benign behavior” [5].
Immediately, it is clear that the PRADA method is a candidate for being identified as a special case of the OMED.
The defense works under the observation that queries requested by an adversarial client are likely to have a distribution that differs from the characteristic distribution of queries from a benign client.
In PRADA, this benign characteristic was chosen to be the property that the distribution over hamming distances between each query in the requested batch should be normally distributed.
This choice is backed by observational evidence that certain popular attacks such as the attack of [24] do not satisfy this condition.
Hence, PRADA tries to satisfy completeness and soundness with respect to the property of all distributions that have a pairwise hamming distance normally distributed (e.g. the uniform distribution over { 0 , 1 } n ).
3) VarDetect: The work of [6] proposes a MED called
VarDetect which is designed “to continuously observational the distribution of queries to [the model] from each user” [6].
Specifically, VarDetect trains a Variational Autoencoder (VAE) to map the “problem domain” (PD) dataset distribution (the
PD distribution mimics the distribution of data that was used to train the underlying model) and the adversarial “outlier” (O) data distribution (the distribution of attacker queries) to distinct regions in latent space.
Benign clients are assumed to query from the PD distribution while adverse clients are assumed to query from an O distribution.
VarDetect purports to separate these two by computing the maximum mean discrepancy (MMD) between the latent mapping of the client’s queries and that of the PD distribution (the MMD test flags the client if the result is above a certain threshold).
IV. A TTACKS ON E FFICIENT OMED S FROM N ATURAL C OVERT L EARNING
In this section, we will consider the question:
Can we efficiently realize the provable security guarantees outlined in the previous section?
Towards a negative answer, we will introduce an attack on the OMED technique for provable security, via a connection to
Covert Learning [13].
Our attack will generate a distribution of examples which is computationally indistinguishable from a distribution in the property that is accepted by the OMED.
In other words, the attacker operates (computationally) indistinguishably from a benign client, in the eyes of the OMED.
Still, the attack classifies as adverse, and the labelled queries allow the attacker to extract a model with high accuracy.
A. Natural Covert Learning
We will focus on a special case of Covert Learning, which we call natural Covert Learning .
As described in
Section I-C, a natural Covert Learning algorithm, essentially, is a membership query learning algorithm that satisfies the normal PAC-learning guarantees with respect to an example distribution D , with the added property that distribution over the membership queries and labels requested by the algorithms is computationally indistinguishable from examples sampled according to D .
More formally, let C =
{C n } n ∈ N be a boolean concept class, and let D = { D n } n ∈ N be a sequence of distributions where each D n is a distribution over { 0 , 1 } n .
Definition IV.1 (Natural Covert Learning) .
We say that A is an natural Covert Learning algorithm for C with respect to D if for every n ∈ N , f ∈ C n , ϵ, δ > 0 , A satisfies the following when given membership query access to f :
• Learning.
For the random variable h =
A O f ( n, ϵ, δ ) we have
[Equation]
• Privacy.
Let the query complexity of A be m .
For every p.p.t.
adversary Adv , and S ∼ E X ( f, D n , m ) ,
[Equation]
where T ( A O f ) denotes the distribution over the queries made by A and the responses by the oracle.
We say that A is efficient if it runs in time polynomial in n, ϵ − 1 , δ − 1 , and the number of queries m is bounded by polynomial in n, ϵ − 1 , δ − 1 .
We note how natural Covert Learning is defined here with respect to a sequence of example distributions, rather than a sequence of sets of example distributions.
This keeps the definition simpler.
One can consider for example natural covert learning algorithms for C with respect to the sequence of uniform distributions {U n } n ∈ N , each over { 0 , 1 } n .
B. Natural Covert Learning Attack
In this section, our goal is to show that the existence of a natural Covert Learning algorithm for a particular concept class implies inadequacy of a polynomial time OMED for a class of models equal to the concept class.
More specifically, we prove that satisfying soundness for an OMED is impossible, for any reasonable completeness parameter.
This suffices to rule out obtaining provable security against model extraction by an OMED by instantiating Theorem III.9.
Theorem IV.2 .
Suppose that there exists an efficient natural
Covert Learning algorithm A (with query complexity m ) for a concept class C , with respect to the sequence of distributions
D = { D n } n ∈ N .
Then there exists an adverse client C such that for any n ∈ N , any poly( n ) -OMED M for C that is δ -complete with respect to a property P n that contains E X ( f, D n , m ) , and any ϵ, δ A > 0 , and f ∈ C n ,
[Equation]
where negl( n ) denotes a negligible function of n and δ A is the failure probability of A .
Proof.
Let D A be the distribution over the set of m examples which is queried by A .
Define the adverse client C ∗ =
( D A , A ) .
Let S ∼ E X
( f, D n , m ) and S A ∼ E X ( f, D A , m ) .
Using the fact that by our Covert Learning assumption we have that for every p.p.t.
adversary Adv ,
[Equation]
and M , f are polynomial time computable, we get that
[Equation]
Then, we can conclude by independence of events
[Equation]
The last equation follows by the δ -completeness property of
M , and the fact that the covert learning algorithm succeeds to extract under the experiment with probability at least 1 − δ A .
Therefore, C ∗ is a client that satisfies the implication in the theorem statement.
Now from this attack, a generic incompleteness theorem follows, essentially because the attack works against any efficient OMED.
Corollary IV.3 .
Suppose that there exists an efficient natural
Covert Learning algorithm A (with query complexity m ) for a hypothesis class C with respect to the sequence of distributions
D = { D n } n ∈ N .
Then if for any n ∈ N , M is a poly( n ) -
OMED for C n and is δ -complete with respect to a property { E X ( f, D n , m ) } , then M is not (1 − δ − negl( n )) -sound with respect to P n .
Proof.
By Theorem IV.2 there exists a { D n } -adverse client
C =
( D A , A ) such that for any δ -complete OMED M (with respect to { D n } ) for C n , it holds that for any f ∈ C n , ϵ > 0 ,
[Equation]
where negl( n ) is a negligible function of n and δ C is the probability C fails to extract.
δ C =
δ A is derived directly from δ A , the failure probability of the natural covert learning algorithm.
Then, we can deduce that
[Equation]
Thus by taking an appropriately large n , M cannot be (1 − δ − negl( n )) -sound for a reasonable choice of δ A such as 1 / 100 , when creating the client C =
( D A , A ) based on A .
C. Concrete Attack and Incompleteness Theorem
In this section, we show that under the subexponential hardness assumption on the standard LPN problem, there exists an attack of the type outlined in the previous section.
Let us first formally introduce our assumption.
Below, let β m ( n ) µ denote the m ( n ) -wise direct product distribution of Bernoulli random variables each with mean µ ∈ [0 , 1] .
Definition IV.4 (Search LPN assumption) .
For µ ∈ (0 , 0 . 5) , n ∈ N , the ( m ( n ) , T ( n )) − SLPN µ,n search assumption states that for every inverter I running in time T ( n ) , [Equation] [Equation] n
[Equation]
where s $ ← Z n 2 , A $ ← Z m ( n ) × n, e $ ← β m ( n ) µ .
Thus, the assumption we adopt is the
[Equation]
SLPN
[Equation]
assumption. The following theorem is implicit in [13].
Let DT [poly( n )] be the set of all decision trees of size poly( n ) .
Theorem IV.5 (Covert Learning of decision trees from [13]) .
Given query access to a function f : { 0 , 1 } n → {− 1 , 1 } , there exists an algorithm A running in time poly( s, 1 /ϵ, log(1 /δ )) and making q ( n ) = poly( n, 1 /ϵ, log(1 /δ ))
query accesses such that, unless the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption does not hold, 1) (Learning) A outputs h : { 0 , 1 } n → {− 1 , 1 } such that
[Equation]
with probability 1 − δ . 2) (Privacy) The distribution over examples requested by A is computationally indistinguishable, but statistically distinguishable, from E X ( f, U n , q ( n )) .
We may now proceed to combine Theorem IV.2 and Theorem IV.5 to obtain a concrete natural Covert Learning attack on models implemented by decision tree classifiers of polynomial size.
Let U =
{U n } n ∈ N , and let L D,f ( h ) =
Pr x ∼ D [ h ( x ) ̸ = f ( x )] .
Theorem IV.6 .
Under the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption, there exists, for any n ∈ N , there exists a U n -adverse client C (for some m = poly( n ) )
such that for any δ -complete
OMED M (with respect to {U n } ) for DT [poly( n )] , it holds that for any f ∈ DT [poly( n )] ,
[Equation]
Proof.
Observe that the algorithm described in Theorem IV.
constitutes a natural Covert Learning algorithm for C with respect to U , and loss function L D,f ( h ) =
Pr x [ h ( x ) ̸ = f ( x )] .
We can set the failure probability of the underlying natural covert learning algorithm for the client to 0.99.
Thus, the statement follows directly from Theorem IV.5 and Theorem IV.2.
Incompleteness Theorem . The previous theorem can be interpreted as the existence of an attack against any OMED for decision tree classifiers with δ -completeness with respect to the uniform property (up to SLPN assumptions).
Hence, we get the following corollary.
Corollary IV.7 .
Under the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption, if M is a poly( n ) -OMED for DT [poly( n )] and
U , then if M is δ -complete
it is not (1 − δ − negl( n )) -sound.
Proof.
By Theorem IV.6, there exists, for any n ∈ N , a U nadverse client C such that for any δ -complete OMED M (with respect to {U n } ) for DT [poly( n )] , it holds that for any f ∈
DT [poly( n )] ,
[Equation]
Then, we can deduce that
[Equation]
which completes the proof.
As an example, our result shows that for any reasonable choice of δ for uniform completeness (e.g. 0.1, meaning that at most 0.1 honest clients are incorrectly rejected), then it is impossible to obtain any even remotely useful soundness guarantee.
Using δ = 0 . 1 , our result shows that not even 0 .fraction of extraction attempts will be detected by the OMED.
More generally, let us recall the provable security theorem, restated below.
Theorem IV.8 (Restated Theorem III.9) . Let M be any p.p.t.
OMED satisfying δ -completeness and γ -soundness for any δ ≥ 1 − 1 / poly( n ) , γ ≤ negl( n ) , with respect to a property of distributions P n .
Then, if a class of ML models F n has no p.p.t.
( η, ϵ ) -heuristic PAC-learning algorithm with respect to any distribution D n ∈ P n , then M is ( η, ϵ ) -secure against model extraction.
The theorem, to be instantiated, requires that the completeness parameter and soundness parameter are noticeably apart.
Hence, it cannot be instantiated under the (2 ω [Equation] n [Equation]
) , 2 ω [Equation] V. C ONCLUSION
In this work, we have (from a theoretical perspective) explored the problem of defending against model extraction attacks using an observational defense.
Our first contribution is the formalization and abstraction of the theory of observational defenses, and in particular the abstraction of the methods implicitly used by the existing literature for how to choose the right benign client query distributions.
We do this with the goal for obtain provable security via assumptions on the hardness of PAC-learning.
Our main conceptual contribution connects existing natural Covert Learning algorithms to the Model Extraction problem to prove that, under cryptographic assumptions, the technique for obtaining provable security via OMEDs cannot work for some natural special cases of the technique (monitoring for uniform and product distributions over examples). Finally, the extension of Covert Learning to product distributions is our main technical contribution.
We suggest the following two directions for future work.
First, it would be interesting to expand the natural Covert
Learning attacks to more distributions than just product distributions.
Doing so would strengthen our incompleteness results further by generalizing the requirements of completeness of the OMED with respect to product distributions.
Second, it would be very interesting to expand natural Covert Learning attacks to other real world classes of ML models, such as neural networks, as many other model extraction attacks target neural networks.
As it currently stands, our attacks only work for ML models computable by polynomial size decision trees or DNFs.
However, we observe that since the natural Covert Learning algorithm of [13] is actually agnostic , meaning that it finds a hypothesis (nearly) as good as the best decision tree, then if the ML model is a neural network that can at least be (weakly) approximated by a decision tree, then some level of extraction would still be possible (if not with arbitrary accuracy).
To this end, it would also be interesting to explore further to what extent the attack of this paper can be used to (weakly) extract some classes of neural networks. A PPENDIX A T ECHNICAL P RELIMINARIES
We use the notion of decision trees and disjunctive normal form formulas (DNFs).
A DNF is function represented by an OR of ANDs. For example, f ( x ) = ( x 1 ∧ x 7 ) ∨ ( x 5 ∧ ¬ x 2 ∧ ¬ x 1 ) ∨ ( ¬ x 4 ∧ x 2 ) where f : { 0 , 1 } n → { 0 , 1 } . The class of DNFs also corresponds to depth AC 0 , that is, depth 2 alternating circuits with AND/OR/NOT gates. The size of the DNF is the number of ANDs (also the number of terms, or disjunctions), while the width of a DNF is the maximum number of variables over all terms.
For the definition of binary decision tree, we refer the reader to Chapter 3 of [25].
The size of a decision tree is identified with the number of leaves on the tree.
A. Computational Indistinguishability
We will use the following standard of computational indistinguishability.
Definition
A.
(Computational
Indistinguishability) .
Let { X n } , { Y n } be sequences of distributions with X n , Y n ranging over { 0 , 1 } m ( n ) for some m ( n ) =
n O (1) .
{ X n } and { Y n } are computationally indistinguishable if for every polynomial time algorithm A and sufficiently large n ,
[Equation]
Often, n is clear from the context, so the subscript is omitted.
B. Learnability
We consider two notions of learnability of boolean concepts.
A boolean concept class C = {C n } n ∈ N is a sequence of sets where C n is a set of boolean functions each taking x ∈ { 0 , 1 } n as input, and outputting a label y ∈ {− 1 , 1 } .
Similarly, a distribution class D = {D n } n ∈ N is a sequence of sets of distributions, where each D n is a set of example distributions over { 0 , 1 } n .
We define the following learning models that are considered in this work.
The oracle E X ( f, D ) samples x ∼ D (for a distribution D over { 0 , 1 } n ) and returns ( x, f ( x )) .
Definition A.2 (Efficient PAC Learning) .
A boolean concept class C is PAC-learnable with respect to a distribution class
D if there exists an algorithm A such that for any n ∈ N , distribution D ∈ D n , concept f ∈ C n , when A is given as input n and ϵ, δ > 0 , plus access to E X ( f, D ) , it outputs a function h such that
[Equation]
We say that C is efficiently PAC-learnable with respect to
D if A runs in time polynomial in n, ϵ − 1 , δ − 1 , and the number of accesses to E X ( f, D ) is bounded by a polynomial in n, ϵ − 1 , δ − 1 .
The following definition of heuristic PAC learning due to
Nanashima [12] can be seen as a variant of many existing average-case learning models, where the distribution over concepts is fixed to be uniform.
In his original work, Nanashima defines a distribution over representation strings over concepts, but in this work it suffices to consider a distribution over actual concepts.
The definition can also be interpreted as requiring
PAC-learning for all but some fraction of concepts in the class, where this fraction is given as an input to the learner.
Definition A.3 (Efficient Heuristic PAC-learning — adapted from [12]) .
Let C =
{C n } n ∈ N be a boolean concept class, and let U n be the uniform distribution over C n .
We say that C is heuristically PAC-learnable with respect to the distribution class D if there exists an algorithm
A such that for any n ∈
N , D ∈ D n , when A is given as input n and ϵ, δ, η > 0 , plus access to E X ( f, D ) for some f sampeld uniformly at random from C n , it outputs a function h such that
[Equation]
We say that C is efficiently heuristically PAC-learnable with respect to D if A runs in time polynomial in n, η − 1 , ϵ − 1 , δ − 1 , and the number of accesses to E X ( f, D ) is bounded by a polynomial in n, η − 1 , ϵ − 1 , δ − 1 .
Additionally, we say that C is (efficiently) ( ϵ ′ , η ′ ) -heuristically
PAC-learnable if we have fixed the parameters
ϵ
=
ϵ ′ , η = η ′ .
C. Basic Fourier Analysis
Let f : { 0 , 1 } n → R .
In particular, we consider functions f : { 0 , 1 } n → {− 1 , 1 } .
Definition A.4 .
For S ⊆ [ n ] , we define χ S : { 0 , 1 } n → R by
[Equation]
Fact
[Equation]
Definition A.6 .
The Fourier expansion of f : { 0 , 1 } n → R is
[Equation]
where ˆ f ( S ) is the Fourier coefficient on S .
A PPENDIX B E XTENSION TO P RODUCT D ISTRIBUTIONS
In this section, we show how to extend the result of [13] (expressed as Theorem IV.5 in this work) to “concise” product distributions over examples, while learning is with respect to the concept class of polynomial size disjunctive normal form formulas (DNFs).
The reason we work with DNFs rather than decision trees (which are a weaker complexity class) is for technical facility later in the section.
The caveat is that the resulting algorithm is in the realizable setting rather than agnostic.
More generally, we show how to construct a natural Covert
Learning algorithm for a concept class C with respect to any ensemble of “concise” product distributions over { 0 , 1 } n , given any natural Covert Learning algorithm for a concept class C ′ over the uniform distribution, provided that C ′ satisfies some efficient “closure” property when composed with a sampling machine for the product distribution.
This general reduction then implies the special case by incorporating the algorithm of [13] that witnesses Theorem IV.5.
Using this extended natural Covert Learning algorithm, we also get a new impossibility result on the viability of OMEDs that monitor product distributions.
Before stating and proving the results, we introduce the necessary tools and definitions.
First, we restrict our attention to “concise” product distributions, which are those that sample x ∈ { 0 , 1 } n by sampling each bit x 1 , · · · x n independently from possibly different Bernoulli random variables, each with mean representable by a limited number of bits.
Definition B.1 ( k -concise product distributions) .
A k -concise product distribution µ p,k,n over { 0 , 1 } n is identified by a list p
= ( p 1 , · · · , p n ) of n k -bit strings, where the distribution µ p,k,n samples x ∈ { 0 , 1 } n by sampling x i to be 1 with probability p i / 2 k and 0 with probability 1 − p i / 2 k , where p i is interpreted as a k -bit integer.
We denote by µ p,k = { µ p,k,n } n ∈ N the ensemble of k -concise product distributions.
We say that the distribution µ p,k,n is concise if k = O (1) .
Now, we prove the following theorem, that gives a reduction from natural Covert Learning with respect to any ensemble of concise product distributions to natural Covert Learning with respect to the uniform distribution.
The reduction also assumes that the concept remains within the designated class, even when composed with the sampling algorithm for the product distribution.
Note, each k -concise product distribution µ p,k,n can be sampled by a (multi-output) DNF ¯ µ p,k,n , which takes as input kn random bits.
Definition B.2 (Closure under composition of sampling) .
We say that a concept class C is closed under composition with µ p,k,n if for any f ∈ C , the function g : { 0 , 1 } poly( n ) → {− 1 , 1 } defined by g ( x ) =
f
(¯ µ p,k,n ( x )) remains contained in
C .
Theorem B.3 (Natural Covert Learning over concise product distributions) .
Let C be a concept class that is closed under composition with µ p,k,n .
If there exists a natural Covert
Learning algorithm for the concept class C with respect to the uniform distribution ensemble U , then there exists a natural
Covert Learning algorithm for C with respect to any ensemble of concise product distribution µ p,k , where k
= O (1) .
The proof of this theorem can be sketched as follows.
We will use an object called a distributional inverter for concise product distributions, which (roughly) is an algorithm that takes as input a sample from x ∼ µ p,k,n , and outputs a uniformly random pre-image z of x , under the sampling function ¯ µ p,k,n .
In other words, the distributional inverter outputs a “fake” sample of coins z used to sample the instance x ∼ µ p,k,n (hence ¯ µ p,k,n ( z ) = x ).
Using this object, we can obtain a new Covert Learning algorithm by transforming the learning problem from being over a product distribution to being over the uniform distribution, but with a new concept being the composition of ¯ µ p,k,n and the original concept.
Since the concept is assumed to be closed under composition with µ p,k,n , then this new learning problem is still handled by the underlying Covert Learning algorithm.
The definition of a distributional inverter is given formally below:
Definition B.4 (Distributional Inverters) .
A function f : { 0 , 1 } ∗ → { 0 , 1 } ∗ is γ -distributionally invertible if there is an efficient probabilistic algorithm I such that the distributions ⟨ x, f ( x ) ⟩ and ⟨ I ( f ( x )) , f ( x ) ⟩ where x ∼ U are statistically indistinguishable within γ for all but finitely many lengths n .
That is,
[Equation]
To prove Theorem B.3, we will use the distributional inverter for concise product distributions of [26]:
Lemma B.5 (Distributional inverter for concise product distributions, from [26]) .
Concise product distributions have distributional inverters.
Moreover, these inverters are computable in AC 0 with auxiliary random bits.
For an overview of how the inverter works, see Section D.
This distributional inverter is denoted by the name ProdInv .
We are now ready to prove Theorem B.3.
Proof of Theorem B.3.
Let r be the number of bits required to sample µ p,k,n with the circuit B p,n,k .
Let g f : { 0 , 1 } r
→ { 0 , 1 } be the composed concept defined by g f ( x ) =
f ( B p,n,k ( x )) .
By assumption, g f ∈ C for every f ∈ C .
Now, since we know that there is a natural Covert Learning algorithm A for C with respect to the uniform distribution, then we can obtain a new algorithm A ′ for C with respect to µ p,k as follows.
1) Input: ϵ, δ, n , an oracle to f ∈ C . 2) Prepare a simulated oracle to g f by using the circuit ¯ µ p,k,n to first generate “intermediate” representations y = ¯ µ p,k,n ( x ) of a query x , and then query the oracle to f for y . 3) Using the simulated oracle, run A (with parameters ϵ/ 2 , δ, r ) to obtain a hypothesis h : { 0 , 1 } r → { 0 , 1 } for g f with respect to the uniform distribution.
Output: h ◦ ProdInv , with statistical closeness parameter γ = ϵ/ 2 .
Let us prove that the above algorithm has the desired learning and privacy guarantees, starting with learning.
First, observe that the simulated oracle from step 1 is equivalent to a membership query oracle for g f .
Thus, the hypothesis h obtained in step 2 satisfies
[Equation]
with probability at least 1 − δ .
This means that if we consider the quantity
[Equation]
x ) (cid:3) then the distributional inversion property of ProdInv (see
Definition B.4) allows one to conclude that
[Equation]
≡
[Equation]
By (1) this final quantity is bounded by ϵ/ 2 , and therefore we can see that
[Equation]
Finally, we need to prove that A ′ satisfies privacy.
To see this, first consider that the distribution over queries that are requested by A ′ is determined by applying ¯ µ p,k,n to each query x ∈ Q , where Q is the set of q ( n ) queries sampled from the distribution over queries D A made by A .
By the privacy guarantee of A , D A is computationally indistinguishable from E X ( f, U , q ( n )) .
Therefore, one can conclude that D A ′ is computationally indistinguishable from E X ( f, µ p,k,n , q ( n )) .
This follows from a simple reduction, which takes into account that µ p,k,n is polynomial time samplable by computing ¯ µ p,k,n .
This suffices to prove the statement.
We will now use Theorem B.3 to obtain Covert Learning for DNFs over concise product distributions.
We will use ideas from Fourier analysis of boolean functions.
See Section A-C for a brief overview of Fourier analysis, and the relevant definitions.
Additionally, we use the fact that polynomial size
DNFs can be learned, using only random examples, from the knowledge of the heavy O (log n )
-degree Fourier coefficients.
The algorithm witnessing this fact is in [27].
Definition B.6 .
Let DNF [ s, w ] be the class of functions f : { 0 , 1 } n → { 0 , 1 } that are computable by a DNF with at most s terms of maximum width w .
DNF [ s ] denotes the class of functions with no limit on the maximum term width.
Theorem B.7 (Natural Covert Learning DNFs over concise product distributions) .
Given query access to a function f ∈ DNF [poly( n )] , there exists an algorithm A running in time poly( n, 1 /ϵ, log(1 /δ )) and making q ( n ) = poly( n, 1 /ϵ, log(1 /δ ))
query accesses such that, unless the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption does not hold, then for any µ p,k for k
= O (1) , ϵ = Ω(1) , then
1) (Learning)
A outputs h : { 0 , 1 } n → { 0 , 1 }
such that Pr x ∼ µ p,k,n (cid:104) h
( x ) ̸ = f ( x ) (cid:105) ≤ ϵ with probability 1 − δ . 2)
(Privacy) The distribution over examples requested by A is computationally indistinguishable from E X ( f, µ p,k,n , q ( n )) .
In order to prove the theorem, we will argue that we can use the algorithm from Theorem IV.5 in place of the generically assumed Covert Learning algorithm in Theorem B.3, and that the class DNF [ s ] maintains the necessary closure under composition condition.
We recall that, as described in the introduction, the algorithm from Theorem IV.5 (Covert
Learning of decision trees) does the following:
1) Use membership query access to a function to collect a list of all parity functions that have large, low-degree Fourier coefficients (with respect to that function).
2) Estimate the magnitude of each of these coefficients to some desired accuracy derived from ϵ given as input.
3) Take the sum of the parity functions weighted by the estimated magnitude, and output 0 if the sign is negative and 1 otherwise.
Hence it
suffices to learn all O (log n )
-degree
Fourier coefficients.
Now, by the result of [27], this suffices to learn any polynomial size DNF in time poly( n ) using further only uniformly random examples.
Proof of Theorem B.7.
To apply Theorem B.3, we only need to argue that for any f ∈ DNF
[poly( n )] we have that g f ∈
DNF [poly( n )] where g f
= f ◦ ¯ µ p,k,n .
After that, we can invoke
Theorem B.3 to complete the proof.
To see that for any f ∈ DNF [poly( n )] , g f ∈ DNF
[poly( n )] , recall that each k -concise product distribution µ p,k,n can be sampled by a (multi-output) DNF ¯ µ p,k,n .
This DNF takes as input kn random bits, and samples all n bits x 1 , · · · x n in parallel by taking the OR of every possible k -bit string (encoded by a k -wise AND) larger than the binary representation p
i (for parallel execution i ∈ [ n ] ).
The resulting DNF is thus of size poly( n ) since k is a constant.
Now, if we consider f ∈ DNF
[poly( n )] , we can write g f ( z )
=
f (¯ µ p,k,n ( z )) .
Initially, this appears to be a depth4 alternating circuit (i.e., a DNF of DNFs).
However, it can be compressed to a DNF with only a polynomial blow-up in size because the DNF for ¯ µ p,k,n is of constant size (since k is a constant).
To see this, observe that every variable in the DNF can be replaced by an explicit encoding by a k -wise AND of a string that maps to a 1 under ¯ µ p,k,n (if the variable is not negated, and 0 otherwise), and then repeating for all possible choices of strings and taking the OR of all the instances.
The resulting circuit is a DNF, and since k is a constant, the number of terms increases only by a polynomial factor, so it remains a member of DNF [poly( n )] .
This proves that the class of polynomial size DNFs is closed under composition with concise product distributions and thus completes the proof of the statement.
Incompleteness theorem for OMED on product distributions.
The theorems obtained in this section so far can now be applied to obtain incompleteness theorems for OMEDs that protect ML models computable by polynomial size DNFs, where the benign property is any concise product distribution.
Theorem B.8 .
Under the (2 ω ( n 12 ) , 2 ω ( n 12 ) ) − SLPN µ,n assumption, if M is a poly( n ) -OMED for DNF [poly( n )] and µ p,k , then if M is δ -complete
it is not (1 − δ − negl( n )) -sound.
Proof.
Define the client C
=
( D A , A ) , where A is the algorithm that witnesses Theorem B.7, and D A is the distribution over q ( n ) queries requested by A .
Fix some failure probability for A , denoted by δ A .
By the learning and privacy guarantees of Theorem B.7, it then follows that, for any n ∈ N , the µ p,k -adverse client
C is such that for any δ -complete OMED M (with respect to { µ p,k,n } ) for DNF [poly( n )] , it holds that for any f ∈
DNF [poly( n )] ,
[Equation]
The first inequality is justified because the distribution over
S is computationally indistinguishable from ( µ p,k,n ) q ( n ) (by the privacy of A ) and M is probabilistic polynomial time.
Then, the second inequality follows by independence of events, and because for ( S, τ ) ∼ Exp C ,f,q ( n ) ,ϵ , we have that τ = extracted with probability at least 1 − δ A (by the learning guarantee of A ), and the δ -completeness of M gives that
[Equation]
] ≥
[Equation]
δ .
Then, we can deduce that
[Equation]
( n ) for appropriately chosen δ A = 1 / 2 (for instance).
This completes the proof.
A PPENDIX C P ROVABLE S ECURITY FROM THE I MPOSSIBILITY OF C OVERT L EARNING
In this section, we show the inverse of Corollary IV.3.
That is, if natural Covert Learning algorithms do not exist, then we can obtain good OMEDs needed for security via
Theorem III.9.
Theorem C.1 .
If there does not exist an efficient natural Covert
Learning algorithm for a concept class C with respect to D = { D n } n ∈ N , then for any n ∈ N , there exists a poly( n ) -OMED
M for a concept class C n that is δ -complete and η -sound with respect to the property { D n } for η ≤ negl( n ) .
Proof.
Since there does not exist an efficient natural Covert
Learning algorithm for the concept class C with respect to
D , then it means that for every learning algorithm
A such that for every f ∈ C n , ϵ, γ > 0 we have that with probability [Equation] γ , [Equation]
[Equation]
[Equation]
, poly
[Equation]
) ,ϵ = extracted , then distribution D
A over queries made by A is efficiently distinguishable from
D .
In other words, there exists p.p.t.
Adv such that for S ∼
[Equation]
f
[Equation]
where T ( A O f ) denotes the distribution over the queries made by A and the responses by the oracle.
This implies that we can produce the desired OMED M by using Adv as follows:
[Equation]
] within an additive factor of µ/ 4 with probability at least 1 − η/ 2 . 2) Using segments of T A O f ) estimate
Pr
Adv , T A O f ) (cid:2)
Adv ( T ( A O f ))
= 1 (cid:3) within an additive factor of µ/ 4 , with probability at least 1 − η/ 2 . 3) Output “reject” if the two estimates are not within µ/ 2 , and “accept” otherwise.
We analyze this process to prove the desired statement, starting with completeness and soundness.
Without loss of generality, assume that Pr Adv ,S
[ Adv ( S )
= 1] ≥ 1 − δ .
Then, it follows that M is δ -complete with respect to P n .
Now, by (3), it must be the case that
[Equation]
Hence, the estimation procedure is guaranteed to distinguish between the two cases with probability least 1 − η (using a union bound).
Using this fact and inspecting step 3 gives ηsoundness.
To see that M is an poly( n ) -OMED specifically, observe that Adv is efficient (and therefore runs in time poly( n ) .
We remark that in the above proof, it is important that
the
OMED has access to poly( n ) · O (log( η − 1 )) examples from the client (in order to complete the estimation procedures).
Ideally, we can reduce this number, as clients may be able to perform model extraction using less queries. A PPENDIX D T HE D ISTRIBUTIONAL I NVERTER OF [26]
In this section, we sketch the process ProdInv of [26] that constitutes a distributional inverter for concise product distributions.
The sketch is based off the description from section 5 of [26].
ProdInv obtains a random preimage of x ∼ µ p,k,n by inverting each bit of x separately.
Thus let x = x 1 x 2 · · · x n .
The following process is the same for each bit of x , so without loss of generality consider x i for any i ∈ [ n ] .
To sample a preimage of x i = 1 , choose a number c ≤ k (where k is the number of bits to describe the probability of 1) such that 2 c − 1 ≤ p i < 2 c , and then choose a uniformly random cbit string y .
Transform y to a k bit string by adding k − c 0s as a prefix.
y is thus a k -bit binary number in the range [0 , · · · , (2 c − 1) / 2 k ] .
Since p i ≥ 2 c − 1 , the number y will be less than or equal to p i with probability at least 1/2.
When this is the case, output y .
In the other case, choose another c -bit string y ′ , and repeat the previous process until either y ′ <
p i is chosen, or the number of attempts is greater than log 1 /γ
(for some pre-determined failure probabiltiy γ ).
Because each attempt is done independently, and produces an output with probability bounded below by 1/2, the number of attempts suffices to suppress the probability of failure (i.e., no output) will to less than or equal to γ .
{+Next, to sample a preimage of x i = 
+}
instead of x i = 1 , just choose y <
1 − p
i in the same way as choosing y < p i before, but instead return p + y .
By the same argument, the number of attempts reduces the failure probability to at most γ .
Finally, to sketch an argument for the statistical closeness of the described distributional inverter, observe that in the case of no failure, the output preimage is distributed exactly uniformly at random from the space of pre-images (also, see lemmaof [26]).
Thus, the statistical closeness follows, by setting γ low enough.
Lastly, combining a direct product of n instances of the bit inverter increases the failure probability by a factor of n , but using a union bound, it follows that setting γ to γ/n handles suppresses this effect.