W HITENING B LACK -B OX N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many deployed learned models are black boxes: given input, returns output.
Internal information about the model, such as the architecture, optimisation procedure, or training data, is not disclosed explicitly as it might contain proprietary information or make the system more vulnerable.
This work shows that such attributes of neural networks can be exposed from a sequence of queries.
This has multiple implications.
On the one hand, our work exposes the vulnerability of black-box neural networks to different types of attacks – we show that the revealed internal information helps generate more effective adversarial examples against the black box model.
On the other hand, this technique can be used for better protection of private content from automatic recognition models using adversarial examples.
Our paper suggests that it is actually hard to draw a line between white box and black box models.
I NTRODUCTION
Black-box models take a sequence of query inputs, and return corresponding outputs, while keeping internal states such as model architecture hidden.
They are deployed as black boxes usually on purpose – for protecting intellectual properties or privacy-sensitive training data.
Our work aims at inferring information about the internals of black box models – ultimately turning them into white box models.
Such a “whitening” of a black box model has many implications.
On the one hand, it has legal implications to intellectual properties (IP) involving neural networks – internal information about the model architecture can be proprietary and a key IP, and the training data may be privacy sensitive.
Disclosing hidden details may also render the model more susceptible to attacks from adversaries.
On the other hand, gaining information about a black-box model can be useful in other scenarios.
E.g. there has been work on utilising adversarial examples for protecting private regions (e.g. faces) in photographs from automatic recognisers (Oh et al., 2017).
In such scenarios, gaining more knowledge on the recognisers will increase the chance of protecting one’s privacy.
Either way, it is a crucial research topic to investigate the type and amount of information that can be gained from a black-box access to a model.
We make a ﬁrst step towards understanding the connection between white box and black box approaches – which were previously thought of as distinct classes.
We introduce the term “model attributes” to refer to various types of information about a trained neural network model.
We group them into three types: (1) architecture (e.g. type of non-linear activation), (2) optimisation process (e.g. SGD or ADAM?), and (3) training data (e.g. which dataset?).
We approach the problem as a standard supervised learning task applied over models .
First, collect a diverse set of white-box models (“meta-training set”) that are expected to be similar to the target black box at least to a certain extend.
Then, over the collected meta-training set, train another model (“metamodel”) that takes a model as input and returns the corresponding model attributes as output.
Importantly, since we want to predict attributes at test time for black-box models, the only information available for attribute prediction is the query input-output pairs.
As will see in the experiments, such input-output pairs allow to predict model attributes surprisingly well.
In summary, we contribute: (1) Investigation of the type and amount of internal information about the black-box model that can be extracted from querying; (2) Novel metamodel methods that not only reason over outputs from static query inputs, but also actively optimise query inputs that can extract more information; (3) Study of factors like size of the meta-training set, quantity and quality of queries, and mismatch between meta-training models and the black box model; (4) Empirical veriﬁcation that revealed information leads to greater susceptibility of a black-box model to an adversarial example based attack.
R ELATED W ORK
There has been a line of work on extracting and exploiting information from black-box learned models.
We ﬁrst describe papers on extracting information ( model extraction and membership inference attacks), and then discuss ones on attacking the network using the extracted information ( adversarial image perturbations (AIP) ).
Model extraction attacks either reconstruct the exact model parameters or build an avatar model that maximises the likelihood of the query input-output pairs from the target model (Tramer et al., 2016;
Papernot et al., 2017).
Tramer et al.
(2016) have shown the efﬁcacy of equation solving attacks and the avatar method in retrieving internal parameters of non-neural network models.
Papernot et al.
(2017) have also used the avatar approach with the end goal of generating adversarial examples.
While the avatar approach ﬁrst assumes model hyperparameters like model family (architecture) and training data, we discriminatively train a metamodel to predict those hyperparameters themselves.
As such, our approach is complementary to the avatar approach.
Membership inference attacks determine if a given data sample has been included in the training data (Ateniese et al., 2015; Shokri et al., 2017).
In particular, Ateniese et al.
(2015) also trains a decision tree metamodel over a set of classiﬁers trained on different datasets.
This work goes far beyond only inferring the training data by showing that even the model architecture and optimisation process can be inferred.
Using the obtained cues, one can launch more effective, focused attacks on the black box.
We use adversarial image perturbations (AIPs) as an example of such attack.
AIPs are small perturbations over the input such that the network is mislead.
Research on this topic has ﬂourished recently after it was shown that the needed amount of perturbation to completely mislead an image classiﬁer is nearly invisible (Szegedy et al., 2014; Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2017).
Most effective AIPs require gradients of the target network.
Some papers proposed different ways to attack black boxes.
They can be grouped into three approaches.
(1) Approximate gradients by numerical gradients (Narodytska & Kasiviswanathan, 2017; Chen et al., 2017).
The caveat is that thousands and millions of queries are needed to compute a single AIP, depending on the image size.
(2) Use the avatar approach to train a white box network that is supposedly similar to the target (Papernot et al., 2016b;a; Hayes & Danezis, 2017).
We note again that our metamodel is complementary to the avatar approach – the avatar network hyperparemters can be determined by the metamodel.
(3) Exploit transferability of adversarial examples; it has been shown that AIPs generated against one network can also fool other networks (Moosavi-Dezfooli et al., 2017; Liu et al., 2017).
Liu et al.
(2017) in particular have shown that generating AIPs against an ensemble of networks make it more transferable.
We show in this work that the AIPs transfer better within an architecture family (e.g. ResNet or DenseNet) than across, and that such a property can be exploited by our metamodel for generating more targetted AIPs.
M ETAMODELS
We want to ﬁnd out the type and amount of internal information about a black-box model that can be revealed from a sequence of queries.
We approach this by ﬁrst building metamodels for predicting model attributes, and then evaluating their performance on black-box models.
Our main approach, metamodel, is described in ﬁgure 1.
In a nutshell, the metamodel is a classiﬁer of classiﬁers.
Speciﬁcally, The metamodel submits n query inputs (cid:2)
x i (cid:3) n i =1 to a black box model f ; the metamodel takes corresponding model outputs (cid:2) f
( x i ) (cid:3) n i =1 as an input, and returns predicted model attributes as output.
As we will describe in detail, the metamodel not only learns to infer model attributes from query outputs from a static set of inputs, but also searches for query inputs that are designed to extract greater amount of information from models.
In this section, our main methods are introduced in the context of MNIST digit classiﬁers.
While
MNIST classiﬁers are not fully representative of any learned model, they have a computational edge: it takes only ﬁve minutes to train each of them with reasonable performance.
We could thus prepare a diverse set of 11k MNIST classiﬁers within 40 GPU days for the meta-training and evaluation of our metamodels.
We stress, however, that the proposed approach is generic with respect to the task, data, and the type of models.
We also focus on 12 model attributes (table 1) that cover hyperparameters for common neural network MNIST classiﬁers, but again the range of predictable attributes are not conﬁned to this list.
3. COLLECTING A DATASET OF CLASSIFIERS
We need a dataset of classiﬁers to train and evaluate metamodels.
We explain how MNIST-NETS has been constructed, a dataset of 11k MNIST digit classiﬁers; the procedure is task and data generic.
BASE NETWORK SKELETON
Every model in MNIST-NETS shares the same convnet skeleton architecture: “ N conv blocks →
M fc blocks → 1 linear classiﬁer”.
Each conv block has the following structure: “ks × ks convolution → optional 2 × 2 max-pooling → non-linear activation”, where ks (kernel size) and the activation type are to be chosen.
Each fc block has the structure: “ (cid:48)(cid:48) linear mapping → non-linear activation → optional dropout” This convnet structure already covers many LeNet (LeCun et al., 1998) variants, one of the best performing architectures on MNIST (mni).
INCREASING DIVERSITY
[Table]
In order to learn generalisable features, the metamodel needs to be trained over a diverse set of models.
The base architecture described above already has several free parameters like the number of layers ( N and
M ), the existence of dropout or maxpooling layers, or the type of nonlinear activation.
Apart from the architectural hyperparameters, we increase diversity along two more axes – optimisation process and the training data.
Along the optimisation axis, we vary optimisation algorithm (SGD, ADAM, or
RMSprop) and the training batch size (64, 128, 256).
We also consider training MNIST classiﬁers on either on the entire MNIST training set (All 0 , 60k), one of the two disjoint halves (Half 0 / 1 , 30k), or one of the four disjoint quarters (Quarter 0 / 1 / 2 / 3 , 15k).
See table 1 for the comprehensive list of 12 model attributes altered in MNIST-NETS .
The number of trainable parameters (#par) and the training data size (size) are not directly controlled but derived from the other attributes.
We also augment MNIST-NETS with ensembles of classiﬁers (ens), whose procedure will be described later.
SAMPLING AND TRAINING
The number of all possible combinations of controllable options in table 1 is 18 , 144 .
We also select random seeds that control the initialisation and training data shufﬂing from { 0 , · · · , 999 } , resulting in 18 , 144 , 000 unique models.
Training such a large number of models is intractable; we have sampled (without replacement) and trained 10 , 000 of them.
All the models have been trained with learning rate 0 . 1 and momentum 0 .
5 for 100 epochs.
It takes around 5 minutes to train each model on a GPU machine (GeForce GTX TITAN); training of 10k classiﬁers has taken 40 GPU days.
PRUNING AND AUGMENTING
In order to make sure that MNIST-NETS realistically represents commonly used MNIST classiﬁers, we have pruned low-performance classiﬁers (validation accuracy < 98% ), resulting in 8 , 582 clas- siﬁers.
Ensembles of trained classiﬁers have been constructed by grouping the identical classiﬁers (modulo random seed).
Given t identical ones, we have augmented MNIST-NETS with 2, · · · , t combinations.
The ensemble augmentation has resulted in 11 , 282 ﬁnal models.
See appendix table 6 for statistics of attributes – due to large sample size all the attributes are evenly covered.
TRAIN - EVAL SPLITS
Attribute prediction can get arbitrarily easy by including the black-box model (or similar ones) in the meta-training set.
We introduce multiple splits of MNIST-NETS with varying requirements on generalization.
Unless stated otherwise, every split has 5 , 000 training (meta-training), 1 , 000 testing (black box), and 5 , 282 leftover models.
The Random (R) split randomly (uniform weights) assigns training and test splits, respectively.
Under R split, the training and test models come from the same distribution.
We introduce harder
Extrapolation (E) splits.
We separate a few attributes between the training and test splits.
They are designed to simulate more difﬁcult domain gaps when the meta-training models are signiﬁcantly different from the black box.
Speciﬁc examples of E splits will be shown in §4.
3. METAMODEL METHODS
The metamodel predicts the attribute of a black-box model f by submitting n query inputs and observing the outputs.
It is trained over a meta-training set (training split).
We propose three approaches for the metamodels – we collectively name them kennen 1 .
See ﬁgure 2 for an overview.
[Figure]
kennen-o ﬁrst selects a ﬁxed set of queries [ x i ] i =1 ··· n from a dataset.
Both during training and testing, always these queries are submitted.
kennen-o learns a classiﬁer m θ to map from the order-sensitive concatenated n query outputs, [ f ( x i )]
i =1 ··· n ( n × 10 dim for MNIST), to the simultaneous prediction of 12 attributes in f .
The training objective is:
[Equation]
where F is the distribution of meta-training models, y a is the ground truth label of attribute a , and L is the cross-entropy loss.
In our experiments, we model the classiﬁer m θ via multilayer perceptron (MLP) with two hidden layers with 50 hidden units.
The last layer consists of 12 parallel linear layers for a simultaneous prediction of attributes.
In our preliminary experiments, MLP has performed better than linear classiﬁers.
The optimisation problem in equation 1 is solved via SGD by approximating the expectation over f ∼ F by an empirical sum over the training split classiﬁers for 200 epochs.
Note that kennen-o can be applied to any type of model (e.g. non-neural networks) with any output structure, as long as the output can be embedded in an Euclidean space.
We will show that this method can effectively extract information from f even if the output is a single label, encoded via one-hot vectors.
KENNEN - I : CRAFT INPUT
kennen-i crafts a single query input that can repurpose a black box f into a model attribute classiﬁer for a single attribute a .
For example, kennen-i for max-pooling layer prediction crafts an input x that is predicted as “1” for MNIST digit classiﬁers with max-pooling layers and “0” for ones without.
See ﬁgure 3 for visual examples.
The training objective is:
[Table]
will predict as if
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Equation]
[Figure]
[Equation]
where the condition x : image ensures the input stays a valid image x ∈ [0 , 1] D with image dimension D .
We use SGD as for kennen-o for 200 epochs.
For each iteration we project x back to [0 , 1] D to enforce the constraint.
We initialise x with a random sample from the MNIST validation set (random noise or uniform gray initilisation gives similar performances).
Unlike kennen-o , kennen-i submits unnatural images to the system, and so may easily be detected.
kennen-o is more realistic when the exploration needs to be stealthy.
Also unlike kennen-o , kennen-i requires end-to-end differentiability of f .
KENNEN - IO : COMBINED APPROACH
In order for kennen-i to optimise multiple different query inputs, simply re-purposing a classiﬁer is not enough – there must be some form of reasoning over their outputs to diversify the input patterns.
Our ﬁnal method kennen-io performs kennen-o on top of the output from kennen-i crafted query inputs.
The optimisation problem is as follows:
[Equation]
To improve stability against covariate shift, we initialise m θ with kennen-o for 200 epochs.
Afterwards, gradient updates of [ x i ] ni =1 and θ alternate every 50 epochs.
W HITENING B LACK -B OX MNIST D IGIT C LASSIFIERS
We have introduced a procedure for constructing a dataset of classiﬁers ( MNIST-NETS ) as well as novel metamodels ( kennen variants) that learn to extract information from black-box classiﬁers.
In this section, we evaluate the ability of kennen to extract information from black-box MNIST digit classiﬁers.
We measure the class-balanced attribute prediction accuracy for each attribute a in the list of 12 attributes in table 1.
ATTRIBUTE PREDICTION
See table 2 for the main results of our metamodels, kennen-o , kennen-i , and kennen-io on the Random split.
Unless stated otherwise, the metamodel is trained with 5 , 000 training split classiﬁers.
Given n = 100 queries with probability output, kennen-o already performs far above the random chance in predicting 12 diverse attributes (73.4% versus 34.9% on average); neural network output indeed contains rich information about the black box.
In particular, the presence of dropout (94.6%) or max-pooling (94.9%) has been predicted with high precision.
It is surprising that optimisation details like optimisation algorithm (71.8%) and training batch size (50.4%) can also be predicted well above the random chance (33.3% for both).
[Figure]
COMPARING METHODS KENNEN - O , KENNEN - I , AND KENNEN - IO
Table 2 shows the comparison of kennen-o , kennen-i
[Figure]
and kennen-io .
kennen-i has a relatively low performance (average 52.7%), but kennen-i relies on a cheap resource: 1 query with single-label output.
kennen-i is also performant at predicting the kernel size (88.5%) and pooling (94.8%), attributes that are closely linked to spatial structure of the input.
kennen-io is superior to kennen-o and kennen-i for all the attributes with average accuracy 80.1%.
4. FACTOR ANALYSIS NUMBER OF TRAINING MODELS
We have trained kennen-o with different number of the meta-training classiﬁers, ranging from 100 to 5 , 000 .
See ﬁgure 4 (left) for the trend.
We observe a diminishing return, but also that the performance has not saturated – collecting larger meta-training set will improve the performance.
NUMBER OF QUERIES
See ﬁgure 4 (middle) for the kennen-o performance against the number of queries with probability output.
The average performance saturates after ∼ 500 queries.
On the other hand, with only ∼queries, we already retrieve ample information about the neural network.
QUALITY OF OUTPUT
Many black-box models return top-k ranking output (e.g. Facebook face recogniser), or single-label output (i.e. top-1 ranking).
We represent top-k ranking outputs by assigning exponentially decaying probabilities up to k digits and a small probability (cid:15) to the remaining.
See table 2 for the kennen-o performance comparison among 100 probability, top-10 ranking, and single label (i.e. top-1) outputs, with average accuracies 73.4%, 69.7%, and 54.4%, respectively.
While performance drops with coarser outputs, when compared to random chance (34.9%),single-label query outputs already leak a great amount of information about the black box.
Figure(right) shows the interpolation from top-1 to top-10 (i.e. top-9) ranking.
We observe a diminishing return as k increases.
4. WHAT IF THE BLACK - BOX IS QUITE DIFFERENT FROM META - TRAINING MODELS ?
So far we have seen results on the Random (R) split.
In realistic scenarios, the meta-training model distribution may not be fully covering possible black box models.
We show how damaging such a scenario is through Extrapolation (E) split experiments.
Results are presented in table 3.
E-split results are presented along three axes, architecture, optimisation, and data.
For example, ﬁrst row presents results when metamodel is trained on shallower nets (2 or 3 conv or fc layers each) compared to the black box model (4 conv and fc layers each).
[Table]
We report relative average accuracies – R-
100% and random chance gives 0%.
Not surprisingly, - split performances are lower than R-split ones; nonetheless, E-split performances are still far above the chance level (relative average accuracies > 70%).
4. D ISCUSSION
We have veriﬁed through our novel kennen metamodels that black-box access to a neural network exposes much internal information.
We have shown that onlysingle-label outputs already “whitens” black boxes to a great deal.
When the black-box classiﬁer is quite different from the meta-training classiﬁers, the performance of our best metamodel – kennen-io – decreases; however, the prediction accuracy for black box internal information is still surprisingly high.
W HITENING AND A TTACKING I MAGE N ET C LASSIFIERS
While MNIST experiments are computationally cheap and a massive number of controlled experiments is possible, we provide additional ImageNet experiments for practical implications on realistic image classiﬁers.
In this section, we use kennen-o introduced in §3 to predict a single attribute of black-box ImageNet classiﬁers – the architecture family (e.g. ResNet or VGG?).
In this section, we go a step further to use the extracted information to attack black boxes with adversarial examples.
5. DATASET OF I MAGE N ET CLASSIFIERS
It is computationally prohibitive to train O (10 k )
ImageNet classiﬁers from scratch as in the previous section.
We have resorted to 19 PyTorch (pyt) pretrained ImageNet classiﬁers.
The 19 classiﬁers come from ﬁve families: S queezenet, V GG, VGG- B atchNorm, R esNet, and D enseNet, each with 2, 4, 4, 5, and 4 variants, respectively (Iandola et al., 2016; Simonyan & Zisserman, 2015;
He et al., 2016; Huang et al., 2017).
5. CLASSIFIER FAMILY PREDICTION
We predict the classiﬁer family (S, V, B, R, D) from the black-box query output, using the method kennen-o , with the same MLP architecture (§3).
kennen-i and kennen-io have not been used for computational reasons, but can also be used in principle.
We conduct 10 cross validations (random sampling of single test network from each family) for evaluation.
We also perform 10 random sampling of the queries from ImageNet validation set.
In total 100 random tries are averaged.
Results: compared to the random chance (20.0%), 100 queries result in high kennen-o performance (90.4%).
With 1 , 000 queries, the prediction performance is even 94.8%.
5. ATTACKING I MAGE N ET CLASSIFIERS
In this section we attack ImageNet classiﬁers with adversarial image perturbations (AIPs).
We show that the knowledge about the black box architecture family makes the attack more effective.
ADVERSARIAL IMAGE PERTURBATION (AIP)
AIPs are carefully crafted additive perturbations on the input image for the purpose of misleading the target model to predict wrong labels (Goodfellow et al., 2015).
Among variants of AIPs, we use efﬁcient and robust GAMAN (Oh et al., 2017).
See appendix ﬁgure 6 for examples of AIPs; the perturbation is nearly invisible.
TRANSFERABILITY OF AIP S
We report misclassiﬁcation rates .
[Table]
Typical AIP algorithms require gradients from the target network, which is not available for a black box.
Mainly three approaches for generating AIPs against black boxes have been proposed: (1) numerical gradient, (2) avatar network, or (3) transferability.
We show that our metamodel strengthens the transferability based attack.
We hypothesize and empirically show that AIPs transfer better within the architecture family than across.
Using this property, we ﬁrst predict the family of the black box (e.g. ResNet), and then generate AIPs against a few instances in the family (e.g. ResNet101,
ResNet152).
The generation of AIPs against multiple targets has been proposed by Liu et al.
but we are the ﬁrst to systemically show that AIPs generalise better within a family when they are generated against multiple instances from the same family.
We ﬁrst verify our hypothesis that AIPs transfer better within a family.
Within-family: we do a leave-one-out cross validation – generate AIPs using all but one instances of the family and test on the holdout.
Not using the exact test black box, this gives a lower bound on the within-family performance.
Across-family: still leave out one random instance from the generating family to match the generating set size with the within-family cases.
We also include the use-all case (Ens): generate AIPs with one network from each family.
See table 4 for the results.
We report the misclassiﬁcation rate , deﬁned as 100 − top-1 accuracy, on 100 random ImageNet validation images.
We observe that the within-family performances dominate the across-family ones (diagonal entries versus the others in each row); if the target black box family is identiﬁed, one can generate more effective AIPs.
Finally, trying to target all network (“Ens”) is not as effective as focusing resources (diagonal entries).
METAMODEL ENABLES MORE EFFECTIVE ATTACKS
We empirically show that the whitening enables more effective attacks.
We consider multiple scenarios.
“White box” means the target model is fully known, and the AIP is generated speciﬁcally for this model.
“Black box” means the exact target is unknown, but we make a distinction when the family is known (“Family black box”).
[Table]
See table 5 for the misclassiﬁcation rates (MC) in different scenarios.
When the target is fully speciﬁed (white box), MC is 100%.
When neither the exact target nor the family is known,
AIPs are generated against multiple families (82.2%).
When the whitening takes place, and
AIPs are generated over the predicted family, attacks become more effective (85.7%).
We almost reach the family-oracle case (86.2%).
5. D ISCUSSION
Our metamodel can predict architecture families for ImageNet classiﬁers with high accuracy.
We additionally show that such whitening enables more focused attack on black-boxes.
C ONCLUSION
We have presented ﬁrst results on the inference of diverse neural network attributes from a sequence of input-output queries.
Our novel metamodel methods, kennen , can successfully predict attributes related not only to the architecture but also to training hyperparameters (optimisation algorithm and dataset) even in difﬁcult scenarios (e.g. single-label output, or a distribution gap between the metatraining models and the target black box).
We have additionally shown in ImageNet experiments that the “whitening” of a black box makes it more vulnerable to adversarial examples.
A PPENDIX A MNIST-NETS STATISTICS
We show the statistics of MNIST-NETS , our dataset of MNIST classiﬁers, in table 6.
B MORE KENNEN - IO RESULTS
We complement the kennen-o results in the main paper (ﬁgure 4) with kennen-io results.
See ﬁgure 5.
Similarly for kennen-o , kennen-io shows a diminishing return as the number of training models and the number of queries increase.
While the performance saturates with 1 ,queries, it does not fully saturate with 5 , 000 training samples.
C VISUAL EXAMPLES OF AIP S
In this section, we show examples of AIPs.
See ﬁgure 6 for the examples of AIPs and the perturbed images.
The perturbation is nearly invisible to human eyes.
We have also generated AIPs with respect to a diverse set of architecture families (S, V, B, R, D, SVBRD) at multiple L 2 norm levels.
See ﬁgure 7; the same image results in a diverse set of patterns depending on the architecture family.
[Table]
[Figure]
pool
[Figure]
ks
[Figure] [Figure]
arch
[Figure]
#conv arch
[Figure]
#par arch
[Figure]
ens opt
[Figure]
alg opt
[Figure]
bs
[Figure]
split
[Figure] [Figure]
Perturbation
Perturbed
Original
Perturbation
Perturbed
O r i g i n a l I m a g e
L 2 = 1 × 10 −
L 2 = 2 × 10 −
L 2 = 5 × 10 −
L 2 = 1 × 10 −
S qu eeze
N e t
V
GG
VGG - B a t c h N o r m
R e s
N e t
D e n s e N e t
A ll ( S V
BR
D )