Original: Inferring Class Label Distribution of Training Data from Classiﬁers: An Accuracy-Augmented Meta-Classiﬁer Attack
Revised: Inferring Class Label Distribution of Training Data from Classiﬁers: An Accuracy-Augmented Meta-Classiﬁer Attack

Original: Abstract
Revised: Abstract

Original: Property inference attacks against machine learning (ML) models aim to inferproperties of the training data that are unrelated to the primary task of the model,and have so far been formulated as binary decision problems, i.e., whether or notthe training data have a certain property.
Revised: Property inference attacks against machine learning (ML) models aim to infer properties of the training data that are unrelated to the primary task of the model, and have so far been formulated as binary decision problems, i.e., whether or not the training data have a certain property.

Original: However, in industrial and healthcare applications, the proportion of labels in the training data is quite often also consideredsensitive information.
Revised: However, in industrial and healthcare applications, the proportion of labels in the training data is quite often also considered sensitive information.

Original: In this paper we introduce a new type of property inferenceattack that unlike binary decision problems in literature, aim at inferring the classlabel distribution of the training data from parameters of ML classiﬁer models.
Revised: In this paper we introduce a new type of property inference attack that unlike binary decision problems in literature, aim at inferring the class label distribution of the training data from parameters of ML classiﬁer models.

Original: Wepropose a method based on shadow training and a meta-classiﬁer trained on theparameters of the shadow classiﬁers augmented with the accuracy of the classiﬁerson auxiliary data.
Revised: We propose a method based on shadow training and a meta-classiﬁer trained on the parameters of the shadow classiﬁers augmented with the accuracy of the classiﬁers on auxiliary data.

Original: We evaluate the proposed approach for ML classiﬁers with fullyconnected neural network architectures.
Revised: We evaluate the proposed approach for ML classiﬁers with fully connected neural network architectures.

Original: We ﬁnd that the proposed meta-classiﬁerattack provides a maximum relative improvement of 52% over state of the art.
Revised: We ﬁnd that the proposed meta-classiﬁer attack provides a maximum relative improvement of 52% over state of the art.

Original:  Introduction
Revised:  Introduction

Original: Classiﬁcation using machine learning (ML) models is increasingly considered for use in a varietyof industrial applications, including outlier detection in manufacturing, bank fraud detection, andvarious diagnostics applications in healthcare.
Revised: Classiﬁcation using machine learning (ML) models is increasingly considered for use in a variety of industrial applications, including outlier detection in manufacturing, bank fraud detection and various diagnostics applications in healthcare.

Original: High quality training data plays a pivotal role inproducing good models that can provide accurate classiﬁcation results but producing high qualitylabeled data sets is often expensive, making trained models a valuable asset.
Revised: High quality training data plays a pivotal role in producing good models that can provide accurate classiﬁcation results, but producing high quality labeled data sets is often expensive, making trained models a valuable asset.

Original: At the same time, it is widely understood that ML models, particularly deep neural networks (DNNs),may learn information about the data they were trained on and could be used by an adversary foridentifying properties of the training data set by analyzing the parameters of the trained model.
Revised: At the same time, it is widely understood that ML models, particularly deep neural networks (DNNs), may learn information about the data they were trained on, and could be used by an adversary for identifying properties of the training data set by analyzing the parameters of the trained model.

Original: Suchattacks, termed property inference attacks Melis et al.
Revised: Such attacks, termed property inference attacks Melis et al.

Original: [2019] have been widely studied from theperspective of the privacy and conﬁdentiality of the input features Rigaki and Garcia [2020].
Revised: [2019] have been widely studied from the perspective of the privacy and conﬁdentiality of the input features Rigaki and Garcia [2020].

Original: Therehas been signiﬁcantly less focus on property inference attacks against training labels, i.e., theirdistribution in the training data set.
Revised: There has been signiﬁcantly less focus on property inference attacks against training labels, i.e., their distribution in the training data set.

Original: Notwithstanding, the class label distribution could be highlysensitive in industrial settings as it could reveal to competitors, e.g., the credibility of loans of bankingcustomers or the production quality of assembly lines, with signiﬁcant impact on a company’svaluation.
Revised: Notwithstanding, the class label distribution could be highly sensitive in industrial settings as it could reveal to competitors, e.g., the credibility of loans of banking customers or the production quality of assembly lines, with signiﬁcant impact on a company’s valuation.

Original: In this work, we explore such property inference attacks on ML models trained for classiﬁcation.
Revised: In this work, we explore such property inference attacks on ML models trained for classiﬁcation.

Original: Thegoal of the adversary is to infer the class label distribution of the training data from the trained MLclassiﬁer model.
Revised: The goal of the adversary is to infer the class label distribution of the training data from the trained ML classiﬁer model.

Original: We consider white-box attacks, i.e., the adversary knows the model architecture andcan access the parameters of the target classiﬁer model.
Revised: We consider white-box attacks, i.e., the adversary knows the model architecture and can access the parameters of the target classiﬁer model.

Original: To motivate further this new class of property inference attack, we discuss two real-world exampleswhere such attacks could cause havoc.
Revised: To motivate further this new class of property inference attack, we discuss two real-world examples where such attacks could cause havoc.

Original: Consider for example the manufacturing context, where apower tool vendor could train a classiﬁer for identifying faulty bolt tightening using data from itscustomers (bolt tightening is often safety critical, e.g., in the car industry) for accelerating productionquality monitoring.
Revised: Consider for example the manufacturing context, where a power tool vendor could train a classiﬁer for identifying faulty bolt tightening using data from its customers (bolt tightening is often safety critical, e.g., in the car industry) for accelerating production quality monitoring.

Original: The proposed type of attack would make the trained model reveal informationabout the production quality of the customer whose data were used for training, which can be usedby competitors for blackmailing and negative publicity.
Revised: The proposed type of attack would make the trained model reveal information about the production quality of the customer whose data were used for training, which can be used by competitors for blackmailing and negative publicity.

Original: In lack of a countermeasure, the feasibilityof the attack is considered a showstopper for this use case of ML.
Revised: In lack of a countermeasure, the feasibility of the attack is considered a showstopper for this use case of ML.

Original: Another example is a bank thattrains a classiﬁer for predicting if a loan would be in default.
Revised: Another example is a bank that trains a classiﬁer for predicting if a loan would be in default.

Original: Our proposed attack could be used forinferring the fraction of default loans of the bank based on the trained classiﬁer, which could be usedfor affecting the bank’s public standing.
Revised: Our proposed attack could be used for inferring the fraction of default loans of the bank based on the trained classiﬁer, which could be used for affecting the bank’s public standing.

Original: Importantly, these use cases involve binary classiﬁcation,where the proposed attack performs best.
Revised: Importantly, these use cases involve binary classiﬁcation, where the proposed attack performs best.

Original: In this paper, we ﬁrstly deﬁne this new property inference attack and advocate the use of a shadowtraining methodology Ateniese et al.
Revised: In this paper, we ﬁrstly deﬁne this new property inference attack and advocate the use of a shadowtraining methodology Ateniese et al.

Original: [2015] for class label distribution inference.
Revised: [2015] for class label distribution inference.

Original: Shadow-training canbe interpreted as a supervised-learning approach to property inference by training a ‘meta-classiﬁer’that takes model parameters as input and outputs the inferred property.
Revised: Shadow-training can be interpreted as a supervised-learning approach to property inference by training a ‘meta-classiﬁer’ that takes model parameters as input and outputs the inferred property.

Original: In order to carry out shadowtraining, a dataset of trained classiﬁers and the corresponding property is used.
Revised: In order to carry out shadowtraining, a dataset of trained classiﬁers and the corresponding property is used.

Original: We propose ameta-classiﬁer whose architecture is invariant to permutations of the neuron connection weights.
Revised: We propose a meta-classiﬁer whose architecture is invariant to permutations of the neuron connection weights.

Original: Furthermore, the meta-classiﬁer also takes as input the accuracy of the trained classiﬁer on auxiliarydata.
Revised: Furthermore, the meta-classiﬁer also takes as input the accuracy of the trained classiﬁer on auxiliary data.

Original: We investigate the performance of the meta-classiﬁer and its dependence on the selection ofthe shadow-training dataset.
Revised: We investigate the performance of the meta-classiﬁer and its dependence on the selection of the shadow-training dataset.

Original: We ﬁnd that the proposed meta-classiﬁer can successfully infer theclass label distribution and that accuracy augmentation is beneﬁcial.
Revised: We ﬁnd that the proposed meta-classiﬁer can successfully infer the class label distribution and that accuracy augmentation is beneﬁcial.

Original: We show empirical results for 2datasets in the case of binary classiﬁers.
Revised: We show empirical results fordatasets in the case of binary classiﬁers.

Original: The empirical results for multi-class classiﬁers is discussedin Appendix A.1.
Revised: The empirical results for multi-class classiﬁers is discussed in Appendix A.1.

Original: In Appendix A.2, we ﬁnd that the attack is robust to random oversampling of theminority class, a technique to combat class imbalance.
Revised: In Appendix A.2, we ﬁnd that the attack is robust to random oversampling of the minority class, a technique to combat class imbalance.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: Property inference attacks have traditionally been posed as binary classiﬁcation problems in theliterature Melis et al.
Revised: Property inference attacks have traditionally been posed as binary classiﬁcation problems in the literature Melis et al.

Original: [2019], and are often based on meta-classiﬁer models irrespective of thetarget classiﬁer model architecture Parisot et al.
Revised: [2019], and are often based on meta-classiﬁer models irrespective of the target classiﬁer model architecture Parisot et al.

Original: [2021], Ganju et al.
Revised: [2021], Ganju et al.

Original: [2018], Ateniese et al.
Revised: [2018], Ateniese et al.

Original: Suri and Evans [2021], Zhou et al.
Revised: Suri and Evans [2021], Zhou et al.

Original: The meta-classiﬁer is trained using shadow-training ,i.e., the adversary trains shadow classiﬁers using labeled shadow-training data sets obtained fromauxiliary data with and without a certain property.
Revised: The meta-classiﬁer is trained using shadow-training , i.e., the adversary trains shadow classiﬁers using labeled shadow-training data sets obtained from auxiliary data with and without a certain property.

Original: The shadow classiﬁers have the same architectureas the target classiﬁer.
Revised: The shadow classiﬁers have the same architecture as the target classiﬁer.

Original: The input to the meta-classiﬁer is a set of features from the trained shadowclassiﬁer models, which are functions of the architecture and the parameters of the shadow classiﬁermodels.
Revised: The input to the meta-classiﬁer is a set of features from the trained shadowclassiﬁer models, which are functions of the architecture and the parameters of the shadow classiﬁer models.

Original: The meta classiﬁer is then trained for binary classiﬁcation of the chosen property.
Revised: The meta classiﬁer is then trained for binary classiﬁcation of the chosen property.

Original: Recentwork proposed to poison the training data set so as to improve attack performance Chase et al.[2021], but the resulting property inference attack is limited to distinguishing between two possibleoutcomes as is the case with rest of the literature in property inference attacks.
Revised: Recent work proposed to poison the training data set so as to improve attack performance Chase et al. [2021], but the resulting property inference attack is limited to distinguishing between two possible outcomes as is the case with rest of the literature in property inference attacks.

Original: Contrary to previouswork on property inference attack, our focus is on inferring the class label distribution, which canbe seen as a regression problem faced by the adversary, and hence distinguishes our work frombinary classiﬁcation-based property inference attacks prevalent in the literature.
Revised: Contrary to previous work on property inference attack, our focus is on inferring the class label distribution, which can be seen as a regression problem faced by the adversary, and hence distinguishes our work from binary classiﬁcation-based property inference attacks prevalent in the literature.

Original: Even for a binaryclassiﬁcation problem, class-label distribution inference is more difﬁcult than property inferencesince the adversary wants to infer more ﬁne-grained information about the dataset than the presenceor absence of a property.
Revised: Even for a binary classiﬁcation problem, class-label distribution inference is more difﬁcult than property inference since the adversary wants to infer more ﬁne-grained information about the dataset than the presence or absence of a property.

Original: Closely related to our work is Zhang et al.
Revised: Closely related to our work is Zhang et al.

Original: [2021], where authors proposed a meta-classiﬁer attackfor inferring the distribution of sensitive features, using an ensemble of binary classiﬁers trainedto decide whether the attribute appears in 10 , 30 , 50 , 70 , 90% of the dataset, but this approach doesnot yield an accurate estimate of the label distribution.
Revised: [2021], where authors proposed a meta-classiﬁer attack for inferring the distribution of sensitive features, using an ensemble of binary classiﬁers trained to decide whether the attribute appears in 10 , 30 , 50 , 70 , 90% of the dataset, but this approach does not yield an accurate estimate of the label distribution.

Original: In the context of online learning, yet closelyrelated to our work is Salem et al.
Revised: In the context of online learning, yet closely related to our work is Salem et al.

Original: [2020] where authors study the reconstruction of samples anddataset inference and solve a class label distribution inference problem similar to ours.
Revised: [2020], where authors study the reconstruction of samples and dataset inference, and solve a class label distribution inference problem similar to ours.

Original: They do,however, assume that the attacker has black-box access to the model before and after model updates which makes their attack model differ signiﬁcantly from ours.
Revised: They do, however, assume that the attacker has black-box access to the model before and after model updates, which makes their attack model differ signiﬁcantly from ours.

Original: 1.2 Organization
Revised: 1.2 Organization

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In Section 2 we describe the problem formulation, andin Section 3 we describe the proposed accuracy-augmented meta-classiﬁer attack .
Revised: In Section 2 we describe the problem formulation, and in Section 3 we describe the proposed accuracy-augmented meta-classiﬁer attack .

Original: In Section 4 wepresent numerical results for the UCI Census income and the MNIST data sets.
Revised: In Section 4 we present numerical results for the UCI Census income and the MNIST data sets.

Original: Section 5 concludesthe paper and discusses future work.
Revised: Section 5 concludes the paper and discusses future work.

Original:  Problem formulation
Revised:  Problem formulation

Original: Consider a target classiﬁer whose task given an input x ∈ R D is to identify the most suitable classfor the input amongst C classes.
Revised: Consider a target classiﬁer whose task given an input x ∈ R D is to identify the most suitable class for the input amongst C classes.

Original: Let the output of the classiﬁer be a vector of length C denoted by y ,where each entry [ y ] i is the probability that the input belongs to a particular class c , c = 1 , 2 , . . .
Revised: Let the output of the classiﬁer be a vector of length C denoted by y , where each entry [ y ] i is the probability that the input belongs to a particular class c , c = 1 , 2 , . . .

Original: C ,i.e., (cid:80) i
Revised: C , i.e., (cid:80) i

Original: [ y ] i = 1 .
Revised: [ y ] i = 1 .

Original: Therefore, the classiﬁer output y lies in the probability simplex ∆ C − 1 , i.e.,y ∈ ∆ C − 1 , also called the the C − 1 unit simplex.
Revised: Therefore, the classiﬁer output y lies in the probability simplex ∆ C − 1 , i.e., y ∈ ∆ C − 1 , also called the the C − 1 unit simplex.

Original: We model the target classiﬁer as a functionf t : R D → ∆ C − 1 parametrized by vector θ .
Revised: We model the target classiﬁer as a function f t : R D → ∆ C − 1 parametrized by vector θ .

Original: Then, the output of the classiﬁer is y =
Revised: Then, the output of the classiﬁer is y =

Original: f t ( x ; θ ) .
Revised: f t ( x ; θ ) .

Original: Theclassiﬁer f t described above is trained on a labeled dataset, D (cid:44) { x i , e i } i =1 , 2 ,...
Revised: The classiﬁer f t described above is trained on a labeled dataset, D (cid:44) { x i , e i } i =1 , 2 ,...

Original: N consisting of Ndata points where class label e i ∈ [0 , 1] C is a coordinate vector so that the entry corresponding toclass c is 1 and the rest are zero.
Revised: N consisting of N data points where class label e i ∈ [0 , 1] C is a coordinate vector so that the entry corresponding to class c is 1 and the rest are zero.

Original: We denote by N c the number of training samples with class label c .
Revised: We denote by N c the number of training samples with class label c .

Original: Then, the class label distribution of the training data is deﬁned as
Revised: Then, the class label distribution of the training data is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We consider cross entropy to be the loss function of the target classiﬁer as it is commonly used inclassiﬁcation problems,
Revised: We consider cross entropy to be the loss function of the target classiﬁer as it is commonly used in classiﬁcation problems,

Original: [Equation]
Revised: [Equation]

Original: The set of weights and biases θ are updated by minimizing the loss function in (2).
Revised: The set of weights and biases θ are updated by minimizing the loss function in (2).

Original: 2.1 Architecture of the target classiﬁer
Revised: 2.1 Architecture of the target classiﬁer

Original: In this work, we consider classiﬁers that are fully connected neural networks.
Revised: In this work, we consider classiﬁers that are fully connected neural networks.

Original: Let L denote thenumber of layers after the input, by m (cid:96) the neurons in layer (cid:96) , and by W 1 ∈ R D × m 1 , W 2 ∈107 R m 1 × m 2 , . . .
Revised: Let L denote the number of layers after the input, by m (cid:96) the neurons in layer (cid:96) , and by W 1 ∈ R D × m 1 , W 2 ∈ R m 1 × m 2 , . . .

Original: W (cid:96) ∈
Revised: W (cid:96) ∈

Original: R m (cid:96) − 1 × m (cid:96) , . . .
Revised: R m (cid:96) − 1 × m (cid:96) , . . .

Original: , W L ∈ R m L × C the connection weight matrices betweeenadjacent layers.
Revised: , W L ∈ R m L × C the connection weight matrices betweeen adjacent layers.

Original: Let the vector b (cid:96) ∈
Revised: Let the vector b (cid:96) ∈

Original: R m (cid:96) denote the biases of neurons in layer (cid:96) , and by σ (cid:96) ( . ) theactivation function for neurons in layer (cid:96) .
Revised: R m (cid:96) denote the biases of neurons in layer (cid:96) , and by σ (cid:96) ( . ) the activation function for neurons in layer (cid:96) .

Original: The activation function for the last layer L is sigmoid (orsoftmax for multi-class) so that the output y of the classiﬁer is normalized to be a probability massfunction.
Revised: The activation function for the last layer L is sigmoid (or softmax for multi-class) so that the output y of the classiﬁer is normalized to be a probability mass function.

Original: Thus, the parameters of the classiﬁer function f t consist of the elements of connectionweight matrices and the biases.
Revised: Thus, the parameters of the classiﬁer function f t consist of the elements of connection weight matrices and the biases.

Original: We can then represent parameter θ ∈ R M by concatenating all thevectorized weight matrices, vec ( W ) =
Revised: We can then represent parameter θ ∈ R M by concatenating all the vectorized weight matrices, vec ( W ) =

Original: [ ww. . .
Revised: [ ww [Equation]

Original: w mn ] , W ∈ R m × n , and biases, i.e.,
Revised: n , and biases, i.e. [Equation]

Original: [Equation]
Revised: [Equation]

Original: 2.2 Attack model
Revised: 2.2 Attack model

Original: We consider a white box attack, i.e., an adversary that has access to the architecture and the parametersθ of the model.
Revised: We consider a white box attack, i.e., an adversary that has access to the architecture and the parameters θ of the model.

Original: This could be the case, e.g., if the ML model is stored in an untrusted public cloud.
Revised: This could be the case, e.g., if the ML model is stored in an untrusted public cloud.

Original: We assume that the adversary cannot observe or interfere with the training process, and it cannottamper with the process of labeling the training data, i.e., the attacker is honest-but-curious.
Revised: We assume that the adversary cannot observe or interfere with the training process, and it cannot tamper with the process of labeling the training data, i.e., the attacker is honest-but-curious.

Original: Inaddition, to create shadow training datasets, it is assumed that the adversary has access to similartraining data that can be used to train shadow classiﬁers that imitate the target classiﬁer f t .
Revised: In addition, to create shadow training datasets, it is assumed that the adversary has access to similar training data that can be used to train shadow classiﬁers that imitate the target classiﬁer f t .

Original: Theaforementioned assumption is common in methods involving a meta-classiﬁer.
Revised: The aforementioned assumption is common in methods involving a meta-classiﬁer.

Original: In the absence of suchdata, synthetic or noisy real-world data could also be used to produce sufﬁciently accurate shadowclassiﬁers such as in Shokri et al.
Revised: In the absence of such data, synthetic or noisy real-world data could also be used to produce sufﬁciently accurate shadow classiﬁers such as in Shokri et al.

Original: The objective of the attacker is to obtain an accurate estimate ˆ p of the class label distribution ofthe data used to train the target classiﬁer.
Revised: The objective of the attacker is to obtain an accurate estimate ˆ p of the class label distribution of the data used to train the target classiﬁer.

Original: We quantify the accuracy of the estimate using the KLdivergence between the estimated and the true distribution,
Revised: We quantify the accuracy of the estimate using the KL divergence between the estimated and the true distribution,

Original: [Equation]
Revised: [Equation]

Original: which is a widely used measure for quantifying how much two distributions differ.
Revised: which is a widely used measure for quantifying how much two distributions differ.

Original: The objective ofthe attacker is thus to minimize D p t (ˆ p ) .
Revised: The objective of the attacker is thus to minimize D p t (ˆ p ) .

Original:  Accuracy-augmented permutation-invariant meta-classiﬁer attack
Revised:  Accuracy-augmented permutation-invariant meta-classiﬁer attack

Original: The attack we propose in the following makes use of a meta-classiﬁer that is trained to produce anestimate ˆ p of the training data class label distribution, when it is fed with the parameters θ of the targetclassiﬁer f t .
Revised: The attack we propose in the following makes use of a meta-classiﬁer that is trained to produce an estimate ˆ p of the training data class label distribution, when it is fed with the parameters θ of the target classiﬁer f t .

Original: Furthermore, the attack also utilizes an auxiliary data set denoted by D aux containing
Revised: Furthermore, the attack also utilizes an auxiliary data set denoted by D aux containing

Original: N aux samples from each class c to evaluate the classiﬁcation accuracy of the target classiﬁer modelfor data in each class.
Revised: N aux samples from each class c to evaluate the classiﬁcation accuracy of the target classiﬁer model for data in each class.

Original: The accuracy on auxiliary data, denoted by a ∈ [0 , 1] C , is also used as aninput to the meta-classiﬁer.
Revised: The accuracy on auxiliary data, denoted by a ∈ [0 , 1] C , is also used as an input to the meta-classiﬁer.

Original: We thus deﬁne the meta classiﬁer as a function g MC : R M + C → ∆ C − 1 ,parametrized by a vector of weights ω MC , which are to be learned.
Revised: We thus deﬁne the meta classiﬁer as a function g MC : R M + C → ∆ C − 1 , parametrized by a vector of weights ω MC , which are to be learned.

Original: The output of the meta-classiﬁeris ˆ p = g MC ( θ , a ; ω MC ) .
Revised: The output of the meta-classiﬁer is ˆ p = g MC ( θ , a ; ω MC ) .

Original: Although the term ‘classiﬁer’ is a misnomer, we retain the term due to theanalogy to the meta-classiﬁer method used in the literature.
Revised: Although the term ‘classiﬁer’ is a misnomer, we retain the term due to the analogy to the meta-classiﬁer method used in the literature.

Revised: 3.1 Meta-classiﬁer training data set

Original: [Figure]
Revised: [Figure]

Original: Figure 1 shows a ﬂow chart of the proposed shadow-training methodology used for training the metaclassiﬁer.
Revised: Figure 1 shows a ﬂow chart of the proposed shadow-training methodology used for training the metaclassiﬁer.

Original: First, a collection of shadow training datasets {SD k } k , k = 1 , 2 , . . .
Revised: First, a collection of shadow training datasets {SD k } k , k = 1 , 2 , . . .

Original: , K , is created; theclass label distribution of data set SD k is p k , i.e., SD k (cid:44) { x i , e i } i =1 , 2 ,...
Revised: , K , is created; the class label distribution of data set SD k is p k , i.e., SD k (cid:44) { x i , e i } i =1 , 2 ,...

Original: N such that 1 N (cid:80) i e i = p k .
Revised: N such that 1 N (cid:80) i e i = p k .

Original: The number of shadow-training sets K and the class label distributions { p k } k =1 , 2 ,...,K are parameters144 chosen by the attacker.
Revised: The number of shadow-training sets K and the class label distributions { p k } k =1 , 2 ,...,K are parameters chosen by the attacker.

Original: The set of distributions { p k } can be chosen in various ways from ∆ C − 1 , e.g.,uniformly sampled or at random, as we discuss later in Section 3.3.
Revised: The set of distributions { p k } can be chosen in various ways from ∆ C − 1 , e.g., uniformly sampled or at random, as we discuss later in Section 3.3.

Original: We use shadow-training dataset SD k with class label distribution p k to train a shadow classiﬁer f kwith the same architecture as the target classiﬁer f t .
Revised: We use shadow-training dataset SD k with class label distribution p k to train a shadow classiﬁer f k with the same architecture as the target classiﬁer f t .

Original: The parameters θ k of the shadow classiﬁer f ktrained using the shadow-training dataset SD k are used as training data samples for the meta-classiﬁeralong with the class label distribution p k as the desired output.
Revised: The parameters θ k of the shadow classiﬁer f k trained using the shadow-training dataset SD k are used as training data samples for the meta-classiﬁer along with the class label distribution p k as the desired output.

Original: In addition, the accuracy of theshadow classiﬁer f k on the auxilliary data, a k , is also used as an input.
Revised: In addition, the accuracy of the shadow classiﬁer f k on the auxilliary data, a k , is also used as an input.

Original: The accuracy for class c isdeﬁned as
Revised: The accuracy for class c is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where 1 { . } is an indicator function which is 1 if the condition in its argument is true and zerootherwise, and (cid:15) is a threshold or meta-parameter used to assign the class label based on the outputof the classiﬁer.
Revised: where 1 { . } is an indicator function which is 1 if the condition in its argument is true and zero otherwise, and (cid:15) is a threshold or meta-parameter used to assign the class label based on the output of the classiﬁer.

Original: The idea behind using accuracy of the shadow classiﬁer is that the accuracyon samples with a certain class-label is higher if that class is over-represented in the trainingdataset and therefore, additional information regarding accuracy could be beneﬁcial in the inferenceof class-label distribution.
Revised: The idea behind using accuracy of the shadow classiﬁer is that the accuracy on samples with a certain class-label is higher if that class is over-represented in the training dataset and therefore, additional information regarding accuracy could be beneﬁcial in the inference of class-label distribution.

Original: The labeled training dataset for the meta-classiﬁer is then deﬁned as
Revised: The labeled training dataset for the meta-classiﬁer is then deﬁned as

Revised: D MC (cid:44)

Original: [Equation]
Revised: [Equation]

Original: Aligned with the goal of the attacker, discussed in Section 2.2, the meta-classiﬁer is trained tominimize the KL-divergence between p k and the estimate ˆ p k = g MC ( θ k , a k ; ω MC ) for all Ksamples.
Revised: Aligned with the goal of the attacker, discussed in Section 2.2, the meta-classiﬁer is trained to minimize the KL-divergence between p k and the estimate ˆ p k = g MC ( θ k , a k ; ω MC ) for all K samples.

Original: Therefore, the loss-function of the meta-classiﬁer is
Revised: Therefore, the loss-function of the meta-classiﬁer is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Observe from (6) that the KL divergence consists of two terms, the cross entropy between distributionsp k and ˆ p k , and the negative entropy of distribution p k .
Revised: Observe from (6) that the KL divergence consists of two terms, the cross entropy between distributions p k and ˆ p k , and the negative entropy of distribution p k .

Original: Of these two, only the cross-entropy terminﬂuences the update of weights ω MC , hence the loss function is effectively
Revised: Of these two, only the cross-entropy term inﬂuences the update of weights ω MC , hence the loss function is effectively

Original: [Equation]
Revised: [Equation]

Original: 3.2 Accuracy-augmented permutation-invariant meta-classiﬁer architecture
Revised: 3.2 Accuracy-augmented permutation-invariant meta-classiﬁer architecture

Original: Recall that the training data set D MC for the meta-classiﬁer consists of collections of weights andbiases.
Revised: Recall that the training data set D MC for the meta-classiﬁer consists of collections of weights and biases.

Original: It is thus important for the meta-classiﬁer to be permutation invariant.
Revised: It is thus important for the meta-classiﬁer to be permutation invariant.

Original: Hence, we use
Revised: Hence, we use

Original: DeepSets Zaheer et al.
Revised: DeepSets Zaheer et al.

Original: [2017] to obtain a succinct representation of the weights and biases θ k thatare fed as input to the meta-classiﬁer, similar to Ganju et al.
Revised: [2017] to obtain a succinct representation of the weights and biases θ k that are fed as input to the meta-classiﬁer, similar to Ganju et al.

Original: DeepSets allows the inputs toundergo a non-linear transformation before they are aggregated (summed) and transformed again.
Revised: DeepSets allows the inputs to undergo a non-linear transformation before they are aggregated (summed) and transformed again.

Original: The transformation before aggregation allows one to create arbitrary moments of the inputs (e.g.,second moment by squaring).
Revised: The transformation before aggregation allows one to create arbitrary moments of the inputs (e.g., second moment by squaring).

Original: The main advantage of this approach is that we learn a set-basedrepresentation that does not depend on the order of neurons in a certain layer.
Revised: The main advantage of this approach is that we learn a set-based representation that does not depend on the order of neurons in a certain layer.

Original: To obtain permutationinvariance, we sum the transformed connection weights and biases of neurons in a layer therebydisregarding the ordering.
Revised: To obtain permutation invariance, we sum the transformed connection weights and biases of neurons in a layer thereby disregarding the ordering.

Original: The proposed meta-classiﬁer architecture is organized as follows.
Revised: The proposed meta-classiﬁer architecture is organized as follows.

Original: • A fully connected single layer network is employed to take as inputs the connection weights and biases of the m 1 neurons in layer (cid:96) = 1 .
Revised: • A fully connected single layer network is employed to take as inputs the connection weights and biases of the m 1 neurons in layer (cid:96) = 1 .

Original: Let the weights of this single layer network be denoted by ω 1 ∈ R D +1 .
Revised: Let the weights of this single layer network be denoted by ω 1 ∈ R D +1 .

Original: The output of the layer is
Revised: The output of the layer is

Original: [Equation]
Revised: [Equation]

Original: where φ  is the activation function employed.
Revised: where φ 1 is the activation function employed.

Original: • For the subsequent layers (cid:96) = 2 , 3 , . . .
Revised: • For the subsequent layers (cid:96) = 2 , 3 , . . .

Original: , L of weights and biases, along with the input of connection weights and biases of neurons , the output from 
Revised: , L of weights and biases, along with the input of connection weights and biases of neurons, the output from the previous layer, q (cid:96) − 1 ∈ R m

Original: the previous

Original: layer, q (cid:96)

Original: − 1 ∈

Original: R

Original: m

Original: [Equation]
Revised: [Equation]

Original: − 1is used as an input
Revised: is used as an input.

Original: More speciﬁcally, let the combined input be
Revised: More speciﬁcally, let the combined input be

Original: [Equation]
Revised: [Equation]

Original: where 1 is a vector of ones of size m (cid:96) .
Revised: where 1 is a vector of ones of size m (cid:96) .

Original: The addition of the outer product ( q (cid:96) − 1 1 (cid:62) ) ensuresthat q (cid:96) − 1 is added to every column of the connection weight matrix W (cid:96) .
Revised: The addition of the outer product ( q (cid:96) − 1 1 (cid:62) ) ensures that q (cid:96) − 1 is added to every column of the connection weight matrix W (cid:96) .

Original: Elements in columnj of W (cid:96) corresponds to the connection weights from all neurons in layer (cid:96) − 1 to neuronj in layer (cid:96) .
Revised: Elements in column j of W (cid:96) corresponds to the connection weights from all neurons in layer (cid:96) − 1 to neuron j in layer (cid:96) .

Original: Now, let the weights of the single layer network for layer (cid:96) be denoted byω (cid:96) ∈
Revised: Now, let the weights of the single layer network for layer (cid:96) be denoted by ω (cid:96) ∈

Original: R m (cid:96) − 1 +1 .
Revised: R m (cid:96) − 1 +1 .

Original: Then its output is given by
Revised: Then its output is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  The transformed weights and biases from all layers are summed, i.e.,(cid:80) [ q 1 ] i , (cid:80) [ q 2 ] i , . . .
Revised: • The transformed weights and biases from all layers are summed, i.e., (cid:80) [ q 1 ] i , (cid:80) [ q 2 ] i , . . .

Original: , (cid:80) [ q L ] i , and the accuracy on the auxiliary dataset a are combined using another fully connected single layer network whose weights are denoted byΩ ρ ∈ R L + C × C .
Revised: , (cid:80) [ q L ] i , and the accuracy on the auxiliary dataset a are combined using another fully connected single layer network whose weights are denoted by Ω ρ ∈ R L + C × C .

Original: Finally, a softmax operation ρ at the output of the fully connected layerensures that the output ˆ p of the meta-classiﬁer is in the probability simplex ∆ C − 1 .
Revised: Finally, a softmax operation ρ at the output of the fully connected layer ensures that the output ˆ p of the meta-classiﬁer is in the probability simplex ∆ C − 1 .

Original: The lastlayer of the meta-classiﬁer is thus
Revised: The last layer of the meta-classiﬁer is thus

Original: [Equation]
Revised: [Equation]

Original: To summarize, the parameters of the meta-classiﬁer consist of the concatenation of L vectors ofweights, one for each layer and a ﬁnal connection weight matrix at the output,
Revised: To summarize, the parameters of the meta-classiﬁer consist of the concatenation of L vectors of weights, one for each layer and a ﬁnal connection weight matrix at the output,

Original: [Equation]
Revised: [Equation]

Original: Now, the proposed architecture for the meta-classiﬁer is different from Ganju et al.
Revised: Now, the proposed architecture for the meta-classiﬁer is different from Ganju et al.

Original: [2018] in twocrucial aspects.
Revised: [2018] in two crucial aspects.

Original: Firstly, the transformed weights and biases from the previous layer i.e. q (cid:96) − 1 are notconcatenated to the weights and biases of layer (cid:96) as in Ganju et al.
Revised: Firstly, the transformed weights and biases from the previous layer i.e. q (cid:96) − 1 are not concatenated to the weights and biases of layer (cid:96) as in Ganju et al.

Original: Instead, in the proposedarchitecture, the combined input adds the output from the previous layer to every column of theconnection weight matrix.
Revised: Instead, in the proposed architecture, the combined input adds the output from the previous layer to every column of the connection weight matrix.

Original: This is done so that the the neuron j in layer (cid:96) − 1 and the transformedoutput [ q (cid:96) − 1 ] j have the same parameter or weight [ ω (cid:96) ]
Revised: This is done so that the the neuron j in layer (cid:96) − 1 and the transformed output [ q (cid:96) − 1 ] j have the same parameter or weight [ ω (cid:96) ]

Original: j in the meta-classiﬁer.
Revised: j in the meta-classiﬁer.

Original: Such an architecture, reduces the number of parameters of the meta-classiﬁer and also helps to combat over-ﬁtting.
Revised: Such an architecture, reduces the number of parameters of the meta-classiﬁer and also helps to combat over-ﬁtting.

Original: In theproposed architecture, ω (cid:96) is of size m (cid:96) + 1 whereas for architecture in Ganju et al.
Revised: In the proposed architecture, ω (cid:96) is of size m (cid:96) + 1 whereas for architecture in Ganju et al.

Original: [2018], ω (cid:96) wouldbe of size 2 m (cid:96) + 1 .
Revised: [2018], ω (cid:96) would be of size 2 m (cid:96) + 1 .

Original: Secondly, the accuracy on the auxiliary dataset is augmented in the last layerwhich is an addition to the architecture in Ganju et al.
Revised: Secondly, the accuracy on the auxiliary dataset is augmented in the last layer which is an addition to the architecture in Ganju et al.

Original: We see that accuracy-augmentationleads to improvement over the architecture in Ganju et al.
Revised: We see that accuracy-augmentation leads to improvement over the architecture in Ganju et al.

Original: Note that even though the architectures of the proposed meta-classiﬁer and the one in Ganju et al.[2018] can be compared as discussed above, the attack goals accomplished by them are very different.
Revised: Note that even though the architectures of the proposed meta-classiﬁer and the one in Ganju et al. [2018] can be compared as discussed above, the attack goals accomplished by them are very different.

Original: In other words, the inference of class label distribution of training data is a novel attack.
Revised: In other words, the inference of class label distribution of training data is a novel attack.

Original: 3.3 Sampling schemes to select shadow training data sets
Revised: 3.3 Sampling schemes to select shadow training data sets

Original: For a target binary classiﬁer, the probability simplex ∆ C − 1 is a line in R 2 between points (0 , 1) and(1 , 0) , and one needs to choose p k from the line segment.
Revised: For a target binary classiﬁer, the probability simplex ∆ C − 1 is a line in R 2 between points (0 , 1) and (1 , 0) , and one needs to choose p k from the line segment.

Original: The attacker could use uniform sampling onthe line segment or it could sample at the ends of the line segment if it has prior information that theclass label distribution is skewed, i.e., training data is imbalanced.
Revised: The attacker could use uniform sampling on the line segment or it could sample at the ends of the line segment if it has prior information that the class label distribution is skewed, i.e., training data is imbalanced.

Original: However, sampling for multi-classclassiﬁers implies choosing p k from a higher dimensional probability simplex.
Revised: However, sampling for multi-class classiﬁers implies choosing p k from a higher dimensional probability simplex.

Original: For example, for a3 -class target classiﬁer, a uniform sampling strategy amounts to choosing points uniformly on the gridwhich is an equilateral triangle formed in R 3 with vertices (0 , 0 , 1) , (0 , 1 , 0) , (1 , 0 , 0) .
Revised: For example, for a 3 -class target classiﬁer, a uniform sampling strategy amounts to choosing points uniformly on the grid which is an equilateral triangle formed in R 3 with vertices (0 , 0 , 1) , (0 , 1 , 0) , (1 , 0 , 0) .

Original: Alternativestrategies include sampling only on the edges of the triangle, and sampling close to the vertices.
Revised: Alternative strategies include sampling only on the edges of the triangle, and sampling close to the vertices.

Original: Amajor challenge in multi-dimensional sampling is that many samples are needed in order to havea certain minimum sample density for all the regions in the simplex, growing exponentially withthe dimension.
Revised: A major challenge in multi-dimensional sampling is that many samples are needed in order to have a certain minimum sample density for all the regions in the simplex, growing exponentially with the dimension.

Original: Since each sample p k corresponds to a shadow training dataset, if the attacker hasno prior information about the class label distribution used by the target classiﬁer then the numberof shadow datasets needed may render the approach infeasible.
Revised: Since each sample p k corresponds to a shadow training dataset, if the attacker has no prior information about the class label distribution used by the target classiﬁer then the number of shadow datasets needed may render the approach infeasible.

Original: This is admittedly a limitation ofthe shadow-training approach for multi-class target classiﬁers.
Revised: This is admittedly a limitation of the shadow-training approach for multi-class target classiﬁers.

Original: An empirical evaluation for 3 -classclassiﬁer is presented in Appendix A.1 to understand what are sufﬁciently good sampling schemes tochoose p k .
Revised: An empirical evaluation for 3 -class classiﬁer is presented in Appendix A.1 to understand what are sufﬁciently good sampling schemes to choose p k .

Original:  Numerical results
Revised:  Numerical results

Original: We present numerical results based on experiments conducted on two datasets, the UCI Censusincome dataset Dua and Graff [2017] and the MNIST handwritten digit classiﬁcation dataset LeCun[1998].
Revised: We present numerical results based on experiments conducted on two datasets, the UCI Census income dataset Dua and Graff [2017] and the MNIST handwritten digit classiﬁcation dataset LeCun [1998].

Original: For the UCI Census income dataset, the task is income classiﬁcation as above or below50 k USD which is a binary classiﬁcation problem.
Revised: For the UCI Census income dataset, the task is income classiﬁcation as above or below 50 k USD which is a binary classiﬁcation problem.

Original: For the MNIST dataset, we study both binaryclassiﬁcation (between digits 0 and 1 ) and the multi-class classiﬁcation.
Revised: For the MNIST dataset, we study both binary classiﬁcation (between digits 0 and 1 ) and the multi-class classiﬁcation.

Original: In the multi-class settingwe consider classiﬁcation among 3 and 4 digits (classes) because of the sampling complexity whengenerating the shadow-training datasets, as discussed in Section 3.3.
Revised: In the multi-class setting we consider classiﬁcation among 3 and 4 digits (classes) because of the sampling complexity when generating the shadow-training datasets, as discussed in Section 3.3.

Original: We evaluate the beneﬁt ofaccuracy augmentation and trimmed architecture by comparing the performance of the proposedmeta-classiﬁer trained with the accuracy on the auxiliary datasets with that of a meta-classiﬁer whosearchitecture is akin to that in Ganju et al.
Revised: We evaluate the beneﬁt of accuracy augmentation and trimmed architecture by comparing the performance of the proposed meta-classiﬁer trained with the accuracy on the auxiliary datasets with that of a meta-classiﬁer whose architecture is akin to that in Ganju et al.

Original: [2018], albeit their goal of the meta-classiﬁer is differentfrom ours.
Revised: [2018], albeit their goal of the meta-classiﬁer is different from ours.

Original: We sample the probability simplex uniformly with step-size ∆ p in each dimension toobtain the label proportions p k for generating shadow-training datasets.
Revised: We sample the probability simplex uniformly with step-size ∆ p in each dimension to obtain the label proportions p k for generating shadow-training datasets.

Original: The test set to evaluate theperformance of the meta-classiﬁer is made of class label distributions with the smallest step-size thatwas used to train the meta-classiﬁer.
Revised: The test set to evaluate the performance of the meta-classiﬁer is made of class label distributions with the smallest step-size that was used to train the meta-classiﬁer.

Original: All the target classiﬁers f t are fully-connected neural networks with L = 3 layers, ReLU activationfunction in the hidden layers, while the last layer has sigmoid and softmax for binary and multi-classclassiﬁcation, respectively.
Revised: All the target classiﬁers f t are fully-connected neural networks with L = 3 layers, ReLU activation function in the hidden layers, while the last layer has sigmoid and softmax for binary and multi-class classiﬁcation, respectively.

Original: As a result, the accuracy-augmented meta-classiﬁers g MC have L = 3weight vectors and 1 connection matrix overall as in (12) and the activation function φ (cid:96) is chosen asthe ELU activation function (exponential linear unit)
Revised: As a result, the accuracy-augmented meta-classiﬁers g MC have L =weight vectors and 1 connection matrix overall as in (12) and the activation function φ (cid:96) is chosen as the ELU activation function (exponential linear unit)

Original: Clevert et al.
Revised: Clevert et al.

Original: 4.1 UCI Census income dataset
Revised: 4.1 UCI Census income dataset

Original: For the UCI Census income dataset, the target classiﬁer is trained to classify the annual income ofpeople being above or below 50 k US dollars based on demographical and employment related featureinputs.
Revised: For the UCI Census income dataset, the target classiﬁer is trained to classify the annual income of people being above or below 50 k US dollars based on demographical and employment related feature inputs.

Original: Some of the feature inputs are categorical and therefore were converted into one-hot vectors.
Revised: Some of the feature inputs are categorical and therefore were converted into one-hot vectors.

Original: The 3 layers of the target classiﬁer have m 1 = 32 , m 2 = 16 , m 3 = 8 neurons.
Revised: The 3 layers of the target classiﬁer have m 1 = 32 , m 2 = 16 , m 3 = 8 neurons.

Original: Since the target is a
Revised: Since the target is a

Original: p
Revised: p

Original: E s t i m a t e ˆ p
Revised: E s t i m a t e ˆ p

Original: MCoutput ˆp ˆ p = p
Revised: MC ˆ ˆ p = p

Original: [Table]
Revised: [Table]

Original: binary classiﬁer, we denote the fraction of the dataset used for training that has income greater than50 k USD by p k and the other class consequently has proportion 1 − p k .
Revised: binary classiﬁer, we denote the fraction of the dataset used for training that has income greater than 50 k USD by p k and the other class consequently has proportion 1 − p k .

Original: The shadow classiﬁers were trained using shadow-training datasets with N = 5000 data points andwith proportions p k uniform on ∆ 1 with ∆ p = 0 .
Revised: The shadow classiﬁers were trained using shadow-training datasets with N = 5000 data points and with proportions p k uniform on ∆ 1 with ∆ p = 0 .

Original: 01 i.e. p k = 0 , 0 .
Revised: 01 i.e. p k = 0 , 0 .

Original: Furthermore,for each value of p k we created 10 different datasets by resampling from the UCI Census incomedataset to account for the heterogeneity of the feature inputs.
Revised: Furthermore, for each value of p k we created 10 different datasets by resampling from the UCI Census income dataset to account for the heterogeneity of the feature inputs.

Original: Out of 10 data sets per proportionp k , 8 were used for training and 2 for testing.
Revised: Out of 10 data sets per proportion p k , 8 were used for training and 2 for testing.

Original: Thus, overall we have K = 800 shadow trainingdatasets and 200 test data sets.
Revised: Thus, overall we have K = 800 shadow training datasets and 200 test data sets.

Original: The proposed accuracy-augmented meta-classiﬁer has 4 parametervectors of dimensions as mentioned in Section 3.2.
Revised: The proposed accuracy-augmented meta-classiﬁer has 4 parameter vectors of dimensions as mentioned in Section 3.2.

Original: Figure 2 shows a scatter plot of the estimatedlabel proportion ˆ p with respect to the actual label proportion p for the UCI Census income dataset.
Revised: Figure 2 shows a scatter plot of the estimated label proportion ˆ p with respect to the actual label proportion p for the UCI Census income dataset.

Original: The ﬁgure shows that the points are close to the p
Revised: The ﬁgure shows that the points are close to the p

Original: = ˆ p straight line, highlighting the efﬁcacy ofthe proposed approach.
Revised: = ˆ p straight line, highlighting the efﬁcacy of the proposed approach.

Original: Figure 3 shows the box-and-whisker plot of KL divergence between theestimated and actual label proportion distribution, D p t (ˆ p ) , for the proposed accuracy-augmentedmeta-classiﬁer and the baseline meta-classiﬁer without accuracy-augmentation and architecturalchanges, as a function of the step size ∆ p .
Revised: Figure 3 shows the box-and-whisker plot of KL divergence between the estimated and actual label proportion distribution, D p t (ˆ p ) , for the proposed accuracy-augmented meta-classiﬁer and the baseline meta-classiﬁer without accuracy-augmentation and architectural changes, as a function of the step size ∆ p .

Original: The test dataset contains 200 cases.
Revised: The test dataset contains 200 cases.

Original: The results show thatthe proposed accuracy-augmented meta-classiﬁer outperforms the baseline meta-classiﬁer, showingthat the reduction in number of parameters and augmentation with accuracy of the auxiliary datais beneﬁcial and improves the estimation capability of the meta-classiﬁer.
Revised: The results show that the proposed accuracy-augmented meta-classiﬁer outperforms the baseline meta-classiﬁer, showing that the reduction in number of parameters and augmentation with accuracy of the auxiliary data is beneﬁcial and improves the estimation capability of the meta-classiﬁer.

Original: The improvement isespecially prominent when ∆ p is large.
Revised: The improvement is especially prominent when ∆ p is large.

Original: This implies that accuracy-augmentation and lower numberof parameters allows the attacker to carry out attacks with a smaller shadow training dataset, i.e., withlower granularity, and still estimate the true label proportion.
Revised: This implies that accuracy-augmentation and lower number of parameters allows the attacker to carry out attacks with a smaller shadow training dataset, i.e., with lower granularity, and still estimate the true label proportion.

Original: In order to gather more intuition, we alsoreport the average mean-squared error in Table 1 between the estimated and true label distribution,
Revised: In order to gather more intuition, we also report the average mean-squared error in Table 1 between the estimated and true label distribution,

Original: [Equation]
Revised: [Equation]

Original: (13) for the baseline meta-classiﬁer and the proposed meta-classiﬁer.
Revised: (13) for the baseline meta-classiﬁer and the proposed meta-classiﬁer.

Original: A monotonic increase with respectto step-size is observed which is to be expected.
Revised: A monotonic increase with respect to step-size is observed which is to be expected.

Original: The table shows a decrease in average MSE whenusing the proposed meta-classiﬁer when compared to the baseline.
Revised: The table shows a decrease in average MSE when using the proposed meta-classiﬁer when compared to the baseline.

Original: In fact, the maximum decrease of
Revised: In fact, the maximum decrease of

Original: MSE is 29 .
Revised: MSE is 29 .

Original: 6% relative to the baseline for a step-size of 0 . 1 .
Revised: 6% relative to the baseline for a step-size of 0 . 1 .

Original: 4.2 MNIST dataset: Classiﬁcation of digits 0 and1
Revised: 4.2 MNIST dataset: Classiﬁcation of digits 0 and

Original: For binary classiﬁcation on the MNIST dataset, we trained a binary target classiﬁer for digits 0 and 1in the dataset.
Revised: For binary classiﬁcation on the MNIST dataset, we trained a binary target classiﬁer for digits 0 andin the dataset.

Original: The input pixels are arranged as a feature vector with dimension D = 784 .
Revised: The input pixels are arranged as a feature vector with dimension D = 784 .

Original: The 3 layersfor the target classiﬁer have m 1 = 128 , m 2 = 32 , m 3 = 16 neurons.
Revised: The 3 layers for the target classiﬁer have m 1 = 128 , m 2 = 32 , m 3 = 16 neurons.

Original: Since the target is a binaryclassiﬁer, we denote the proportion of digit 0 used for training by p k and of digit 1 by 1 − p k .
Revised: Since the target is a binary classiﬁer, we denote the proportion of digit 0 used for training by p k and of digit 1 by 1 − p k .

Original: Similarto the UCI Census income dataset, the shadow-classiﬁer datasets were also created with different
Revised: Similar to the UCI Census income dataset, the shadow-classiﬁer datasets were also created with different

Original: p
Revised: p

Revised: E s t i m a t

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Proposed
Revised: Proposed

Original: [Table]
Revised: [Table]

Original: proportions p k = 0 , 0 . 01 , 0 .
Revised: proportions p k = 0 , 0 . 01 , 0 .

Original: 1 , with N = 4000 data points each.
Revised: 1 , with N = 4000 data points each.

Original: The rest of the evaluationmethodology is the same as for the UCI Census income dataset binary classiﬁer experiment.
Revised: The rest of the evaluation methodology is the same as for the UCI Census income dataset binary classiﬁer experiment.

Original: Figure 4shows the estimated ˆ p versus actual label proportion p .
Revised: Figureshows the estimated ˆ p versus actual label proportion p .

Original: Overall, the meta-classiﬁer is not as accuratefor MNIST as for the UCI Census income classiﬁer.
Revised: Overall, the meta-classiﬁer is not as accurate for MNIST as for the UCI Census income classiﬁer.

Original: We attribute this to the higher complexity of theinput feature space of MNIST, which makes the training of the meta-classiﬁer less effective.
Revised: We attribute this to the higher complexity of the input feature space of MNIST, which makes the training of the meta-classiﬁer less effective.

Original: However,the meta-classiﬁer attack can still satisfactorily estimate the input label distribution for low and highp values, i.e., when the training data set is unbalanced.
Revised: However, the meta-classiﬁer attack can still satisfactorily estimate the input label distribution for low and high p values, i.e., when the training data set is unbalanced.

Original: Figure 5 shows the box-and-whisker plot ofthe KL divergence between the estimated and true label proportion distribution, D p t (ˆ p ) , as a functionof the sampling step-size ∆ p .
Revised: Figure 5 shows the box-and-whisker plot of the KL divergence between the estimated and true label proportion distribution, D p t (ˆ p ) , as a function of the sampling step-size ∆ p .

Original: As for the UCI Census dataset, the proposed accuracy-augmentedmeta classiﬁer outperforms the baseline meta-classiﬁer attack for all values of the step size ∆ p .
Revised: As for the UCI Census dataset, the proposed accuracy-augmented meta classiﬁer outperforms the baseline meta-classiﬁer attack for all values of the step size ∆ p .

Original: Theadvantage of accuracy augmentation and reduced parameters is even clearer for MNIST as comparedto UCI Census dataset.
Revised: The advantage of accuracy augmentation and reduced parameters is even clearer for MNIST as compared to UCI Census dataset.

Original: Table 2 also highlights the superiority of the proposed meta-classiﬁer overthe baseline since the average MSE is lower at all time steps.
Revised: Table 2 also highlights the superiority of the proposed meta-classiﬁer over the baseline since the average MSE is lower at all time steps.

Original: The maximum relative decrease withrespect to the baseline is 52 .
Revised: The maximum relative decrease with respect to the baseline is 52 .

Original: 6% at step-size 0 . 04 .
Revised: 6% at step-size 0 . 04 .

Original:  Conclusions and future work
Revised:  Conclusions and future work

Original: In this paper, we introduced a new type of property inference attack that aims at inferring the classlabel distribution of the training data from weights and biases of a trained classiﬁer that has fullyconnected neural network architecture.
Revised: In this paper, we introduced a new type of property inference attack that aims at inferring the class label distribution of the training data from weights and biases of a trained classiﬁer that has fully connected neural network architecture.

Original: We proposed a shadow-training based accuracy-augmentedmeta-classiﬁer for this purpose.
Revised: We proposed a shadow-training based accuracy-augmented meta-classiﬁer for this purpose.

Original: The empirical results on the UCI Income dataset and the MNISTdataset show the efﬁcacy of the proposed method for class label distribution inference.
Revised: The empirical results on the UCI Income dataset and the MNIST dataset show the efﬁcacy of the proposed method for class label distribution inference.

Original: Possiblefuture research directions include exploring the potential of adaptive or online sampling schemes forshadow-training data sets improving the performance of multi-class meta-classiﬁer attacks, and thedevelopment of mitigation schemes against such meta-classiﬁer attacks.
Revised: Possible future research directions include exploring the potential of adaptive or online sampling schemes for shadow-training data sets improving the performance of multi-class meta-classiﬁer attacks, and the development of mitigation schemes against such meta-classiﬁer attacks.

Original: Future work also includesdeveloping new meta-classiﬁer architecture speciﬁc to other types of deep learning networks.
Revised: Future work also includes developing new meta-classiﬁer architecture speciﬁc to other types of deep learning networks.

Revised: Acknowledgments and Disclosure of Funding

Revised: This work was partly funded by the Vinnova Competence Center for Trustworthy Edge Computing

Revised: Systems and Applications (TECoSA) at KTH and by the Swedish Foundation for Strategic Research through the CLAS project (grant RIT17-0046).

Original: A AppendixA.1 Multi-class target classiﬁer
Revised: A Appendix A.1 Multi-class target classiﬁer

Original: For the multi-class target classiﬁer, we use the MNIST dataset with 3 and 4 digits for classiﬁcation.
Revised: For the multi-class target classiﬁer, we use the MNIST dataset with 3 and 4 digits for classiﬁcation.

Original: For the 3-class and 4-class classiﬁer we used digits 0 , 1 , 2 and 0 , 1 , 2 , 3 , respectively.
Revised: For the 3-class and 4-class classiﬁer we used digits 0 , 1 , 2 and 0 , 1 , 2 , 3 , respectively.

Original: The targetclassiﬁer for the multi-class MNIST classiﬁer has the same architecture as the binary MNIST classiﬁerwith the only difference being the number of outputs and the use of softmax as the activation functionat the last layer.
Revised: The target classiﬁer for the multi-class MNIST classiﬁer has the same architecture as the binary MNIST classiﬁer with the only difference being the number of outputs and the use of softmax as the activation function at the last layer.

Original: The architecture of the meta-classiﬁer attack is also the same as in the binary MNISTcase, with the difference that the number of outputs is C for ( C = 3 and C = 4 , respectively), andsoftmax is the activation function ρ at the last layer of the meta-classiﬁer.
Revised: The architecture of the meta-classiﬁer attack is also the same as in the binary MNIST case, with the difference that the number of outputs is C for ( C = 3 and C = 4 , respectively), and softmax is the activation function ρ at the last layer of the meta-classiﬁer.

Original: Figure 6 shows the sampledpoints p k for the uniformly sampled label distributions employed to generate data for the shadowclassiﬁers with different p k .
Revised: Figure 6 shows the sampled points p k for the uniformly sampled label distributions employed to generate data for the shadow classiﬁers with different p k .

Original: As discussed in Section 3.3, for the 3-class target classiﬁer, p k must bedrawn from the probability simplex ∆ 2 .
Revised: As discussed in Section 3.3, for the 3-class target classiﬁer, p k must be drawn from the probability simplex ∆ 2 .

Original: Figure 7 shows the box-and-whisker plot of KL divergencebetween the actual and the estimated distributions for the meta-classiﬁer with accuracy augmentationand without, as a function of the step size ∆ p .
Revised: Figure 7 shows the box-and-whisker plot of KL divergence between the actual and the estimated distributions for the meta-classiﬁer with accuracy augmentation and without, as a function of the step size ∆ p .

Original: While the overall accuracy of the attack is worsethan that for binary classiﬁcation, the proposed accuracy augmentation is clearly beneﬁcial for theattack.
Revised: While the overall accuracy of the attack is worse than that for binary classiﬁcation, the proposed accuracy augmentation is clearly beneﬁcial for the attack.

Original: Table 3 shows the average MSE for the proposed and the baseline meta-classiﬁer, and showsan improvement at all step sizes; the maximum decrease relative to the baseline is 14 .
Revised: Table 3 shows the average MSE for the proposed and the baseline meta-classiﬁer, and shows an improvement at all step sizes; the maximum decrease relative to the baseline is 14 .

Original: 9% at step-size0 .
Revised: 9% at step-size 0 . 15 .

Original: To explore the inﬂuence of the sampling pattern of p k on shadow training, we investigate two differentsampling patterns: a) p k sampled in the middle of the probability simplex and b) p k sampled along the3 edges of the probability simplex.
Revised: To explore the inﬂuence of the sampling pattern of p k on shadow training, we investigate two different sampling patterns: a) p k sampled in the middle of the probability simplex and b) p k sampled along the 3 edges of the probability simplex.

Original: In Figure 8 and Figure 9 we show a heatmap of the KL divergenceof the estimated distribution with respect to the actual distribution, D p
Revised: In Figure 8 and Figure 9 we show a heatmap of the KL divergence of the estimated distribution with respect to the actual distribution, D p

Original: = D KL ( p || ˆ p ) with respect tothe true distribution p .
Revised: = D KL ( p || ˆ p ) with respect to the true distribution p .

Original: For easier visualization, only the ﬁrst 2 dimensions of p are plotted since thethird dimension is dependent on the ﬁrst two, [ p ] 3 = 1 − [ p ] 1 − [ p ] 2 .
Revised: For easier visualization, only the ﬁrst 2 dimensions of p are plotted since the third dimension is dependent on the ﬁrst two, [ p ] 3 = 1 − [ p ] 1 − [ p ] 2 .

Original: The color is proportional to374 the KL divergence D p .
Revised: The color is proportional to the KL divergence D p .

Original: Figure 8 shows that when the meta-classiﬁer is trained on samples from themiddle of the probability simplex, the KL divergence is lower in the middle than along the edges, i.e.when values of [ p ] 1 and [ p ] 2 are either large or small.
Revised: Figure 8 shows that when the meta-classiﬁer is trained on samples from the middle of the probability simplex, the KL divergence is lower in the middle than along the edges, i.e. when values of [ p ] 1 and [ p ] 2 are either large or small.

Original: However, in Figure 9, even though edges ofthe probability simplex were sampled to train the meta-classiﬁer, the KL divergence is still high atthe edges.
Revised: However, in Figure 9, even though edges of the probability simplex were sampled to train the meta-classiﬁer, the KL divergence is still high at the edges.

Original: The overall range of KL divergences seems to be smaller in this case compared to the onesampled in the middle.
Revised: The overall range of KL divergences seems to be smaller in this case compared to the one sampled in the middle.

Original: Overall, it seems that sub-sampling from the probability simplex for the sake
Revised: Overall, it seems that sub-sampling from the probability simplex for the sake

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [ p ][ p ] 0 .0 .0 .0 .
Revised: [ p ][ p ] 0 .0 .0 .0 .

Original: D p
Revised: D p

Original: ∆ p = 0 .
Revised: ∆ p = 0 .

Original: [Figure]
Revised: [Figure]

Original: [ p [Figure] [Figure] 8 D p of training does not adversely affect the ability of the attacker to infer label distributions.
Revised: 8 D p of training does not adversely affect the ability of the attacker to infer label distributions.

Original: Therefore,an attacker could potentially choose a region from the probability simplex based on prior knowledgeand only train the meta-classiﬁer using shadow classiﬁers with label distribution sampled from thisregion.
Revised: Therefore, an attacker could potentially choose a region from the probability simplex based on prior knowledge and only train the meta-classiﬁer using shadow classiﬁers with label distribution sampled from this region.

Original: This is one of the ways to reduce the sampling complexity.
Revised: This is one of the ways to reduce the sampling complexity

Original: For the 4-class classiﬁer between digits 0 , 1 , 2 , 3 , we trained the meta-classiﬁer on samples p kuniformly sampled from the ∆ 3 simplex with step sizes ∆ p = 0 .
Revised: For the 4-class classiﬁer between digits 0 , 1 , 2 , 3 , we trained the meta-classiﬁer on samples p k uniformly sampled from the ∆ 3 simplex with step sizes ∆ p = 0 .

Original: Figure 10 shows the boxand-whisker plots of the KL divergences obtained for the baseline and the proposed meta-classiﬁers.
Revised: Figure 10 shows the boxand-whisker plots of the KL divergences obtained for the baseline and the proposed meta-classiﬁers.

Original: The overall range of values is smaller when compared to the baseline.
Revised: The overall range of values is smaller when compared to the baseline.

Original: For better understanding, wealso report the average MSE for the uniform sampling case in Table 4 for the proposed meta-classiﬁerand the baseline.
Revised: For better understanding, we also report the average MSE for the uniform sampling case in Table 4 for the proposed meta-classiﬁer and the baseline.

Original: From these values, it is evident that accuracy augmentation and architecturalchanges are beneﬁcial even in higher dimensional settings, and the maximum decrease in MSErelative to the baseline is 5 .
Revised: From these values, it is evident that accuracy augmentation and architectural changes are beneﬁcial even in higher dimensional settings, and the maximum decrease in MSE relative to the baseline is 5 .

Original: 4% at step-size 0 . 4 .
Revised: 4% at step-size 0 . 4 .

Original: However, we see that the overall performance hasslightly deteriorated in comparison to binary and 3-class classiﬁers.
Revised: However, we see that the overall performance has slightly deteriorated in comparison to binary and 3-class classiﬁers.

Original: This is to be expected given thedimensionality of the problem and the computational difﬁculty of producing shadow training datasetsthat are representative of all possible label proportions, due to the large number of samples requiredfor the same sampling density as the 3 -class classiﬁer.
Revised: This is to be expected given the dimensionality of the problem and the computational difﬁculty of producing shadow training datasets that are representative of all possible label proportions, due to the large number of samples required for the same sampling density as the 3 -class classiﬁer.

Original: A potential approach to address the issue ofsampling complexity would be to use prior knowledge and only sample from certain regions from theprobability simplex in higher dimensions which is the subject of our future work.
Revised: A potential approach to address the issue of sampling complexity would be to use prior knowledge and only sample from certain regions from the probability simplex in higher dimensions which is the subject of our future work.

Original: A.2 Random oversampling as a countermeasure
Revised: A.2 Random oversampling as a countermeasure

Original: The imbalance of class labels is often addressed prior to training a classiﬁer by random oversamplingof the minority class Chawla et al.
Revised: The imbalance of class labels is often addressed prior to training a classiﬁer by random oversampling of the minority class Chawla et al.

Original: [2002], Ling and Li [1998].
Revised: [2002], Ling and Li [1998].

Original: Through random oversampling, thelabel distribution used for training becomes uniform, and thus the true label distribution p becomeshidden.
Revised: Through random oversampling, the label distribution used for training becomes uniform, and thus the true label distribution p becomes hidden.

Original: Hence, random oversampling could be a simple countermeasure to the considered class labeldistribution inference attacks.
Revised: Hence, random oversampling could be a simple countermeasure to the considered class label distribution inference attacks.

Original: However, since random oversampling leads to the repetition of datasamples from minority classes, the parameters of the target classiﬁer could still be similar to theparameters of a target classiﬁer trained on the original imbalanced dataset.
Revised: However, since random oversampling leads to the repetition of data samples from minority classes, the parameters of the target classiﬁer could still be similar to the parameters of a target classiﬁer trained on the original imbalanced dataset.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: We now investigate if class label distribution attacks are effective despite random oversampling, i.e.,if the true label distribution p can be estimated by a meta-classiﬁer based on the parameters of atarget classiﬁer that was instead trained on over-sampled data with induced uniform label distributionp o .
Revised: We now investigate if class label distribution attacks are effective despite random oversampling, i e., if the true label distribution p can be estimated by a meta-classiﬁer based on the parameters of a target classiﬁer that was instead trained on over-sampled data with induced uniform label distribution p o .

Original: We use the UCI Census income dataset and binary classiﬁcation for the evaluation.
Revised: We use the UCI Census income dataset and binary classiﬁcation for the evaluation.

Original: The datasets with label distributions p ∈ (0 , 0 . 2] ∪ [0 . 8 , 1) were considered imbalanced and used those fortraining target classiﬁers by oversampling the minority class, i.e., p o = 0 .
Revised: The data sets with label distributions p ∈ (0 , 0 . 2] ∪ [0 . 8 , 1) were considered imbalanced and used those for training target classiﬁers by oversampling the minority class, i e., p o = 0 .

Original: We then used theaccuracy-augmented meta-classiﬁer that was trained in Section 4.1 with ∆ p = 0 .
Revised: We then used the accuracyaugmented meta-classiﬁer that was trained in Section 4.1 with ∆ p = 0 .

Original: 01 in order toinfer the class label distribution p , i.e., the attacker is not aware of random oversampling.
Revised: in order to infer the class label distribution p , i.e., the attacker is not aware of random oversampling.

Original: Figure11 shows the estimated label distribution plotted with respect to the original label distribution p .
Revised: Figure 11 shows the estimated label distribution plotted with respect to the original label distribution p .

Original: It shows that the proposed meta-classiﬁer can successfully infer the true label distribution despiteoversampling, hence oversampling may not be an effective countermeasure against the proposedmeta-classiﬁer attack.
Revised: It shows that the proposed meta-classiﬁer can successfully infer the true label distribution despite oversampling, hence oversampling may not be an effective countermeasure against the proposed meta-classiﬁer attack.

Original: In addition, if the attacker is aware of random oversampling, it could re-trainthe meta-classiﬁer by generating shadow-training data sets with random oversampling and labels asthe original imbalanced distribution and further improve the inference accuracy.
Revised: In addition, if the attacker is aware of random oversampling, it could re-train the meta-classiﬁer by generating shadow-training data sets with random oversampling and labels as the original imbalanced distribution and further improve the inference accuracy.

Original: This is veriﬁed next.
Revised: This is veriﬁed next.

Original: We consider training the meta-classiﬁer to be aware of random oversampling, i.e., in the shadowtraining data set for the meta-classiﬁer, the parameters of the shadow classiﬁers that are trainedon random oversampling adjusted shadow training data sets are labeled with the original labeldistribution.
Revised: We consider training the meta-classiﬁer to be aware of random oversampling, i.e., in the shadow training data set for the meta-classiﬁer, the parameters of the shadow classiﬁers that are trained on random oversampling adjusted shadow training data sets are labeled with the original label distribution.

Original: As before, this experiment is performed on the UCI Census income data.
Revised: As before, this experiment is performed on the UCI Census income data.

Original: Figure.
Revised: Figure.

Original: 12shows the KL divergence for a meta-classiﬁer that is not aware of random oversampling and for onethat is aware of it.
Revised: 12 shows the KL divergence for a meta-classiﬁer that is not aware of random oversampling and for one that is aware of it.

Original: The ﬁgure shows that the meta classiﬁer that was trained on the oversampledshadow training data sets achieves better accuracy, i.e. lower KL divergence.
Revised: The ﬁgure shows that the meta classiﬁer that was trained on the oversampled shadow training data sets achieves better accuracy, i.e. lower KL divergence.

Revised: A.3 Limitations of the Attack

Revised: The limitations of the proposed meta-classiﬁer attack can be grouped into two categories: the architecture of the target classiﬁer and the shadow-training methodology.

Revised: The proposed methodology is applicable to deep architectures with many layers.

Revised: However, if the target classiﬁer is a convolutional neural network (CNN), permutation invariance does not hold so one would need appropriate metaclassiﬁer architectures other than DeepSets to also encode the spatial dependencies of the ﬁlter weights and biases.

Revised: In Parisot et al.

Revised: [2021], a multi-layer perceptron is proposed as meta-classiﬁer when the target classiﬁer is a CNN, but it unfortunately does not perform the encoding and could lead to loss of attack efﬁcacy.

Revised: A direction of future work would be to develop speciﬁc meta-classiﬁers for CNNs that perform well.

Revised: The shadow training methodology itself has limitations, as it does not scale well with the number of classes (as discussed in Section 3.3) and with the number of model parameters in the target classiﬁer.

Revised: Increasing the number of classes requires more shadow training data sets, hence longer training time, while increasing the number of model parameters increases the number of parameters of the meta classiﬁer, making it more challenging to be trained.