Original: A DVERSARIAL REPRESENTATION LEARNING FOR SYNTHETIC REPLACEMENT OF PRIVATE ATTRIBUTES
Revised: A DVERSARIAL REPRESENTATION LEARNING FOR SYNTHETIC REPLACEMENT OF PRIVATE ATTRIBUTES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Data privacy is an increasingly important aspect of many real-world big data analytics tasks.
Revised: Data privacy is an increasingly important aspect of many real-world big data analytics tasks.

Original: Data sources that contain sensitive information may have immense potential which could be unlocked using privacy enhancing transformations, but current methods often fail to produce convincing output.
Revised: Data sources that contain sensitive information may have immense potential which could be unlocked using privacy enhancing transformations, but current methods often fail to produce convincing output.

Original: Furthermore, ﬁnding the right balance between privacy and utility is often a tricky trade-off.
Revised: Furthermore, ﬁnding the right balance between privacy and utility is often a tricky trade-off.

Original: In this work, we propose a novel approach for data privatization, which involves two steps: in the ﬁrst step, it removes the sensitive information, and in the second step, it replaces this information with an independent random sample.
Revised: In this work, we propose a novel approach for data privatization, which involves two steps: in the ﬁrst step, it removes the sensitive information, and in the second step, it replaces this information with an independent random sample.

Original: Our method builds on adversarial representation learning which ensures strong privacy by training the model to fool an increasingly strong adversary.
Revised: Our method builds on adversarial representation learning which ensures strong privacy by training the model to fool an increasingly strong adversary.

Original: While previous methods only aim at obfuscating the sensitive information, we ﬁnd that adding new random information in its place strengthens the provided privacy and provides better utility at any given level of privacy.
Revised: While previous methods only aim at obfuscating the sensitive information, we ﬁnd that adding new random information in its place strengthens the provided privacy and provides better utility at any given level of privacy.

Original: The result is an approach that can provide stronger privatization on image data, and yet be preserving both the domain and the utility of the inputs, entirely independent of the downstream task.
Revised: The result is an approach that can provide stronger privatization on image data, and yet be preserving both the domain and the utility of the inputs, entirely independent of the downstream task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Increasing capacity and performance of modern machine learning models lead to increasing amounts of data required for training them (Goodfellow et al., 2016).
Revised: Increasing capacity and performance of modern machine learning models lead to increasing amounts of data required for training them (Goodfellow et al., 2016).

Original: However, collecting and using large datasets which may contain sensitive information about individuals is often impeded by increasingly strong privacy laws protecting individual rights, and the infeasibility of obtaining individual consent.
Revised: However, collecting and using large datasets which may contain sensitive information about individuals is often impeded by increasingly strong privacy laws protecting individual rights, and the infeasibility of obtaining individual consent.

Original: Giving privacy guarantees on a dataset may let us share data, while protecting the rights of individuals, and thus unlocking the large beneﬁts for individuals and for society that big datasets can provide.
Revised: Giving privacy guarantees on a dataset may let us share data, while protecting the rights of individuals, and thus unlocking the large beneﬁts for individuals and for society that big datasets can provide.

Original: In this work, we propose a technique for selective obfuscation of image datasets.
Revised: In this work, we propose a technique for selective obfuscation of image datasets.

Original: The aim is to provide the original data as detailed as possible while making it hard for an adversary to detect speciﬁc sensitive attributes.
Revised: The aim is to provide the original data as detailed as possible while making it hard for an adversary to detect speciﬁc sensitive attributes.

Original: The proposed solution is agnostic to the downstream task, with the objective to make the data as private as possible given a distortion constraint.
Revised: The proposed solution is agnostic to the downstream task, with the objective to make the data as private as possible given a distortion constraint.

Original: This issue has previously been addressed using adversarial representation learning with some success: a ﬁlter model is trained to obfuscate sensitive information while an adversary model is trained to recover the information (Edwards & Storkey, 2016).
Revised: This issue has previously been addressed using adversarial representation learning with some success: a ﬁlter model is trained to obfuscate sensitive information while an adversary model is trained to recover the information (Edwards & Storkey, 2016).

Original: In the current work, we demonstrate that it is easier to hide sensitive information if you replace it with something else : a sample which is independent from the input data.
Revised: In the current work, we demonstrate that it is easier to hide sensitive information if you replace it with something else : a sample which is independent from the input data.

Original: Aside from the adversary module, our proposed solution includes two main components: one ﬁlter model that is trained to remove the sensitive attribute, and one generator model that inserts a synthetically generated new value for the sensitive attribute.
Revised: Aside from the adversary module, our proposed solution includes two main components: one ﬁlter model that is trained to remove the sensitive attribute, and one generator model that inserts a synthetically generated new value for the sensitive attribute.

Original: The generated sensitive attribute is entirely independent from the sensitive attribute in the original input image.
Revised: The generated sensitive attribute is entirely independent from the sensitive attribute in the original input image.

Original: Following a body of work in privacy-related adversarial learning we evaluate the proposed model on faces from the
Revised: Following a body of work in privacy-related adversarial learning we evaluate the proposed model on faces from the

Original: CelebA dataset (Liu et al., 2015), and consider, for example, the smile or gender of a person to be the sensitive attribute.
Revised: CelebA dataset (Liu et al., 2015), and consider, for example, the smile or gender of a person to be the sensitive attribute.

Original: The smile is an attribute that carries interesting aspects in the transformations of a human face.
Revised: The smile is an attribute that carries interesting aspects in the transformations of a human face.

Original: The obvious change reside close to the mouth when a person smiles, but also other subtle changes occur: eyelids tighten, dimples show and the skin wrinkles.
Revised: The obvious change reside close to the mouth when a person smiles, but also other subtle changes occur: eyelids tighten, dimples show and the skin wrinkles.

Original: The current work includes a thorough analysis of the dataset, including correlations of such features.
Revised: The current work includes a thorough analysis of the dataset, including correlations of such features.

Original: These correlations make the task interesting and challenging, reﬂecting the real difﬁculty that may occur when anonymizing data.
Revised: These correlations make the task interesting and challenging, reﬂecting the real difﬁculty that may occur when anonymizing data.

Original: What is the right trade-off between preserving the utility as deﬁned by allowing information about other attributes to remain, and removing the sensitive information?
Revised: What is the right trade-off between preserving the utility as deﬁned by allowing information about other attributes to remain, and removing the sensitive information?

Original: In our setup, the adversary can make an arbitrary number of queries to the model.
Revised: In our setup, the adversary can make an arbitrary number of queries to the model.

Original: For each query another sample will be produced from the distribution of the sensitive data, while keeping as much as possible of the non-sensitive information about the requested data point.
Revised: For each query another sample will be produced from the distribution of the sensitive data, while keeping as much as possible of the non-sensitive information about the requested data point.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Privacy-preserving machine learning has been studied from a number of different angles.
Revised: Privacy-preserving machine learning has been studied from a number of different angles.

Original: Some work assumes access to a privacy-preserving mechanism, such as bounding boxes for faces, and studies how to hide people’s identity by blurring (Oh et al., 2016a), removing (Orekondy et al., 2018) or generating the face of other people (Hukkelås et al., 2019) in their place.
Revised: Some work assumes access to a privacy-preserving mechanism, such as bounding boxes for faces, and studies how to hide people’s identity by blurring (Oh et al., 2016a), removing (Orekondy et al., 2018) or generating the face of other people (Hukkelås et al., 2019) in their place.

Original: Other work assumes access to the utility-preserving mechanism and proposes to obfuscate everything except what they want to retain (Alharbi et al., 2019).
Revised: Other work assumes access to the utility-preserving mechanism and proposes to obfuscate everything except what they want to retain (Alharbi et al., 2019).

Original: This raises the question: how do we ﬁnd the pixels in an image that need to be modiﬁed to preserve privacy with respect to some attribute?
Revised: This raises the question: how do we ﬁnd the pixels in an image that need to be modiﬁed to preserve privacy with respect to some attribute?

Original: Furthermore, Oh et al.
Revised: Furthermore, Oh et al.

Original: (2016b) showed that blurring or removing the head of a person has a limited effect on privacy.
Revised: (2016b) showed that blurring or removing the head of a person has a limited effect on privacy.

Original: The ﬁnding is crucial; we cannot rely on modiﬁcations of an image such as blurring or overpainting to achieve privacy .
Revised: The ﬁnding is crucial; we cannot rely on modiﬁcations of an image such as blurring or overpainting to achieve privacy .

Original: An adversarial set-up instead captures the signals that the adversary uses, and can attain a stronger privacy.
Revised: An adversarial set-up instead captures the signals that the adversary uses, and can attain a stronger privacy.

Original: Adversarial learning is the process of training a model to fool an adversary (Goodfellow et al., 2014).
Revised: Adversarial learning is the process of training a model to fool an adversary (Goodfellow et al., 2014).

Original: Both models are trained simultaneously, and become increasingly good at their respective task during training.
Revised: Both models are trained simultaneously, and become increasingly good at their respective task during training.

Original: This approach has been successfully used to learn image-to-image transformations (Isola et al., 2017; Choi et al., 2018), and synthesis of properties such as facial expressions (Song et al., 2017; Tang et al., 2019).
Revised: This approach has been successfully used to learn image-to-image transformations (Isola et al., 2017; Choi et al., 2018), and synthesis of properties such as facial expressions (Song et al., 2017; Tang et al., 2019).

Original: Privacy-preserving adversarial representation learning utilize this paradigm to learn representations of data that hide sensitive information (Edwards & Storkey, 2016;
Revised: Privacy-preserving adversarial representation learning utilize this paradigm to learn representations of data that hide sensitive information (Edwards & Storkey, 2016;

Original: Zhang et al., 2018;
Revised: Zhang et al., 2018;

Original: Xie et al., 2017; Beutel et al., 2017; Raval et al., 2017).
Revised: Xie et al., 2017; Beutel et al., 2017; Raval et al., 2017).

Original: Bertran et al.
Revised: Bertran et al.

Original: (2019) minimize the mutual information between the utility variable and the input image data conditioned on the learned representation.
Revised: (2019) minimize the mutual information between the utility variable and the input image data conditioned on the learned representation.

Original: Roy & Boddeti (2019) maximize the entropy of the discriminator output rather than minimizing the log likelihood, which is beneﬁcial for stronger privacy.
Revised: Roy & Boddeti (2019) maximize the entropy of the discriminator output rather than minimizing the log likelihood, which is beneﬁcial for stronger privacy.

Original: Osia et al.
Revised: Osia et al.

Original: (2020) approached the problem using an information-bottleneck.
Revised: (2020) approached the problem using an information-bottleneck.

Original: Wu et al.
Revised: Wu et al.

Original: Ren et al.
Revised: Ren et al.

Original: (2018), and Wang et al.
Revised: (2018), and Wang et al.

Original: (2019) learn transformations of video that respect a privacy budget while maintaining performance on a downstream task.
Revised: (2019) learn transformations of video that respect a privacy budget while maintaining performance on a downstream task.

Original: Tran et al.
Revised: Tran et al.

Original: (2018) proposed an approach for pose-invariant face recognition.
Revised: (2018) proposed an approach for pose-invariant face recognition.

Original: Similar to our work, their approach used adversarial learning to disentangle speciﬁc attributes in the data.
Revised: Similar to our work, their approach used adversarial learning to disentangle speciﬁc attributes in the data.

Original: Oh et al.
Revised: Oh et al.

Original: (2017) trained a model to add a small amount of noise to the input to hide the identity of a person.
Revised: (2017) trained a model to add a small amount of noise to the input to hide the identity of a person.

Original: Xiao et al.
Revised: Xiao et al.

Original: (2020) learn a representation from which it is hard to reconstruct the original input,but from which it is possible to predict a predeﬁned task.
Revised: (2020) learn a representation from which it is hard to reconstruct the original input,but from which it is possible to predict a predeﬁned task.

Original: The method provides control over which attributes that is preserved, but no control over which attributes that are being censored.
Revised: The method provides control over which attributes that is preserved, but no control over which attributes that are being censored.

Original: That is, it puts more emphasis on preserving utility than privacy, which is not always desired.
Revised: That is, it puts more emphasis on preserving utility than privacy, which is not always desired.

Original: All of these, with the exception of Edwards & Storkey (2016) (see below), depend on knowing the downstream task labels.
Revised: All of these, with the exception of Edwards & Storkey (2016) (see below), depend on knowing the downstream task labels.

Original: Our work has no such dependency: the data produced by our method is designed to be usable regardless of downstream task.
Revised: Our work has no such dependency: the data produced by our method is designed to be usable regardless of downstream task.

Original: In Edwards & Storkey (2016), a limited experiment is included which does not depend on the downstream task.
Revised: In Edwards & Storkey (2016), a limited experiment is included which does not depend on the downstream task.

Original: In this experiment, they remove sensitive text which was overlaid on images, a task which is much simpler than the real-world problem considered in the current work.
Revised: In this experiment, they remove sensitive text which was overlaid on images, a task which is much simpler than the real-world problem considered in the current work.

Original: The overlaid text is independent of the underlying image, and therefore the solution does not require a trade-off between utility and privacy which is the case in most real settings.
Revised: The overlaid text is independent of the underlying image, and therefore the solution does not require a trade-off between utility and privacy which is the case in most real settings.

Original: Furthermore, we also replace the sensitive information with synthetic information which we show further strengthens the privacy.
Revised: Furthermore, we also replace the sensitive information with synthetic information which we show further strengthens the privacy.

Original: Like in the current work, Huang et al.
Revised: Like in the current work, Huang et al.

Original: (2017, 2018) use adversarial learning to minimize the mutual information between the private attribute and the censored image under a distortion constraint.
Revised: (2017, 2018) use adversarial learning to minimize the mutual information between the private attribute and the censored image under a distortion constraint.

Original: Our solution extends and improves upon these ideas with a modular design consisting of a ﬁlter that is trained to obfuscate the data, and a generator that further enhances the privacy by adding new independently sampled synthetic information for the sensitive attributes.
Revised: Our solution extends and improves upon these ideas with a modular design consisting of a ﬁlter that is trained to obfuscate the data, and a generator that further enhances the privacy by adding new independently sampled synthetic information for the sensitive attributes.

Original:  P RIVACY - PRESERVING ADVERSARIAL REPRESENTATION LEARNING
Revised:  P RIVACY - PRESERVING ADVERSARIAL REPRESENTATION LEARNING

Original: In the current work, we propose a novel solution for utility-preserving privacy-enhancing transformations of data: we use privacy-preserving representation learning to obfuscate information in the input data, and output results that retain the information and structure of the input.
Revised: In the current work, we propose a novel solution for utility-preserving privacy-enhancing transformations of data: we use privacy-preserving representation learning to obfuscate information in the input data, and output results that retain the information and structure of the input.

Original: 3. P ROBLEM SETTING
Revised: 3. P ROBLEM SETTING

Original: Generative adversarial privacy (GAP) (Huang et al., 2018) was proposed as a method to provide privacy in images under a distortion constraint, and will be used as the baseline in the current work.
Revised: Generative adversarial privacy (GAP) (Huang et al., 2018) was proposed as a method to provide privacy in images under a distortion constraint, and will be used as the baseline in the current work.

Original: In
Revised: In

Original: GAP, one assumes a joint distribution P ( X, S ) of public data points X and sensitive private attributes
Revised: GAP, one assumes a joint distribution P ( X, S ) of public data points X and sensitive private attributes

Original: S where S is typically correlated with X .
Revised: S where S is typically correlated with X .

Original: The authors deﬁne a privacy mechanism X (cid:48)
Revised: The authors deﬁne a privacy mechanism X (cid:48)

Original: =
Revised: =

Original: f ( X, Z 1 ) where Z 1 is the source of noise or randomness in f .
Revised: f ( X, Z 1 ) where Z 1 is the source of noise or randomness in f .

Original: Let h f ( X (cid:48) ) be an adversary’s prediction of the sensitive attribute S from the privatized data X (cid:48)
Revised: Let h f ( X (cid:48) ) be an adversary’s prediction of the sensitive attribute S from the privatized data X (cid:48)

Original: according to a decision rule h f .
Revised: according to a decision rule h f .

Original: The performance of the adversary is thus measured by a loss function (cid:96) f
Revised: The performance of the adversary is thus measured by a loss function (cid:96) f

Original: ( h f ( f ( x, z 1 )) , s ) and the expected loss of the adversary with respect to X , S and Z 1 is
Revised: ( h f ( f ( x, z 1 )) , s ) and the expected loss of the adversary with respect to X , S and Z 1 is

Original: [Equation]
Revised: [Equation]

Original: where p ( z 1 ) is the source of noise.
Revised: where p ( z 1 ) is the source of noise.

Original: The privacy mechanism f will be trained to be privacy-preserving and utility-preserving.
Revised: The privacy mechanism f will be trained to be privacy-preserving and utility-preserving.

Original: That is, it should be hard for an analyst to infer S from X (cid:48) , but X (cid:48) should be minimally distorted with respect to X .
Revised: That is, it should be hard for an analyst to infer S from X (cid:48) , but X (cid:48) should be minimally distorted with respect to X .

Original: Huang et al.
Revised: Huang et al.

Original: (2018) formulate this as a constrained minimax problem
Revised: (2018) formulate this as a constrained minimax problem

Original: [Equation]
Revised: [Equation]

Original: where the constant (cid:15) 1 ≥ 0 deﬁnes the allowed distortion for the privatizer and d ( · , · ) is some distortion measure.
Revised: where the constant (cid:15) 1 ≥ 0 deﬁnes the allowed distortion for the privatizer and d ( · , · ) is some distortion measure.

Original: In the current work, f will be referred to as the ﬁlter since the purpose of it is to ﬁlter the sensitive information from x .
Revised: In the current work, f will be referred to as the ﬁlter since the purpose of it is to ﬁlter the sensitive information from x .

Original: A potential limitation with this formulation is that it only obfuscates the sensitive information in x which may make it obvious to the adversary that x (cid:48) is a censored version of x .
Revised: A potential limitation with this formulation is that it only obfuscates the sensitive information in x which may make it obvious to the adversary that x (cid:48) is a censored version of x .

Original: Instead, we propose to replace the sensitive information with a new independent value s (cid:48) .
Revised: Instead, we propose to replace the sensitive information with a new independent value s (cid:48) .

Original: 3. O UR CONTRIBUTION
Revised: 3. O UR CONTRIBUTION

Original: We extend the ﬁlter with a generator module g , deﬁned as X (cid:48)(cid:48)
Revised: We extend the ﬁlter with a generator module g , deﬁned as X (cid:48)(cid:48)

Original: =
Revised: =

Original: g ( f ( X, Z 1 ) , S (cid:48) , Z 2 ) where S (cid:48) denotes the random variable of the new synthetic value for the sensitive attribute.
Revised: g ( f ( X, Z 1 ) , S (cid:48) , Z 2 ) where S (cid:48) denotes the random variable of the new synthetic value for the sensitive attribute.

Original: Z 1 and Z 2 denote the sources of randomness in f and g respectively.
Revised: Z 1 and Z 2 denote the sources of randomness in f and g respectively.

Original: The discriminator h g is trained to predict s when the input is a real image, and to predict the “ fake ” output when the input comes from g as in the learning setup in Salimans et al.
Revised: The discriminator h g is trained to predict s when the input is a real image, and to predict the “ fake ” output when the input comes from g as in the learning setup in Salimans et al.

Original: The objective of the generator g ( x (cid:48) , s (cid:48) , z 2 ) is to generate a new synthetic (independent) sensitive attribute s (cid:48) in x (cid:48) , that will fool the discriminator h g .
Revised: The objective of the generator g ( x (cid:48) , s (cid:48) , z 2 ) is to generate a new synthetic (independent) sensitive attribute s (cid:48) in x (cid:48) , that will fool the discriminator h g .

Original: We deﬁne the loss of the discriminator h g as
Revised: We deﬁne the loss of the discriminator h g as

Original: [Equation]
Revised: [Equation]

Original: where p ( z 1 , z 2 ) is the source of noise, p ( s (cid:48) ) is the assumed distribution of the synthetic sensitive attributes s (cid:48) , fake is the fake class, and (cid:96) g is the loss function.
Revised: where p ( z 1 , z 2 ) is the source of noise, p ( s (cid:48) ) is the assumed distribution of the synthetic sensitive attributes s (cid:48) , fake is the fake class, and (cid:96) g is the loss function.

Original: We formulate this as a constrained minimax problem
Revised: We formulate this as a constrained minimax problem

Original: [Equation]
Revised: [Equation]

Original: where the constant (cid:15) 2 ≥ 0 deﬁnes the allowed distortion for the generator.
Revised: where the constant (cid:15) 2 ≥ 0 deﬁnes the allowed distortion for the generator.

Original: [Figure]
Revised: [Figure]

Original: In Figure 1 we show the difference between (a) minimizing log-likelihood of the adversary, (b) maximizing entropy of the adversary, and (c) maximizing the entropy of the adversary and also synthetically replace the sensitive attribute with a random sample.
Revised: In Figure 1 we show the difference between (a) minimizing log-likelihood of the adversary, (b) maximizing entropy of the adversary, and (c) maximizing the entropy of the adversary and also synthetically replace the sensitive attribute with a random sample.

Original: 3. I MPLEMENTATION
Revised: 3. I MPLEMENTATION

Original: Let D =
Revised: Let D =

Original: ( x i , s i ) } ni =1 be a dataset of samples ( x i , s i ) which are assumed to be identically and independently distributed according to some unknown joint distribution P ( X, S ) .
Revised: ( x i , s i ) } ni =1 be a dataset of samples ( x i , s i ) which are assumed to be identically and independently distributed according to some unknown joint distribution P ( X, S ) .

Original: We assume that the sensitive attribute is binary and takes values in { 0 , 1 } .
Revised: We assume that the sensitive attribute is binary and takes values in { 0 , 1 } .

Original: However, the proposed approach can easily be extended to categorical sensitive attributes.
Revised: However, the proposed approach can easily be extended to categorical sensitive attributes.

Original: g ( X (cid:48) , S (cid:48) , Z 2 ; θ g ) using convolutional neural networks of the UNet (Ronneberger et al., 2015) architecture parameterized by θ f and θ g , respectively.
Revised: g ( X (cid:48) , S (cid:48) , Z 2 ; θ g ) using convolutional neural networks of the UNet (Ronneberger et al., 2015) architecture parameterized by θ f and θ g , respectively.

Original: (See
Revised: (See

Original: Appendix A.1 for details).
Revised: Appendix A.1 for details).

Original: The discriminators h f ( X (cid:48) ; φ f ) and h g ( X (cid:48)(cid:48) ; φ g ) are modeled using ResNet-18 (He et al., 2016) and a modiﬁed version which we refer to as ResNet-10 1 , respectively.
Revised: The discriminators h f ( X (cid:48) ; φ f ) and h g ( X (cid:48)(cid:48) ; φ g ) are modeled using ResNet-18 (He et al., 2016) and a modiﬁed version which we refer to as ResNet-10 1 , respectively.

Original: The last fully connected layer has been replaced with a two and three class output layer for each model, respectively.
Revised: The last fully connected layer has been replaced with a two and three class output layer for each model, respectively.

Original: As suggested by Roy & Boddeti (2019), we choose the ﬁlter discriminator loss (cid:96) f to be the the negative entropy.
Revised: As suggested by Roy & Boddeti (2019), we choose the ﬁlter discriminator loss (cid:96) f to be the the negative entropy.

Original: Intuitively, this leads to f learning to make the adversary
Revised: Intuitively, this leads to f learning to make the adversary

Original: h
Revised: h

Original: f confused about the sensitive attribute rather than to make it certain of the wrong value.
Revised: f confused about the sensitive attribute rather than to make it certain of the wrong value.

Original: For completeness, we also include experiments where (cid:96) f is the categorical cross-entropy, as are (cid:96) h f and (cid:96) g .
Revised: For completeness, we also include experiments where (cid:96) f is the categorical cross-entropy, as are (cid:96) h f and (cid:96) g .

Original: The distortion measure d is deﬁned as the L 2 -norm, and p ( s (cid:48) ) is assumed to be the uniform distribution U{ 0 , 1 } .
Revised: The distortion measure d is deﬁned as the L 2 -norm, and p ( s (cid:48) ) is assumed to be the uniform distribution U{ 0 , 1 } .

Original: The hyperparameters consist of the learning rate lr , the quadratic penalty term coefﬁcient λ , the distortion constraint (cid:15) , and the ( β 1 , β 2 ) parameters to Adam (Kingma & Ba, 2014).
Revised: The hyperparameters consist of the learning rate lr , the quadratic penalty term coefﬁcient λ , the distortion constraint (cid:15) , and the ( β 1 , β 2 ) parameters to Adam (Kingma & Ba, 2014).

Original: Details of the training setup can be found in Appendix A.3, and the full code is published on GitHub 2 .
Revised: Details of the training setup can be found in Appendix A.3, and the full code is published on GitHub 2 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we describe our experiments, the datasets used, and the evaluation metrics.
Revised: In this section we describe our experiments, the datasets used, and the evaluation metrics.

Original: Synthetic data.
Revised: Synthetic data.

Original: We introduce and apply the method on a synthetic dataset to illustrate the difference between optimizing the ﬁlter for log-likelihoood and entropy, and why adding the generator is important.
Revised: We introduce and apply the method on a synthetic dataset to illustrate the difference between optimizing the ﬁlter for log-likelihoood and entropy, and why adding the generator is important.

Original: The synthetic data consists D train
Revised: The synthetic data consists D train

Original: = { ( x i , s i ) } ni =1 , x ∈ R 2 and s ∈ { 0 , 1 } , drawn from two different normal distributions
Revised: = { ( x i , s i ) } ni =1 , x ∈ R 2 and s ∈ { 0 , 1 } , drawn from two different normal distributions

Original: N ( µ 1 , σ 1 ) and N ( µ 2 , σ 2 ) , where µ 1 =
Revised: N ( µ 1 , σ 1 ) and N ( µ 2 , σ 2 ) , where µ 1 =

Original: [ − 1 , 1] , and σ 1 =7]] , and µ 2 = [1 , − 1] , and σ 2 =
Revised: [ − 1 , 1] , and σ 1 = and µ 2 = [1 , − 1] , and σ 2 =

Original: Similarily, we construct a test sethttps:/
Revised: Similarily, we construct a test set

Original: D test = { ( x i , s i ) } mi =1 , and we let n = 400 , 000 and m =2 , 560 .
Revised: D test = { ( x i , s i ) } mi =1 , and we let n = 400 , 000 and m =

Original: The points are classiﬁed according to which normal distribution they belong to, and we consider this the secret attribute, i.e, s = 

Revised: The points are classiﬁed according to which normal distribution they belong to, and we consider this the secret attribute, i.e, s =

Original: if x ∼ N ( µ 1 , σ 1 ) and, s = 1 if x ∼ N ( µ 2 , σ 2 ) .
Revised: if x ∼ N ( µ 1 , σ 1 ) and, s = 1 if x ∼ N ( µ 2 , σ 2 ) .

Original: We sample s (cid:48) ∼ U ( { 0 , 1 } ) .
Revised: We sample s (cid:48) ∼ U ( { 0 , 1 } ) .

Original: We run the experiment with distortion constraints (cid:15) ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 } and run each experiment ﬁve times.
Revised: We run the experiment with distortion constraints (cid:15) ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 } and run each experiment ﬁve times.

Original: CelebA.
Revised: CelebA.

Original: The CelebA dataset 3 (Liu et al., 2015) consists of 202,599 face images of size 218x pixels and 40 binary attribute annotations per image, such as age (old or young), gender, if the image is blurry, if the person is bald, etc.
Revised: The CelebA dataset 3 (Liu et al., 2015) consists of 202,599 face images of size 218x pixels and 40 binary attribute annotations per image, such as age (old or young), gender, if the image is blurry, if the person is bald, etc.

Original: The dataset has an ofﬁcial split into a training set of 162, images, a validation set of 19,867 images and a test set of 19,962 images.
Revised: The dataset has an ofﬁcial split into a training set of 162, images, a validation set of 19,867 images and a test set of 19,962 images.

Original: We resize all images to 64x64 for the quantitative experiments, and to 128x128 pixels for the qualitative experiments, and normalize all pixel values to the region [0 , 1] .
Revised: We resize all images to 64x64 for the quantitative experiments, and to 128x128 pixels for the qualitative experiments, and normalize all pixel values to the region [0 , 1] .

Original: We use a higher resolution for the qualitative results to make subtle visual changes more apparent.
Revised: We use a higher resolution for the qualitative results to make subtle visual changes more apparent.

Original: Filtering and replacement of sensitive data.
Revised: Filtering and replacement of sensitive data.

Original: Let D train =
Revised: Let D train =

Original: ( x i , s i } ni =1 be a set of training data where x i denotes facial image i and s i ∈ { 0 , 1 } denotes the sensitive attribute.
Revised: ( x i , s i } ni =1 be a set of training data where x i denotes facial image i and s i ∈ { 0 , 1 } denotes the sensitive attribute.

Original: Further let
Revised: Further let

Original: D test = { ( x i , s i ) } mi =1 be the held out test data.
Revised: D test = { ( x i , s i ) } mi =1 be the held out test data.

Original: For the purpose of evaluation, we assume access to a number of utility attributes u ( j ) ∈ { 0 , 1 } for each x and j ∈ U .
Revised: For the purpose of evaluation, we assume access to a number of utility attributes u ( j ) ∈ { 0 , 1 } for each x and j ∈ U .

Original: The following attributes provided with the dataset will be used in the CelebA experiments: gender , lipstick , age , high cheekbones , mouth slightly open , heavy makeup , smiling .
Revised: The following attributes provided with the dataset will be used in the CelebA experiments: gender , lipstick , age , high cheekbones , mouth slightly open , heavy makeup , smiling .

Original: In each experiment, one of these will be selected as the sensitive attribute.
Revised: In each experiment, one of these will be selected as the sensitive attribute.

Original: The rest will be considered the utility attributes, not used for training, but allow for a utility score in the evaluation.
Revised: The rest will be considered the utility attributes, not used for training, but allow for a utility score in the evaluation.

Original: This score shows how well non-sensitive attributes are preserved in the transformation.
Revised: This score shows how well non-sensitive attributes are preserved in the transformation.

Original: We compute an average over ﬁxed classiﬁers, trained to predict each respective utility attribute on the original training data, when evaluated on the censored data.
Revised: We compute an average over ﬁxed classiﬁers, trained to predict each respective utility attribute on the original training data, when evaluated on the censored data.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: We train the models using D train with lr = 0 . 0005 , λ = 10 5 , (cid:15) ∈ { 0 . 03 , 0 . 02 , 0 . 01 , 0 .
Revised: We train the models using D train with lr = 0 . 0005 , λ = 10 5 , (cid:15) ∈ { 0 . 03 , 0 . 02 , 0 . 01 , 0 .

Original: 001 } , and ( β 1 , β 2 ) =
Revised: 001 } , and ( β 1 , β 2 ) =

Original: Let the training data censored by the ﬁlter be D (cid:48) train = { ( x (cid:48)
Revised: Let the training data censored by the ﬁlter be D (cid:48) train = { ( x (cid:48)

Original: i , s i ) } ni
Revised: i , s i ) } ni

Original: =1 , where x (cid:48)
Revised: =1 , where x (cid:48)

Original: i =
Revised: i =

Original: f ( x i , z (1) i ; θ f ) , and by both the ﬁlter and the generator be D (cid:48)(cid:48) train = { ( x (cid:48)(cid:48) i , s i ) } where x (cid:48)(cid:48)
Revised: f ( x i , z (1) i ; θ f ) , and by both the ﬁlter and the generator be D (cid:48)(cid:48) train = { ( x (cid:48)(cid:48) i , s i ) } where x (cid:48)(cid:48)

Original: i =
Revised: i =

Original: g ( x (cid:48)
Revised: g ( x (cid:48)

Original: i , s (cid:48)
Revised: i , s (cid:48)

Original: i , z (2) i ; θ g ) , z (1) i , z (2) i ∼ N ( 0 , 1 ) , and s (cid:48)
Revised: i , z (2) i ; θ g ) , z (1) i , z (2) i ∼ N ( 0 , 1 ) , and s (cid:48)

Original: i ∼ U{ 0 , 1 } .
Revised: i ∼ U{ 0 , 1 } .

Original: We do the same transformations to the test data and denote them D (cid:48) test and D (cid:48)(cid:48) test respectively.
Revised: We do the same transformations to the test data and denote them D (cid:48) test and D (cid:48)(cid:48) test respectively.

Original: Computational requirements.
Revised: Computational requirements.

Original: Each experiment was performed on a Tesla V100 SXM2 32 GB in a DGX-1 machine.
Revised: Each experiment was performed on a Tesla V100 SXM2 32 GB in a DGX-1 machine.

Original: The training was restricted to 100 epochs which takes about 13 hours.
Revised: The training was restricted to 100 epochs which takes about 13 hours.

Original: We run each experiment with negative entropy loss three times, and present the average over these runs, and we run each experiment with log-likelihood loss ﬁve times and present the average over these.
Revised: We run each experiment with negative entropy loss three times, and present the average over these runs, and we run each experiment with log-likelihood loss ﬁve times and present the average over these.

Original: Evaluation.
Revised: Evaluation.

Original: To evaluate the privacy loss for our method we train an adversarial classiﬁer to predict the ground truth sensitive attribute given a training set of images censored by each privatization method, and then we evaluate each adversary on a test set of censored images.
Revised: To evaluate the privacy loss for our method we train an adversarial classiﬁer to predict the ground truth sensitive attribute given a training set of images censored by each privatization method, and then we evaluate each adversary on a test set of censored images.

Original: If an adversary can predict the ground truth sensitive attribute of censored images, that it has not seen before, with high accuracy, then the privacy loss is high.
Revised: If an adversary can predict the ground truth sensitive attribute of censored images, that it has not seen before, with high accuracy, then the privacy loss is high.

Original: Let adv ( s | h ( x )) denote an adversary trained on the censored training set to predict the ground truth attribute s given censored image h
Revised: Let adv ( s | h ( x )) denote an adversary trained on the censored training set to predict the ground truth attribute s given censored image h

Original: ( x ) then the privacy loss is deﬁned as
Revised: ( x ) then the privacy loss is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: To evaluate the utility score on the CelebA data, we use ﬁxed classiﬁers that have been pre-trained on the original data to predict a set of utility attributes that are considered non-sensitive.
Revised: To evaluate the utility score on the CelebA data, we use ﬁxed classiﬁers that have been pre-trained on the original data to predict a set of utility attributes that are considered non-sensitive.

Original: If these utility attributes are predictable with high accuracy after privatization of the image, then we consider the method to have a high utility score.
Revised: If these utility attributes are predictable with high accuracy after privatization of the image, then we consider the method to have a high utility score.

Original: Let fix ( u | x (cid:48)(cid:48) ) denote a ﬁxed classiﬁer trained on the original training set to predict the ground truth attribute u , then the utility score is deﬁned as 
Revised: Let fix ( u | x (cid:48)(cid:48) ) denote a ﬁxed classiﬁer trained on the original training set to predict the ground truth attribute u , then the utility score is deﬁned as [Equation] [Equation] ( u (

Original: [Equation]
Revised: [Equation]

Revised: | h ( x )

Original:  [Figure]
Revised: [Equation] = u ( j ) ] [Equation] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: To evaluate the utility score on the synthetic data, we compute
Revised: To evaluate the utility score on the synthetic data, we compute

Original: [Equation]
Revised: [Equation]

Original: where we normalize with (cid:15) max = 2 to get distances into the range [0 , 1] and convert it to a similarity by subtracting the average over these distances from one for consistency in the trade-off curves.
Revised: where we normalize with (cid:15) max = 2 to get distances into the range [0 , 1] and convert it to a similarity by subtracting the average over these distances from one for consistency in the trade-off curves.

Original: Where we let h ( x ) =
Revised: Where we let h ( x ) =

Original: f ( x, z 1 ; θ f ) to evaluate the baseline, and h ( x )
Revised: f ( x, z 1 ; θ f ) to evaluate the baseline, and h ( x )

Original: =
Revised: =

Original: g
Revised: g

Original: ( f ( x, z 1 ; θ f ) , s (cid:48) , z 2 ; θ g ) to evaluate our method.
Revised: ( f ( x, z 1 ; θ f ) , s (cid:48) , z 2 ; θ g ) to evaluate our method.

Original: We then plot the utility score against the privacy loss for different distortion budgets and get a trade-off curve.
Revised: We then plot the utility score against the privacy loss for different distortion budgets and get a trade-off curve.

Original: Similar evaluation method are used in Roy & Boddeti (2019); Bertran et al.
Revised: Similar evaluation method are used in Roy & Boddeti (2019); Bertran et al.

Original: In particular, Roy & Boddeti (2019) plot accuracy of an adversary predicting the sensitive attribute against the accuracy of a classiﬁer predicting the target attribute.
Revised: In particular, Roy & Boddeti (2019) plot accuracy of an adversary predicting the sensitive attribute against the accuracy of a classiﬁer predicting the target attribute.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section we present quantitative and qualitative results on the facial image experiments.
Revised: In this section we present quantitative and qualitative results on the facial image experiments.

Original: Synthetic data.
Revised: Synthetic data.

Original: Figure 2 shows the trade-off between privacy loss and utility score for the experiment on the synthetic dataset.
Revised: Figure 2 shows the trade-off between privacy loss and utility score for the experiment on the synthetic dataset.

Original: Our method consistently outperforms the baseline for all given distortion budgets.
Revised: Our method consistently outperforms the baseline for all given distortion budgets.

Original: Figure 3 shows the original data (orange for s=0, and blue for s=1) together with the censored data for the three different transforms (red for baseline with likelihood, green for baseline with entropy, and purple for the proposed method).
Revised: Figure 3 shows the original data (orange for s=0, and blue for s=1) together with the censored data for the three different transforms (red for baseline with likelihood, green for baseline with entropy, and purple for the proposed method).

Original: When minimizing log-likelihood we see clear clusters on opposite side of the decision boundary, when maximizing entropy we see that all points approach the decision boundary, and when adding the generator we see that the points
Revised: When minimizing log-likelihood we see clear clusters on opposite side of the decision boundary, when maximizing entropy we see that all points approach the decision boundary, and when adding the generator we see that the points

Original: [Figure]
Revised: [Figure]

Original: mapped
Revised: mapped

Original: CelebA. Figure 4 shows the trade-off between privacy loss and utility score when evaluated on censored images.
Revised: CelebA. Figure 4 shows the trade-off between privacy loss and utility score when evaluated on censored images.

Original: Our method consistently has a higher utility at any given level of privacy compared to the baseline.
Revised: Our method consistently has a higher utility at any given level of privacy compared to the baseline.

Original: Remember: these are strong adversaries required to run tagged training data through the privacy mechanism to be able to train.
Revised: Remember: these are strong adversaries required to run tagged training data through the privacy mechanism to be able to train.

Original: Additional results can be found in Appendix A.2.
Revised: Additional results can be found in Appendix A.2.

Original: To further show that explicitly optimizing for privacy in the privatization mechanism is necessary we have conducted a similar experiment using StarGAN Choi et al.
Revised: To further show that explicitly optimizing for privacy in the privatization mechanism is necessary we have conducted a similar experiment using StarGAN Choi et al.

Original: (2018) to randomly change the sensitive attribute in the image (results in suppl.).
Revised: (2018) to randomly change the sensitive attribute in the image (results in suppl.).

Original: We then evaluate the censored images, which look very convincing to a human, using adversarial classiﬁers.
Revised: We then evaluate the censored images, which look very convincing to a human, using adversarial classiﬁers.

Original: The adversaries can successfully detect the sensitive attributes with an accuracy of roughly 90%.
Revised: The adversaries can successfully detect the sensitive attributes with an accuracy of roughly 90%.

Original: For the weight λ rec of the cycle consistency loss we explored values { 0 , 5 , 10 , 50 } .
Revised: For the weight λ rec of the cycle consistency loss we explored values { 0 , 5 , 10 , 50 } .

Original: We obtain similar scores when we exclude the ﬁlter part of our method and use only the generator part to censor the images (see Appendix A.2).
Revised: We obtain similar scores when we exclude the ﬁlter part of our method and use only the generator part to censor the images (see Appendix A.2).

Original: In Table 1 we present the results of evaluating the accuracy of fix ( s |· ) on the dataset { x (cid:48)(cid:48)
Revised: In Table 1 we present the results of evaluating the accuracy of fix ( s |· ) on the dataset { x (cid:48)(cid:48)

Original: i , s (cid:48) i } mi = where x (cid:48)(cid:48) i
Revised: i , s (cid:48) i } mi = where x (cid:48)(cid:48) i

Original: = g ( x (cid:48)
Revised: = g ( x (cid:48)

Original: i , s (cid:48)
Revised: i , s (cid:48)

Original: i z (2) i ) is the image censored with our method and s (cid:48)
Revised: i z (2) i ) is the image censored with our method and s (cid:48)

Original: i is the new synthetic attribute uniformly sampled from { 0 , 1 } .
Revised: i is the new synthetic attribute uniformly sampled from { 0 , 1 } .

Original: That is, we measure how often the classiﬁer predict the new synthetic attribute s (cid:48)
Revised: That is, we measure how often the classiﬁer predict the new synthetic attribute s (cid:48)

Original: i when applied to x (cid:48)(cid:48) i .
Revised: i when applied to x (cid:48)(cid:48) i .

Original: We can see that with (cid:15) = 0 .
Revised: We can see that with (cid:15) = 0 .

Original: 001 the method is on average able to fool the classiﬁer 82 .
Revised: 001 the method is on average able to fool the classiﬁer 82 .

Original: 4% of the time for the smiling attribute, and this increases with larger distortion budget (cid:15) to a success rate of 91 .
Revised: 4% of the time for the smiling attribute, and this increases with larger distortion budget (cid:15) to a success rate of 91 .

Original: 2% on average with (cid:15) = 0 .
Revised: 2% on average with (cid:15) = 0 .

Original: The results are similar when the images have been censored with respect to the attributes gender , lipstick , and age , but these require a larger distortion budget.
Revised: The results are similar when the images have been censored with respect to the attributes gender , lipstick , and age , but these require a larger distortion budget.

Original: [Table]
Revised: [Table]

Original: = 1 ).
Revised: = 1 ).

Original: The model is able to generate a synthetic smiling attribute while maintaining much of the structure in the image.
Revised: The model is able to generate a synthetic smiling attribute while maintaining much of the structure in the image.

Original: These images were generated from a model trained using 128x128 pixels.
Revised: These images were generated from a model trained using 128x128 pixels.

Original: Figure 5 shows, from the top row to the bottom row, the input image x , the censored image x (cid:48) , the censored image x (cid:48)(cid:48) with the synthetic attribute s (cid:48)
Revised: Figure 5 shows, from the top row to the bottom row, the input image x , the censored image x (cid:48) , the censored image x (cid:48)(cid:48) with the synthetic attribute s (cid:48)

Original: = 

Revised: =

Original: (non-smiling) and the censored image x (cid:48)(cid:48) with the synthetic attribute s (cid:48)
Revised: (non-smiling) and the censored image x (cid:48)(cid:48) with the synthetic attribute s (cid:48)

Original: = 1 (smiling).
Revised: = 1 (smiling).

Original: A value of (cid:15) = 0 .
Revised: A value of (cid:15) = 0 .

Original: 001 is used in the ﬁrst four columns, and (cid:15) = 0 .
Revised: 001 is used in the ﬁrst four columns, and (cid:15) = 0 .

Original: 01 in the last four columns.
Revised: 01 in the last four columns.

Original: The images censored by our method look sharper and it is less obvious that they are censored.
Revised: The images censored by our method look sharper and it is less obvious that they are censored.

Original: We can see that the method convincingly generates non-smiling faces and smiling faces while most of the other parts of the image is intact.
Revised: We can see that the method convincingly generates non-smiling faces and smiling faces while most of the other parts of the image is intact.

Original: These images are sampled from models trained on images of 128x128 pixels resolution.
Revised: These images are sampled from models trained on images of 128x128 pixels resolution.

Original: See Figure 7 in Appendix A.2 for corresponding samples on the same input images, but using gender as the sensitive attribute.
Revised: See Figure 7 in Appendix A.2 for corresponding samples on the same input images, but using gender as the sensitive attribute.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In Figure 1, we show how a privatization mechanism is affected by an adversarial setup and its training criterion.
Revised: In Figure 1, we show how a privatization mechanism is affected by an adversarial setup and its training criterion.

Original: Minimizing the log likelihood of the adversary can be interpreted as attempting to make the adversary certain of the wrong value of the sensitive attribute and may lead to transforming data with a certain sensitive value to be transformed to similar outputs.
Revised: Minimizing the log likelihood of the adversary can be interpreted as attempting to make the adversary certain of the wrong value of the sensitive attribute and may lead to transforming data with a certain sensitive value to be transformed to similar outputs.

Original: Following the gradient of the entropy loss instead leads to the adversary becoming less certain and thus leads to more privacy.
Revised: Following the gradient of the entropy loss instead leads to the adversary becoming less certain and thus leads to more privacy.

Original: Using this approach to remove sensitive information, and then adding random information makes an even stronger privacy, as we take another step adding randomness to the output.
Revised: Using this approach to remove sensitive information, and then adding random information makes an even stronger privacy, as we take another step adding randomness to the output.

Original: This intuition is conﬁrmed in the experimental results on the synthetic data, as demonstrated by Figure 2.
Revised: This intuition is conﬁrmed in the experimental results on the synthetic data, as demonstrated by Figure 3.

Original: Our approach successfully transforms the datapoints into a distribution which is nearly indistinguishable from the input distribution, and where datapoints with the different sensitive values are completely mixed.
Revised: Our approach successfully transforms the datapoints into a distribution which is nearly indistinguishable from the input distribution, and where datapoints with the different sensitive values are completely mixed.

Original: Our method consistently outperforms the baseline, ensuring a higher level of privacy while maintaining more of the useful information (see Figure 4).
Revised: Our method consistently outperforms the baseline, ensuring a higher level of privacy while maintaining more of the useful information (see Figure 4).

Original: For all sensitive attributes that we consider and at nearly all given distortion budgets ( (cid:15) ∈ { 0 . 02 , 0 . 01 , 0 .
Revised: For all sensitive attributes that we consider and at nearly all given distortion budgets ( (cid:15) ∈ { 0 . 02 , 0 . 01 , 0 .

Original: we observe both a higher privacy and a higher utility for our method.
Revised: we observe both a higher privacy and a higher utility for our method.

Original: For some of the attributes ( gender , lipstick ), at extreme distortions ( (cid:15) = 0 . 03 ) our method has a utility score which is not better than the baseline, but still provide considerably better privacy.
Revised: For some of the attributes ( gender , lipstick ), at extreme distortions ( (cid:15) = 0 . 03 ) our method has a utility score which is not better than the baseline, but still provide considerably better privacy.

Original: Furthermore, we can observe that using the entropy loss function for the ﬁlter beneﬁts both the baseline and our method.
Revised: Furthermore, we can observe that using the entropy loss function for the ﬁlter beneﬁts both the baseline and our method.

Original: This shows that our method makes it more difﬁcult for the adversary to see through the privatization step at each given distortion budget.
Revised: This shows that our method makes it more difﬁcult for the adversary to see through the privatization step at each given distortion budget.

Original: To show the effect of the ﬁlter we have conducted experiments where we only use the generator to privatize the images (see Appendix A.2), which does not seem to provide any privacy.
Revised: To show the effect of the ﬁlter we have conducted experiments where we only use the generator to privatize the images (see Appendix A.2), which does not seem to provide any privacy.

Original: A generator without the ﬁlter may learn to either pass the original image without modiﬁcation (when s (cid:48)
Revised: A generator without the ﬁlter may learn to either pass the original image without modiﬁcation (when s (cid:48)

Original: = s ) or to transform the image into a random other value ( s (cid:48) (cid:54) = s ).
Revised: = s ) or to transform the image into a random other value ( s (cid:48) (cid:54) = s ).

Original: If the transformed image is indistinguishable from a real image this is not a problem, but otherwise we can easily reverse the privatization by detecting if the image is real or not.
Revised: If the transformed image is indistinguishable from a real image this is not a problem, but otherwise we can easily reverse the privatization by detecting if the image is real or not.

Original: The ﬁlter step mitigates this by always removing the sensitive data in the image, forcing the generator to synthesize new data.
Revised: The ﬁlter step mitigates this by always removing the sensitive data in the image, forcing the generator to synthesize new data.

Original: Since the censored image is now guaranteed to be synthetic, we are no longer susceptible the simple real/fake attack.
Revised: Since the censored image is now guaranteed to be synthetic, we are no longer susceptible the simple real/fake attack.

Original: Similarily, we conduct experiments where we use StarGAN to censor the images, but observe no privacy when using this standard attribute manipulation method.
Revised: Similarily, we conduct experiments where we use StarGAN to censor the images, but observe no privacy when using this standard attribute manipulation method.

Original: A reason could be that the cyclic consistency loss term actually encourage image transformations that are easily invertible given the original attribute, which is not desirable in a privacy setting.
Revised: A reason could be that the cyclic consistency loss term actually encourage image transformations that are easily invertible given the original attribute, which is not desirable in a privacy setting.

Original: This motivates our approach which is explicit about the privacy objective.
Revised: This motivates our approach which is explicit about the privacy objective.

Original: In Table 1, we see that the ﬁxed smile classiﬁer is fooled by our privatization mechanism in 82.4% to 91.2% of the data points in the test set (depending on the distortion (cid:15) ).
Revised: In Table 1, we see that the ﬁxed smile classiﬁer is fooled by our privatization mechanism in 82.4% to 91.2% of the data points in the test set (depending on the distortion (cid:15) ).

Original: These results indicate that it may be harder for an adversarially trained classiﬁer to predict the sensitive attribute when it has been replaced with something else, as compared to simply removed.
Revised: These results indicate that it may be harder for an adversarially trained classiﬁer to predict the sensitive attribute when it has been replaced with something else, as compared to simply removed.

Original: We assume that this is due to the added variability in the data.
Revised: We assume that this is due to the added variability in the data.

Original: Or intuitively: it is easier to “blend in” with other images that have similar demonstrations of smiles.
Revised: Or intuitively: it is easier to “blend in” with other images that have similar demonstrations of smiles.

Original: The fact that many important attributes in facial images correlate leads to the reﬂection that disentangling the underlying factors of variation is not entirely possible.
Revised: The fact that many important attributes in facial images correlate leads to the reﬂection that disentangling the underlying factors of variation is not entirely possible.

Original: For example, in CelebA, lipstick is highly correlated with female gender.
Revised: For example, in CelebA, lipstick is highly correlated with female gender.

Original: This means that if we want to hide all information about the presence of lipstick we also need to hide the gender (and other correlating attributes).
Revised: This means that if we want to hide all information about the presence of lipstick we also need to hide the gender (and other correlating attributes).

Original: This problem is further analysed in Table 3 in Appendix A.2.
Revised: This problem is further analysed in Table 3 in Appendix A.2.

Original: A strength of our method is that it is domain-preserving , this allows a utility provider to use the censored image in existing algorithms without modiﬁcations.
Revised: A strength of our method is that it is domain-preserving , this allows a utility provider to use the censored image in existing algorithms without modiﬁcations.

Original: Since the method also preserves utility this may also allow stacking the privatization mechanism to censor multiple attributes in an image.
Revised: Since the method also preserves utility this may also allow stacking the privatization mechanism to censor multiple attributes in an image.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work we have presented a strong privacy-preserving transformation mechanism for image data which is learned using an adversarial setup.
Revised: In this work we have presented a strong privacy-preserving transformation mechanism for image data which is learned using an adversarial setup.

Original: While previous work on adversarial representation learning has focused on removing information from a representation, our approach extends on this and can also generate new information in its place that looks realistic and gives further privacy properties compared to the baseline.
Revised: While previous work on adversarial representation learning has focused on removing information from a representation, our approach extends on this and can also generate new information in its place that looks realistic and gives further privacy properties compared to the baseline.

Original: We evaluate our method using adversarially trained classiﬁers, and our results show that not only do we provide stronger privacy with regards to sensitive attributes, but we also preserve non-sensitive attributes of the image at a higher rate.
Revised: We evaluate our method using adversarially trained classiﬁers, and our results show that not only do we provide stronger privacy with regards to sensitive attributes, but we also preserve non-sensitive attributes of the image at a higher rate.

Original: The results show that the synthetically added attribute gives stronger privacy properties and helps fooling the adversary in the challenging setting where the adversary is allowed to be trained using the output of the privacy mechanism.
Revised: The results show that the synthetically added attribute gives stronger privacy properties and helps fooling the adversary in the challenging setting where the adversary is allowed to be trained using the output of the privacy mechanism.

Original: A A PPENDIX A. A DDITIONAL DETAILS
Revised: A A PPENDIX A. A DDITIONAL DETAILS

Original: An overview of the setup can be seen in Figure 6.
Revised: An overview of the setup can be seen in Figure 6.

Original: We use the UNet Ronneberger et al.
Revised: We use the UNet Ronneberger et al.

Original: , illustrated on the right in Figure 6, architecture for both the ﬁlter and the generator.
Revised: , illustrated on the right in Figure 6, architecture for both the ﬁlter and the generator.

Original: The orange blocks are convolution blocks each of which, except for the last block, consist of a convolution layer, a batch normalization layer and a rectiﬁed linear activation unit, repeated twice in that order.
Revised: The orange blocks are convolution blocks each of which, except for the last block, consist of a convolution layer, a batch normalization layer and a rectiﬁed linear activation unit, repeated twice in that order.

Original: The number of output channels of the convolution layers in each block has been noted in the ﬁgure.
Revised: The number of output channels of the convolution layers in each block has been noted in the ﬁgure.

Original: The last convolution block with a 3 channel output (the RGB image) consists of only a single convolutional layer followed by a sigmoid activation.
Revised: The last convolution block with a 3 channel output (the RGB image) consists of only a single convolutional layer followed by a sigmoid activation.

Original: The green blocks denote either a max pooling layer with a kernel size of two and a stride of two if marked with “/2” or a nearest neighbor upsampling by a factor of two if marked with “2x”.
Revised: The green blocks denote either a max pooling layer with a kernel size of two and a stride of two if marked with “/2” or a nearest neighbor upsampling by a factor of two if marked with “2x”.

Original: The blue block denotes an embedding layer, which takes as input the categorical value of the sensitive attribute and outputs a dense embedding of 128 dimensions.
Revised: The blue block denotes an embedding layer, which takes as input the categorical value of the sensitive attribute and outputs a dense embedding of 128 dimensions.

Original: It is then followed by a linear projection and a reshaping to match the spatial dimensions of the output of the convolution block to which it is concatenated, but with a single channel.
Revised: It is then followed by a linear projection and a reshaping to match the spatial dimensions of the output of the convolution block to which it is concatenated, but with a single channel.

Original: The same type of linear projection is applied on the 1024 dimensional noise vector input, but this projection and reshaping matches both the spatial and channel dimensions of the output of the convolutional block to which it is concatenated.
Revised: The same type of linear projection is applied on the 1024 dimensional noise vector input, but this projection and reshaping matches both the spatial and channel dimensions of the output of the convolutional block to which it is concatenated.

Original: Concatenation is in both cases done along the channel dimension.
Revised: Concatenation is in both cases done along the channel dimension.

Original: A. A DDITIONAL RESULTS
Revised: A. A DDITIONAL RESULTS

Original: The results in Table 2 show that using the ﬁlter together with the generator ( g ◦ f ) give a stronger privatization guarantee than using only the generator (without the ﬁlter) or only the baseline (the ﬁlter without the generator).
Revised: The results in Table 2 show that using the ﬁlter together with the generator ( g ◦ f ) give a stronger privatization guarantee than using only the generator (without the ﬁlter) or only the baseline (the ﬁlter without the generator).

Original: the baseline, only the generator, and our method for varying distortion (cid:15) for the smiling attribute.
Revised: the baseline, only the generator, and our method for varying distortion (cid:15) for the smiling attribute.

Original: Closer to 50 means more privacy.
Revised: Closer to 50 means more privacy.

Original: Only using the generator does not privide any privacy, but using it together with the ﬁlter provides the best privacy.
Revised: Only using the generator does not privide any privacy, but using it together with the ﬁlter provides the best privacy.

Original: Lower is better.
Revised: Lower is better.

Original: [Table]
Revised: [Table]

Original: In Figure 7 we show additional qualitative results when the attribtue gender is considered sensitive.
Revised: In Figure 7 we show additional qualitative results when the attribtue gender is considered sensitive.

Original: Table 3 shows the correlations between classiﬁer predictions on a pair of attributes when one attribute has been synthetically replaced.
Revised: Table 3 shows the correlations between classiﬁer predictions on a pair of attributes when one attribute has been synthetically replaced.

Original: [Table]
Revised: [Table]

Original: For example, in the CelebA dataset lipstick is highly correlated with female gender.
Revised: For example, in the CelebA dataset lipstick is highly correlated with female gender.

Original: This means that if we want to hide all information about whether or not the person is wearing lipstick we also need to hide its gender (and other correlating attributes).
Revised: This means that if we want to hide all information about whether or not the person is wearing lipstick we also need to hide its gender (and other correlating attributes).

Original: This problem can be seen in Table 3 where changing whether or not a person is wearing lipstick correlates with changes of gender.
Revised: This problem can be seen in Table 3 where changing whether or not a person is wearing lipstick correlates with changes of gender.

Original: The question is: if we censor an attribute in an image, how does that correlate with changes of other attributes in the image?
Revised: The question is: if we censor an attribute in an image, how does that correlate with changes of other attributes in the image?

Original: In the lipstick column of Table 3 we have censored the attribute lipstick.
Revised: In the lipstick column of Table 3 we have censored the attribute lipstick.

Original: We then make predictions on whether or not the person in the censored image is wearing lipstick, and compute the correlation between these predictions and predictions for the attributes for each row.
Revised: We then make predictions on whether or not the person in the censored image is wearing lipstick, and compute the correlation between these predictions and predictions for the attributes for each row.

Original: We can see that changes in lipstick correlate negatively with changes in gender and positively with makeup.
Revised: We can see that changes in lipstick correlate negatively with changes in gender and positively with makeup.

Original: This highlights the problem of disentangling these underlying factors of variation.
Revised: This highlights the problem of disentangling these underlying factors of variation.

Original: Table 4 shows the Pearson correlations between the smiling attribute and 37 other attributes in the
Revised: Table 4 shows the Pearson correlations between the smiling attribute and 37 other attributes in the

Original: CelebA dataset.
Revised: CelebA dataset.

Original: [Table]
Revised: [Table]

Original: A. T RAINING S ETUP
Revised: A. T RAINING S ETUP

Original: Algorithm 1 demonstrates the training setup for our method.
Revised: Algorithm 1 demonstrates the training setup for our method.

Original: Note that when (cid:96) f is negative entropy it does not depend on the 1 −
Revised: Note that when (cid:96) f is negative entropy it does not depend on the 1 −

Original: s i argument, but is simply computed from the output of the discriminator
Revised: s i argument, but is simply computed from the output of the discriminator

Original: h f .
Revised: h f .

Original: However, when (cid:96) f is cross entropy we optimize the ﬁlter such that the discriminator h f is fooled into predicting the complement class 1 − s .
Revised: However, when (cid:96) f is cross entropy we optimize the ﬁlter such that the discriminator h f is fooled into predicting the complement class 1 − s .

Original: This means that we need to assume s to be a binary attribute when using this loss.
Revised: This means that we need to assume s to be a binary attribute when using this loss.

Original: When using the negative entropy we do not need to make this assumption, which is a strength of the proposed method.
Revised: When using the negative entropy we do not need to make this assumption, which is a strength of the proposed method.

Original: Algorithminput: D ,
Revised: Algorithminput: D ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ← (cid:15) repeat
Revised: ← (cid:15) repeat

Original: Draw m samples uniformly at random from the dataset
Revised: Draw m samples uniformly at random from the dataset

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: Draw m samples from the noise distribution
Revised: Draw m samples from the noise distribution

Original: [Equation]
Revised: [Equation]

Original: Draw m samples from the synthetic distribution
Revised: Draw m samples from the synthetic distribution

Original: [Equation]
Revised: [Equation]

Original: Compute censored and synthetic data
Revised: Compute censored and synthetic data

Original: [Equation]
Revised: [Equation]

Original: Compute ﬁlter and generator losses
Revised: Compute ﬁlter and generator losses

Original: [Equation]
Revised: [Equation]

Original: (cid:48)(cid:48)
Revised: (cid:48)(cid:48)

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: d ( x (cid:48)(cid:48)
Revised: d ( x (cid:48)(cid:48)

Original: i , x i ) − (cid:15), 0)
Revised: i , x i ) − (cid:15), 0)

Original: Update ﬁlter and generator parameters θ f ← Adam (Θ f ( θ f ); lr, β 1 , β 2 ) θ g ← Adam (Θ g ( θ g ); lr, β 1 , β 2 )
Revised: Update ﬁlter and generator parameters θ f ← Adam (Θ f ( θ f ); lr, β 1 , β 2 ) θ g ← Adam (Θ g ( θ g ); lr, β 1 , β 2 )

Original: Compute discriminator losses Φ f ( φ f ) = 1 m (cid:80) mi =1 (cid:96) h
Revised: Compute discriminator losses Φ f ( φ f ) = 1 m (cid:80) mi =1 (cid:96) h

Original: f
Revised: f

Original: ( h f ( x (cid:48)
Revised: ( h f ( x (cid:48)

Original: i ; φ f ) , s i )
Revised: i ; φ f ) , s i )

Original: Φ g ( φ g ) =
Revised: Φ g ( φ g ) =

Original: 1 m m (cid:88) i
Revised: 1 m m (cid:88) i

Original: = (cid:96) g
Revised: = (cid:96) g

Original: ( h g ( x (cid:48)(cid:48) i ; φ g ) , fake ) +m m (cid:88) i
Revised: ( h g ( x (cid:48)(cid:48) i ; φ g ) , fake ) +m m (cid:88) i

Original: = (cid:96) g
Revised: = (cid:96) g

Original: ( h g ( x i ; φ g ) , s i )
Revised: ( h g ( x i ; φ g ) , s i )

Original: Update discriminator parameters φ f ← Adam (Φ f ( φ f ); lr, β 1 , β 2 ) φ g ← Adam (Φ g ( φ g ); lr, β 1 , β 2 ) until stopping criterion return θ f , θ g , φ f , φ g
Revised: Update discriminator parameters φ f ← Adam (Φ f ( φ f ); lr, β 1 , β 2 ) φ g ← Adam (Φ g ( φ g ); lr, β 1 , β 2 ) until stopping criterion return θ f , θ g , φ f , φ g