Original: PrivE : Empirical Privacy Evaluation of Synthetic DataGenerators
Revised: TAPAS : a Toolbox for Adversarial Privacy Auditing of Synthetic Data

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Personal data collected at scale promises to improve decision-making and accelerateinnovation.
Revised: Personal data collected at scale promises to improve decision-making and accelerate innovation.

Original: However, sharing and using such data raises serious privacy concerns.
Revised: However, sharing and using such data raises serious privacy concerns.

Original: A promising solution is to produce synthetic data, artificial records to share insteadof real data.
Revised: A promising solution is to produce synthetic data, artificial records to share instead of real data.

Original: Since synthetic records are not linked to real persons, this intuitivelyprevents classical re-identification attacks.
Revised: Since synthetic records are not linked to real persons, this intuitively prevents classical re-identification attacks.

Original: However, this is insufficient to protectprivacy.
Revised: However, this is insufficient to protect privacy.

Original: We here present PrivE , a toolbox of attacks to evaluate synthetic dataprivacy under a wide range of scenarios.
Revised: We here present TAPAS , a toolbox of attacks to evaluate synthetic data privacy under a wide range of scenarios.

Original: These attacks include generalizations ofprior works and novel attacks.
Revised: These attacks include generalizations of prior works and novel attacks.

Original: We also introduce a general framework for reasoningabout privacy threats to synthetic data and showcase PrivE on several examples.
Revised: We also introduce a general framework for reasoning about privacy threats to synthetic data and showcase TAPAS on several examples.

Original:  Introduction
Revised:  Introduction

Original: Synthetic data generation is a promising technology to enable access to sensitive data and acceleratemachine learning projects while protecting user privacy, and has attracted a lot of attention fromresearch and industry.
Revised: Synthetic data generation is a promising technology to enable access to sensitive data and accelerate machine learning projects while protecting user privacy and has attracted a lot of attention from research and industry.

Original: The key idea is to produce and share semantically, and possibly statistically,accurate artificial data sets instead of the real data.
Revised: The key idea is to produce and share semantically and possibly statistically, accurate artificial data sets instead of the real data.

Original: While this intuitively seems to protect individuals, synthetic records are not automatically private.
Revised: While this intuitively seems to protect individuals, synthetic records are not automatically private.

Original: Apart from the obvious pitfall that large models can memorize real records, reproducing manystatistics of the original data can enable reconstruction attacks [1, 2].
Revised: Apart from the obvious pitfall that large models can memorize real records, reproducing many statistics of the original data can enable reconstruction attacks [1, 2].

Original: Further, in some cases, amotivated attacker can infer sensitive information about real records based only on synthetic ones [3].
Revised: Further, in some cases, a motivated attacker can infer sensitive information about real records based only on synthetic ones [3].

Original: Adversarial evaluation of privacy is a promising approach, which has the ability to take into accounta large range of potential attackers, with different knowledge of the real dataset and synthetic datageneration algorithm, and different attack goals.
Revised: Adversarial evaluation of privacy is a promising approach, which has the ability to take into account a large range of potential attackers, with different knowledge of the real dataset and synthetic data generation algorithm, and different attack goals.

Original: However, existing attacks are limited in scope,usually applying to one specific threat model.
Revised: However, existing attacks are limited in scope, usually applying to one specific threat model.

Original: In this paper, we present PrivE , a toolbox for evaluating the privacy of synthetic data generators.
Revised: In this paper, we present TAPAS , a toolbox for evaluating the privacy of synthetic data generators.

Original: We first propose a threat modeling framework to define privacy attacks.
Revised: We first propose a threat modeling framework to define privacy attacks.

Original: This framework modelsa wide range of attacker background knowledge and goals, allowing for a context-aware privacyanalysis.
Revised: This framework models a wide range of attacker background knowledge and goals, allowing for a context-aware privacy analysis.

Original: Second, we describe the library of attacks implemented in PrivE .
Revised: Second, we describe the library of attacks implemented in TAPAS .

Original: Finally, we showcase theprivacy evaluation of PrivE , applying our toolbox to several attack scenarios and generators.
Revised: Finally, we showcase the privacy evaluation of TAPAS , applying our toolbox to several attack scenarios and generators.

Original: Wealso highlight gaps in current research on empirical privacy evaluation.
Revised: We also highlight gaps in current research on empirical privacy evaluation.

Revised: The toolbox is open source, and its code is available at https://github.com/alan-turing-institute/privacy-sdg-toolbox .

Original:  Background2.1 Synthetic Data Generation
Revised:  Background 2.1 Synthetic Data Generation

Original: Denote by X a set of potential records, and by D =
Revised: Denote by X a set of potential records, and by D =

Original: (cid:83) N ∈ N ∪{ 0 } X N the set of finite collections (withrepeats) taken from X .
Revised: (cid:83) N ∈ N ∪{ 0 } X N the set of finite collections (with repeats) taken from X .

Original: A synthetic data generation model (SDG or generator ) is a randomized 1function G : Ω ×D → D that takes as input a ( real ) dataset D ( r ) ∈ D and produces a synthetic dataset
Revised: A synthetic data generation model (SDG or generator ) is a randomizedfunction G : Ω ×D → D that takes as input a ( real ) dataset D ( r ) ∈ D and produces a synthetic dataset

Original: D ( s ) =
Revised: D ( s ) =

Original: G ( D ( r ) ) .
Revised: G ( D ( r ) ) .

Original: The goal is for the synthetic data D ( s ) =
Revised: The goal is for the synthetic data D ( s ) =

Original: G ( D ( r ) ) to be a useful replacement forthe real data D ( r ) for given data science tasks, without revealing sensitive information about D ( r ) .
Revised: G ( D ( r ) ) to be a useful replacement for the real data D ( r ) for given data science tasks, without revealing sensitive information about D ( r ) .

Original: Generators typically operate in two steps: a training step, where a parameter θ ∈ Θ is adapted to
Revised: Generators typically operate in two steps: a training step, where a parameter θ ∈ Θ is adapted to

Original: D ( r ) , and a sampling step, where synthetic records are sampled i.i.d.
Revised: D ( r ) , and a sampling step, where synthetic records are sampled i.i.d.

Original: from a distribution p θ on X .
Revised: from a distribution p θ on X .

Original: 2.2 Privacy attacks on synthetic data
Revised: 2 Privacy attacks on synthetic data

Original: Adversarial approaches can evaluate the privacy of synthetic data, either as an alternative or complement to formal guarantees.
Revised: Adversarial approaches can evaluate the privacy of synthetic data, either as an alternative or complement to formal guarantees.

Original: Stadler et al.
Revised: Stadler et al.

Original: [3] proposed a black-box attack against synthetic data thatrelies on shadow modeling.
Revised: [3] proposed a black-box attack against synthetic data that relies on shadow modeling.

Original: Lu et al.
Revised: Lu et al.

Original: [4] and Yale et al.
Revised: [4] and Yale et al.

Original: [5] evaluate privacy using the distance betweensynthetic and real records.
Revised: [5] evaluate privacy using the distance between synthetic and real records.

Original: In some settings, attacks against other privacy-enhancing technologies canapply to synthetic data, including Generative Adversarial Networks [6, 7].
Revised: In some settings, attacks against other privacy-enhancing technologies can apply to synthetic data, including Generative Adversarial Networks [6, 7].

Original: Commonly, generators useaggregate statistics from real data when generating synthetic data.
Revised: Commonly, generators use aggregate statistics from real data when generating synthetic data.

Original: In this case, attacks developedagainst aggregate statistics apply, e.g. [8, 9, 10].
Revised: In this case, attacks developed against aggregate statistics apply, e.g. [8, 9, 10].

Original: Adversarial approaches have three key advantages: (1) Bug detection : attacks can exploit bugsin implementations of theoretically secure mechanisms (e.g. leaky categories [3]); (2) Contextawareness : attacks can incorporate assumptions on what the attacker knows, and what they try tolearn; and (3) Comparison across models : generators can be effectively compared without requiringcomparable or tight formal privacy guarantees.
Revised: Adversarial approaches have three key advantages: (1) Bug detection : attacks can exploit bugs in implementations of theoretically secure mechanisms (e.g. leaky categories [3]); (2) Context awareness : attacks can incorporate assumptions on what the attacker knows, and what they try to learn; and (3) Comparison across models : generators can be effectively compared without requiring comparable or tight formal privacy guarantees.

Original: 2.3 Differential Privacy
Revised: 2.3 Differential Privacy

Original: A common approach to privacy is to ensure that the generator satisfies formal guarantees such asdifferential privacy (DP), introduced by Dwork et al.
Revised: A common approach to privacy is to ensure that the generator satisfies formal guarantees such as differential privacy (DP), introduced by Dwork et al.

Original: DP applies to randomized mechanisms
Revised: DP applies to randomized mechanisms

Original: M : D × Ω → S and requires that distributions induced by M are close when datasets vary slightly.
Revised: M : D × Ω → S and requires that distributions induced by M are close when datasets vary slightly.

Original: More precisely, we define two datasets to be neighbors (denoted “ ∼ = ”) if they differ by the addition ordeletion of one record.
Revised: More precisely, we define two datasets to be neighbors (denoted “ ∼ = ”) if they differ by the addition or deletion of one record.

Original: Definition 1 (Differential Privacy) .
Revised: Definition 1 (Differential Privacy) .

Original: A randomized mechanism M : D × Ω → S over datasetsprovides ( ε, δ ) − Differential Privacy if, for all d ∼ = d ′ ∈ D and all S ⊂ S ,
Revised: A randomized mechanism M : D × Ω → S over datasets provides ( ε, δ ) − Differential Privacy if, for all d ∼ = d ′ ∈ D and all S ⊂ S ,

Original: [Equation]
Revised: [Equation]

Original: The parameter ε is called the privacy budget .
Revised: The parameter ε is called the privacy budget .

Original: Smaller ε implies more privacy, and typically less utilityof the mechanism.
Revised: Smaller ε implies more privacy, and typically less utility of the mechanism.

Original: A valuable property of differential privacy is post-processing : if M is ( ε, δ ) − DPthen, for any randomised operation F : Ω × O → O ′ , the composition F ◦ M is also ( ε, δ ) − DP.
Revised: A valuable property of differential privacy is post-processing : if M is ( ε, δ ) − DP then, for any randomised operation F : Ω × O → O ′ , the composition F ◦ M is also ( ε, δ ) − DP.

Original: Therefore, no operation performed on the output can “break” the privacy guarantees.
Revised: Therefore, no operation performed on the output can “break” the privacy guarantees.

Original: Differential privacy guarantees resistance to membership inference attacks , where an attacker attemptsto infer whether a target record x is in D ( r ) .
Revised: Differential privacy guarantees resistance to membership inference attacks , where an attacker attempts to infer whether a target record x is in D ( r ) .

Original: Indeed, suppose an attacker seeks to distinguish betweenthe cases D ( r )
Revised: Indeed, suppose an attacker seeks to distinguish between the cases D ( r )

Original: =
Revised: =

Original: d and D ( r ) =
Revised: d and D ( r ) =

Original: d ′ =
Revised: d ′ =

Original: d ∪ { x } , for a target x and dataset d .
Revised: d ∪ { x } , for a target x and dataset d .

Original: Kairouz et al.
Revised: Kairouz et al.

Original: [12] showa tradeoff between the attacker’s true and false positive rates, which applies even in the worst-caseattack model where the target x and the dataset d = D ( r ) \ { x } are known.
Revised: [12] show a tradeoff between the attacker’s true and false positive rates, which applies even in the worst-case attack model where the target x and the dataset d = D ( r ) \ { x } are known.

Original: Theorem 1 ([12]) .
Revised: Theorem 1 ([12]) .

Original: Let M : Ω × D
Revised: Let M : Ω × D

Original: → O satisfy ( ε, δ ) − differential privacy and d, d ′ ∈ D beneighboring datasets ( d ∼ = d ′ ).
Revised: → O satisfy ( ε, δ ) − differential privacy and d, d ′ ∈ D be neighboring datasets ( d ∼ = d ′ ).

Original: Then, for any randomized attacker A : Ω × O → { d, d ′ } ,
Revised: Then, for any randomized attacker A : Ω × O → { d, d ′ } ,

Original: [Equation]
Revised: [Equation]

Original: , where TP A = P [ A ( M ( d ))
Revised: where TP A = P [ A ( M ( d ))

Original: =
Revised: =

Original: d ] and FP A = P
Revised: d ] and FP A = P

Original: [ A ( M ( d ′ ))
Revised: [ A ( M ( d ′ ))

Original: =
Revised: =

Original: d ] are true and false positive rates.
Revised: d ] are true and false positive rates.

Original: However, this inequality is not generally tight.
Revised: However, this inequality is not generally tight.

Original: One can thus define the effective ε eff ( δ ) :
Revised: One can thus define the effective ε eff ( δ ) :

Original: [Equation]
Revised: [Equation]

Original: For any d ∼ = d ′ , Theorem 1 shows ε ( δ ) ≥ ε eff ( δ ; d, d ′ ) for all δ ≥ 0 ; however ε eff can be used toprove that a privacy analysis is tight [13, 14].
Revised: For any d ∼ = d ′ , Theorem 1 shows ε ( δ ) ≥ ε eff ( δ ; d, d ′ ) for all δ ≥ 0 ; however ε eff can be used to prove that a privacy analysis is tight [13, 14].

Original: Further, ε eff can measure privacy protection in differentcontexts, i.e. under different assumptions about the attacker A .
Revised: Further, ε eff can measure privacy protection in different contexts, i.e. under different assumptions about the attacker A .

Original: Specifically, for synthetic data,many SDGs are made differentially private with respect to their parameters θ , usually through noiseaddition during model training.
Revised: Specifically, for synthetic data, many SDGs are made differentially private with respect to their parameters θ , usually through noise addition during model training.

Original: As sampling the model is a form of post-processing, D ( s ) =
Revised: As sampling the model is a form of post-processing, D ( s ) =

Original: G θ ( D ( r ) )is equally ( ε, δ ) -DP, but this guarantee is most likely not tight, as sampling introduces additionalrandomness.
Revised: G θ ( D ( r ) ) is equally ( ε, δ ) -DP, but this guarantee is most likely not tight, as sampling introduces additional randomness.

Original: Thus, ε eff can help evaluate more realistic quantitative privacy guarantees.
Revised: Thus, ε eff can help evaluate more realistic quantitative privacy guarantees.

Original:  Adversarial toolbox
Revised:  Adversarial toolbox

Original: PrivE is a toolbox for adversarial evaluation of synthetic data.
Revised: TAPAS is a toolbox for adversarial evaluation of synthetic data.

Original: We here introduce our threat modelingframework, which forms the foundation of the PrivE analysis.
Revised: We here introduce our threat modeling framework, which forms the foundation of the TAPAS analysis.

Original: We then detail the library of attacksimplemented within the toolbox.
Revised: We then detail the library of attacks implemented within the toolbox.

Original: We finally review the reporting options provided by PrivE .
Revised: We finally review the reporting options provided by TAPAS .

Original: 3.1 Threat Modeling
Revised: 3.1 Threat Modeling

Original: A threat model (or attack model) defines the setup in which attacks take place, and what an attackeris assumed to know.
Revised: A threat model (or attack model) defines the setup in which attacks take place, and what an attacker is assumed to know.

Original: This defines what attacks can be performed on a synthetic dataset.
Revised: This defines what attacks can be performed on a synthetic dataset.

Original: The goal offormally defining a threat model is to evaluate the privacy guarantees of synthetic data contextually ,i.e. under assumptions as to what a realistic attacker might know.
Revised: The goal of formally defining a threat model is to evaluate the privacy guarantees of synthetic data contextually , i.e. under assumptions as to what a realistic attacker might know.

Original: We propose a modular threat modeling framework, defined by three attributes of the attacker thatcan be modified independently: (1) their knowledge of the real dataset D ( r ) , (2) their knowledge ofthe generator G , and (3) what they are trying to learn.
Revised: We propose a modular threat modeling framework, defined by three attributes of the attacker that can be modified independently: (1) their knowledge of the real dataset D ( r ) , (2) their knowledge of the generator G , and (3) what they are trying to learn.

Original: PrivE implements this modular framework,enabling users to evaluate the privacy of generators in a wide range of scenarios within the toolbox.
Revised: TAPAS implements this modular framework, enabling users to evaluate the privacy of generators in a wide range of scenarios within the toolbox.

Original: 3.1.1 Knowledge of the dataset
Revised: 3.1.1 Knowledge of the dataset

Original: The attacker is assumed to have (potentially) uncertain knowledge about the real data D ( r ) .
Revised: The attacker is assumed to have (potentially) uncertain knowledge about the real data D ( r ) .

Original: Thisis captured by a prior over datasets, D ( r ) ∼ π D , similarly to the framework of Li et al.
Revised: This is captured by a prior over datasets, D ( r ) ∼ π D , similarly to the framework of Li et al.

Original: Twocommon examples from prior work are Auxiliary and Exact knowledge.
Revised: Two common examples from prior work are Auxiliary and Exact knowledge.

Original: In the former, the dataset is arandom subset of a larger (“population”) dataset D (pop) .
Revised: In the former, the dataset is a random subset of a larger (“population”) dataset D (pop) .

Original: This is a common assumption for attackerknowledge (see, e.g., [16, 3]).
Revised: This is a common assumption for attacker knowledge (see, e.g., [3, 16]).

Original: In the latter, the attacker knows that the dataset is one of two datasets( d and d ′ ) that differ only in one entry.
Revised: In the latter, the attacker knows that the dataset is one of two datasets ( d and d ′ ) that differ only in one entry.

Original: This is typically used to verify DP guarantees [14].
Revised: This is typically used to verify DP guarantees [14].

Original: PrivE optimizes for targeted attacks (i.e. attacks that concern one specific record t ) by combiningprior knowledge of the dataset excluding t (so D ( r ) − t ∼ π d ′ ) and knowledge of the target t , which areassumed to be independent from each other.
Revised: TAPAS optimizes for targeted attacks (i.e. attacks that concern one specific record t ) by combining prior knowledge of the dataset excluding t (so D ( r ) − t ∼ π d ′ ) and knowledge of the target t , which are assumed to be independent from each other.

Original: For instance, if the attacker assumes P ( t ∈ D ( r ) ) = 0 . 8 ,and otherwise t is replaced by another record t ′ , while the remainder D − t is sampled from π ′ , PrivEcombines these to give the prior π D : d (cid:55)→ 0 . {+
+}
Revised: For instance, if the attacker assumes P ( t ∈ D ( r ) ) = 0 . 8 , and otherwise t is replaced by another record t ′ , while the remainder D − t is sampled from π ′ , TAPAS combines these to give the prior π D : d (cid:55)→ 0 . {+
+}

Original: · I { t ∈ d } π ′ ( d \ { t } ) + 0 .
Revised: · I { t ∈ d } π ′ ( d \ { t } ) + 0 .

Original: 2 · I { t ′ ∈
Revised: 2 · I { t ′ ∈

Original: d } π ′ ( d \ { t ′ } ) .
Revised: d } π ′ ( d \ { t ′ } ) .

Original: 3.1.2 Knowledge of the generator
Revised: 3.1.2 Knowledge of the generator

Original: It is important to model the information that the attacker has about the generator.
Revised: It is important to model the information that the attacker has about the generator.

Original: Typically, there arethree situations:
Revised: Typically, there are three situations:

Original: • No-box , where the attacker has no information about the generator.
Revised: • No-box , where the attacker has no information about the generator.

Original: • Black-box , where the attacker has exact knowledge of the generator and is able to apply the function G to arbitrary datasets.
Revised: • Black-box , where the attacker has exact knowledge of the generator and is able to apply the function G to arbitrary datasets.

Original: This is the most common assumption, as it is consistent with good security practices.
Revised: This is the most common assumption, as it is consistent with good security practices.

Original: • White-box , which is an extension of black-box where the attacker additionally has access to trained parameters of the generators, such as model weights.
Revised: • White-box , which is an extension of black-box where the attacker additionally has access to trained parameters of the generators, such as model weights.

Original: This additional informationcan be used, e g. to apply tailored attacks to the specific generative model used by G .
Revised: This additional information can be used, e g. to apply tailored attacks to the specific generative model used by G .

Original: PrivE supports an additional setup, which we call Uncertain-box , where the generator functionis parameterized by “meta-parameters” γ ∈ Γ , so D ( s ) =
Revised: TAPAS supports an additional setup, which we call Uncertain-box , where the generator function is parameterized by “meta-parameters” γ ∈ Γ , so D ( s ) =

Original: G Γ ( D ( r ) , γ ) .
Revised: G Γ ( D ( r ) , γ ) .

Original: The attacker knows thegenerator function G Γ , but has uncertainty on the meta-parameter γ ∼ π G .
Revised: The attacker knows the generator function G Γ , but has uncertainty on the meta-parameter γ ∼ π G .

Original: PrivE focuses on black-,uncertain- and no-box threat models, all three of which can be modeled as ( G Γ , π G ) .
Revised: TAPAS focuses on black-, uncertain- and no-box threat models, all three of which can be modeled as ( G Γ , π G ) .

Original: 3.1.3 Attacker goal
Revised: 3.1.3 Attacker goal

Original: Through their attack, the attacker aims to infer private information about the real dataset.
Revised: Through their attack, the attacker aims to infer private information about the real dataset.

Original: This can berepresented by a function mapping a dataset to a decision g : D → S .
Revised: This can be represented by a function mapping a dataset to a decision g : D → S .

Original: Three common categories ofattacks are:
Revised: Three common categories of attacks are:

Original: • Targeted Membership Inference (MIA) for a target record t : assess whether the target is inthe real dataset, g : D ( r )
Revised: • Targeted Membership Inference (MIA) for a target record t : assess whether the target is in the real dataset, g : D ( r )

Original: (cid:55)→ I { t ∈ D ( r ) } .
Revised: (cid:55)→ I { t ∈ D ( r ) } .

Original: • Targeted Attribute Inference (AIA) for an attributea and incomplete target record t − a :
Revised: • Targeted Attribute Inference (AIA) for an attribute

Original:  find the value v of a such that the completed record t − a | v is in the data, g : D ( r ) (cid:55)→v s.t. t − a | v ∈ D ( r ) .
Revised: a and incomplete target record t − a : find the value v of a such that the completed record t − a | v is in the data, g : D ( r ) (cid:55)→ v s.t. t − a | v ∈ D ( r ) .

Original: This assumes the datasets are categorical , i.e. X = X 1 × · · · × X k .
Revised: This assumes the datasets are tabular , i e. X = X 1 × · · · × X k .

Revised: In this paper, we focus on categorical sensitive attributes,

Original:  • Reconstruction : a special case where the attacker aims to reconstruct the entire real dataset g : D ( r ) (cid:55)→ D ( r ) .
Revised: i e., where |X a | is finite. • Reconstruction : a special case where the attacker aims to reconstruct the entire real dataset g : D ( r ) (cid:55)→ D ( r ) .

Original: PrivE currently focuses on the first two classes of attacks, as does most literature on privacy attackson synthetic data.
Revised: TAPAS currently focuses on the first two classes of attacks, as does most literature on privacy attacks on synthetic data.

Original: Reconstruction attacks are currently an underexplored research area.
Revised: Reconstruction attacks are currently an underexplored research area.

Original: 3.2 Evaluating attacks
Revised: 3.2 Evaluating attacks

Original: For a given attack goal, an attack is a (potentially randomized) function of a synthetic dataset thatoutputs a decision, A : D × Ω → S .
Revised: For a given attack goal, an attack is a (potentially randomized) function of a synthetic dataset that outputs a decision, A : D × Ω → S .

Original: The success of an attack is captured by a criterion C :
Revised: The success of an attack is captured by a criterion C :

Original: D ( r ) × S → R for one specific dataset and decision.
Revised: D ( r ) × S → R for one specific dataset and decision.

Original: This can simply be whether the guess is correct
Revised: This can simply be whether the guess is correct

Original: C : ( D ( r ) , s ) (cid:55)→ I { g ( D ( r ) )
Revised: C : ( D ( r ) , s ) (cid:55)→ I { g ( D ( r ) )

Original: =
Revised: =

Original: s } .
Revised: s } .

Original: The success rate of an attack A , for a specific real dataset D ( r ) ∗ andgenerator meta-parameter γ ∗ is E D ( s ) ∼G Γ ( D ( r ) ∗ ,γ ∗ ) (cid:2) C (cid:0) D ( r ) ∗ ,
Revised: The success rate of an attack A , for a specific real dataset D ( r ) ∗ and generator meta-parameter γ ∗ is E D ( s ) ∼G Γ ( D ( r ) ∗ ,γ ∗ ) (cid:2) C (cid:0) D ( r ) ∗ ,

Original: A ( D ( s ) )
Revised: A ( D ( s ) )

Original: (cid:1)(cid:3) .
Revised: (cid:1)(cid:3) .

Original: In practice, PrivE uses adistribution over “real” datasets D ( r ) ∗ ∼ π ∗ D in this evaluation, to incorporate uncertainty in g ( D ( r ) ) .
Revised: In practice, TAPAS uses a distribution over “real” datasets D ( r ) ∗ ∼ π ∗ D in this evaluation, to incorporate uncertainty in g ( D ( r ) ) .

Original: Base Rate The attacker’s prior π D defines a base rate for a threat model, defined as themaximum success rate of an attack A base that does not have access to the synthetic dataset :max s ∈S E D ( r ) ∼ π D
Revised: Rate The attacker’s prior π D defines a base rate for a threat model, defined as the maximum success rate of an attack A base that does not have access to the synthetic dataset : max s ∈S E D ( r ) ∼ π D

Original: [ C ( D ( r ) , s )] .
Revised: [ C ( D ( r ) , s )] .

Original: In PrivE , threat models are defined with a known base rate, typically|S| − 1 for categorical decisions.
Revised: In TAPAS , threat models are defined with a known base rate, typically |S| − 1 for categorical decisions.

Original: Simulation Under a given threat model, the attacker is able to simulate the context, and generatetraining samples:
Revised: Simulation Under a given threat model, the attacker is able to simulate the context, and generate training samples:

Original: [Equation]
Revised: [Equation]

Original: These samples can be used to select parameters θ ∈ Θ of the attack A θ , e.g. to optimize C by takingθ
Revised: These samples can be used to select parameters θ ∈ Θ of the attack A θ , e.g. to optimize C by taking θ ∗ ∈ arg max θ ∈ Θ (cid:80)

Original: [Equation]
Revised: ki =1 C (cid:0) D ( r ) i , A θ ( D ( s ) i ) (cid:1) .

Original: 3.3 Library of Attacks
Revised: 3.3 Library of Attacks

Original: PrivE implements a range of attacks, or more formally, classes of attacks defined by (meta-)parameters.
Revised: TAPAS implements a range of attacks, or more formally, classes of attacks defined by meta-parameters.

Original: We focus on threat models with finite decision sets ( |S| = n cat ∈ N ), and writewlog S=
Revised: We focus on threat models with finite decision sets ( |S| = n cat ∈ N ), and write without loss of generality S

Original:  { s 1 , . . .
Revised: = { s 1 , . . .

Original: , s n cat } .
Revised: , s n cat } .

Original: We call these “label-inference attacks”.
Revised: We call these “label-inference attacks”.

Original: In this setup, PrivE definesattacks as functions mapping a synthetic dataset to a vector of scores 2 A : D → R n cat .
Revised: In this setup, TAPAS defines attacks as functions mapping a synthetic dataset to a vector of scores 2 A : D → R n cat .

Original: A decisionis then based on the score, typically arg max i σ i ( D ) , but more complex decisions can also be considered.
Revised: A decision is then based on the score, typically arg max i σ i ( D ) , but more complex decisions can also be considered.

Original: If n cat = 2 (“binary attacks”), an attack can be summarized by a score and a threshold 3 as:
Revised: If n cat = 2 (“binary attacks”), an attack can be summarized by a score and a thresholdas:

Original: A σ,τ ( D ) :
Revised: A σ,τ ( D ) :

Original: = I { σ ( D ) ≥ τ } .
Revised: = I { σ ( D ) ≥ τ } .

Original: Some attacks apply specifically to targeted membership (MIA) or attribute (AIA) inference.
Revised: Some attacks apply specifically to targeted membership (MIA) or attribute (AIA) inference.

Original: Targeted
Revised: Targeted

Original: MIAs are binary attacks where the score is proportional to the likelihood of membership.
Revised: MIAs are binary attacks where the score is proportional to the likelihood of membership.

Original: Targeted
Revised: Targeted

Original: AIAs are label-inference attacks where the decision set is the set of possible a values { v 1 , . .
Revised: AIAs are label-inference attacks where the decision set is the set of possible a values { v 1 , . .

Original: . , v l } .
Revised: . , v l } .

Original: Shadow-Modeling Attacks In a shadow modeling attack, the attacker first generates a large numberof “real” datasets D ( r ) i ∼ π D and synthetic datasets D ( s )
Revised: Shadow-Modeling Attacks In a shadow modeling attack, the attacker first generates a large number of “real” datasets D ( r ) i ∼ π D and synthetic datasets D ( s )

Original: i =
Revised: i =

Original: G ( D ( r ) i ) , then trains a classifier C θ topredict g ( D ( r ) i ) from D ( s ) i .
Revised: G ( D ( r ) i ) , then trains a classifier C θ to predict g ( D ( r ) i ) from D ( s ) i .

Original: This procedure requires a classifier over sets C θ : D → S .
Revised: This procedure requires a classifier over sets C θ : D → S .

Original: Stadler et al.
Revised: Stadler et al.

Original: [3] use a two-stage classifier combining a fixed set feature map ϕ : D → R l , witha “classical” classifier C θ : R l → S as C θ
Revised: [3] use a two-stage classifier combining a fixed set feature map ϕ : D → R l , with a “classical” classifier C θ : R l → S as C θ

Original: =
Revised: =

Original: C θ ◦ ϕ .
Revised: C θ ◦ ϕ .

Original: PrivE implements the three feature setsof Stadler et al.
Revised: TAPAS implements the three feature sets of Stadler et al.

Original: (basic statistics, histograms, and correlations), as well as additional feature maps.
Revised: (basic statistics, histograms, and correlations), as well as additional feature maps.

Original: In particular, we propose and implement the targeted counting queries feature map, Q t,S : D (cid:55)→1 | D | (cid:80) x ∈ D
Revised: In particular, we propose and implement the targeted counting queries feature map, Q t,S : D (cid:55)→| D | (cid:80) x ∈ D

Original: I { x s = t s ∀ s ∈ S } , defined for a random subset of attributes S .
Revised: I { x s = t s ∀ s ∈ S } , defined for a random subset of attributes S .

Original: We show in Section 4 thatusing this feature map empirically outperforms the feature maps of Stadler et al.
Revised: We show in Section 4 that using this feature map empirically outperforms the feature maps of Stadler et al.

Original: Local Neighborhood Attacks Given a distance over records d : X ×X → R + , this class of attacksuses synthetic data near a target record t to make a decision.
Revised: Local Neighborhood Attacks Given a distance over records d : X ×X → R + , this class of attacks uses synthetic data near a target record t to make a decision.

Original: The intuition is that the neighborhood oft is most likely to be influenced by the presence of t in D ( r ) .
Revised: The intuition is that the neighborhood of t is most likely to be influenced by the presence of t in D ( r ) .

Original: Distance to closest synthetic record is a common heuristic privacy metric for synthetic data [4,5].
Revised: Distance to closest synthetic record is a common heuristic privacy metric for synthetic data [4, 5].

Original: It corresponds to a targeted membership inference attack with target t and score σ ( D ) =
Revised: It corresponds to a targeted membership inference attack with target t and score σ ( D ) =

Original: −
Revised: −

Original: min x ∈ D
Revised: min x ∈ D

Original: d ( x, t ) .
Revised: d ( x, t ) .

Original: This generalizes the direct lookup attack (“is the real record in the data?”).
Revised: This generalizes the direct lookup attack (“is the real record in the data?”).

Original: Note that while this looks like a classical de-identification attack, presence of t in D ( s ) does notautomatically imply a privacy violation .
Revised: Note that while this looks like a classical de-identification attack, presence of t in D ( s ) does not automatically imply a privacy violation .

Original: This attack is an issue if and only if the record t is morelikely to be in D ( s ) when it is also in D ( r ) .
Revised: This attack is an issue if and only if the record t is more likely to be in D ( s ) when it is also in D ( r ) .

Original: We extend this attack to targeted attribute inference, where the score for value v i for the sensitiveattribute is σ i ( D ) =
Revised: We extend this attack to targeted attribute inference, where the score for value v i for the sensitive attribute is σ i ( D ) =

Original: −
Revised: −

Original: min x ∈ D
Revised: min x ∈ D

Original: d ( x, t − a | v i ) .
Revised: d ( x, t − a | v i ) .

Original: Note that the distance d can also be trained.
Revised: Note that the distance d can also be trained.

Original: For instance, Zhang et al.
Revised: For instance, Zhang et al.

Original: [17] propose a no-box attackagainst synthetic health records, using representation learning to train a similarity metric.
Revised: [17] propose a no-box attack against synthetic health records, using representation learning to train a similarity metric.

Original: Inference-on-Synthetic attacks This class of attacks relies on the idea that generators can “overfit”to D ( r ) .
Revised: Inference-on-Synthetic attacks This class of attacks relies on the idea that generators can “overfit” to D ( r ) .

Original: In that case, it is possible to train a model f θ to perform an attack directly on synthetic data.
Revised: In that case, it is possible to train a model f θ to perform an attack directly on synthetic data.

Original: For membership inference attacks, this involves fitting a density L θ : X → R + to D ( s ) .
Revised: For membership inference attacks, this involves fitting a density L θ : X → R + to D ( s ) .

Original: This can beany statistical model, e.g. a Gaussian mixture.
Revised: This can be any statistical model, e.g. a Gaussian mixture.

Original: Some generators (e.g. [18, 19]) explicitly fit such amodel, which in white-box scenarios can be used directly.
Revised: Some generators (e.g. [18, 19]) explicitly fit such a model, which in white-box scenarios can be used directly.

Original: Here, σ ( D ) =
Revised: Here, σ ( D ) =

Original: L θ ( t ) .
Revised: L θ ( t ) .

Original: For attribute inference, the attacker trains a classifier C θ : X → R n cat on D ( s ) to predict thesensitive attribute a from other attributes ( x − a ).
Revised: For attribute inference, the attacker trains a classifier C θ : X → R n cat on D ( s ) to predict the sensitive attribute a from other attributes ( x − a ).

Original: The score is the prediction score for the target t − a ,so σ ( D ) =
Revised: The score is the prediction score for the target t − a , so σ ( D ) =

Original: C θ ( t − a ) .
Revised: C θ ( t − a ) .

Original: This corresponds to a privacy metric called Correct Attribution Probability(CAP) [20].
Revised: This corresponds to a privacy metric called Correct Attribution Probability (CAP) [20].

Original: This attack suffers from the base rate problem: correlations between sensitive andnon-sensitive attributes can make this attack successful even if t ̸∈ D ( r ) .
Revised: This attack suffers from the base rate problem: correlations between sensitive and non-sensitive attributes can make this attack successful even if t ̸∈ D ( r ) .

Original: Whether this is a privacyviolation is a contentious point in the community.
Revised: Whether this is a privacy violation is a contentious point in the community.

Original: PrivE addresses this by randomizing the sensitiveattribute of the target records uniformly at random when training the classifier.
Revised: TAPAS addresses this by randomizing the sensitive attribute of the target records uniformly at random when training the classifier.

Original: 3.4 Summarizing results
Revised: 3.4 Summarizing results

Original: PrivE provides several analyses of the outcome of attacks (called reports ).
Revised: TAPAS provides several analyses of the outcome of attacks (called reports ).

Original: These reports aggregatethe result of a range of attacks run on many test pairs ( D ( r ) i , D ( s ) i ) .
Revised: These reports aggregate the result of a range of attacks run on many test pairs ( D ( r ) i , D ( s ) i ) .

Original: Three main reports are:
Revised: Three main reports are:

Original: • Classification metrics : accuracy, true/false positive rates, AUROC, as well as privacy gainmetrics from Stadler et al.
Revised: • Classification metrics : accuracy, true/false positive rates, AUROC, as well as privacy gain metrics from Stadler et al.

Original: • ROC curves : For binary attacks with scores, the true positive rate vs. false positive rates fora range of thresholds.
Revised: • ROC curves : For binary attacks with scores, the true positive rate vs. false positive rates for a range of thresholds.

Original: • Effective epsilon ε eff :
Revised: • Effective epsilon ε eff :

Original: This report first greedily selects an attack and threshold on a 10%subset of testing samples, then performs the procedure of Jagielski et al.
Revised: This report first greedily selects an attack and threshold on a 10% subset of testing samples, then performs the procedure of Jagielski et al.

Original: [14] to estimate astatistically significant lower bound on ε eff using the 90% remaining samples.
Revised: [14] to estimate a statistically significant lower bound on ε eff using the 90% remaining samples.

Original: This can beused to empirically verify differential privacy.
Revised: This can be used to empirically verify differential privacy.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original:  Examples
Revised:  Examples

Original: In this section, we showcase how PrivE can be used to evaluate the privacy of several generators.
Revised: In this section, we showcase how TAPAS can be used to evaluate the privacy of several generators.

Original: Our goal is to demonstrate how the toolbox can and should be used, rather than a thorough analysisof the privacy of synthetic data.
Revised: Our goal is to demonstrate how the toolbox can and should be used, rather than a thorough analysis of the privacy of synthetic data.

Original: Still, we show existing challenges in privacy evaluation.
Revised: Still, we show existing challenges in privacy evaluation.

Revised: We demonstrate the toolbox in a setting where privacy is particularly important, working with personal data.

Revised: We apply the toolbox to the Office for National Statistics’ 2011 Census Microdata Teaching

Original: We use the 1%teaching file of  the  England and Wales Census 4 , a dataset of 500 k records with 15categorical attributes.
Revised: File 4 , which contains an anonymised random sample of 1% of responses from the 2011 Census of England and Wales, a total of ~570,000 records with 15 categorical attributes.

Revised: The data is already recognised as non-disclosive, but our toolbox could be used in the future production of similar micro-data files.

Original: We consider three generators: PrivBayes [18], MST [19], and CT-GAN [21].
Revised: We consider three generators: PrivBayes [18], MST [19], and CT-GAN [21].

Revised: The first two respectively provide ε − and ( ε, δ ) − DP, while the last one does not.

Original: Experiment 1 : we consider an attacker performing an attribute-inference attack with auxiliaryknowledge of the dataset and black-box knowledge of the generator.
Revised: Experiment 1 : we consider an attacker performing an attribute-inference attack with auxiliary knowledge of the dataset and black-box knowledge of the generator.

Original: We split the dataset in two equalrandom parts (auxiliary and testing), from which “real” datasets of 5000 records are sampled.
Revised: We split the dataset in two equal random parts (auxiliary and testing), from which “real” datasets of 5000 records are sampled.

Original: Astarget, we select an “outlier” record, by choosing the record with lowest log-likelihood (assumingattributes were independently drawn from their respective empirical distributions) from a randomsample of 1000 records.
Revised: As target, we select an “outlier” record, by choosing the record with lowest log-likelihood (assuming attributes were independently drawn from their respective empirical distributions) from a random sample of 1000 records.

Original: We attack each generator independently, using ε = 10 and δ = 10 − 5 for DPgenerators.
Revised: We attack each generator independently, using ε = 10 and δ = 10 − 5 for DP generators.

Original: We apply a range of attacks: a local neighborhood attack with Hamming distance, aninference-on-synthetic attack with a random forest classifier, the Groundhog attack, the Groundhogattack with logistic regression as a learner, and a shadow-modeling attack with the random-queryfeature we propose.
Revised: We apply a range of attacks: a local neighborhood attack with Hamming distance, an inference-on-synthetic attack with a random forest classifier, the Groundhog attack, the Groundhog attack with logistic regression as a learner, and a shadow-modeling attack with the random-query feature we propose.

Original: We compare the resulting accuracy, privacy gain and AUC of these attacks infigure 1(a).
Revised: We compare the resulting accuracy, privacy gain and AUC of these attacks in figure 1(a).

Original: Experiment 2 : we demonstrate empirical estimation of ε eff on MST [19] with ε = 10 .
Revised: Experiment 2 : we demonstrate empirical estimation of ε eff on MST [19] with ε = 10 .

Original: We usethe exact knowledge setup, where the attacker knows that the dataset is either D + = d − t ∪ { t } or
Revised: We use the exact knowledge setup, where the attacker knows that the dataset is either D + = d − t ∪ { t } or

Original: D − = d − t ∪ { t ′ } for an arbitrary d of size 499 sampled from the full dataset, the same target t andanother arbitrary record t ′ .
Revised: D − = d − t ∪ { t ′ } for an arbitrary d of size 499 sampled from the full dataset, the same target t and another arbitrary record t ′ .

Original: We choose this setup to detect violations (since empirical estimates of
Revised: We choose this setup to detect violations (since empirical estimates of

Original: DP are computationally intensive [22]).
Revised: DP are computationally intensive [22]).

Original: We generate 1000 synthetic “training” datasets D + and D − to train the attacks, and 2500 synthetic “testing” datasets to evaluate ε eff .
Revised: We generate 1000 synthetic “training” datasets D + and D − to train the attacks, and 2500 synthetic “testing” datasets to evaluate ε eff .

Original: In figure 1(b), weshow the ROC curves produced by PrivE .
Revised: In figure 1(b), we show the ROC curves produced by TAPAS .

Original: We find that the shadow-modeling attack with randomquery features outperforms all other attacks, and is used for ε eff estimation.
Revised: We find that the shadow-modeling attack with random-query features outperforms all other attacks (with an AUC of 0.70, whereas the Groundhog attack achieves

Original: Our ε eff estimationprocedure produces a 95% confidence interval of ε eff ∈ [0 . 86 , 1 . 39] .
Revised: [Figure]

Original: Surprisingly, ε =10 is not inthis confidence interval.
Revised: , ε = not in this

Original: Three main factors may be responsible for this gap: (1) the privacy analysisof the method is not tight, (2) the privacy analysis is not tight to differentiate the specific datasets D −and D + , and (3) the attack used for ε eff estimation is not optimal.
Revised: Three main factors may be responsible for this gap: (1) the privacy analysis of the method is not tight; (2) the privacy analysis is not tight to differentiate the specific datasets D − and D + (which were sampled randomly, rather than specifically designed worst-case datasets); and (3) the attack used for ε eff estimation is not optimal.

Original: Further work is needed to assesswhich of these factors is the cause.
Revised: Further work is needed to assess which of these factors is the cause.

Revised: Author contributions

Revised: Conceptualization: All authors; Methodology: FH, JJ, AE, JG, CM, CRS; Software: FH, JJ, JG, CM, CRS;

Revised: Writing (initial): FH, SC, AE, LS; Writing (revisions): FH, OD; Project Administration: FH, JJ.