Revised: ζ -DP:

Original: C OMPLEX - VALUED DEEP LEARNING WITH DIFFERENTIAL PRIVACY
Revised: C OMPLEX -V ALUED D IFFERENTIAL P RIVACY AND

Revised: ITS

Revised: A PPLICATIONS

Original: Anonymous authors
Revised: TO N EURAL N ETWORK

Revised: T RAINING

Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present ζ -DP, an extension of differential privacy (DP) to complex-valued functions.
Revised: We present ζ -DP, an extension of differential privacy (DP) to complex-valued functions.

Original: After introducing the complex Gaussian mechanism, whose properties we characterise in terms of p ε, δ q -DP and R´enyi-DP, we present ζ -DP stochastic gradient descent ( ζ -DP-SGD), a variant of DP-SGD for training complex-valued neural networks.
Revised: After introducing the complex Gaussian mechanism, whose properties we characterise in terms of p ε, δ q -DP, R´enyi DP and Gaussian DP, we present ζ -DP stochastic gradient descent ( ζ -DP-SGD), a variant of DP-SGD for training complex-valued neural networks.

Original: We experimentally evaluate ζ -DP-SGD on three complexvalued tasks, i e. electrocardiogram classiﬁcation, speech classiﬁcation and magnetic resonance imaging (MRI) reconstruction.
Revised: We experimentally evaluate ζ -DP-SGD on three complex-valued tasks, i e. electrocardiogram classiﬁcation, speech classiﬁcation and magnetic resonance imaging (MRI) reconstruction.

Original: Moreover, we provide ζ -DP-SGD benchmarks for a large variety of complex-valued activation functions and on a complex-valued variant of the  MNIST dataset.
Revised: Moreover, we benchmark the performance of our methods on real- and complex-valued variants of the CIFAR-10 and MNIST datasets as well as a large range of complex-valued activation functions.

Original: Our experiments demonstrate that DP training of complex-valued neural networks is possible with rigorous privacy guarantees and excellent utility 
Revised: Our experiments demonstrate that DP training of complexvalued neural networks is possible with rigorous privacy guarantees and excellent utility and represents a promising technique to mitigate privacy-utility trade-offs

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability to harness diverse, feature-rich datasets for algorithm training can allow the scientiﬁc community to create machine learning (ML) models capable of solving challenging data-driven tasks.
Revised: The ability to harness diverse, feature-rich datasets for algorithm training can allow the scientiﬁc community to create machine learning (ML) models capable of solving challenging data-driven tasks.

Original: These include the creation of robust autonomous vehicles (Rao & Frtunikj, 2018), early-stage cancer discovery (Cruz & Wishart, 2006) or disease survival prediction (Rau et al., 2018).
Revised: These include the creation of robust autonomous vehicles (Rao & Frtunikj, 2018), earlystage cancer discovery (Cruz & Wishart, 2006) or disease survival prediction (Rau et al., 2018).

Original: A subclass of these ML problems is able to proﬁt particularly from the ability to execute deep learning workﬂows over complex-valued datasets, such as magnetic resonance imaging (MRI) (Virtue et al., 2017) or time-series data (Fan & Xiong, 2013; Kociuba & Rowe, 2016).
Revised: A subclass of these ML problems is able to proﬁt particularly from the ability to execute deep learning workﬂows over complex-valued datasets, such as magnetic resonance imaging (MRI) (Virtue et al., 2017) or time-series data (Fan & Xiong, 2013; Kociuba & Rowe, 2016).

Original: Complex-valued deep learning has seen increased traction in the past years, owing in part to the improved support by ML frameworks and the broader availability of graphics processing unit (GPU) hardware able to tackle the increased computational requirements (Bassey et al., 2021).
Revised: Complex-valued deep learning has seen increased traction in the past years, owing in part to the improved support by ML frameworks and the broader availability of graphics processing unit (GPU) hardware able to tackle the increased computational requirements (Bassey et al., 2021).

Original: However, since complex numbers are often used to represent signals derived from sensitive biological or medical records (Cole et al., 2020; K ¨ ustner et al., 2020; Peker, 2016), privacy constraints can render such datasets hard to obtain.
Revised: However, since complex numbers are often used to represent signals derived from sensitive biological or medical records (Cole et al., 2020; K¨ustner et al., 2020; Peker, 2016), privacy constraints can render such datasets hard to obtain.

Original: The resulting data scarcity impairs effective model training, prompting the adoption of regulationcompliant and privacy-preserving methods for data access.
Revised: The resulting data scarcity impairs effective model training, prompting the adoption of regulationcompliant and privacy-preserving methods for data access.

Original: Distributed computation methods such as federated learning (FL) (Koneˇcn´y et al., 2016) can partially address this requirement by only requiring participants to share results of the local computation rather than exchange data over the network.
Revised: Distributed computation methods such as federated learning (FL) (Koneˇcn´y et al., 2016) can partially address this requirement by only requiring participants to share results of the local computation rather than exchange data over the network.

Original: However, FL on its own has repeatedly been shown to be insufﬁcient in the task of privacy protection (Geiping et al., 2020;
Revised: However, FL on its own has repeatedly been shown to be insufﬁcient in the task of privacy protection (Geiping et al., 2020;

Original: Yin et al., 2021).
Revised: Yin et al., 2021).

Original: Thus, bridging the gap between data protection and utilisation for algorithmic training requires methods able to offer objective privacy guarantees.
Revised: Thus, bridging the gap between data protection and utilisation for algorithmic training requires methods able to offer objective privacy guarantees.

Original: Differential privacy (DP) (Dwork et al., 2014) has established itself as the cornerstone of such techniques and has been deployed in contexts like the US Census (Abowd, 2018) and distributed learning on mobile devices (Cormode et al., 2018).
Revised: Differential privacy (DP) (Dwork et al., 2014) has established itself as the cornerstone of such techniques and has been deployed in contexts like the US Census (Abowd, 2018) and distributed learning on mobile devices (Cormode et al., 2018).

Original: DP’s purview has been expanded to encompass deep learning through the introduction of DP stochastic gradient descent (DP-SGD)
Revised: DP’s purview has been expanded to encompass deep learning through the introduction of DP stochastic gradient descent (DP-SGD)

Original: (Abadi et al., 2016), allowing for the training of deep neural networks on private data.
Revised: (Abadi et al., 2016), allowing for the training of deep neural networks on private data.

Original: So far, however, the application of DP to complex-valued ML tasks remains drastically under-explored.
Revised: So far, however, the application of DP to complex-valued ML tasks remains drastically under-explored.

Original: Our work attempts to address this challenge through the following contributions:
Revised: Our work attempts to address this challenge through the following contributions:

Original: We extend DP to the complex domain through a collection of techniques we refer to as ζ -DP.
Revised: We extend DP to the complex domain through a collection of techniques we refer to as ζ -DP.

Original: We use this term instead of “complex-valued DP” for brevity and to avoid confusion with the abbreviation “cDP”, which is already used for concentrated DP (Dwork & Rothblum, 2016).
Revised: We use this term instead of “complex-valued DP” for brevity and to avoid confusion with the abbreviation “cDP”, which is already used for concentrated DP (Dwork & Rothblum, 2016).

Original: The letter ζ alludes to the complex-valued Riemann ζ function and is intended to convey the notion of “continuation” to the complex domain.
Revised: The letter ζ alludes to the complex-valued Riemann ζ function and is intended to convey the notion of “continuation” to the complex domain.

Original: We deﬁne and discuss the complex Gaussian Mechanism (GM) in Section 4.1 and show that its properties generalise corresponding results on real-valued functions.
Revised: We deﬁne and discuss the complex Gaussian Mechanism (GM) in Section 4.1 and show that its properties generalise corresponding results on real-valued functions.

Original: This allows us to interpret the complex GM through the lens of previous work on p ε, δ q -DP and R´enyi-DP (RDP).
Revised: This allows us to interpret the complex GM through the lens of previous work on p ε, δ q -DP, R´enyi-DP (RDP) and Gaussian DP (GDP).

Original: To enable the design and privacy-preserving training of complex-valued deep learning models, we introduce ζ -DP-SGD in Section 4.2. 4.
Revised: To enable the design and privacy-preserving training of complex-valued deep learning models, we introduce ζ -DP-SGD in Section 4.2. 4.

Original: Finally, in Section 5 we experimentally evaluate our techniques on several real-life neural network training tasks, i e. speech classiﬁcation, abnormality detection in electrocardiograms and magnetic resonance imaging (MRI) reconstruction.
Revised: In Section 5 we experimentally evaluate the aforementioned techniques on several real-life neural network training tasks, i.e. speech classiﬁcation, abnormality detection in electrocardiograms and magnetic resonance imaging (MRI) reconstruction.

Original: Moreover, we establish baselines for future work by providing benchmark results on a complex-valued variant of the MNIST dataset and on complex neural network activation functions both with and without ζ -DP-SGD
Revised: Moreover, we establish baselines for future work by providing benchmark results on real- and complex-valued variants of the MNIST and CIFAR-10 datasets and on complex neural network activation functions both with and without ζ -DP-SGD.

Revised: We conclude that our DP analysis and the utilisation of tight sensitivity accounting through the Wirtinger/ CR -calculus allows practitioners to avail themselves of the higher learning capacity of neural networks with complex-valued weights to achieve improved utility with the same privacy guarantees as corresponding real-valued architectures.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Prior work has addressed several challenges in non-private complex-valued deep learning, including the introduction of appropriate activation functions, and has presented applications in domains such as MRI reconstruction (K ¨ ustner et al., 2020) or time series analysis (Fink et al., 2014).
Revised: Prior work has addressed several challenges in non-private complex-valued deep learning, including the introduction of appropriate activation functions and has presented applications in domains such as MRI reconstruction (K ¨ ustner et al., 2020) or time series analysis (Fink et al., 2014).

Original: For a detailed overview of methodology and applications, we refer to Hirose (2012); Bassey et al. (2021).
Revised: For a detailed overview of methodology and applications, we refer to Hirose (2012); Bassey et al. (2021).

Original: Until recently, deep learning frameworks did not fully support complex arithmetic and automatic differentiation.
Revised: Until recently, deep learning frameworks did not fully support complex arithmetic and automatic differentiation.

Original: Hence, previous works (Trabelsi et al., 2017; Nazarov & Burnaev, 2020) express C  as R 2  and use two real-valued channels rather than complex ﬂoating-point numbers.
Revised: Hence, previous works (Trabelsi et al., 2017; Nazarov & Burnaev, 2020) express C n as R 2 n and use two real-valued channels rather than complex ﬂoating-point numbers.

Original: This approach can lead to a spurious increase in function sensitivity  and, by extension, to the addition of excessive noise in the private setting, adversely impacting utility.
Revised: This approach can lead to a spurious increase in function sensitivity by incorrectly computing gradient magnitudes and, by extension, to the addition of excessive noise in the private setting, adversely impacting utility.

Original: Our work speciﬁcally addresses this shortcoming through the use of  Wirtinger/ CR -calculus
Revised: Our work speciﬁcally addresses this shortcoming through the use of complex-valued weights and the Wirtinger/ CR -calculus

Original: (Wirtinger, 1927; Kreutz-Delgado, 2009).
Revised: (Wirtinger, 1927; Kreutz-Delgado, 2009).

Original: A limited number of studies have utilised DP techniques in conjunction with complex-valued data (Fan & Xiong, 2013; Fioretto et al., 2019), however, to our knowledge none has formalised a notion of complex-valued DP or investigated neural network applications.
Revised: Only a limited number of studies have utilised DP techniques in conjunction with complex-valued data (Fan & Xiong, 2013; Fioretto et al., 2019), however, to our knowledge none has formalised a notion of complex-valued DP or investigated neural network applications.

Original: The p ε, δ q -deﬁnition of DP and the Gaussian mechanism are essential to our formalism, and details on their real-valued deﬁnitions can be found in Dwork et al.
Revised: The p ε, δ q -deﬁnition of DP and the Gaussian mechanism are essential to our formalism, and details on their real-valued deﬁnitions can be found in Dwork et al.

Original: As stated above, DP-SGD was introduced by Abadi et al.
Revised: As stated above, DP-SGD was introduced by Abadi et al.

Original: R´enyi-DP (RDP) was introduced by Mironov (2017) as a relaxation of p ε, δ q -DP with favourable properties under composition, rendering it particularly useful for DP-SGD privacy accounting. 
Revised: R´enyi-DP (RDP) was introduced by Mironov (2017) as a relaxation of p ε, δ q -DP with favourable properties under composition, rendering it particularly useful for DP-SGD privacy accounting. Gaussian DP (GDP) was introduced by Dong et al.

Revised: and tailored to the speciﬁc properties of the Gaussian mechanism, for which it provides a tight privacy analysis.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We begin by introducing key terminology required in the rest of our work.
Revised: We begin by introducing key terminology required in the rest of our work.

Original: We assume that a trusted analyst in possession of sensitive data wishes to publish the results of some analysis performed on this data while offering the individuals to whom the data belongs a DP guarantee.
Revised: We assume that a trusted analyst in possession of sensitive data wishes to publish the results of some analysis performed on this data while offering the individuals to whom the data belongs a DP guarantee.

Original: We will refer to the set of all sensitive records as the sensitive database D , whereby we assume that one individual’s data is only present in the database once.
Revised: We will refer to the set of all sensitive records as the sensitive database D , whereby we assume that one individual’s data is only present in the database once.

Original: Let X , the metric space of all sensitive databases, be equipped with the Hamming metric d X and let D P X .
Revised: Let X , the metric space of all sensitive databases, be equipped with the Hamming metric d X and let D P X .

Original: D ’s adjacent database D 1 can be constructed from D by adding or removing exactly one database row (that is, one individual’s data), such that d X p D, D 1 q “ 1 .
Revised: D ’s adjacent database D 1 can be constructed from D by adding or removing exactly one database row (that is, one individual’s data), such that d X p D, D 1 q “ 1 .

Original: The analyst executes a query (function) f , for example a mean calculation, over the database.
Revised: The analyst executes a query (function) f , for example a mean calculation, over the database.

Original: We ﬁrst deﬁne the sensitivity of f :
Revised: We ﬁrst deﬁne the sensitivity of f :

Original: Deﬁnition 1 (Sensitivity ∆ of f ) .
Revised: Deﬁnition 1 (Sensitivity ∆ of f ) .

Original: Let X and d X be deﬁned as above.
Revised: Let X and d X be deﬁned as above.

Original: f maps the elements of X to elements of a metric space Y equipped with a metric d Y .
Revised: f maps the elements of X to elements of a metric space Y equipped with a metric d Y .

Original: The (global) sensitivity ∆ of f is then deﬁned as:
Revised: The (global) sensitivity ∆ of f is then deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The maximum is taken over all adjacent database pairs in X .
Revised: The maximum is taken over all adjacent database pairs in X .

Original: When Y is the Euclidean space and d Y is the L 2 metric, ∆ is referred to as the L 2 sensitivity.
Revised: When Y is the Euclidean space and d Y is the L 2 metric, ∆ is referred to as the L 2 sensitivity.

Original: We will only use the L 2 sensitivity in this work.
Revised: We will only use the L 2 sensitivity in this work.

Original: In private data analysis and ML, we are often concerned with differentiable functions; for Lipschitzcontinuous query functions, the equivalence of the Lipschitz constant and the L 2 -sensitivity (Raskhodnikova & Smith, 2016) can be exploited:
Revised: In private data analysis and ML, we are often concerned with differentiable functions; for Lipschitzcontinuous query functions, the equivalence of the Lipschitz constant and the L 2 -sensitivity (Raskhodnikova & Smith, 2016) can be exploited:

Original: Deﬁnition 2 (Lipschitz constant K of f ) .
Revised: Deﬁnition 2 (Lipschitz constant K of f ) .

Original: Let X, Y, d X and d Y be deﬁned as above.
Revised: Let X, Y, d X and d Y be deﬁned as above.

Original: Then f is said to be K -Lipschitz continuous if and only if a non-negative real number K f exists for which the following holds:
Revised: Then f is said to be K -Lipschitz continuous if and only if a non-negative real number K f exists for which the following holds:

Original: [Equation]
Revised: [Equation]

Original: Evidently, K f ” ∆ by Equation (1) and the deﬁnition of adjacency.
Revised: Evidently, K f ” ∆ by Equation (1) and the deﬁnition of adjacency.

Original: Moreover, let D be the differential operator; then K f “ sup } D p f q} , where } ¨ } is the operator norm (O’Searcoid, 2006).
Revised: Moreover, let D be the differential operator; then K f “ sup } D p f q} , where } ¨ } is the operator norm (O’Searcoid, 2006).

Original: Therefore, for a scalar-valued query function, ∆ ” K f ” sup } ∇ f } 2 .
Revised: Therefore, for a scalar-valued query function, ∆ ” K f ” sup } ∇ f } 2 .

Original: A DP mechanism adds noise to the results of f calibrated to its sensitivity.
Revised: A DP mechanism adds noise to the results of f calibrated to its sensitivity.

Original: Here, we provide the deﬁnition of the (real-valued) Gaussian mechanism (GM):
Revised: Here, we provide the deﬁnition of the (real-valued) Gaussian mechanism (GM):

Original: Deﬁnition 3 (Gaussian mechanism) .
Revised: Deﬁnition 3 (Gaussian mechanism) .

Original: The Gaussian mechanism M operates on the results of a query function f : R n ÞÑ R d with sensitivity ∆ over a sensitive database D by outputting f p D q` ξ , where ξ „ N p 0 , σ 2 I d q .
Revised: The Gaussian mechanism M operates on the results of a query function f : R n ÞÑ R d with sensitivity ∆ over a sensitive database D by outputting f p D q` ξ , where ξ „ N p 0 , σ 2 I d q .

Original: Here, I d denotes the identity matrix with d diagonal elements and σ 2 is the variance of Gaussian noise calibrated to ∆ .
Revised: Here, I d denotes the identity matrix with d diagonal elements and σ 2 is the variance of Gaussian noise calibrated to ∆ .

Original: The application of the GM with properly calibrated noise satisﬁes p ε, δ q -DP:
Revised: The application of the GM with properly calibrated noise satisﬁes p ε, δ q -DP:

Original: Deﬁnition 4 ( p ε, δ q -DP) .
Revised: Deﬁnition 4 ( p ε, δ q -DP) .

Original: The randomised mechanism M preserves p (cid:15), δ q -DP if, for all pairs of inputs D and D 1 and all subsets S of M ’s range:
Revised: The randomised mechanism M preserves p (cid:15), δ q -DP if, for all pairs of inputs D and D 1 and all subsets S of M ’s range:

Original: [Equation]
Revised: [Equation]

Original: A number of relaxations have been proposed to characterise the properties of the GM, of which
Revised: A number of relaxations have been proposed to characterise the properties of the GM, of which

Original: R´enyi DP is arguably the most widely employed in DP deep learning frameworks owing to its favourable properties under composition.
Revised: R´enyi DP is arguably the most widely employed in DP deep learning frameworks owing to its favourable properties under composition.

Original: Deﬁnition 5 (R ´ enyi DP) .
Revised: Deﬁnition 5 (R´enyi DP) .

Original: M preserves p α, ρ q -R´enyi-DP (RDP) if, for all pairs of inputs D and D 1 :
Revised: M preserves p α, ρ q -R´enyi-DP (RDP) if, for all pairs of inputs D and D 1 :

Original: [Equation]
Revised: [Equation]

Original: where D α denotes the R´enyi divergence of order α ą 1  .
Revised: where D α denotes the R´enyi divergence of order α ą 1 between M p f p D qq and M p f p D 1 qq .

Revised: At α “ 1 , the divergence is deﬁned by continuity as the Kullback-Leibler divergence.

Revised: We note that the term “between” is an abuse of terminology, as the R ´ enyi divergence is asymmetric.

Revised: In general, we will use D α p p, q q to denote sup t D α p p, q q , D α p q, p qu .

Revised: Gaussian DP (GDP) was introduced as a variant of f-DP by Dong et al.

Revised: (2019) speciﬁcally tailored to the properties of the GM, and provides the tightest possible characterisation of its properties.

Original: ζ -DP
Revised: Relying on statistical hypothesis testing, f-DP interprets DP through a trade-off function between the Type I and Type II statistical errors faced by an adversary trying to determine whether one of the adjacent databases contains the individual or not. GDP is a specialisation of f-DP when the trade-off-function has the form G µ : “ T p N p 0 , 1 q , N p µ, 1 qq . Deﬁnition 6 (Gaussian DP) . M preserves µ -Gaussian DP (GDP) if, for all pairs of adjacent databases D and D 1 :

Revised: [Equation]

Revised: [Equation]

Revised: where T denotes a trade-off function.

Revised: In this case, G µ p α q “ Φ ` Φ ´ 1 p 1 ´

Revised: α q ´ µ q ˘ where α is the Type I statistical error and Φ is the cumulative distribution of the standard, real-valued normal distribution.

Original: In this section we introduce ζ -DP, an extension of DP to complex-valued query functions and mechanisms.
Revised: In this section we introduce ζ -DP, an extension of DP to complex-valued query functions and mechanisms.

Original: ζ -DP generalises real-valued DP and allows the re-use of prior theoretical results and software implementations.
Revised: ζ -DP generalises real-valued DP and allows the re-use of prior theoretical results and software implementations.

Original: 4. T HE COMPLEX G AUSSIAN MECHANISM
Revised: 4. T HE COMPLEX G AUSSIAN MECHANISM

Original: We begin by introducing a variant of the GM suitable to query functions with codomain C .
Revised: We begin by introducing a variant of the GM suitable to query functions with codomain C .

Original: Deﬁnition 6 (Complex Gaussian mechanism) .
Revised: Deﬁnition 7 (Complex Gaussian mechanism) .

Original: The complex Gaussian mechanism M C on f : C n ÞÑ C d outputs f p D q` ξ , where ξ „ N C p 0 , σ 2 I d q and N C p 0 , σ 2 q denotes circularly symmetric complexvalued Gaussian noise with variance σ 2 .
Revised: The complex Gaussian mechanism M C on f : C n ÞÑ C d outputs f p D q` ξ , where ξ „ N C p 0 , σ 2 I d q and N C p 0 , σ 2 q denotes circularly symmetric complexvalued Gaussian noise with variance σ 2 .

Original: Of note, a random variable X „ N C p 0 , σ 2 q can be constructed by independently drawing two random variables A, B from a real-valued normal distribution N p 0 , σ 2 2 q and outputting X “ A ` B i , where i is the imaginary unit. 
Revised: Of note, a random variable X „ N C p 0 , σ 2 q can be constructed by independently drawing two random variables A, B from a real-valued normal distribution N p 0 , σ 2 2 q and outputting X “ A ` B i , where i is the imaginary unit. This property is unique to the complex GM and leverages the fact that the inner product in C n is non-bilinear to add noise scaled by ? 2 ¨ σ to each component of the complex number. Naively “simulating” C n as R 2 n , the latter being equipped with a bi-linear inner product, would instead require the addition of independent Gaussian noise to each component of a vector with resulting scale 2 ¨ σ .

Original: We now state ourtwo main theoretical results:
Revised: We now state our main theoretical results and proof sketches.

Revised: Detailed proofs can be found in

Revised: Appendix A.1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f : C n
Revised: Let f : C n

Original: ÞÑ C
Revised: ÞÑ C

Original: d be a query function with sensitivity ∆ .
Revised: d be a query function with sensitivity ∆ .

Original: Then, M C preserves p ε, δ p ε qq -DP if and only if the following holds @
Revised: Then, M C preserves p ε, δ p ε qq -DP if and only if the following holds @

Original: ε ą 0 , δ P r 0 , 1 s :
Revised: ε ą 0 , δ P r 0 , 1 s :

Original: [Equation]
Revised: [Equation]

Original: where Φ denotes the cumulative distribution function of the standard (real-valued) normal distribution.
Revised: where Φ denotes the cumulative distribution function of the standard (real-valued) normal distribution.

Original: Proof.
Revised: Proof (Sketch).

Original: The claim represents a generalisation of the Analytic Gaussian Mechanism (Balle & Wang, 2018) to M C .

Original: It sufﬁces to show that the magnitude of the privacy-loss random variable Ω is bounded by ε with probability 1 ´ δ .
Revised: It sufﬁces to show that the magnitude of the privacy-loss random variable Ω is bounded by ε with probability 1 ´ δ .

Original: As shown in Dwork & Rothblum (2016) and Balle & Wang (2018), given some ﬁxed output O P C d , Ω is given by:

Original: [Equation]

Original: where log is the natural logarithm, and is distributed as:

Original: [Equation]

Original: As } f p D

Original: [Equation]

Original: 1 qq

Original: [Equation] P R , where H denotes the Hermitian transpose, Ω has a real-valued mean and hence follows a real-valued normal distribution, even when O P C . From here, the proof proceeds identically to the proof to Theorem 8 of (Balle & Wang, 2018).
Revised: This holds, as Ω is distributed as a real-valued normal distribution even when O P C , where O is any output of f , seeing as the mean of Ω is distributed as p f p D q ´ f p D 1 qqp f p D q ´ f p D 1 qq H P R , where H denotes the Hermitian transpose.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let f be deﬁned as above.
Revised: Let f be deﬁned as above.

Original: Then, M C preserves p α, ρ q -RDP if:
Revised: Then, M C preserves p α, ρ q -RDP if:

Original: [Equation]
Revised: [Equation]

Original: We will rely on the following fact about the R ´ enyi divergence of order α between arbitrary distributions: Corollary 1 (Deﬁnition 2 in (Van Erven & Harremos, 2014)) . Let P and Q be two arbitrary distributions deﬁned on a measurable space p X , F q with densities p p x q and q p x q .
Revised: Proof (Sketch).

Original: Then, for α ą 1 : [Equation] In particular, for two  normal distributions with means µ 0  and µ 1  and common variance σ 2 I :
Revised: The R´enyi divergence of order α ą 1 between two circularly symmetric, complexvalued normal distributions with means µ 0 P C n and µ 1 P C n and common variance σ 2 I n is:

Original: [Equation]
Revised: [Equation]

Original: where x¨y denotes the inner product.
Revised: where x¨y denotes the inner product.

Original: We can now prove Theorem 2: Proof.
Revised: Theorem 3.

Original: By Deﬁnition 6 and the additive property of  the Gaussian distribution, the density functions of M C on f p D q and f p D 1 q follow a circularly symmetric complex-valued Gaussian distribution with means f p D q and f p D 1 q and common covariance matrix σ 2 I .
Revised: Let f be deﬁned as above, ∆ be the L 2 -sensitivity of f and σ be the standard deviation of M C ’s noise. Let G µ be the trade-off function of a µ -GDP real-valued GM. Then, if λ “ ∆ { σ , M C preserves µ -GDP if and only if λ ď µ .

Original: By substituting in equation (10):
Revised: Thus, to preserve µ -GDP, one must choose σ 2 such that:

Original: [Equation]
Revised: [Equation]

Original: Hence, to preserve p α, ρ q -RDP, it sufﬁces to choose σ 2 such that
Revised: Proof (Sketch). We will show that T p N C p f p D q , σ q , N C p f p D 1 q , σ qq ě G µ .

Original: [Equation]
Revised: [Equation]

Revised: ) which follows from the fact that } f p D q ´ f p D 1 q} 2 ď ∆ and that G µ is strictly monotonically decreasing in µ .

Original: These ﬁndings allow for a seamless transfer of results which apply to real-valued functions to the complex domain.
Revised: These ﬁndings allow for a seamless transfer of results which apply to real-valued functions to the complex domain.

Original: In particular, they yield the following insights
Revised: In particular, they yield the following insights:

Original: The complex GM inherits all properties of the p ε, δ q and RDP interpretations of the realvalued GM, such as composition and sub-sampling ampliﬁcation.
Revised: The complex GM inherits all properties of the real-valued GM, such as composition and sub-sampling ampliﬁcation, as well as the tight analysis afforded by GDP.

Original: The complex GM, like the real-valued GM, is fully characterised by the sensitivity ∆ and the magnitude of the noise σ . 3.
Revised: The complex GM, like the real-valued GM, is fully characterised by the sensitivity ∆ and the magnitude of the noise σ . 3.

Original: The GM “naturally ﬁts” ζ -DP due to the  convenient properties of the circularly symmetric complex-valued Gaussian distribution.
Revised: The GM “naturally ﬁts” ζ -DP due to the aforementioned convenient properties of the circularly symmetric complex-valued Gaussian distribution.

Original: As a counterexample, a complex-valued Laplace random variable is naturally non-circular in the complex (and multivariate) case, even when constructed from independent distributions (Kotz et al., 2001).
Revised: As an additional counterexample, a complex-valued Laplace random variable is naturally non-circular in the complex (and multivariate) case, even when constructed from independent distributions (Kotz et al., 2001).

Original: Moreover, the utilisation of the L 1 -metric on the output space of f is disadvantageous, as even for scalar (complex) outputs, the L 1 sensitivity can be higher than the L 2 sensitivity.
Revised: Moreover, the utilisation of the L 1 -metric on the output space of f is disadvantageous, as even for scalar (complex) outputs, the L 1 sensitivity can be higher than the Lsensitivity.

Original: Lastly, the utilisation of elliptical  noise is inherently unable to satisfy p ε, 0 q -DP in any dimension ą 1 (Reimherr & Awan, 2019).
Revised: Lastly, the utilisation of elliptical Laplace noise is inherently unable to satisfy p ε, 0 q -DP in any dimension ą 1 (Reimherr & Awan, 2019).

Original: We thus leave the introduction of alternative strategies for obtaining p ε, 0 qDP in the complex-valued setting to future investigation.
Revised: We thus leave the introduction of alternative strategies for obtaining p ε, 0 q -DP in the complex-valued setting to future investigation.

Original: We conclude this section by introducing a modiﬁcation of the DP stochastic gradient descent (DPSGD) algorithm, which will be employed in our experimental evaluation.
Revised: We conclude this section by introducing a modiﬁcation of the DP stochastic gradient descent (DPSGD) algorithm, which will be employed in our experimental evaluation.

Original: 4. ζ -DP-SGD
Revised: 4. ζ -DP-SGD

Original: The DP-SGD algorithm (Abadi et al., 2016) represents an application of the GM to the training of deep neural networks.
Revised: The DP-SGD algorithm (Abadi et al., 2016) represents an application of the GM to the training of deep neural networks.

Original: Using the terminology above, each training step of the neural network (whose loss function, in this setting, represents the query ) leads to the release of a privatised gradient.
Revised: Using the terminology above, each training step of the neural network (which, in this setting, represents the query ) leads to the release of a privatised gradient.

Original: Evidently, the noise magnitude of the GM must be calibrated to the sensitivity of the loss function.
Revised: Evidently, the noise magnitude of the GM must be calibrated to the sensitivity of the loss function.

Original: However, most neural network loss functions have a Lipschitz constant which is too high to preserve DP while maintaining acceptable utility (and –generally –the Lipschitz constant of neural networks is NP-hard to compute (Scaman & Virmaux, 2018)).
Revised: However, most neural network loss functions have a Lipschitz constant which is too high to preserve DP while maintaining acceptable utility (and –generally –the Lipschitz constant of neural networks is NP-hard to compute (Scaman & Virmaux, 2018)).

Original: Thus, DP-SGD (Abadi et al., 2016) artiﬁcially induces a bounded sensitivity condition by clipping the L 2 -norm of the gradient to a pre-deﬁned value.
Revised: Thus, DP-SGD (Abadi et al., 2016) artiﬁcially induces a bounded sensitivity condition by clipping the L 2 -norm of the gradient to a pre-deﬁned value.

Original: Areal-valued loss function  is required for minimisation  as the complex plane –contrary to the real number line– does not admit a natural ordering.
Revised: A loss function with real-valued outputs is required for minimisation (even when the function’s arguments are complex-valued) as the complex plane –contrary to the real number line– does not admit a natural ordering.

Original: Our implementation of the algorithm makes use of Wirtinger (or CR -) calculus
Revised: Our implementation of the algorithm makes use of Wirtinger (or CR -) calculus

Original: (Kreutz-Delgado, 2009) for gradient computations similar to previous works on complex-valued deep learning (Virtue et al., 2017; Boeddeker et al., 2017).
Revised: (Kreutz-Delgado, 2009) for gradient computations similar to previous works on complex-valued deep learning (Virtue et al., 2017; Boeddeker et al., 2017).

Original: This technique, discussed in detail Appendix A.1, provides several beneﬁts: It relaxes the requirement for component functions to be holomorphic (that is, differentiable in the complex sense ), only requiring them to be individually differentiable with respect to their real and imaginary components (differentiable in the real sense ).
Revised: This technique, discussed in detail Appendix A.5, provides several beneﬁts: It relaxes the requirement for component functions to be holomorphic (that is, differentiable in the complex sense ), only requiring them to be individually differentiable with respect to their real and imaginary components (differentiable in the real sense ).

Original: For holomorphic functions C ÞÑ C , CR -derivatives nevertheless recover the correct derivative deﬁnition.
Revised: For holomorphic functions C ÞÑ C , CR -derivatives nevertheless recover the correct derivative deﬁnition.

Original: Thus, CR -derivatives can also be used to compute the global sensitivity in the C ÞÑ C -case via the Lipschitz constant.
Revised: Thus, CR -derivatives can also be used to compute the global sensitivity in the C ÞÑ C -case via the Lipschitz constant.

Original: More importantly, for functions C ÞÑ R , they lead to a correct gradient magnitude calculation, whereas expressing complex-valued functions as vector-valued functions in R 2 n , a technique often employed in complex-valued neural network training (Trabelsi et al., 2017), can incur an undesirable multiplicative sensitivity increase which would diminish the utility of ζ -DP-SGD.
Revised: More importantly, for functions C ÞÑ R , they lead to a correct gradient magnitude calculation, whereas expressing complex-valued functions as vector-valued functions in R 2 n , a technique often employed in complex-valued neural network training (Trabelsi et al., 2017), can incur an undesirable multiplicative sensitivity increase which would diminish the utility of ζ -DP-SGD.

Original: We exemplify this phenomenon and the noise savings CR -calculus can enable in Appendix A.1.
Revised: We exemplify this phenomenon and the noise savings CR -calculus can enable in Appendix A.5 and Section 5.1.

Original: ζ -DP-SGD is presented in Algorithm 1 and relies on a modiﬁcation of the gradient clipping step: we clip the conjugate gradient , which represents the direction of steepest ascent for areal-valued loss function L :
Revised: ζ -DP-SGD is presented in Algorithm 1 and relies on a modiﬁcation of the gradient clipping step: we clip the conjugate gradient , which represents the direction of steepest ascent for a loss function L with real-valued outputs and complex-valued arguments:

Original: [Equation]
Revised: [Equation]

Original: (13) where ` θ 1 , . .
Revised: where ` θ 1 , . .

Original: , θ n ˘ is the conjugate weight vector.
Revised: , θ n ˘ is the conjugate weight vector.

Original: Throughout this section, we present results from the experimental evaluation of ζ -DP-SGD. Details on dataset preparation and training can be found in Appendix A.2. 5. B ENCHMARKING ζ -DP-SGD ON P HASE MNIST
Revised: We remark that, due to the aforementioned properties of the complex GM, the algorithm is compatible with other ﬁrst-order optimisers (such as Adam), as well as other clipping techniques, such as per-layer clipping (McMahan et al., 2018).

Original: The MNIST dataset (LeCun et al., 2010) is widely used as a benchmark dataset in real-valued DP-SGD literature. As a means of comparison, we thus begin our experimental evaluation with results on an adapted, complex-valued version of MNIST, which we term PhaseMNIST. In brief, for
Revised: Moreover, newer methods for analysing mechanism composition, such as the Fourier Accountant (Koskela et al., 2020) can be used.

Original: Algorithm 
Revised: Algorithm 1 ζ -

Original: Require: Database with samples t x 1 , . . .
Revised: Require: Database with samples t x 1 , . . .

Original: , x N u P C n , neural network with loss function L and weight vector θ P C n .
Revised: , x N u P C n , neural network with loss function L and weight vector θ P C n .

Original: Hyperparameters: learning rate η t , noise variance σ 2 , sampling probability
Revised: Hyperparameters: learning rate η t , noise variance σ 2 , sampling probability

Original: R , gradient norm bound B , total steps T .
Revised: R , gradient norm bound B , total steps T .

Original: Initialize θ 0 randomly for t P r T s do
Revised: Initialize θ 0 randomly for t P r T s do

Original: Draw a lot L t with sampling probability R using Poisson or uniform sampling
Revised: Draw a lot L t with sampling probability R using Poisson or uniform sampling

Original: Compute per-sample conjugate gradient
Revised: Compute per-sample conjugate gradient

Original: For each i P L t , compute g t p x
Revised: For each i P L t , compute g t p x

Original: i q Ð ∇L p θ t , x i q
Revised: i q Ð ∇L p θ t , x i q

Original: Clip conjugate gradient q g t p x i q Ð g t p x i q{ max ´ 1 , } g t p x i q} 2 B ¯
Revised: Clip conjugate gradient q g t p x i q Ð g t p x i q{ max ´ 1 , } g t p x i q} 2 B ¯

Original: Apply the Complex Gaussian Mechanism and average r g t Ð 1 L `ř i q g t p x i q ` N C p 0 , σ 2 B 2 I L q ˘
Revised: Apply the Complex Gaussian Mechanism and average r g t Ð 1 L `ř i q g t p x i q ` N C p 0 , σ 2 B 2 I L q ˘

Original: Descend θ t ` 1 Ð θ t ´ η t r g t end for
Revised: Descend θ t ` 1 Ð θ t ´ η t r g t end for

Original: Output updated neural network weight vector θ T and compute the privacy cost.  
Revised: Output updated neural network weight vector θ T and compute the privacy cost.  E XPERIMENTAL EVALUATION

Original: each example of the original MNIST dataset with label L R P t 0 , . . . , 9 u , we obtain the imaginary component by selecting an image with label L I such that L R ` L I “ 9 resulting in an input image arrangement p 0 , 9 q , p 1 , 8 q , . .
Revised: Throughout this section, we present results from the experimental evaluation of ζ -DP-SGD. Details on dataset preparation and training can be found in Appendix A.6. All ε -values are converted (losslessly) from GDP.

Original: , p 9 , 0 q .
Revised: RDP guarantees are slightly looser and provided in Appendix A.2.

Original: Only the label of the real-valued image is used. The results are summarised in Table 1, where we also provide baselines for real-valued MNIST training on the same architecture (with real-valued weights).
Revised: We also provide an experimental evaluation of commonly used complex-valued activation functions for neural network training with ζ -DP-SGD in Appendix A.3. Lastly, we provide additional benchmark experiments demonstrating a complex-valued variant of the MNIST dataset in Appendix A.4.

Original: [Table]
Revised: We begin by demonstrating that the utilisation of the complex Gaussian mechanism and ζ -DP-SGD provides increased model utility for the same privacy budget, even on tasks with real-valued inputs in which no additional information from the input’s imaginary component is available. Moreover, we empirically conﬁrm that, for ζ -DP-SGD, leveraging the Wirtinger/ CR -calculus for gradient computations with respect to complex-valued weights results in a lower sensitivity penalty, and thus improved performance compared to “simulating” complex-valued neural networks with real-valued weights in R 2 n similar to Trabelsi et al. Further theoretical and experimental evaluation of this phenomenon can be found in Appendix A.5.

Original: The complex-weighted neural networks reached 99% accuracy with a low privacy budget consumption of 0 .
Revised: We selected the CIFAR-10 dataset (Krizhevsky et al., 2009), a challenging dataset in the (realvalued) DP setting.

Original: We assume this to be due to the increased amount of information provided by the alignment with a second image as well as the higher entropic capacity of the network due to the complex-valued weights.
Revised: We selected the model architecture recently reported by Papernot et al. (2020), which we equipped with either complex-valued weights (for training in C or with an additional weight matrix (for training in R 2 n ). We trained to the same ε values as Papernot et al.

Original: A similar phenomenon was observed by Scardapane et al.
Revised: (in RDP), but modiﬁed each image such that the real component contained the image and the imaginary component was zero-ﬁlled (for training in C ), or added an additional trailing zero-ﬁlled channel (for training in R 2 n ). The results were identical when the imaginary/second channel components were ﬁlled with the same real-valued image. Table 1 summarises these results.

Original: 5. P RIVACY - PRESERVING ELECTROCARDIOGRAM ABNORMALITY DETECTION ON WEARABLE DEVICES
Revised: Without DP, the increased learning capacity of the complex-valued weights led to improved test set performance compared to real-valued weights, with identical performance for training in C and in R 2 n . When DP was employed, the model trained with ζ -DP-SGD achieved the best performance, outperforming both the real-valued DP model and the R 2 n “simulated” complex-valued DP model. This result underscores the aforementioned theoretical ﬁnding that the complex Gaussian mechanism (contrary to the naive application of a real-valued multivariate Gaussian mechanism in R 2 n ) provides improved performance for the same privacy budget even in tasks with real-valued input data. [Table]

Original:  The advent of wearable devices incorporating electrocardiography (ECG) sensors has provided consumers the ability to detect signs of an abnormal heart rhythm.
Revised: [Table] P RIVACY - PRESERVING ELECTROCARDIOGRAM ABNORMALITY DETECTION ON WEARABLE DEVICES The advent of wearable devices incorporating electrocardiography (ECG) sensors has provided consumers the ability to detect signs of an abnormal heart rhythm.

Original: In this section, we demonstrate the utilisation of a small neural network architecture suitable for deployment, e.g. to a mobile device connected to such a biosensor, to be trained on ECG data from the the China Physiological Signal
Revised: In this section, we demonstrate the utilisation of a small neural network architecture suitable for deployment, e.g. to a mobile device connected to such a biosensor, to be trained on ECG data from the the China Physiological Signal

Original: Challenge (CPSC) 2018 challenge dataset (Liu et al., 2018).
Revised: Challenge (CPSC) 2018 challenge dataset (Liu et al., 2018).

Original: We selected the task of automated
Revised: We selected the task of automated

Original: Left Bundle Branch Block (LBBB) detection, formulated as a binary classiﬁcation task against a normal (sinus) rhythm.
Revised: Left Bundle Branch Block (LBBB) detection, formulated as a binary classiﬁcation task against a normal (sinus) rhythm.

Original: This task is clinically relevant, as the sudden appearance of LBBB can herald acute coronary syndrome which requires urgent attention to avert myocardial infarction.
Revised: This task is clinically relevant, as the sudden appearance of LBBB can herald acute coronary syndrome which requires urgent attention to avert myocardial infarction.

Original: As
Revised: As

Original: ECG data constitutes personal health information, its protection is mandated both legally and ethically.
Revised: ECG data constitutes personal health information, its protection is mandated both legally and ethically.

Original: We utilised ζ -DP-SGD for training a complex-valued neural network on Fourier-transformed
Revised: We utilised ζ -DP-SGD for training a complex-valued neural network on Fourier-transformed

Original: ECG acquisitions.
Revised: ECG acquisitions.

Original: We adopt this strategy as it can beneﬁt from two key properties of the Fourier transform: ECG data can contain high-frequency noise which is irrelevant for diagnosis and can be reduced using Fourier ﬁltering.
Revised: We adopt this strategy as it can beneﬁt from two key properties of the Fourier transform: ECG data can contain high-frequency noise which is irrelevant for diagnosis and can be reduced using Fourier ﬁltering.

Original: Concurrently, this technique compresses the signal, which can drastically reduce the amount of data transferred.
Revised: Concurrently, this technique compresses the signal, which can drastically reduce the amount of data transferred.

Original: Table 2 shows classiﬁcation results and Figureshows exemplary source data.[Table] [Figure] [Table]
Revised: Table 1 shows classiﬁcation results and Figureshows exemplary source data.

Original: [Figure]

Original: 5. D IFFERENTIALLY PRIVATE SPEECH COMMAND CLASSIFICATION FOR VOICE ASSISTANT APPLICATIONS
Revised: 5. D IFFERENTIALLY PRIVATE SPEECH COMMAND CLASSIFICATION FOR VOICE ASSISTANT APPLICATIONS

Original: In recent years, voice assistants have gained popularity in consumer applications such as home speakers, and rely heavily on ML.
Revised: In recent years, voice assistants have gained popularity in consumer applications such as home speakers, and rely heavily on ML.

Original: Recordings collected from users for training speech processing algorithms can be used in impersonation attacks, resulting in successful identity theft (Sweet, 2016) or in acoustic attacks , which trigger unintended behaviour in voice assistants (Yuan et al., 2018;
Revised: Recordings collected from users for training speech processing algorithms can be used in impersonation attacks, resulting in successful identity theft (Sweet, 2016) or in acoustic attacks , which trigger unintended behaviour in voice assistants (Yuan et al., 2018;

Original: Carlini et al., 2016).
Revised: Carlini et al., 2016).

Original: Protecting privacy in this setting is therefore paramount to increase trust and applicability, as well as safeguard both users and systems from adversarial interference.
Revised: Protecting privacy in this setting is therefore paramount to increase trust and applicability, as well as safeguard both users and systems from adversarial interference.

Original: Convolutional neural networks (CNNs) have been demonstrated to yield state-of-the-art performance on spectrogram-transformed audio data (Palanisamy et al., 2020).
Revised: Convolutional neural networks (CNNs) have been demonstrated to yield state-of-the-art performance on spectrogram-transformed audio data (Palanisamy et al., 2020).

Original: However, this and other works (Zhou et al., 2021) typically discard the imaginary components.
Revised: However, this and other works (Zhou et al., 2021) typically discard the imaginary components.

Original: We here experimentally demonstrate the differentially private training of a 2-dimensional CNN directly on the complex spectrogram data.
Revised: We here experimentally demonstrate the DP training of a 2-dimensional CNN directly on the complex spectrogram data.

Original: We utilised a subset of the SpeechCommands dataset (Warden, 2018), speciﬁcally samples from the categories “Yes”, “No”, “Up”, “Down”, “Left”, “Right”, “On”, “Off”, “Stop”, and “Go”, summing up to 8000 examples.
Revised: We utilised a subset of the SpeechCommands dataset (Warden, 2018), speciﬁcally samples from the categories “Yes”, “No”, “Up”, “Down”, “Left”, “Right”, “On”, “Off”, “Stop”, and “Go”, summing up to 8000 examples.

Original: We transformed each waveform signal to a complex-valued 2-D spectrogram and used ζ -DP-SGD to train a complex-valued CNN.
Revised: We transformed each waveform signal to a complex-valued 2-D spectrogram and used ζ -DP-SGD to train a complex-valued CNN.

Original: These results are summarised in Table 3 and Figure 2.[Table]
Revised: These results are summarised in Table 1 and Figure 2.

Original: 5. B ENCHMARKING COMPLEX - VALUED ACTIVATION FUNCTIONS FOR ζ -DP-SGD

Original: A number of specialised activation functions, designed for utilisation with complex-valued neural networks, have been proposed in literature.

Original: To guide practitioner choice in our newly proposed setting of ζ -DP-SGD training, we here provide activation function benchmarks on the SpeechCommands dataset used in the previous section.

Original: Table 4 summarises these results.

Original: We consistently found the inverted Gaussian (iGaussian) activation function to perform best in the ζ -DP-SGD setting.

Original: This may be in part due to its bounded magnitude, thereby recapitulating the effect Papernot et al.

Original: (2020) discuss for real-valued networks,

Original: i

Original: [Table]

Original: e. that bounded activation functions lead to improved performance in DP-SGD.

Original: We leave the further investigation of this ﬁnding to future work.

Original: Activation function

Original: [Table]

Original: Trainable Cardioid (per-feature bias)

Original: [Table]

Original: Trainable ModReLU (per-feature bias)

Original: [Table]

Original: Trainable Cardioid (single bias)

Original: [Table]

Original: 5. MRI RECONSTRUCTION
Revised: 5. MRI RECONSTRUCTION

Original: MRI is an important medical imaging modality and has been studied extensively in the context of deep learning (Akc¸akaya et al., 2019; Hammernik et al., 2018; K¨ustner et al., 2020; Muckley et al., 2020).
Revised: MRI is an important medical imaging modality and has been studied extensively in the context of deep learning (Akc¸akaya et al., 2019; Hammernik et al., 2018; K¨ustner et al., 2020; Muckley et al., 2020).

Original: MRI data is acquired in the so-called k -space.
Revised: MRI data is acquired in the so-called k -space.

Original: Sampling only a subset of k -space data allows for a considerable speed-up in acquisition time, beneﬁting patient comfort and costs, however, typically leads to image artifacts, which reduce the diagnostic quality of the resulting MR images.
Revised: Sampling only a subset of k -space data allows for a considerable speed-up in acquisition time, beneﬁting patient comfort and costs, however, typically leads to image artifacts, which reduce the diagnostic quality of the resulting MR images.

Original: Although neural networks have the ability to produce high-quality reconstructions, their usage for this task has been shown to sometimes lead to the appearance of spurious image content from the fully-sampled reference images the models have been originally trained on (Hammernik et al., 2021;
Revised: Although neural networks have the ability to produce high-quality reconstructions, their usage for this task has been shown to sometimes lead to the appearance of spurious image content from the fully-sampled reference images the models have been originally trained on (Hammernik et al., 2021;

Original: Muckley et al., 2020; Shimron et al., 2021).
Revised: Muckley et al., 2020; Shimron et al., 2021).

Original: DP could counteract such hallucination as it is designed to limit the effect of individual training examples on model training.
Revised: DP could counteract such hallucination as it is designed to limit the effect of individual training examples on model training.

Original: However, this positive effect of DP may be counterbalanced by an unacceptable decrease in the diagnostic suitability of the reconstructed images.
Revised: However, this positive effect of DP may be counterbalanced by an unacceptable decrease in the diagnostic suitability of the reconstructed images.

Original: In this section, we investigate the ramiﬁcations of DP on the quality of MRI reconstructions.
Revised: In this section, we investigate the ramiﬁcations of DP on the quality of MRI reconstructions.

Original: For this purpose, we trained a complex-valued U-Net model architecture on the task of reconstructing single-coil knee MRI images from the fastMRI dataset (Zbontar et al., 2018) using pseudo-random k -space sampling at 4 ˆ acceleration.
Revised: For this purpose, we trained a complex-valued U-Net model architecture on the task of reconstructing single-coil knee MRI images from the fastMRI dataset (Zbontar et al., 2018) using pseudo-random k -space sampling at 4 ˆ acceleration.

Original: We observed a nearly equivalent performance in the non-DP and the ζ -DP-SGD settings, whereby the non-DP model enjoyed a ă 2% performance advantage in all metrics.
Revised: We observed a nearly equivalent performance in the non-DP and the ζ -DP-SGD settings, whereby the non-DP model enjoyed a ă 2% performance advantage in all metrics.

Original: Moreover, to assess the diagnostic suitability of the reconstructed images, we asked a diagnostic radiologist who was blinded to whether or not ζ -DP-SGD was used, to compare the resulting scans.
Revised: Moreover, to assess the diagnostic suitability of the reconstructed images, we asked a diagnostic radiologist who was blinded to whether or not ζ -DP-SGD was used, to compare the resulting scans.

Original: No differences in diagnostic suitability were observed by the expert in any of the reconstructed images.
Revised: No differences in diagnostic suitability were observed by the expert in any of the reconstructed images.

Original: We thus conclude that –at least with respect to image quality– DP can indeed match the non-private training of MRI reconstruction models, even atan ε value of ă 1 ; we intend to investigate its effect on preventing training data hallucination into reconstructed images in future work.
Revised: We thus conclude that –at least with respect to image quality– DP can indeed match the non-private training of MRI reconstruction models, even at ε ! 1 ; we intend to investigate its effect on preventing training data hallucination into reconstructed images in future work.

Original: Results from these experiments are summarised in Table 5 and Figure 3.
Revised: Results from these experiments are summarised in Table 2 and Figure 3.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work presents ζ -DP, an extension of DP to the complex domain and introduces key building blocks of DP  training, namely the complex Gaussian mechanism and ζ -DP-SGD.
Revised: Our work presents ζ -DP, an extension of DP to the complex domain and introduces key building blocks of DP model training, namely the complex Gaussian mechanism and ζ -DP-SGD.

Original: Our experimentson real-world tasks demonstrate that the training of DP complex-valued neural networks is possible with high utility under tight privacy guarantees.
Revised: Our experiments demonstrate that the training of DP complex-valued neural networks is possible with high utility under tight privacy guarantees.

Original: This may –in part– be attributable to  the increased learning capacity of complex-valued models resulting from incorporating two “degrees of freedom” (real and imaginary) per trainable model parameter.
Revised: Our theoretical analysis allows us to leverage the increased learning capacity of complex-weighted neural networks. We thus show improved learning performance compared to scalar real-valued networks and to networks using two real-valued channels to approximate complex numbers, hence, ignoring the speciﬁc relationship between the real and imaginary complex components. Lastly, Wirtinger calculus and Gaussian DP allow us to derive the tight sensitivity estimates and privacy bounds for the complex Gaussian mechanism.

Original: On the ﬂip side, both complex-valued deep learning and DP incur a considerable computational performance penalty.
Revised: We acknowledge some limitations of our work: Both complex-valued deep learning and DP incur a considerable computational performance penalty.

Original: Despite steadily improving complex number support, current deep learning frameworks have not yet implemented a full palette of complex-valued layers and activation functions.
Revised: Despite steadily improving complex number support, current deep learning frameworks have not yet implemented a full palette of complex-valued layers and activation functions.

Original: Moreover, the software framework utilised to computationally realise ζ -DP-SGD in our work relies on multithreading, which suffers from considerable overhead compared to implementations utilising vector instructions and hardware.
Revised: Moreover, the software framework utilised to computationally realise ζ -DP-SGD in our work relies on multithreading, which suffers from considerable overhead compared to implementations utilising vector instructions and/or bespoke hardware.

Original: We discuss the topic of software implementation and provide computational performance benchmarks in Appendices A.3 and A.4.
Revised: We discuss the topic of software implementation and provide computational performance benchmarks in Appendices A.7 and A.8.

Original: Our conclusions highlight a requirement for mature software frameworks able to offer feature and performance parity with their real-valued counterparts. We focus on the Gaussian mechanism and the p ε, δ q -DP/RDP interpretations in this work, believing them to be the most relevant for deep learning applications. The formalisation of other DP mechanisms and interpretations (such as f-DP (Dong et al., 2019)) is a promising future research direction. Such future work could beneﬁt from improved privacy accounting (interestingly also relying on complex numbers, e.g. Zhu et al.
Revised: Our investigation highlights a requirement for mature software frameworks able to offer feature and performance parity with their real-valued counterparts, which we intend to develop and publish as free-and-open-source software in the near future.

Original: to diminish –as much as possible– the utility gap to nonprivate training. In conclusion, we contend that DP and other privacy-enhancing technologies can increase the amount of data available for scientiﬁc study, and are optimistic that our work represents a worthwhile contribution to their implementation in a broad variety of tasks.
Revised: In conclusion, we contend that ζ -DP represents a promising future research direction for complexvalued and even for purely real-valued tasks, and the improved privacy-utility trade-offs resulting from our work represent a worthwhile contribution to the implementation of DP to a broad variety of relevant tasks, which can help to increase the amount of data available for scientiﬁc studies.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: Our work follows all applicable ethical research standards and laws.
Revised: Our work follows all applicable ethical research standards and laws.

Original: All experiments were conducted on publicly available datasets.
Revised: All experiments were conducted on publicly available datasets.

Original: No new data concerning human or animal subjects was generated during our investigation.
Revised: No new data concerning human or animal subjects was generated during our investigation.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We adhere to ICLRs reproducibility standards and include all necessary information to reproduce our experimental and theoretical results either in the main manuscript or in the Appendix.
Revised: We adhere to ICLRs reproducibility standards and include all necessary information to reproduce our experimental and theoretical results either in the main manuscript or in the Appendix.

Original: Theoretical results and proofs can be found in the main manuscript, Section 4 and additional information can be found in Appendix A.1.
Revised: Theoretical results and proofs can be found in the main manuscript, Section 4 and additional information can be found in Appendix A.5.

Original: Details of dataset preparation and analysis can be found in Appendix
Revised: Details of dataset preparation and analysis can be found in Appendix

Original: A.2.
Revised: A.6.

Original: Speciﬁcally, it contains details about the used datasets, their number of samples, all training, validation and test splits, as well as preprocessing steps.
Revised: Speciﬁcally, it contains details about the used datasets, their number of samples, all training, validation and test splits, as well as preprocessing steps.

Original: Furthermore, we describe model architectures, employed optimisers, learning rates and the number of epochs for which models were trained.
Revised: Furthermore, we describe model architectures, employed optimisers, learning rates and the number of epochs for which models were trained.

Original: Lastly, for all DP trainings we provide the noise multipliers, L 2 clipping norms and sampling rates, as well as the δ -values at which the ε -values were calculated.
Revised: Lastly, for all DP trainings we provide the noise multipliers, L 2 clipping norms and sampling rates, as well as the δ -values at which the ε -values were calculated.

Original: Software implementation details and computational resources used can be found in Appendices A.3 and A. 
Revised: Software implementation details and computational resources used can be found in Appendices A.7 and A. A A PPENDIX A. P ROOFS OF THEORETICAL RESULTS

Revised: Proof of Theorem 1.

Original: A A PPENDIX A. W IRTINGER / CR -C ALCULUS
Revised: The claim represents a generalisation of the Analytic Gaussian Mechanism (Balle & Wang, 2018) to M C . It sufﬁces to show that the magnitude of the privacy-loss random variable Ω is bounded by ε with probability 1 ´ δ . As shown in Dwork & Rothblum (2016) and Balle & Wang (2018), given some ﬁxed output O P C d , Ω is given by:

Revised: [Equation]

Revised: where log is the natural logarithm, and is distributed as:

Revised: [Equation]

Revised: As } f p D q ´ f p D 1 q} 22 “ p f p D q ´ f p D 1 qqp f p D q ´ f p D 1 qq H P R , where H denotes the Hermitian transpose, Ω has a real-valued mean and hence follows a real-valued normal distribution, even when

Revised: O P C .

Revised: From here, the proof proceeds identically to the proof to Theorem 8 of (Balle & Wang, 2018).

Revised: For proving Theorem 2, we will rely on the following fact about the R ´ enyi divergence of order α between arbitrary distributions:

Revised: Corollary 1 (Deﬁnition 2 in (Van Erven & Harremos, 2014)) .

Revised: Let P and Q be two arbitrary distributions deﬁned on a measurable space p X , F q with densities p p x q and q p x q .

Revised: Then, for α ą 1 :

Revised: [Equation]

Revised: [Equation]

Revised: In particular, for two normal distributions with means µ 0 and µ 1 and common variance σ 2 I :

Revised: [Equation]

Revised: where x¨y denotes the inner product.

Revised: We can now prove Theorem 2:

Revised: Proof of Theorem 2.

Revised: By Deﬁnition 7 and the additive property of the Gaussian distribution, the density functions of M C on f p D q and f p D 1 q follow a circularly symmetric complex-valued Gaussian distribution with means f p D q and f p D 1 q and common covariance matrix σ 2 I .

Revised: By substituting in equation (17):

Revised: [Equation]

Revised: Hence, to preserve p α, ρ q -RDP, it sufﬁces to choose σ 2 such that σ 2 ě α ∆ 2 { 2 ρ .

Revised: Proof of Theorem 3.

Revised: We recall that the trade-off function between M C p f p D qq and M C p f p D 1 qq is given by:

Revised: [Equation]

Revised: and is strictly monotonically decreasing in µ such that if µ 1 ě µ 2 , then G µ 1 ď G µ 2 .

Revised: To preserve µ -GDP, we require that:

Revised: [Equation]

Revised: intuitively, that the outputs of M C are at least as hard to distinguish as the distributions

Revised: N p 0 , 1 q and N p µ, 1 q given a single draw.

Revised: We have:

Revised: [Equation]

Revised: We require

Revised: [Equation]

Revised: where we have used the fact that } f p D q ´ f p D 1 q} 2 ď ∆ and the strict monotonicity of G µ .

Revised: Thus, to preserve µ -GDP, one must choose σ 2 such that σ 2 ě p ∆ { µ q 2 .

Revised: A. R´ ENYI DP ANALYSIS OF THE MAIN MANUSCRIPT EXPERIMENTS

Revised: Table 3 provides an overview of the differences between the privacy bounds of RDP and GDP for the experiments in the main manuscript.

Revised: We recall that GDP provides a tight analysis of the Gaussian mechanism (Dong et al., 2019) and a lossless conversion between GDP and p ε, δ q -DP is possible, whereas RDP cannot be losslesly converted.

Revised: Of note, we are already using the improved conversion formula from RDP to p ε, δ q -DP shown in Balle et al.

Revised: (2020) instead of the original formula proposed in Mironov (2017).

Revised: [Table]

Revised: [Table]

Revised: A. B ENCHMARKING COMPLEX - VALUED ACTIVATION FUNCTIONS FOR ζ -DP-SGD

Revised: A number of specialised activation functions designed for utilisation with complex-valued neural networks have been proposed in literature.

Revised: To guide practitioner choice in our newly proposed setting of ζ -DP-SGD training, we here provide activation function benchmarks on the SpeechCommands dataset used in Section 5.3 of the main manuscript.

Revised: Table 4 summarises these results.

Revised: We consistently found the inverted Gaussian (iGaussian) activation function to perform best in the ζDP-SGD setting.

Revised: This may be in part due to its bounded magnitude, thereby recapitulating the effect

Revised: Papernot et al.

Revised: (2020) discuss for real-valued networks, i.e. that bounded activation functions lead to improved performance in DP-SGD.

Revised: We leave the further investigation of this ﬁnding to future work.

Revised: A. B ENCHMARKING ζ -DP-SGD ON P HASE MNIST

Revised: The MNIST dataset (LeCun et al., 2010) is widely used as a benchmark dataset in real-valued DPSGD literature.

Revised: We here therefore show the experimental evaluation of an adapted, complex-valued version of MNIST, which we term PhaseMNIST.

Revised: The details of how this dataset can be constructed as well as details on the used model architectures are provided in Appendix A.6.

Revised: In brief, for each example of the original MNIST dataset with label L R P t 0 , . .

Revised: , 9 u , we obtain the imaginary component by selecting an image with label L I such that L R ` L I “ 9 resulting in an input image arrangement p 0 , 9 q , p 1 , 8 q , . . .

Revised: , p 9 , 0 q .

Revised: Only the label of the real-valued image is used.

Revised: The results are summarised in Table 5, where we also provide baselines for real-valued MNIST training on the same architecture (with real-valued weights).

Revised: For simplicity, we have included the RDP and GDP guarantees in the same table.

Revised: [Table]

Revised: [Table]

Revised: As with the CIFAR-10 benchmarks in the main manuscript, we found that ζ -DP-SGD outperformed the real-valued network.

Revised: In this case, additional information is available from the imaginary component of the image in addition to the higher entropic capacity of the network due to the complexvalued weights.

Revised: A similar phenomenon was observed by Scardapane et al.

Original: In this section, we present key results from Wirtinger (or CR -) calculus which are used in our work.
Revised: In this section, we present key results from Wirtinger (or CR -) calculus which are used in our work.

Original: For a detailed treatment, we refer to Kreutz-Delgado (2009).
Revised: For a detailed treatment, we refer to Kreutz-Delgado (2009).

Original: Consider a function f : C ÞÑ C .
Revised: Consider a function f : C ÞÑ C .

Original: As for real-valued functions, the derivative of f at a point z P C can be deﬁned as:
Revised: As for real-valued functions, the derivative of f at a point z P C can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: If this limit is deﬁned for the (inﬁnitely many) series approaching z , f is called complex differentiable (equivalently, differentiable in the complex sense ).
Revised: If this limit is deﬁned for the (inﬁnitely many) series approaching z , f is called complex differentiable (equivalently, differentiable in the complex sense ).

Original: If, in addition, f 1 p z q exists everywhere in the neighbourhood U of z , f is called holomorphic .
Revised: If, in addition, f 1 p z q exists everywhere in the neighbourhood U of z , f is called holomorphic .

Original: It is also possible to write z “ x ` y i and to then express f as two real-valued functions u and v of the variables x and y :
Revised: It is also possible to write z “ x ` y i and to then express f as two real-valued functions u and v of the variables x and y :

Original: [Equation]
Revised: [Equation]

Original:  f 1 can then be written as:
Revised: (25) f 1 can then be written as:

Original: [Equation]
Revised: [Equation]

Original: If this derivative exists at z , f is called differentiable in the real sense .
Revised: If this derivative exists at z , f is called differentiable in the real sense .

Original: This interpretation represents C as R 2 or, more generally, for vector-valued functions, C n as R 2 n .
Revised: This interpretation represents C as R 2 or, more generally, for vector-valued functions, C n as R 2 n .

Original: The Cauchy-Riemann equations state that, for f to be holomorphic, it must satisfy:[Equation]
Revised: The Cauchy-Riemann equations state that, for f to be holomorphic, it must satisfy:

Original: [Equation]
Revised: [Equation]

Original: As discussed above, the complex plane does not admit a natural ordering.
Revised: As discussed above, the complex plane does not admit a natural ordering.

Original: Hence, the minimisation of a complex-valued function is not deﬁned.
Revised: Hence, the minimisation of a complex-valued function is not deﬁned.

Original: Therefore, for complex-valued deep learning, we only consider real-valued (loss-) functions f : C ÞÑ R .
Revised: Therefore, for complex-valued deep learning, we only consider real-valued (loss-) functions f : C ÞÑ R .

Original: By equation (15), v p x, y q i “ 0 .
Revised: By equation (25), v p x, y q i “ 0 .

Original: Thus, by the
Revised: Thus, by the

Original: Cauchy-Riemann equations, such a real-valued function is only holomorphic if:
Revised: Cauchy-Riemann equations, such a real-valued function is only holomorphic if:

Original: [Equation]
Revised: [Equation]

Original: This means that any holomorphic real-valued function must be constant, which invalidates its usefulness for optimisation.
Revised: This means that any holomorphic real-valued function must be constant, which invalidates its usefulness for optimisation.

Original: The Wirtinger/ CR -derivatives provide an alternative interpretation of the
Revised: The Wirtinger/ CR -derivatives provide an alternative interpretation of the

Original: Cauchy-Riemann equations which allows us to consider holomorphicity and differentiability in the real sense separately.
Revised: Cauchy-Riemann equations which allows us to consider holomorphicity and differentiability in the real sense separately.

Original: Thus, they recover the usefulness of interpreting C n as R 2 n while preventing multiplicative penalties on the gradient norm as a consequence of following this interpretation “too closely”.
Revised: Thus, they recover the usefulness of interpreting C n as R 2 n while preventing multiplicative penalties on the gradient norm as a consequence of following this interpretation “too closely”.

Original: We will motivate this somewhat informal notion with an example below.
Revised: We will motivate this somewhat informal notion with an example below.

Original: The
Revised: The

Original: Wirtinger/ CR -derivatives 1 of f are deﬁned as:
Revised: Wirtinger/ CR -derivatives 1 of f are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: An immediate consequence of this deﬁnition is that the Cauchy-Riemann equations can be expressed as:
Revised: An immediate consequence of this deﬁnition is that the Cauchy-Riemann equations can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Therefore, if a function f is holomorphic, BB z corresponds to the derivative in the complex sense (that is, d f d z ) while, if f is differentiable in the real sense, both BB z and BB z are valid (and are conjugates of each other).
Revised: Therefore, if a function f is holomorphic, BB z corresponds to the derivative in the complex sense (that is, d f d z ) while, if f is differentiable in the real sense, both BB z and BB z are valid (and are conjugates of each other).

Original: As stated above, it can be shown that the steepest ascent of f is aligned with BB z .
Revised: As stated above, it can be shown that the steepest ascent of f is aligned with BB z .

Original: In this sense, BB z fulﬁls the role of the ∇ operator for real, scalar-valued loss functions.
Revised: In this sense, BB z fulﬁls the role of the ∇ operator for real, scalar-valued loss functions.

Original: Evidently, compared to the actual gradient of f in the real sense, the following relationship holds:
Revised: Evidently, compared to the actual gradient of f in the real sense, the following relationship holds:

Original: [Equation]
Revised: [Equation]

Original: However, re-deﬁning ∇ f : “ BB z is desirable (and correct, as shown by Brandwood (1983)) .
Revised: However, re-deﬁning ∇ f : “ BB z is desirable (and correct, as shown by Brandwood (1983)) .

Original: We will motivate this requirement with an example: Let f C be a function such that:
Revised: We will motivate this requirement with an example: Let f C be a function such that:

Original: [Equation]
Revised: [Equation]

Original: The Wirtinger/ CR -derivative of f C is B f C B z
Revised: The Wirtinger/ CR -derivative of f C is B f C B z

Original: “ z , whose L 2 -norm is } z } 2 “ | z | “ [Equation] | 2 ` | y | 2 .
Revised: “ z , whose L 2 -norm is } z } 2 “ | z | “ a | x | 2 ` | y | 2 .

Original: The same output can be realised by interpreting f as a function of a real-valued vector a “ p x, y q T [Equation]
Revised: The same output can be realised by interpreting f as a function of a real-valued vector a “ p x, y q T :

Original: [Equation]
Revised: [Equation]

Original: The gradient of f R is ∇ f R “ p 2 x, 2 y q , whose L 2 -norm is } ∇ f R 
Revised: The gradient of f R is ∇ f R “ p 2 x, 2 y q , whose L 2 -norm is } ∇ f R } “

Original: [Equation]
Revised: [Equation]

Original: a | x | 2 ` | y | 2 = 2 | z | .
Revised: a | x | 2 ` | y | 2 = 2 | z | .

Original: This undesirable multiplicative penalty, which would translate to a superﬂuous multiplicative increase in the noise scale of the GM to preserve DP, is a consequence of “ignoring” the connection between real and imaginary part inherent to complex numbers, but not to components of vectors.
Revised: This undesirable multiplicative penalty, which would translate to a superﬂuous multiplicative increase in the noise scale of the GM to preserve DP, is a consequence of “ignoring” the connection between real and imaginary part inherent to complex numbers, but not to components of vectors.

Original: In fact, zz is neither equivalent to z 2 (as would be the case if z P R where @ a P R , a “ a ), nor is it equivalent to x a , a y , a P R 2 , as  complex multiplication lacks the bilinearity inherent to a real inner product space.
Revised: In fact, zz is neither equivalent to z 2 (as would be the case if z P R where @ a P R , a “ a ), nor is it equivalent to x a , a y , a P R 2 , as the complex inner product lacks the bilinearity inherent to its real-valued counterpart.

Original: Both complications are avoided by the redeﬁnition of the Wirtinger/ CR -derivative as the gradient used for optimisation, which prompts its utilisation in our work. 
Revised: Both complications are avoided by the re-deﬁnition of the Wirtinger/ CR -derivative as the gradient used for optimisation, which prompts its utilisation in our work. To exemplify this ﬁnding in the training of neural networks, we conducted the following experiment:

Revised: We generated one thousand points using the function f p z q “ sin p

Original: As a note to practitioners, certain deep learning frameworks silently re-scale the Wirtinger/ CR -gradient by 2 ˆ to avoid user confusion by alower effective learning rate.
Revised: z q ` N C p 0 , 1 q , z P C . We then trained two separate neural networks to ﬁt the sine function and measured the norms of the gradient vectors after 100 training steps (when gradient norms were empirically highest). One network included complex-valued weights ( C ), the other simulated complex-valued weights with an additional weight matrix per layer ( R 2 n ). Both networks utilised the Cardioid activation function and were trained with the Adam optimiser with a learning rate of 0 .

Revised: 01 using a set of identical random seeds over 100 repetitions for 1000 steps.

Revised: We utilised the Mean Squared Error (MSE) loss function given by f p y , ˆy q “ 12 } y ´ ˆy } 2 , p y , ˆy q P C .

Revised: For the C network, we calculated CR -derivatives, while for the R 2 n network, standard gradients were obtained using the PyTorch automatic differentiation system.

Revised: Figure 4 shows the gradient norms of the two networks.

Revised: The R 2 n -network had signiﬁcantly higher average gradient norms (Student t -test p “ 0 . 0043 ) over the 100 repetitions.

Revised: As a note to practitioners, certain deep learning frameworks silently re-scale the Wirtinger/ CRgradient by 2 ˆ to avoid user confusion by a lower effective learning rate.

Original: To ascertain a correct implementation, we therefore recommend examining this behaviour by testing the gradient norm of known functions.
Revised: To ascertain a correct implementation, we therefore recommend examining this behaviour by testing the gradient norm of known functions.

Original: A. D ATASET PREPARATION AND M  
Revised: A. D ATASET PREPARATION AND M ODEL T RAINING A.6. CIFAR-

Original: T RAINING A.2. P HASE MNIST

Original: Dataset construction As described in the main manuscript, PhaseMNIST is intended as a benchmark dataset for complex-valued computer vision tasks and contains images of handwritten digits from 0 to 9 2 .
Revised: Dataset preparation We utilised the CIFAR-10 dataset as described in Krizhevsky et al.

Original: The training set consists of 60000 images and the testing set of 10000 images. For each example of the original MNIST dataset, from which PhaseMNIST is constructed, we performed the following procedure: Let L R P t 0 , . .
Revised: For complex-valued training in C , we either zeroed or copied each image into the imaginary component, while for training in R 2 n , two separate training channels were used for each tensor.

Original: . , 9 u be the label corresponding to the real-valued image. We then constructed the imaginary component by (deterministically) sampling uniformly withreplacement from the set of images whose label L I satisﬁes L R ` L I “ 9 .
Revised: Results in Table 1 are for the zero imaginary/second channel component, but were identical (within ă 1% for all metrics) for the case where the imaginary component was ﬁlled with the same image.

Original: We used the label of the real-valued image as the label of the overall training example.
Revised: Model training For real-valued non-DP training, we used the model described in Papernot et al.

Original: Model training We used a complex-valued model consisting of three fully connected layers withp 784 , 256 , 128 q units and an output layer of 10 units. The Cardioid activation function was used between layers and the Softmax activation function after the output layer.
Revised: (2020) (see Table 2) with the SGD optimiser at a learning rate of 0 .

Original: The model was trained withthe Stochastic Gradient Descent optimiser at a learning rate of 0 .
Revised: 1 with a momentum term of 0 .and a cosine learning rate scheduler.

Original: 1 both for ζ -DP-SGD and for non-private training. The non-private model converged after 3 epochs, whereas the ζ -DP-SGD model required 10 epochs to achieve the same accuracy.
Revised: The hyperbolic tangent (TanH) activation function was used. For complex-valued non-DP training, we utilised the same architecture and activation functions, albeit with complex weights, or with an additional weight matrix per layer.

Original: The noise multiplier was set to 1 . 1 and the L 2 clipping norm to 1 . 0  .
Revised: For DP training, we used an L 2 clipping norm of 1 . 0 , a noise multiplier of 0 . 61 , a sampling rate of 0 .

Original: The ε value was calculated at a δ “ 10 ´ 5 .
Revised: 005 and trained for 20 epochs.

Original: A sampling rate of 0 . 
Revised: The SGD optimiser at a learning rate of 0 . 1 with a momentum term of 0 . 9 and a cosine learning rate scheduler was used. Interestingly, this arrangement led to a higher test set performance than reported by Papernot et al. (2020) in the non-DP setting, however we were unable to reproduce the test set accuracy reported in the real-valued DP setting.

Original: 01 was used for ζ -DP-SGD, and a batch size of 64 for non-DP training.
Revised: For training in R 2 n , we note that multivariate Gaussian noise was used instead of circularly symmetric complex-valued Gaussian noise, and “standard” gradient computations were used instead of Wirtinger derivatives.

Original: A.2. ECG D ATASET
Revised: A.6. ECG D ATASET

Original: Dataset preparation We utilised the China Physiological Signal Challenge 2018 (Liu et al., 2018) dataset for this task.
Revised: Dataset preparation We utilised the China Physiological Signal Challenge 2018 (Liu et al., 2018) dataset for this task.

Original: We used the normal and left bundle branch block classes and channel 3 .
Revised: We used the normal and left bundle branch block classes and channel 3 .

Original: The
Revised: The

Original: ECGs were loaded from the provided Matlab format using the SciPy library and trimmed or padded to a length of 5000 .
Revised: ECGs were loaded from the provided Matlab format using the SciPy library and trimmed or padded to a length of 5000 .

Original: The numpy Fast Fourier Transform implementation was used whereby the signal was pre-trimmed to length 512 .
Revised: The numpy Fast Fourier Transform implementation was used whereby the signal was pre-trimmed to length 512 .

Original: The ﬁnal dataset consisted of 1012 training examples and 113 testing examples.
Revised: The ﬁnal dataset consisted of 1012 training examples and 113 testing examples.

Original: Model training We implemented a complex-valued fully-connected neural network architecture consisting of input/hidden layers with p 512 , 256 , 128 q ( ζ -DP-SGD) units and a single output unit.
Revised: Model training We implemented a complex-valued fully-connected neural network architecture consisting of input/hidden layers with p 512 , 256 , 128 q ( ζ -DP-SGD) units and a single output unit.

Original: The cReLU activation function was used both in the non-DP and the ζ -DP-SGD setting.
Revised: The cReLU activation function was used both in the non-DP and the ζ -DP-SGD setting.

Original: The output layer implemented the magnitude operation followed by a logistic sigmoid activation function.
Revised: The output layer implemented the magnitude operation followed by a logistic sigmoid activation function.

Original: Models were trained using the SGD optimiser at a learning rate of 0 .
Revised: Models were trained using the SGD optimiser at a learning rate of 0 .

Original: 08 with an L 2 regularisation of 5 ˆ 10 ´ 3 for non-DP training and a learning rate of 0 .
Revised: 08 with an L 2 regularisation of 5 ˆ 10 ´ 3 for non-DP training and a learning rate of 0 .

Original: for ζ -DP-SGD training, respectively.
Revised: for ζ -DP-SGD training, respectively.

Original: A batch size of 64 was used for non-private training and a sampling rate of 0 .
Revised: A batch size of 64 was used for non-private training and a sampling rate of 0 .

Original: 063 at a noise multiplier of 5 and an L 2 clipping norm of 0 .
Revised: 063 at a noise multiplier of 5 and an L 2 clipping norm of 0 .

Original: 5 for ζ -DP-SGD.
Revised: 5 for ζ -DP-SGD.

Original: ε was calculated at a δ of 1 1012 1 . 1 « 5 ˆ 10 ´where 1012 is the number of training samples.
Revised: ε was calculated at a δ of [Table] where 1012 is the number of training samples.

Original: Both models were trained for 100 epochs.
Revised: Both models were trained for 100 epochs.

Original: A.2. S PEECH COMMAND CLASSIFICATION DATASET
Revised: A.6. S PEECH COMMAND CLASSIFICATION DATASET

Original: Dataset preparation We used a subset of the SpeechCommands dataset (Warden, 2018) as described above, consisting of 2000 samples each from the categories “Yes”, “No”, “Up”, “Down”, “Left”, “Right”, “On”, “Off”, “Stop”, and “Go”.
Revised: Dataset preparation We used a subset of the SpeechCommands dataset (Warden, 2018) as described above, consisting of 2000 samples each from the categories “Yes”, “No”, “Up”, “Down”, “Left”, “Right”, “On”, “Off”, “Stop”, and “Go”.

Original: Of these, 7200 examples were used as the training test and 800 as the testing set.
Revised: Of these, 7200 examples were used as the training test and 800 as the testing set.

Original: The waveform data was decoded using the TensorFlow library and, where necessary, padded to a length of 16000 samples.
Revised: The waveform data was decoded using the TensorFlow library and, where necessary, padded to a length of 16000 samples.

Original: The TensorFlow implementation of the Short time Fourier Transform function was used with a frame length of 255 and a frame step of 128 .
Revised: The TensorFlow implementation of the Short time Fourier Transform function was used with a frame length of 255 and a frame step of 128 .

Original: Model training For this task, we employed a complex-valued 2D CNN consisting using ﬁlters of size 3 ˆ 3 without zero-padding and a stride of 1 .
Revised: Model training For this task, we employed a complex-valued 2D CNN consisting using ﬁlters of size 3 ˆ 3 without zero-padding and a stride of 1 .

Original: The convolutional layers had p 8 , 16 , 32 , 64 q output ﬁlters, whereby a MaxPooling layer was used between the second layer and the third layer and an adaptive MaxPooling layer after the ﬁnal convolutional layer.
Revised: The convolutional layers had p 8 , 16 , 32 , 64 q output ﬁlters, whereby a MaxPooling layer was used between the second layer and the third layer and an adaptive MaxPooling layer after the ﬁnal convolutional layer.

Original: The convolutional block was followed by a fully connected layer with 64 units and an output layer of 8 units.
Revised: The convolutional block was followed by a fully connected layer with 64 units and an output layer of 8 units.

Original: Both employed the iGaussian activation function.
Revised: Both employed the iGaussian activation function.

Original: The non-DP model was trained at a batch size of 64 for 10 epochs at a learning rate of 0 .
Revised: The non-DP model was trained at a batch size of 64 for 10 epochs at a learning rate of 0 .

Original: 1 using the Stochastic Gradient Descent optimiser, whereas the ζ -DP-SGD network was trained using a sampling rate of 0 . 009 for 5 epochs with the same learning rate and optimiser, a noise multiplier of 1 and an L 2 clipping norm of 2 .
Revised: 1 using the Stochastic Gradient Descent optimiser, whereas the ζ -DP-SGD network was trained using a sampling rate of 0 . 009 for 5 epochs with the same learning rate and optimiser, a noise multiplier of 1 and an L 2 clipping norm of 2 .

Original: We calculated ε at a δ -value of 10 ´ 5 .
Revised: We calculated ε at a δ -value of 10 ´ 5 .

Original: A
Revised: A.6. FAST MRI KNEE DATASET

Original: FAST MRI KNEE DATASET

Original: Dataset preparation We utilised the single coil knee MRI dataset of the fastMRI challenge proposed by Zbontar et al.
Revised: Dataset preparation We utilised the single coil knee MRI dataset of the fastMRI challenge proposed by Zbontar et al.

Original: We used the reference implementation 3 , and employed the default settings using an acceleration rate of 4 ˆ and 8% of densely sampled k -space center lines in the mask.
Revised: We used the reference implementation 2 , and employed the default settings using an acceleration rate of 4 ˆ and 8% of densely sampled k -space center lines in the mask.

Original: Masks are sampled pseudo-randomly during training time.
Revised: Masks are sampled pseudo-randomly during training time.

Original: The dataset offers 34742 train and 7135 validation images.
Revised: The dataset offers 34742 train and 7135 validation images.

Original: Model training We changed the U-Net network to use complex-valued weights and accept complex-valued inputs instead of the magnitude image employed in the original example.
Revised: Model training We changed the U-Net network to use complex-valued weights and accept complex-valued inputs instead of the magnitude image employed in the original example.

Original: We replaced the original ReLU activation functions with CReLU.
Revised: We replaced the original ReLU activation functions with CReLU.

Original: In the DP setting, we used a noise multiplier of 1 . 0 , an L 2 clipping norm of 1 . 0 and a sampling rate of 3 ˆ 10 ´ 5 and calculated the ε at a δ of 10 ´ 5 .
Revised: In the DP setting, we used a noise multiplier of 1 . 0 , an L 2 clipping norm of 1 . 0 and a sampling rate of 3 ˆ 10 ´ 5 and calculated the ε at a δ of 10 ´ 5 .

Original: The learning rate was set to 0 .
Revised: The learning rate was set to 0 .

Original: 001 using the RMSProp optimiser and a stepwise learning rate scheduler.
Revised: 001 using the RMSProp optimiser and a stepwise learning rate scheduler.

Original: We trained both in the non-private and the ζ -DP-SGD setting for 30 epochs and disabled the collection of running statistics in the BatchNormalisation layers to render them compatible with DP.
Revised: We trained both in the non-private and the ζ -DP-SGD setting for 30 epochs and disabled the collection of running statistics in the Batch Normalisation layers to render them compatible with DP (in the utilised library, sample sizes of 1 are used, so Batch Normalisation corresponds to Instance Normalisation in this setting). A.6. P HASE MNIST

Revised: Dataset construction As described in Appendix A.4 above, PhaseMNIST is intended as a benchmark dataset for complex-valued computer vision tasks and contains images of handwritten digits from 0 to 9 3 .

Revised: The training set consists of 60000 images and the testing set of 10000 images.

Revised: For each example of the original MNIST dataset, from which PhaseMNIST is constructed, we performed the following procedure:

Revised: Let L R P t 0 , . .

Original: A. S OFTWARE LIBRARIES AND COMPUTATIONAL RESOURCES USED
Revised: . , 9 u be the label corresponding to the real-valued image. We then constructed the imaginary component by (deterministically) sampling uniformly with replacement from the set of images whose label L I satisﬁes L R ` L I “ 9 . We used the label of the real-valued image as the label of the overall training example. Model training We used a complex-valued model consisting of three fully connected layers with p 784 , 256 , 128 q units and an output layer of 10 units.

Revised: The Cardioid activation function was used between layers and the Softmax activation function after the output layer.

Revised: The model was trained with the Stochastic Gradient Descent optimiser at a learning rate of 0 .

Revised: 1 both for ζ -DP-SGD and for non-private training.

Revised: The non-private model converged after 3 epochs, whereas the ζ -DP-SGD model required 10 epochs to achieve the same accuracy.

Revised: The noise multiplier was set to 1 . 1 and the

Revised: L 2 clipping norm to 1 . 0 .

Revised: The ε value was calculated at a δ “ 10 ´ 5 .

Revised: A sampling rate of 0 .

Revised: 001 was used for ζ -DP-SGD, and a batch size of 64 for non-DP training.

Revised: A. S OFTWARE LIBRARIES AND COMPUTATIONAL RESOURCES USED

Original: Implementations of the DP-SGD algorithm, and –by extension– ζ -DP-SGD require access to perexample gradients.
Revised: Implementations of the DP-SGD algorithm, and –by extension– ζ -DP-SGD require access to perexample gradients.

Original: We utilised the deepee software library (Ziller et al., 2021) to implement ζDP-SGD, as it is compatible with arbitrary neural network architectures, including such containing complex-valued weights.
Revised: We utilised the deepee software library (Ziller et al., 2021) to implement ζDP-SGD, as it is compatible with arbitrary neural network architectures, including such containing complex-valued weights.

Original: We report results using uniform without replacement sampling and using the RDP option provided by deepee .
Revised: We report results using uniform without replacement sampling and using the RDP option provided by deepee .

Original: For complex-valued neural network components, the PyTorch
Revised: For complex-valued neural network components, the PyTorch

Original: Complex library (Chatterjee et al., 2021) with PyTorch 1.9 were used.
Revised: Complex library (Chatterjee et al., 2021) with PyTorch 1.9 were used.

Revised: Standard PyTorch was also used to create the R 2 n model architectures for the experiments in Section 5.1.

Original: TensorFlow 2.4 was used for loading data and the Short Time Fourier Transforms discussed above, but no neural network components were used from this library.
Revised: TensorFlow 2.4 was used for loading data and the Short Time Fourier Transforms discussed above, but no neural network components were used from this library.

Original: Experiments were carried out in Python 3.8.5 on a single workstation computer running Ubuntu Linux 20.04 and equipped with a single NVidia Quadro RTX 8000 GPU, 12 CPU cores and 64 GB of RAM.
Revised: Experiments were carried out in Python 3.8.5 on a single workstation computer running Ubuntu Linux 20.04 and equipped with a single NVidia Quadro RTX 8000 GPU, 12 CPU cores and 64 GB of RAM.

Original: A. C OMPUTATIONAL CONSIDERATIONS
Revised: A. C OMPUTATIONAL CONSIDERATIONS

Original: We conclude by presenting a systematic evaluation of the computational considerations incurred by the utilisation of complex-valued neural networks and by the implementation of ζ -DP-SGD using the above-mentioned libraries.
Revised: We conclude by presenting a systematic evaluation of the computational considerations incurred by the utilisation of complex-valued neural networks and by the implementation of ζ -DP-SGD using the above-mentioned libraries.

Original: Two main sources of computational overhead arise between realvalued and complex-valued neural networks.
Revised: Two main sources of computational overhead arise between realvalued and complex-valued neural networks.

Original: Complex numbers are internally represented as a pair of 32 -bit ﬂoating point numbers.
Revised: Complex numbers are internally represented as a pair of 32 -bit ﬂoating point numbers.

Original: This affects inputs and neural network weights.
Revised: This affects inputs and neural network weights.

Original: Moreover, even though a complex-valued architecture may contain the same number of parameters as its real-valued counterpart, an increased number of computational operations is required in C .
Revised: Moreover, even though a complex-valued architecture may contain the same number of parameters as its real-valued counterpart, an increased number of computational operations is required in C .

Original: For instance, the operation z 2 requires a single multiplication operation in R .
Revised: For instance, scalar multiplication requires a single multiplication operation in R .

Original: However, in C it can require up tomultiplications and 2 additions, depending if vector hardware is used and whether complex ﬂoating point instructions are implemented in the respective framework (e.g., cuDNN ).
Revised: However, in C it can require up to 4 multiplications (although this can be reduced to 3 multiplications (Higham, 1992)). Performance moreover depends on whether vector hardware is used and whether complex ﬂoating point instructions are implemented in the respective framework (e.g., cuDNN ).

Original: Table A.4 shows results for individual matrix multiplication operations and convolutions with real/complex-valued inputs and weight matrices.
Revised: Table A.8 shows results for individual matrix multiplication operations and convolutions with real/complex-valued inputs and weight matrices.

Original: [Table]
Revised: [Table]

Original: ζ -DP-SGD carries additional overhead as it requires per-sample gradients.
Revised: ζ -DP-SGD carries additional overhead as it requires per-sample gradients.

Original: In the utilised deepee framework, this is realised through dispatching one computation thread per example in the minibatch (more precisely, lot ) to perform a forward and backward pass, which incurs substantial overhead compared to pure vectorisation.
Revised: In the utilised deepee framework, this is realised through dispatching one computation thread per example in the minibatch (more precisely, lot ) to perform a forward and backward pass, which incurs substantial overhead compared to pure vectorisation.

Original: These results are shown in Table A.4.
Revised: These results are shown in Table A.8.

Original: Of note, for the non-private models, the computation time includes the forward pass, backward pass, loss gradient calculation (Mean Squared Error against a vector of dimensions p 64 , 1 q ) and weight update (Stochastic Gradient
Revised: Of note, for the non-private models, the computation time includes the forward pass, backward pass, loss gradient calculation (Mean Squared Error against a vector of dimensions p 64 , 1 q ) and weight update (Stochastic Gradient

Original: Descent).
Revised: Descent).

Original: For the ζ -DP-SGD model, the following additional steps occur between the loss gradient calculation and the weight update: gradient clipping, averaging of per-sample gradients, noise application.
Revised: For the ζ -DP-SGD model, the following additional steps occur between the loss gradient calculation and the weight update: gradient clipping, averaging of per-sample gradients, noise application.

Original: Moreover, the deepee framework requires an additional step between the weight update and the subsequent batch.
Revised: Moreover, the deepee framework requires an additional step between the weight update and the subsequent batch.

Original: [Table]
Revised: [Table]