Original: Differentially Private CutMix for Split Learning with Vision Transformer
Revised: Differentially Private CutMix for Split Learning with Vision Transformer

Original: Abstract
Revised: Abstract

Original: Recently, vision transformer (ViT) has started to outpace the conventional CNN in computer vision tasks.
Revised: Recently, vision transformer (ViT) has started to outpace the conventional CNN in computer vision tasks.

Original: Considering privacy-preserving distributed learning with
Revised: Considering privacy-preserving distributed learning with

Original: ViT, federated learning (FL) communicates models, which becomes ill-suited due to ViT’s large model size and computing costs.
Revised: ViT, federated learning (FL) communicates models, which becomes ill-suited due to ViT’s large model size and computing costs.

Original: Split learning (SL) detours this by communicating smashed data at a cut-layer, yet suffers from data privacy leakage and large communication costs caused by high similarity between ViT’s smashed data and input data.
Revised: Split learning (SL) detours this by communicating smashed data at a cut-layer, yet suffers from data privacy leakage and large communication costs caused by high similarity between ViT’s smashed data and input data.

Original: Motivated by this problem, we propose DP-CutMixSL , a differentially private (DP) SL framework by developing DP patch-level randomized CutMix (DP-CutMix) , a novel privacy-preserving inter-client interpolation scheme that replaces randomly selected patches in smashed data.
Revised: Motivated by this problem, we propose DP-CutMixSL , a differentially private (DP) SL framework by developing DP patch-level randomized CutMix (DP-CutMix) , a novel privacy-preserving inter-client interpolation scheme that replaces randomly selected patches in smashed data.

Original: By experiment, we show that DP-CutMixSL not only boosts privacy guarantees and communication efficiency, but also achieves higher accuracy than its Vanilla SL counterpart.
Revised: By experiment, we show that DP-CutMixSL not only boosts privacy guarantees and communication efficiency, but also achieves higher accuracy than its Vanilla SL counterpart.

Original: Theoretically, we analyze that DP-CutMix amplifies Rényi DP (RDP), which is upper-bounded by its Vanilla Mixup counterpart.
Revised: Theoretically, we analyze that DP-CutMix amplifies Rényi DP (RDP), which is upper-bounded by its Vanilla Mixup counterpart.

Original:  Introduction
Revised:  Introduction

Original: Motivation: Privacy-Preserving Distributed ML for ViT Edge devices such as phones, cameras, and e-health wearables generate the sheer amount of fresh data [1].
Revised: Motivation: Privacy-Preserving Distributed ML for ViT Edge devices such as phones, cameras, and e-health wearables generate the sheer amount of fresh data [1].

Original: To exploit these user data for machine learning (ML) without violating data privacy, federated learning (FL) is gaining increasing attention, which keeps raw data locally stored while only exchanging and averaging model parameters across devices [2, 3].
Revised: To exploit these user data for machine learning (ML) without violating data privacy, federated learning (FL) is gaining increasing attention, which keeps raw data locally stored while only exchanging and averaging model parameters across devices [2, 3].

Original: In particular, FL has been notably successful in computer vision tasks with the de facto standard convolutional neural network (CNN) architectures [4, 5].
Revised: In particular, FL has been notably successful in computer vision tasks with the de facto standard convolutional neural network (CNN) architectures [4, 5].

Original: However, recently vision transformer (ViT) has been aggressively taking over the throne of CNN [6], questioning the effectiveness of FL.
Revised: However, recently vision transformer (ViT) has been aggressively taking over the throne of CNN [6], questioning the effectiveness of FL.

Original: In fact, ViT is often larger than CNN, and this bodes ill for FL by imposing excessive energy and communication burdens on devices [7, 8].
Revised: In fact, ViT is often larger than CNN, and this bodes ill for FL by imposing excessive energy and communication burdens on devices [7, 8].

Original: Alternatively, split learning (SL) can cope with large models via model partitioning [9, 10].
Revised: Alternatively, split learning (SL) can cope with large models via model partitioning [9, 10].

Original: In SL, each device locally stores only a tiny fraction of the entire model, and offloads the rest to a parameter server, between which devices exchange their cut-layer forward activations with the server, referred to as smashed data .
Revised: In SL, each device locally stores only a tiny fraction of the entire model, and offloads the rest to a parameter server, between which devices exchange their cut-layer forward activations with the server, referred to as smashed data .

Original: Notwithstanding, ViT commonly lacks pooling and convolutional layers [6, 11, 12], making smashed data similar to their raw data as visualized in Fig.
Revised: Notwithstanding, ViT commonly lacks pooling and convolutional layers [6, 11, 12], making smashed data similar to their raw data as visualized in Fig.

Original: 4 of Appendix A. This may entail huge costs and privacy leakage as opposed to its counterpart SL with CNN [13, 14, 15].
Revised: 4 of Appendix A. This may entail huge costs and privacy leakage as opposed to its counterpart SL with CNN [13, 14, 15].

Original: Mixer ×
Revised: Mixer ×

Original: λ× λ0 0 1 1 1 0 1 1 1 1 01 1 

Revised: λ× λ0 0 1 1 1 0 1 1 1 1 01 1

Original: Server
Revised: Server

Original: Client
Revised: Client

Original: Client0 Shared pseudorandom binary mask generator @Clients, Server mixing ratio 1 Gaussian DP mechanism @Clients 2 σ 2 Cutout smashed data @Clients 0 0 1 1 1 0 1 1 1 1 01 1 1λλ3 DP-CutMix smashed data @Server
Revised: Client0 Shared pseudorandom binary mask generator @Clients, Server mixing ratio 1 Gaussian DP mechanism @Clients 2 σ 2 Cutout smashed data @Clients 0 0 1 1 1 0 1 1 1 1 01 1 1λλ3 DP-CutMix smashed data @Server

Original: DP smahsed
Revised: DP smahsed

Original: [Figure]
Revised: [Figure]

Original: DP
Revised: DP

Original: [Figure]
Revised: [Figure]

Original: Contributions: DP-CutMixSL To address the aforementioned issues, inspired from the patchfied smashed data in ViT [6] and the CutMix technique [16], we propose DP-CutMixSL 2 , a differentially private (DP) SL framework with ViT via patch-level randomized CutMix.
Revised: Contributions: DP-CutMixSL To address the aforementioned issues, inspired from the patchfied smashed data in ViT [6] and the CutMix technique [16], we propose DP-CutMixSL 2 , a differentially private (DP) SL framework with ViT via patch-level randomized CutMix.

Original: As Fig.
Revised: As Fig.

Original: 1 demonstrates, following the Gaussian DP mechanism [17, 18, 19], each device in DP-CutMixSL first injects random
Revised: 1 demonstrates, following the Gaussian DP mechanism [17, 18, 19], each device in DP-CutMixSL first injects random

Original: Gaussian noise into smashed data, followed by punching randomly selected patches, yielding Cutout smashed data as analogous to those of Cutout [20].
Revised: Gaussian noise into smashed data, followed by punching randomly selected patches, yielding Cutout smashed data as analogous to those of Cutout [20].

Original: These Cutout smashed data are uploaded to and put together by the server, resulting in DP-CutMix smashed data that continue feed-forward propagation.
Revised: These Cutout smashed data are uploaded to and put together by the server, resulting in DP-CutMix smashed data that continue feed-forward propagation.

Original: Compared to SL with the Gaussian DP mechanism ( DP-SL ), we theoretically prove that the proposed randomized CutMix operation in DP-CutMixSL amplifies the DP guarantee of smashed data, by up to its upper-bound baseline DP-MixSL obtained by replacing CutMix with
Revised: Compared to SL with the Gaussian DP mechanism ( DP-SL ), we theoretically prove that the proposed randomized CutMix operation in DP-CutMixSL amplifies the DP guarantee of smashed data, by up to its upper-bound baseline DP-MixSL obtained by replacing CutMix with

Original: Mixup [21] that simply superimposes the entire patches from each of different smashed data.
Revised: Mixup [21] that simply superimposes the entire patches from each of different smashed data.

Original: By experiment, we show that DP-CutMixSL achieves the highest accuracy, followed by DP-MixSL and
Revised: By experiment, we show that DP-CutMixSL achieves the highest accuracy, followed by DP-MixSL and

Original: DP-SL.
Revised: DP-SL.

Original: It is worth noting that while most of the existing works apply Cutout and CutMix at pixel levels for intra-dataset interpolations [22, 23], we utilize them at patch levels for privacy-preserving inter-dataset interpolations across different devices, i.e., privacy-preserving distributed ML.
Revised: It is worth noting that while most of the existing works apply Cutout and CutMix at pixel levels for intra-dataset interpolations [22, 23], we utilize them at patch levels for privacy-preserving inter-dataset interpolations across different devices, i.e., privacy-preserving distributed ML.

Original:  DP-CutMixSL: Patch-Level Randomized CutMix Operations for ViT
Revised:  DP-CutMixSL: Patch-Level Randomized CutMix Operations for ViT

Original: The major difference between ViT and CNN can be summarized as follows: i)
Revised: The major difference between ViT and CNN can be summarized as follows: i)

Original: As shown in Fig.of Appendix A, ViT has less feature distortion for the input data of the hidden representation (i.e. smashed data) due to the absence of a pooling layer, ii)
Revised: As shown in Fig.of Appendix A, ViT has less feature distortion for the input data of the hidden representation (i.e. smashed data) due to the absence of a pooling layer, ii)

Original: Due to its own self-attention mechanism driven by embedding process, ViT captures global spatial information whereas CNN focuses on local spatial information, iii)
Revised: Due to its own self-attention mechanism driven by embedding process, ViT captures global spatial information whereas CNN focuses on local spatial information, iii)

Original: The above operations of ViT run at patch-level.
Revised: The above operations of ViT run at patch-level.

Original: At first, i) implies that regularization of the hidden representation in ViT is as efficient as in the input data.
Revised: At first, i) implies that regularization of the hidden representation in ViT is as efficient as in the input data.

Original: Conversely, the mutual information about the input data of the hidden representation is high, leading to data privacy leakage.
Revised: Conversely, the mutual information about the input data of the hidden representation is high, leading to data privacy leakage.

Original: Next, due to the property of ViT to learn global spatial information mentioned in ii), ViT has more robustness to large-scale noise applied to the fraction of the image [25], thereby it is suitable for Cutout [20] or CutMix regularization.
Revised: Next, due to the property of ViT to learn global spatial information mentioned in ii), ViT has more robustness to large-scale noise applied to the fraction of the image [25], thereby it is suitable for Cutout [20] or CutMix regularization.

Original: Finally, iii) suggests the possibility of a patch-scale regularizer.
Revised: Finally, iii) suggests the possibility of a patch-scale regularizer.

Original: Integrating the above yields a common solution, patch-level randomized
Revised: Integrating the above yields a common solution, patch-level randomized

Original: CutMix of hidden representations, short for patch CutMix .
Revised: CutMix of hidden representations, short for patch CutMix .

Original: Let i and C be a subscript for a client and a set of clients, respectively.
Revised: Let i and C be a subscript for a client and a set of clients, respectively.

Original: As observed in Fig.
Revised: As observed in Fig.

Original: 1, a mixer, which may be a third-party entity, first generates random sequences M i with the mixing ratio λ i ∈ [0 , 1] ∀ i ∈ C following a Dirichlet multinomial distribution [26], where (cid:80) i λ i = 1 .
Revised: 1, a mixer, which may be a third-party entity, first generates random sequences M i with the mixing ratio λ i ∈ [0 , 1] ∀ i ∈ C following a Dirichlet multinomial distribution [26], where (cid:80) i λ i = 1 .

Original: For instance, if a smashed data s i consists of N patches, M i is a random binary sequence of length N to control the on-off of each patch, and its non-zero element is ⌈
Revised: For instance, if a smashed data s i consists of N patches, M i is a random binary sequence of length N to control the on-off of each patch, and its non-zero element is ⌈

Original: λ i · N ⌉ .
Revised: λ i · N ⌉ .

Original: Then, the i -th client acquires the Cutout smashed data ¯ s i by masking the smashed data ( ¯ s i =
Revised: Then, the i -th client acquires the Cutout smashed data ¯ s i by masking the smashed data ( ¯ s i =

Original: M i ⊙ s i ), obtained by passing the input data through the lower model segment, via the random sequence downloaded from the mixer.
Revised: M i ⊙ s i ), obtained by passing the input data through the lower model segment, via the random sequence downloaded from the mixer.

Original: When the Cutout smashed data and its label are uploaded to the server,
Revised: When the Cutout smashed data and its label are uploaded to the server,

Original: Algorithm requirements: w =
Revised: Algorithm requirements: w =

Original: [ w c,i , w s ] T ( w c,i : lower model segment, w s : upper model segment)
Revised: [ w c,i , w s ] T ( w c,i : lower model segment, w s : upper model segment)

Original: η : learning rate while w not converged do /*Runs on mixer*/ samples { a 1 , .., a n } ∼ Dir( ¯ α ) generates pseudo random sequences M i for all i ▷ Pseudorandom binary mask generation unicasts M i to i -th client for all i /*Runs on client
Revised: η : learning rate while w not converged do /*Runs on mixer*/ samples { a 1 , .., a n } ∼ Dir( ¯ α ) generates pseudo random sequences M i for all i ▷ Pseudorandom binary mask generation unicasts M i to i -th client for all i /*Runs on client

Original: i ∈ C */ generates smashed data s i by passing input data
Revised: i ∈ C */ generates smashed data s i by passing input data

Original: x i through w c,i produces ¯ s i by masking s i via M i
Revised: x i through w c,i produces ¯ s i by masking s i via M i

Original: ▷ Cutout smashed data produces ¯ s ′ i by applying Gaussian mechanism ▷ DP-Cutout smashed data uploads ¯ s ′
Revised: ▷ Cutout smashed data produces ¯ s ′ i by applying Gaussian mechanism ▷ DP-Cutout smashed data uploads ¯ s ′

Original: i to the server /*Runs on server*/ produces ˜ s ′ i via ¯ s ′
Revised: i to the server /*Runs on server*/ produces ˜ s ′ i via ¯ s ′

Original: i aggregation for all i ▷ DP-CutMix smashed data generates loss (cid:80) i
Revised: i aggregation for all i ▷ DP-CutMix smashed data generates loss (cid:80) i

Original: L i by passing ˜ s ′ i through w s in parallel updates w s via w s ← w s − η · ∇ w s ( (cid:80) i L i )
Revised: L i by passing ˜ s ′ i through w s in parallel updates w s via w s ← w s − η · ∇ w s ( (cid:80) i L i )

Original: ▷ Upper model segment update unicasts i -th cut-layer gradient to i -th client for all i /*Runs on client i ∈ C */ updates w c,i via w c,i ← w c,i − η · ∇ w c,i ( (cid:80) i L i )
Revised: ▷ Upper model segment update unicasts i -th cut-layer gradient to i -th client for all i /*Runs on client i ∈ C */ updates w c,i via w c,i ← w c,i − η · ∇ w c,i ( (cid:80) i L i )

Original: ▷ Lower model segment update end while (a) Raw images.
Revised: ▷ Lower model segment update end while (a) Raw images.

Original: [Figure]
Revised: [Figure]

Original: we assume that a gaussian mechanism is applied to them, generating the following DP-Cutout smashed data and label containing white gaussian noise of N s and N y , respectively:
Revised: we assume that a gaussian mechanism is applied to them, generating the following DP-Cutout smashed data and label containing white gaussian noise of N s and N y , respectively:

Original: [Equation]
Revised: [Equation]

Original: The server aggregates DP-Cutout smashed data from all clients and generates DP-CutMix smashed data in the following way:
Revised: The server aggregates DP-Cutout smashed data from all clients and generates DP-CutMix smashed data in the following way:

Original: [Equation]
Revised: [Equation]

Original: Next, the rest of DP-CutMixSL’s operation, equal to that of Vanilla SL, performing FP & BP on the server-side model follows.
Revised: Next, the rest of DP-CutMixSL’s operation, equal to that of Vanilla SL, performing FP & BP on the server-side model follows.

Original: The said operation of DP-CutMixSL is detailed by the pseudo code of
Revised: The said operation of DP-CutMixSL is detailed by the pseudo code of

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Fig.
Revised: Fig.

Original: 2 also provides image samples of smashed data as well as input data to which the proposed patch CutMix is applied compared to those of Mixup and Vanilla CutMix.
Revised: 2 also provides image samples of smashed data as well as input data to which the proposed patch CutMix is applied compared to those of Mixup and Vanilla CutMix.

Original: As a result, DP-CutMixSL can benefit both in terms of privacy leakage and communication cost, in a way that only fraction of the smashed data is shared to the server, even ejected with gaussian noise.
Revised: As a result, DP-CutMixSL can benefit both in terms of privacy leakage and communication cost, in a way that only fraction of the smashed data is shared to the server, even ejected with gaussian noise.

Original: Note that random sequences used for smashed data masking are mutually exclusive and collectively exhaustive at the patch-level, so that there are no blank patches in DP-CutMix smashed data.
Revised: Note that random sequences used for smashed data masking are mutually exclusive and collectively exhaustive at the patch-level, so that there are no blank patches in DP-CutMix smashed data.

Original:  DP-CutMixSL: Differential Privacy Analysis
Revised:  DP-CutMixSL: Differential Privacy Analysis

Original: Let D =
Revised: Let D =

Original: ( s 1 , y 1 ) , .., ( s n , y n ) } be a set consisting of n clients’ pairs of smashed data s i ∈ R N × P 2 × C and
Revised: ( s 1 , y 1 ) , .., ( s n , y n ) } be a set consisting of n clients’ pairs of smashed data s i ∈ R N × P 2 × C and

Original: the corresponding label y ∈ R L is a one-hot encoding vector, where P denotes the size of patch, respectively.
Revised: the corresponding label y ∈ R L is a one-hot encoding vector, where P denotes the size of patch, respectively.

Original: We assume that the each element of the smashed data and ground-truth label is upper bounded as follows: s i ∈ [0 , ∆] D s and y i ∈ [0 , 1] D y , where D s = NP 2 C and D y =
Revised: We assume that the each element of the smashed data and ground-truth label is upper bounded as follows: s i ∈ [0 , ∆] D s and y i ∈ [0 , 1] D y , where D s = NP 2 C and D y =

Original: L .
Revised: L .

Original: In addition, λ i is the mixing ratio of the i -th client, and N s and N y are white gaussian noise with dimensions D s and D y , respectively, i.e., N s ∼ N (0 , σ 2 s
Revised: In addition, λ i is the mixing ratio of the i -th client, and N s and N y are white gaussian noise with dimensions D s and D y , respectively, i.e., N s ∼ N (0 , σ 2 s

Original: I D s )
Revised: I D s )

Original: and N y ∼ N (0 , σ 2 y
Revised: and N y ∼ N (0 , σ 2 y

Original: I D y )
Revised: I D y )

Original: for some ( σ s , σ y ) .
Revised: for some ( σ s , σ y ) .

Original: Then, we derive the Rényi differential privacy (RDP) [19] of the proposed DP-CutMixSL, which is compared with those of DP-SL and DP-MixSL as follows.
Revised: Then, we derive the Rényi differential privacy (RDP) [19] of the proposed DP-CutMixSL, which is compared with those of DP-SL and DP-MixSL as follows.

Original: Theorem 1.
Revised: Theorem 1.

Original: For a given order α , the RDP privacy budgets ϵ o ( α ) , ϵ Mix ( α ) , and ϵ CutMix ( α ) of
Revised: For a given order α , the RDP privacy budgets ϵ o ( α ) , ϵ Mix ( α ) , and ϵ CutMix ( α ) of

Original: DP-SL, DP-MixSL and DP-CutMixSL satisfy ϵ Mix ( α ) ≤ ϵ CutMix ( α ) ≤
Revised: DP-SL, DP-MixSL and DP-CutMixSL satisfy ϵ Mix ( α ) ≤ ϵ CutMix ( α ) ≤

Original: ϵ o ( α ) where:
Revised: ϵ o ( α ) where:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Sketch of Proof.
Revised: Sketch of Proof.

Original: For each technique, we derive its output representation, followed by calculating the RDP bound using the output via the Rényi divergence formula for a multivariate Gaussian distribution [18].
Revised: For each technique, we derive its output representation, followed by calculating the RDP bound using the output via the Rényi divergence formula for a multivariate Gaussian distribution [18].

Original: Applying this to both the smashed data and the label and combining them via the sequential composition rule completes the proof.
Revised: Applying this to both the smashed data and the label and combining them via the sequential composition rule completes the proof.

Original: The details are deferred to Appendix B.
Revised: The details are deferred to Appendix B.

Original: ■
Revised: ■

Original: Since λ i ∈ [0 , 1] , DP-MixSL achieves the highest RDP guarantee (i.e., tightest RDP bound) compared to DP-CutMixSL and DP-SL, with the help of the inherent distortion property of interpolations [27, 28, 29].
Revised: Since λ i ∈ [0 , 1] , DP-MixSL achieves the highest RDP guarantee (i.e., tightest RDP bound) compared to DP-CutMixSL and DP-SL, with the help of the inherent distortion property of interpolations [27, 28, 29].

Original: It is worth noting that the case only when max i ∈ C
Revised: It is worth noting that the case only when max i ∈ C

Original: λ i = 1 , i.e., a single client scenario with | C | = 1 , the equality conditions ϵ Mix ( α ) =
Revised: λ i = 1 , i.e., a single client scenario with | C | = 1 , the equality conditions ϵ Mix ( α ) =

Original: ϵ CutMix ( α ) =
Revised: ϵ CutMix ( α ) =

Original: ϵ SL ( α ) hold.
Revised: ϵ o ( α ) hold.

Original: In other words, none of equality does not hold for multi clients.
Revised: In other words, none of equality does not hold for multi clients.

Original: Note here that we focus only on the RDP guarantees of smashed data.
Revised: Note here that we focus only on the RDP guarantees of smashed data.

Original: Smashed data are vulnerable to reconstruction attacks [30], threatening the privacy of raw data, which is discussed in Appendix C.
Revised: Smashed data are vulnerable to reconstruction attacks [30], threatening the privacy of raw data, which is discussed in Appendix C.

Original:  Numerical Evaluation
Revised:  Numerical Evaluation

Original: In this section, we measure the accuracy, RDP bound ( ϵ ), and scalability of DP-CutMixSL compared to those of SplitFed [14], DP-SL, DP-MixSL, and etc.
Revised: In this section, we measure the accuracy, RDP bound ( ϵ ), and scalability of DP-CutMixSL compared to those of SplitFed [14], DP-SL, DP-MixSL, and etc.

Original: In Table 1, both the CIFAR-10 dataset [31] and the Fashion-MNIST dataset [32] are utilized under three types of models: ViT-tiny [33], PiT-tiny [34], and VGG-16 [35].
Revised: In Table 1, both the CIFAR-10 dataset [31] and the Fashion-MNIST dataset [32] are utilized under three types of models: ViT-tiny [33], PiT-tiny [34], and VGG-16 [35].

Original: Here, PiT is a transformer structure equipped with a pooling layer and is a model between ViT and CNN.
Revised: Here, PiT is a transformer structure equipped with a pooling layer and is a model between ViT and CNN.

Original: For all SL algorithms, we assume that the cut-layer is located after embedding process.
Revised: For all SL algorithms, we assume that the cut-layer is located after embedding process.

Original: Other parameters especially for RDP calculation are as follows: patch size
Revised: Other parameters especially for RDP calculation are as follows: patch size

Original: N = 64 , D s = 20 , D y
Revised: N = 64 , D s = 20 , D y

Original: = 10 , ∆ = 0 .
Revised: = 10 , ∆ = 0 .

Original: 05 , λ i = 1 /n ∀ i (uniform), and RDP parameter α = 2 .
Revised: 2 , λ i = 1 /n ∀ i (uniform), and RDP parameter α = 2 .

Original: Table 1 shows the top-1 accuracy for several SL methods including the proposed CutMixSL in a noiseless environment.
Revised: Table 1 shows the top-1 accuracy for several SL methods including the proposed CutMixSL in a noiseless environment.

Original: As seen at Table 1, except for one case, where SL w. Mixup is used with CIFAR-10 dataset and VGG-16 model, the CutMixSL outperforms other state-of-the-art SL algorithms in terms of top-1 accuracy.
Revised: As seen at Table 1, except for one case, where SL w. Mixup is used with CIFAR-10 dataset and VGG-16 model, the CutMixSL outperforms other state-of-the-art SL algorithms in terms of top-1 accuracy.

Original: This is rooted in the difference between ViT and CNN mentioned in Sec.
Revised: This is rooted in the difference between ViT and CNN mentioned in Sec.

Original: When learning spatial information, ViT focuses on globality due to the selfattention mechanism, whereas CNN focuses on locality.
Revised: When learning spatial information, ViT focuses on globality due to the selfattention mechanism, whereas CNN focuses on locality.

Original: Hence, a patch CutMix in which certain patches are replaced by patches of other smashed data may cause significant information loss in CNN.
Revised: Hence, a patch CutMix in which certain patches are replaced by patches of other smashed data may cause significant information loss in CNN.

Original: On the other hand, interpolation such as mixup is less likely to yield large information loss, thereby
Revised: On the other hand, interpolation such as mixup is less likely to yield large information loss, thereby

Original: CNN and ViT are suitable for mixup and patch CutMix, respectively.
Revised: CNN and ViT are suitable for mixup and patch CutMix, respectively.

Original: Comparing CutMixSL and SL w. vanilla CutMix in Table 1 proves that patch-level random punching is more efficient than bounding box-based process.
Revised: Comparing CutMixSL and SL w. Vanilla CutMix in Table 1 proves that patch-level random punching is more efficient than bounding box-based process.

Original: This is because the random punching method is a regularizer well suited to ViT, where all operations operate at patch-level, unlike vanilla CutMix’s bounding box where the size is regardless of patch size.
Revised: This is because the random punching method is a regularizer well suited to ViT, where all operations operate at patch-level, unlike Vanilla CutMix’s bounding box where the size is regardless of patch size.

Original: From a dropout [36] perspective, the random punching method of CutMixSL is more similar to dropout than mixup or vanilla CutMix, leading to accuracy gains. 
Revised: From a dropout [36] perspective, the random punching method of CutMixSL is more similar to dropout than mixup or Vanilla CutMix, leading to accuracy gains. [Table] [Figure]

Original: Fig.
Revised: Fig.

Original: 3a shows the effect of noise variance on accuracy and ϵ .
Revised: 3a shows the effect of noise variance on accuracy and ϵ .

Original: In terms of accuracy, DP-CutMixSL is the best, except when the noise variance is 16 / 255 .
Revised: In terms of accuracy, DP-CutMixSL is the best, except when the noise variance is 16 / 255 .

Original: Compared to DP-MixSL, DP-CutMixSL has[Table] [Figure] superior performance in all cases.
Revised: Compared to DP-MixSL, DP-CutMixSL has superior performance in all cases.

Original: Looking at ϵ , however, DP-CutMixSL has a tighter RDP bound compared to DP-SL, but has a larger ϵ in comparison with DP-MixSL, showing the accuracy-privacy trade-off .
Revised: Looking at ϵ , however, DP-CutMixSL has a tighter RDP bound compared to DP-SL, but has a larger ϵ in comparison with DP-MixSL, showing the accuracy-privacy trade-off .

Original: In Fig.
Revised: In Fig.

Original: 3b, the size of a mixing group, a set of clients taking a mixup or CutMix, varies both in DP-CutMixSL and DP-MixSL.
Revised: 3b, the size of a mixing group, a set of clients taking a mixup or CutMix, varies both in DP-CutMixSL and DP-MixSL.

Original: In both DP-CutMixSL and DP-MixSL, the accuracy and ϵ decrease as the mixing group size increases, also resulting in the accuracy-privacy trade-off.
Revised: In both DP-CutMixSL and DP-MixSL, the accuracy and ϵ decrease as the mixing group size increases, also resulting in the accuracy-privacy trade-off.

Original: The decrease in ϵ according to the mixing group size can be explained by the "Hiding in the crowd" effect [37], and the decrease in accuracy is also explained in connection with the information loss mentioned above.
Revised: The decrease in ϵ according to the mixing group size can be explained by the "Hiding in the crowd" effect [37], and the decrease in accuracy is also explained in connection with the information loss mentioned above.

Original: Fig.
Revised: Fig.

Original: 3c shows the curve of scalability, in terms of accuracy increase according to the number of clients.
Revised: 3c shows the curve of scalability, in terms of accuracy increase according to the number of clients.

Original: In
Revised: In

Original: Fig.
Revised: Fig.

Original: 3c, all SL techniques including CutMixSL guarantee scalability when the client increases fromto 10, and the accuracy of CutMixSFL, which introduced SplitFed’s weight averaging of lower model segment to CutMixSL, is further improved.
Revised: 3c, all SL techniques including CutMixSL guarantee scalability when the client increases fromto 10, and the accuracy of CutMixSFL, which introduced SplitFed’s weight averaging of lower model segment to CutMixSL, is further improved.

Original:  Concluding Remarks
Revised:  Concluding Remarks

Original: In this work, we proposed DP-CutMixSL for privacy-preserving distributed ML for ViT by exploiting
Revised: In this work, we proposed DP-CutMixSL for privacy-preserving distributed ML for ViT by exploiting

Original: CutMix for inter-dataset interpolations.
Revised: CutMix for inter-dataset interpolations.

Original: We theoretically analyzed its DP guarantee, and numerically showed its achieving the highest accuracy compared to two baselines, DP-SL and DP-MixSL.
Revised: We theoretically analyzed its DP guarantee, and numerically showed its achieving the highest accuracy compared to two baselines, DP-SL and DP-MixSL.

Original: While we focus only on generating a single CutMix output for two or more inputs, it is possible to generate multiple outputs by changing the mixing ratio for interpolations with finer resolution as done in [38], which could be an interesting topic for future study.
Revised: While we focus only on generating a single CutMix output for two or more inputs, it is possible to generate multiple outputs by changing the mixing ratio for interpolations with finer resolution as done in [38], which could be an interesting topic for future study.

Original: Furthermore, based on the simulation results showing the effectiveness of the proposed method even in the presence of several pooling and convolutional layers, it is worth investigating other patchified architectures in different domains for future research.
Revised: Furthermore, based on the simulation results showing the effectiveness of the proposed method even in the presence of several pooling and convolutional layers, it is worth investigating other patchified architectures in different domains for future research.

Revised: Lastly, for label privacy leakage resulting from gradient in BP, it can be prevented by

Revised: GradPerturb of [39], compromising accuracy, but the ordering is expected to be the same.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)
Revised: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)

Original: (No. 2021-0-02208, SubTHz Augmented Routing and Transmission for 6G).Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-02208, Sub-THz Augmented Routing and Transmission for 6G).
Revised: (No. 2021-0-02208, Sub-

Original: B RDP Analysis
Revised: B RDP Analysis

Original: Before going further, the definition of RDP is as follows:
Revised: Before going further, the definition of RDP is as follows:

Original: Definition B.1 ( ( α, ϵ ) -RDP [19]) .
Revised: Definition B.1 ( ( α, ϵ ) -RDP [19]) .

Original: A randomized mechanism f : D → R is said to have ϵ
Revised: A randomized mechanism f : D → R is said to have ϵ

Original: -R ´e nyi differential privacy of order α , or ( α, ϵ )-RDP for short, if for any adjacent D,
Revised: -R ´e nyi differential privacy of order α , or ( α, ϵ )-RDP for short, if for any adjacent D,

Original: D ′ ∈ D it holds that
Revised: D ′ ∈ D it holds that

Original: [Equation]
Revised: [Equation]

Original: A strong privacy guarantee implies that one cannot distinguish whether D or D ′ was used to produce an outcome of mechanism.
Revised: A strong privacy guarantee implies that one cannot distinguish whether D or D ′ was used to produce an outcome of mechanism.

Original: B.1 DP-SL
Revised: B.1 DP-SL

Original: Here, our baseline DP mechanism is Gaussian mechanism.
Revised: Here, our baseline DP mechanism is Gaussian mechanism.

Original: Then, the output of DP-SL, which are the smashed data and its label to which gaussian noise is applied, respectively, is as follows:
Revised: Then, the output of DP-SL, which are the smashed data and its label to which gaussian noise is applied, respectively, is as follows:

Original: [Equation]
Revised: [Equation]

Original: where N s ∼ N (0 , σ 2 s
Revised: where N s ∼ N (0 , σ 2 s

Original: I D s )
Revised: I D s )

Original: and N y ∼ N (0 , σ 2 y
Revised: and N y ∼ N (0 , σ 2 y

Original: I D y )
Revised: I D y )

Original: for some ( σ s , σ y ) .
Revised: for some ( σ s , σ y ) .

Original: By using the Definition B.1 and Rényi divergence formula from [18], the RDP bound of gaussian mechanism M for DP-SL, ϵ o ( α ) can be expressed as:
Revised: By using the Definition B.1 and Rényi divergence formula from [18], the RDP bound of gaussian mechanism M for DP-SL, ϵ o ( α ) can be expressed as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where M ( D ) ∼ N ( µ D X , σ 2 X ) and M ( D ′ ) ∼ N ( µ D ′ X , σ 2 X ) .
Revised: where M ( D ) ∼ N ( µ D X , σ 2 X ) and M ( D ′ ) ∼ N ( µ D ′ X , σ 2 X ) .

Original: Let s D
Revised: Let s D

Original: ( i,k ) denote the k -th element of smashed data s D i in dataset D , where s D i =
Revised: ( i,k ) denote the k -th element of smashed data s D i in dataset D , where s D i =

Original: [ s D ( i, 1) , ..., s D ( i,D s ) ] .
Revised: [ s D ( i, 1) , ..., s D ( i,D s ) ] .

Original: Then, s ′D i obtained by passing s D i through (8) follows a gaussian distribution with mean s D i and variance σ 2 s (In element perspective, s ′D ( i,k ) ∼ N ( s D ( i,k ) , σ 2 s ) for k ∈
Revised: Then, s ′D i obtained by passing s D i through (8) follows a gaussian distribution with mean s D i and variance σ 2 s (In element perspective, s ′D ( i,k ) ∼ N ( s D ( i,k ) , σ 2 s ) for k ∈

Original: [ D s ] ).
Revised: [ D s ] ).

Original: For two sets of smashed data D and D ′ where only the i ′ -th smashed data is different, (10) becomes:
Revised: For two sets of smashed data D and D ′ where only the i ′ -th smashed data is different, (10) becomes:

Original: [Equation]
Revised: [Equation]

Original: Considering the element-wise upper bound of DP-SL’s smashed data yields the following formula:
Revised: Considering the element-wise upper bound of DP-SL’s smashed data yields the following formula:

Original: [Equation]
Revised: [Equation]

Original: Therefore, gaussian mechanism for smashed data in DP-SL is ( α, ϵ o ) -RDP, where
Revised: Therefore, gaussian mechanism for smashed data in DP-SL is ( α, ϵ o ) -RDP, where

Original: [Equation]
Revised: [Equation]

Original: Likewise, RDP bound of groud-truth label in DP-SL can be calculated, yielding¯ ϵ o ( α ) as below:
Revised: Likewise, RDP bound of groud-truth label in DP-SL can be calculated, yielding ϵ o ( α ) as below:

Original: [Equation]
Revised: [Equation]

Original: B.2 DP-MixSL
Revised: B.2 DP-MixSL

Original: The output of DP-MixSL can be expressed as ˆ s = (cid:80) ni
Revised: The output of DP-MixSL can be expressed as ˆ s = (cid:80) ni

Original: =

Revised: =

Original: λ i s ′
Revised: λ i s ′

Original: i , where { λ 1 , ..., λ n } is a set of mixing ratio of each client’s smashed data.
Revised: i , where { λ 1 , ..., λ n } is a set of mixing ratio of each client’s smashed data.

Original: Also, DP-MixSL executes mixup operation similar to its corresponding label.
Revised: Also, DP-MixSL executes mixup operation similar to its corresponding label.

Original: Then, for two adjacent datasets D and D ′ , (10) becomes:
Revised: Then, for two adjacent datasets D and D ′ , (10) becomes:

Original: [Equation]
Revised: [Equation]

Original: This bound can be maximized when the i ′ -th
Revised: This bound can be maximized when the i ′ -th

Original: mixing ratio is the largest value of λ i for all i ∈ C .
Revised: mixing ratio is the largest value of λ i for all i ∈ C .

Original: In this case, from 15, we have
Revised: In this case, from 15, we have

Original: [Equation]
Revised: [Equation]

Original: By applying the same process above to the label, we have
Revised: By applying the same process above to the label, we have

Original: [Equation]
Revised: [Equation]

Original: B.3 DP-CutMixSL
Revised: B.3 DP-CutMixSL

Original: The output of DP-CutMixSL can be represented by ˜ s = (cid:80) ni
Revised: The output of DP-CutMixSL can be represented by ˜ s = (cid:80) ni

Original: =1 M i ⊙ s
Revised: =1 M i ⊙ s

Original: ′ i , whereas its operation on label is the same as in DP-MixSL.
Revised: ′ i , whereas its operation on label is the same as in DP-MixSL.

Original: For two adjacent datasets with only one smashed data different,
Revised: For two adjacent datasets with only one smashed data different,

Original: DP-CutMixSL only needs to calculate bounds only for the element in which the smashed data is masked, in contrast to DP-MixSL, where the smashed data is melted in the whole element.
Revised: DP-CutMixSL only needs to calculate bounds only for the element in which the smashed data is masked, in contrast to DP-MixSL, where the smashed data is melted in the whole element.

Original: That is, assuming that the number of 1 elements included in the i ′ -th
Revised: That is, assuming that the number of 1 elements included in the i ′ -th

Original: mask
Revised: mask

Original: is N i ′ , the following inequality is derived from (10):
Revised: is N i ′ , the following inequality is derived from (10):

Original: [Equation]
Revised: [Equation]

Original: (18) has an upper bound as shown below when λ i ′ is the maximum among λ
Revised: 19) (19) has an upper bound as shown below when λ i ′ is the maximum among λ

Original: i ∀ i :
Revised: i ∀ i :

Original: [Equation]
Revised: [Equation]

Original: Then, we have¯ ϵ CutMix ( α ) for DP-CutMixSL mechanism as follows:
Revised: Then, we have ϵ CutMix ( α ) for DP-CutMixSL mechanism as follows:

Original: [Equation]

Original: If we take ϵ only from the data aspect, ϵ CutMix ( α ) becomes:

Original: [Equation]
Revised: [Equation]

Original: leading to inequality in Theorem 1.

Original: [Table]
Revised: [Table]

Original: C Robustness to Reconstruction Attack
Revised: C Robustness to Reconstruction Attack

Original: Table 2 shows loss between raw data and reconstructed data generated from different types of mixing methods or datasets.
Revised: Table 2 shows loss between raw data and reconstructed data generated from different types of mixing methods or datasets.

Original: To restore raw data from reconstructed data, we utilize a decoder model, comprised of two convolutional layers with additional interpolation methods to adaptively match the dimension to the aimed data size.
Revised: To restore raw data from reconstructed data, we utilize a decoder model, comprised of two convolutional layers with additional interpolation methods to adaptively match the dimension to the aimed data size.

Original: For comparison, we train the decoder model with training datasets of two different sizes.
Revised: For comparison, we train the decoder model with training datasets of two different sizes.

Original: As a result, regardless of the training dataset size, the reconstruction loss was large in the following order: Cutout, patch CutMix, Mixup, and smashed data.
Revised: As a result, regardless of the training dataset size, the reconstruction loss was large in the following order: Cutout, patch CutMix, Mixup, and smashed data.

Original: In other words, it has robustness against reconstruction attacks in that order.
Revised: In other words, it has robustness against reconstruction attacks in that order.

Original: Except for smashed data, Mixup is most vulnerable to reconstruction attack, because information leakage occurs over the entire area even though linear interpolation is taken.
Revised: Except for smashed data, Mixup is most vulnerable to reconstruction attack, because information leakage occurs over the entire area even though linear interpolation is taken.

Original: The proposed patch CutMix has relatively robustness since it can inject large-size noise into the local information necessary for reconstruction, thanks to its inherent masking, but is upper bounded on the Cutout, which discards a part of the image corresponding to the mask.
Revised: The proposed patch CutMix has relatively robustness since it can inject large-size noise into the local information necessary for reconstruction, thanks to its inherent masking, but is upper bounded on the Cutout, which discards a part of the image corresponding to the mask.

Original: Finally, the larger the training dataset size, the better the decoder model is trained, which reduces the overall reconstruction loss.
Revised: Finally, the larger the training dataset size, the better the decoder model is trained, which reduces the overall reconstruction loss.