Original: P ANNING FOR G OLD IN F EDERATED L EARNING : T ARGETED T EXT E XTRACTION UNDER A RBITRARILY L ARGE -S CALE A GGREGATION
Revised: P ANNING FOR G OLD IN F EDERATED L EARNING : T ARGETED T EXT E XTRACTION UNDER A RBITRARILY L ARGE -S CALE A GGREGATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As federated learning (FL) matures, privacy attacks against FL systems in turn become more numerous and complex.
Revised: As federated learning (FL) matures, privacy attacks against FL systems in turn become more numerous and complex.

Original: Attacks on language models have progressed from recovering single sentences in simple classification tasks to recovering larger parts of user data.
Revised: Attacks on language models have progressed from recovering single sentences in simple classification tasks to recovering larger parts of user data.

Original: Current attacks against federated language models are sequence-agnostic and aim to extract as much data as possible from an FL update - often at the expense of fidelity for any particular sequence.
Revised: Current attacks against federated language models are sequence-agnostic and aim to extract as much data as possible from an FL update - often at the expense of fidelity for any particular sequence.

Original: Because of this, current attacks fail to extract any meaningful data under large-scale aggregation.
Revised: Because of this, current attacks fail to extract any meaningful data under large-scale aggregation.

Original: In realistic settings, an attacker cares most about a small portion of user data that contains sensitive personal information, for example sequences containing the phrase “my credit card number is ...”.
Revised: In realistic settings, an attacker cares most about a small portion of user data that contains sensitive personal information, for example sequences containing the phrase “my credit card number is ...”.

Original: In this work, we propose the first attack on FL that achieves targeted extraction of sequences that contain privacycritical phrases, whereby we employ maliciously modified parameters to allow the transformer itself to filter relevant sequences from aggregated user data and encode them in the gradient update.
Revised: In this work, we propose the first attack on FL that achieves targeted extraction of sequences that contain privacycritical phrases, whereby we employ maliciously modified parameters to allow the transformer itself to filter relevant sequences from aggregated user data and encode them in the gradient update.

Original: Our attack can effectively extract sequences of interest even against extremely large-scale aggregation.
Revised: Our attack can effectively extract sequences of interest even against extremely large-scale aggregation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Industrial machine learning models are often trained on large sets of user data.
Revised: Industrial machine learning models are often trained on large sets of user data.

Original: In a traditional centralized training paradigm, this is done by aggregating user data into a large repository.
Revised: In a traditional centralized training paradigm, this is done by aggregating user data into a large repository.

Original: Unfortunately, when user data contains personal information in the form of text, images, or other media, dataset aggregation leads to significant security, regulatory, and liability risks.
Revised: Unfortunately, when user data contains personal information in the form of text, images, or other media, dataset aggregation leads to significant security, regulatory, and liability risks.

Original: Against this backdrop, federated learning (FL) has emerged as a popular way to train models with decentralized data , that is without the need for a central party to host a dataset.
Revised: Against this backdrop, federated learning (FL) has emerged as a popular way to train models with decentralized data , that is without the need for a central party to host a dataset.

Original: By exchanging only model gradients, user devices collaboratively train a model without the direct exchange of plaintext data.
Revised: By exchanging only model gradients, user devices collaboratively train a model without the direct exchange of plaintext data.

Original: In many applications, FL is slower than centralized training (Bonawitz et al., 2019), but the privacy benefits outweigh the costs, especially in next-word text prediction which requires training on private text from smartphones (Hard et al., 2019).
Revised: In many applications, FL is slower than centralized training (Bonawitz et al., 2019), but the privacy benefits outweigh the costs, especially in next-word text prediction which requires training on private text from smartphones (Hard et al., 2019).

Original: Privacy through federated learning is sometimes taken for granted.
Revised: Privacy through federated learning is sometimes taken for granted.

Original: In reality, the actual privacy achieved by federated learning systems depends on a large number of factors and parameters – model size, architecture, number of users, the aggregation scheme, and more.
Revised: In reality, the actual privacy achieved by federated learning systems depends on a large number of factors and parameters – model size, architecture, number of users, the aggregation scheme, and more.

Original: Attacks against privacy in FL probe this boundary, empirically discovering pitfalls that should be considered and avoided when designing federated protocols (Phong et al., 2017; Melis et al., 2019;
Revised: Attacks against privacy in FL probe this boundary, empirically discovering pitfalls that should be considered and avoided when designing federated protocols (Phong et al., 2017; Melis et al., 2019;

Original: Geiping et al., 2020).
Revised: Geiping et al., 2020).

Original: In this work, we study the security of federated learning systems involving transformer architectures (Vaswani et al., 2017) which form the backbone of many recent advancements in natural language processing (Brown et al., 2020; Dosovitskiy et al., 2021; Jumper et al., 2021), and especially applications in text, which represent a key point of interest in many modern applications of federated learning (Paulik et al., 2021; Dimitriadis et al., 2022).
Revised: In this work, we study the security of federated learning systems involving transformer architectures (Vaswani et al., 2017) which form the backbone of many recent advancements in natural language processing (Brown et al., 2020; Dosovitskiy et al., 2021; Jumper et al., 2021), and especially applications in text, which represent a key point of interest in many modern applications of federated learning (Paulik et al., 2021; Dimitriadis et al., 2022).

Original: Our main threat model of interest is the untrusted server scenario, also known as the malicious server scenario, in which the server may make changes to model parameters in order to break user privacy.
Revised: Our main threat model of interest is the untrusted server scenario, also known as the malicious server scenario, in which the server may make changes to model parameters in order to break user privacy.

Original: This is in contrast to the honest-butcurious threat model, in which no malicious changes are permitted to the model training protocol.
Revised: This is in contrast to the honest-butcurious threat model, in which no malicious changes are permitted to the model training protocol.

Original: Untrusted server scenarios are of critical importance from a user-centric privacy perspective.
Revised: Untrusted server scenarios are of critical importance from a user-centric privacy perspective.

Original: Hey , do you want me to put the drinks on your tab ?
Revised: Hey , do you want me to put the drinks on your tab ?

Original: Yeah , can you use the credit card 4744 67 88 55 52 

Revised: Yeah , can you use the credit card 4744 67 88 55 52

Original: Oof did I tell you that my cat woke me up tonight ?
Revised: Oof did I tell you that my cat woke me up tonight ?

Original: Hey , do you want me to put the drinks on your tab ?
Revised: Hey , do you want me to put the drinks on your tab ?

Original: Yeah , can you use the credit card 4744 67 88 55 52 

Revised: Yeah , can you use the credit card 4744 67 88 55 52

Original: Oof did I tell you that my cat woke me up tonight ?
Revised: Oof did I tell you that my cat woke me up tonight ?

Original: Hey , do you want me to put the drinks on your tab ?
Revised: Hey , do you want me to put the drinks on your tab ?

Original: Yeah , can you use the credit card 4744 67 88 55 52 

Revised: Yeah , can you use the credit card 4744 67 88 55 52

Original: Oof did I tell you that my cat woke me up tonight ?
Revised: Oof did I tell you that my cat woke me up tonight ?

Original: Tagging through modified MHSA
Revised: Tagging through modified MHSA

Original: Input Text Seqs.
Revised: Input Text Seqs.

Original: [Figure]
Revised: [Figure]

Original: If a federated system is supposed to uphold privacy, then ideally this privacy can be guaranteed without having to assert full trust in the server.
Revised: If a federated system is supposed to uphold privacy, then ideally this privacy can be guaranteed without having to assert full trust in the server.

Original: After all, a perfectly trustworthy server could run the simplest FL protocol:
Revised: After all, a perfectly trustworthy server could run the simplest FL protocol:

Original: Centralize all user data, promise not to share it, train a centralized model, and delete all data.
Revised: Centralize all user data, promise not to share it, train a centralized model, and delete all data.

Original: Another way to look at untrusted server threats is to view them as a glimpse of worstcase dataset security.
Revised: Another way to look at untrusted server threats is to view them as a glimpse of worstcase dataset security.

Original: Even if we believe a server will uphold privacy, we might wonder about the worst-case loss in privacy that would arise if this server is even briefly compromised (through either classical security breaches or poisoning attacks (Bagdasaryan et al., 2019)) and acts maliciously.
Revised: Even if we believe a server will uphold privacy, we might wonder about the worst-case loss in privacy that would arise if this server is even briefly compromised (through either classical security breaches or poisoning attacks (Bagdasaryan et al., 2019)) and acts maliciously.

Original: A major strategy available to a malicious server is to modify the current state of a machine learning model as it is being trained, and then broadcast this corrupted model to the users.
Revised: A major strategy available to a malicious server is to modify the current state of a machine learning model as it is being trained, and then broadcast this corrupted model to the users.

Original: As the model is directly executed on each user device, this can be considered an analogue to untrusted code instructions that are being evaluated on a user’s private data (OWASP, 2022; Fowl et al., 2022).
Revised: As the model is directly executed on each user device, this can be considered an analogue to untrusted code instructions that are being evaluated on a user’s private data (OWASP, 2022; Fowl et al., 2022).

Original: Despite the inherent power that a malicious server has, extracting user data is still extremely difficult when gradients are aggregated over many users, in which case the averaged gradient does not contain enough entries to record the whole global training batch.
Revised: Despite the inherent power that a malicious server has, extracting user data is still extremely difficult when gradients are aggregated over many users, in which case the averaged gradient does not contain enough entries to record the whole global training batch.

Original: For this reason, existing attacks on text models only recover user data in scenarios where the number of model parameters is significantly larger than the number of tokens in a user update (Fowl et al., 2022;
Revised: For this reason, existing attacks on text models only recover user data in scenarios where the number of model parameters is significantly larger than the number of tokens in a user update (Fowl et al., 2022;

Original: Gupta et al., 2022; Dimitrov et al., 2022; Pasquini et al., 2021).
Revised: Gupta et al., 2022; Dimitrov et al., 2022; Pasquini et al., 2021).

Original: In some cases, attacks can siphon random examples of user data, but only through a large number of repeated queries (Wen et al., 2022).
Revised: In some cases, attacks can siphon random examples of user data, but only through a large number of repeated queries (Wen et al., 2022).

Original: In this work, we discuss a novel attack on text models whereby a malicious server is able to pick and choose which data to encode and extract from the model gradient, even with industrial-scale aggregation.
Revised: In this work, we discuss a novel attack on text models whereby a malicious server is able to pick and choose which data to encode and extract from the model gradient, even with industrial-scale aggregation.

Original: The attacker selects a trigger phrase, such as “credit card number” or “social security number,” and extracts all tokens of user data that follow the occurrence of this trigger.
Revised: The attacker selects a trigger phrase, such as “credit card number” or “social security number,” and extracts all tokens of user data that follow the occurrence of this trigger.

Original: We call this process, in which we sift selected phrases out of a large corpus of user data, “panning.”
Revised: We call this process, in which we sift selected phrases out of a large corpus of user data, “panning.”

Original: In comparison to existing attacks, this attack does not degrade when very many user updates are securely aggregated (Bonawitz et al., 2017).
Revised: In comparison to existing attacks, this attack does not degrade when very many user updates are securely aggregated (Bonawitz et al., 2017).

Original: For this reason, panning is an essential shift in capabilities for attacks against transformer-based models in federated learning.
Revised: For this reason, panning is an essential shift in capabilities for attacks against transformer-based models in federated learning.

Original:  B ACKGROUND AND A PPLICATION E XAMPLES
Revised:  B ACKGROUND AND A PPLICATION E XAMPLES

Original: Text models were the first and most successful systems where federated learning has been used in industrial settings.
Revised: Text models were the first and most successful systems where federated learning has been used in industrial settings.

Original: These applications include keystroke prediction Hard et al.
Revised: These applications include keystroke prediction Hard et al.

Original: Ramaswamy et al.
Revised: Ramaswamy et al.

Original: (2019), settings search Bonawitz et al.
Revised: (2019), settings search Bonawitz et al.

Original: (2019), news personalization (Paulik et al., 2021), and improved messenger services on Android (Google, 2022).
Revised: (2019), news personalization (Paulik et al., 2021), and improved messenger services on Android (Google, 2022).

Original: In the latter case, the documentation of
Revised: In the latter case, the documentation of

Original: Google messages 1 , describes federated learning being used to improve ML models related to Google messages, for example “smart reply” features.
Revised: Google messages 1 , describes federated learning being used to improve ML models related to Google messages, for example “smart reply” features.

Original: FL is active by default on all Android devices with this app.
Revised: FL is active by default on all Android devices with this app.

Original: According to the documentation, user conversations are protected only through secure aggregation (Bonawitz et al., 2017), which “ can’t reveal your conversations or content to Google or anyone else [...], grouping many similar adjustments together so that Google can’t inspect an adjustment from a single device ”.
Revised: According to the documentation, user conversations are protected only through secure aggregation (Bonawitz et al., 2017), which “ can’t reveal your conversations or content to Google or anyone else [...], grouping many similar adjustments together so that Google can’t inspect an adjustment from a single device ”.

Original: In this work, we argue that this system of protecting privacy leaves users open to targeted extraction attacks from a malicious server update sent to the user, provided the system uses a transformer-based machine learning model.
Revised: In this work, we argue that this system of protecting privacy leaves users open to targeted extraction attacks from a malicious server update sent to the user, provided the system uses a transformer-based machine learning model.

Original: Known attacks against privacy in federated language models are not capable of breaking privacy in such large aggregation settings.
Revised: Known attacks against privacy in federated language models are not capable of breaking privacy in such large aggregation settings.

Original: The earliest attacks against transformers in Zhu et al. (2019b);
Revised: The earliest attacks against transformers in Zhu et al. (2019b);

Original: Deng et al.
Revised: Deng et al.

Original: (2021) operate in the ”honest-but-curious” threat model.
Revised: (2021) operate in the ”honest-but-curious” threat model.

Original: These attacks have resulted in only limited success, recovering data only from single sequences.
Revised: These attacks have resulted in only limited success, recovering data only from single sequences.

Original: In practice, “single sequences” imply that user updates are not securely aggregated, each individual user sends an individual update that represents only their data.
Revised: In practice, “single sequences” imply that user updates are not securely aggregated, each individual user sends an individual update that represents only their data.

Original: Recent developments in Dimitrov et al.
Revised: Recent developments in Dimitrov et al.

Original: and Gupta et al.
Revised: and Gupta et al.

Original: (2022) have pushed the envelope, reconstructing more and more text fragments, even from multiple sequences in this restricted threat model.
Revised: (2022) have pushed the envelope, reconstructing more and more text fragments, even from multiple sequences in this restricted threat model.

Original: On the other side, malicious server threat models, for example in Pasquini et al.
Revised: On the other side, malicious server threat models, for example in Pasquini et al.

Original: (2021) and Fowl et al.
Revised: (2021) and Fowl et al.

Original: (2022) can recover more information and identify or respectively reconstruct from aggregates of hundreds of sequences.
Revised: (2022) can recover more information and identify or respectively reconstruct from aggregates of hundreds of sequences.

Original: These attacks all attempt to recover all user data from an aggregated gradient, putting equal emphasis on the fidelity of each reconstructed sequence.
Revised: These attacks all attempt to recover all user data from an aggregated gradient, putting equal emphasis on the fidelity of each reconstructed sequence.

Original: This strategy inevitably decreases the capability of previous attacks to recover accurate individual sequences as the number of sequences and tokens increases, and eventually the extracted sequences become meaningless combinations of tokens.
Revised: This strategy inevitably decreases the capability of previous attacks to recover accurate individual sequences as the number of sequences and tokens increases, and eventually the extracted sequences become meaningless combinations of tokens.

Original: Attacks that break large scale aggregation generally require additional assumptions.
Revised: Attacks that break large scale aggregation generally require additional assumptions.

Original: Lam et al.
Revised: Lam et al.

Original: (2021) show that aggregation can be breached if side-channel metadata is available to the server,
Revised: (2021) show that aggregation can be breached if side-channel metadata is available to the server,

Original: Pasquini et al.
Revised: Pasquini et al.

Original: (2021) show that if users can be addressed separately by the server and each user only has limited data, malicious parameters with zero-gradients can be used to suppress all other users and break aggregation.
Revised: (2021) show that if users can be addressed separately by the server and each user only has limited data, malicious parameters with zero-gradients can be used to suppress all other users and break aggregation.

Original: This trick can also be used for targeted membership inference, checking if a user owns a specific datapoint.
Revised: This trick can also be used for targeted membership inference, checking if a user owns a specific datapoint.

Original: Further, Wen et al.
Revised: Further, Wen et al.

Original: (2022) shows that if malicious servers are allowed to send multiple queries, then they can “fish” for single data points, even from aggregated data.
Revised: (2022) shows that if malicious servers are allowed to send multiple queries, then they can “fish” for single data points, even from aggregated data.

Original: Yet, in a real-life scenario, among all data a user computes and aggregates model updates with, often only a few of them contain information valuable to a potential attacker.
Revised: Yet, in a real-life scenario, among all data a user computes and aggregates model updates with, often only a few of them contain information valuable to a potential attacker.

Original: For example, out of a 50sentence online conversation between a user and a bank agent, an attacker may only be interested in the particular sentence that contains the user’s social security number.
Revised: For example, out of a 50sentence online conversation between a user and a bank agent, an attacker may only be interested in the particular sentence that contains the user’s social security number.

Original: This poses the question of whether it is possible for an attacker to dedicate the capacity of the gradient to only a limited number of target sequences with a specific set of keywords or triggers.
Revised: This poses the question of whether it is possible for an attacker to dedicate the capacity of the gradient to only a limited number of target sequences with a specific set of keywords or triggers.

Original: Threat Model - Untrusted Server Our threat model contains two parties.
Revised: Threat Model - Untrusted Server Our threat model contains two parties.

Original: First, a user (or group of aggregated users) that owns text, which contains private information following a set of keywords
Revised: First, a user (or group of aggregated users) that owns text, which contains private information following a set of keywords

Original: K = { k 1 , . . . , k n } .
Revised: K = { k 1 , . . . , k n } .

Original: Second, a malicious server that aims to “pan”, e.g., perform targeted extraction of user tokens that immediately follow one of these keywords.
Revised: Second, a malicious server that aims to “pan”, e.g., perform targeted extraction of user tokens that immediately follow one of these keywords.

Original: We assume that the number of aggregated sequences may be unlimited, but the number of sequences matching the combination of keywords is limited.
Revised: We assume that the number of aggregated sequences may be unlimited, but the number of sequences matching the combination of keywords is limited.

Original: We assume that the federated learning exchange is otherwise secure: Both parties agree beforehand on a transparent implementation for both model architecture and user-side protocol that is vetted by public examination.
Revised: We assume that the federated learning exchange is otherwise secure: Both parties agree beforehand on a transparent implementation for both model architecture and user-side protocol that is vetted by public examination.

Original: The only attack vector for the server is the model update sent indiscriminately to all users in the group.
Revised: The only attack vector for the server is the model update sent indiscriminately to all users in the group.

Original: In this way, the presence of the attack may be non-obvious because it does not require deploying any new executable code, and it is embedded in network parameters that are seldom, if ever, inspected.
Revised: In this way, the presence of the attack may be non-obvious because it does not require deploying any new executable code, and it is embedded in network parameters that are seldom, if ever, inspected.

Original: Even stronger threat models have also been investigated (Frey, 2021; Boenisch et al., 2021; Fowl et al., 2021).
Revised: Even stronger threat models have also been investigated (Frey, 2021; Boenisch et al., 2021; Fowl et al., 2021).

Original: Threat models considered in Frey (2021) allow the server to execute arbitrary pieces of code on user devices, while Fowl et al.
Revised: Threat models considered in Frey (2021) allow the server to execute arbitrary pieces of code on user devices, while Fowl et al.

Original: ; Boenisch et al.
Revised: ; Boenisch et al.

Original: (2021) assumes a server can maliciously modify model architectures.
Revised: (2021) assumes a server can maliciously modify model architectures.

Original: We also note that the role of the attacker need not be played by the party that owns the server, and any individual/party that has access to the model update anywhere in the pipeline can assume the role.
Revised: We also note that the role of the attacker need not be played by the party that owns the server, and any individual/party that has access to the model update anywhere in the pipeline can assume the role.

Original: Particularly, a man-in-the-middle attacker can inject the attack into model parameters before broadcasting to users, retrieve the resulting unsecured user update, and replace the outgoing update with seemingly normal model gradients.
Revised: Particularly, a man-in-the-middle attacker can inject the attack into model parameters before broadcasting to users, retrieve the resulting unsecured user update, and replace the outgoing update with seemingly normal model gradients.

Original: Remark (Peer-to-Peer Attacks between Users.) .
Revised: Remark (Peer-to-Peer Attacks between Users.) .

Original: This threat model applies even if the server is fully benign: If in any round of training, a group of malicious users can control enough updates to replace the model parameters (Bagdasaryan et al., 2019;
Revised: This threat model applies even if the server is fully benign: If in any round of training, a group of malicious users can control enough updates to replace the model parameters (Bagdasaryan et al., 2019;

Original: Bhagoji et al., 2019) in a single round of training, then the server will unwittingly distribute these malicious model parameters in the next round of training, collect model updates from other, vulnerable users, aggregate them and redistribute the updated model.
Revised: Bhagoji et al., 2019) in a single round of training, then the server will unwittingly distribute these malicious model parameters in the next round of training, collect model updates from other, vulnerable users, aggregate them and redistribute the updated model.

Original: Because an attack through “panning” is invariant to aggregation sizes, this means that the group of attackers can then discover data from vulnerable users in the next round of training, while the server remains unaware of any breach privacy.
Revised: Because an attack through “panning” is invariant to aggregation sizes, this means that the group of attackers can then discover data from vulnerable users in the next round of training, while the server remains unaware of any breach privacy.

Original:  M ETHOD
Revised:  M ETHOD

Original: Here we discuss the possibility of a“panning” attack that performs extraction of target sequences that contain a specific set of keywords.
Revised: Here we discuss the possibility of a“panning” attack that performs extraction of target sequences that contain a specific set of keywords.

Original: The attack we describe below is specific to transformer architectures and based on modifications to attention mechanisms and linear layers.
Revised: The attack we describe below is specific to transformer architectures and based on modifications to attention mechanisms and linear layers.

Original: The key ideas can be summarized as follows:
Revised: The key ideas can be summarized as follows:

Original: Modification of multi-head attention blocks (MHSA) to “tag” the tokens of a sequence that contains specific keywords, as in Section 3.2.
Revised: Modification of multi-head attention blocks (MHSA) to “tag” the tokens of a sequence that contains specific keywords, as in Section 3.2.

Original: Modification of linear layers to detect the tag and trigger capture of token embeddings belonging to target sequences, while filtering out non-target sequences, see Section 3.3.
Revised: Modification of linear layers to detect the tag and trigger capture of token embeddings belonging to target sequences, while filtering out non-target sequences, see Section 3.3.

Original: Post-processing the relevant embeddings into target sequences, as in Section 3.4.
Revised: Post-processing the relevant embeddings into target sequences, as in Section 3.4.

Original: An overview of our attack is presented in Figure 2.
Revised: An overview of our attack is presented in Figure 2.

Original: The discussed attack draws on a family of works describing “analytic” attacks against models in federated learning that we describe in Section 3.1.
Revised: The discussed attack draws on a family of works describing “analytic” attacks against models in federated learning that we describe in Section 3.1.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: As discussed in Phong et al.
Revised: As discussed in Phong et al.

Original: ; Geiping et al.
Revised: ; Geiping et al.

Original: Fowl et al.
Revised: Fowl et al.

Original: (2021), the gradient of a linear layer parameterized by ( W , b ) whose forward pass is y
Revised: (2021), the gradient of a linear layer parameterized by ( W , b ) whose forward pass is y

Original: = W f + b reveals information about its input f .
Revised: = W f + b reveals information about its input f .

Original: In particular, for any given row W m and the related bias entry b m , we have
Revised: In particular, for any given row W m and the related bias entry b m , we have

Original: [Equation]
Revised: [Equation]

Original: where L ( y ) denotes the downstream loss computed on the layer output y .
Revised: where L ( y ) denotes the downstream loss computed on the layer output y .

Original: An attacker can then simply reconstruct f by computing f = ∇ W m L ⊘ ∂L∂b m , where ⊘ denotes entry-wise division as.
Revised: An attacker can then simply reconstruct f by computing f = ∇ W m L ⊘ ∂L∂b m , where ⊘ denotes entry-wise division as.

Original: However, attackers cannot easily apply this trick to transformers in federated learning.
Revised: However, attackers cannot easily apply this trick to transformers in federated learning.

Original: Firstly, transformers add together token embeddings and positional embeddings, and have residual connections that add the output of multi-head self-attention (MHSA) to these mixed embeddings before applying the first fully-connected linear layer.
Revised: Firstly, transformers add together token embeddings and positional embeddings, and have residual connections that add the output of multi-head self-attention (MHSA) to these mixed embeddings before applying the first fully-connected linear layer.

Original: In other words, the individual input f i,j of the first linear layer of the first encoder block is
Revised: In other words, the individual input f i,j of the first linear layer of the first encoder block is

Original: [Equation]
Revised: [Equation]

Original: where e i,j is the embedding of the j -th token in the i -th sequence, p j is the j -th positional embedding, and MHSA ( { e i,j ′ + p j ′ } ℓ j ′ =1 , j ) is the j -th entry after applying MHSA on sequence i .
Revised: where e i,j is the embedding of the j -th token in the i -th sequence, p j is the j -th positional embedding, and MHSA ( { e i,j ′ + p j ′ } ℓ j ′ =1 , j ) is the j -th entry after applying MHSA on sequence i .

Original: Secondly,  assuming the model update is computed over B sequences of length ℓ , an attacker only has access to the average gradient aggregated over all Bℓ tokens, The aggregated gradients of the first linear layer are instead
Revised: Secondly, [Equation] assuming the model update is computed over B sequences of length ℓ , an attacker only has access to the average gradient aggregated over all Bℓ tokens, The aggregated gradients of the first linear layer are instead

Original: [Equation]
Revised: [Equation]

Original: Recovering an individual f i,j from this gradient cocktail is non-trivial.
Revised: Recovering an individual f i,j from this gradient cocktail is non-trivial.

Original: In summary, to recover a user’s input sequences, an attacker needs to (1) isolate out individual mixed embedding f i,j , (2) split the mixed embedding f i,j into a token embedding e i,j and position p
Revised: In summary, to recover a user’s input sequences, an attacker needs to (1) isolate out individual mixed embedding f i,j , (2) split the mixed embedding f i,j into a token embedding e i,j and position p

Original: i,j , and (3) match e i,j to the most probable word.
Revised: i,j , and (3) match e i,j to the most probable word.

Original: If all tokens are uniformly represented in the gradient, then step (1) may be impossible.
Revised: If all tokens are uniformly represented in the gradient, then step (1) may be impossible.

Original: Below, we show how the network can be put into a state where the gradient’s information capacity is sparingly used to encode targeted sequences.
Revised: Below, we show how the network can be put into a state where the gradient’s information capacity is sparingly used to encode targeted sequences.

Original: Big Picture : To extract data from a transformer model update, the attacker needs to first extract individual mixed embeddings from aggregated (batch averaged) gradients, group these into sentences, and then split the mixed embeddings into their position and token.
Revised: Big Picture : To extract data from a transformer model update, the attacker needs to first extract individual mixed embeddings from aggregated (batch averaged) gradients, group these into sentences, and then split the mixed embeddings into their position and token.

Original: 3. T AGGING TARGET SEQUENCES
Revised: 3. T AGGING TARGET SEQUENCES

Original: In this section, we describe how an attacker could modify parameters in the attention block to “tag” the mixed embedding that belongs to a sequence that contains the selected keywords
Revised: In this section, we describe how an attacker could modify parameters in the attention block to “tag” the mixed embedding that belongs to a sequence that contains the selected keywords

Original: K = { k 1 , . . .
Revised: K = { k 1 , . . .

Original: , k N } .
Revised: , k N } .

Original: We illustrate the modification in Figure 3.
Revised: We illustrate the modification in Figure 3.

Original: For simplicity, we start by detailing this construction with only one keyword k .
Revised: For simplicity, we start by detailing this construction with only one keyword k .

Original: We refer to a sequence that contains k as either the “target” sequence or “relevant” sequence.
Revised: We refer to a sequence that contains k as either the “target” sequence or “relevant” sequence.

Original: Other sequences are called “non-targeted” or “irrelevant.”
Revised: Other sequences are called “non-targeted” or “irrelevant.”

Original: To facilitate the construction, we assume the norm of token embedding e ( k ) associated with the target word k to be larger than the norm of all other token and positional embeddings.
Revised: To facilitate the construction, we assume the norm of token embedding e ( k ) associated with the target word k to be larger than the norm of all other token and positional embeddings.

Original: The server can easily meet this assumption as both embedding matrices are also part of the model parameters under their control.
Revised: The server can easily meet this assumption as both embedding matrices are also part of the model parameters under their control.

Original: With this assumption, we design a self-attention mechanism inspired by Fowl et al.
Revised: With this assumption, we design a self-attention mechanism inspired by Fowl et al.

Original: (2022) that tags the target sequences by imprinting e ( k ) into every mixed embedding in the target sequence.
Revised: (2022) that tags the target sequences by imprinting e ( k ) into every mixed embedding in the target sequence.

Original: To be more specific, let ( W Q , b Q ) , ( W K , b K ) , ( W V , b V ) be the weight matrices and biases of query, key, and value layers of MSHA in the first encoder block respectively.
Revised: To be more specific, let ( W Q , b Q ) , ( W K , b K ) , ( W V , b V ) be the weight matrices and biases of query, key, and value layers of MSHA in the first encoder block respectively.

Original: We set each of the parameters as follows,
Revised: We set each of the parameters as follows,

Original: [Equation]
Revised: [Equation]

Original: where α is a large positive value (e.g. 10 8 ) and T d ′ is the truncated identity matrix with pre-defined dimension d ′ .
Revised: where α is a large positive value (e.g. 10 8 ) and T d ′ is the truncated identity matrix with pre-defined dimension d ′ .

Original: W K and b K form an identity operator, and thus the key token K (and others) pass through unaltered.
Revised: W K and b K form an identity operator, and thus the key token K (and others) pass through unaltered.

Original: W Q and b Q are designed so that the query vectors Q are each identical copies of e ( k ) .
Revised: W Q and b Q are designed so that the query vectors Q are each identical copies of e ( k ) .

Original: Finally, W V and b V keep only the first d ′ entries of the input embedding, and set all other positions in each embedding to zero.
Revised: Finally, W V and b V keep only the first d ′ entries of the input embedding, and set all other positions in each embedding to zero.

Original: Let’s evaluate the results of the MHSA computation based on this construction.
Revised: Let’s evaluate the results of the MHSA computation based on this construction.

Original: If a length- ℓ sequence { e j } ℓj =1 contains the target k at position j ′ , then the attention weight softmax ( QK ⊤ √ d ) will be large, forcing every word in the sentence to attend to the target word.
Revised: If a length- ℓ sequence { e j } ℓj =1 contains the target k at position j ′ , then the attention weight softmax ( QK ⊤ √ d ) will be large, forcing every word in the sentence to attend to the target word.

Original: In fact, when α is large enough, attention to the target is near-absolute.
Revised: In fact, when α is large enough, attention to the target is near-absolute.

Original: Thus, the resulting mixed embedding after the MHSA is:
Revised: Thus, the resulting mixed embedding after the MHSA is:

Original: [Equation]
Revised: [Equation]

Original: In plain language, the first d ′ entries of the keyword embedding vector are added to the first d ′ positions of every embedding in the sequence.
Revised: In plain language, the first d ′ entries of the keyword embedding vector are added to the first d ′ positions of every embedding in the sequence.

Original: We say that the sequence is now tagged with the keyword.
Revised: We say that the sequence is now tagged with the keyword.

Original: For causal language models, applying the construction in Equation (10) on masked MHSA turns the keyword into a trigger, where the attention imprints e ( k ) to tokens that appear after the keyword in the target sequences.
Revised: For causal language models, applying the construction in Equation (10) on masked MHSA turns the keyword into a trigger, where the attention imprints e ( k ) to tokens that appear after the keyword in the target sequences.

Original: Coupling this tagging approach with a filter introduced later in Section 3.3, one can perform targeted extraction of sub-sequences that start with the keyword k .
Revised: Coupling this tagging approach with a filter introduced later in Section 3.3, one can perform targeted extraction of sub-sequences that start with the keyword k .

Original: On the other hand, if the sequence is irrelevant, or in the case of causal MHSA, if the mixed embedding appears before k , then the imprinted embedding will be random from e 1 to e ℓ .
Revised: On the other hand, if the sequence is irrelevant, or in the case of causal MHSA, if the mixed embedding appears before k , then the imprinted embedding will be random from e 1 to e ℓ .

Original: This difference allows our malicious modification to tag the mixed embedding in target sequences by a keyword-specific imprint and we will discuss in the next section how we filter out irrelevant sequences based on this imprint
Revised: This difference allows our malicious modification to tag the mixed embedding in target sequences by a keyword-specific imprint and we will discuss in the next section how we filter out irrelevant sequences based on this imprint

Original: We note that extending the above construction to multiple keywords is straightforward.
Revised: We note that extending the above construction to multiple keywords is straightforward.

Original: In particular, for N keywords k 1 , . . .
Revised: In particular, for N keywords k 1 , . . .

Original: , k N , one can simply repeat the above constructions for a separate head in the
Revised: , k N , one can simply repeat the above constructions for a separate head in the

Original: MHSA block for each keyword and imprint each keyword-specific signature to different positions.
Revised: MHSA block for each keyword and imprint each keyword-specific signature to different positions.

Original: Big Picture : The attention mechanism of a transformer is repurposed to attend strongly to the attacker’s keywords, and add these keyword embeddings to all other entries in the sequences, thereby tagging them as important.
Revised: Big Picture : The attention mechanism of a transformer is repurposed to attend strongly to the attacker’s keywords, and add these keyword embeddings to all other entries in the sequences, thereby tagging them as important.

Original: 3. F ILTERING NON - TARGETED TOKENS
Revised: 3. F ILTERING NON - TARGETED TOKENS

Original: Targeted tokens are those appearing in a relevant sequence containing a keyword.
Revised: Targeted tokens are those appearing in a relevant sequence containing a keyword.

Original: All other tokens in other sequences are not targets and should be filtered out.
Revised: All other tokens in other sequences are not targets and should be filtered out.

Original: We want to remove them from the gradient aggregation to enable the attacker to isolate the few remaining target tokens in the gradient.
Revised: We want to remove them from the gradient aggregation to enable the attacker to isolate the few remaining target tokens in the gradient.

Original: We provide an illustration of the key idea in Figure 3.
Revised: We provide an illustration of the key idea in Figure 3.

Original: Our approach draws inspiration from Fowl et al.
Revised: Our approach draws inspiration from Fowl et al.

Original: (2021), who performs gradient separation on aggregated gradients uncovered in Equation (3).
Revised: (2021), who performs gradient separation on aggregated gradients uncovered in Equation (3).

Original: We construct a malicious linear “imprint” layer by assigning to each row of W identical copies of a measurement vector w .
Revised: We construct a malicious linear “imprint” layer by assigning to each row of W identical copies of a measurement vector w .

Original: We then set the entries of the bias vector b to form an increasing sequence b m < b m +1 .
Revised: We then set the entries of the bias vector b to form an increasing sequence b m < b m +1 .

Original: If the linear layer is followed by a ReLU activation, the aggregated gradient becomes
Revised: If the linear layer is followed by a ReLU activation, the aggregated gradient becomes

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: where g m ( f i,j )
Revised: where g m ( f i,j )

Original: = 1 if ⟨ w , f i,j ⟩ + b m > 0 and g m ( f i,j ) = 0 otherwise.
Revised: = 1 if ⟨ w , f i,j ⟩ + b m > 0 and g m ( f i,j ) = 0 otherwise.

Original: Intuitively, the ReLU outputs from this layer are like a thermometer for measuring the inner product ⟨ w , f i,j > ⟩ .
Revised: Intuitively, the ReLU outputs from this layer are like a thermometer for measuring the inner product ⟨ w , f i,j > ⟩ .

Original: Because the bias entries are increasing, more ReLUs will be active if the value of
Revised: Because the bias entries are increasing, more ReLUs will be active if the value of

Original: W f i,j is larger.
Revised: W f i,j is larger.

Original: The m -th ReLU turns on when ⟨ w , f i,j > ⟩ lies above the cutoff defined by b m . .
Revised: The m -th ReLU turns on when ⟨ w , f i,j > ⟩ lies above the cutoff defined by b m . .

Original: The gradient for the row W m aggregates over all tokens with inner products above b m .
Revised: The gradient for the row W m aggregates over all tokens with inner products above b m .

Original: Now suppose that just one token f ′ out of all f i,j uniquely satisfies g m ( f ′ )
Revised: Now suppose that just one token f ′ out of all f i,j uniquely satisfies g m ( f ′ )

Original: = 1 and g m +

Revised: = 1 and g m +

Original: ( f ′ )
Revised: ( f ′ )

Original: = 0 .
Revised: = 0 .

Original: Then we can separate out the individual gradient for this token by computing
Revised: Then we can separate out the individual gradient for this token by computing

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: We can now uniquely recover f ′ just as we did from with a batch size of 1 in Equation (1).
Revised: We can now uniquely recover f ′ just as we did from with a batch size of 1 in Equation (1).

Original: Therefore, if the attacker can accurately estimate the distribution of ⟨ w f i,j ⟩ , they can then construct b accordingly to maximize the chance for every f i,j to fall uniquely in one of the bins ( b m , b m +1 ) .
Revised: Therefore, if the attacker can accurately estimate the distribution of ⟨ w f i,j ⟩ , they can then construct b accordingly to maximize the chance for every f i,j to fall uniquely in one of the bins ( b m , b m +1 ) .

Original: It was noted by Fowl et al.
Revised: It was noted by Fowl et al.

Original: that the distribution of ⟨ w, ( e i,j + p j ) ⟩ is approximately Gaussian if every entry of w is a random sample from a standard Gaussian.
Revised: that the distribution of ⟨ w, ( e i,j + p j ) ⟩ is approximately Gaussian if every entry of w is a random sample from a standard Gaussian.

Original: Assuming b has M entries in total, we can optimize the attack by setting b m = − Φ − 1 ( mM ) where Φ − 1 ( · ) is the inverse of estimated Gaussian distribution.
Revised: Assuming b has M entries in total, we can optimize the attack by setting b m = − Φ − 1 ( mM ) where Φ − 1 ( · ) is the inverse of estimated Gaussian distribution.

Original: Under this choice, each bin ( b m , b m +1 ) approximately contains the same probably mass.
Revised: Under this choice, each bin ( b m , b m +1 ) approximately contains the same probably mass.

Original: However, observe that the number of bins upper-bounds the capability of an attacker to recover user data; if too many samples pass through the bins will become oversaturated.
Revised: However, observe that the number of bins upper-bounds the capability of an attacker to recover user data; if too many samples pass through the bins will become oversaturated.

Original: Therefore, directly attempting to recover all f i,j becomes less effective against large-scale aggregation.
Revised: Therefore, directly attempting to recover all f i,j becomes less effective against large-scale aggregation.

Original: To address the issue, we propose a set of modifications for linear layers to pan for relevant mixed embeddings.
Revised: To address the issue, we propose a set of modifications for linear layers to pan for relevant mixed embeddings.

Original: Here, we describe how to program the first linear layer of the first encoder block, and discuss the extension to multiple encoder blocks in Appendix A.
Revised: Here, we describe how to program the first linear layer of the first encoder block, and discuss the extension to multiple encoder blocks in Appendix A.

Original: The key idea is to design w so that targeted tokens produce a large value of ⟨ w , f ⟩ while untargeted tokens produce a small value.
Revised: The key idea is to design w so that targeted tokens produce a large value of ⟨ w , f ⟩ while untargeted tokens produce a small value.

Original: We then assign entries of b so that ReLUs only activate for tokens in the target distribution.
Revised: We then assign entries of b so that ReLUs only activate for tokens in the target distribution.

Original: To be more concrete, we scale the norm of e ( k ) , the embedding of the given keyword, to β > 1 , where β controls the ”signal strength” of the relevant mixed embedding, and helps to separate the irrelevant embedding as detailed later.
Revised: To be more concrete, we scale the norm of e ( k ) , the embedding of the given keyword, to β > 1 , where β controls the ”signal strength” of the relevant mixed embedding, and helps to separate the irrelevant embedding as detailed later.

Original: We also sample the random measurement vector w from standard
Revised: We also sample the random measurement vector w from standard

Original: Gaussian and mask out the first d ′ entries.
Revised: Gaussian and mask out the first d ′ entries.

Original: Finally, we construct w ′ as
Revised: Finally, we construct w ′ as

Original: [Equation]
Revised: [Equation]

Original: By setting each row of W to ( w + w ′ ) , we have
Revised: By setting each row of W to ( w + w ′ ) , we have

Original: [Equation]
Revised: [Equation]

Original: by Eq.
Revised: by Eq.

Original: 5 where j ′ is the position the MHSA attends to.
Revised: 5 where j ′ is the position the MHSA attends to.

Original: Assuming e j and p j are both zero-mean (these can be chosen at will by the server), the measurement distribution of irrelevant embeddings remains zero-mean, while a mean shift of ( β ∥ w ′ ∥ ) 2 can be observed in the distribution of the mixed target embeddings.
Revised: Assuming e j and p j are both zero-mean (these can be chosen at will by the server), the measurement distribution of irrelevant embeddings remains zero-mean, while a mean shift of ( β ∥ w ′ ∥ ) 2 can be observed in the distribution of the mixed target embeddings.

Original: We show the separation between both distributions empirically in Figure 4.
Revised: We show the separation between both distributions empirically in Figure 4.

Original: The distinction allows us to adapt the binning strategy from Fowl et al.
Revised: The distinction allows us to adapt the binning strategy from Fowl et al.

Original: (2022) to focus on the distribution of relevant target embeddings.
Revised: (2022) to focus on the distribution of relevant target embeddings.

Original: This allows the attack to condense all other embeddings into a small number of bins and dedicate the remaining bins to separating individual target embeddings.
Revised: This allows the attack to condense all other embeddings into a small number of bins and dedicate the remaining bins to separating individual target embeddings.

Original: Big Picture : The attack leads to a bimodal distribution of embeddings.
Revised: Big Picture : The attack leads to a bimodal distribution of embeddings.

Original: Targeted embeddings produce large activations in the linear imprint layer, while untargeted embeddings produce low activations.
Revised: Targeted embeddings produce large activations in the linear imprint layer, while untargeted embeddings produce low activations.

Original: 3. P OST - PROCESSING MIXED EMBEDDINGS INTO MEANINGFUL SEQUENCES
Revised: 3. P OST - PROCESSING MIXED EMBEDDINGS INTO MEANINGFUL SEQUENCES

Original: With individual mixed embeddings f i,j isolated into separate ReLU bins, we then need to extract the corresponding sequence index i , position index j and the actual word from the vocabulary for each f i,j .
Revised: With individual mixed embeddings f i,j isolated into separate ReLU bins, we then need to extract the corresponding sequence index i , position index j and the actual word from the vocabulary for each f i,j .

Original: The tagging we describe above is used to label tokens based on the keyword that made them a target Section 3.2.
Revised: The tagging we describe above is used to label tokens based on the keyword that made them a target Section 3.2.

Original: We can also introduce a second tagging mechanism that imprints positional information onto each token.
Revised: We can also introduce a second tagging mechanism that imprints positional information onto each token.

Original: To be more concrete, we construct a positional imprint using a second head in the MHSA block:  
Revised: To be more concrete, we construct a positional imprint using a second head in the MHSA block:  

Original: [Equation]
Revised: [Equation]

Original: 0 otherwise.
Revised: 0 otherwise.

Original: For a given sequence { f j } ℓj =1 with mixed-in embeddings { e j } ℓj =1 , the positional imprint outputs
Revised: For a given sequence { f j } ℓj =1 with mixed-in embeddings { e j } ℓj =1 , the positional imprint outputs

Original: [Equation]
Revised: [Equation]

Original: where entries d +1 to 2 d of the first token in every sequence are imprinted to mixed embeddingin the same sequence.
Revised: where entries d +1 to 2 d of the first token in every sequence are imprinted to mixed embeddingin the same sequence.

Original: We then group f i,j into sequences with constrained K-means clustering (Bradley et al., 2000) using entries d + 1 to 2 d that encode sentence identity.
Revised: We then group f i,j into sequences with constrained K-means clustering (Bradley et al., 2000) using entries d + 1 to 2 d that encode sentence identity.

Original: Constrained K -means allows us to constraint the size of each cluster, which is no larger than sequence length ℓ .
Revised: Constrained K -means allows us to constraint the size of each cluster, which is no larger than sequence length ℓ .

Original: After grouping embeddings into sequences, the next step is to recover positions.
Revised: After grouping embeddings into sequences, the next step is to recover positions.

Original: As mixed embeddings in the same sequences are guaranteed to have distinct positions, we perform position recovery on a sequence-by-sequence basis.
Revised: As mixed embeddings in the same sequences are guaranteed to have distinct positions, we perform position recovery on a sequence-by-sequence basis.

Original: For a set of mixed embeddings { f j ′ } grouped in the same sequence, identifying the most suitable unique position j of each f j ′ can be viewed as a maximum weight bipartite matching problem, where we define the weight of a potential match ( f j ′ , j ) to be the correlation between f j ′ and p j .
Revised: For a set of mixed embeddings { f j ′ } grouped in the same sequence, identifying the most suitable unique position j of each f j ′ can be viewed as a maximum weight bipartite matching problem, where we define the weight of a potential match ( f j ′ , j ) to be the correlation between f j ′ and p j .

Original: We can solve the matching problem efficiently (Kuhn, 1955) and obtain the pure token embedding e i,j by subtracting the recovered positional embedding from f i,j .
Revised: We can solve the matching problem efficiently (Kuhn, 1955) and obtain the pure token embedding e i,j by subtracting the recovered positional embedding from f i,j .

Original: The final step is to associate each recovered token embedding e i,j with the actual words.
Revised: The final step is to associate each recovered token embedding e i,j with the actual words.

Original: (Fowl et al., 2022) discovered that the frequency of tokens presented in user updates can be estimated accurately.
Revised: (Fowl et al., 2022) discovered that the frequency of tokens presented in user updates can be estimated accurately.

Original: The estimated frequency dictates the maximum number of e i,j a specific word v can be associated with.
Revised: The estimated frequency dictates the maximum number of e i,j a specific word v can be associated with.

Original: Then, taking the constraint into account, matching e i,j to the most suitable word v can be formulated as another maximum weight bipartite matching problem, where the weight of a potential match ( e i,j , v ) is the correlation between e i,j and e v .
Revised: Then, taking the constraint into account, matching e i,j to the most suitable word v can be formulated as another maximum weight bipartite matching problem, where the weight of a potential match ( e i,j , v ) is the correlation between e i,j and e v .

Original: Big Picture : Individually recovered mixed embeddings (which are the sum of a position embedding and a token embedding) can be grouped by sequence.
Revised: Big Picture : Individually recovered mixed embeddings (which are the sum of a position embedding and a token embedding) can be grouped by sequence.

Original: The tokens in a sequence can then be separated into token ID and position ID by solving bipartite matching problems.
Revised: The tokens in a sequence can then be separated into token ID and position ID by solving bipartite matching problems.

Original:  E MPIRICAL E VALUATION OF THE A TTACK
Revised:  E MPIRICAL E VALUATION OF THE A TTACK

Original: In this section, we empirically evaluate our proposed attack on different transformer architectures commonly used in real-world applications.
Revised: In this section, we empirically evaluate our proposed attack on different transformer architectures commonly used in real-world applications.

Original: In particular, we consider the smallest variant of GPT2 (Radford et al., 2019) with 124 million parameters, which is a causal language model with masked self-attention.
Revised: In particular, we consider the smallest variant of GPT2 (Radford et al., 2019) with 124 million parameters, which is a causal language model with masked self-attention.

Original: We also consider a variant of BERT (Devlin et al., 2019) with 110 million parameters, which was used in previous works (Deng et al., 2021; Zhu et al., 2019a) for attack evaluation.
Revised: We also consider a variant of BERT (Devlin et al., 2019) with 110 million parameters, which was used in previous works (Deng et al., 2021; Zhu et al., 2019a) for attack evaluation.

Original: Unlike
Revised: Unlike

Original: GPT-2, BERT is trained as a masked language model which allows bidirectional self-attention.
Revised: GPT-2, BERT is trained as a masked language model which allows bidirectional self-attention.

Original: We evaluate on the wikitext dataset (Merity et al., 2016), which we partition into articles.
Revised: We evaluate on the wikitext dataset (Merity et al., 2016), which we partition into articles.

Original: We simulate each user by sampling text from an article.
Revised: We simulate each user by sampling text from an article.

Original: We focus on fedSGD, where each user performs a single gradient step and sends a model update to the server.
Revised: We focus on fedSGD, where each user performs a single gradient step and sends a model update to the server.

Original: Note that many related protocols, such as federated averaging (McMahan et al., 2017), can be converted back to fedSGD, if the server is malicious and can control the number of local update steps or local batch size.
Revised: Note that many related protocols, such as federated averaging (McMahan et al., 2017), can be converted back to fedSGD, if the server is malicious and can control the number of local update steps or local batch size.

Original: We perform quantitative evaluations on our method with a range of different metrics.
Revised: We perform quantitative evaluations on our method with a range of different metrics.

Original: We measure success based on BLEU scores (Papineni et al., 2002),
Revised: We measure success based on BLEU scores (Papineni et al., 2002),

Original: ROUGE-L (Lin, 2004), and total accuracy (where we only count exact matches of both token ID and positions ID).
Revised: ROUGE-L (Lin, 2004), and total accuracy (where we only count exact matches of both token ID and positions ID).

Original: We provide additional details in Appendix A.
Revised: We provide additional details in Appendix A.

Original: 4. C OMPARING WITH T RANSFORMER - BASED A TTACKS FOR M ALICIOUS S ERVERS
Revised: 4. C OMPARING WITH T RANSFORMER - BASED A TTACKS FOR M ALICIOUS S ERVERS

Original: We begin by comparing our proposed method with Decepticons (Fowl et al., 2022), a recent transformer-based attack for the malicious server threat model, across various batch size and sequence lengths.
Revised: We begin by comparing our proposed method with Decepticons (Fowl et al., 2022), a recent transformer-based attack for the malicious server threat model, across various batch size and sequence lengths.

Original: To simulate the scenario where only a small number of sequences contain privacycritical information, we select the first 3 sequences from each batch and replace one of the tokens of each sequence with a target keyword.
Revised: To simulate the scenario where only a small number of sequences contain privacycritical information, we select the first 3 sequences from each batch and replace one of the tokens of each sequence with a target keyword.

Original: For experiments on BERT, the replaced token is randomly selected.
Revised: For experiments on BERT, the replaced token is randomly selected.

Original: For experiments on GPT-2, we instead replace one of the first four tokens with a trigger keyword, which allows us to assess the impact of imperfect imprints created by masked selfattention.
Revised: For experiments on GPT-2, we instead replace one of the first four tokens with a trigger keyword, which allows us to assess the impact of imperfect imprints created by masked selfattention.

Original: Due to causal language modeling, an attacker can only recover tokens following the trigger keywords for GPT-2.
Revised: Due to causal language modeling, an attacker can only recover tokens following the trigger keywords for GPT-2.

Original: The quantitative results are then evaluated only on these target sequences.
Revised: The quantitative results are then evaluated only on these target sequences.

Original: We first fix the sequence length, and compare our method with Decepticons, while varying batch size in Figure 5.
Revised: We first fix the sequence length, and compare our method with Decepticons, while varying batch size in Figure 5.

Original: We see that the performance of Decepticons drops significantly as the batch size increases beyond 128 , however the proposed panning attack mechanism is able to maintain a steady reconstruction quality on the targeted sequences as the batch size increases.
Revised: We see that the performance of Decepticons drops significantly as the batch size increases beyond 128 , however the proposed panning attack mechanism is able to maintain a steady reconstruction quality on the targeted sequences as the batch size increases.

Original: In this sense, the new attack really is independent of aggregation size , and rather only depends on the number of keyword occurrences.
Revised: In this sense, the new attack really is independent of aggregation size , and rather only depends on the number of keyword occurrences.

Original: We push this further in the next section.
Revised: We push this further in the next section.

Original: 4. E XPERIMENTS WITH L ARGE -S CALE A GGREGATION AND M ULTIPLE K EYWORDS
Revised: 4. E XPERIMENTS WITH L ARGE -S CALE A GGREGATION AND M ULTIPLE K EYWORDS

Original: Next, we evaluate our proposed methods under more extreme conditions.
Revised: Next, we evaluate our proposed methods under more extreme conditions.

Original: First, we assess the capability of our method to capture relevant sequences under extremely large-scale aggregation.
Revised: First, we assess the capability of our method to capture relevant sequences under extremely large-scale aggregation.

Original: We fix the sequence length to 32 , and experiments with batch size up to 2048 , and summarize the results in
Revised: We fix the sequence length to 32 , and experiments with batch size up to 2048 , and summarize the results in

Original: Figure 6.
Revised: Figure 6.

Original: As demonstrated in the figure, the fidelity of our recovered relevant sequences remains high and is still largely agnostic to batch size.
Revised: As demonstrated in the figure, the fidelity of our recovered relevant sequences remains high and is still largely agnostic to batch size.

Original: This observation validates the danger of the tagging and filtering strategy, which can capture tokens in targeted sequences even against extremely large-scale aggregation.
Revised: This observation validates the danger of the tagging and filtering strategy, which can capture tokens in targeted sequences even against extremely large-scale aggregation.

Original: We note that at this level, it becomes infeasible to even compare to Decepticons, due to the increased computational complexity to run the algorithm proposed therein for 65536 tokens.
Revised: We note that at this level, it becomes infeasible to even compare to Decepticons, due to the increased computational complexity to run the algorithm proposed therein for 65536 tokens.

Original: Additionally, we investigate how the number of keywords K impacts the performance of our attack.
Revised: Additionally, we investigate how the number of keywords K impacts the performance of our attack.

Original: We perform experiments on K ∈ { 1 , 2 , 3 , 4 , 5 , 6 } , We randomly replace a length- K sub-sequence with a length- K target phrase in each of the first 3 sentences.
Revised: We perform experiments on K ∈ { 1 , 2 , 3 , 4 , 5 , 6 } , We randomly replace a length- K sub-sequence with a length- K target phrase in each of the first 3 sentences.

Original: We present results in Figure 7, and demonstrate that our attack is able to recover sequences relevant to a various number of keywords.
Revised: We present results in Figure 7, and demonstrate that our attack is able to recover sequences relevant to a various number of keywords.

Original: We note that both GPT-2 and BERT include 12 heads and allow us to target 11 keywords, which is enough to target most sensitive information in the real world.
Revised: We note that both GPT-2 and BERT include 12 heads and allow us to target 11 keywords, which is enough to target most sensitive information in the real world.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we describe a novel vulnerability for transformers used in a federated setting.
Revised: In this paper, we describe a novel vulnerability for transformers used in a federated setting.

Original: This vulnerability opens the door to an attack through malicious parameter vectors that can “pan” for sequences that contain private information based on specified keywords.
Revised: This vulnerability opens the door to an attack through malicious parameter vectors that can “pan” for sequences that contain private information based on specified keywords.

Original: The attack allows an adversary to accurately capture sentences that contain the keywords out of aggregated updates from thousands of sequences.
Revised: The attack allows an adversary to accurately capture sentences that contain the keywords out of aggregated updates from thousands of sequences.

Original: The attack injects malicious parameters into the transformer whereby sensitive sequences are “tagged” with a unique signature, and filtered to recover these targeted sequences.
Revised: The attack injects malicious parameters into the transformer whereby sensitive sequences are “tagged” with a unique signature, and filtered to recover these targeted sequences.

Original: This attack reflects a notable shift in the capabilities of data reconstruction attacks in federated learning as the attack succeeds under seemingly arbitrary amounts of aggregation.
Revised: This attack reflects a notable shift in the capabilities of data reconstruction attacks in federated learning as the attack succeeds under seemingly arbitrary amounts of aggregation.

Original: E THICS AND M ITIGATIONS S TATEMENT
Revised: E THICS AND M ITIGATIONS S TATEMENT

Original: We describe an attack that significantly extends the capabilities of data extraction attacks for transformer models in federated learning.
Revised: We describe an attack that significantly extends the capabilities of data extraction attacks for transformer models in federated learning.

Original: However, this attack only extends the capabilities of a malicious server and furthers our understanding that privacy cannot be guaranteed against such a server with only secure aggregation.
Revised: However, this attack only extends the capabilities of a malicious server and furthers our understanding that privacy cannot be guaranteed against such a server with only secure aggregation.

Original: Yet, in examples such as the message application discussed in Section 2, the described attack in this paper does not actually extend the capabilities of the server, as the federated learning system considered therein is implemented on a good-will basis without the ability for a user to inspect and vet the implementation of the protocol.
Revised: Yet, in examples such as the message application discussed in Section 2, the described attack in this paper does not actually extend the capabilities of the server, as the federated learning system considered therein is implemented on a good-will basis without the ability for a user to inspect and vet the implementation of the protocol.

Original: From a security perspective, the server is so far not constrained to protocol, and is already capable of downloading user datawithout federated learning.
Revised: From a security perspective, the server is so far not constrained to protocol, and is already capable of downloading user datawithout federated learning.

Original: Both with and without our attack, the server in this example remains with current implementations only legally bounded to minimize their use of user data.
Revised: Both with and without our attack, the server in this example remains with current implementations only legally bounded to minimize their use of user data.

Original: In terms of mitigation opportunities, two lines of defense algorithms are possible.
Revised: In terms of mitigation opportunities, two lines of defense algorithms are possible.

Original: The first line of defense algorithms involves (automatic) inspection of model parameters, and the second line of algorithms leverages differential privacy.
Revised: The first line of defense algorithms involves (automatic) inspection of model parameters, and the second line of algorithms leverages differential privacy.

Original: We provide discussions for each separately.
Revised: We provide discussions for each separately.

Original: Mitigations through Inspection The vulnerability induced via malicious modifications of transformer parameters, as discussed in this work exhibits identifiable characteristics.
Revised: Mitigations through Inspection The vulnerability induced via malicious modifications of transformer parameters, as discussed in this work exhibits identifiable characteristics.

Original: Yet, there are several problems with relying on parameter inspection as a defense to this attack.
Revised: Yet, there are several problems with relying on parameter inspection as a defense to this attack.

Original: First, awareness of this type of attack is necessary to know what to look for in terms of “conspicuous” parameters, which we hope to provide with this work.
Revised: First, awareness of this type of attack is necessary to know what to look for in terms of “conspicuous” parameters, which we hope to provide with this work.

Original: Second, parameter inspection, at least on the server side, is not implemented in any major FL framework (see Bonawitz et al. (2021); Dimitriadis et al.
Revised: Second, parameter inspection, at least on the server side, is not implemented in any major FL framework (see Bonawitz et al. (2021); Dimitriadis et al.

Original: As such, the point of even being able to inspect parameters on a user side would require a user to have “rooted” their device - making this defense currently infeasible for the average user concerned about privacy.
Revised: As such, the point of even being able to inspect parameters on a user side would require a user to have “rooted” their device - making this defense currently infeasible for the average user concerned about privacy.

Original: Finally, inspection defenses naturally put users on the “back foot” in a back-and-forth between ever-adapting attackers and constantly defending users.
Revised: Finally, inspection defenses naturally put users on the “back foot” in a back-and-forth between ever-adapting attackers and constantly defending users.

Original: Both parties need to agree about the implementation of all defenses, so that the defenses used on the user side are always known to the attacker, i.e. the server.
Revised: Both parties need to agree about the implementation of all defenses, so that the defenses used on the user side are always known to the attacker, i.e. the server.

Original: On the other hand though, knowledge of attacks is limited until such vulnerabilities are published.
Revised: On the other hand though, knowledge of attacks is limited until such vulnerabilities are published.

Original: Because of the asymmetry induced by these sorts of defenses, we strongly advise against relying only on fixed sets of parameter inspection rules to avoid attacks like this.
Revised: Because of the asymmetry induced by these sorts of defenses, we strongly advise against relying only on fixed sets of parameter inspection rules to avoid attacks like this.

Original: Mitigations through Differential Privacy We argue that strong user-level differential privacy, i.e. mechanisms applied directly on the user device to all outgoing model updates, remains the defense of choice against the attacks in the untrusted or malicious server threat model that we consider here.
Revised: Mitigations through Differential Privacy We argue that strong user-level differential privacy, i.e. mechanisms applied directly on the user device to all outgoing model updates, remains the defense of choice against the attacks in the untrusted or malicious server threat model that we consider here.

Original: Common differential-privacy-based defense algorithms involve gradient noising and clipping, and we provide additional experiments to evaluate the effectiveness of these defenses Appendix C.
Revised: Common differential-privacy-based defense algorithms involve gradient noising and clipping, and we provide additional experiments to evaluate the effectiveness of these defenses Appendix C.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We perform all our experiments based on the official implementation provided in Fowl et al.
Revised: We perform all our experiments based on the official implementation provided in Fowl et al.

Original: We provide details of parameters and evaluations for our experiments, as well as pre-processing steps for the wikitext dataset in Appendix A. 
Revised: We provide details of parameters and evaluations for our experiments, as well as pre-processing steps for the wikitext dataset in Appendix A. A CKNOWLEDGMENTS

Revised: This work is supported by ONR MURI program, DARPA GARD (HR00112020007), and the National Science Foundation (IIS-2212182 and DMS-1912866).

Revised: Further support was provided by Capital One Bank.

Original: A A DDITIONAL D ETAILS
Revised: A A DDITIONAL D ETAILS

Original: In this section we first describe how to reprogram multiple linear layers as hinted in Section 3.
Revised: In this section, we first describe how to reprogram multiple linear layers as hinted in Section 3.3.

Original: We then discuss variants of our construction for MHSA and linear layers that improve performance in practice.
Revised: We then discuss variants of our construction for MHSA and linear layers that improve performance in practice.

Original: A. E XTENSION TO M ULTIPLE L INEAR L AYERS
Revised: A. E XTENSION TO M ULTIPLE L INEAR L AYERS

Original: Linear layers in encoder blocks other than the first can provide more bins for a finer-grain gradient separation.
Revised: Linear layers in encoder blocks other than the first can provide more bins for a finer-grain gradient separation.

Original: Assuming we have H encoder blocks, We can apply the same construction technique for every W ( h ) of the first linear layer of the h -th encoder block.
Revised: Assuming we have H encoder blocks, We can apply the same construction technique for every W ( h ) of the first linear layer of the h -th encoder block.

Original: For bias, we instead compute b ( h ) m = − Φ − 1 ( Hh + m HM ) , where we essentially concatenate all the bias vectors, assign bin thresholds on the concatenated bias with HM entries, and separate it back to corresponding linear layers.
Revised: For bias, we instead compute b ( h ) m = − Φ − 1 ( Hh + m HM ) , where we essentially concatenate all the bias vectors, assign bin thresholds on the concatenated bias with HM entries, and separate it back to corresponding linear layers.

Original: Additionally, we need to disable MHSA blocks other than the first by setting the output to zero, and also partially disable the feed-forward block of each encoder block by setting all output entries except for the last to zero.
Revised: Additionally, we need to disable MHSA blocks other than the first by setting the output to zero, and also partially disable the feed-forward block of each encoder block by setting all output entries except for the last to zero.

Original: Note that the output of feed-forward cannot be entirely disabled, or the gradient to the first linear layer will become zero.
Revised: Note that the output of feed-forward cannot be entirely disabled, or the gradient to the first linear layer will become zero.

Original: These additional treatments allow inputs of each feed-forward block to stay the same except for the last entry through residual connection.
Revised: These additional treatments allow inputs of each feed-forward block to stay the same except for the last entry through residual connection.

Original: A. A LTERNATIVE C ONSTRUCTION OF MHSA B LOCK AND L INEAR L AYER
Revised: A. A LTERNATIVE C ONSTRUCTION OF MHSA B LOCK AND L INEAR L AYER

Original: Here we provide an alternative construction of the MHSA block.
Revised: Here we provide an alternative construction of the MHSA block.

Original: In particular, we instead use the following,
Revised: In particular, we instead use the following,

Original: [Equation]
Revised: [Equation]

Original: (10) where the key difference is that the new T d ′ is instead a truncated shift matrix that shifts the entries d ′ +1 to 2 d ′ of the input to the first d ′ entries.
Revised: where the key difference is that the new T d ′ is instead a truncated shift matrix that shifts the entries d ′ +1 to 2 d ′ of the input to the first d ′ entries.

Original: By additionally setting the first d ′ entries of each token embedding and position embedding to be zero, the resulting mixed embedding f i,j from Equation (2) will have their first d ′ entries contain only the imprint information.
Revised: By additionally setting the first d ′ entries of each token embedding and position embedding to be zero, the resulting mixed embedding f i,j from Equation (2) will have their first d ′ entries contain only the imprint information.

Original: On the other hand, the original f i,j whose first d ′ entries are a mixture of imprint and original token/positional embedding, To apply this construction we also need set w ′ in Section 3.3 as w ′ =
Revised: On the other hand, the original f i,j whose first d ′ entries are a mixture of imprint and original token/positional embedding, To apply this construction we also need set w ′ in Section 3.3 as w ′ =

Original: T d ′ e ( k ) instead.
Revised: T d ′ e ( k ) instead.

Original: Comparing to the original construction, this alternative provides better performance in practice.
Revised: Comparing to the original construction, this alternative provides better performance in practice.

Original: B A DDITIONAL E XPERIMENT D ETAILS
Revised: B A DDITIONAL E XPERIMENT D ETAILS

Original: In this section we provide additional details for our experiments.
Revised: In this section we provide additional details for our experiments.

Original: We start by detailing hyperparameters.
Revised: We start by detailing hyperparameters.

Original: We set α = 10 12 for GPT-2 and α = 10 8 for BERT across all experiments.
Revised: We set α = 10 12 for GPT-2 and α = 10 8 for BERT across all experiments.

Original: For GPT-2, we set β = 10 for GPT-2 if batch size is smaller than 256 and β = 3 for GPT-2 if otherwise.
Revised: For GPT-2, we set β = 10 for GPT-2 if batch size is smaller than 256 and β = 3 for GPT-2 if otherwise.

Original: For BERT, we set β = 10 for all single keyword experiments, and we set β = 5 K for experiments on multiple keywords, where K is the number of keywords.
Revised: For BERT, we set β = 10 for all single keyword experiments, and we set β = 5 K for experiments on multiple keywords, where K is the number of keywords.

Original: We set d ′ , the block size to be truncated and shifted as described in Section 3.2, to 32 .
Revised: We set d ′ , the block size to be truncated and shifted as described in Section 3.2, to 32 .

Original: Next, we describe our procedure to process articles into sequences of tokens for each user.
Revised: Next, we describe our procedure to process articles into sequences of tokens for each user.

Original: For experiments on GPT-2, the articles are tokenized with GPT-2 (BPE) tokenizer.
Revised: For experiments on GPT-2, the articles are tokenized with GPT-2 (BPE) tokenizer.

Original: For experiments on
Revised: For experiments on

Original: BERT, we tokenize the articles with the original BERT tokenizer.
Revised: BERT, we tokenize the articles with the original BERT tokenizer.

Original: Then, given a sequence length ℓ , we concatenate all the words in an article into one array, and partition the array into sequences of length ℓ .
Revised: Then, given a sequence length ℓ , we concatenate all the words in an article into one array, and partition the array into sequences of length ℓ .

Original: The left-over words are discarded.
Revised: The left-over words are discarded.

Original: Then, given the batch size B , we keep the first B sequences for each user and discard the rest.
Revised: Then, given the batch size B , we keep the first B sequences for each user and discard the rest.

Original: The user then computes their model update based on the remaining B sequences.
Revised: The user then computes their model update based on the remaining B sequences.

Original: For each quantitative metric, the reported result is obtained over the average over the first 10 users with enough data.
Revised: For each quantitative metric, the reported result is obtained over the average over the first 10 users with enough data.

Original: That is, for a given combination of batch size and sequence length, a user only contributes to the final result if his/her data is enough to fill in the required number of tokens.
Revised: That is, for a given combination of batch size and sequence length, a user only contributes to the final result if his/her data is enough to fill in the required number of tokens.

Original: Discussion about FedAVG:
Revised: Discussion about FedAVG:

Original: In this work we focus on fedSGD as a template for federated learning.
Revised: In this work we focus on fedSGD as a template for federated learning.

Original: However, this restriction is not as limiting as it might seem: A malicious server can side-step the restriction of FedAVG if the hyperparameters of the FedAVG protocol can be configured so that either the user runs only a single local update step or runs the update on a large enough batch size.
Revised: However, this restriction is not as limiting as it might seem: A malicious server can side-step the restriction of FedAVG if the hyperparameters of the FedAVG protocol can be configured so that either the user runs only a single local update step or runs the update on a large enough batch size.

Original: Both variants effectively turn FedAVG into FedSGD.
Revised: Both variants effectively turn FedAVG into FedSGD.

Original: C A DDITIONAL E XPERIMENTS FOR D IFFERENTIAL P RIVACY
Revised: C A DDITIONAL E XPERIMENTS FOR D IFFERENTIAL P RIVACY

Original: Here we provide experiments where a user additionally adds Gaussian noise and performs gradient clipping before sending the model update to server.
Revised: Here we provide experiments where a user additionally adds Gaussian noise and performs gradient clipping before sending the model update to server.

Original: Notice that the presence of noise makes the measurement by Eqn.
Revised: Notice that the presence of noise makes the measurement by Eqn.

Original: 7 noisy.
Revised: 7 noisy.

Original: Therefore, we only perform embedding recovery using Equation (7) when the magnitude of difference between bias gradient is larger than a certain threshold.
Revised: Therefore, we only perform embedding recovery using Equation (7) when the magnitude of difference between bias gradient is larger than a certain threshold.

Original: The experiment is performed on a 3-layer transformer as ribed in (Wang et al., 2021), where we clip gradient norm to 1 and compare performance on different noise scales from { 1 e − 6 , 1 e − 7 , 1 e − 8 , 1 e − 9 , 1 e − 10 } , and fix the threshold to be 1 e − 6 .
Revised: The experiment is performed on a 3-layer transformer as ribed in (Wang et al., 2021), where we clip gradient norm to 1 and compare performance on different noise scales from { 1 e − 6 , 1 e − 7 , 1 e − 8 , 1 e − 9 , 1 e − 10 } , and fix the threshold to be 1 e − 6 .

Original: The experiment also uses batch size of 2 and sequence length of 32, where one of the sequences contains the target word.
Revised: The experiment also uses batch size of 2 and sequence length of 32, where one of the sequences contains the target word.

Original: The experiment results are presented in Figure 8.
Revised: The experiment results are presented in Figure 8.

Original: The results show that the attacker is still able to succeed when the noise scale is relatively small, but predictably fails if the gradient becomes sufficiently noisy.
Revised: The results show that the attacker is still able to succeed when the noise scale is relatively small, but predictably fails if the gradient becomes sufficiently noisy.

Original: We also note that defense based on noisy gradient is more likely to succeed against attack on larger transformer models due to the inherently smaller gradient magnitude.
Revised: We also note that defense based on noisy gradient is more likely to succeed against attack on larger transformer models due to the inherently smaller gradient magnitude.

Original: Overall, as promised by theory, applying stronger differential privacy eventually enforces user’s privacy, and we believe related defense algorithms are more practically attractive.
Revised: Overall, as promised by theory, applying stronger differential privacy eventually enforces user’s privacy, and we believe related defense algorithms are more practically attractive.

Original: However, it is also known that stronger differential privacy degrades the utility of the final model.
Revised: However, it is also known that stronger differential privacy degrades the utility of the final model.

Original: Finding a good balance between privacy and utility remains an important research direction.
Revised: Finding a good balance between privacy and utility remains an important research direction.

Original: D A DDITIONAL E XPERIMENTS
Revised: D A DDITIONAL E XPERIMENTS

Original: To verify that the attack is stable across sequence lengths, we run a series of additional experiments, where we fix the batch size and compare across sequence length in Figure 9.
Revised: To verify that the attack is stable across sequence lengths, we run a series of additional experiments, where we fix the batch size and compare across sequence length in Figure 9.

Original: The figure shows that our proposed attack also outperforms Decepticons across all combinations of model and sequence length (although the performance of Decepticons is less sensitive to the change of sequence length).
Revised: The figure shows that our proposed attack also outperforms Decepticons across all combinations of model and sequence length (although the performance of Decepticons is less sensitive to the change of sequence length).