Original: Training a Tokenizer for Free with Private Federated Learning
Revised: Training a Tokenizer for Free with Private Federated Learning

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Federated learning with differential privacy,i.e. private federated learning (PFL), makes itpossible to train models on private data distributed across users’ devices without harmingprivacy.
Revised: Federated learning with differential privacy, i.e. private federated learning (PFL), makes it possible to train models on private data distributed across users’ devices without harming privacy.

Original: PFL is efﬁcient for models, such asneural networks, that have a ﬁxed number ofparameters, and thus a ﬁxed-dimensional gradient vector.
Revised: PFL is efficient for models, such as neural networks, that have a fixed number of parameters, and thus a fixed-dimensional gradient vector.

Original: Such models include neural-netlanguage models, but not tokenizers, the topicof this work.
Revised: Such models include neural-net language models, but not tokenizers, the topic of this work.

Original: Training a tokenizer requiresfrequencies of words from an unlimited vocabulary, and existing methods for ﬁnding anunlimited vocabulary need a separate privacybudget.
Revised: Training a tokenizer requires frequencies of words from an unlimited vocabulary, and existing methods for finding an unlimited vocabulary need a separate privacy budget.

Original: A workaround is to train the tokenizer on publicly available data.
Revised: A workaround is to train the tokenizer on publicly available data.

Original: However, in this paperwe ﬁrst show that a tokenizer trained on mismatched data results in worse model performance compared to a privacy-violating “oracle” tokenizer that accesses user data, with perplexity increasing by 20 %.
Revised: However, in this paper we first show that a tokenizer trained on mismatched data results in worse model performance compared to a privacy-violating “oracle” tokenizer that accesses user data, with perplexity increasing by 20 %.

Original: We also show thatsub-word tokenizers are better suited to thefederated context than word-level ones, sincethey can encode new words, though with moretokens per word.
Revised: We also show that subword tokenizers are better suited to the federated context than word-level ones, since they can encode new words, though with more tokens per word.

Original: Second, we propose a novel method to obtaina tokenizer without using any additional privacy budget.
Revised: Second, we propose a novel method to obtain a tokenizer without using any additional privacy budget.

Original: During private federated learningof the language model, we sample from themodel, train a new tokenizer on the sampledsequences, and update the model embeddings.
Revised: During private federated learning of the language model, we sample from the model, train a new tokenizer on the sampled sequences, and update the model embeddings.

Original: We then continue private federated learning,and obtain performance within 1 % of the “oracle” tokenizer.
Revised: We then continue private federated learning, and obtain performance within 1 % of the “oracle” tokenizer.

Original: Since this process trains the tokenizeronly indirectly onprivate data, we canuse the “postprocessing guarantee” of differential privacy and thus use no additional privacybudget.
Revised: We show that, since this process trains the tokenizer on the server using data for which the privacy loss has already been accounted for, our method spends no additional privacy budget.

Original:  Introduction
Revised:  Introduction

Original: Learning a language model (LM) requires text datathat in many situations is private, resides on people’s devices, and should stay there.
Revised: Learning a language model (LM) requires text data that in many situations is private, resides on people’s devices, and should stay there.

Original: In federatedlearning (McMahan et al., 2017), a central server learns a model by receiving statistics, like parameter updates, from many devices.
Revised: In federated learning (McMahan et al., 2017), a central server learns a model by receiving statistics, like parameter updates, from many devices.

Original: Though devices send only statistics and not the raw data, federated learning by itself can leak information about the data (Shokri et al., 2017; Song et al., 2017).
Revised: Though devices send only statistics and not the raw data, federated learning by itself can leak information about the data (Shokri et al., 2017; Song et al., 2017).

Original: Private federated learning (PFL) (McMahan et al., 2018;
Revised: Private federated learning (PFL) (McMahan et al., 2018;

Original: Geyer et al., 2017) uses differential privacy (Dwork et al., 2006, 2014) to mitigate the privacy leaks by limiting the user’s impact on the ﬁnal model.
Revised: Geyer et al., 2017) uses differential privacy (Dwork et al., 2006, 2014) to mitigate the privacy leaks by limiting the user’s impact on the final model.

Original: It is known how to train neural-net language models using PFL (McMahan et al., 2018).
Revised: It is known how to train neural-net language models using PFL (McMahan et al., 2018).

Original: However, an important part of language modeling is tokenization: turning a text into a sequence of symbols from a ﬁxed-size symbol set.
Revised: However, an important part of language modeling is tokenization: turning a text into a sequence of symbols from a fixed-size symbol set.

Original: To obtain a tokenizer, published research on private federated learning of language models uses either of two approaches, neither of which are satisfactory.
Revised: To obtain a tokenizer, published research on private federated learning of language models uses either of two approaches, neither of which are satisfactory.

Original: One approach is to train the tokenizer on user data directly.
Revised: One approach is to train the tokenizer on user data directly.

Original: The commonly-used LEAF dataset (Caldas et al., 2018) and works relying on it (Li et al., 2021;
Revised: The commonly-used LEAF dataset (Caldas et al., 2018) and works relying on it (Li et al., 2021;

Original: Hu et al., 2021; Yu et al., 2020) assume access to the training data to create the tokenizer.
Revised: Hu et al., 2021; Yu et al., 2020) assume access to the training data to create the tokenizer.

Original: This is not relevant to real-world use cases and undermines user privacy.
Revised: This is not relevant to real-world use cases and undermines user privacy.

Original: The other approach is to use public data to obtain the tokenizer (McMahan et al., 2018).
Revised: The other approach is to use public data to obtain the tokenizer (McMahan et al., 2018).

Original: This is sensible from a privacy perspective, but as we show the resulting distribution mismatch harms performance, resulting in 10%-20% drop compared to using an “oracle” tokenizer trained directly on users’ private data.
Revised: This is sensible from a privacy perspective, but as we show the resulting distribution mismatch harms performance, resulting in 10%-20% drop compared to using an “oracle” tokenizer trained directly on users’ private data.

Original: There are two common types of tokenization, which are affected by mismatched distributions in different ways: word and sub-word tokenization.
Revised: There are two common types of tokenization, which are affected by mismatched distributions in different ways: word and sub-word tokenization.

Original: Figure 1 illustrates these.
Revised: Figure 1 illustrates these.

Original: A word-level tokenizer produces a symbol for each word, and assigns an out-of-vocabulary token (OOV) to any unseen word.
Revised: A word-level tokenizer produces a symbol for each word, and assigns an out-of-vocabulary token (OOV) to any unseen word.

Original: Text from mismatched distributions will generally contain unseen words, which means the correct word cannot be predicted, and the context becomes less meaningful when predicting the“test [Figure] covid”
Revised: Text from mismatched distributions will generally contain unseen words, which means the correct word cannot be predicted, and the context becomes less meaningful when predicting the

Original: [Figure]
Revised: [Figure]

Original: next word.
Revised: next word.

Original: Sub-word tokenization, on the otherhand, splits some words into multiple smaller tokens.
Revised: Sub-word tokenization, on the other hand, splits some words into multiple smaller tokens.

Original: This type of tokenization is generally chosento minimize the average number of tokens per wordon training data.
Revised: This type of tokenization is generally chosen to minimize the average number of tokens per word on training data.

Original: Current centrally trained modelsuse sub-word tokenization such as Byte-Pair Encoding (Sennrich et al., 2016), SentencePiece (Kudoand Richardson, 2018), or WordPieces (Schusterand Nakajima, 2012).
Revised: Current centrally trained models use sub-word tokenization such as Byte-Pair Encoding (Sennrich et al., 2016), SentencePiece (Kudo and Richardson, 2018), or WordPieces (Schuster and Nakajima, 2012).

Original: Nevertheless, mismatchedtokenizations in sub-word methods cause an increase in the number of tokens per word, and thusdecrease the amount of context the model can useto predict the distribution of the next word.
Revised: Nevertheless, mismatched tokenizations in sub-word methods cause an increase in the number of tokens per word, and thus decrease the amount of context the model can use to predict the distribution of the next word.

Original: In this work we present a general framework toapproach training language models in private federated learning by including tokenization as part ofthe training pipeline.
Revised: In this work we present a general framework to approach training language models in private federated learning by including tokenization as part of the training pipeline.

Original: Our contributions are: (1) weuncover the performance gaps when the models usethe tokenizer obtained from a different distributionvs the tokenizer obtained from the underlying distribution.
Revised: Our contributions are: (1) we uncover the performance gaps when the models use the tokenizer obtained from a different distribution vs the tokenizer obtained from the underlying distribution.

Original: For word-level tokenization we show thata tokenizer trained on public data reduces the nextword prediction accuracy of 10–20 % compared toa tokenizer estimated on user data.
Revised: For word-level tokenization we show that a tokenizer trained on public data reduces the nextword prediction accuracy of 10–20 % compared to a tokenizer estimated on user data.

Original: (2) We demonstrate signiﬁcant beneﬁts of switching tokenizersfrom word to sub-word level, thus eliminating theout-of-vocabulary problem.
Revised: (2) We demonstrate significant benefits of switching tokenizers from word to sub-word level, thus eliminating the out-of-vocabulary problem.

Original: (3) We propose a newmethod that samples data from an existing model,e.g. from the prior PFL run, and uses that data toinitialize a new tokenizer.
Revised: (3) We propose a new method that samples data from an existing model, e.g. from the prior PFL run, and uses that data to initialize a new tokenizer.

Original: Our approach can update the tokenizer between iterations of the same PFL run by modifying model embeddings withnew tokenizations and signiﬁcantly boosting performance.
Revised: Our approach can update the tokenizer between iterations of the same PFL run by modifying model embeddings with new tokenizations and significantly boosting performance.

Original: Crucially, since the language model istrained with differential privacy, the “postprocessing guarantee” of differential privacy means thattraining the tokenizer with our approach does notuse any additional privacy budget.
Revised: Crucially, since the additional processing is done entirely on the server, training the tokenizer with our approach does not use any additional privacy budget.

Original:  Private federated learning
Revised:  Private federated learning

Original: Machine-learned models work best if they are trained on the correct distribution of the data, in this paper text data.
Revised: Machine-learned models work best if they are trained on the correct distribution of the data, in this paper text data.

Original: In many scenarios text data is private and contained on people’s devices, and should stay there.
Revised: In many scenarios text data is private and contained on people’s devices, and should stay there.

Original: To train a global model without harming privacy, we use federated learning (McMahan et al., 2017) with differential privacy (Dwork et al., 2006, 2014).
Revised: To train a global model without harming privacy, we use federated learning (McMahan et al., 2017) with differential privacy (Dwork et al., 2006, 2014).

Original: Federated learning involves devices sending not the data, but statistics, e.g. model gradients, computed on that data.
Revised: Federated learning involves devices sending not the data, but statistics, e.g. model gradients, computed on that data.

Original: To train neural networks, the standard algorithm is federated averaging (McMahan et al., 2017).
Revised: To train neural networks, the standard algorithm is federated averaging (McMahan et al., 2017).

Original: At each iteration t , the server randomly selects a subset of m participants S m and distributes the current global model M t .
Revised: At each iteration t , the server randomly selects a subset of m participants S m and distributes the current global model M t .

Original: Each participant takes a number of gradient steps to train on their private data and submits the sum G ti of the gradients to the server.
Revised: Each participant takes a number of gradient steps to train on their private data and submits the sum G ti of the gradients to the server.

Original: The server takes a step (with step size η ) in the direction of the average gradient to create the new global model:
Revised: The server takes a step (with step size η ) in the direction of the average gradient to create the new global model:

Original: [Equation]
Revised: [Equation]

Original: 2. Federated Learning with Differential 
Revised: 2. Federated Learning with Differential Privacy

Original: Privacy

Original: The global model M t +1 might still reveal private information including user participation in training (Shokri et al., 2017; Song et al., 2017; Melis et al., 2019).
Revised: The global model M t +1 might still reveal private information including user participation in training (Shokri et al., 2017; Song et al., 2017; Melis et al., 2019).

Original: To mitigate this threat, we can combine federated learning with differential privacy (DP) (Dwork et al., 2006, 2014), to give private federate learning (McMahan et al., 2018).
Revised: To mitigate this threat, we can combine federated learning with differential privacy (DP) (Dwork et al., 2006, 2014), to give private federate learning (McMahan et al., 2018).

Original: Differential privacy gives a strong guarantee: it limits the advantage that a computationally unconstrained adversary has in inferring whether an individual’s data is contained in the data set that the statistics are computed from.
Revised: Differential privacy gives a strong guarantee: it limits the advantage that a computationally unconstrained adversary has in inferring whether an individual’s data is contained in the data set that the statistics are computed from.

Original: ( (cid:15), δ ) -differential privacy parametrizes this advantage by (cid:15) (the maximum privacy loss) and δ (a slack term).
Revised: ( ϵ, δ ) -differential privacy parametrizes this advantage by ϵ (the maximum privacy loss) and δ (a slack term).

Original: The common mechanism to provide differential privacy in a federated learning setting is the Gaussian mechanism that uses the moments accountant (Abadi et al., 2016).
Revised: The common mechanism to provide differential privacy in a federated learning setting is the Gaussian mechanism that uses the moments accountant (Abadi et al., 2016).

Original: For each participant, the model parameters are clipped to a norm S , i.e., multiplied by min (1 , S/ (cid:107) G t (cid:107) 2 ) , to bound the sum’s sensitivity to any individual’s data.
Revised: Each participant clips its gradients to a norm S , i.e., multiplied by min (1 , S/ ∥ G t ∥ 2 ) , to bound the sum’s sensitivity to any individual’s data.

Original: Second, Gaussian noise N (0 , σ 2 ) is added to theﬁnal sum.
Revised: Second, Gaussian noise N (0 , σ 2 ) is added to the sum.

Original:  How much privacy budget is spent  depends on the variance σ 2 relative to the magnitude of individual updates, the total population,thenumber of contributions in each iteration, and thetotal number of iterations (for more details, see 
Revised: 1 How much privacy budget is spent in one iteration depends on the variance σ 2 relative to the magnitude of individual updates, the total population, and the number of contributions (for more details, see McMahan et al., 2018; Balle et al., 2018).

Original: McMahan et al., 2018; Balle et al., 2018).
Revised: The moments accountant keeps track of this in terms of the Rényi differential privacy (Mironov, 2017). What is learned in one iteration is allowed to affect the query in the next iteration, and this increases the budget (in terms of Rényi DP) merely linearly. This is called adaptive composition , and it is crucial both to standard private federate learning (where the model changes every iteration as in (1)) and to the method we propose.

Original:  Privately ﬁnding vocabulary items
Revised: 2. Privately finding vocabulary items

Original: Central differential privacy with the Gaussianmechanism and the moments accountant is efﬁcient in terms of utility vs privacy loss, but it doescome with restrictions.
Revised: Central differential privacy with the Gaussian mechanism and the moments accountant is efficient in terms of utility vs privacy loss, but it does come with restrictions.

Original: The sum of individual contributions, which the noise is added to, must be ofﬁnite and ﬁxed size.
Revised: The sum of individual contributions, which the noise is added to, must be of finite and fixed size.

Original: This is not a problem for training neural networks.
Revised: This is not a problem for training neural networks.

Original: However, training a tokenizerrequires frequencies for an exponential-size set ofsequences, as does training a traditional N -grammodel.
Revised: However, training a tokenizer requires frequencies for an exponential-size set of sequences, as does training a traditional N -gram model.

Original: Differentially private algorithms to compute histograms over sets of elements (e.g. words)distributed over devices are called “heavy hitters”algorithms (Bassily et al., 2017; Zhu et al., 2020;
Revised: Differentially private algorithms to compute histograms over sets of elements (e.g. words) distributed over devices are called “heavy hitters” algorithms (Bassily et al., 2017; Zhu et al., 2020;

Original: Apple, 2017).
Revised: Apple, 2017).

Original: These algorithms require a separate and large privacy budget.
Revised: These algorithms require a separate and large privacy budget.

Original: In section 5 we willcompare with a heavy hitters algorithm.
Revised: In section 5 we will compare with a heavy hitters algorithm.

Original: Another way of ﬁnding vocabulary items privately is to train a neural-net generative model.
Revised: Another way of finding vocabulary items privately is to train a neural-net generative model.

Original: Beaufays et al.
Revised: Beaufays et al.

Original: (2019) trains a separate, character-level
Revised: (2019) trains a separate, character-level

Original: LSTM model to generate the new words.
Revised: LSTM model to generate the new words.

Original: However, the proposed method is only shown to workfor discover OOV s in a word-level model and alsorequires separate training and a privacy budget.
Revised: However, the proposed method is only shown to work for discover OOV s in a word-level model and also requires separate training and a privacy budget.

Original:  Tokenization in Language Modeling
Revised:  Tokenization in Language Modeling

Original: A language model is a model that assigns probabilities to sequences of tokens.
Revised: A language model is a model that assigns probabilities to sequences of tokens.

Original: In this paper, itis always an autoregressive model with parameters θ : P θ ( s ) =
Revised: In this paper, it is always an autoregressive model with parameters θ : P θ ( s ) =

Original: P θ ( t 2 | t 1 = BOS ) · P θ ( t 3 | t 1 =
Revised: P θ ( t 2 | t 1 = BOS ) · P θ ( t 3 | t 1 =

Original: BOS , t 2 ) · · · P θ ( t n = EOS | t 1 = BOS , . . .
Revised: BOS , t 2 ) · · · P θ ( t n = EOS | t 1 = BOS , . . .

Original: , t n − 1 ) ,where each term in this equation is normalizedover all possible values of the current token.
Revised: , t n − 1 ) , where each term in this equation is normalized over all possible values of the current token.

Original: Localnormalization is useful when decoding input, likein speech recognition or a keyboard (Hard et al.,2018).
Revised: Local normalization is useful when decoding input, like in speech recognition or a keyboard (Hard et al., 2018).

Original: For this paper, we assume that a corpus issegmented into sentences.
Revised: For this paper, we assume that a corpus is segmented into sentences.

Original: A tokenizer τ then converts each sentence s in the dataset into a sequenceof n tokens τ ( s ) =
Revised: A tokenizer τ then converts each sentence s in the dataset into a sequence of n tokens τ ( s ) =

Original: [ BOS , t 2 , .., t n − 1 , EOS ] , whichis fed into the language model.
Revised: [ BOS , t 2 , .., t n − 1 , EOS ] , which is fed into the language model.

Original: There are two typesof tokenization, highlighted in Figure 1: word-leveland sub-word-level.
Revised: There are two types of tokenization, highlighted in Figure 1: word-level and sub-word-level.

Original: Using a sub-word tokenizerwill be key to the algorithm this paper proposes.
Revised: Using a sub-word tokenizer will be key to the algorithm this paper proposes.

Original: The next section will discuss the two typesof tokenizers and their consequences for out-ofvocabulary tokens and the performance of language models based in them.
Revised: The next section will discuss the two types of tokenizers and their consequences for out-ofvocabulary tokens and the performance of language models based in them.

Original: Section 3.2 will discuss the complex topic of how to compare performance across different tokenizations.
Revised: Section 3.2 will discuss the complex topic of how to compare performance across different tokenizations.

Original: 3. Word-level vs sub-word-level 
Revised: 3. Word-level vs sub-word-level tokenization

Original: tokenization

Original: The type of tokenization that papers about language models in federated learning commonly use is word-level tokenization (McMahan et al., 2017).
Revised: The type of tokenization that papers about language models in federated learning commonly use is word-level tokenization (McMahan et al., 2017).

Original: For a vocabulary of size N the tokenizer assigns a unique token for top- N most popular words in the dataset while other words receive an out-ofvocabulary token OOV , as highlighted in Figure 1.
Revised: For a vocabulary of size N the tokenizer assigns a unique token for top- N most popular words in the dataset while other words receive an out-ofvocabulary token OOV , as highlighted in Figure 1.

Original: Some papers (e.g. McMahan et al., 2018) build the tokenizer from a publicly available dataset, others including the LEAF benchmark (Caldas et al., 2018) build the tokenizer from users’ training data.
Revised: Some papers (e.g. McMahan et al., 2018) build the tokenizer from a publicly available dataset, others including the LEAF benchmark (Caldas et al., 2018) build the tokenizer from users’ training data.

Original: OOV tokens in the word history make it harder for a language model to predict the next word.
Revised: OOV tokens in the word history make it harder for a language model to predict the next word.

Original: The other type of tokenization is sub-word tokenization, for which there are two popular schemes: byte-pair encoding (BPE) (Sennrich et al., 2016) and WordPieces (Schuster and Nakajima, 2012).
Revised: The other type of tokenization is sub-word tokenization, for which there are two popular schemes: byte-pair encoding (BPE) (Sennrich et al., 2016) and WordPieces (Schuster and Nakajima, 2012).

Original: We focus on BPE which unlike WordPieces guarantees the absence of OOVs as there exists a token for every byte.
Revised: We focus on BPE which unlike WordPieces guarantees the absence of OOVs as there exists a token for every byte.

Original: However, the number of tokens required to encode each word can change signiﬁcantly depending on the dataset that the tokenizer was trained on.
Revised: However, the number of tokens required to encode each word can change significantly depending on the dataset that the tokenizer was trained on.

Original: As highlighted in Figure 1, a tokenizer trained on data from before the COVIDpandemic would generate multiple tokens for the word “covid”.
Revised: As highlighted in Figure 1, a tokenizer trained on data from before the COVIDpandemic would generate multiple tokens for the word “covid”.

Original: Generating longer token sequences makes it harder for the language model to keep track of the context, degrading its performance.
Revised: Generating longer token sequences makes it harder for the language model to keep track of the context, degrading its performance.

Original: Even LSTMs and transformers, which in theory can use arbitrarily long history, have imperfect memory.
Revised: Even LSTMs and transformers, which in theory can use arbitrarily long history, have imperfect memory.

Original: 3. Evaluating language models across 
Revised: 3. Evaluating language models across tokenizations

Original: tokenizations

Original: Comparing language models across tokenizations is a complex problem.
Revised: Comparing language models across tokenizations is a complex problem.

Original: For example, when comparing word-level language models using perplexity, often OOVs are ignored which gives an edge to the language model with more OOVs, which is the opposite of what is desired.
Revised: For example, when comparing word-level language models using perplexity, often OOVs are ignored which gives an edge to the language model with more OOVs, which is the opposite of what is desired.

Original: The following sections detail the problems when comparing sub-word language models.
Revised: The following sections detail the problems when comparing sub-word language models.

Original: 3.2. Comparing 
Revised: 3.2. Comparing word-level with sub-word

Original: word -level with sub-word

Original: Since a word-level language model has a closedvocabulary, it outputs probabilities only on invocabulary words, artiﬁcially lowering the perplexity of closed-vocabulary LMs, particularly on datawith a large number of OOVs.
Revised: Since a word-level language model has a closed vocabulary, it outputs probabilities only on invocabulary words, artificially lowering the perplexity of closed-vocabulary LMs, particularly on data with a large number of OOVs.

Original: Removing thosesame words in evaluating a sub-word languagemodel, would disadvantage it.
Revised: Removing those same words in evaluating a sub-word language model, would disadvantage it.

Original: A better alternative, which this paper will use,is to compare model performance the word-levelaccuracy.
Revised: A better alternative, which this paper will use, is to compare model performance the word-level accuracy.

Original: The most accurate way would be to ﬁndthe word with the highest probability by summingover sequences of tokens.
Revised: The most accurate way would be to find the word with the highest probability by summing over sequences of tokens.

Original: However, we choose asimpler, though less accurate method (similar to
Revised: However, we choose a simpler, though less accurate method (similar to

Original: Likhomanenko et al., 2019): repeatedly generatethe best tokens within each word’s bounds and onlyaccept the word as accurate if all generated tokenswere correct.
Revised: Likhomanenko et al., 2019): repeatedly generate the best tokens within each word’s bounds and only accept the word as accurate if all generated tokens were correct.

Original: 3.2. Comparing sub-word with sub-word
Revised: Comparing sub-word with sub-word

Original: It is possible to meaningfully compare perplexitiesof two language models with different sub-wordtokenizations (Mielke, 2019).
Revised: It is possible to meaningfully compare perplexities of two language models with different sub-word tokenizations (Mielke, 2019).

Original: Though the languagemodel assigns probability mass to all token sequences, a single sentence can have multiple corresponding token sequences, only one of which willbe chosen by the tokenizer.
Revised: Though the language model assigns probability mass to all token sequences, a single sentence can have multiple corresponding token sequences, only one of which will be chosen by the tokenizer.

Original: Some of the probabilitymass will therefore be lost to never-occurring tokensequences.
Revised: Some of the probability mass will therefore be lost to never-occurring token sequences.

Original: However, it is unfeasible to sum overall token sequences (Likhomanenko et al., 2019).
Revised: However, it is unfeasible to sum over all token sequences (Likhomanenko et al., 2019).

Original: The danger with comparing perplexities directlyis that since models with different tokenizers operate on different sets of tokens the number of tokensneeded to encode each sentence is different in general (Mielke, 2019).
Revised: The danger with comparing perplexities directly is that since models with different tokenizers operate on different sets of tokens the number of tokens needed to encode each sentence is different in general (Mielke, 2019).

Original: Nevertheless, note that allmodels assign a probability to a sentence (with theapproximation above).
Revised: Nevertheless, note that all models assign a probability to a sentence (with the approximation above).

Original: To compute the perplexityin such a way that it can be compared across tokenizers, use the same denominator in computingthe perplexity: the number of words in the sentenceinstead of number of tokens, which depends on thetokenizer.
Revised: To compute the perplexity in such a way that it can be compared across tokenizers, use the same denominator in computing the perplexity: the number of words in the sentence instead of number of tokens, which depends on the tokenizer.

Original: Therefore we deﬁne the perplexity as:ppl
Revised: Therefore we define the perplexity as:

Original: [Equation]
Revised: [Equation]

Original: (cid:107) s (cid:107) [Equation] (cid:19) (2)where (cid:107) s (cid:107) w counts the number of words in thesentence s .
Revised: where ∥ s ∥ w counts the number of words in the sentence s .

Original: To generalize from a single sentenceto a dataset, replace s with the concatenation of allsentences in the dataset.
Revised: To generalize from a single sentence to a dataset, replace s with the concatenation of all sentences in the dataset.

Original:  Learning a Tokenizer with Private 
Revised:  Learning a Tokenizer with Private Federated Learning

Original: Federated Learning

Original: Problem deﬁnition.
Revised: Problem definition.

Original: We aim to obtain a tokenizer that works well on users’ federated data without compromising user privacy.
Revised: We aim to obtain a tokenizer that works well on users’ federated data without compromising user privacy.

Original: First, we aim to ﬁnd the appropriate tokenization scheme, and second, given the tokenization scheme obtain the right approximation of user data to train the tokenizer.
Revised: First, we aim to find the appropriate tokenization scheme, and second, given the tokenization scheme obtain the right approximation of user data to train the tokenizer.

Original: Setting We focus on a common application of federated learning: training a language model, parameterized by θ , using federated learning with differential privacy.
Revised: Setting. We focus on a common application of federated learning: training a language model, parameterized by θ , using federated learning with differential privacy.

Original: In our setting each user u i has a dataset d i of private texts from a private distribution of user data D .
Revised: In our setting, each user u i has a dataset d i of private texts from a private distribution of user data D .

Original: The trained model will be evaluated against a held-out dataset D test , e.g. a mix of all user data, which in practice must be replaced by federated evaluation.
Revised: The trained model will be evaluated against a held-out dataset D test , e.g. a mix of all user data, which in practice must be replaced by federated evaluation.

Original: We assume that the central server does not have access to the user data distribution D and can only approximate it with the publicly available dataset
Revised: We assume that the central server does not have access to the user data distribution D and can only approximate it with the publicly available dataset

Original: D pub .
Revised: D pub .

Original: We assume the public data is some commonly available dataset, such as Wikipedia (Merity et al., 2017).
Revised: We assume the public data is some commonly available dataset, such as Wikipedia (Merity et al., 2017).

Original: The tokenizer trained on this public data will be τ pub .
Revised: The tokenizer trained on this public data will be τ pub .

Original: For comparison we assume the existence of an oracle tokenizer τ o initialized on users’ training data D .
Revised: For comparison we assume the existence of an oracle tokenizer τ o initialized on users’ training data D .

Original: Papers that study language models in federated learning commonly use word-level tokenization.
Revised: Papers that study language models in federated learning commonly use word-level tokenization.

Original: While some papers (e.g. McMahan et al., 2018), build the vocabulary using publicly available dataset, others (e.g. Yu et al., 2020; Caldas et al., 2018) explicitly use the federated training data, even though in real-world scenarios the analogous data would be unavailable and it violates privacy guarantees when used in PFL (Li et al., 2021).
Revised: While some papers (e.g. McMahan et al., 2018), build the vocabulary using publicly available dataset, others (e.g. Yu et al., 2020; Caldas et al., 2018) explicitly use the federated training data, even though in real-world scenarios the analogous data would be unavailable and it violates privacy guarantees when used in PFL (Li et al., 2021).

Original: 4. Sampling from a PFL-trained language 
Revised: 4. Sampling from a PFL-trained language model

Original: model

Original: To address the problem of learning a good tokenizer we ﬁrst propose to use a sub-word tokenizer with an open vocabulary.
Revised: To address the problem of learning a good tokenizer we first propose to use a sub-word tokenizer with an open vocabulary.

Original: This allows the language model trained with such a tokenizer to represent any word, if inefﬁciently.
Revised: This allows the language model trained with such a tokenizer to represent any word, if inefficiently.

Original: It is then possible to query the language model to ﬁnd new words as the model can utilize this open vocabulary.
Revised: It is then possible to query the language model to find new words as the model can utilize this open vocabulary.

Original: This is the core of the Algorithm 1 that this paper introduces.
Revised: This is the core of the Algorithm 1 that this paper introduces.

Original: Figure 2 shows the proposed pipeline.
Revised: Figure 2 shows the proposed pipeline.

Original: A language model is trained with private federated learning.
Revised: A language model is trained with private federated learning.

Original: This results (on the left) in a model matched with an old, stale tokenizer.
Revised: This results (on the left) in a model matched with an old, stale tokenizer.

Original: The next block queriesthe language model to produce a better tokenizer,with a method that section 4.2 will detail.
Revised: The next block queries the language model to produce a better tokenizer, with a method that section 4.2 will detail.

Original: The blockafter that updates the language model for the newtokenizer, using reasonable guesses for the newparameters.
Revised: The block after that updates the language model for the new tokenizer, using reasonable guesses for the new parameters.

Original: This results in a new LM-tokenizercombination that can be trained further with PFL.
Revised: This results in a new LM-tokenizer combination that can be trained further with PFL.

Original: We assume that the language model obtainedwith the stale tokenizer is trained with a certainprivacy budget.

Original: The postprocessing guarantee ofdifferential privacy(Dwork, 2011) means that thesteps other than private federated learning do notconsume any further budget.
Revised: Adaptive composition (see Mironov, 2017) of differential privacy makes it possible to run a serverside process between iterations without spending additional privacy budget.

Original: The function U PDATEin Algorithm 1 performs the on-server steps.
Revised: The function U PDATE in Algorithm 1 performs the on-server steps.

Original: Thefollowing sections will give more detail.
Revised: The following sections will give more detail.

Original: 4. New tokenizer from a trained LM
Revised: 4. New tokenizer from a trained LM

Original: Training a tokenizer requires text data.
Revised: Training a tokenizer requires text data.

Original: Since theraw data is not available, we propose to instead sample from the LM matched with the stale tokenizer,as detailed in Algorithm 1.
Revised: Since the raw data is not available, we propose to instead sample from the LM matched with the stale tokenizer, as detailed in Algorithm 1.

Original: The S AMPLE T OKENSfunction samples from the language model, drawing sequences of tokens according to the probabilities that the model assigns to them.
Revised: The S AMPLE T OKENS function samples from the language model, drawing sequences of tokens according to the probabilities that the model assigns to them.

Original: The S AMPLEfunction then converts these sequences in the old tokenization into word sequences, by decoding withτ pub .
Revised: The S AMPLE function then converts these sequences in the old tokenization into word sequences, by decoding with τ pub .

Original: Once a large enough corpus of word-levelsentences has been produced, training a tokenizerproceeds as normally (the T RAIN T OKENIZER function is not speciﬁed).
Revised: Once a large enough corpus of word-level sentences has been produced, training a tokenizer proceeds as normally (the T RAIN T OKENIZER function is not specified).

Original: 4. Adapting the language model to the new 
Revised: 4. Adapting the language model to the new tokenizer

Original: tokenizer

Original: After a new tokenizer τ has been trained, the language model, trained with τ pub , must be updatedto work with the new tokenizer.
Revised: After a new tokenizer τ has been trained, the language model, trained with τ pub , must be updated to work with the new tokenizer.

Original: Neural-net language models use an embedding layer to convertthe provided tokens into multi-dimensional vectors.
Revised: Neural-net language models use an embedding layer to convert the provided tokens into multi-dimensional vectors.

Original: It is the embedding vectors that are most importantto modify when changing the tokenization.
Revised: It is the embedding vectors that are most important to modify when changing the tokenization.

Original: The rest of the model only consumes the embedding vector.
Revised: The rest of the model only consumes the embedding vector.

Original: It is not possible to ﬁnd the optimal parameters without further training of both embeddings and other layers, but we propose an algorithm to ﬁnd a reasonable starting point, in the function
Revised: It is not possible to find the optimal parameters without further training of both embeddings and other layers, but we propose an algorithm to find a reasonable starting point, in the function

Original: R EMAP ( τ, τ pub ) in Algorithm 1.
Revised: R EMAP ( τ, τ pub ) in Algorithm 1.

Original: R EMAP iterates over the tokens from the new tokenizer τ and creates the mapping from the tokens’ embedding in the public tokenizer τ pub to the new token’s embedding.
Revised: R EMAP iterates over the tokens from the new tokenizer τ and creates the mapping from the tokens’ embedding in the public tokenizer τ pub to the new token’s embedding.

Original: In some cases it is a one-toone mapping, but when the new token accumulates multiple tokens in τ pub we split the weight equally between each token.
Revised: In some cases it is a one-toone mapping, but when the new token accumulates multiple tokens in τ pub we split the weight equally between each token.

Original: Once we have the mapping map we modify the embedding layer of the model by performing matrix multiplication, i.e. θ.
Revised: Once we have the mapping map we modify the embedding layer of the model by performing matrix multiplication, i.e. θ.

Original: embedding = map · θ. embedding .
Revised: embedding = map · θ. embedding .

Original: The resulting model can accept the tokens from the new tokenizer τ , and can participate in future training in federated learning.
Revised: The resulting model can accept the tokens from the new tokenizer τ , and can participate in future training in federated learning.

Original:  Experiments
Revised:  Experiments

Original: We evaluate our approach by ﬁrst looking at performance of tokenizers trained on the distributions matched and mismatched to real data, we then test the proposed federated sampling on different datasets for federated learning.
Revised: We evaluate our approach by first looking at performance of tokenizers trained on the distributions matched and mismatched to real data, we then test the proposed federated sampling on different datasets for federated learning.

Original: 5. Experimental setup.
Revised: 5. Experimental setup

Original: We use two datasets common in the federated learning literature (Kairouz et al., 2019).
Revised: We use two datasets common in the federated learning literature (Kairouz et al., 2019).

Original: While both use English, there is nothing about our experiments that is speciﬁc to this language, and multilingual datasets can further beneﬁt from using SentencePiece tokenization (Kudo and Richardson, 2018),.
Revised: While both use English, there is nothing about our experiments that is specific to this language, and multilingual datasets can further benefit from using SentencePiece tokenization (Kudo and Richardson, 2018).

Original: • Reddit data – this dataset is taken from the LEAF benchmark (Caldas et al.
Revised: • Reddit data – this dataset is taken from the LEAF benchmark (Caldas et al.,

Original: and
Revised: and

Original: contains over a million users that have multiple posts on the Reddit platform.

Original: As proposed

Original: Algorithm
Revised: Algorithm

Original: Inputs: model θ , current sentence s , new tokenizer τ , public tokenizer τ pub , size of the sampled dataset corpus _ size .
Revised: Inputs: model θ , current sentence s , new tokenizer τ , public tokenizer τ pub , size of the sampled dataset corpus _ size .

Original: function S AMPLE T OKENS ( θ, s ) t next ∼ θ t k | s if t next =
Revised: function S AMPLE T OKENS ( θ, s ) t next ∼ θ t k | s if t next =

Original: EOS then return s ++ t next else return S AMPLE T OKENS ( θ, s ++ t next ) function S AMPLE ( θ, τ ) return τ.
Revised: EOS then return s ++ t next else return S AMPLE T OKENS ( θ, s ++ t next ) function S AMPLE ( θ, τ ) return τ.

Original: decode(
Revised: decode(

Original: S AMPLE T OKENS ( θ, [ BOS ])) function R EMAP ( τ pub , τ ) map =
Revised: S AMPLE T OKENS ( θ, [ BOS ])) function R EMAP ( τ pub , τ ) map =

Original: zeros( τ. size , τ pub .
Revised: zeros( τ. size , τ pub .

Original: size) for token , tid ← τ. vocab do tokens
Revised: size) for token , tid ← τ. vocab do tokens

Original: = τ pub .
Revised: = τ pub .

Original: decode(token) for token ← tokens do tid pub =
Revised: decode(token) for token ← tokens do tid pub =

Original: τ pub .
Revised: τ pub .

Original: vocab[token] map[tid pub , tid] = 1 / len(tokens) return map function U PDATE ( θ, τ pub ) while len(corpus) < corpus _ size do corpus ← S AMPLE ( θ, ∅ , l max ) τ
Revised: vocab[token] map[tid pub , tid] = 1 / len(tokens) return map function U PDATE ( θ, τ pub ) while len(corpus) < corpus _ size do corpus ← S AMPLE ( θ, ∅ , l max ) τ

Original: = T RAIN T OKENIZER (corpus) map =
Revised: = T RAIN T OKENIZER (corpus) map =

Original: R EMAP ( τ pub , τ ) θ.
Revised: R EMAP ( τ pub , τ ) θ.

Original: embedding = map · θ.
Revised: embedding = map · θ.

Original: embedding return θ, τ
Revised: embedding return θ, τ

Original: by
Revised: contains over a million users that have multiple posts on the Reddit platform.

Original: LEAF , we limit each user to contain at most 1600 tokens and use 10 % of users for faster training
Revised: As proposed by LEAF, we limit each user to contain at most 1600 tokens and use 10 % of users for faster training.

Original: • StackOverﬂow data – this data is taken from Kaggle (Kaggle, 2021) and processed with the TensorFlow Federated framework.
Revised: • StackOverflow data – this data is taken from Kaggle (Kaggle, 2021) and processed with the TensorFlow Federated framework.

Original: The train split of the dataset contains 342k users and we select at most 1600 tokens per user.
Revised: The train split of the dataset contains 342k users and we select at most 1600 tokens per user.

Original: Model parameters.
Revised: Model parameters.

Original: We use an LSTM model with

Original:  3 layers, and total parameters of 14M. We also use a Transformer language model (Vaswani et al., 2017) with 6 layers and the same total number of parameters as the LSTM (see Appendix A).
Revised: We use an LSTM model with 3 layers, and total parameters of 14M. We also use a Transformer language model (Vaswani et al., 2017) with 6 layers and the same total number of parameters as the LSTM (see Appendix A).

Original: Each model is trained from scratch.
Revised: Each model is trained from scratch.

Original: Hyper-parameters.
Revised: Hyper-parameters

Original: We set the privacy budgetto
Revised: We set the privacy budget

Original: (cid:15)
Revised: to ϵ

Original: = 2 and δ = 10 −
Revised: = 2 and δ = 10 −

Original: – a common privacyregime (Kairouz et al., 2019).
Revised: a common privacy regime (Kairouz et al., 2019).

Original: For the “heavy hitters” baseline we use local DP with an additionalprivacy budget of (cid:15) = 8 . {+
+}
Revised: For the “heavy hitters” baseline we use local DP with an additional privacy budget of ϵ = 8 . {+
+}

Original: The overall population for the moments accountant is assumed to be 10m.
Revised: The overall population for the moments accountant is assumed to be 10m.

Original: We use a cohort size of 20 , 000 for each round and train all models for 5 , 000 iterations.
Revised: We use a cohort size of 20 , 000 for each round and train all models for 5 , 000 iterations.

Original: We use
Revised: We use

Original: Adam (Kingma and Ba, 2015) for central optimization with learning rate set to 0.5.
Revised: Adam (Kingma and Ba, 2015) for central optimization with learning rate set to 0.5.

Original: For the clients we use SGD and train for 1 local epoch with batch size set to 16 and local learning rate set to 0.1, and an L 2 clipping bound for DP of 0 . 5 .
Revised: For the clients we use SGD and train for 1 local epoch with batch size set to 16 and local learning rate set to 0.1, and an L 2 clipping bound for DP of 0 . 5 .

Original: Vocabulary size.
Revised: Vocabulary size.

Original: We assume that the tokenizer has a moderate vocabulary size such as 10,000 tokens (we experiment with larger vocabularies in Appendix A).
Revised: We assume that the tokenizer has a moderate vocabulary size such as 10,000 tokens (we experiment with larger vocabularies in Appendix A).

Original: Smaller vocabularies reduce model size and, therefore, might be better for deployment on devices and communication with the global server.
Revised: Smaller vocabularies reduce model size and, therefore, might be better for deployment on devices and communication with the global server.

Original: Tokenizer details.
Revised: Tokenizer details.

Original: To train an initial tokenizer  we use a popular and public Wikipedia dataset (Merity et al., 2017).
Revised: To train an initial tokenizer (on the server) we use a popular and public Wikipedia dataset (Merity et al., 2017).

Original: It may seem like the distribution of Wikipedia data is artiﬁcially far from the distributions of Reddit and StackOverﬂow data.
Revised: It may seem like the distribution of Wikipedia data is artificially far from the distributions of Reddit and StackOverflow data.

Original: However, the server might not have the right prior possibly due to a natural distribution shift (Miller et al., 2020) of typed texts (such as an emerging topic of which there were plenty recently).
Revised: However, the server might not have the right prior possibly due to a natural distribution shift (Miller et al., 2020) of typed texts (such as an emerging topic of which there were plenty recently).

Original: We use BPE and WordLevel tokenization algorithms from the HuggingFace Tokenizer library (Huggingface, 2021).
Revised: We use BPE and WordLevel tokenization algorithms from the HuggingFace Tokenizer library (Huggingface, 2021).

Original: Each user post is surrounded by special tokens BOS and EOS .
Revised: Each user post is surrounded by special tokens BOS and EOS .

Original: We also tried
Revised: We also tried

Original: WordPieces tokenization which has slightly better performance than BPE but cannot encode all words and is therefore less applicable in FL.
Revised: WordPieces tokenization which has slightly better performance than BPE but cannot encode all words and is therefore less applicable in FL.

Original: Note on splitting data.
Revised: Note on splitting data.

Original: Whereas the original LEAF dataset for Reddit proposes to split each user’s data we argue that in real life not every user might have a chance to participate in the training.
Revised: Whereas the original LEAF dataset for Reddit proposes to split each user’s data we argue that in real life not every user might have a chance to participate in the training.

Original: Therefore, we split users into two distinct training and test sets and evaluate the model on data from the users who have never participated in the training.
Revised: Therefore, we split users into two distinct training and test sets and evaluate the model on data from the users who have never participated in the training.

Original: This results in notably increased test perplexity but provides a clear separation between training and inference modes.
Revised: This results in notably increased test perplexity but provides a clear separation between training and inference modes.

Original: 5. Comparing tokenization schemes
Revised: 5. Comparing tokenization schemes

Original: Table 1 summarizes experiments that use different tokenization schemes.
Revised: Table 1 summarizes experiments that use different tokenization schemes.

Original: We compute statistics on tokenizers: the average share of OOV tokens for theword-level scheme and the average number of tokens required to encode one word for the sub-word scheme. To compare the effect of each tokenizer
Revised: We compute statistics on tokenizers: the average share of OOV tokens for the

Original: [Table]
Revised: [Table]

Revised: word-level scheme and the average number of tokens required to encode one word for the sub-word scheme.

Original:  on the PFL-trained model, we report word-level[Table] accuracy, for the reasons described in Section 3.2.[Table]
Revised: To compare the effect of each tokenizer on the PFL-trained model, we report word-level accuracy, for the reasons described in Section 3.2.

Original: The “wiki” tokenizers are trained on the Wikipediadata, and the “oracle” tokenizers directly on thetraining data.
Revised: The “wiki” tokenizers are trained on the Wikipedia data, and the “oracle” tokenizers directly on the training data.

Original: Word-level tokenization provides high word accuracy when it is trained using “oracle” user training data.
Revised: Word-level tokenization provides high word accuracy when it is trained using “oracle” user training data.

Original: However, when the word-level has accessto only public “wiki” dataset that mismatches userdistribution the performance signiﬁcantly drops: by26 % for Reddit and 10 % for StackOverﬂow witha signiﬁcant increase in out-of-vocabulary share.
Revised: However, when the word-level has access to only public “wiki” dataset that mismatches user distribution the performance significantly drops: by 26 % for Reddit and 10 % for StackOverflow with a significant increase in out-of-vocabulary share.

Original: However, BPE tokenizers that use public data perform more consistently and outperform the wordlevel models trained on public data, but still requirea large number of tokens per each word.
Revised: However, BPE tokenizers that use public data perform more consistently and outperform the wordlevel models trained on public data, but still require a large number of tokens per each word.

Original: 5. Learning a tokenizer with sampling
Revised: 5. Learning a tokenizer with sampling

Original: A key part of the proposed algorithm is the sampling from a model that uses a public tokenizerτ pub , but is trained with private federated learningand should represent the words in the actual data.
Revised: A key part of the proposed algorithm is the sampling from a model that uses a public tokenizer τ pub , but is trained with private federated learning and should represent the words in the actual data.

Original: The sampling is implemented as in Algorithm 1.
Revised: The sampling is implemented as in Algorithm 1.

Original: First, Figure 3 shows samples from the languagemodels on the two data sets.
Revised: First, Figure 3 shows samples from the language models on the two data sets.

Original: Although clearly thesamples are less coherent than the underlying data,it seems plausible that the word occurrences matchthat data.
Revised: Although clearly the samples are less coherent than the underlying data, it seems plausible that the word occurrences match that data.

Original: Second, Table 2 further investigates the properties of the sampled text.
Revised: Second [Table] Table 2 further investigates the properties of the sampled text.

Original: The “BPE sample” rowsrefer to the method proposed in this paper.
Revised: The “BPE sample” rows refer to the method proposed in this paper.

Original: A language model with the “wiki” tokenizer istrainedwith PFL on the ﬁrst half of the training data. Thensamples are drawn from this language model. Then,the language model is trainedfrom scratch on the
Revised: A language model with the “wiki” tokenizer is trained

Original: [Table]
Revised: [Table]

Original: second half of the training data.
Revised: with PFL on the first half of the training data.

Original: [Table]
Revised: Then samples are drawn from this language model.

Original: The “BPE Heavy hitters” rows refer to training [Table]
Revised: Then, the language model is trained from scratch on the second half of the training data.

Original:  with a differentially private “heavy hitters” algo- [Table] rithm (Apple, 2017).Each of the population of
Revised: The “BPE Heavy hitters” rows refer to training with a differentially private “heavy hitters” algorithm (Apple, 2017).

Original: [Table]

Original:  the users from the ﬁrst half of the training set con- [Table] tributes three words from the from the Wikipedia dataset, with a local privacy budget of (cid:15) = 8 .
Revised: Each of the population of the users from the first half of the training set contributes three words from the from the Wikipedia dataset, with a local privacy budget of ϵ = 8 .

Original: Just like for the sampling approach, the language model is then trained from scratch on the second half of the training data.
Revised: Just like for the sampling approach, the language model is then trained from scratch on the second half of the training data.

Original: First, we examine the difference between the real training data and the data used to train the tokenizers.
Revised: First, we examine the difference between the real training data and the data used to train the tokenizers.

Original: The column “Data KLD” shows the KL divergence from the user “oracle” training data to the sampled data.
Revised: The column “Data KLD” shows the KL divergence from the user “oracle” training data to the sampled data.

Original: The KL divergence is computed from the unigram counts, which are relevant for training a tokenizer, over the top 10,000 wordsfrom the training data and with add-1 smoothing. The KL divergence to the training data itself, which the oracle tokenizer is trained on, is 0 by deﬁnition.
Revised: The KL divergence is computed from the unigram counts, which are relevant for training a tokenizer, over the top 10,000 words

Original: Reddit i would love to know why we may already live in a consolation subreddit and the aforementioned it will almost always be done on the warrior sheet shows from the west .
Revised: Reddit i would love to know why we may already live in a consolation subreddit and the aforementioned it will almost always be done on the warrior sheet shows from the west .

Original: i
Revised: i

Original: StackOverﬂow json results are : can anyone provide a complete sample response ( lists of descendants list ) to my page depending on future python functions .
Revised: StackOverflow json results are : can anyone provide a complete sample response ( lists of descendants list ) to my page depending on future python functions .

Original: in web apps that require patient for many
Revised: in web apps that require patient for many

Original: P e r p l e x it y
Revised: P e r p l e x it y

Original: Central iteration
Revised: Central iteration

Original: Baseline 1k 2k 3k 4k
Revised: Baseline 1k 2k 3k 4k

Original: [Figure]
Revised: [Figure]

Original: dataset

Original: P e r p l e x it y
Revised: P e r p l e x it y

Original: Central iteration
Revised: Central iteration

Original: Baseline 1k 2k 3k 4k(
Revised: Baseline 1k 2k 3k 4k

Original: [Figure]
Revised: [Figure]

Original: dataset
Revised: from the training data and with add-1 smoothing. The KL divergence to the training data itself, which the oracle tokenizer is trained on, is 0 by definition.

Original: The KL divergence between the actual data andthe Wikipedia data, on the other hand, is around 1,for both datasets.
Revised: The KL divergence between the actual data and the Wikipedia data, on the other hand, is around 1, for both datasets.

Original: Both the heavy hitters algorithmand the algorithm we propose in this paper ﬁnd adistribution close to the real distribution.
Revised: Both the heavy hitters algorithm and the algorithm we propose in this paper find a distribution close to the real distribution.

Original: For sub-word tokenizers, the number of tokensper word is relevant.
Revised: For sub-word tokenizers, the number of tokens per word is relevant.

Original: Even though they can represent unseen words by multiple tokens, a languagemodel trained on top of that has a harder task giventhe longer context on average.
Revised: Even though they can represent unseen words by multiple tokens, a language model trained on top of that has a harder task given the longer context on average.

Original: The oracle tokenizerhas the lowest number of tokens per words and the“wiki” tokenizer the highest.
Revised: The oracle tokenizer has the lowest number of tokens per words and the “wiki” tokenizer the highest.

Original: The “BPE sample”tokenizer comes very close to the oracle tokenizer.
Revised: The “BPE sample” tokenizer comes very close to the oracle tokenizer.

Original: However, the  heavy hitters experiment showsmuch smaller gain in performance, i.e. better than“wiki” tokenizer but still worse than our proposedsampling method.
Revised: However, the local-DP heavy hitters experiment shows much smaller gain in performance, i.e. better than “wiki” tokenizer but still worse than our proposed sampling method.

Original: Furthermore, it requires a separate privacy budget allocated for the run, whilesampling can operate on existing prior model.
Revised: Furthermore, it requires a separate privacy budget allocated for the run, while sampling can operate on existing prior model.

Original: 5. Iterative updates
Revised: 5. Iterative updates

Original: This part implements Algorithm 1 completely.
Revised: This part implements Algorithm 1 completely.

Original: Weagain initialize the tokenizer on publicly availabledata.
Revised: We again initialize the tokenizer on publicly available data.

Original: We then train the language model with PFL.
Revised: We then train the language model with PFL.

Original: At a point during training, we retrain the tokenizerby sampling.
Revised: At a point during training, we retrain the tokenizer by sampling.

Original: Unlike in the previous section, weupdate the language model by remapping its embedding layer, and continue training.
Revised: Unlike in the previous section, we update the language model by remapping its embedding layer, and continue training.

Original: We samplethe same data before and after changing the tokenizer.
Revised: We sample the same data before and after changing the tokenizer.

Original: Figure 4 shows the results for changing tokenizers at different times.
Revised: Figure 4 shows the results for changing tokenizers at different times.

Original: The “Baseline” curve represents the model trained using public tokenizerτ pub from Wikipedia data.
Revised: The “Baseline” curve represents the model trained using public tokenizer τ pub from Wikipedia data.

Original: Each of the other curvestakes the system from the “Baseline” curve at a different iteration.
Revised: Each of the other curves takes the system from the “Baseline” curve at a different iteration.

Original: As expected, the initial remappingof the embedding layer is not perfect and needs ﬁnetuning.
Revised: As expected, the initial remapping of the embedding layer is not perfect and needs finetuning.

Original: The graph also shows the tradeoff in when to change tokenizers: too early, e.g. after only 1000 iterations, and the tokenizer is not representative enough yet; too late, e.g. after 4000 iterations, and there is not enough time to converge again.
Revised: The graph also shows the tradeoff in when to change tokenizers: too early, e.g. after only 1000 iterations, and the tokenizer is not representative enough yet; too late, e.g. after 4000 iterations, and there is not enough time to converge again.

Original:  Conclusion
Revised:  Conclusion

Original: This paper has proposed a method that allows a tokenizer to be found together with a language model using private federated learning.
Revised: This paper has proposed a method that allows a tokenizer to be found together with a language model using private federated learning.

Original: First, it has shown that a mismatched tokenizer can cause a signiﬁcant performance degradation.
Revised: First, it has shown that a mismatched tokenizer can cause a significant performance degradation.

Original: The key to improving this is to use a sub-word tokenizer which allows new words to be represented as a sequence of tokens.
Revised: The key to improving this is to use a sub-word tokenizer which allows new words to be represented as a sequence of tokens.

Original: Then, a language model trained with PFL can represent the private data.
Revised: Then, a language model trained with PFL can represent the private data.

Original: This paper has presented a method to produce a new tokenizer from that model, and to convert the model to work with the new tokenizer.
Revised: This paper has presented a method to produce a new tokenizer from that model without spending additional privacy budget, and to convert the model to work with the new tokenizer.

Original: When this is trained further with private federated learning, it outperforms the language model with the mismatched tokenizer, and gets close to one with the oracle tokenizer.
Revised: When this is trained further with private federated learning, it outperforms the language model with the mismatched tokenizer, and gets close to one with the oracle tokenizer.

Original: Personalization and Fairness.
Revised: Personalization and Fairness.

Original: The problem of out-of-vocabulary words might be more acute for some users that use unique vocabulary, such as dialect, and impact individual performance.
Revised: The problem of out-of-vocabulary words might be more acute for some users that use unique vocabulary, such as dialect, and impact individual performance.

Original: Therefore good tokenizers can beneﬁt personalization in federated models (Li et al., 2021; Yu et al., 2020)
Revised: Therefore good tokenizers can benefit personalization in federated models (Li et al., 2021; Yu et al., 2020).

Original: A Impact of hyperparameters
Revised: A Impact of hyperparameters

Original: This section examines different hyperparameters.
Revised: This section examines different hyperparameters.

Original: A. Experimental design
Revised: A. Experimental design

Original: First, consider the choice to train the public tokenizer on Wikipedia data.
Revised: First, consider the choice to train the public tokenizer on Wikipedia data.

Original: To examine the effectof using a more conversational style corpus.
Revised: To examine the effect of using a more conversational style corpus.

Original: To dothis, Table 3 takes a subset of the numbers from
Revised: To do this, Table 3 takes a subset of the numbers from

Original: Table 2 and adds a scenario where a tokenizer on
Revised: Table 2 and adds a scenario where a tokenizer on

Original: StackOverﬂow data is used with Reddit data andvice versa.
Revised: StackOverflow data is used with Reddit data and vice versa.

Original: The cross-dataset numbers are highlighted bold in the table.
Revised: The cross-dataset numbers are highlighted bold in the table.

Original: First, in terms of the KL divergence the StackOverﬂow data seems a slightly better model forthe Reddit distribution than the Wikipedia data is.
Revised: First, in terms of the KL divergence the StackOverflow data seems a slightly better model for the Reddit distribution than the Wikipedia data is.

Original: However, when using PFL to train on Reddit data,but with a StackOverﬂow-trained tokenizer, theperplexity deteriorates compared to the Wikipediatrained tokenizer.
Revised: However, when using PFL to train on Reddit data, but with a StackOverflow-trained tokenizer, the perplexity deteriorates compared to the Wikipediatrained tokenizer.

Original: Second, the reverse experimentlooks a bit better but not hugely better.
Revised: Second, the reverse experiment looks a bit better but not hugely better.

Original: Thoughthe KL divergence from the StackOverﬂow datato the Reddit data is signiﬁcantly better than the
Revised: Though the KL divergence from the StackOverflow data to the Reddit data is significantly better than the

Original: KL divergence to the Wikipedia data, some of thatadvantage disappears in the ﬁnal trained model.
Revised: KL divergence to the Wikipedia data, some of that advantage disappears in the final trained model.

Original: [Table]
Revised: [Table]

Original: Data Data
Revised: Data Data

Original: [Table]

Original: Wikipedia

Original: [Table]
Revised: [Table]

Original: BPE
Revised: BPE

Original: StackOverﬂow
Revised: StackOverflow

Original: [Table]
Revised: [Table]

Revised: StackOverflow

Revised: BPE

Revised: Wikipedia 1.

Original: [Table]
Revised: [Table]

Original: BPE
Revised: BPE

Original: Reddit 
Revised: Reddit 0. BPE StackOverflow 108. BPE

Original: [Table]
Revised: [Table]

Original: BPE sample

Original: [Table]
Revised: [Table]

Revised: Then

Original: [Table]
Revised: [Table]

Original: Then, consider the choice of vocabulary size, here the number of distinct tokens.
Revised: consider the choice of vocabulary size [Table] here the number of distinct tokens.

Original: Table 4 shows the perplexities for the baseline (“Wiki”) and ceiling (“oracle”) experiments.
Revised: Table 4 shows the perplexities for the baseline (“Wiki”) and ceiling (“oracle”) experiments.

Original: Though the absolute numbers change, the trends do not change.
Revised: Though the absolute numbers change, the trends do not change.

Original: Similarly for changing model architectures.
Revised: Similarly for changing model architectures.

Original: This paper has presented results on an LSTM model.
Revised: This paper has presented results on an LSTM model.

Original: Table 5 shows results on a Transformer model.
Revised: Table 5 shows results on a Transformer model.

Original: Again, though the absolute numbers change, the trends do not change.
Revised: Again, though the absolute numbers change, the trends do not change.

Original: A. Other hyperparameters
Revised: A. Other hyperparameters

Original: We consider two hyperparameter choices for experiments: ﬁrst, the privacy budget, and secondly, the cohort size.
Revised: We consider two hyperparameter choices for experiments: first, the privacy budget, and secondly, the cohort size.

Original: Figure 5 shows the effect of different privacyModel
Revised: Figure 5 shows the effect of different privacy

Original: Reddit

Original: StackOverﬂow architecture

Original: Wiki Oracle Wiki Oracle
Revised: [Table]

Original: Transformer 261.

Original: LSTM 276.

Original: parameters.
Revised: parameters.

Original: The effects are not huge, but clearlydifferential privacy does impede learning somewhat.
Revised: The effects are not huge, but clearly differential privacy does impede learning somewhat.

Original: Figure 6 shows the effect of differing cohortsizes.
Revised: Figure 6 shows the effect of differing cohort sizes.

Original: A larger cohort size implies a better signal-tonoise ratio when training with differential privacy.
Revised: A larger cohort size implies a better signal-tonoise ratio when training with differential privacy.

Original: However, for practical reasons it is preferable forcohorts to be smaller.
Revised: However, for practical reasons it is preferable for cohorts to be smaller.

Original: 10,000 is a happy mediumbetween good performance and practicality.
Revised: 10,000 is a happy medium between good performance and practicality.

Original: Also,again, though the absolute numbers change, thetrends do not change.
Revised: Also, again, though the absolute numbers change, the trends do not change.