Original: L OSSLESS D ATA C OMPRESSION WITH T RANSFORMER
Revised: L OSSLESS D ATA C OMPRESSION WITH T RANSFORMER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers have replaced long-short term memory and other recurrent neural networks variants in sequence modeling.
Revised: Transformers have replaced long-short term memory and other recurrent neural networks variants in sequence modeling.

Original: It achieves state-of-the-art performance on a wide range of tasks related to natural language processing, including language modeling, machine translation, and sentence representation.
Revised: It achieves state-of-the-art performance on a wide range of tasks related to natural language processing, including language modeling, machine translation, and sentence representation.

Original: Lossless compression is another problem that can beneﬁt from better sequence models.
Revised: Lossless compression is another problem that can beneﬁt from better sequence models.

Original: It is closely related to the problem of online learning of language models.
Revised: It is closely related to the problem of online learning of language models.

Original: But, despite this ressemblance, it is an area where purely neural network based methods have not yet reached the compression ratio of state-of-the-art algorithms.
Revised: But, despite this ressemblance, it is an area where purely neural network based methods have not yet reached the compression ratio of state-of-the-art algorithms.

Original: In this paper, we propose a Transformer based lossless compression method that match the best compression ratio for text.
Revised: In this paper, we propose a Transformer based lossless compression method that match the best compression ratio for text.

Original: Our approach is purely based on neural networks and does not rely on hand-crafted features as other lossless compression algorithms.
Revised: Our approach is purely based on neural networks and does not rely on hand-crafted features as other lossless compression algorithms.

Original: We also provide a thorough study of the impact of the different components of the
Revised: We also provide a thorough study of the impact of the different components of the

Original: Transformer and its training on the compression ratio.
Revised: Transformer and its training on the compression ratio.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Lossless compression is a class of compression algorithms that allows for the perfect reconstruction of the original data.
Revised: Lossless compression is a class of compression algorithms that allows for the perfect reconstruction of the original data.

Original: In the last decades, statistical methods for lossless compression have been dominated by PAQ-type approaches (Mahoney, 2005).
Revised: In the last decades, statistical methods for lossless compression have been dominated by PAQ-type approaches (Mahoney, 2005).

Original: The structure of these approaches is similar to the Prediction by Partial Matching (PPM) of Cleary & Witten (1984) and are composed of two separated parts: a predictor and an entropy encoding.
Revised: The structure of these approaches is similar to the Prediction by Partial Matching (PPM) of Cleary & Witten (1984) and are composed of two separated parts: a predictor and an entropy encoding.

Original: Entropy coding scheme like arithmetic coding (Rissanen & Langdon, 1979) are optimal and most of the compression gains are coming from improving the predictor.
Revised: Entropy coding scheme like arithmetic coding (Rissanen & Langdon, 1979) are optimal and most of the compression gains are coming from improving the predictor.

Original: The predictor estimates the probability of a token given its past, which is a standard sequence modeling problem.
Revised: The predictor estimates the probability of a token given its past, which is a standard sequence modeling problem.

Original: In most PAQ-type approaches, this predictor relies heavily on hand-crafted features, but recent work has used the close connection between sequence modeling and machine learning to add neural networks based methods in the predictor.
Revised: In most PAQ-type approaches, this predictor relies heavily on hand-crafted features, but recent work has used the close connection between sequence modeling and machine learning to add neural networks based methods in the predictor.

Original: In particular,
Revised: In particular,

Original: CMIX (Knoll, 2014) has successfully added a Long-Short Term Memory (LSTM, Hochreiter &
Revised: CMIX (Knoll, 2014) has successfully added a Long-Short Term Memory (LSTM, Hochreiter &

Original: Schmidhuber (1997)) to the predictor of PAQ8.
Revised: Schmidhuber (1997)) to the predictor of PAQ8.

Original: However, these approaches are still heavily relying on hand-crafted features along with the neural networks, and purely neural network based models are still far from the state of the art (Bellard, 2019).
Revised: However, these approaches are still heavily relying on hand-crafted features along with the neural networks, and purely neural network based models are still far from the state of the art (Bellard, 2019).

Original: In this work, we aim at building a purely neural network based model that compete the state of the art.
Revised: In this work, we aim at building a purely neural network based model that compete the state of the art.

Original: We follow Knoll & de Freitas (2011) and formulate the token prediction problem solved by the predictor as the online learning of a sequence model.
Revised: We follow Knoll & de Freitas (2011) and formulate the token prediction problem solved by the predictor as the online learning of a sequence model.

Original: As opposed to previous work, we propose to use a Transformer (Vaswani et al., 2017a) to replace entirely the predictor of PAQ8 and its handcrafted features.
Revised: As opposed to previous work, we propose to use a Transformer (Vaswani et al., 2017a) to replace entirely the predictor of PAQ8 and its handcrafted features.

Original: Transformer has emerged as the standard model for sequence modeling in natural language processing, achieving state-of-the-art performance in numerous applications, including, but not limited to, language modeling (Dai et al., 2019; Sukhbaatar et al., 2019b), machine translation (Vaswani et al., 2017a;
Revised: Transformer has emerged as the standard model for sequence modeling in natural language processing, achieving state-of-the-art performance in numerous applications, including, but not limited to, language modeling (Dai et al., 2019; Sukhbaatar et al., 2019b), machine translation (Vaswani et al., 2017a;

Original: Ott et al., 2018), or sentence representation (Devlin et al., 2018; Yang et al., 2019).
Revised: Ott et al., 2018), or sentence representation (Devlin et al., 2018; Yang et al., 2019).

Original: As opposed to the recurrent neural networks (RNNs, Elman (1990)) used in lossless compression, they are able to capture very long-term dependencies by allowing direct connections between long-distance symbols.
Revised: As opposed to the recurrent neural networks (RNNs, Elman (1990)) used in lossless compression, they are able to capture very long-term dependencies by allowing direct connections between long-distance symbols.

Original: However, the training of a Transformer is unstable and slow, limiting its application to the online setting of lossless compression.
Revised: However, the training of a Transformer is unstable and slow, limiting its application to the online setting of lossless compression.

Original: We propose several improvements to its architecture and training to accelerate and stabilize its online training.
Revised: We propose several improvements to its architecture and training to accelerate and stabilize its online training.

Original: We show that our neural network approach matches the state of the art on the competitive benchmark enwik8 used in the Hutter prize (Mahoney, 2005).
Revised: We show that our neural network approach matches the state of the art on the competitive benchmark enwik8 used in the Hutter prize (Mahoney, 2005).

Original: We also provide a thorough study of all the important component of a Transformer and its training to achieve this performance.
Revised: We also provide a thorough study of all the important components of a Transformer and its training to achieve this performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we brieﬂy introduce statistical based methods for lossless compression, and focus on
Revised: In this section, we brieﬂy introduce statistical based methods for lossless compression, and focus on

Original: PAQ based compression methods as well as purely neural network based methods.
Revised: PAQ based compression methods as well as purely neural network based methods.

Original: Statistical based methods.
Revised: Statistical based methods.

Original: A popular statistical based method for lossless data compression is the Prediction by Partial Matching (PPM) of Cleary & Witten (1984).
Revised: A popular statistical based method for lossless data compression is the Prediction by Partial Matching (PPM) of Cleary & Witten (1984).

Original: It is based on a predictor and a entropy coding, that often takes the form of an arthimetic coding (Rissanen & Langdon, 1979;
Revised: It is based on a predictor and a entropy coding, that often takes the form of an arthimetic coding (Rissanen & Langdon, 1979;

Original: Witten et al., 1987b), but could also be a Huffman coding (Huffman, 1952b).
Revised: Witten et al., 1987b), but could also be a Huffman coding (Huffman, 1952b).

Original: There are many variants of PPM such as PPM*C (Cleary & Teahan, 1997) or cPPMI-64 Shkarin (2002).
Revised: There are many variants of PPM such as PPM*C (Cleary & Teahan, 1997) or cPPMI-64 Shkarin (2002).

Original: Another statistical based method is the stochastic sequence memoizer of Gasthaus et al.
Revised: Another statistical based method is the stochastic sequence memoizer of Gasthaus et al.

Original: (2010) that is based on a nonparametric Bayesian model.
Revised: (2010) that is based on a nonparametric Bayesian model.

Original: In particular, the stochastic memoizer of Wood et al.
Revised: In particular, the stochastic memoizer of Wood et al.

Original: (2009) shares many similarities with PPM and reaches similar compression performance.
Revised: (2009) shares many similarities with PPM and reaches similar compression performance.

Original: However, most of these models are now surpassed by PAQ based methods like CMIX (Knoll, 2014).
Revised: However, most of these models are now surpassed by PAQ based methods like CMIX (Knoll, 2014).

Original: PAQ based compression methods.
Revised: PAQ based compression methods.

Original: The series of PAQ algorithms, including PAQ8 (Mahoney, 2005), are statistical based methods designed to operate at bit-level, making it applicable to any type of data (Salomon & Motta, 2010).
Revised: The series of PAQ algorithms, including PAQ8 (Mahoney, 2005), are statistical based methods designed to operate at bit-level, making it applicable to any type of data (Salomon & Motta, 2010).

Original: There are also different version of PAQ8 specialised in different data types for better compression performance.
Revised: There are also different version of PAQ8 specialised in different data types for better compression performance.

Original: PAQ uses a context mixer that follows the same structure as PPM.
Revised: PAQ uses a context mixer that follows the same structure as PPM.

Original: However, unlike PPM, the predictor of the context mixers combines the probability of different prediction models (Knoll & de Freitas, 2011).
Revised: However, unlike PPM, the predictor of the context mixers combines the probability of different prediction models (Knoll & de Freitas, 2011).

Original: More recently, a PAQ based model called CMIX (Knoll, 2014) has been the state of the art for lossless text compression in terms of compression rate.
Revised: More recently, a PAQ based model called CMIX (Knoll, 2014) has been the state of the art for lossless text compression in terms of compression rate.

Original: As opposed to PAQ8, CMIX uses a Long Short Term Memory network (LSTM, Hochreiter & Schmidhuber (1997)) in their probabilistic model.
Revised: As opposed to PAQ8, CMIX uses a Long Short Term Memory network (LSTM, Hochreiter & Schmidhuber (1997)) in their probabilistic model.

Original: The context mixer used in CMIX is also based on a gated linear network (Veness et al., 2017).
Revised: The context mixer used in CMIX is also based on a gated linear network (Veness et al., 2017).

Original: Neural Network based compression method.
Revised: Neural Network based compression method.

Original: More recently, there has been some attempts to use purely neural network based approaches for the probabilistic model.
Revised: More recently, there has been some attempts to use purely neural network based approaches for the probabilistic model.

Original: Notably Knoll released lstm-compress (Knoll, 2015), which uses only the LSTM introduced in CMIX and the preprocessing of CMIX.
Revised: Notably Knoll released lstm-compress (Knoll, 2015), which uses only the LSTM introduced in CMIX and the preprocessing of CMIX.

Original: Bellard (2019) investigates the performance of pure neural approaches with modern and large neural networks such as LSTM and Transformer.
Revised: Bellard (2019) investigates the performance of pure neural approaches with modern and large neural networks such as LSTM and Transformer.

Original: As well as our approach, all of these approaches use arithmetic coding as entropy coding scheme.
Revised: As well as our approach, all of these approaches use arithmetic coding as entropy coding scheme.

Original:  L OSSLESS DATA COMPRESSION
Revised:  L OSSLESS DATA COMPRESSION

Original: In this section, we brieﬂy introduce the core components of statistical methods for lossless compression.
Revised: In this section, we brieﬂy introduce the core components of statistical methods for lossless compression.

Original: As opposed to lossy compression where it is allowed to lose information about the orginal data in the reconstruction, lossless compression methods restore the data to its original value.
Revised: As opposed to lossy compression where it is allowed to lose information about the orginal data in the reconstruction, lossless compression methods restore the data to its original value.

Original: Lossless compression methods are composed of two main elements: a predictor and an entropy coding scheme.
Revised: Lossless compression methods are composed of two main elements: a predictor and an entropy coding scheme.

Original: They are used successively to encode an input stream of tokens s 0 , . . .
Revised: They are used successively to encode an input stream of tokens s 0 , . . .

Original: , s T .
Revised: , s T .

Original: More precisely, for each token, the predictor ﬁrst evaluates its probability given its past, then the entropy coding scheme uses this probability to store the token into a lossless code.
Revised: More precisely, for each token, the predictor ﬁrst evaluates its probability given its past, then the entropy coding scheme uses this probability to store the token into a lossless code.

Original: More precisely, if we assume that t tokens s 0 , . . .
Revised: More precisely, if we assume that t tokens s 0 , . . .

Original: , s t − 1 are already compressed into a lossless code, the predictor computes the probability of the next token to be equal to its value s t given all the preceding tokens, i.e., the predictor computes p ( s t | s t − 1 , . . .
Revised: , s t − 1 are already compressed into a lossless code, the predictor computes the probability of the next token to be equal to its value s t given all the preceding tokens, i.e., the predictor computes p ( s t | s t − 1 , . . .

Original: , s 0 ) .
Revised: , s 0 ) .

Original: Then, the entropy coding scheme encodes the token s t into a lossless code of bit-length log 2 p ( s t | s t − 1 , . . .
Revised: Then, the entropy coding scheme encodes the token s t into a lossless code of bit-length log 2 p ( s t | s t − 1 , . . .

Original: , s 0 ) .
Revised: , s 0 ) .

Original: Once the token s t has been encoded, these two steps are repeated on following tokens until the end of the sequence.
Revised: Once the token s t has been encoded, these two steps are repeated on following tokens until the end of the sequence.

Original: The operations performed by the predictor are identical during compression and decompression phases, i.e., in both cases it predicts the next token given all the preceding tokens.
Revised: The operations performed by the predictor are identical during compression and decompression phases, i.e., in both cases it predicts the next token given all the preceding tokens.

Original: The only difference is that, during the compression, the preceding tokens are taken from the input ﬁle, while during the decompression, they are generated by the entropy coding scheme.
Revised: The only difference is that, during the compression, the preceding tokens are taken from the input ﬁle, while during the decompression, they are generated by the entropy coding scheme.

Original: Note that, as opposed to the predictor, the role of the entropy coding scheme differs during the two phases since, during decompression, it acts as a decoder, i.e., given the compressed representation and the sequence of predicted probability distributions ( p ( s t | s t − 1 , . . .
Revised: Note that, as opposed to the predictor, the role of the entropy coding scheme differs during the two phases since, during decompression, it acts as a decoder, i.e., given the compressed representation and the sequence of predicted probability distributions ( p ( s t | s t − 1 , . . .

Original: , s 0 )) 

Revised: , s 0 ))

Original: ≤ t ≤ T it restores the original sequence of tokens.
Revised: ≤ t ≤ T it restores the original sequence of tokens.

Original: There are many efﬁcient and optimal methods for the entropy coding scheme.
Revised: There are many efﬁcient and optimal methods for the entropy coding scheme.

Original: The source coding theorem Shannon (1948) has introduced the fundamental idea that any probability distribution can be encoded into a lossless code.
Revised: The source coding theorem Shannon (1948) has introduced the fundamental idea that any probability distribution can be encoded into a lossless code.

Original: For a given data point, the bit-length of this code is equal to the negative log probability estimated by the model.
Revised: For a given data point, the bit-length of this code is equal to the negative log probability estimated by the model.

Original: Thus, efﬁcient probabilistic models are crucial to design effective compression scheme since the smallest codes are obtained when the model estimates the true data distribution.
Revised: Thus, efﬁcient probabilistic models are crucial to design effective compression scheme since the smallest codes are obtained when the model estimates the true data distribution.

Original: Depending on the type of probabilistic model considered, several compression schemes have been developed to implement this idea in practice, namely Huffman coding Huffman (1952a), arithmetic coding Witten et al.
Revised: Depending on the type of probabilistic model considered, several compression schemes have been developed to implement this idea in practice, namely Huffman coding Huffman (1952a), arithmetic coding Witten et al.

Original: (1987a)
Revised: (1987a)

Original: and asymmetric coding Duda (2009).
Revised: and asymmetric coding Duda (2009).

Original: The basic idea of all these entropy coding schemes is to assign shorter codes to the more likely tokens.
Revised: The basic idea of all these entropy coding schemes is to assign shorter codes to the more likely tokens.

Original: In this work, we use the arithmetic coding scheme.
Revised: In this work, we use the arithmetic coding scheme.

Original:  O UR APPROACH
Revised:  O UR APPROACH

Original: In this section, we introduce our approach to lossless data compression using neural networks.
Revised: In this section, we introduce our approach to lossless data compression using neural networks.

Original: First, we brieﬂy review transformer networks, which are at the core of our method.
Revised: First, we brieﬂy review transformer networks, which are at the core of our method.

Original: We then discuss modiﬁcations to the standard model, to make it more suitable to long sequence modeling.
Revised: We then discuss modiﬁcations to the standard model, to make it more suitable to long sequence modeling.

Original: Finally, we propose a strategy to make the online learning of large models more efﬁcient.
Revised: Finally, we propose a strategy to make the online learning of large models more efﬁcient.

Original: 4. T RANSFORMER N ETWORKS
Revised: 4. T RANSFORMER N ETWORKS

Original: Transformer networks were introduced by Vaswani et al.
Revised: Transformer networks were introduced by Vaswani et al.

Original: (2017a)
Revised: (2017a)

Original: in the context of machine translation.
Revised: in the context of machine translation.

Original: These models are made of a sequence of identical blocks, each comprising two sublayers.
Revised: These models are made of a sequence of identical blocks, each comprising two sublayers.

Original: The ﬁrst sublayer, called SelfAttention , is based on the attention mechanism from Bahdanau et al.
Revised: The ﬁrst sublayer, called SelfAttention , is based on the attention mechanism from Bahdanau et al.

Original: More speciﬁcally, given a sequence of T hidden states of dimension d , represented by the matrix X ∈ R d × T , the output of this layer is given by
Revised: More speciﬁcally, given a sequence of T hidden states of dimension d , represented by the matrix X ∈ R d × T , the output of this layer is given by

Original: [Equation]
Revised: [Equation]

Original: where W v , W k and W q are the parameters of the sublayer used to compute the values, keys and queries of the attention mechanism.
Revised: where W v , W k and W q are the parameters of the sublayer used to compute the values, keys and queries of the attention mechanism.

Original: Multiple self-attention mechanisms are usually applied in parallel, leading to the multi-head self-attention sublayer.
Revised: Multiple self-attention mechanisms are usually applied in parallel, leading to the multi-head self-attention sublayer.

Original: The second sublayer, called FFN , is a fully connected feed forward network with R E LU activation, which is applied independently at each time step.
Revised: The second sublayer, called FFN , is a fully connected feed forward network with R E LU activation, which is applied independently at each time step.

Original: Each sublayer is followed by a skip connection and a layer norm operator, to ﬁnally obtain:
Revised: Each sublayer is followed by a skip connection and a layer norm operator, to ﬁnally obtain:

Original: Z = LayerNorm ( SelfAttention ( H ) + H ) , and
Revised: Z = LayerNorm ( SelfAttention ( H ) + H ) , and

Original: Y = LayerNorm ( FFN ( Z )
Revised: Y = LayerNorm ( FFN ( Z )

Original: + Z ) .
Revised: + Z ) .

Original: Following Dai et al.
Revised: Following Dai et al.

Original: (2019), we use relative positional encodings at each layer, as well as a cache mechanism, to make the processing of long sequences more efﬁcient.
Revised: (2019), we use relative positional encodings at each layer, as well as a cache mechanism, to make the processing of long sequences more efﬁcient.

Original: In the following, we will refer to this model as Transformer-XL.
Revised: In the following, we will refer to this model as Transformer-XL.

Original: We refer the reader to Vaswani et al.
Revised: We refer the reader to Vaswani et al.

Original: (2017a)
Revised: (2017a)

Original: and Dai et al.
Revised: and Dai et al.

Original: for a more detailed introduction to these models.
Revised: for a more detailed introduction to these models.

Original: 4. M ODIFICATIONS TO THE T RANSFORMER -XL MODEL
Revised: 4. M ODIFICATIONS TO THE T RANSFORMER -XL MODEL

Original: We propose several modiﬁcations to the Transformer-XL model that we found to improve the compression rate empirically.
Revised: We propose several modiﬁcations to the Transformer-XL model that we found to improve the compression rate empirically.

Original: First, we follow Devlin et al.
Revised: First, we follow Devlin et al.

Original: and use the GELU activation of Hendrycks & Gimpel (2016) instead of ReLUs.
Revised: and use the GELU activation of Hendrycks & Gimpel (2016) instead of ReLUs.

Original: Second, as opposed to Transformer-XL (Dai et al., 2019), we do not use Adaptive Inputs (Baevski & Auli, 2018; ? ) to tie the weights of embeddings and the classiﬁer.
Revised: Second, as opposed to Transformer-XL (Dai et al., 2019), we do not use Adaptive Inputs (Baevski & Auli, 2018) to tie the weights of embeddings and the classiﬁer.

Original: Finally, we use different attention span size across all layers as it suggested by Sukhbaatar et al. (2019a).
Revised: Finally, we use different attention span size across all layers as was suggested by Sukhbaatar et al. (2019a).

Original: In particular, an observation from this work is that attention span can be short in the ﬁrst half of the network while it has to be longer in the last layers.
Revised: In particular, an observation from this work is that attention span can be short in the ﬁrst half of the network while it has to be longer in the last layers.

Original: We exploit this observation by restricting the attention span of the ﬁrst layers of the Transformer and is extended linearly with the depth.
Revised: We exploit this observation by restricting the attention span of the ﬁrst layers of the Transformer and is extended linearly with the depth.

Original: More speciﬁcally, we restrict the attention size of the ﬁrst 6 layers to be equal to 6 × k for the k -th layer.
Revised: More speciﬁcally, we restrict the attention size of the ﬁrst 6 layers to be equal to 6 × k for the k -th layer.

Original: 4. A DDING n - GRAMS AS INPUT
Revised: 4. A DDING n - GRAMS AS INPUT

Original: We also propose to enrich the input of the transformer with n -gram information.
Revised: We also propose to enrich the input of the transformer with n -gram information.

Original: More speciﬁcally, at time step t , the input of the model is all the n -grams for n between 1 and N of the form
Revised: More speciﬁcally, at time step t , the input of the model is all the n -grams for n between 1 and N of the form

Original: [Table]
Revised: [Table]

Original: x t − n +1 , ..., x t .
Revised: x t − n +1 , ..., x t .

Original: Concretely, we have an embedding vector for each n -gram, and for a given time step, we average the vectors corresponding to the n -grams of different lengths.
Revised: Concretely, we have an embedding vector for each n -gram, and for a given time step, we average the vectors corresponding to the n -grams of different lengths.

Original: For large N , there are many rare n -grams, leading to an important increase of parameters and memory consumption.
Revised: For large n , there are many rare n -grams, leading to an important increase of parameters and memory consumption.

Original: A potential solution would be to only keep the most frequent ones, but this would require computing a dictionary of n -grams and add it to the archive.
Revised: A potential solution would be to only keep the most frequent ones, but this would require computing a dictionary of n -grams and add it to the archive.

Original: Instead, we propose to hash the n -grams into a ﬁxed number of bins, allowing to constrain the memory requirement, without having to increase the size of the archive.
Revised: Instead, we propose to hash the n -grams into a ﬁxed number of bins, allowing to constrain the memory requirement, without having to increase the size of the archive.

Original: 4. T RAINING WITH OR WITHOUT REVISITS
Revised: 4. T RAINING WITH OR WITHOUT REVISITS

Original: During the compression phase (resp. decompression), once a token has been encoded (resp. decoded), it can be used to update the probabilistic model.
Revised: During the compression phase (resp. decompression), once a token has been encoded (resp. decoded), it can be used to update the probabilistic model.

Original: This means that compression using statistical models can be framed as an online learning problem.
Revised: This means that compression using statistical models can be framed as an online learning problem.

Original: Traditionally, previously encoded tokens are used only once to update the probabilistic model.
Revised: Traditionally, previously encoded tokens are used only once to update the probabilistic model.

Original: However, nothing prevents re-using the tokens to compute multiple updates of the model, besides memory consumption.
Revised: However, nothing prevents re-using the tokens to compute multiple updates of the model, besides memory consumption.

Original: Indeed, to be able to do so, one needs to store the data being compressed to be able to revisit it.
Revised: Indeed, to be able to do so, one needs to store the data being compressed to be able to revisit it.

Original: In the following we propose to explore this strategy to learn better probabilistic models based on transformer networks.
Revised: In the following we propose to explore this strategy to learn better probabilistic models based on transformer networks.

Original: The main motivation is that large overparametrized neural networks usually needs many passes over the data to be trained.
Revised: The main motivation is that large overparametrized neural networks usually needs many passes over the data to be trained.

Original: In practice, every F compressed tokens, we re-use the last M compressed tokens to further train the network.
Revised: In practice, every F compressed tokens, we re-use the last M compressed tokens to further train the network.

Original: This strategy will improve the quality of the probabilistic model, leading to better predictions and compression rate of the data not yet processed.
Revised: This strategy will improve the quality of the probabilistic model, leading to better predictions and compression rate of the data not yet processed.

Original: This phase is determined by the frequency of revisit F and the number of tokens revisited M .
Revised: This phase is determined by the frequency of revisit F and the number of tokens revisited M .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we investigate the inﬂuence of different parameters on the performance of our approach.
Revised: In this section, we investigate the inﬂuence of different parameters on the performance of our approach.

Original: We also provide numerical evidence that a purely neural based approach can be competitive with state-of-the-art methods in terms of compression ratio.
Revised: We also provide numerical evidence that a purely neural based approach can be competitive with state-of-the-art methods in terms of compression ratio.

Original: 5. I MPLEMENTATION DETAILS
Revised: 5. I MPLEMENTATION DETAILS

Original: Dataset.
Revised: Dataset.

Original: The standard dataset for text compression is enwik8 , notably used for the Hutter Prize.
Revised: The standard dataset for text compression is enwik8 , notably used for the Hutter Prize.

Original: It is composed of 100MB of English Wikipedia data.
Revised: It is composed of 100MB of English Wikipedia data.

Original: In the compression setting, we report the bit per character ( BPC ) obtained the ﬁrst time we visit each token of the full dataset.
Revised: In the compression setting, we report the bit per character ( BPC ) obtained the ﬁrst time we visit each token of the full dataset.

Original: This differs from language modeling where we report the BPC on withheld data.
Revised: This differs from language modeling where we report the BPC on withheld data.

Original: Preprocessing.
Revised: Preprocessing.

Original: We use the CMIX preprocessor where all uppercase letters are converted to lowercase ones preceded by an escape code.
Revised: We use the CMIX preprocessor where all uppercase letters are converted to lowercase ones preceded by an escape code.

Original: This reduces the size of the vocabulary from 205 characters to 165 characters, while increasing the number of symbols by approximately 3 % on enwik8 .
Revised: This reduces the size of the vocabulary from 205 characters to 165 characters, while increasing the number of symbols by approximately 3 % on enwik8 .

Original: 10M 100M
Revised: 10M 100M

Original: Number of parameters (log scale) 1.
Revised: Number of parameters (log scale) 1.

Original: B it p e r c h a r ac t e r dim= dim= dim=
Revised: B it p e r c h a r ac t e r dim= dim= dim=

Original: Number of layers 1.
Revised: Number of layers 1.

Original: dim= dim= dim=     

Revised: dim= dim= dim=

Original: Percentage of data 1.
Revised: Percentage of data 1.

Original: dim= dim= dim=
Revised: dim= dim= dim=

Original: [Figure]
Revised: [Figure]

Original: Percentage
Revised: Percentage

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: B it p e r c h a r ac t e r
Revised: B it p e r c h a r ac t e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: BPC 1.
Revised: BPC 1.

Original: Transformer.
Revised: Transformer.

Original: We consider Transformers with either 12 , 18 or 24 layers, a hidden size d of either 256 , 512 or 1024 , and a number of heads between 4 , 8 and 16 .
Revised: We consider Transformers with either 12 , 18 or 24 layers, a hidden size d of either 256 , 512 or 1024 , and a number of heads between 4 , 8 and 16 .

Original: The dimension of the feedforward d f is 4 d .
Revised: The dimension of the feedforward d f is 4 d .

Original: We consider several size for the span, i.e., 256 , 512 and 768 , but in practice longer spans work better.
Revised: We consider several size for the span, i.e., 256 , 512 and 768 , but in practice longer spans work better.

Original: The optimizer used to train the Transformer impacts greatly the compression ratio.
Revised: The optimizer used to train the Transformer impacts greatly the compression ratio.

Original: Our best results were obtained with Adam optimizer Kingma & Ba (2015) with the correction introduced in Reddi et al.
Revised: Our best results were obtained with Adam optimizer Kingma & Ba (2015) with the correction introduced in Reddi et al.

Original: (2018) and its variant Adamw Loshchilov & Hutter (2017).
Revised: (2018) and its variant Adamw Loshchilov & Hutter (2017).

Original: The impact of the different hyperparameters involved in these optimizer is investigated in the following subsections.
Revised: The impact of the different hyperparameters involved in these optimizer is investigated in the following subsections.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: In table 1 we summarize the performance obtained with transformer networks in this report.
Revised: In table 1 we summarize the performance obtained with transformer networks in this report.

Original: the fact that other approaches require the transmission of a dictionary to the decoder makes the comparison difﬁcult: in most cases it is not accounted in the performance although it can represent up to 400kb of uncompressed data.
Revised: the fact that other approaches require the transmission of a dictionary to the decoder makes the comparison difﬁcult: in most cases it is not accounted in the performance although it can represent up to 400kb of uncompressed data.

Original: in order to obtain a comparison that makes sense, we take into account the size of the dictionary used in the different methods after compression with gzip -9 .
Revised: in order to obtain a comparison that makes sense, we take into account the size of the dictionary used in the different methods after compression with gzip -9 .

Original: the code used to train the different networks also needs to be transmitted to the decoder, but its size is negligible.
Revised: the code used to train the different networks also needs to be transmitted to the decoder, but its size is negligible.

Original: Percentage of data 1.
Revised: Percentage of data 1.

Original: B it p e r c h a r ac t e r
Revised: B it p e r c h a r ac t e r

Original: Fixed Learning Rate
Revised: Fixed Learning Rate

Original: LR=2eLR=5eLR=1ePercentage of data 1.
Revised: LR=2eLR=5eLR=1ePercentage of data 1.

Original: B it p e r c h a r ac t e r
Revised: B it p e r c h a r ac t e r

Original: Learning Rate Schedules
Revised: Learning Rate Schedules

Original: Fixed LR
Revised: Fixed LR

Original: Linear LR
Revised: Linear LR

Original: 5. I MPACT OF THE NUMBER OF PARAMETERS
Revised: 5. I MPACT OF THE NUMBER OF PARAMETERS

Original: First, we investigate the inﬂuence of the model size, as well as its depth and layer dimension, on the compression rate.
Revised: First, we investigate the inﬂuence of the model size, as well as its depth and layer dimension, on the compression rate.

Original: In many previous work, it has been observed that large overparametrized models lead to state-of-the-art performance.
Revised: In many previous work, it has been observed that large overparametrized models lead to state-of-the-art performance.

Original: For example, Vaswani et al.
Revised: For example, Vaswani et al.

Original: (2017b) notice that large transformers with up to 160 M parameters obtain better results than smaller networks.
Revised: (2017b) notice that large transformers with up to 160 M parameters obtain better results than smaller networks.

Original: Similarly, for language modeling, Dai et al.
Revised: Similarly, for language modeling, Dai et al.

Original: ; Sukhbaatar et al. (2019a); Radford et al.
Revised: ; Sukhbaatar et al. (2019a); Radford et al.

Original: (2019) present better results with larger networks.
Revised: (2019) present better results with larger networks.

Original: However, our online learning setting is different from these works, and a natural question is whether this observation is also true for compression.
Revised: However, our online learning setting is different from these works, and a natural question is whether this observation is also true for compression.

Original: More speciﬁcally, it could be possible that smaller networks would learn faster, as less parameters needs to be trained.
Revised: More speciﬁcally, it could be possible that smaller networks would learn faster, as less parameters needs to be trained.

Original: In Figure 1, we report the compression rate on enwik8 obtained by transformers of various sizes.
Revised: In Figure 1, we report the compression rate on enwik8 obtained by transformers of various sizes.

Original: The rest of the parameters, such as the optimization parameters, are identical for all runs and are close to the best we have found.
Revised: The rest of the parameters, such as the optimization parameters, are identical for all runs and are close to the best we have found.

Original: First, we observe that, as in the supervised learning setting, larger models do obtain better performance.
Revised: First, we observe that, as in the supervised learning setting, larger models do obtain better performance.

Original: In particular, our best model was also the largest one, containing almost 350 M parameters.
Revised: In particular, our best model was also the largest one, containing almost 350 M parameters.

Original: More surprisingly, larger models also obtained better compression ratio, even at the very beginning of the compression phase.
Revised: More surprisingly, larger models also obtained better compression ratio, even at the very beginning of the compression phase.

Original: In Figure 1, we also report the BPC averaged over the last 1 M characters processed by the model (instantaneous compression rate).
Revised: In Figure 1, we also report the BPC averaged over the last 1 M characters processed by the model (instantaneous compression rate).

Original: We compare three networks made oflayers: a small network with 4 heads, a hidden state of dimension d = 256 , a medium network, with 8 heads and a hidden state of dimension 512 and ﬁnally, a large network, with 16 heads and a hidden state of dimension 1024 .
Revised: We compare three networks made oflayers: a small network with 4 heads, a hidden state of dimension d = 256 , a medium network, with 8 heads and a hidden state of dimension 512 and ﬁnally, a large network, with 16 heads and a hidden state of dimension 1024 .

Original: Even after processing only one percent of the data, the large model have a better compression rate than the small and medium models.
Revised: Even after processing only one percent of the data, the large model have a better compression rate than the small and medium models.

Original: Finally, we study the inﬂuence of the depth of the network on the compression rate.
Revised: Finally, we study the inﬂuence of the depth of the network on the compression rate.

Original: In Figure 1, we report results obtained by networks of various depths, all the other dimensions being equal.
Revised: In Figure 1, we report results obtained by networks of various depths, all the other dimensions being equal.

Original: Here, we observe that using models deeper than 12 layers do not really improve the compression rate.
Revised: Here, we observe that using models deeper than 12 layers do not really improve the compression rate.

Original: In particular, as opposed to other sequence modeling tasks, it seems more efﬁcient to consider wider networks than deeper ones, for a constant number of parameters.
Revised: In particular, as opposed to other sequence modeling tasks, it seems more efﬁcient to consider wider networks than deeper ones, for a constant number of parameters.

Original: 5. I MPACT OF THE CONTEXT SIZE
Revised: 5. I MPACT OF THE CONTEXT SIZE

Original: The size of the context in transformer networks determines the maximum number of previous hidden states the model can use to predict the next symbol.
Revised: The size of the context in transformer networks determines the maximum number of previous hidden states the model can use to predict the next symbol.

Original: In Table 2, we compare the compression rate obtained by using different context sizes, from 256 to 768.
Revised: In Table 2, we compare the compression rate obtained by using different context sizes, from 256 to 768.

Original: We show that larger context size results in better performance overall.
Revised: We show that larger context size results in better performance overall.

Original: A larger memory provides to the network more information, but it
Revised: A larger memory provides to the network more information, but it

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: B it p e r c h a r ac t e r
Revised: B it p e r c h a r ac t e r

Original: Learning rate warmup=1.25% warmup=0.25% 0.
Revised: Learning rate warmup=1.25% warmup=0.25% 0.

Original: B it p e r c h a r .
Revised: B it p e r c h a r .

Original: Beta_1 Parameter 0.
Revised: Beta_1 Parameter 0.

Original: B it p e r c h a r .
Revised: B it p e r c h a r .

Original: Gradient Clipping could also be more difﬁcult to exploit: the most useful information to predict the next character is usually located in the few preceding characters.
Revised: Gradient Clipping could also be more difﬁcult to exploit: the most useful information to predict the next character is usually located in the few preceding characters.

Original: In particular, we observe that at the beginning of the compression phase, models with shorter contexts tend to perform better.
Revised: In particular, we observe that at the beginning of the compression phase, models with shorter contexts tend to perform better.

Original: To illustrate this, we report in Figure 2 the instantaneous BPC for models with context sizes 256 and 768.
Revised: To illustrate this, we report in Figure 2 the instantaneous BPC for models with context sizes 256 and 768.

Original: Finally, it is important to note that the computation time and the memory scale linearly with the size of the context.
Revised: Finally, it is important to note that the computation time and the memory scale linearly with the size of the context.

Original: 5. I MPACT OF THE LEARNING RATE SCHEDULE
Revised: 5. I MPACT OF THE LEARNING RATE SCHEDULE

Original: In general, the learning rate is a parameter having an important inﬂuence on the convergence speed of neural networks.
Revised: In general, the learning rate is a parameter having an important inﬂuence on the convergence speed of neural networks.

Original: In Figure 3, we report the compression rate, averaged over the last one million characters, obtained by using different ﬁxed learning rate with the AMSGrad algorithm.
Revised: In Figure 3, we report the compression rate, averaged over the last one million characters, obtained by using different ﬁxed learning rate with the AMSGrad algorithm.

Original: As expected, a higher learning rate leads to a better compression rate at the beginning of the run, but a smaller learning seems important to obtain good performance at the end of the compression phase.
Revised: As expected, a higher learning rate leads to a better compression rate at the beginning of the run, but a smaller learning seems important to obtain good performance at the end of the compression phase.

Original: In order to have the best of both worlds, we use a linear decay schedule, such that the learning rate is equal to zero at the end of the compression phase.
Revised: In order to have the best of both worlds, we use a linear decay schedule, such that the learning rate is equal to zero at the end of the compression phase.

Original: In Figure 3, we observe that this scheme outperforms ﬁxed learning rate schedules over most of the compression.
Revised: In Figure 3, we observe that this scheme outperforms ﬁxed learning rate schedules over most of the compression.

Original: Overall, the best ﬁxed learning rate leads to a compression rate of 1.38 BPC , while linear decay get to 1.36 BPC .
Revised: Overall, the best ﬁxed learning rate leads to a compression rate of 1.38 BPC , while linear decay get to 1.36 BPC .

Original: When training transformers, it is standard practice to use a warmup phase at the beginning of learning, during which the learning rate increases from zero to its peak value (Vaswani et al., 2017b).
Revised: When training transformers, it is standard practice to use a warmup phase at the beginning of learning, during which the learning rate increases from zero to its peak value (Vaswani et al., 2017b).

Original: We follow this practice, meaning that our linear learning schedule is determined by two parameters: the peak learning rate and the length of the warump phase.
Revised: We follow this practice, meaning that our linear learning schedule is determined by two parameters: the peak learning rate and the length of the warump phase.

Original: We report the inﬂuence of these two parameters on the overall compression rate in Figure 4.
Revised: We report the inﬂuence of these two parameters on the overall compression rate in Figure 4.

Original: It appears that a longer warmup phase is detrimental to the performance of transformer based data compression.
Revised: It appears that a longer warmup phase is detrimental to the performance of transformer based data compression.

Original: 5. O THER OPTIMIZER PARAMETERS
Revised: 5. O THER OPTIMIZER PARAMETERS

Original: Finally we found that the compression ratio is sensitive to the optimizer parameters.
Revised: Finally we found that the compression ratio is sensitive to the optimizer parameters.

Original: In this section, we evaluate the impact of the following parameters of the Adam optimizer: β 1 , β 2 and ε .
Revised: In this section, we evaluate the impact of the following parameters of the Adam optimizer: β 1 , β 2 and ε .

Original: We report the BPC over the full enwik8 data for different values of β 1 in Figure 4.
Revised: We report the BPC over the full enwik8 data for different values of β 1 in Figure 4.

Original: The fact that small β 1 s give the best results is surprising, as usually, values close to 1.0 are used.
Revised: The fact that small β 1 s give the best results is surprising, as usually, values close to 1.0 are used.

Original: For example, the default βvalue in PyTorch is 0 . 9 , and gives a compression rate of 1 . 39 BPC , compared to 1 .
Revised: For example, the default βvalue in PyTorch is 0 . 9 , and gives a compression rate of 1 . 39 BPC , compared to 1 .

Original: 36 for the optimal value of β 1 .
Revised: 36 for the optimal value of β 1 .

Original: On the other hand, we found the β 2 and ε parameters to have little inﬂuence on the compression rate, and standard values to give the best results.
Revised: On the other hand, we found the β 2 and ε parameters to have little inﬂuence on the compression rate, and standard values to give the best results.

Original: Finally, Figure 4 shows that gradient clipping is helpful, but the performance are not very sensitive to its value.
Revised: Finally, Figure 4 shows that gradient clipping is helpful, but the performance are not very sensitive to its value.

Original: [Figure]
Revised: [Figure]

Original: B it p e r c h a r ac t e r
Revised: B it p e r c h a r ac t e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Number of vectors
Revised: Number of vectors

Original: [Figure]
Revised: [Figure]

Original: 20k 1.
Revised: 20k 1.

Original: 50k 1.
Revised: 50k 1.

Original: 80k 1. 80k 1.
Revised: 80k 1. 80k 1.

Original: No n -gram information 1.
Revised: No n -gram information 1.

Original: 5. I MPACT OF PREPROCESSING AND n - GRAMS
Revised: 5. I MPACT OF PREPROCESSING AND n - GRAMS

Original: We only apply a minimal preprocessing step to the data, consisting in replacing uppercase characters by lowercase ones, with an additional escape code.
Revised: We only apply a minimal preprocessing step to the data, consisting in replacing uppercase characters by lowercase ones, with an additional escape code.

Original: This simple step, also used by other compression algorithms such as CMIX , leads to an improvement of 0.01 BPC .
Revised: This simple step, also used by other compression algorithms such as CMIX , leads to an improvement of 0.01 BPC .

Original: We also considered adding n -gram information to the input of the transformer model.
Revised: We also considered adding n -gram information to the input of the transformer model.

Original: At each time step t , the input corresponds to the current character, as well as the n -grams ending at position t .
Revised: At each time step t , the input corresponds to the current character, as well as the n -grams ending at position t .

Original: To avoid storing a dictionary of n -grams in the archive, we instead hash the n -grams into one a ﬁxed number of bins, each bin being associated to a learned embedding vector.
Revised: To avoid storing a dictionary of n -grams in the archive, we instead hash the n -grams into one a ﬁxed number of bins, each bin being associated to a learned embedding vector.

Original: We report the effect of adding n -grams in Tableand Figure 5.
Revised: We report the effect of adding n -grams in Tableand Figure 5.

Original: We show that adding this information lead to an improvement of 0 .
Revised: We show that adding this information lead to an improvement of 0 .

Original: 025 BPC , and this improvement can still be observed at the end of the compression phase.
Revised: 025 BPC , and this improvement can still be observed at the end of the compression phase.

Original: By combining the optimal parameters derived from the different experiments reported in this section, we can compress enwik8 using 1 .
Revised: By combining the optimal parameters derived from the different experiments reported in this section, we can compress enwik8 using 1 .

Original: 33 bits per character.
Revised: 33 bits per character.

Original: While this is far from the performance of CMIX , it improves previous results obtained with transformer networks (Bellard, 2019).
Revised: While this is far from the performance of CMIX , it improves previous results obtained with transformer networks (Bellard, 2019).

Original: 5. I MPACT OF REVISITING DATA
Revised: 5. I MPACT OF REVISITING DATA

Original: Finally, we study the impact of revisiting data already compressed, to improve the quality of the model and its predictions on data to be processed.
Revised: Finally, we study the impact of revisiting data already compressed, to improve the quality of the model and its predictions on data to be processed.

Original: Overall, this is a critical strategy that allows to improve the compression rate from 1.33 BPC to 1.20 BPC , for the full enwik8 dataset.
Revised: Overall, this is a critical strategy that allows to improve the compression rate from 1.33 BPC to 1.20 BPC , for the full enwik8 dataset.

Original: For this, every 512 k compressed characters, we re-use the last 12 .
Revised: For this, every 512 k compressed characters, we re-use the last 12 .

Original: 8 M characters with batch size ofto further train the network.
Revised: 8 M characters with batch size ofto further train the network.

Original: During the revisit phase, we use dropout regularization, with a rate of p = 0 .
Revised: During the revisit phase, we use dropout regularization, with a rate of p = 0 .

Original: 25 , so that the model does not overﬁt the past data.
Revised: 25 , so that the model does not overﬁt the past data.

Original: Introducing the use of dropout during the revisit improves the compression rate by approximately 0.05 BPC .
Revised: Introducing the use of dropout during the revisit improves the compression rate by approximately 0.05 BPC .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we explore the application of transformer networks to the problem of lossless data compression, which is closely related to the online learning of sequence models.
Revised: In this paper, we explore the application of transformer networks to the problem of lossless data compression, which is closely related to the online learning of sequence models.

Original: One challenge of using large neural networks to this task is the fact that many passes over the data are usually required to train them.
Revised: One challenge of using large neural networks to this task is the fact that many passes over the data are usually required to train them.

Original: We thus propose to periodically revisit tokens that were already compressed to further train the model, and improve its predictions.
Revised: We thus propose to periodically revisit tokens that were already compressed to further train the model, and improve its predictions.

Original: We also perform a thorough study of the impact of various hyper-parameters, such as the model architecture, on the compression rate.
Revised: We also perform a thorough study of the impact of various hyper-parameters, such as the model architecture, on the compression rate.

Original: Based on this study, we show that a transformer based approach obtains state-of-the-art results on the competitive enwik8 benchmark, wihtout using hand crafted features.
Revised: Based on this study, we show that a transformer based approach obtains state-of-the-art results on the competitive enwik8 benchmark, without using hand crafted features.