Paper:
Transformers have replaced long-short term memory and other recurrent neural networks variants in sequence modeling. It achieves state-of-the-art performance on a wide range of tasks related to natural language processing, including language modeling, machine translation, and sentence representation. Lossless compression is another problem that can benefit from better sequence models. It is closely related to the problem of online learning of language models. But, despite this ressemblance, it is an area where purely neural network based methods have not yet reached the compression ratio of state-of-the-art algorithms. In this paper, we propose a Transformer based lossless compression method that match the best compression ratio for text. Our approach is purely based on neural networks and does not rely on hand-crafted features as other lossless compression algorithms. We also provide a thorough study of the impact of the different components of the Transformer and its training on the compression ratio.

Review:
Summary: 
The paper investigates using the transformer architecture for neural network-based lossless compression of text. The resulting model, obtained through a thorough investigation of the architecture hyper-parameters are on par with standard SOTA compression. The paper is well-written. In particular the authors have done a great job reviewing existing compression literature and positioning their method within the space of prior work.
Recommendation: Weak Reject
While the paper considers an interesting application of the Transformer architecture, and is well-written, it is of limited novelty. Specifically, the bulk of the paper is concerned with describing experimental results of a thorough (but standard) hyper-parameter search - considering things like Transformer context size, learning rate (schedule), number of layers and key, value, query dimensionality; and does not offer any new architectural modifications / insights.
Furthermore, only a single dataset - enwik8 - is considered in the experimental validation and little attention is given to the description of the dataset split and any distribution differences between splits. Taken together, the existing experimental setup potentially creates an unfair advantage for the neural network-based methods - while the standard methods can be expected to perform similarly across a wide range of datasets / texts, the neural-network based methods have been trained and tested on very similar data and could be expected to perform well on these data, but not in case of a distributional shift (e.g. compressing legal texts instead of Wikipedia). The paper does not answer the question of whether or not this is true.
Furthermore, similar to autoregressive models, transformers are known to be slow at inference time. I expect this to lead to very slow decoding. Therefore, methods in table 1 should be compared in compression/decompression time to give a better overview of the practical impact of this work. 
Taken together, in its current form the paper may be better suited for a workshop publication rather than a full conference paper.
Major comments:
1. For reasons mentioned above, the paper should include additional experimental evaluation. In particular, it should consider the effect of training the model on one dataset, but evaluating it on another dataset; and discuss how differences in performance (if any) compare to standard methods.
2. Compression/decompression times of the proposed method should be compared against the other compression methods in table 1. I expect the proposed transformer to be slow at decompressing.
3. The paper does not contain the loss that the transformer model was used to optimize. I assume that it is the softmax cross entropy, but this is worth mentioning explicitly. It would also be worthwhile to explain the training procedure - for how many epochs was the model trained (see also next question), what was the dataset size? 
4. Description of the “training with revisits” is not very clear. My understanding is that it resembles a pass through the data, where some of it is considered again at specific intervals. My first assessment is that this should not be necessary - the data should already be considered multiple times during the training process.
a) The authors should provide a more detailed description of the training-with-revisits procedure, contrasting it specifically with a procedure where revisits are not done (i.e. normal training).
b) If the goal of the revisits training is to observe some training examples more than once, then it would be very interesting if simply training for a longer time (several epochs == passes through the data) has a similar effect.
c) Is there any motivation for the choice of the revisits hyper-parameters F and M? Was a different batch size used during the revisits training? Is the learning rate evolved during the revisits training phase or is it still decayed?
Minor comments:
1. There is some prior work on using Neural Networks for lossless image compression (e.g. [1], [2]. [3] that achieves SOTA compression ratios compared to standard methods. It may be interesting for the readers to mention these results. In particular the authors’ statement that “[...] purely neural network based models are still far from state of the art [...]” may give the wrong impression to the readers.
2. The authors mention that they “[...] propose several improvements to its (the Transformer) architecture and training to accelerate and stabilize [...] training”. In my view, the experiments described in the paper resemble a hyper-parameter search more than architectural improvements. The authors may want to clarify in the text which specific improvements they refer to.
3. Page 1, last paragraph: “[...] of all the important component [...]” -> “[...] of all the important components [...]”
4. Page 3: “[...] attention span size across all layers as it suggested [...]” -> “[...] attention span size across all layers as was suggested [...]”
5. Page 3: Missing references.
6. Page 3: Use of small n and capital N when talking about n-grams. Should be made consistent.
7. Page 8 (Conclusion): “wihtout” -> “without”
[1] F. H. Kingma, P. Abbeel, and J. Ho. Bit-Swap: recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning (ICML), 2019.
[2] Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer Discrete Flows and Lossless Compression. arXiv e-prints, 2019.
[3] Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with Flows via Local Bits-Back Coding. arXiv e-prints, 2019.