Paper:
Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations, longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT . Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large.

Review:
The authors present ALBERT, a modification of the BERT architecture with substantially fewer parameters. They show that despite being much smaller, the performance is very strong and achieves state of the art on a variety of different tasks. There are several ideas proposed here: embedding factorization, sharing layers, and sentence ordering as a training objective. 
1. The point that naively increasing the size of the BERT architecture does not work is a good one, but the authors don't acknowledge that this is tied up in the effect of regularization. Cross layer parameter sharing has a regularization effect that simply scaling up BERT large to x-large or such sizes does not have. This is also an issue with the authors making the statement that they are the first to show that dropout is harmful for Transformers. This is a large generalization that seems to be a special case of not only the regularized architecture they propose but also the large quantity of data that the model still underfits to.
2. The authors propose embedding factorization to reduce the number of parameters in the embedding dimension. This is very intuitive, but the authors do not cite or compare to related approaches. I understand these models are computationally intensive and thus do not expect large quantities of detailed ablations. However, this kind of dimensionality reduction has been explored with other techniques, for example for knowledge distillation, quantization, or even adaptive input/softmax (and with subword as well, not just whole word modeling). These techniques have also been applied to machine translation models, which do not use them to learn rare words. I believe a better discussion of these methods should be added to the paper, as this is not a novel proposition.
3. A large takeaway I have from this paper is that parameter size is not a good metric. While ALBERT is substantially smaller, the authors do not make it clear that this model is very slow at inference time due to the large size. This raises several questions: is it better to have models that are deeper or more wide? Can the authors actually report the latency in a comparative table next to BERT? Can the authors provide a sense of how large this model is in MB - e.g. presumably a goal of less parameters would be to have a model with less memory, but then the decision between memory and latency that different models make should be made more clear.
4. Section 4.8 is not clear. Exactly how much data, in terms of GB of uncompressed text, is used here? Is it the data of XLNet and RoBERTa, so larger than both of those settings individually? Further, the authors train for 1 million steps. This is larger than both XLNet and RoBERTa, is that correct? Or there is some detail about the size of the batch that actually makes it comparable? The many small tables where the changes are not clearly delineated makes it difficult to compare results.