Paper:
Character-level language modeling based on Transformer has brought great success by alleviating limitation of recursive operation. However, existing Transformer-based models require substantial computational resources, which hinders the usability of character-level language models in applications with limited resources. In this paper, we propose a lightweight model, called GroupTransformer , that factorizes the calculation paths by grouped embedding operators. Additionally, Group-Transformer employs inter-group linear operators to prevent performance degradation from the group strategy. With comparison experiments on about five times larger and the best performing LSTM-based models and compatible parameter size of Transformer-based models, we show that GroupTransformer has better performance on two benchmark tasks, enwik8 and text8. Further experiments including ablation studies and qualitative analysis revealed that the proposed work contributes to the effective lightweight model for practical application. The implementation code will be available.

Review:
Summary: This paper proposes a lightweight alternative to the design of self-attention based Transformers on character-level language modeling (LM). The approach was motivated by the similar technique that has been applied on group convolutions, but with a few notable differences too, such as inter-group mixing and low-rank approximation (which also appeared in ConvNets before, but this still strkes me as a difference in the Transformer context). Via experiments on two large-scale char-level LM datasets as well as a relatively extensive set of ablative experiments, the authors demonstrated the effectiveness of their approach. 
Pros:
+ A very well-written paper. Most of the math symbols in the paper come with clear dimensionalities, which make it very easy to follow. The description for the methodology is also pretty clear. 
+ Well-designed experiments. Enwik-8 and text8, while widely used to benchmark Transformers these days, are still very challenging large-scale tasks. The authors also provide a series of ablative studies comparing the group-transformer with the original transformer in Table 3.
+ Table 2 and Figure 3 (in the Appendix) are pretty strong proof of the effectiveness of the approach (at least on character-level language modeling). 
================================
A few questions/issues/comments:
1. For the key/value computation, why did you still keep the "more complex/expensive" $D_\text{model}^2$ design? You explained in the paper that they could "come from other source domain", but in the specific case of character-level language modeling (in which you are just using a decoder Transformer without encoder-decoder attention), I don't think this is a problem. Why not make $\mathbf{k}_{gh}$ and $\mathbf{v}_{gh}$ something similar to how you compute the query? Or alternatively, why don't you make them low-rank too, as in the feed-forward layer? This difference in design seems strange to me.
2. In Section 3.4, you mentioned that the Group-Transformer (I'll call it GT for simplicity below) has resource complexity $O(D_\text{model}^2/G)$ whereas the original Transformer has complexity $O(D_\text{model}^2)$. However, this is not true by your design of the key/value module, and by your own analysis in Appendix B.1, where you still have a $2 D_\text{model}^2$ term. Therefore, I suggest reworking on Section 3.4, as the big-O complexity of the parameter space should be the same. (This again makes me curious about question (1) above...)
3. Section 4.1 says that you only explored group size from {2, 4}. How did you pick this number? Why not 8 groups or more? As the 2-group option only saves about 10%-15% of the parameters (according to your analysis in Appendix B), it's actually not a large difference. Meanwhile, it seems 2-group is always better than 4-group. While I guess the 8-group option would certain make the model size very small, I'm very curious to see how good/bad it is when you match the # of parameters of an 8-group GT with a {2,4}-group GT.
4. As the "lightweight" property of GT is what you are focusing on, could you also show/approximate the number of FLOPs used by LSTMs in Table 1? While LSTMs use more parameters, they don't use as much computation as do the Transformers (which has needs to form a $O(L^2)$ matrix in the self-attention module, where $L$ is the sequence length). Also, I think it's important to show the actual (wall-clock) runtime comparison of GT with Transformer-XL and the best LSTM model(s).
5. I find it a bit strange (and slightly disappointing) that this method does not generalize that well to word-level language modeling, as none of the designs introduced in the paper are specific to "character"-level modeling alone. How's the performance of GT if you forget about the word embedding compression for a while (e.g., use a large embedding size, such as 500 like in prior works)? Some recent work [1] seems to suggest that very small Transformer-XL (only 4M parameters + a normal embedding) can achieve a perplexity around 35, too.
------------------------------------
Some issues that did not really affect the score:
6. In Secton 3.2 (currently at the bottom of page 3), maybe add the dimensionality of $\mathbf{x}$ (which should be $D_\text{model}$) just for clarity, as you are omitting the "time" dimension (of a sequence) and only considering a single time step.
7. Right after Eq. (2), the first $\mathbf{W}_{gh}^\text{m-intra}$ should be $\mathbf{W}_{gh}^\text{o-intra}$.
8. In Eq. (4) (and the sentence following it), $W_{hg}^\text{f2}$ shouldn't have a reference to $h$, as the reference to heads should only be in the self-attention.
9. Eq. (7) intra -> inter.
10. Some descriptions in Appendix A are confusing. For instance, you didn't really define function $\text{Shuffle}(\cdot)$, and it took me a while to realize you mean transposing the 0th and 2nd dimension of a $G \times M \times G$ matrix. Similarly, the $\text{Concat}(\cdot)$ function in Eq. (7) is "undefined", in the sense that its input is already a $G \times M$ matrix (each row is a $1 \times M$ vector). I think what you want is to vectorize it to shape $1 \times (M * G)$ , and $\mathbf{W}_g^\text{intra[2]}$ should have shape $(M * G) \times \bar{D}_\text{group}$. I suggest you revise an clarify this part.
6. I'm curious (and wonder if you've tried this): What if you increase the model size of the Group-Transformer to be as large as the original Transformer on enwik-8 and text8 (e.g., 40M)? How does the GT perform? While Table 3 is indeed convincing, the result obtained by GT is still far from the actual SOTA (e.g., obtained by Child et al. [2] with a much larger model). Would be interesting to compare how a model "as large" would do.
------------------------------------
Overall, I think this is a promising strategy that seems to work very well on character-level language modeling. My only major concerns are some of the specifics of the design of the methodology (e.g., the key/value part) and the failure of the approach to generalize to the very-relevant domain such as word-level LM.
[1] https://arxiv.org/abs/1909.01377
[2] https://arxiv.org/abs/1904.10509