Paper:
We propose R2D2 layers, a new neural block for training efficient NLP models. Our proposed method is characterized by a dynamic weight diffusion mechanism which learns to reuse and reduce parameters in the conventional transformation layer, commonly found in popular Transformer/LSTMs models. Our method is inspired by recent Quaternion methods which share parameters via the Hamilton product. This can be interpreted as a neural and learned approximation of the Hamilton product which imbues our method with increased flexibility and expressiveness, i.e., we are no longer restricted by the 4D nature of Quaternion weight sharing. We conduct extensive experiments in the NLP domain, showing that R2D2 (i) enables a parameter savings of up to 2× to 16× with minimal degradation of performance and (ii) outperforms other parameter savings alternative such as low-rank factorization and Quaternion methods.

Review:
Paper Summary:
This paper proposes to train smaller models by decomposing the weights of fully connected networks as the product of smaller matrices, along with a reordering/transpose of the outcome. The experiments shows that models with less parameters yield comparable performance with their larger counterparts.
Review Summary:
The method is technically sound and the paper reads well. Experiments demonstrate the efficacy of the method, although some ablations are missing (see below). The paper is however not clear on the ultimate objective of the method (speed/accuracy/generalisation?) and does not compare with alternatives.
Detailed Review:
The introduction does not make clear if your motivation to make model smaller is training speed, inference speed, memory usage, generalization accuracy. Please clarify.
The explanation of the method, i.e. Section 2.2.1, is not clear, in particular for the mapping \psi. I feel it would cleared if somewhere in the paper there was an equation with the element-wise correspondence, i.e. H_{?,?} = \sum_k A_i,k S_k,j
In that section, you should introduce that n is a hyperparameter before using it as well.
In that section, you could also discuss parameter initialization, and whether this model can use weight decay over H or A/S. it is also not clear to me if you control the norm ratio between A and S given the weight magnitude is over parameterized.
The experimental section lack a validation/ablation study to help the reader understand the interplay between the number of blocks and the number of latent dimensions. It will also be good to show learning curves to compare training speed of different parameterization. 
Also no training errors are reported, does your method can be seen as a regularizer, i.e. is training objective closer to valid objective when n grows? Did you have to change other regularization parameters like dropout.
To me the main weakness of the paper lies in the lack of comparison with alternatives. Replacing fully connected layers with alternative has a rich literature that the authors ignore.
I feel it is necessary to compare the approach with
(i) block diagonal approaches, popular since ResNext for convolutions but equally applicable to linear layers. https://arxiv.org/abs/1611.05431
(ii) other form of structured sparsity. https://arxiv.org/abs/1902.09574 (survey). https://arxiv.org/abs/1812.08301 (squantizer) https://arxiv.org/abs/1802.08435 (block sparsity)...
(iii) distillation of large models into smaller models.  https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1702.01802
(iv) it might not be necessary to compare, but at least mentioning approaches which predict weights from a meta network would be good.  https://arxiv.org/abs/1609.09106
As a reviewer, I am a bit annoyed that made no effort to have a decent list of related work and that they delegate that work to the reviewers to do so.
Details:
"transformation layer": this is not common terminology, please prefer linear layer or fully-connected layer.
please define all acronyms, e.g. FC.
The experimental section does not define \alpha (end of page 6).