  The advent of the Transformer can arguably be described as a driving force
behind many of the recent advances in natural language processing. However,
despite their sizeable performance improvements, as recently shown, the model
is severely over-parameterized, being parameter inefficient and computationally
expensive to train. Inspired by the success of parameter-sharing in pretrained
deep contextualized word representation encoders, we explore parameter-sharing
methods in Transformers, with a specific focus on encoder-decoder models for
sequence-to-sequence tasks such as neural machine translation. We perform an
analysis of different parameter sharing/reduction methods and develop the
Subformer, a parameter efficient Transformer-based model which combines the
newly proposed Sandwich-style parameter sharing technique - designed to
overcome the deficiencies in naive cross-layer parameter sharing for generative
models - and self-attentive embedding factorization (SAFE). Experiments on
machine translation, abstractive summarization, and language modeling show that
the Subformer can outperform the Transformer even when using significantly
fewer parameters.
