  Transformers have outperformed recurrent neural networks (RNNs) in natural
language generation. This comes with a significant computational overhead, as
the attention mechanism scales with a quadratic complexity in sequence length.
Efficient transformer variants have received increasing interest from recent
works. Among them, a linear-complexity recurrent variant has proven well suited
for autoregressive generation. It approximates the softmax attention with
randomized or heuristic feature maps, but can be difficult to train or yield
suboptimal accuracy. This work aims to convert a pretrained transformer into
its efficient recurrent counterpart, improving the efficiency while retaining
the accuracy. Specifically, we propose a swap-then-finetune procedure: in an
off-the-shelf pretrained transformer, we replace the softmax attention with its
linear-complexity recurrent alternative and then finetune. With a learned
feature map, our approach provides an improved tradeoff between efficiency and
accuracy over the standard transformer and other recurrent variants. We also
show that the finetuning process needs lower training cost than training these
recurrent variants from scratch. As many recent models for natural language
tasks are increasingly dependent on large-scale pretrained transformers, this
work presents a viable approach to improving inference efficiency without
repeating the expensive pretraining process.
