  One of the challenges for current sequence to sequence (seq2seq) models is
processing long sequences, such as those in summarization and document level
machine translation tasks. These tasks require the model to reason at the token
level as well as the sentence and paragraph level. We design and study a new
Hierarchical Attention Transformer-based architecture (HAT) that outperforms
standard Transformers on several sequence to sequence tasks. In particular, our
model achieves stateof-the-art results on four summarization tasks, including
ArXiv, CNN/DM, SAMSum, and AMI, and we push PubMed R1 & R2 SOTA further. Our
model significantly outperforms our document-level machine translation baseline
by 28 BLEU on the WMT19 EN-DE document translation task. We also investigate
what the hierarchical layers learn by visualizing the hierarchical
encoder-decoder attention. Finally, we study hierarchical learning on
encoder-only pre-training and analyze its performance on classification
downstream tasks.
