  Transfer learning in natural language processing (NLP), as realized using
models like BERT (Bi-directional Encoder Representation from Transformer), has
significantly improved language representation with models that can tackle
challenging language problems. Consequently, these applications are driving the
requirements of future systems. Thus, we focus on BERT, one of the most popular
NLP transfer learning algorithms, to identify how its algorithmic behavior can
guide future accelerator design. To this end, we carefully profile BERT
training and identify key algorithmic behaviors which are worthy of attention
in accelerator design.
  We observe that while computations which manifest as matrix multiplication
dominate BERT's overall runtime, as in many convolutional neural networks,
memory-intensive computations also feature prominently. We characterize these
computations, which have received little attention so far. Further, we also
identify heterogeneity in compute-intensive BERT computations and discuss
software and possible hardware mechanisms to further optimize these
computations. Finally, we discuss implications of these behaviors as networks
get larger and use distributed training environments, and how techniques such
as micro-batching and mixed-precision training scale. Overall, our analysis
identifies holistic solutions to optimize systems for BERT-like models.
