High Performance Natural Language Processing (2020)

Abstract: Scale has played a central role in the rapid progress natural language processing has enjoyed in recent years. While benchmarks are dominated by ever larger models, efficient hardware use is critical for their widespread adoption and further progress in the field. In this cutting-edge tutorial, we will recapitulate the state-of-the-art in natural language processing with scale in perspective. After establishing these foundations, we will cover a wide range of techniques for improving efficiency, including knowledge distillation, quantization, pruning, more efficient architectures, along with case studies and practical implementation tricks.

## tutorial proposal recent advances in natural language processing @xcite ; @xcite ; @xcite ; @xcite , among many others) have substantially improved model capabilities. notably, pre-trained checkpoints can be fine-tuned without substantial task specific modifications to create powerful models for a wide range of tasks @xcite . for many applications, production systems with models up to date with the state-of-the-art are meeting high quality bars for adoption across a wide variety of language tasks. however, the ever larger computational requirements of such cutting-edge models-which quickly approximates the scale of a trillion parameters @xcite )-imposes challenges to their widespread adoption and further progress in the field. this has driven increasing attention to methods that allow more efficient use of hardware, through techniques such as knowledge distillation @xcite , quantization @xcite , pruning @xcite , and architectural changes @xcite al. (2020); wang et al. (2020b); katharopoulos et al. (2020); zaheer et al. (2020) , among others). altogether, these techniques are promising avenues for more efficient natural language processing. this tutorial starts with an introduction covering recent trends in nlp with scale in perspective, and covers foundational knowledge such as the transformer architecture @xcite and the fine-tuning paradigm. we then move to core techniques for improving efficiency, including knowledge distillation, quantization and pruning, later covering recent work on architectural improvements, focusing on the move towards self-attention with linear complexity. then, we dive into case studies by examining specific models such as iandola et al. (2020) and @xcite . finally, we end with practical implementation considerations including model and data parallelism, gradient accumulation and floating point precision, ending the tutorial with closing notes and a questions and answers section. we outline the structure of this tutorial in