From back to the roots into the gated woods: Deep learning for NLP (2021)

Abstract: Deep neural networks have revolutionized many fields, including Natural Language Processing. This paper outlines teaching materials for an introductory lecture on deep learning in Natural Language Processing (NLP). The main submitted material covers a summer school lecture on encoder-decoder models. Complementary to this is a set of jupyter notebook slides from earlier teaching, on which parts of the lecture were based on. The main goal of this teaching material is to provide an overview of neural network approaches to natural language processing, while linking modern concepts back to the roots showing traditional essential counterparts. The lecture departs from count-based statistical methods and spans up to gated recurrent networks and attention, which is ubiquitous in today’s NLP.

## introduction in 2015, the "deep learning tsunami" hit our field @xcite . in 2011, neural networks were not really "a thing" yet in nlp; they were even hardly taught. i remember when in 2011 andrew ng introduced the free online course on machine learning, which soon after led daphne koller and him to start massive online education: "many scientists think the best way to make progress [..] is through learning algorithms called neural networks". 1 ten years later, it is hard to imagine any nlp education not touching upon neural networks and in particular, representation learning. while neural networks have undoubtedly pushed the field, it has led to a more homogenized field; some lecturers even question whether to include pre-neural methods. i believe it is essential to teach statistical foundations besides modern dl, ie., to go back to the roots, to better be equipped for the future. 2 1 accessed march 15, 2021: https://bit.ly/ 2ozh2mz ## complementary notebooks of earlier material this lecture evolved from a series of lectures given earlier, amongst which a short course given in malta in 2019, and a msc-level course i taught at the university of groningen (language technology project). to complement the keynote slides of the summer school lecture provided here, earlier jupyter notebooks can be found at the website. these cover a subset of the material above. ## conclusions this short paper outlines teaching material for an introductory lecture on deep learning for nlp. by releasing this teaching material, i hope to contribute material that fellow researchers find useful when teaching introductory courses on dl for nlp. for comments and suggestions to improve upon this material, please reach out to me.