Contemporary NLP Modeling in Six Comprehensive Programming Assignments (2021)

Abstract: We present a series of programming assignments, adaptable to a range of experience levels from advanced undergraduate to PhD, to teach students design and implementation of modern NLP systems. These assignments build from the ground up and emphasize full-stack understanding of machine learning models: initially, students implement inference and gradient computation by hand, then use PyTorch to build nearly state-of-the-art neural networks using current best practices. Topics are chosen to cover a wide range of modeling and inference techniques that one might encounter, ranging from linear models suitable for industry applications to state-of-the-art deep learning models used in NLP research. The assignments are customizable, with constrained options to guide less experienced students or open-ended options giving advanced students freedom to explore. All of them can be deployed in a fully autogradable fashion, and have collectively been tested on over 300 students across several semesters.

## introduction this paper presents a series of assignments designed to give a survey of modern nlp through the lens of system-building. these assignments provide hands-on experience with concepts and implementation practices that we consider critical for students to master, ranging from linear feature-based models to cutting-edge deep learning approaches. the assignments are as follows: a1. sentiment analysis with linear models @xcite on the stanford sentiment treebank @xcite . a2. sentiment analysis with feedforward "deep averaging" networks @xcite using glove embeddings @xcite . a3. hidden markov models and linear-chain conditional random fields (crfs) for named entity recognition (ner) @xcite , using features similar to those from @xcite . a4. character-level rnn language modeling @xcite . a5. semantic parsing with seq2seq models @xcite on the geoquery dataset @xcite . a6. reading comprehension on squad @xcite using a simplified version of the drqa model @xcite , similar to bidaf @xcite . a1-a5 come with autograders. these train each student's model from scratch and evaluate performance on the development set of each task, verifying whether their code behaves as intended. the autograders are bundled to be deployable on gradescope using their docker framework. 2 these coding assignments can also be supplemented with conceptual questions for hybrid assignments, though we do not distribute those as part of this release. other courses and materials several other widely-publicized courses like stanford cs224n and cmu cs 11-747 are much more "neural-first" views of nlp: their assignments delve more heavily into word embeddings and low-level neural implementation like backpropagation. by contrast, this course is designed to be a survey that also covers topics like linear classification, generative modeling (hmms), and structured inference. other hands-on courses discussed in prior teaching nlp papers @xcite @xcite make some similar choices about how to blend linguistics and cs concepts, but our desire to integrate deep learning as a primary (but not the sole) focus area guides us towards a different set of assignment topics. ## design principles this set of assignments was designed after we asked ourselves, what should a student taking nlp know how to build? nlp draws on principles from machine learning, statistics, linguistics, algorithms, and more, and we set out to expose students to a range of ideas from these disciplines through the lens of implementation. this choice follows the "text processing first" @xcite or "core tools" @xcite views of the field, with the idea that students can study undertake additional study of particular topic areas and quickly get up to speed on modeling approaches given the building blocks presented here. ## deployment these assignments have been used in four different versions of an nlp survey course: an upperlevel undergraduate course, a masters level course (delivered online), and two phd-level courses. in the online ms course, these constitute the only assessment. for courses delivered in a traditional classroom format, we recommend choosing a subset of the assignments and supplementing with additional written assignments testing conceptual understanding. our undergrad courses use a1, a2, a4, and a final project based on a6. we use additional written assignments covering word embedding techniques, syntactic parsing, machine translation, and pre-trained models. our phd-level courses use a1, a2, a3, a5, and an independent final project. the assignments also support further "extension" options: for example, in a3, beam search is presented as optional and students can also explore syllabus logistics we typically provide students around 2 weeks per assignment. their submission either consists of just the code or a code with a brief report, depending on the course format. students collaborate on assignments through a discussion board on piazza as well as in person. we have relatively low incidence of students copying code, assessed using moss over several semesters.