Interactive Assignments for Teaching Structured Neural NLP (2021)

Abstract: We present a set of assignments for a graduate-level NLP course. Assignments are designed to be interactive, easily gradable, and to give students hands-on experience with several key types of structure (sequences, tags, parse trees, and logical forms), modern neural architectures (LSTMs and Transformers), inference algorithms (dynamic programs and approximate search) and training methods (full and weak supervision). We designed assignments to build incrementally both within each assignment and across assignments, with the goal of enabling students to undertake graduate-level research in NLP by the end of the course.

## overview our course contains five implementation projects focusing on neural methods for structured prediction tasks in nlp. over a range of tasks from language modeling to machine translation to syntactic and semantic parsing, the projects cover methods such as lstms and transformers, dynamic programming, beam search, and weak supervision (learning from denotations). our aim was to let students incrementally and interactively develop models and see their effectiveness on real nlp datasets. section 2 describes the tasks and objectives of the projects in more detail. links to assignments are available at https://sites.google.com/ view/nlp-assignments . ## findings in initial course offerings an initial iteration of the course was taught to 60 students, and an offering for over 100 students is in progress. overall, we have found the projects to be a great success. in the first iteration of the course, 81% of students completed the course and submitted all five projects. from a mid-semester survey, students reported taking 19.88 hours on average to complete project 1. we observed that students with no prior experience programming with deep learning frameworks took significantly longer on the projects and required more assistance. in future semesters, we intend to strengthen the deep learning prerequisites required for the course to ensure adequate background for success. the online project infrastructure worked with minimal issues. while the colab platform does have the downside of timeouts due to inactivity, we believe the use of free gpu resources outweighed this cost. by encouraging students to download checkpoints after training runs, they were able to avoid re-training after each timeout. a handful of students who spent very long stretches of time on the projects in a single day reported having temporary limits placed on gpu usage (e.g. after 8+ hours of continuous use), but these limits could be circumvented by logging in with a separate account. most students were able to successfully complete the majority of each assignment, but there remained some point spread to distinguish performance for grading (mean 94%, standard deviation 12%). due to the use of autograding, instructor code grading effort totaled less than several hours per project. one aspect of grading that we are continuing to improve is the open-ended exploration report, and we plan to better define and communicate expectations in future semesters by introducing a clear rubric for the report section. overall, these projects proved a valuable resource for the success of our course and for preparing students for nlp research. at the end of last year's course, one student submitted an extension of their exploration work on one project to emnlp and presented at the conference.