Learning How To Learn NLP : Developing Introductory Concepts Through Scaffolded Discovery (2021)

Abstract: We present a scaffolded discovery learning approach to introducing concepts in a Natural Language Processing course aimed at computer science students at liberal arts institutions. We describe some of the objectives of this approach, as well as presenting specific ways that four of our discovery-based assignments combine specific natural language processing concepts with broader analytic skills. We argue this approach helps prepare students for many possible future paths involving both application and innovation of NLP technology by emphasizing experimental data navigation, experiment design, and awareness of the complexities and challenges of analysis.

## introduction discovery learning describes a pedagogical framing where, instead of introducing students to a skill and then using assessments to explicitly practice that skill, students are given a broad objective and "discover" pertinent concepts and skills in pursuit of that objective. though pedagogical research yields unimpressive results for pure discovery learning @xcite , several works support the effectiveness of guided discovery learning @xcite , where students are provided scaffolding for how to develop their solutions and regular opportunities for feedback or validation. this type of instructional approach offers the opportunity to exercise creativity and to take more ownership of their own sensemaking process. in this paper, we present a proof of concept of the merits of scaffolded discovery learning by describing our implementation of four guided discovery learning exercises 1 to anchor the first four weeks of an undergraduate natural language processing course. we also detail why we have found this 1 materials for these four assignments are available for use at https://github.com/discovernlp . ## motivation there are several strategic benefits to a discovery learning approach for nlp. first, the traditional computer science approach to nlp instruction tends to emphasize programming and/or core algorithms first @xcite , which requires students to jump immediately into implementing widely accepted algorithms for particular tasks. while this may lead to a more satisfying performance outcome, it may miss the opportunity for students to engage with why particular choices, such as how to tokenize text or how to smooth an n-gram model, actually make sense from a linguistic perspective. these skills are important when transferring to new domains, where the optimal choices might change and require further assessment. second, the choice of which model is the mostly "widely accepted" has been shown to change rapidly. examining the curricular recommendations made by the acm/ieee joint task force for computing curricula over time, we see speech recognition and parsing as emphasized topics in 2001, but no specific discussion of n-gram models (acm/ieee, 2001). in contrast, probabilistic models and ngrams appear in the 2013 version, while speech recognition disappears as an emphasized topic and parsing remains on the syllabus in vaguer terms (acm/ieee, 2013). while familiarity with these different problem spaces is helpful for building context and approaches to new problem domains, it is hard to predict if the depth of knowledge required to fully implement a conditional random field (crf) or a deep neural net for machine translation will be useful beyond the next few years. in contrast, our discovery-learning labs focus on how to set up experimental data and protocols, select and interpret appropriate metrics, use those metrics to investigate what textual phe-nomena they actually embody, and react to that knowledge with possible design interventions. we have found these skills not only generalize better across different models and problem domains, but also dominate the actual time spent doing our nlp work as researchers. third, this strategy of teaching is more compatible with the structure of the curricula at the undergraduate-only institutions where we teach. upper-level courses such as natural language processing have shallow prerequisites and students have uneven backgrounds in math, computer science and linguistics. this challenge is shared by both computer science and linguistics undergraduate programs @xcite and graduate courses aimed at an information studies audience @xcite . emphasizing skills of evaluation and model discovery over detailed model analyses helps ensure that students comfortable with data structures and some probability will be able to succeed and develop meaningful skills in the course. in our context, it has the additional benefit of inviting a liberal arts approach to critique the nature of what we evaluate in nlp tasks, as well as to consider the implications of the deployment of language technologies. ## learning objectives we present four labs as a central element of the first four weeks of a natural language course. in developing these exercises, we emphasize three skills we hope students develop through these assignments: analysis of quantitative results, text exploration, and generalizing concepts to new models. while some of these fall into the classic "text processing first" paradigm of course offered as an applied linguistics approach by @xcite , it does not do so at the exclusion of understanding the "why" of related algorithms and models. notably, none of these skills specify particular models in natural language processing, though the labs we present do have some alignment with different classic nlp course topics. this is a deliberate alternative approach: by de-emphasizing outcomes focused on "knowing" particular algorithms and models, we provide more opportunity to practice skills to approach and interpret new models that students may choose to explore later in the course. analysis of quantitative results. our work aims to support a progression not in terms of complexity of models, but in the levels of critique students apply to computational work on text. by building from some of the basic challenges of what it means to read, process, and write text in a computer through the development of increasingly complex metrics, students naturally have the opportunity to progress from making comments based on initially observed phenomena to making the case for whether results are surprising using typical metrics of the field. unlike work that primarily privileges meeting certain minimum accuracy thresholds, these exercises intentionally include analyses of unimpressive results, helping to emphasize important lessons about how seemingly large quantitative differences don't always indicate significance in the vast, sparse universe of language. exploration of the text. natural language processing benchmarks often present results in ways that obscure the contents of the data being evaluated, especially in cases where test data is totally hidden. students new to the field of nlp often do not have strong intuitions around how much variation exists even in a limited problem domain. examining system inputs and outputs helps them develop intuitions about language data and what it means to be "unusual" in a highly sparse space. our discovery approach helps reinforce the importance of including both quantitative experimental results and qualitative discussions using examples about model behavior with respect to the text, as both contribute to an understanding of what a model has (or hasn't) done. thinking beyond individual models. an important component of practicing natural language processing is implementing models and comparing their performance. however, current "state of the art" models often are built using a combination of different fairly detailed neural architectural choices. engaging with implementing these can require either many prerequisite courses, many days of in-class time to establish working knowledge of neural networks for sequential data, or a strategy of teaching how to code the models that eschews why to use these pieces. further, with little confidence that these models won't be obsolete in two years, it can be an expensive investment to set up curricular materials for these topics without knowing whether they will still merit reuse the next time the course is offered. because our approach de-emphasizes the specific models implemented in favor of understanding broader skills around evaluation and comparison, we believe it will better prepare students to teach themselves and to ask meaningful research questions when approaching new work. ## course format the format of the two courses using these labs include both a lecture-focused larger class format and a smaller discussion-based format. in both, students are expected to read some text from the textbook, jurafsky and martin (2020), supplemented by papers on relevant nlp topics. outside of course readings, the lab work constitutes the vast majority of the work completed outside of class for the first half of the semester. in both formats, we reserve one day a week for specifically setting up and starting lab assignments. students attend "lab day" in order to start work on the lab exercise for that week. the lab relies on the concepts discussed in class. though the lab assignment is started during the lab period, the expectation is that the lab work will be completed outside of class and due the following week. ## lab assignment progression the following subsections describe the progression of four laboratory assignments intended for the first four weeks of the course. each lab assignment has a learning outcome related to analytical skills in natural language processing. each of the four labs presented also focuses on introducing content from @xcite . these assignments are implemented in python, with starter code provided through github. the assignments can be coupled with autograders for the code managed through gradescope. 2 when used, the autograders (which account for less than half the points in the student's final write-up) can serve as extra scaffolding, as they provide a useful on-demand tool for students to check whether their code is correct before writing their analysis. each lab is accompanied by an extensive lab write-up, which breaks the exploration topic into individual implementation goals, check-ins, and questions for students to address as they venture through the assignment. these are designed to be self-contained: while references to concepts or definitions described in the textbook reading may appear, most formulas and terms specific to the questions in the lab are re-introduced as part of the write-up. the text alternates between providing 2 https://www.gradescope.com/ textual definitions and descriptions, coding instructions, and analysis questions. as the lab progresses, the questions grow more open-ended, culminating in prompts to make sense of results and what they signal about how well a model fits the text. ## observations in many ways, this format has helped us to feel liberated in how we approach topics later in the semester. in the most recent offering of the course, for instance, the latter half focuses on student projects and presentations on student-determined topics covering a range of popular tasks like machine translation and question answering and more wide-ranging topics like text-to-speech systems and computational social science. in other semesters, the course has continued with structured weekly (or bi-weekly) lab assignments, often culminating in a lab assignment that introduces a shared task that students then work on as a final project. overall, we find that the foundational skills introduced in the first four weeks prepare students well for all of these paths through the rest of the course. this approach admittedly releases the opportunity to do a deeper technical dive with the class into modern models; while students tend to discuss lstms, bert, and other popular modern models, they are not expected to fully present these topics in their presentations. while we had initially worried about whether this would provide too little infrastructure to make sense of neural models, we have been happy with how students use the skills we emphasized earlier to uncover known controversies in the field of nlp. we have witnessed advanced discussions initiated by the students about topics such as how strong baselines can be in questionanswering datasets, the possible shortcomings of metrics such as bleu, rouge, and pyramid, and what it would mean to reach human parity for machine translation. taking a discovery approach is not without risks, especially when students may get stuck or confused. we have learned some lessons about where more guidance may be needed for students. first, unsurprisingly, file i/o often can be remarkably picky, and we have found that the i/o portions of these labs can be fairly brittle when students make choices that may have been reasonable in previous classes with fairly rudimentary file processing. further illustration around ways to load and save data, manipulate numpy and spacy objects, and otherwise use libraries can greatly reduce time students spend stuck on smaller bugs instead of nlp ques-tions. additionally, students often ask a number of questions about how much analysis is sufficient and which aspects to focus on, often not finding engagement with the text intuitive. sample write-ups from the first lab or two may help illustrate this, as well as more development on the prompts for the write-up to elicit thoughtful analyses. ## conclusions we hope that the description of this class format and assignment progression helps motivate the feasibility of a discovery-based approach to teaching natural language processing. we are delighted to share the initial assignment instructions, lab files, and student gradescope image publicly on github at https://github.com/ discovernlp , with additional files for autograding through gradescope available on request.