Gaining Experience with Structured Data: Using the Resources of Dialog State Tracking Challenge 2 (2021)

Abstract: This paper describes a class project for a recently introduced undergraduate NLP course that gives computer science students the opportunity to explore the data of Dialog State Tracking Challenge 2 (DSTC 2). Student background, curriculum choices, and project details are discussed. The paper concludes with some instructor advice and final reflections.

## introduction one of the consequences of the data explosion of the past twenty-five years is the likelihood that a large number of computer scientists and computing professionals will have the opportunity to develop analytical tools for processing large, structured datasets during their careers. this is of course especially true in the domain of nlp. a variety of nlp application domains can serve as an opportunity to provide undergraduate students with a chance to gain experience with the processing of structured data in order to solve an interesting "real world" problem. this paper describes a class project for an undergraduate nlp course that gives students the opportunity to explore the data of dialog state tracking challenge 2 (dstc 2). student background, curriculum choices, and project details are discussed. the paper concludes with some advice for instructors who might be interested in incorporating a dstc 2 based project into their course and final reflections about student feedback. ## background at east carolina university, a natural language processing course was added to our curriculum during academic year 2015-2016. it is a junior/senior level course for which the prerequisites are data structures and introductory statistics. significant factors that influenced the decisions about curriculum and pedagogy during the initial offerings of the course (spring semesters 2017 through 2019) include the following. based on these factors, natural language processing with python @xcite was chosen as the textbook for the course. it had been the basis for a successful independent study course with a duke university undergraduate during spring 2013. notable advantages of this choice include the following. a main disadvantage of this text is its limited discussion of linguistic theory, but that can be addressed by other assigned readings. with the cooperation of the department chair, the class size was restricted ( 15 in 2017, 25 in 2018, and 29 in 2019 ). this enabled the opportunity to provide more individualized learning opportunities. the primary such opportunity was the class project using the dstc 2 data that was assigned for the spring 2018 and spring 2019 offerings of the course. discussion of this project will be the focus of the remainder of the paper. ## dstc 2 overview for dstc 2 a general discussion of the challenge and challenge results are provided in @xcite . the ground rules for the challenge are specified in @xcite . a summary of the details most relevant to its use for a class project is provided below. ## possible activities with the dstc 2 data for a group of students with sufficient technical background with python and/or machine learning, a class project that allows students to develop their own dialog state tracking system is certainly feasible. students could start from scratch and enhance one of the rule-based baseline trackers that are provided in the dstc 2 dataset or else develop or enhance a machine learning approach for tracking (e.g. @xcite , @xcite )) foot_1 . in either case students should base their approach on a data-driven analysis of the nature of the dialogs. however, this is not the only possible use of the data. in a circumstance where students do not have the necessary background to develop their own tracker, they can still engage with the data by developing their own analysis tools to glean information useful for studying other aspects of dialog processing such as miscommunication handling. given the instructor's personal research interests and the students' background, this seemed to be the best way to proceed with a project as described next. ## in-class project activities about midway through the semester after completing the first five chapters of natural language processing with python, a week of class is taken to introduce students to dialog state tracking and the project. the goals for that week of class include the following. there are a few other in-class activities that relate more specifically to the project requirements. they will be mentioned in the following section that discusses the project requirements. ## project requirements students are required to submit a separate python program for carrying out each of the following tasks. more detailed discussion about each of these activities will follow. ## classroom reinforcement of their research efforts given the challenging nature of this project to our students, it would have been unwise to have spent the final weeks of the semester with excessive presentation of new material. to reinforce the goal of student exposure to the process of research and to connect their work to the research community, two 50 minute class periods were used looking at four papers from the 2014 sigdial conference where the work from dstc 2 was presented. class time was spent watching the videos of the original conference presentations of these papers @xcite , @xcite , @xcite , and @xcite ). these are available at https://www.superlectures. com/sigdial2014/ . this classroom activity was conducted two weeks prior to the end of the semester. the final week was spent being available for additional consultation about projects as well as spending class time on answering questions about nlp that were posed by students at the beginning of the semester. this was a chance to help them see what they had learned during the semester, and to understand better what remains an unsolved problem. ## advice for instructors besides in an introductory undergraduate nlp class for computer science majors with limited background in machine learning and python, i believe a project based on the dstc 2 data can be used in a variety of contexts. if interested in using the dstc 2 data for a class project, i would suggest the following steps. once you've successfully completed these tasks, you should be well on your way to developing your own project with this data. feel free to consult the author of this paper as needed. ## lessons learned student feedback was largely anecdotal and quite favorable. while it is certainly the case that in response to the "what would you change to improve the course?", the most common answer was to have more time spent on classifiers/machine learning, students did enjoy working on the project. the most interesting piece of feedback that was acquired was from a student who originally submitted an incomplete project during the first offering of the assignment (spring 2018). the student was allowed an extra couple of weeks to complete the project. at some point during fall semester 2018 the student took the time to communicate with me to let me know that the student had used the project as part of a job interview when asked to give a technical presentation about a previous project the student had completed. the student said, ". . . the final project in your nlp class was by far the most interesting one that i've been assigned in college . . . ". while this was gratifying feedback, the more important outcome is the belief that the course and project did achieve the goals of giving students experience with self-directed learning and engagement with the tools of research. we are fortunate in nlp to have a wide variety of interesting problems available to us that will naturally intrigue our students regardless of their original interest in nlp. if one wishes to use dialog processing as the interesting problem, the dstc 2 data is a valuable resource for use in the classroom. regardless, i would encourage instructors to pick a task that excites them. our teaching is much stronger when we are demonstrating the need for self-directed learning ourselves. our research provides such a means. ## acknowledgements i would like to thank my computer science department chair, dr. venkat gudivada, for allowing me to teach the initial offerings of the course with a restricted enrollment. this enabled me to offer more individualized exploratory research opportunities with our undergraduates. none of this would have been possible without the existence of the dstc 2 challenge. a special thanks goes to the dstc 2 organizers, matthew henderson, blaise thomson, and jason williams as well as to the university of cambridge, microsoft research, and sigdial (special interest group on discourse and dialogue) for their sponsorship and support of the challenge. a final thanks goes to the anonymous reviewers who helped me better understand what would be of most value to the readers of this paper.