Flamingos and Hedgehogs in the Croquet-Ground: Teaching Evaluation of NLP Systems for Undergraduate Students (2021)

Abstract: This report describes the course Evaluation of NLP Systems, taught for Computational Linguistics undergraduate students during the winter semester 20/21 at the University of Potsdam, Germany. It was a discussion-based seminar that covered different aspects of evaluation in NLP, namely paradigms, common procedures, data annotation, metrics and measurements, statistical significance testing, best practices and common approaches in specific NLP tasks and applications.

## course content and format i started the online meetings with a wrap-up and feedback about the previous week's content. then, i randomly split them into groups of 3 or 4 participants in breakout sessions so that they could discuss a worksheet together for about 45 minutes. i encouraged them to use this occasion to profit from the interaction and brainstorming with their 2 https://briemadu.github.io/evalnlp/schedule peers and exchange arguments and thoughts. after the meeting, they had one week to write down their solutions individually and submit it. in the second half of the semester, they divided into 4 groups to analyze how evaluation is being done in specific nlp tasks. for larger groups, other nlp tasks can be added. they prepared group presentations and discussion topics according to general guidelines and an initial bibliography that they could expand. students provided anonymous feedback about each other's presentations for me and i then shared it with the presenters, to have the chance to filter abusive or offensive comments. the last lecture was a tutorial about useful metrics available in scikit-learn and nltk python libraries using jupyter notebook @xcite . finally, they had six weeks to work on a final project. students could select one of the following three options: i) a critical essay on the development and current state of evaluation in nlp, discussing the positive and negative aspects and where to go from here; ii) a hands-on detailed evaluation of an nlp system of their choice, which could be, for example, an algorithm they implemented for another course; or iii) a summary of the course in the format of a small newspaper. ## participants seventeen bachelor students of computational linguistics attended the course. at the university of potsdam, this seminar falls into the category of a module called methods of computational linguistics, which is intended for students in the 5 th semester of their bachelor course. still, one student in the 3 rd and many students in higher semesters also took part. by the 5 th semester, students are expected to have completed introductory courses on linguistics (phonetic and phonology, syntax, morphology, semantics and psycho-and neurolinguistics), computational linguistics techniques, computer science and programming (finite state automata, advanced python and other courses of their choice), introduction to statistics and empirical methods and foundations of mathematics and logic, as well as varying seminars related to computational linguistics. although there were no formal requirements for taking this course, students should preferably be familiar some common tasks and practices in nlp and the basics of statistics. ## outcomes i believe this course successfully introduced students to several fundamental principles of evaluation in nlp. the quality of their submissions, especially the final project, was, in general, very high. by knowing how to properly manage flamingos and hedgehogs, they will hopefully be spared the sentence "off with their head!" as they continue their careers in nlp. the game is not very difficult when one learns the rules. students gave very positive feedback at the end of the semester about the content, the literature and the format. they particularly enjoyed the opportunity to discuss with each other, saying it was good to exchange what they recalled from the reading. they also stated that what they learned contributed to their understanding in other courses and improved their ability to document and evaluate models they implement. the course was also useful for them to start reading more scientific literature. in terms of improvements, they mentioned that the weekly workload could be reduced. they also reported that the reading for the week when we covered statistical significance testing was too advanced. still, they could do the worksheet since it did not dive deep into the theory. the syllabus, slides and suggested readings are available on the course's website. 3 the references here list the papers and books used to put together the course and has no ambition of being exhaustive. in case this course is replicated, the references should be updated with the most recent papers. i can share the worksheets and guidelines for the group presentation and the project upon request. feedback from readers is very welcome.