Teaching Through Tagging — Interactive Lexical Semantics (2021)

Abstract: In this paper we discuss an ongoing effort to enrich students’ learning by involving them in sense tagging. The main goal is to lead students to discover how we can represent meaning and where the limits of our current theories lie. A subsidiary goal is to create sense tagged corpora and an accompanying linked lexicon (in our case wordnets). We present the results of tagging several texts and suggest some ways in which the tagging process could be improved. Two authors of this paper present their own experience as students. Overall, students reported that they found the tagging an enriching experience. The annotated corpora and changes to the wordnet are made available through the NTU multilingual corpus and associated wordnets (NTU-MC).

## introduction this paper introduces a method of incorporating lexical semantic research into the teaching of semantics, as a form of experiential learning @xcite . the main goal is to lead students to discover how we can represent meaning and where the limits of our current theories lie. a subsidiary goal is to create sense tagged corpora and an accompanying linked lexicon (in our case wordnets). the first author (francis) teaches hg2002: semantics and pragmatics, a core course in linguistics with 70-100 students. the course is survey-oriented @xcite , summarising various theories without dwelling on any overarching theme. it is easy for students to become bewildered by the variety of concepts, particularly in the absence of concrete applications. to alleviate this, from 2011, each semester a text was introduced, which students would try to analyse using the various approaches. the decision was also influenced by our university's encouraging stance towards involving students in research. the ntu computational linguistics lab is heavily involved in lexical semantics and wordnets, including building and extending wordnets and sense tagged corpora for multiple languages. we thus integrated some tagging into the course as a way to give students hands-on experience with a semantics-oriented research project. this course later formed the base for a general elective (ge), which is offered to any student in the university. 1 this was an interdisciplinary course, developed and co-taught with colleagues from the english and chinese departments. to make it more appealing, we focused on sherlock holmes -hg8011: detecting meaning with sherlock holmes. roughly half the course deals with interpreting the texts using semantics, a quarter with placing the stories in their literary context, including a discussion of fan-fiction, and the rest with sherlock in film and in translation. the course has proved popular, with over 200 students every time it is offered, and long wait lists. the pedagogical goals for both courses were fourfold: p-1 apply semantic theories to real-world texts p-2 show students the difficulties of defining and identifying senses. for example they need to look at more than prototypical cases; identify gaps in the lexicons and add new entries; and consider the problems of tokenization and mwes. p-3 expose them to annotation and resource building (common sources of employment for humanities students) p-4 teach the students about inter-annotator agreement there were four main research goals: r-1 produce sense tagged corpora, with all concepts disambiguated, in multiple languages r-2 experiment with how to sense tag: what is the best interface? what information do annotators need? one project that is very similar to our annotation in spirit is the georgetown university multilayer corpus (gum: @xcite . gum is collected and expanded by students as part of the curriculum in ling-367 computational corpus linguistics at georgetown university. the course has around 20 students, mainly postgraduate. the students are more computational than in our courses, so there is more emphasis on using external tools to annotate. the corpus selection is opportunistic, aiming to represent different communicative purposes, while coming from sources that are readily and openly available (mostly creative commons licenses). the results of this project show that high quality, richly annotated resources can be created effectively as part of a linguistics curriculum. the main difference is that the course at georgetown is specifically about corpora, while for my courses, the corpora are not the main focus. wordnets have also been used widely in teaching computational linguistics @xcite @xcite , but as far as we know this is the first time they have been a core part of a general linguistics course. the paper is structured as follows: in section 2 we describe the actual practice of annotation. in section 3 we present the student experience. in section 4 we look at what work is necessary to make the corpus ready for release. we finish with some conclusions and ideas for future work in section 5. ## annotating texts there is some overlap between the linguistics and ge course, but enough differences that we will describe them separately. our university teaches in english, and the majority of the students are na-tive speakers of english, although we have some international students (more in the ge class). many students are also fluent in another mother tongue (mainly mandarin chinese, standard malay and sometimes tamil) and many of the linguistics students have studied a second language to a level in which they can annotate meaning (with japanese and korean being the most popular). ## the student experience in this section we provide a summary of students' feedback. additionally, two students, one each from the linguistics and general elective classes talk about their experience, both as students and later as research assistants. foot_1 ## quality control and expert tagging given that the annotation that happens in our classrooms is done by untrained students from diverse backgrounds and often lacking linguistic intuition, it is not surprising that our corpus needs to go through multiple layers of quality control before being suitable for release. the large majority of this quality control is done by student ras. this usually happens in phases, and each phase (or ra) focuses on a particular task. these different tasks include: i) review comments left by students during their tagging exercise (e.g. references to possible metaphors, named entities, etc.); ii) review and fix the corpus where problems concerning lemmatization or corpus structure were flagged (i.e. e tags); iii) review and address reported gaps in the wordnet coverage (i.e. w tags); iv) ensure students made adequate use of the tag x (i.e. using it only for words that should not be tagged); and v) review and retag any mistakes in the student annotations. much of this work ends up rejecting the suggestions made by students, as they often identify real issues without finding the best solution, due to unfamiliarity with wordnets. to accomplish these tasks, student ras make use of a set of tools not usually available to other students, including the targeted tagging tools (introduced above); the corpus fixer which allows the annotator to change the tokenization, pos and lemmatization, as well as to add new multi word expressions; and omwedit which allows the annotator to add to or change the wordnets. @xcite . some of the non-intuitive aspects of these tools require some training before they can be used but, most importantly, require a deeper understanding of many layers of lexical analysis (e.g. pos tags, lemmatization, multi-word expressions, etc.). student ras without a computational background are often both baffled and amused with problems caused by pos and lemmatization issues (e.g. when words like graves are lemmatized as graf through a misapplication of the same rule that produces shelf from shelves), but are quick to grasp these more mechanical aspects of the quality control process. most of the other tasks involve more difficult problems, such as judging whether an expression is compositional or not, or whether a distinction in meaning is significant enough to warrant the creation of a new synset. wordnets are fairly complex, and our student ras learn about it on the job. the task of changing a wordnet feels quite daunting at first, and it only becomes easier once our ras get familiarized with the wordnet's structure. once the decision to create a new synset is made, other layers of complexity arise. our ras have to balance the coverage of new senses (i.e. how broad or narrow should the new synset be -taking into consideration other existing synsets). finding the appropriate semantic links between new and preexisting synsets is also not always straightforward. if the decision is to try to use an existing synset to accommodate a missing sense, then there are other issues to take into account. the main concern is the extent to which an existing synset can be edited to accommodate this alternative meaning. this often requires detailed lexicographic work, observing examples inside and outside our corpus to determine if the proposed changes are warranted by real data. many of the more difficult decisions are discussed within larger lab meetings, where multiple student ras and senior lab members join in. as it was discussed above, some of the problems encountered by our ras end up deserving a more in depth treatment or discussion, and are taken up by smaller focused teams within our lab, or as the topic of a project/dissertation. every time we teach one of the courses described above, a new set of data requiring quality control is created. from our experience, this amounts to roughly 3-4 weeks full-time work for a trained annotator for 600-700 sentences of text. this is often done taking into consideration the written reports submited by students (when available), which also gives our ras an insight into the common problems that faced student annotators. whenever possible, these insights are also used to improve the documentation made available to students during their annotation task -with the goal of making this documentation intuitive for students who may feel overwhelmed by the amount of information they need to absorb. this is not the most efficient way to annotate text, but a good result is obtained in the end, and we can involve many students. one problem we found was that as we refined the tokenization and wordnet guidelines, the corpora got out of sync. for example, when we added pronouns, we had to go back and tag them in the older corpora. more interestingly, we occasionally change our tokenization guides: long-legged we used to tokenize as long and -legged but now tokenize as long, -, -legged. we also need a new tag for the noun: nnd (noun inflected like a pas-participle), which we lemmatize to leg. we are currently working on further using the tagged corpora to find examples in this class; as a source text in corpus linguistics, and for the digital edition of the tagged stories. ## conclusion and future work we need more annotated text: linking text to analysis is an important task. we expect linking to lead to changes in the linked resource: it is important to support this. access to more data makes more interesting projects possible. students learn a lot by attempting real tasks, and enjoy working on interesting stories. we can take advantage of this to improve the quantity and quality of our wordnets and corpora. one of the goals of this paper is to encourage other similar courses around the world to integrate similar strategies to annotate more text. we have had success supporting colleagues at the university of pisa in order to tag an italian translation of the speckled band as art of a semantics course. we would like to like to coordinate with more lecturers in other countries to extend the task to other languages. this is also why we commit to open-source practices, and make both our data and our tools 4