Completing the P rinceton Annotated Gloss Corpus Project (2019)

Abstract: In the Princeton WordNet Gloss Corpus, the word forms from the definitions (“glosses”) in WordNet’s synsets are manually linked to the context-appropriate sense in the WordNet. The glosses then become a sense-disambiguated corpus annotated against WordNet version 3.0. The result is also called a semantic concordance, which can be seen as both a lexicon (WordNet extension) and an annotated corpus. In this work we motivate and present the initial steps to complete the annotation of all open-class words in this corpus. Finally, we introduce a freely-available annotation interface built as an Emacs extension, and evaluate a preliminary annotation effort.

## introduction the princeton wordnet gloss corpus is a corpus of the manually annotated synset definitions (glosses) from the princeton wordnet (pwn) @xcite . the corpus is available for download in the pwn website as one of the standoff packages that supplement the wordnet 3.0 release. 1 although it has been already recognized as a precious resource, the project of semantically tagging all pwn glosses was not finished. according to the pwn website, the corpus contains 206,711 words (including collocations) yet to be disambiguated. in simple terms, our goal is to complete the disambiguation of all open-class words in this corpus, and here we present our preliminary findings and methodological decisions. previous efforts address this same goal in older versions of pwn using automatic or semi-automatic methods @xcite . here we 1 http://wordnetcode.princeton.edu/ glosstag.shtml aim at high-quality human annotation of the glosses, leveraging the lessons learned and directives developed for the project in princeton but adapting them to our tools and priorities. data is available at https://github.com/ own-pt/glosstag using the same open license used by princeton for the current version of the data. the definitional glosses were introduced in pwn primarily to help humans identify the meaning of the synsets, but recently, many word sense disambiguation (wsd) algorithms use the network structure of pwn in combination with the glosses to improve the identification of the most plausible sense for a given word in a corpus @xcite @xcite . by semantically disambiguating the words in the glosses, we add pointers from each word to its synset, and this increases the connectivity between the wordnet synsets by approximately an order of magnitude, hopefully improving the performance of these algorithms. another reason for such an effort is to ensure the completeness of pwn. by completeness we mean the property of a lexico-semantic resource that all words used in the definitions of the concepts are also themselves explained in this same resource. hopefully, this completeness could also help us ensure quality in our long-term endeavor, the expansion of pwn to highly technical domains such as those of the geosciences, agriculture, and law. once more concepts are added or redefined, we will redefine and add glosses that we intend to disambiguate, forcing us to use the newly added senses in a productive cycle of editing, testing, and correcting. we begin this paper by discussing the original dataset and how we interpreted it converting to a more friendly format. next we describe our annotation interface and some of our implementation decisions. we continue by discussing some of the issues we encountered while sense-tagging the glosses corpus. finally, we evaluate our ongoing annotation work, and discuss related work and conclude. ## sense tagging semantically tagging (or sense annotating) a corpus is a task of constructing a semantic concordance -a textual corpus and a lexicon so combined that every content word in the text is linked to its appropriate sense in the lexicon @xcite . two different strategies for building a semantic concordance are known: the sequential and the targeted approaches. @xcite presented one of the first tools developed for supporting the work on building a semantic concordance with pwn, the con-text. the tool was constructed to support sequential tagging. in this approach, the annotator starts with the corpus and proceeds through it word by word. this procedure has the advantage of immediately revealing deficiencies in the lexicon: missing words, missing senses, and indistinguishable definitions. the sequential process was chosen because of their priorities at that time, as they aimed to make substantial improvements in the pwn. another tool supporting the sequential approach to building semantic concordances was described by @xcite . the tool was introduced after a brief survey on other tools for sense tagging, none of them actively maintained and freely available at that time. in the targeted approach, the work starts with the lexicon: we focus on a polysemous word, extract all sentences from the corpus in which that word occurs, categorize the instances and write definitions for each missing sense, and create a pointer between each instance of the word and its appropriate sense in the lexicon; we then repeat the process by choosing another word to focus on. the targeted approach has the advantage of concentrating the annotation effort on a single word, producing better definitions. however, the previously listed flaws in the lexicon would not appear so straightforwardly in this targeted strategy. consequently, this strategy has the potential of being more successful when the lexicon has already reached a more stable stage. the targeted strategy was the one chosen for the wordnet gloss corpus initial phase; it is described in the original annotators' guidelines that we had access to, and we have decided to follow it as close as possible. the original wordnet gloss corpus project employed an interface called mantag, implemented in the perl programming language. 2 unfortunately, the tool has many dependencies on legacy code that we were not able to solve. for our continuation of the wordnet gloss corpus annotation project, we decided to implement a serverless application that can be used offline. this decision reflects the prevailing understanding that semantic annotation is a difficult task that is best done individually and in an environment conducive to concentration. in our tool, each annotator can perform their work independently, making annotations on overlapping parts of the corpus or not. the annotators' data can then be consolidated, possibly including discussions aiming at agreement in the cases where annotations diverge. we have also differed in our technology of choice compared to @xcite . instead of choosing a web framework we have decided to implement our annotation tool in the extensible and free text editor emacs, foot_1 taking advantage of the editor's support for multiple platforms and its rich ecosystem. the annotation interface needs no internet access, depends only on emacs and its libraries, and can be run either from a graphical interface or from a terminal window. the annotation interface works as follows: given the directory where the data files are stored, it indexes all tokens to be annotated by their lemmas. this index is persisted to disk so that this indexing does not need to be re-run. the user is then prompted for a lemma and (optionally) a pos tag; if any matching pairs of lemma and pos tag are found in the index, a new buffer is opened, containing the glosses where the targeted lemma was found. colours differentiate token's status: preannotated tokens are shown in one color, while tokens yet to be annotated are shown in another; tokens annotated in the current iteration are also shown in a different color. multiword expressions are marked by subscripts in their constituent tokens (whether they are adjacent or not), while sense and pos annotations are shown as superscripts. the annotation interface offers the user the following capabilities: the first capability is the main functionality of the tool. when selected, the user is asked to confirm the token's pos tag, and then a dialogue box is shown with all possible senses to that lemma and pos tag pair, along with their defined terms and glosses. the user can then select or deselect a sense, or explicitly say that there is no sense for that word in wordnet (see figure 1 ). all other commands are there to allow the correct sense annotation of a word. in case its lemma is wrong, there is no way of presenting the user with the correct sense options unless the lemma is corrected; if a token is part of a multiword expression but is not already marked as so, annotators are able to mark it themselves. the dismantling of a multiword expression is necessary for the cases where the token is wrongly assigned as part of a multiword expression, as rock and bass in example 2b, where both are marked as part of the multiword expression rock bass (a kind of fish). all commands are available through customizable mnemonic keyboard shortcuts or by a menu. ## data preparation the princeton wordnet gloss corpus is distributed in two different xml formats: standoff and merged files. 4 we choose to work with the merged files because they are more concise and are precisely described by a document type description (dtd). we have split this data into files containing 100 glosses each, with one annotated gloss per line encoded as an s-expression, a notation for tree-like data. every wordnet gloss contains a sense definition. the gloss can be preceded by a domain classification fragment and/or an auxiliary fragment (usually in parenthesis, but not always), and optionally followed by more auxiliary fragments and zero or more examples. in the original xml, all these components are marked up with nesting elements. the tokens are marked up with parts of speech, potential lemma forms, and (optionally) a small set of semantic classes (indicating whether the token is punctuation, abbreviation, acronym, number, year, currency, or some kind of symbol). collocations are delimited by special markup which can even indicate discontiguous forms. words and collocations that have been disambiguated are further annotated with word-net sense keys. to facilitate the implementation of the interface, we have adopted a flat data format where a gloss is a list of tokens, each one of them represented by a property list (see listing 1). all nesting elements for boundary-marking tokens in the xml files were converted to key-values pairs in the respective tokens. further details are publicly provided in a readme file along with the data itself. the data conversion is followed by a validation step to ensure that our understanding of the data was right and that no information was lost. although the xml validation using the dtd takes care of many validation issues, we did find encoding errors and nonexistent sense-keys in the corpus. for the encoding errors, before the conversion, we searched for and replaced invalid characters by utf-8 legal codes. most of the cases of invalid sense keys turned out to be instances of adjective satellites whose sense keys had been wrongly marked with synset type 3 instead of 5, but some cases were tokens marked with (:ofs "02744323" :pos "n" :keys (("arterial road%1:06:00::" . "arterial road")) :gloss "a major or main route" :tokens ((:kind :def :action :open) (:kind :wf :form "a" :lemma "a" :pos "dt" :tag "ignore") (:kind :wf :form "major" :pos "jj" :tag "man" :lemma "major%1|major%2|major%3" :senses (("major%3:00:06::" . "major"))) (:kind :wf :form "or" :lemma "or" :pos "cc" listing 1: property list encoding of wordnet 3.0 synset 02744323-n an undocumented and non-existent sense key purposefully ignored%0:00:00::. the name suggests that this was a virtual sense, created as a way of manually marking tokens as to be ignored in the sense annotation. a case like the annotation of 'ng' in example 1a foot_3 seems to support this view. however there is also evidence to believe that the non-existent sense key was created to mark cases where the appropriate sense for a word did not exist in wordnet, as in 'designating' in example 1b. we also found four cases where a 'purposefully ignored' sense was assigned together with some other sense; these we have revised and corrected manually. these cases include the aforementioned example 1a (where it had also been tagged as the unit nanogram), example 1c (where waves also had been tagged as "(physics) a movement up and down or back and forth"), and example 1d (where mediterranean was also tagged as "of or relating to or characteristic of or located near the mediterranean sea"). ## challenges the challenges of this project encompass many aspects: the amount of work, the particularities of the glosses compared to sentences in an ordinary text, and the mismatch between the 'continuous' sense boundaries of words in utterances and the 'discrete' boundaries defined by a lexicon. the princeton wordnet gloss corpus contains 117,659 glosses composed by definitions and examples, comprising more than 1,621,129 tokens. so far, 449,355 tokens have been annotated, 118,856 of them automatically. considering only the taggable tokens, i.e., the open-class words, 206,711 tokens are estimated to remain untagged. from these untagged tokens, we have so far annotated approximately 500 tokens during the development of our tool and training of the annotators. to deal with the amount of work in the next phases of this project, we plan to prioritize the annotation by focusing on domain-specific words most relevant to other projects of our team. the sense of a word in a text is determined by its context -the more context information we use, the easier is the determination of the right sense of its polysemous words. compared to other corpora, synset glosses provide relatively little context. figure 2 summarizes the sizes of glosses (number of characters) by part-of-speech. as we can see, the majority of the glosses has less than 100 characters (76% of them). moreover, most of the glosses are not complete sentences, e.g. 'secured with bastions or fortifications.' the annotator has to carefully consider the words that are being defined by the gloss and its relations to other synsets, in order to compensate for these obstacles. for some cases, such as 'allomorphs' in 'pertaining to allomorphs' and 'park' in 'the young man was caught soliciting in the park' the unique viable solution is to allow multiple sense annotation, as described in section 2. regardless of the nature of the sentences, disambiguation of senses is a notoriously hard task that may not be disconnected from the constant revision of the lexical resource being used as a sense inventory. this line of thought is supported by the way previous work on building semantic concordances was conducted. @xcite , carried out sense annotation while expanding and refining pwn, with the annotations continually signalling omissions and inaccuracies in the resource. we have already noted some cases of inconsistencies in pwn such as the case of 'deposit' presented in figure 3 . the dashed red lines point to the two possible senses for the word 'deposit' (bold) in the synset 01576001-v. note that although the synset 01528069-v seems to be the best option, as it is the direct hyperonym of 01576001-v. the synset 01575675-v is the best matching considering the example in 01576001-v and its hyponym 01988755-v. this situation suggests that 01576001-v should have a different position in the network. more recently, @xcite has already pointed out 'word senses are only ever relative to a set of interests' and @xcite emphasizes this point, remarking that dictionaries (or wordnets) and corpora are in two different levels: "dictionaries and wordnets are metalinguistic generalizations, while corpora are real texts; dictionaries and wordnets include decontextualized isolated items, corpora consist of contextualized continuous text." some cases of multiword expressions (mwe) seem to support our belief that sense annotation and pwn maintenance should be joint work. first, we need to define and enforce heuristics to determine when a given word sequence is a multiword expression (being sense annotated as a single entity), and when its component tokens should be annotated individually. the compositionality and conventionality criteria from @xcite may help, however these criteria are not as clear-cut as we would like them to be. take the case of 'first degree' and the example 'all of the terms in a linear equation are of the first degree' in its definition (synset 05861716-n); we can annotate it as 'first degree' (this same sense being defined in the synset where the example is given); but there is no sense for 'second degree', or 'third degree', which are equally valid. this leads us to consider that it should be annotated individually, and that the 'first degree' sense should be removed from pwn. one can conclude that sense tagging the pwn glosses is a never-ending task, but we believe it is possible (and useful) to achieve definitional completeness in restrictive domains. the question that we face is how to make it feasible and synchronized with the changes in the lexicon (senses, words, and relations). admittedly, we will need to implement tools for tracking the changes in the dictionary and signal for re-annotation all potentially affected glosses. finally, the challenges related to the corpus' size and to mwes also interact. to make the annotation process easier we would like to have a certain degree of automation. we have inherited from the original data expressions that have been incorrectly tagged as mwes, as in the bold words in the sentences 2a and 2b. while it is easy to recognize and fix this kind of error, the other way around is more challenging: identifying an expression that should be added or that is already defined in the lexicon. ## evaluation we have carried out a preliminary annotation effort to test our interface, train our annotators, and refine our guidelines. in this section we report the issues we found and the results we obtained. we have trained four annotators and instructed them to annotate all glosses in which one of these three words occurred: 'derivation' (9 occurrences in 8 glosses), 'formation' (153 occurrences in 146 glosses), and 'incompatible' (8 occurrences in 7 glosses). the word 'derivation' has eight senses available from the pwn, while 'formation' has seven senses and the adjective 'incompatible' has nine senses. these example words were chosen to balance frequency and polysemy degree. after the annotation, two sessions of discussion were conducted to consolidate annotation decisions. we must note that because training the annotators was the goal of this experiment, the results presented here are still very preliminary. considering all three words, only half of the occurrences presented full agreement among annotators. but partial agreements are reasonably common as we can see in the tables 1 and 2. the 'ctx' column (for context) numbers the occurrences of a given word; when the number is a decimal it means that there is more than one occurrence of that word in the same context. the other columns number the possible sense an annotator could choose, including a label for the absence of a suitable sense in pwn ('n'). particularly interesting is that all of the eight occurrences of 'incompatible' have almost always been annotated with the most generic sense "not compatible" (00508192-a). nevertheless, annotators reported this to be the hardest among the three words to annotate. even to reach a consensus on the proper sense afterwards was a hard task. when we consider the word 'formation,' there are 82 occurrences with full agreement ( the case of 'formation' is in agreement with results from @xcite that say "the degree of difficulty involved in resolving individual senses is a greater performance factor than the degree of polysemy.". it also suggests a two-step sequential approach to annotation: first the annotators agree on each synset's scope and only then do they proceed to the actual annotation process. this two-step approach will be the object of a future investigation. as for multiword expressions, expressions such as 'military formation', 'geological formation', 'reticular formation', and 'reaction formation' are removed from the above quantitative analysis but we have discussed them. the expression 'military formation' stands out in many glosses. the expression exists as a mwe but a similar expression, 'naval formation' does not, with both appearing in the gloss "the side of military or naval formation". annotators discussed whether 'naval formation' should be considered a mwe or whether 'military formation' should not be considered one. an annotator's familiarity with a particular domain also plays a role in the annotation process, affecting both the senses assigned and the decisions regarding which collocations should be considered mwes. for instance, the expression 'rock formation' is not part of pwn, but it appears many times in the corpus (see example 3a). although some of us believe the expression should be added to pwn, it is not in the lexicon yet, and so, three annotators chose the sense '(geology) the geological features of the earth' for the word 'formation' in all occurrences of the expression. this decision was understandable if we consider that the word 'rock', in one of its senses, naturally evokes the domain 'geology'. the same can also be said for the word 'continent' in example 3b. but one annotator, a geology expert, consistently took the sense 'a particular spatial arrangement' for the word 'formation' in this expression. his decision was based on the strict interpretation of 'geological formation' as a domain-specific concept also described in https://en.wikipedia.org/ wiki/geological_formation and reinforced by the fact that 'geological formation' in pwn has 'physical object' as its hyperonym, not 'formation' (as a process). another issue identified in this small experiment was that of an annotator consistency. in the definition of 'male bonding' and 'female bonding,' the word 'formation' appears in a very similar way ('the formation of a close personal relationship between men/women'), but one of the annotators was not consistent in the annotation of its sense in the two glosses. finally, tables 1 and 3 show that some annotators have already identified missing senses in pwn (column n). the recognition that pwn contains a substantial amount of knowledge within its glosses was made clear in @xcite . these articles describe the work on some of the standoff files distributed in the pwn website, foot_4 including the 'logical forms' of the glosses. the authors also mention the use of the 'logical forms' produced years before by another team at the university of texas. 7 in the extended wordnet @xcite , the disambiguation of the glosses was done automatically over the pwn 2.0 glosses. although their initial plan was to "develop a tool that takes as input the current or future versions of wordnet and automatically generates an extended wordnet that provides several important enhancements intended to remedy the present limitations of word-net", the project does not seem to be maintained anymore. in @xcite the authors reported that the logical forms generated at that stage are not of high quality in general. further use of the extended wordnet was reported in @xcite . the most relevant work to our present effort is the original princeton wordnet gloss corpus project, our starting point. 8 unfortunately, we are not aware of any publications resulting from the project except the readme file distributed with the data and the annotators' guidelines. 9 here we emphasize the manual process, focus on the creation of a semantic concordance of pwn glosses and pwn itself in the same lines of @xcite . we are also following as close as possible the directives devised by the princeton team when they started the original princeton wordnet gloss corpus. similar to @xcite , our primary goal is the development of better word-sense disambiguation methods and algorithms that can take advantage of the annotated glosses for better results on a domain-specific corpus, such as the one described in @xcite . ## conclusion in this paper we describe our resuming of the princeton wordnet glosstag corpus project. 10 we have assembled a team and created an annotation interface, and have begun our work. as put by @xcite , the semantic annotation of corpora helps improve both the coverage and the precision of the semantic resource being used in the annotation. this work is thus part of our effort in expanding and improving wordnet-like resources in an application-driven and domainspecific way, initially focusing on oil & gas domain applications. besides a continuous annotation effort, future work mostly involves improvements in the annotation interface 11 and in annotation methodologies. with respect to the annotation tool, we intend to start supporting the sequential annotation style discussed in section 2, and to improve its performance. the methodological work involves developing processes for the revision of syntactic annotation (part-of-speech tags, lemmas, and mwe tagging) and for updating the corpus when the underlying wordnet changes. additionally, we also intend to develop querying and visualization tools to support the annotation and the wordnet's expansion work.