Semantic Parsing and Sense Tagging the P rinceton W ord N et Gloss Corpus (2023)

Abstract: In 2008, the Princeton team released the last version of the “Princeton Annotated Gloss Corpus”. In this corpus. The word forms from the definitions and examples (glosses) of Princeton WordNet are manually linked to the context-appropriate sense in WordNet. However, the annotation was not complete, and the dataset was never officially released as part of WordNet 3.0, remaining as one of the standoff files available for download. Eleven years later, in 2019, one of the authors of this paper restarted the project aiming to complete the sense annotation of the approximately 200 thousand word forms not yet annotated. Here, we provide additional motivations to complete this dataset and report the progress in the work and evaluations. Intending to provide an extra level of consistency in the sense annotation and a deep semantic representation of the definitions and examples promoting WordNet from a lexical resource to a lightweight ontology, we now employ the English Resource Grammar (ERG), a broad-coverage HPSG grammar of English to parse the sentences and project the sense annotations from the surface words to the ERG predicates. We also report some initial steps on upgrading the corpus to WordNet 3.1 to facilitate mapping the data to other lexical resources.

## introduction in the "princeton annotated gloss corpus" (glosstag), the content word forms in the definitions and examples in word-net foot_0 @xcite are manually linked to the context-appropriate sense in word-net itself. thus, the glosses are a sensedisambiguated corpus, and wordnet is the dictionary against which the corpus was annotated. the corpus is available for download on the princeton wordnet website as a standoff package supplementing the word-net 3.0 release. although it has already been recognized as a precious resource, not all words have been annotated yet. according to the statistics in the website, foot_1 by 2008, the corpus contains 206,711 words (including collocations and multi-word expressions) yet to be disambiguated. in @xcite , one of the authors reported initial efforts to complete the annotation of the corpus (release 2019). this paper reports the progress on this endeavor (release 2022) with improvements in the methodology and evaluation. the glosses were introduced in @xcite ; before them, senses were distinguished only by synonyms and semantic relations. from this same reference,"as the number of words in wordnet increased, it became increasingly difficult for us, purely on the basis of synonyms, to keep all the different word senses distinct." on the other hand, introducing glosses has its problems. first, it is not always easy to write good definitions, and second, glosses introduce information redundancy. "in the course of incorporating this kind of explanatory information, we have all acquired greater respect for traditional lexicographers." "the (somewhat idealistic) hope was that the definition of any word could be inferred from its position in this network of semantic relations and that definitional glosses would be redundant @xcite if more distinguishing features could be indicated by pointers representing additional semantic relations, the glosses would become even more redundant. an imaginable test of the system would then be to write a computer program that would synthesize glosses from the information provided by the pointers." @xcite nevertheless, since its introduction, researchers have found many applications for wordnet glosses. @xcite @xcite propose disambiguating the content words in the glosses to increase the semantic connections among the words and to establish relations among them between different syntactic categories to support common-sense reasoning. in one example, they explain how the disambiguation of the words 'eat' and 'food' in the definitions of the adjective 'hungry,' the verb 'eat,' and the noun 'refrigerator' establish a semantic path between the concepts expressed by the words. thus, one can infer from 'being hungry' the action of 'going to the refrigerator.' increasing the semantic connections among wordnet synsets also improves the results of many word sense disambiguation (wsd) algorithms that use the network structure of wordnet to identify the most plausible sense for the words in a context @xcite @xcite . by disambiguating the words in the glosses, we add pointers between synsets a and b whenever we annotated a word with a sense from a in b's definition. with that approach, we can increase the connectivity between the word-net synsets by approximately an order of magnitude. the disambiguation of words in the glosses can also improve wordnet and provide completeness and consistency. for instance, the initial versions of wordnet do not contain relations that indicate how words like 'racquet', 'ball', and 'net', and the concepts behind them, are part of another concept that can be expressed by 'court game' @xcite . in wordnet 3.0 the 'domain relations' between synsets were introduced to alleviate this so-called 'tennis problem' of wordnet @xcite , but the disambiguated gloss of the synset {tennis, lawn_tennis} (1) would already enrich the connections among the concepts. another desired property is that all words used in the definitions are defined in this same resource. hopefully, this completeness could also help us ensure quality in our longterm endeavor during the expansion of word-net to highly technical domains. once more concepts are added or redefined, the glosses would be refined and disambiguated, forcing us to use the newly added senses in a productive cycle of editing, testing, and correcting. in our approach, we aim at high-quality human annotation, leveraging the lessons learned and directives developed for the project in princeton but adapting them to our tools. data is available using the same open license used by princeton for the initial version of the data @xcite . in @xcite , we reported the initial steps of data preparation, our annotation interface's implementation, and a preliminary experiment on the inter-annotator agreement. the result was called glosstag 2019 (the 2019 release of the corpus). here we present the glosstag 2022. in section 2, we discuss some inconsistencies in the data identified during the annotation; mainly, we ensure that glosstag 2022 sentences are all effectively derived from the wordnet 3.0 glosses. in section 3, we explain why we employed the english resource grammar (erg) to parse the sentences into syntactic and semantic structures, hopefully more consistent than previous pre-processing annotations in glosstag 2008 and manually constructed logical forms @xcite @xcite . we also explain the projection of the sense annotations from the surface words to the erg predicates. this helped us improve consistency and facilitate sense annotation, mainly of verbs, where senses are normally related to specific syntactic valence alternations. in section 4 we evaluated the ukb word-sense disambiguation algorithm @xcite in the annotation of the glosstag itself. the idea was to test the feasibility of having an algorithm give hints to the sense annotator and produce intermediary 'silver' versions of the data in future releases. in section 5 we make initial considerations about the challenge to migrate the glosstag 2022 annotations to wordnet 3.1. we presented some final considerations and future work in section 6. ## data validation and preparation as reported in @xcite , from the glosstag 2008 xml files, we built the glosstag 2019 json-lines files with one json object per line for each gloss. the transformation was done mainly to facilitate the data ingestion in the backend of our annotation tool with elementary validations. in glosstag 2019, we focused on the annotation job and assessing its complexity only. nevertheless, the work over the last three years reveals inconsistencies, and data need to be prepared to be combined with their semantic representation obtained from the english resource grammar. in @xcite , the au-thors briefly mentioned the sense tagging of the wordnet glosses. the description is not detailed enough to conclude if they are describing the glosstag 2008. the best documentation about the dataset is provided as comments on a dtd file that specifies the wordnet document type of the xml files in the 'merged' folder of the dataset. 4 every 'synset' element in the dtd contains three 'gloss' child elements. one with the attribute 'orig', is a string that allegedly matches the gloss of this same synset in the wordnet 3.0 db files. the second, with the attribute 'text', is also a string, with extra spaces indicating the tokenization and quotes encoded in utf-8. the third, with the attribute 'wsd', is the one that holds the actual annotations of words and collocations in child elements. the glosses in wordnet 3.0 can be preceded by a domain classification fragment and/or an auxiliary fragment, both usually in parenthesis and optionally followed by more auxiliary fragments and zero or more examples. for sense tagging purposes, the original annotators ignored the classification fragments, as the information is normally repeated in the usage, region, and category pointers. the auxiliary fragments are always secondary to the primary sense of the synset; they can precede or follows the definition but can also be embedded within the definition. auxiliary fragments are tagged with 'ignore' or 'arg.' those assigned with the tag 'ignore' are ignored for sense tagging and contain mainly grammatical or usage information, some qualifying text such as a year born, time, date range, or a chemical or other symbols. in (2) we show the gloss of the synset {wash} (verb) with a fragment assigned with tag 'arg', the argument or typical argument (in green), for the preceding verb (in blue). they are set off in this way so that the syntax of the definition fits that of the lemma (the defining verb is intransitive if the lemma is intransitive). inside the definitions and examples, we have the 'wf' (word form), 'cf' ('wf' that are part of one or more collocations), and 'mwf' elements. the 'wf' and 'cf' are marked up with parts of speech and potential lemma forms at wordnet 3.0. the collocations are marked in a way that can even indicate discontiguous forms. the 'wf' can also be annotated with some semantic classes: punctuation, year, chemical name, number, time, currency, abbreviations, or mathematical symbol. the 'mwf' are multi-word forms composed by 'wf' and 'cf' children that can also be annotated with semantic classes: date, date/numeric range, numeric form, currency, measurement, mathematical formula, and other groups of symbols. the 'wf' and 'cf' that have been disambiguated are further annotated with wordnet sense keys and the flag indicating if the annotation was done automatically or manually. furthermore, 'wf' and 'cf' elements may contain a separator attribute with the character separating the corresponding form from the next in print. valid values for this attribute are hyphen, empty string, and space for hyphenated words not in wordnet, contractions that get split (in 'cf' forms), and cases where no space follows the form. the default value is a space, not explicitly assigned. we should be able to reconstruct the original text of the glosses in the wordnet db files using the separators, but this was not true for approximately 2100 glosses; we found and fixed some mismatches also caused by extra semi-colons added in the end of the examples. we know almost nothing about how the original text of the glosses was processed to produce all these mark ups in the glosstag 2008. what are the tokenization criteria? how were the semantic classes identified? how the definitions and examples segments were identified in a given gloss text. moreover, some essential details are provided in the dtd. the part-of-speech (pos) tags were automatically assigned only to word forms in the definitions, not in the examples, ap-parently because examples were supposed to be partially annotated; according to another comment, that says 'only synset terms in examples should be sense tagged,' but we have a more ambitious goal. once the tokenization issues were solved, and data was confirmed to correspond to the original wordnet 3.0 db files, we split the glosses into sentences (definitions and examples). approximately 758 examples in wordnet 3.0 are quotes such as (3), that is, quotes followed by the author's name or source. we removed the quote marks and moved the author's name (or title of the publication) to the metadata associated with the example. finally, we calculated the text span of each word form (also known in the literature as token ranges) in the sentences. as we will see in the rest of the paper, once we parse the sentences with erg to produce the semantic representations, we need to match the predicates obtained from the erg analysis with the word forms in the glosstag 2022 using the text spans. that is the reason for such careful considerations about tokenization. the missing pos in the examples were obtained from erg analysis. ## parsing with english resource grammar the english resource grammar (erg) @xcite is a broad-coverage, general-purpose computational grammar that, combined with specialized tools, can map running english text to highly normalized logical-form representations of meaning. erg is a linguistically precise hpsgbased grammar of english and semantically grounded in minimal recursion semantics (mrs) @xcite , which is a form of flat semantic representation capable of supporting underspecification. the erg is developed as part of the international deep linguistic processing with hpsg initiative (delph-in) 5 and can be executed by some parsing and realization systems, including the lkb grammar engineering environment @xcite , as well as the more efficient ace parser, 6 for applications. we grouped the glosstag 2022 sentences in profiles, test suites, collections of test items for judging the performance of an implemented grammar within delph-in. while the original purpose of test suites is to aid in grammar development, they are more generally useful for batch processing. after creating the profiles with 2000 sentences each, we processed them with the ace parser in a cluster, running each profile in parallel. it took about 30 minutes. for each sentence, we asked for the top-best analysis of erg. glosstag 2022 contains 165,976 sentences; from these, only 5,282 were not parsed by erg. using some heuristics (the most productive one is adding an extra 'x' in sentences ending with the preposition 'of', e.g. "get the votes of x "), we were able to parse roughly 600 more sentences (only 2% are not parsed). since erg is a computational grammar and sentences are typically ambiguous, we can have hundreds or thousands of readings for each sentence. we stored only the top-best analysis according to the pretrained parsing ranking model distributed 5 https://github.com/delph-in/docs/wiki/ 6 https://github.com/delph-in/docs/wiki/ acetop with erg. 7 this is not to say that all analyses were the expected ones, but informal evaluation gives us some great expectations. in a future experiment, we aim to employ fftb @xcite for gradually treebanking all sentences. fftb allows the selection of an arbitrary tree from the 'full forest' without enumerating/unpacking all analyses in the parsing stage. the treebanking of all sentences would ensure the data's quality and the actual evaluation of the parsing selection model. we aim to turn glosstag 2022 into a dynamically annotated treebank @xcite . for each item (sentence) in a profile, once it was processed, we have the derivation tree and the semantic representation mrs. 8 figure 1 presents one mrs. predicate-argument structure is expressed in a bag of n-ary elementary predications (ep) linked together by typed variables. 9 the predicate symbols can be divided into surface predicates and abstract predicates. surface predicates follow a naming convention where the symbol is composed of three components, called 'lemma', 'pos' (mostly align with a crude inventory of word classes (n)oun, (q)uantifier, (v)erb and (a)djective, etc), and 'sense' (coarse-grained senses, erg only marks those sense distinctions that are morphosyntactically marked). surface predicates, by convention, are marked by a leading underscore and are exclusively introduced by lexical entries from the grammar, whose orthography is a (possibly inflected) form of the lemma field in the predicate. the predicate _ palmately/rb _ u _ unknown is a generic predicate instantiated by erg for dealing with the unknown word. 10 the numbers following the predicate name indicate the text span to which the ep corresponds. 11 7 we are ignoring details about all other parameters that control the ace parser. note that the glosstag 2008 was already used by many wsd approaches @xcite . it has been used as a dataset for training supervised wsd algorithms. 12 and also to increase the connectivity among synsets by @xcite . in this section, we used ukb @xcite , a graph-based approach for wsd. it applies random walks, e.g., personalized pagerank, on the knowledge base (kb) graph to rank the vertices according to the given context. ukb has been shown to 12 replacing the well-known but controversial sem-cor (semantic concordance), a subset of the brown corpus @xcite and other small corpora used in the previous semeval tasks. figure 2 presents the glosstag information of the same definition processed by erg and presented in figure 1 in a tabular format. to produce the ukb input (figure 3 ), we have to consolidate the information obtained from erg with the glosstag annotations, which is not easy. mwe must be combined into a single token, and all tokens must have lemma and pos so that ukb can disambiguate them. but in figure 2 , tokens 9-10 are not marked as 'cf' (mwe). token 11 was tagged as an adjective but manually disambiguated and analyzed by erg as a verb. on the other hand, the mwe 'leaf shape' (tokens 3-5) matches with the erg analysis that identified the expression with the 'compound' predicate. 13 13 we are skipping details related to obtaining the lemmas rather and leaf _ shape from the predicates figure 2: glosstag 2022 tabular presentation of a sentence. the lines starting with hash contains sentence metadata. each word is presented in a line, column 1 is the identifier, column 2 is the word type, column 3 the annotation flag, column 4 the text span, column 5 the part-of-speech tag (when available), column 6 the form in the sentence, column 7 the possible wordnet 3.0 lemmas and column 8 the sense, when annotated. figure 3 presents the ukb inputs for the sentence from figures 1 and 2 . two consecutive lines represent each context. the first line contains the context identifier, whereas the second one contains the words to be disambiguated. each element in a context has four mandatory fields; lemma and pos are the most important ones. ukb then disambiguates all the words from the input in a single run. ukb can deal with partially disambiguated contexts and use the provided concept identifiers (synset identifiers) to give extra information in the disambiguation of the remaining tokens. given that, we generated two contexts for each sentence. in the first context in figure 3 , we included one extra token, the synset identifier. for evaluation, we only considered the words in the glosstag which are associated with at least one sense. 14 we have looked for a match by checking if ukb generated sense is a subset of the senses provided in the annotations. the total number of words disambiguated and considered for evaluation using ukb was 819,533. among them, the ones with senses that were also disambiguated by ukb sum up to 442,782. the majority of the ukb errors involved words that are highly polysemic. for example, the verb 'make' has 52 senses in wordnet 3.0. synset 00891038-v has the definition "assure the success of" and example "a good review by this critic will make your play!". ukb does not annotate the correct sense of 'make' in the example, even in the context where the synset identifier itself is added as an extra fake word. finally, we have definitions such as "of or relating to taxonomy" (03018498-a) with only two content words, not enough information to ukb. after some error additional analysis, we have found some necessary improvements for further evaluation. ukb did not find many lemmas in wordnet 3.0 because they were not lower-cased properly. many cases of mwe were not annotated in the glosstag nor detected correctly by erg also need to be fixed. the mapping of erg com15 and glosstag globs needs improvements. nevertheless, we can safely conclude that adding the synset identifier as an additional word in the context helps ukb. it seems to justify the use of ukb to automatically annotate missing senses and thus generate a 'silver' release of glosstag 2022. ## conclusion in this paper, we presented the glosstag 2022 release. the project is hosted in the https://github.com/own-pt/glosstag repository, and it will be updated in the following days. as put by @xcite , the semantic annotation of corpora helps improve both the coverage and the precision of the semantic resource being used in the annotation. this work is thus part of our effort in expanding and improving wordnet-like resources in an application-driven and domain-specific way. besides continuing the manual annotation, we plan to improve the annotation interface 16 and experiment with alternative wsd methods @xcite . concerning the annotation tool, we intend to improve its performance and make it a wordnet editor, allowing the sense annotation to influence wordnet improvements. we also aim for a workflow with feedback between annotation and erg analysis, one supporting the other. additionally, we also intend to develop querying and visualization tools. finally, we need to finish the migration to wordnet 3.1 before forking it from the princeton official release (or further mapping to ( mccrae et al., 2020) ) for changes driven by the annotation.