Grammatical Role Embeddings for Enhancements of Relation Density in the P rinceton W ord N et (2018)

Abstract: In this paper we present an approach for training verb subatom embeddings. For each verb we learn several embeddings rather than only one. These embeddings include the verb itself as well as embeddings for each grammatical role of this verb. To give an example, for the verb ‘to give’ we learn four embeddings: one for the lemma ‘give’, one for the subject, one for the direct object and one for the indirect object. We have exploited these grammatical role embeddings in order to add new syntagmatic relations to WordNet. The evaluation of the new relations quality has been done extrinsically through the Knowledge-based Word Sense Disambiguation task.

## introduction in this paper we present an approach to extending the knowledge graph, based on princeton english wordnet (pwn) - @xcite -with syntagmatic relations. our aim is to improve the knowledge-based word sense disambiguation (kwsd). in several papers we showed that adding syntagmatic relations from syntactic and semantic annotated corpora improves the performance of kwsd - @xcite and @xcite . the main types of syntagmatic relations extracted from these corpora are the ones corresponding to the grammatical roles: verb-subject (has-subj), verb-direct object (has-dobj) and verb-indirect object (has-iobj). although we managed to extract good sets of new relations, the main problem is that corpora annotated with semantic and syntactic information contain only a fraction of all the possible syntagmatic relations. the inheritance over the hierarchies of pwn is problematic because the hierarchies of pwn are not monotonic. for that reason, in this paper we use feature learning in low dimensional vectors of real numbers known as embeddings. word embeddings play an important role in the new stream of natural language processing applications, providing latent features for lexical items. it is expected that the necessary features are encoded within the embedding space. for example, a verb embedding represents information for its valency frame elements. unfortunately, we can check this information only indirectly. in the paper we report embeddings on the subatom level foot_0 that make explicit some of the features related to the semantic selectional restrictions on grammatical roles of words in text. thus our goal is not to learn an embedding for a verb, but rather embeddings for the participants in the event (or state) denoted by that verb. such an explicit embedding of the valency frame elements has many potential applications. in this work we exploit these embeddings for adding new syntagmatic relations to pwn with the aim to improve applications such as kwsd. evaluation in the paper is performed by automatically extending word-net with ranked relations within the context of kwsd. we show that adding higher ranked relations improves the performance of kwsd. further we provide manual inspection and validation of the new relations that also supports the feasibility of our approach. our approach is similar to the approach of @xcite who started with the lexical function model where each functional lexical item is represented via n+1 tensor if it is an n-ary functor. in order to escape from using tensor with three and more dimensions they proposed a representation where for each argument a matrix is used. each matrix determines the incorporation of the corresponding argument semantics into the compositional semantics of the whole phrase. our method is also similar to the other popular methods for relation extraction. the main difference is that we do not implement relation embeddings, but rather a general embedding for one of the entities involved in the relation. also we work with relations that are not present in the knowledge source we extend -pwn in our case. in this way we hope that our method is applicable also to the underresourced languages. the structure of the paper is as follows: section 2 briefly discusses related work. in section 3 we present our motivation to extend wordnet with syntagmatic relations. section 4 outlines an example of subatom sentential semantics based on the ideas behind minimal recursion semantics. section 5 describes the mechanism for creating grammatical role embeddings. in section 6 the experiment setup is presented and the results are discussed. the last section concludes the paper. ## related work the success of kwsd approaches apparently depends on the quality of the knowledge graph -whether the knowledge represented in terms of nodes and relations (arcs) between them is sufficient for the algorithm to pick the correct senses of ambiguous words. several extensions of the knowledge graph constructed on the basis of wordnet have been proposed and implemented. with respect to the extension of wordnet with syntagmatic information there exist many works such as @xcite and @xcite . here we present in more detail only one approach similar to ours. it is described in @xcite and explores the extraction of syntactically supported semantic relations from manually annotated corpora. in this line of research semcor - @xcite , being a semantically annotated corpus, was processed with the minipar dependency parser and the subject-verb and object-verb relations were consequently extracted. the new relations were represented on several levels: as word-to-class and class-to-class relations. the extracted selectional relations were then added to wordnet and used in the wsd task. the main differences with the approach described here are as follows: we used a bigger set of relations (since it includes also indirectobject-to-verb relations). apart from that, the new relations reported in this paper are not added as selectional relations, but as semantic relations between the corresponding synsets. this means that the specific syntactic role of the participant is not taken into account, but only the connectedness between the participant and the event is registered in the knowledge graph. also, in our work we use embeddings as filters, instead of the selectional restrictions approach undertaken in @xcite . in the range of distributional semantics, the representation of word semantics for compositionality was suggested as n+1 dimension tensors for n-ary functor words. for example, an adjective is treated as a function over the modified noun. in order to implement this idea in practice, the semantics of the adjective is represented as a matrix which by multiplication with the vector representation of the noun produces the semantics of the noun phrases. thus, if we assume 300-size vectors for representation of nouns, the adjectives are represented as a 300 × 300 matrix. for transitive verbs the representation is a 300 × 300 × 300 tensor. this approach is called lexical function model by @xcite . however, it has been criticized because the number of parameters to be learned exponentially increases. in order to solve this problem @xcite proposed a representation of functor words as a vector of a vector for the semantics of the word itself and a matrix for each of its arguments -⟨ -→ a , 2 1 a, . . . , 2n a ⟩. each of the matrices corresponds to a function-argument relation, such as subject-verb, noun-modifying adjective, etc. the compositional semantics of a phrase is defined as sum of the vector for the semantic functor and the multiplication of the vectors for the arguments and the corresponding matrix. this approach is called practical lexical function (plf) model. @xcite demonstrate the feasibility of plf by testing it on several benchmarks that represent different aspects of sentence-level semantics composition. our main goal is to learn features for the prototypical grammatical roles. in principle, they might be constructed from the plf representation, but the derivation of the prototypical roles representation would require an additional mechanism of abstraction. we hope that our representation will facilitate the selectional restrictions of the corresponding predicates. this is not possible to be done directly by the plf model. one direction of future research is to combine both approaches. this could be done by learning argument matrices that work in combination with the argument vector and the grammatical role vector. our intuition is that such matrices could not be attached to a specific lexical unit, but to a class of lexical units. there is also a huge number of works on extending world knowledge oriented graphs with new relations (see @xcite , and @xcite among others). the main difference in our case is that we do not learn instances of the required relations from the corpora, but we learn semantic restrictions over the arguments of the relations. the candidate relations are generated from knowledge base itself (wordnet here). ## wordnet extensions with new relations as mentioned above, in our previous works we extended pwn with syntagmatic relations using semantically annotated corpora such as semcor. the idea was that if there is a subject-verb syntactic relation in the corpus, and the related verb and noun are manually annotated with synset ids from pwn, we could reliably assume that there is a semantic relation between the noun and the verb synsets in pwn. at a more general level we call this relation has-participant. it is directed from the verb to the noun synset. in order to draw a distinction between the different participants in an event (state), we use subrelations named after their grammatical roles: has-subj, has-dobj, and has-iobj. 2 adding a has-participant relation between two synsets in wordnet imposes two questions: (1) does this relation hold for more specific synsets? (2) does this relation generalize to more general synsets? in our previous research on extending wordnet with new relations from semantically and syntactically annotated corpora - @xcite and @xcite -we showed that using inference over the wordnet hierarchy adds new appropriate relations between verb synsets and noun synsets. especially with respect to the has-participant relation, we assume that the relation holds when the noun synset is substituted with a hyponymic synset and that it also holds when the verb synset is substituted with a hypernymic synset. we noticed that in many cases such an inheritance is not correct. for example, if we have "a doctor operates a patient", it does not entail that all doctors can operate. thus we cannot reliably substitute the synset for 'doctor' with each of its hyponymic synsets. it is also true that the verb synset allows many more participants than the instances in the corpus. for example, "a surgeon cures a patient" does not imply that only hyponymic synsets are appropriate to substitute 'surgeon'. thus, although the extraction from syntactically and semantically annotated corpora is a reliable method for adding syntagmatic relations to wordnet, their generalization to all possible syntagmatic relations is problematic. another problem is that such manually annotated corpora are relatively small and many verbs and nouns do not appear in them. thus, we need a new mechanism for selection of appropriate noun synsets for participants of verbs. in this paper we used subatom semantic embeddings for checking which ones are appropriate. such subatom semantic embeddings for each verbal synset are constructed for the appropriate grammatical roles: subject, direct object and indirect object. having these embeddings, we rank each noun synset in pwn with respect to the corresponding grammatical role. the closer the noun synset embedding to the grammatical role embedding, the more appropriate is the noun synset as a participant for the corresponding grammatical roles in the selected verbal synset. in the rest of the paper we present some additional motivation why such subatomic embeddings are useful, how we could train and evaluate them. ## minimal recursion semantics an additional piece of motivation for subatom semantic embeddings is the construction of a logical form for a sentence. in many semantic theories the lexical semantics is represented not only by using predicates from first order logic, but by exploring a more complicated schema which would allow access to a more detailed representation of the semantic interpretation. as an illustration of such a kind of semantics we assume minimal recursion semantics (mrs) - @xcite . an mrs structure is a @xmath0, r, c>, where gt is the top handle, r is a bag of eps (elementary predicates) and c is a bag of handle constraints, such that there is no handle h that outscopes gt. each elementary predication contains exactly four components: (1) a handle which is the label of the ep; (2) a relation; (3) a list of zero or more ordinary variable arguments of the relation; and (4) a list of zero or more handles corresponding to scopal arguments of the relation (i.e., holes). here is an example of an mrs structure for the sentence "every dog chases some white cat." <h0, {h1:every(x,h2,h3), h2:dog(x), h4:chase(e, x, y), h5:some(y,h6,h7), h6:white(y), h6:cat(y)}, {}> the top handle is h0. the quantifiers are represented as the relations every(x, y, z) and some(x, y, z), where x is the bound variable, y and z are handles determining the restriction and the body of the quantifier. the conjunction of two or more relations is represented by sharing the same handle (h6 above). the outscope relation is defined as a transitive closure of the immediate outscope relation between two elementary predications -ep immediately outscopes ep' iff one of the scopal arguments of ep is the label of ep'. in the example the set of handle constraints is empty, which means that the representation is underspecified with respect to the scope of both quantifiers. in order to use semantic embeddings over mrs structures we need to determine the interactions of the latent features for each of the predicate arguments. for example, the features from the embeddings for 'every', 'dog', and 'chase' have to agree on the common argument denoted by the variable 'x'. in order to control this interaction in a better way, we would like for each multiargument predicate to learn an embedding per argument. thus for the above mrs structure we will need to have embeddings for 'x', 'y', 'e', 'h0', ... 'h7'. when we have them, we would like also to create an embedding related to the first argument of 'every'. the argument of 'dog' and the second argument of 'chase' have to "agree". our long-term goal is to train such subatom embeddings. here we present an approach for learning such embeddings for grammatical roles. then we use these embeddings for extending of wordnet with syntagmatic relations, as it was described above. ## grammatical role embeddings from parsed corpora in our first experiment we learned subatom semantic embeddings on the basis of dependency-parsed corpora. we determined the arguments as wordforms in the text. as an example, for the above mentioned case we used the position of 'dog'. in order to generalize over the different word forms in the different examples in the corpus we substituted the wordforms for the corresponding argument with a pseudoword form. for example, in the above sentence we generated the following variations with pseudoword forms for the different arguments of the different predicates: every subj_chase chases some white cat. every dog chases some white obj_chase. and many more. having learned embeddings for these pseudowords, we assume that they represent the selectional features for the corresponding grammatical roles of the verbs. the actual corpus we have used is wack-ypedia_en corpus - @xcite . the wackypedia_en corpus was reparsed with a more recent version of the stanford corenlp dependency parser. the dependency of type "collapsed-cc" was selected, which collapses several dependency relations in order to obtain direct dependencies between content words, and in addition propagates dependencies involving conjuncts. for instance, a parse of the sentence "the dog runs and barks" would result in the relations nsubj(dog, runs) and nsubj(dog, barks). this type of dependency allows for a token to have multiple head words. the head word of each noun phrase subject, as well as direct and indirect object, is then replaced by its predicate role and its governing verb's lemma (subj_run, subj_barkboth for the noun 'dog'). when a token has more than one head word suitable for substitution, copies of the sentence are created for each alternative replacement. for the relation has-subj we use the dependency relations 'nsubj' and 'nsubjpass'; for the relation has-dobj we use the dependency relation 'dobj'; and for the relation has-iobj we use the dependency relation 'iobj'. in order to minimize some errors we enforced a condition that the dependency word should be a noun. ## experiments and results in this section we describe the experimental set up and the results. ## conclusion the paper presents an approach for learning features by subtomic semantic representation. it is useful for addition of syntagmatic relations to wordnet. our longterm plan is to design a learning approach for each semantic argument of predicates in a logical form. the results here are the first steps in this direction. in future we plan to do the following: (1) to include more arguments in the learning process like arguments of relational nouns and adjectives. they will impose mutual constraints on the learned features; (2) to experiment with different algorithms for learning of embeddings such as @xcite , where it is possible to select arbitrary contexts. such contexts could be more appropriate for grammatical role embeddings learning; (3) to improve sense annotation in order to respectively improve sense embeddings; (4) to evaluate the subatom embeddings in other tasks such as coreference resolution, neural network word sense disambiguation; (5) to perform tuning to the linguistic knowledge already represented in wordnet, framenet and other lexical resources as well as manually annotated corpora, by techniques similar to retrofitting; and (6) to develop compositional semantics over this representation.