Learning Contextualized Box Embeddings with Prototypical Networks (2024)

Abstract: This paper proposes ProtoBox, a novel method to learn contextualized box embeddings. Unlike an ordinary word embedding, which represents a word as a single vector, a box embedding represents the meaning of a word as a box in a high-dimensional space: that is suitable for representing semantic relations between words. In addition, our method aims to obtain a “contextualized” box embedding, which is an abstract representation of a word in a specific context. ProtoBox is based on Prototypical Networks, which is a robust method for classification problems, especially focusing on learning the hypernym–hyponym relation between senses. ProtoBox is evaluated on three tasks: Word Sense Disambiguation (WSD), New Sense Classification (NSC), and Hypernym Identification (HI). Experimental results show that ProtoBox outperforms baselines for the HI task and is comparable for the WSD and NSC tasks.

## introduction word embedding is an abstract representation of a word, usually as a vector in a high dimensional space. nowadays, word embeddings are widely used in models based on deep learning. word embedding can represent the meaning not only of a word itself @xcite but also of a word in a context. for example, bert @xcite is often used to obtain vector representations of words in a given sentence. in this paper, we call such word embeddings "contextualized word embeddings." in addition, box embedding @xcite is a kind of word embedding, which represents a word not with a point but with an area in vector space. while an ordinary word embedding is primary used to measure the similarity between two words, box embeddings can be used to capture other semantic relations between 1 our code is available at: https://github.com/iehok/ protobox . words such as that between a hypernym and a hyponym. however, past studies of box embeddings did not well consider the context of the word, that is, the box embedding was not contextualized. contextualized word embeddings can be regarded as "sense embeddings," since a word may have two or more senses and convey one of those possible senses in a specific context. word sense disambiguation (wsd) @xcite is a task that aims to identify the meaning of a word in a context. most previous work on wsd has focused only on the senses in a predefined inventory and has ignored new (not predefined) senses. however, senses of words change day by day and new senses are constantly created @xcite . it is preferable that a wsd system can handle a new sense by classifying a word even if it is being used in a new sense, and, if possible, explaining the meaning of the new sense. in this paper, we propose protobox, a method to produce a contextualized box embedding of a word in a given context. in general, box embeddings can represent hypernym-hyponym relations between 1 words, as illustrated in figure 1 @xcite . if a box of word w a subsumes the box of another word w b , w a can be regarded as a hypernym of w b . such relations between words can be represented as taxonomy (figure 1 (b) ). our protobox can produce contextualized box embeddings. for example, a box embedding of x in the sentence "the x hopped across the grass." can be obtained as shown in @xcite . contextualized box embeddings enables us to judge that x has a new sense when the box embedding of x does not overlap any other box embeddings of fine-grained senses such as "cat" and "dog". in addition, "animal" can be identified as a hypernym of x, since the box embedding of x is subsumed by that of "animal". identification of a hypernym can provide a rough explanation of a new sense, i.e., x is a kind of an animal. furthermore, protobox can expand the existing taxonomy by adding a new node x to the structure as shown in figure 1 (b) . we evaluate protobox with three tasks: wsd, new sense classification (nsc), and hypernym identification (hi). three datasets of different domains, one is large and two are small, are used to thoroughly evaluate our proposed method. experimental results show that protobox is better than or comparable to the baselines for wsd and nsc, and always outperforms the baselines for hi. the contributions of this paper are summarized as follows: ## proposed method this section describes the details of protobox, our proposed method to train contextualized box embeddings. we first explain box embeddings, as background, in subsection 3.1, then explain proto-box in the succeeding subsections. ## applications of protobox this section describes how protobox is applied to three tasks: word sense disambiguation, new sense classification, and hypernym identification. ## conclusion this paper proposed protobox, an expansion of metricwsd to learn contextualized box embeddings. the representations of words in a context were changed from single vectors in metricwsd to box embeddings in our protobox, since box embeddings are suitable to represent semantic relations between senses such as the hypernym-hyponym relation. additionally, we proposed a method to construct episodes to train the model to produce the contextualized box embeddings. we evaluated pro-tobox on three tasks: word sense disambiguation (wsd), new sense classification (nsc), and hypernym identification (hi). protobox outperformed the baselines in terms of all evaluation metrics in the hi task. this was reasonable, since protobox was designed to take the hypernym-hyponym relation into account when training the contextualized box embeddings. in addition, protobox achieved a performance comparable with the baselines for the other sense related tasks, wsd and nsc. in the future, the scalability of protobox should be improved. as reported in subsection 5.3, the performance of protobox was degraded when the number of sense instances was large. a more ef-ficient and precise method to learn contextualized box embeddings should be investigated. in addition, the definition of the prototype representation (the box embedding of a sense) can be reconsidered. currently, the prototype representation is an average of box embeddings of the elements in the support set. however, it can be a box that includes all the elements in the support set. it is worth to explore better ways to obtain the prototype representation.