Graph-based Relation Mining for Context-free Out-of-vocabulary Word Embedding Learning (2023)

Abstract: The out-of-vocabulary (OOV) words are difficult to represent while critical to the performance of embedding-based downstream models. Prior OOV word embedding learning methods failed to model complex word formation well. In this paper, we propose a novel graph-based relation mining method, namely GRM, for OOV word embedding learning. We first build a Word Relationship Graph (WRG) based on word formation and associate OOV words with their semantically relevant words, which can mine the relational information inside word structures. Subsequently, our GRM can infer high-quality embeddings for OOV words through passing and aggregating semantic attributes and relational information in the WRG, regardless of contextual richness. Extensive experiments demonstrate that our model significantly outperforms state-of-the-art baselines on both intrinsic and downstream tasks when faced with OOV words.

## introduction pre-trained word embedding models, such as word2vec @xcite and bert @xcite , can not only boost the performance of downstream tasks but also accelerate the convergence of downstream models @xcite . however, in real-world scenarios, the pre-trained models trained with generic large-scaled corpora may encounter a lot of words never seen before in downstream tasks due to domain specificity. these outof-vocabulary (oov) words rarely appear, resulting in a scarcity of their contexts, while traditional word embedding methods require a large number of contexts to learn high-quality word embeddings @xcite . the oov words may cause a dramatic performance degradation in downstream tasks because of their poor word embeddings @xcite @xcite , which leads to the oov problem. thus, it is vital to explore an effective way of learning high-quality oov word embeddings in natural language processing. traditional methods for tackling the oov problem injected sub-units of words into the training process of pre-trained word embedding models to get the sub-unit embeddings and then calculated the oov word embedding as a summation of them @xcite @xcite . these methods require training from scratch, which are time consuming. to save computing resources, two categories of methods have been proposed. methods in the first category attempted to fully utilize limited contextual information carried by oov words directly without modifying the training process of background models @xcite @xcite . these methods are often lightweight, but they cannot deal with some frequently occurring situations where the oov words are extremely context-less. to break the limitation of contexts, methods in the other category learned word embeddings for oov words through fine-grained sub-units or morphemes to model word formation implicitly without using contexts @xcite @xcite . however, the word formation can be complex and highly internally structured @xcite , rendering simple simulations cannot represent the word formation well. in the situation of context absence, it's meaningful to utilize word formation for oov words since most language vocabularies are derived from the creation of new words on the basis of old ones @xcite . intuitively, humans can guess the meaning of an oov word based on its complex word formation and association with similar words, as shown in figure 1. however, the measures of word formation are varied, and the relationships inside word structures are sophisticated. taking this into account, we introduce a word relationship graph (wrg) to imitate word formation and word association for better capturing the relational information of word-internal structure, which logically simulates human learning habits when facing an oov word. in light of these considerations, we propose a graph-based relation mining (grm) model on the basis of wrg to learn embeddings for oov words without contexts, which can help mine the relational information about complex word formation. our method can also explore additional semantic information by associating each oov word with its relevant words. to achieve these, we transport and incorporate relational information and semantic attributes by graph neural network (gnn). noteworthy, we use the graph structure to find more reasonable positive sample pairs for contrastive learning, forcing every node embedding to be more informative. the contributions of our work can be summarized as follows: ## proposed method within the existing methods, mimick @xcite used a lightweight post-processing learning paradigm, which attempted to mimic the vector space of a background embedding model for oov words and can therefore be applied to different types of embedding models. the mimick paradigm sought to maximize the similarity between the inferred embeddings produced by the oov word embedding model and the original embeddings derived from the background embedding model. we follow this mimick learning paradigm to mine the relational information about word formation. compared to other data structures, graph structure can model complex data compositions well. therefore, we construct a wrg to model word formation rules and associate other semantically related words. the relational information and relevant semantic attributes can be transported and aggregated on wrg by gnn. besides, we utilize graph structure in the process of positive sample pairs selection for contrastive learning, which can provide the flexibility to obtain more reasonable positive sample words. ## experiments in this section, we carry out extensive experiments on several widely-used text datasets varying in scale to test different methods, which can be categorized into intrinsic and extrinsic evaluators. furthermore, we plug grm into static and contextual word embedding models to show the gains brought by grm. finally, we conduct qualitative analysis and ablation study on grm. ## model feasibility for other languages in this section, we discuss the feasibility of our grm model to other languages. in the foregoing section, we introduced our grm model that splits oov words into wordpieces, then constructs wrg around oov words, which can associate relevant words through wordpieces. due to the design of the model, our grm matches with the properties of an agglutinative language, such as japanese or korean, which forms words by stringing morphemes together directly. fusional language is more difficult to process than agglutinative one because the morphemes are usually linked together. the language explored in our paper, english, is a fusional language with some agglutinative properties. it can be observed that grm performs quite well on the fusional language by reasonable segmentation of words, which indicates that the application effectiveness of grm to other languages depends on the rationality of word decomposition only. theoretically, the graph structure of wrg in grm can cope with various complex word formations, thus grm can infer highquality embeddings for oov words though capturing the relational information inside the word structures and associating other relevant words. ## conclusion in this paper, we present a graph-based method named grm for oov word embedding learning. we creatively propose to model word formation through using wrg which can help to mine relationships inside word structures and associate relevant words. we demonstrate our superiority over baseline models through word similarity, word analogy, ner, and pos tagging tasks. besides, our grm model can be easily incorporated into static and contextual pre-trained embedding models, and help them alleviate the oov problem effectively. furthermore, on the qualitative analysis, we observe that grm can discover the semantic relational information between words, which validates the ability of grm to recover relationship information between words. our code and supplementary materials are available in public at: https://github.com/liangzrtvjivo/grm .