Cross-Modal Contextualized Hidden State Projection Method for Expanding of Taxonomic Graphs (2022)

Abstract: Taxonomy is a graph of terms organized hierarchically using is-a (hypernymy) relations. We suggest novel candidate-free task formulation for the taxonomy enrichment task. To solve the task, we leverage lexical knowledge from the pre-trained models to predict new words missing in the taxonomic resource. We propose a method that combines graph-, and text-based contextualized representations from transformer networks to predict new entries to the taxonomy. We have evaluated the method suggested for this task against text-only baselines based on BERT and fastText representations. The results demonstrate that incorporation of graph embedding is beneficial in the task of hyponym prediction using contextualized models. We hope the new challenging task will foster further research in automatic text graph construction methods.

## introduction in this paper, we focus on taxonomic structures which are quite relevant in many natural language processing (nlp) tasks such as lexical entailment @xcite and entity linking @xcite to represent the relations between products or employees. taxonomies are tree-like structures where words are considered as nodes (synsets) and the edges are the relations between them. such kinds of relationship is called a hypo-hypernym relationship. for instance, let us consider two words: "apple" and "fruit". the former word is hyponym ("child") to the latter and the latter is hypernym ("parent") to the former. many approaches have been proposed to automatically update existing taxonomies @xcite @xcite . however, we argue about one crucial limitation of the existing setups questioning their usefulness in real-world application. in the traditional taxonomy enrichment task setting the system is provided with the candidate (orphan) to add and the task is to find the correct place for it in the existing taxonomy. compiling lists with the new words to add is extremely important but inherently challenging: it might be not clear to which of the multiple sources we would give our preference: neologisms, teenage slang from the internet or professional jargon. on the contrary, large pre-trained language models such as bert @xcite , elmo @xcite , gpt @xcite already contain information about the majority of terms in a language. for instance, many probing studies @xcite @xcite show that a vast amount of linguistic information is encoded inside large transformer networks, e.g. syntax or lexical semantics. in our study, we assume that the huge amount of knowledge from pre-trained models can be leveraged to predict new words missing in taxonomic resources. we suggest a novel candidate-free task formulation for taxonomy enrichment, arguing that compiling word lists may be redundant. information about new words is already present in the large pre-trained networks. there would be not need in compiling lists of "parents" to predict hyponyms either, as language models should be able to predict words only if necessary. furthermore, we propose a cross-modal contextualized hidden state projection method (chsp) for candidate-free taxonomy enrichment. the approach includes several stages: (i) learning embeddings of wordnet taxonomy, (ii) projecting them into the hidden states space of bert, and (iii) decoding them back to text candidates. thus, the contribution of our work is three-fold: ## related work there has been two major competitions that have introduced the task of taxonomy enrichment: se-meval 2016 @xcite and russe-2020 @xcite . however, their formulations both required a predefined list of candidates. a detailed overview of taxonomyrelated papers is presented in @xcite ; @xcite . at the same time there exists a lot of research on how suitable is bert for capturing and transferring information about hypo-hypernym relationship @xcite ; @xcite ; @xcite . for instance, @xcite examine hypernymy knowledge encoded in bert representations. in their experiments bert demonstrated the ability to correctly retrieve hypernyms, however, they argue that it does not necessarily follow that bert is capable of systematic generalisation. another paper about bert's knowledge of hypernymy @xcite applies several patterns to predict possible hypernym candidates: "[mask], such as x" and "my favorite [mask] is x". such prompts often elicit correct hypernyms from bert. however, bert still fails in 43% of cases, therefore, the authors claim that bert has limited understanding of hypernymy. there exist many more hearst patterns @xcite that aim to identify hypo-hypernym relationship in unlabeled texts @xcite . we compare baselines with some of them in section 6. @xcite examine the influence of context-aware word representation models for lexical units and frame role expansion task. this task is related to our setting in a sense of generation of meaningful substitutes with preservation of content. we adopt their context-aware methods for our task. in our case the meaningful substitute will be generated for a masked hyponym with preservation of meaning represented in projected embeddings (see section 4). ## taxonomy enrichment task we formulate taxonomy enrichment in a new way avoiding the need of pre-supplied candidates (cf. fig. 1 ) making it more challenging yet realistic. given a taxonomy @xmath0, r, t} ⊆ e × r × e, the task is to predict new nodes @xmath1, n ̸ ⊆ e, which are not yet included in the taxonomy t , starting from the current node h @xmath2. ## cross-modal contextualized hidden state projection method the main idea of the paper is to predict new words using knowledge preserved in bert and enhance the word generation process with graph information. fig. 2 demonstrates the overall architecture of the chsp approach that we use to solve the task. first, we train a graph representation model to compute graph embeddings. furthermore, we learn a projection layer to transform target graph embeddings to the bert vector space. then we apply the projected embeddings as input to the masked language modelling part of bert model. the prediction head generates new lemmas that are treated as candidate hyponyms for parent nodes. this process results in gradual joining of graph and textual modalities. ## baselines in our experiments we are using three baselines: 1. fasttext (nearest neighbours); 2. bert (parent embeddings on inference); 3. three patterns from @xcite . ## experiments our experiments can be categorised by following features: source graph embeddings, usage of context structure, replacement layer and replacement strategy. this section is divided into two parts. the first subsection compares various combinations of chsp configurations. the second subsection analyzes performance of the best chsp configurations against the baselines. ## error analysis we can categorise common errors into several groups: failing to differentiate the real meaning of the hypernym, prediction of synonymical/same domain words instead of hyponyms, weakness of multi-token generator. the first type of errors is related to incorrect recognition of a rare meaning of a synset and mistaking of it for a more common one. for example, for hypernym "depression.n.10" (pushing down) the correct prediction would be "click". however, almost all results are medical related predictions, e.g., headache, coma, schizophrenia. an example of the second type of errors might be predictions of multi-token pipeline with graph-bert embeddings for "jazz_musician.n.01" hypernym. while the correct answer is "syncopator", top produced predictions are "singer", "dj", which obviously come from the same music-related domain. for multi-token node2vec we observed a lot of cases where one strong word was produced and further multi-token hypothesis would retain this first word and simply permute other different words. example output for test hypernym "suburb.n.01": suburb, suburb suburbs, suburbs, suburb suburban, suburb suburbs suburban, etc. because of the weak multi-token decoding mechanism, many predictions failed. for example, none of the setups managed to produce adequate hyponyms for "berry.n.01", because all correct answers are multi-token in bert vocabulary. all in all, the results are diverse an controversial. for instance, for instance, the model can generate candidates that are correct but they are not yet included to the taxonomy. in this case, the evaluation system will still mark them as incorrect. therefore, as future work we plan not only improve current methods but also perform human evaluation of the results. another reason for the absolute low scores is the way the test set was generated. while in @xcite the data is selected from the well-known domains like "pets", "food", "sport", our test set is generated randomly and thus comprises rare terms, which may be harder to process. at the same time, simple examples like "beverage" or "meal" gain better scores. as future work we want to tackle the problem of rare terms. ## conclusion in this work, we presented a novel candidate-free task formulation for taxonomy enrichment. the contribution is three-fold: task proposal, according dataset and test of multi-modal approach. we performed a computational study of various methods using knowledge from bert. we compared different graph-based embeddings on the task and projected them to the bert vector space. then we identified the best position for the projected graph embedding to be injected to the bert model. the results demonstrate that incorporation of graph embedding is beneficial in the task of hyponym prediction using bert. nevertheless, the bert architecture does not allow us to easily operate with multi-token words and the pipeline accumulates errors in each component. this may be room for improvement for generative models like gpt or t5 and their prompt-tuning. all in all, the proposed task is proven to be very challenging paving the way for future research.