TEAM : A multitask learning based Taxonomy Expansion approach for Attach and Merge (2022)

Abstract: Taxonomy expansion is a crucial task. Most of Automatic expansion of taxonomy are of two types, attach and merge. In a taxonomy like WordNet, both merge and attach are integral parts of the expansion operations but majority of study consider them separately. This paper proposes a novel mult-task learning-based deep learning method known as Taxonomy Expansion with Attach and Merge (TEAM) that performs both the merge and attach operations. To the best of our knowledge this is the first study which integrates both merge and attach operations in a single model. The proposed models have been evaluated on three separate WordNet taxonomies, viz., Assamese, Bangla, and Hindi. From the various experimental setups, it is shown that TEAM outperforms its state-of-the-art counterparts for attach operation, and also provides highly encouraging performance for the merge operation.

## introduction taxonomy, such as the wordnet, is a crucial resource for developing nlp related technologies, as it plays a vital role in various text processing tasks such as information retrieval, information extraction, text classification, summarization, etc. @xcite @xcite ) @xcite . as most of the wordnets are manually curated, it often suffers from the problem of limited coverage. therefore, an automatic taxonomy expansion is a crucial problem to handle the above issue. for taxonomy expansion, wordnet in particular, may need two types of operations; (i) merge, where a new concept 1 is merged to an existing node, and (ii) attach, where a new concept is inserted as a new node. figure 1 illustrates these two operations where the word mango is inserted as a new concept with the attach operation, and the figure 1 : example of wordnet taxonomy expansion with attach and merge operations to include new terms "mango" and "nutrient". though both of these operations are integral parts of a wordnet taxonomy expansion, all of the existing studies on taxonomy expansion have considered expansion with either attach operation @xcite @xcite @xcite @xcite or merge operation @xcite @xcite @xcite @xcite , but not together. realizing the need to apply both the operation, semeval-2016:task 14 (semantic taxonomy enrichment) @xcite includes a call for expansion with both attach and merge operations. however, none of the submissions incorporate both operations in a single model. motivated by the above observations, in this study, we propose an integrated deep learningbased method, namely, taxonomy expansion with attach and merge (team), which performs both the attach and merge operations in a multitasklearning framework. though most of the existing studies consider the expansion a regression problem @xcite @xcite , considering that our method performs both the attach and merge operation in a single model, it can also be considered a classification task. as a result, we propose two versions of team, namely, team-rg: regression, and team-cl: classification to perform with explicit and implicit rankings. the proposed models have been evaluated on three different wordnet taxonomies, viz., assamese, bangla, and hindi. from the various experimental setups, it is observed that the proposed team-rg and team-cl outperform their baselines counterparts for attach operation, and also obtained encouraging performance for merge operation as well. the major contributions of the paper are summarized as follows: to the best of our knowledge, it is the first integrated model to perform both the attach and merge operations in a single model. ## taxonomy expansion -attach and merge in this study, we have considered wordnets as our target taxonomies. a wordnet may be defined by a collection of concepts connected by various semantic relationships such as hypernymy, hyponymy, troponymy, etc., where each concept is further defined by a set of attributes such as definition, synonyms, examples, etc @xcite . in this study, we have considered only the hypernymy relation and the definition and synonymy attributes. in order to be able to apply the proposed model, we first transform the original wordnet taxonomy into an experimental intermediate taxonomy ( directed unweighted acyclic graph) @xmath0, e) where v represents the set of concepts and e represents the set of hypernymy relations between the concepts. a concept @xmath1, s v ) where d v represents the definition of the concept, and s v represents the set of associated synonyms. an edge @xmath2: (v p hyper ---→ v ch ). the taxonomy t is arranged in a hierarchical manner with directed edges in e, as shown in figure 1 . given the taxonomy t and a query concept @xmath3, s q ), the attach and the merge expansion operations are defined below. attach (a) -an attach operation is performed when the concept q is not present in t. the objective of the attach operation is to identify the best matching parent node in taxonomy network known as anchor concept @xmath4, s a ), and insert a new concept q with an edge e : (a hyper ---→ q). in a taxonomy network, a parent node represents a more generic concept of its children. after an attach operation i.e., insertion of q in t under the anchor a, the expanded taxonomy is updated as follows. merge (m) -a merge operation is performed when an equivalent concept @xmath5, s a ) of the query q (i.e., d a ≡ d q ) is already present in t, but the synset s q is not present in a (i.e., s q ∩ s @xmath6. the objective of the merge operation is to identify the best matching concept @xmath7, s a ), known as the anchor concept, in the taxonomy network t and add the synset s q to s a . it neither creates a new node nor adds a new edge. it only updates the synset of the anchor concept. after the merge operation, the updated anchor concept in the expanded taxonomy can be expressed as follows. our objective is to develop an integrated model that performs both attach and merge operations for taxonomy expansion. since we have two tasks to unify in a single model, we resort to a multi-task learning framework known as taxonomy expansion framework with attach and merge (team). this joint learning objective facilitates information flow so that the two tasks can aid each other. also, we are interested in deciding which expansion operation is to perform given a triplet (expansion task classification) and retrieving the ranked list of candidates the table shows positive and negative training instances corresponding to the query concept "rock" for both operations attach and merge. ## experiments and results here we give you an overview of our experiment settings and provide the detailed reproducibility information in the sections c, d, e of the appendix. metrics. we use mean rank (mr), hit@k, and mean reciprocal rank (mrr) to evaluate the ranks of the retrieved results obtained from different models, for the test queries. like taxo-expan @xcite evaluation strategy, we scale the mrr score by a factor of 10 to highlight the discrepancy of the performances among different methods. further, we use accuracy, micro/ macro f1, precision, recall, and f-scores to evaluate a method's prediction capability to decide which operation among merge (m), attach (a), and no-operation (n) needs to be performed. baselines. we choose two most recent benchmark sota taxonomy-expansion frameworks tax-oexpan @xcite and triplet matching network(tmn) @xcite as the competing methods. as taxo-expan and tmn outperform semeval-2016 @xcite , we have not included semeval-2016 as baseline in this study. in terms of learning objective, taxo-expan is similar to ours. it uses ego-treebased anchor features for matching query features in a regression-based setting. tmn captures finegrained relationship dynamics of query and anchor concepts using channel-wise gating mechanismbased attention learning. all datasets and our model implementations are available at: https://github.com/ barnal/team ## results here we report the classification and ranking results of the competing methods. we also compare and contrast among the variants of our team framework. apart from the two versions of the team, namely, team-rg and team-cl, we have task-specific model variants specified asattach-a, merge-m and merge+attach-ma. here, (attach+merge) means simultaneously optimizing for both the tasks. ## related works existing methods for taxonomy expansion can be divided into two categories: relying on alignment between multiple taxonomies @xcite , @xcite , ponzetto and navigli (2009), and @xcite ] or relying on machine learning-based rating sub-graphs. further, the latter category can be divided into two sub-categories (1) by expanding synonymy relations/merge (2) by expanding hypernymy relations/attach. synonymy-based taxonomy expansion leverages synonymy relations of the taxonomy. given a seed taxonomy, the distributional approach discovers synonyms by representing strings with their distributional feature and learning a classifier to predict the relation between strings [ @xcite , @xcite , @xcite ]. most of the recent taxonomy expansion approaches are based on hypernymy expansion. these methods attempt to determine the attachment position by scoring between several nodes. recently numerous methods have been proposed to solve this problem @xcite , @xcite , @xcite , @xcite , @xcite . @xcite . hence, all the existing taxonomy expansion approaches expand a taxonomy either by merge operation(synonymy expansion) or by attach operation(hypernymy expansion). however, particular to wordnet expansion it is an integrated task of merge and attach operation. we are the first to study the problem of taxonomy expansion using both the attach and merge taxonomy expansion operations in a single model. a detailed related study is in section a of the appendix. ## conclusion in this paper, we proposed an integrated framework called taxonomy expansion with attach and merge (team) for expanding taxonomy with attach and merge operations together. we built two multi-task learning-based variants of team, namely, team-regression and team-classification, which solve the taxonomy expansion problem as regression and classification, respectively. our proposed methods learned to predict the taxonomy expansion operation (merge, attach, or no-operation) to perform and provided a ranked list of candidates. we evaluated the effectiveness of team on wordnet taxonomies of three distinct languages, viz., assamese, bangla, and hindi. in various experimental setups, the proposed team-rg and team-cl outperformed its state of the art for attach operation and provided a highly encouraging performance on merge operation. we had also investigated the performance of the proposed model with out-ofvocabulary concepts. in the future, we plan to investigate the response of the proposed model with different types of taxonomies and wordnet of different languages. another future research possibility can be to explore the response of this model using advanced contextual encoders.