Representing and Reconstructing P hy SH : Which Embedding Competent? (2020)

Abstract: Recent advances in natural language processing make embedding representations dominate the computing language world. Though it is taken for granted, we actually have limited knowledge of how these embeddings perform in representing the complex hierarchy of domain scientific knowledge. In this paper, we conduct a comprehensive comparison of well-known embeddings’ capability in capturing the hierarchical Physics knowledge. Several key findings are: i, Poincaré embeddings do outperform if trained on PhySH taxonomy, but it fails if trained on co-occurrence pairs which are extracted from raw text. ii, No algorithm can properly learn hierarchies from the more realistic case of co-occurrence pairs, which contains more noisy relations other than hierarchical relations. iii, Our statistic analysis of Poincaré embedding’s representation of PhySH shows successful hierarchical representation share two characteristics: firstly, upper-level terms have a smaller semantic distance to root; secondly, upper-level hypernym-hyponym pairs should be further apart than lower-level hypernym-hyponym pairs.

## introduction concept hierarchy or taxonomy foot_0 is highly organized and expertly curated hierarchical hypernymhyponym sets. how to effectively represent these terms with the hierarchical relation is the main hurdle for automatically taxonomy construction and other downstream applications. though embeddings have been taken for granted in most nlp pipelines, none of the previous work has fully explored which embeddings can capture hierarchical scientific knowledge. even though poincaré embedding is proved to have a better ability to capture hierarchical relations, it is learned based on existing wordnet hypernym-hyponym pairs. it is never been tested in the scientific domain. in this paper, we conduct a comprehensive comparison of well-known embeddings' performance in reconstructing physical subject headings (physh) from raw aps datasets. our main contributions are mainly three-fold: firstly, for the first time, we compare mainstream embeddings' capability to represent and reconstruct physical subject headings (physh) both from raw text and physh. secondly, our experiment shows poincaré embedding is not sufficient for taxonomy induction from raw text. thirdly, we explore the characteristics of successful representation of physh, which might be the inspiration for better taxonomy construction algorithms. ## related work representations for concept hierarchy. representations for concept hierarchy has been receiving quite growing interests in recent years @xcite @xcite . it is the basis of automaticly taxonomy construction. in the survey study of @xcite , there are pattern-based @xcite @xcite methods and distributional @xcite @xcite @xcite methods use hand-crafted rule-based, co-occurrence features,syntactic features or graph features to learn representations of hierarchical pairs. they also apply pretrained neural laguage models such as word2vec @xcite , glove @xcite . recently, poincaré embedding @xcite is proposed to better represent hierarchical relations. following works like @xcite use lorentzian distance to replace the poincaré metric, @xcite extends poincaré embedding to apply in raw text with re-parameterization technique, @xcite and @xcite introduce hyperbolic embeddings in word embeddings like skipgram and glove. effectively in reconstructing wordnet though, the poincaré embedding is not quite perfect yet @xcite . it has only been tested on wordnet reconstruction and hyperlex entailment @xcite . whether it is an effective tool in representing hierarchical relations from raw domain text need to be further explored. ## method in our pipeline (figure 1 ), we follow three steps: raw text and physh preprocessing; learn various embeddings with different hyperparameters; evaluate embeddings by reconstructing physh. we evaluate the following embeddings: • word embeddings: cbow and skipgram @xcite , fasttext @xcite , glove @xcite foot_1 . taxonomy reconstruction: we follow @xcite to reconstruct taxonomy based on embedding vectors. for each embedding vector in poincaré disk space, which is denoted as the norm of each vector can measure the radius of each vector, while the hyperbolic distance can measure the closeness of two vectors. the closest two are assigned as hypernym-hyponym pairs. the hyperbolic distance of two vector points u, @xmath0. the distance could only tell how semantically close are the node pairs (u, v). but which one is the parent node is not answered. one property that makes hyperbolic space outstanding for the hierarchical structure is that the hyperbolic disc area and circle length grow exponentially with its radius. node with the smaller norm is the higherlevel term. ## conclusion and future work we further demystify what is the success of poincaré embeddings in reconstructing physh from physh. the future work would be how to design a powerful taxonomy induction algorithm which could benifit from the characteristics of our paper.