H yper T ext: Endowing F ast T ext with Hyperbolic Geometry (2020)

Abstract: Natural language data exhibit tree-like hierarchical structures such as the hypernym-hyponym hierarchy in WordNet. FastText, as the state-of-the-art text classifier based on shallow neural network in Euclidean space, may not represent such hierarchies precisely with limited representation capacity. Considering that hyperbolic space is naturally suitable for modelling tree-like hierarchical data, we propose a new model named HyperText for efficient text classification by endowing FastText with hyperbolic geometry. Empirically, we show that HyperText outperforms FastText on a range of text classification tasks with much reduced parameters.

## introduction fasttext @xcite is a simple and efficient neural network for text classification, which achieves comparable performance to many deep models like char-cnn @xcite and vdcnn @xcite , with a much lower computational cost in training and inference. however, natural language data exhibit tree-like hierarchies in several respects @xcite such as the taxonomy of wordnet. in euclidean space the representation capacity of a model is strictly bounded by the number of parameters. thus, conventional shallow neural networks (e.g., fasttext) may not represent tree-like hierarchies efficiently given limited model complexity. fortunately, hyperbolic space is naturally suitable for modeling the tree-like hierarchical data. theoretically, hyperbolic space can be viewed as a continuous analogue of trees, and it can easily embed trees with arbitrarily low distortion @xcite . experimentally, @xcite first used the poincaré ball model to embed hierarchical data into hyperbolic space and average pooling linear classifier poincar embedding word embedding einstein midpoint hyperbolic classifier m bius linear visualize midpoint tree-like embedding achieved promising results on learning word embeddings in wordnet. inspired by their work, we propose hypertext for text classification by endowing fasttext with hyperbolic geometry. we base our method on the poincaré ball model of hyperbolic space. specifically, we exploit the poincaré ball embedding of words or ngrams to capture the latent hierarchies in natural language sentences. further, we use the einstein midpoint @xcite as the pooling method to emphasize semantically specific words which usually contain more information but occur less frequently than general ones @xcite . finally, we employ möbius linear transformation @xcite to play the part of the hyperbolic classifier. we evaluate the performance of our approach on text classification task using ten standard datasets. we observe hypertext outperforms fasttext on eight of them. besides, hypertext is much more parameterefficient. across different tasks, only 17% ∼ 50% parameters of fasttext are needed for hypertext to achieve comparable performance. meanwhile, the computational cost of our model increases moderately (2.6x in inference time) over fasttext. ## conclusion we have shown that hyperbolic geometry can endow the shallow neural networks with the ability to capture the latent hierarchies in natural language. the empirical results indicate that hypertext consistently outperforms fasttext on a variety of text classification tasks. on the other hand, hyper-text requires much less parameters to retain performance on par with fasttext, which means neural networks in hyperbolic space could have a stronger representation capacity.