Exploring Semantic Capacity of Terms (2020)

Abstract: We introduce and study semantic capacity of terms. For example, the semantic capacity of artificial intelligence is higher than that of linear regression since artificial intelligence possesses a broader meaning scope. Understanding semantic capacity of terms will help many downstream tasks in natural language processing. For this purpose, we propose a two-step model to investigate semantic capacity of terms, which takes a large text corpus as input and can evaluate semantic capacity of terms if the text corpus can provide enough co-occurrence information of terms. Extensive experiments in three fields demonstrate the effectiveness and rationality of our model compared with well-designed baselines and human-level evaluations.

## introduction terms are not all considered equal. for instance, in computer science, the meaning scope of artificial intelligence or computer architecture is broader than that of linear regression. to study this phenomenon, in this paper, we introduce semantic capacity, which is a scalar value to characterize the meaning scope of a term. a good command of semantic capacity will give us more insight into the granularity of terms and allow us to describe things more precisely, which is a crucial step for downstream tasks such as keyword extraction @xcite @xcite ) and semantic analysis @xcite . figure 1 shows the fingerprint visualization of a computer scientist, which is generated by elsevier fingerprint engine 1 , a popular system that creates * asterisk indicates equal contribution. work done while visiting university of illinois at urbana-champaign. from the example, we can find that there exist some non-ideal terms, such as learning whose semantic capacity is too high, backpropagation whose semantic capacity is too low, and even irrelevant terms such as color. understanding semantic capacity of terms will help us to choose better terms to describe entities. besides, combining with other techniques like word similarity, semantic capacity can also help keyword replacement. for instance, to describe the computer scientist depicted in figure 1 , if the audience is a layman of computer science, we should use terms with high semantic capacity like artificial intelligence. but for an expert in the corresponding domain, we can select terms with low semantic capacity like object recognition to make the fingerprint more precise. however, there are countless terms in human language, which means that it is extremely hard to investigate semantic capacity for all existing terms. besides, semantic capacity of terms is also ambiguous in different domains. for instance, cheminformatics may be considered as a term with low semantic capacity in computer science and a term with high semantic capacity in chemistry. on the other hand, the information on terms we acquire is usually very limited and/or noisy. although semantic taxonomies such as wordnet @xcite provide rich semantic relations between words, the information is still limited, and these knowledge bases are expensive to maintain and extend. besides, there exists some research work that models hierarchical structures of terms automatically, but most of them suffer from low recall or insufficient precision. for instance, hypernymy discovery @xcite @xcite aims at finding is-a relations in textual data. if we can find all the hypernymy pairs and construct a perfect tree structure that includes every term, the problem of semantic capacity can be solved to some extent. however, as far as we know, this is almost impossible in state of the art. the above analysis shows that we should focus the problem on a specific domain and cover as many terms as possible with easily accessible information. besides, we should also consider user requirements and deal with terms that are not included at first. therefore, we propose a two-step model that only takes a text corpus as input and can evaluate semantic capacity of terms, provided that the text corpus can give enough co-occurrence signals. our model consists of the offline construction process and the online query process. the offline construction process measures semantic capacity of terms in a specific semantic space, which narrows the problem to a specific domain and reduces the complexity of the problem to a practical level. the online query process deals with users' queries and evaluates newly added terms that users are interested in. to learn semantic capacity of terms with simple co-occurrences between terms, we introduce the semantic capacity association hypothesis and propose the lorentz model with normalized pointwise mutual information, where terms are placed in the hyperbolic space with a novel combination of normalized pointwise mutual information. finally, norms of embeddings are interpreted as semantic capacity of terms. the main contributions of our work are summarized as follows: the code and data are available at https:// github.com/c3sr/semantic-capacity. ## methodology in this section, we introduce the definition of semantic capacity and describe our model in detail. the overview of our model is shown in figure 2 . ## experiments in this section, we conduct experiments to validate the effectiveness of our model. ## related work our work is related to research on lexical semantics @xcite . among them, hypernymy, also known as is-a relation, has been studied for a long time. a well-known method is the hearst patterns @xcite , which extracts hypernymy pairs from a text corpus by hand-crafted lexicosyntactic patterns. inspired by the hearst patterns, some other pattern-based-methods like @xcite are proposed successively. on the other hand, hypernymy discovery based on distributional approaches has also attracted widespread interest @xcite @xcite . the techniques our model based on are related to research on learning representations of symbolic data in the hyperbolic space @xcite @xcite . since text preserves natural hierarchical structures, @xcite design a framework that learns word and sentence embeddings in an unsupervised manner from text corpora, @xcite propose poincaré glove to learn word embeddings based on the glove algorithm in the hyperbolic space, @xcite use poincaré embeddings to improve exiting methods to domain-specific taxonomy induction, and @xcite propose a method to predict missing hypernymy relations and correct wrong extractions for hearst patterns based on the hyperbolic entailment cones @xcite . ## conclusion in this paper, we explore semantic capacity of terms. we first introduce the definition of semantic capacity and propose the semantic capacity association hypothesis. after that, we propose a two-step model to investigate semantic capacity of terms, which consists of the offline construction and the online query processes. the offline construction process places domain-specific terms in the hyperbolic space by our proposed lorentz model with npmi, and the online query process deals with user requirements, where semantic capacity is interpreted by norms of embeddings. extensive experiments with datasets from three fields demonstrate the effectiveness and rationality of our model compared with well-designed baselines and human-level evaluations. in addition, while semantic capacity studied in this paper is restricted to a specific domain, we believe the notion of semantic capacity can be ex-tended to all terms in human language. the extension of the scope will be the future work.