Knowledge-Enhanced Named Entity Disambiguation for Short Text (2020)

Abstract: Named entity disambiguation is an important task that plays the role of bridge between text and knowledge. However, the performance of existing methods drops dramatically for short text, which is widely used in actual application scenarios, such as information retrieval and question answering. In this work, we propose a novel knowledge-enhanced method for named entity disambiguation. Considering the problem of information ambiguity and incompleteness for short text, two kinds of knowledge, factual knowledge graph and conceptual knowledge graph, are introduced to provide additional knowledge for the semantic matching between candidate entity and mention context. Our proposed method achieves significant improvement over previous methods on a large manually annotated short-text dataset, and also achieves the state-of-the-art on three standard datasets. The short-text dataset and the proposed model will be publicly available for research use.

## introduction name entity disambiguation (ned) aims to associate each entity mention in the text with its corresponding entity in the knowledge graph @xcite . it plays an important role in many text-related artificial intelligent tasks such as recommendation and conversation, since it works as a bridge between text and knowledge. in decades, researchers devoted their efforts to ned in many ways, including the rule-based methods @xcite , the conventional statistic methods @xcite and the deep learning methods @xcite . on formal text, state-of-theart methods achieve high performance thanks to the well-written utterance and rich context. however, experiments show that the performance of these methods degrades dramatically on informal text, for example, the short text widely used in many real application scenarios such as information retrieval and human-machine interaction. it is difficult for existing methods to make decisions on the non-standard utterance without adequate context. the discrimination procedure of ned depends on sufficient context in the input text, which is usually noisy and scarce in the short text used in information retrieval and human-machine interaction. for example, an analysis based on search engine logs demonstrates that a search query contains 2.35 words on average @xcite . such short text could not provide adequate context which is necessary for ned models. in recent years, many efforts improve ned by exploiting more powerful models and richer context information @xcite . these methods mainly focus on the better utilization of existing context. therefore, they can not improve ned effectively on short text since the problem of information shortage still exists. intuitively, it is hard for ned to achieve essential improvement on short text if it can not exploit external information to enhance the recognition procedure. in information scarce situations, human beings can still perform recognition by association with related external information, such as commonsense or domain-specific knowledge. it inspires us that, ned on short text could be improved if appropriate external knowledge can be retrieved and considered. we propose a novel knowledge-enhanced ned model, where the prediction procedure of ned is enhanced by two kinds of knowledge formalized as two different kgs. the one kind is conceptual knowledge formalized as a conceptual kg, it is used to augment the representation of the entity mention by giving the mention a concept embedding. the other kind is factual knowledge formalized as a factual kg, it is used to augment the representation of each candidate entity by giving the entity an entity embedding. the augmented which singer sang li bai? ( wedding photos of li bai and sa beining? ) [ ] [ ] [ ] [ ] [ ] [ ] * li bai ( ) has 21 ambiguous entities in kb . . . . ( how to play the hero li bai? ) ( what famous poems written by li bai? ) ## task definition and baseline model ned is a fundamental task in the area of natural language processing and knowledge base. it aims to associate each entity mention in the given text with its corresponding entity in the given kg. formally, given a kg g and a piece of text t , it assigns each mention @xmath0, or with the symbol φ indicating that there is no corresponding entity. the disambiguation procedure can be formalized as matching between the context of the mention and each candidate entities. here, the function e returns the entity candidate set for a given mention, and the function c returns the context of the given mention. the function s is used to evaluate the matching degree between context and candidate, and is usually implemented as matching networks. if the entity candidate set is empty or the highest matching score is bellow a given threshold, the function f returns φ for the given mention. the conditions g and t in the functions e and c are omitted in the equation for simplicity. we adopt the deep structured semantic model (dssm) @xcite as the baseline model for ned @xcite . based on a selfattention matching network, dssm maps the candidate entities and the context to the same semantic space, and finds the candidate entity that best semantically matches the context. the representation learning for both entities and contexts is enhanced product attention fc sim loss self attention lstm embedding candidate entity (ce) mention and context (mc) entity context embedding entity text meta entity relation entity embedding knowledge feature network graph construction conceptualize dynamic conceptualization entity context emb conceptualization emb entity emb context emb knowledge base score(ce, mc) who is the singer of li bai? li bai by word2vec @xcite . in this work, we focus on the problem of ned itself, that is, predicting the right entity candidate for each given entity mention. the entity mentions needed for ned are simply derived from the results of named entity recognition (ner). ## knowledge-enhanced ned model for short text used in information acquisition scenarios such as information retrieval and question answering, the lack of both lexical and syntactical information obstacles the precise disambiguation of entity mentions. for human beings, however, the problem of information scarcity does not hinder the disambiguation procedure. this is because there are many implicit assumptions and apriori knowledges in these information acquisition scenarios, which can be effectively considered by association and imagination during disambiguation procedure. inspired by this, we propose a knowledge-enhanced ned model (k-ned) for short text, where two kinds of knowledge are introduce to provide additional information for better disambiguation performance. figure 2 gives the overall architecture of the model. the overall procedure of the k-ned model is a pipeline including feature extraction and semantic matching, where the former is composed of two sub-procedures, taking charge of feature extraction for mention context and candidate entity, respectively. rather than considering only the utterance of the input text, the feature extraction procedure also considers external knowledge for better representation. in details, the feature extraction sub-procedure for mention context is enhanced by conceptual knowledge formalized as a conceptual kg, which augments the representation of the mention context by giving each word a concept embedding; while the sub-procedure for candidate entity is enhanced by factual knowledge formalized as a factual kg, which augments the representation of each candidate entity by giving the entity an entity embedding. the augmented representation is used in the following semantic matching procedure for better prediction. the major difference of the k-ned model is the introduction of external knowledge in the representation learning procedure. for the representation learning procedure, it simply uses the pre-trained word2vec language model to take charge of the conventional utterance representation learning. for the semantic matching procedure, it directly adopt the self-attention matching network based on dssm. given a mention m and a candidate entity e, the word2vec-based module gives two representation vectors, r lm m and r lm e , while the kg-based modules give another two representation vectors, r kg m and r kg e . the concatenation of the four representation vectors is fed into the matching network to obtain the matching degree. based on the utterance of m or e, the word2vec-based representation vector r lm m or r lm e is obtained by averaging the hidden representations for the words or characters in the utterance. we omit the detailed descriptions of the word2vec-based feature extraction and the dssm-based semantic matching owing to space limitations. in the following subsections, we describe in details the computation procedures for the kgbased feature extraction. ## experiments and analysis in this section, we first introduce the experimental dataset, and construction methods of the dataset we published with this paper. then, we present evaluation metrics, the experiments conducted for both the english and the chinese datasets with existing approaches, and we analyze the experimental results in detail. ## #avgnumofentityproperties 9 data annotation method: we annotated the entire short text in the dataset by crowd-sourcing. the same data was repeatedly labeled by three domain experts, then reviewed and released by additional experts. the average precision of annotating entities is about 95.2%. the evaluation method of dataset is as follows: given an input of a short text q, the annotated entities is e @xmath1, e 2 , e 3 , .... by comparing the outputs e q with additional expertsannotated set e @xmath2, e 2 , e 3 , ..., precision p is defined as follows. comparison with previous datasets: as summarized in ## related works many efforts have been devoted to ned in recent years. some methods @xcite @xcite exploit the learning to rank framework (ltr) @xcite to rank the candidate entities, taking advantage of the relationships between all candidates. most commonly used ranking models are the pairwise framework (perceptron @xcite , ranksvm(chingpei lee, 2014)) and the listwise framework (listnet @xcite ). @xcite proposed a named entity linking method based on a probabilistic topic model @xcite , which employs the conceptual topic model to map words and mentioned entities into the same topic space. @xcite used a graphbased collaborative entity linking model. @xcite proposed using random walks for entity linking. some models choosed to rely solely on the context of the links to learn entity representations, such as @xcite , and some methods used a pipeline of existing annotators to filter entity candidates such as @xcite . different from these conventional work, we use multiple sources of information and a deep structured semantic model to achieve better ned performance. many efforts have been devoted to ned for queries, such as @xcite and @xcite . some approaches try to solve ned by making extensive use of deep neural networks @xcite , or by adopting distributed representations of words or entities @xcite . other existing approaches take advantage of global context, which captures the coherence between mapped entities of the related keywords in a document @xcite . in @xcite , the neural network model uses attention mechanism to focus on the contextual entities to be disambiguated. in @xcite , the distributed representation of contexts models the relationships between words and entities or between documents and entities, where the distances between various vectors provides useful information for disambiguation. different from these work where complicated techniques or features are used, we adopt external knowledge including factual and conceptual knowledge graphs for better ned performance. @xcite proposes a method to train entity embedding for entity similarity, but this method relies on a dense knowledge map, we use the text and relationship information of entity to model the similarity between entity and context to improve the effect of ned. there are also previous work using concept or type information to improve ned performance. the models of @xcite @xcite try to map short text to a concept space, and then generate comprehensive concept vectors to represent the short text. @xcite constructs a type ontology and a type classifier to map entities to a closed type ontology. @xcite proposes to modeling context explicitly by entity concept, but we use a complementary way of coarse-grained and fine-grained to dynamically predict the concept according to the context and improve the effect of disambiguation. @xcite studies named entity recognition (ner) and named entity linking (nel) for tweets. unlike these work, our method uses fine-grained entity concepts and predicts concepts more accurately by using an advanced knowledge feature network. ## conclusion we propose a knowledge-enhanced approach to short text entity disambiguation. through bridging and facilitating semantic understanding of the finegrained concept associated to a mentioned entity and entity embedding, the performance of entity disambiguation can be significantly improved. the experimental results demonstrate that our approach outperforms existing sota methods on english and chinese datasets for this task. at the same time, we constructed a large-scale manual-annotated chinese dataset for short text entity disambiguation, which has been released with the paper for use by researchers. as a future direction of research, we plan to explore better conceptualization and semantic understanding methods, and further improve the performance of the short text entity disambiguation task. we intend to continue to update our chinese dataset. in the future, we will use more modern embedding(such as bert) or encoder(such as transformer) to obtain better embedding, and we also plan to conduct experiments to verify the effectiveness of our methods in other tasks related to semantic understanding such as q&a, dialogue, etc.