Named Entity Recognition for Social Media Texts with Semantic Augmentation (2020)

Abstract: Existing approaches for named entity recognition suffer from data sparsity problems when conducted on short and informal texts, especially user-generated social media content. Semantic augmentation is a potential way to alleviate this problem. Given that rich semantic information is implicitly preserved in pre-trained word embeddings, they are potential ideal resources for semantic augmentation. In this paper, we propose a neural-based approach to NER for social media texts where both local (from running text) and augmented semantics are taken into account. In particular, we obtain the augmented semantic information from a large-scale corpus, and propose an attentive semantic augmentation module and a gate module to encode and aggregate such information, respectively. Extensive experiments are performed on three benchmark datasets collected from English and Chinese social media platforms, where the results demonstrate the superiority of our approach to previous studies across all three datasets.

## introduction the increasing popularity of microblogs results in a large amount of user-generated data, in which texts are usually short and informal. how to effectively understand these texts remains a challenging task since the insights are hidden in unstructured forms of social media posts. thus, named entity recognition (ner) is a critical step for detecting proper entities in texts and providing support for downstream natural language processing (nlp) tasks @xcite . however, ner in social media remains a challenging task because (i) it suffers from the data sparsity problem since entities usually represent a small part of proper names, which makes the task hard to be generalized; (ii) social media texts do not follow strict syntactic rules @xcite . to tackle these challenges, previous studies tired to leverage domain information (e.g., existing gazetteer and embeddings trained on large social media text) and external features (e.g., part-of-speech tags) to help with social media ner @xcite . however, these approaches rely on extra efforts to obtain such extra information and suffer from noise in the resulted information. for example, training embeddings for social media domain could bring a lot unusual expressions to the vocabulary. inspired by studies using semantic augmentation (especially from lexical semantics) to improve model performance on many nlp tasks @xcite @xcite , it is also a potential promising solution to solving social media ner. figure 1 shows a typical case. "chris", supposedly tagged with "person" in this example sentence, is tagged as other labels in most cases. therefore, in the predicting process, it is difficult to label "chris" correctly. a sound solution is to augment the semantic space of "chris" through its similar words, such as "jason" and "mike", which can be obtained by existing pre-trained word embeddings from the general domain. in this paper, we propose an effective approach to ner for social media texts with semantic augmentation. in doing so, we augment the semantic space for each token from pre-trained word embedding models, such as glove @xcite and tencent embedding @xcite , and encode semantic information through an attentive semantic augmentation module. then we apply a gate module to weigh the contribution of the augmentation module and context encoding module in the ner process. to further improve ner performance, we also attempt multiple types of pre-trained word embeddings for feature extraction, which has been demonstrated to be effective in previous studies @xcite @xcite @xcite . to evaluate our approach, we conduct experiments on three benchmark datasets, where the results show that our model outperforms the stateof-the-arts with clear advantage across all datasets. ## the proposed model the task of social media ner is conventionally regarded as sequence labeling task, where an input sequence @xmath0, x 2 , • • • , x n with n tokens is annotated with its corresponding ne labels @xmath1, y 2 , • • • , y n in the same length. following this paradigm, we propose a neural model with semantic augmentation for the social media ner. figure 2 shows the architecture of our model, where the backbone model and the semantic augmentation module are illustrated in white and yellow backgrounds, respectively. for each token in the input sentence, we firstly extract the most similar words of the token according to their pre-trained embeddings. then, the augmentation module use an attention mechanism to weight the semantic information carried by the extracted words. afterwards, the weighted semantic information is leveraged to enhance the backbone model through a gate module. in the following text, we firstly introduce the encoding procedure for augmenting semantic information. then, we present the gate module to incorporate augmented information into the backbone model. finally, we elaborate the tagging procedure for ner with the aforementioned enhancement. ## conclusion in this paper, we proposed a neural-based approach to enhance social media ner with semantic augmentation to alleviate data sparsity problem. particularly, an attentive semantic augmentation module is suggested to encode semantic information and a gate module is applied to aggregate such information to tagging process. experiments conducted on three benchmark datasets in english and chinese show that our model outperforms previous studies and achieves the new state-of-the-art result.