Named Entity Recognition Only from Word Embeddings (2020)

Abstract: Deep neural network models have helped named entity recognition achieve amazing performance without handcrafting features. However, existing systems require large amounts of human annotated training data. Efforts have been made to replace human annotations with external knowledge (e.g., NE dictionary, part-of-speech tags), while it is another challenge to obtain such effective resources. In this work, we propose a fully unsupervised NE recognition model which only needs to take informative clues from pre-trained word embeddings.We first apply Gaussian Hidden Markov Model and Deep Autoencoding Gaussian Mixture Model on word embeddings for entity span detection and type prediction, and then further design an instance selector based on reinforcement learning to distinguish positive sentences from noisy sentences and then refine these coarse-grained annotations through neural networks. Extensive experiments on two CoNLL benchmark NER datasets (CoNLL-2003 English dataset and CoNLL-2002 Spanish dataset) demonstrate that our proposed light NE recognition model achieves remarkable performance without using any annotated lexicon or corpus.

## introduction named entity (ne) recognition is a major natural language processing task that intends to identify words or phrases that contain the names of per (person), org (organization), loc (location), etc. recent advances in deep neural models allow us to build reliable ne recognition systems @xcite @xcite @xcite . however, these existing methods require large amounts of manually annotated data for training supervised models. there have been efforts to deal with the lack of annotation data in ne recognition, @xcite train a weak supervision model and use label propagation methods to identify more entities of each type; @xcite employ deep active learning to efficiently select the set of samples for labeling, thus greatly reduce the annotation budget; @xcite @xcite @xcite use partially annotated data or external resources such as ne dictionary, knowledge base, pos tags as a replacement of hand-labeled data to train distant supervision systems. however, these methods still have certain requirements for annotation resources. unsupervised models have achieved excellent results in the fields of part-of-speech induction @xcite , dependency parsing @xcite , etc. whereas the development of unsupervised ne recognition is still kept unsatisfactory. @xcite design a knowledge-augmented language model for unsupervised ne recognition, they perform ne recognition by controlling whether a particular word is modeled as a general word or as a reference to an entity in the training of language models. however, their model still requires type-specific entity vocabularies for computing the type probabilities and the probability of the word under given type. early unsupervised ne systems relied on labeled seeds and discrete features @xcite , open web text @xcite , shallow syntactic knowledge @xcite , etc. word embeddings learned from unlabeled text provide representation with rich syntax and semantics and have shown to be valuable as features in unsupervised learning prob-lems @xcite . in this work, we propose an ne recognition model with word embeddings as the unique feature source. we separate the entity span detection and entity type prediction into two steps. we first use gaussian hidden markov model (gaussian-hmm) to learn the latent markov process among ne labels with the iob tagging scheme and then feed the candidate entity mentions to a deep autoencoding gaussian mixture model (dagmm) @xcite for their entity types. we further apply bilstm and an instance selector based on reinforcement learning @xcite to refine annotated data. different from existing distant supervision systems @xcite @xcite , which generate labeled data from ne lexicons or knowledge base which are still from human annotation, our model may be further enhanced by automatically labeling data with gaussian-hmm and dagmm. the contribution of this paper is that we propose a fully unsupervised ne recognition model that depends on no external resources or annotation data other than word embeddings. the empirical results show that our model achieves remarkable results on @xcite spanish benchmark datasets. the rest of this paper is organized as follows. the next section introduces our proposed basic model in detail. section 3 further gives a refinement model. experimental results are reported in section 4, followed by related work in section 5. the last section concludes this paper. ## model as shown in figure 1 , the first layer of the model is a two-class clustering layer, which initializes all the words in the sentences with 0 and 1 tags, where 0 and 1 represent non-ne and ne, respectively. the second layer is a gaussian-hmm used to generate the boundaries of an entity mention with iob tagging (inside, outside and beginning). the representation of each candidate entity span is further fed into a deep autoencoding gaussian mixture model (dagmm) to identify the entity types. ## refinement the annotations obtained from the above procedure are noisy, we apply reinforcement learning (rl) @xcite to distinguish positive sentences from noisy sentences and refine these coarse-grained annotations. the rl model has two modules: an ne tagger and an instance selector. ## experiments we conduct experiments foot_1 on two standard ner datasets: conll 2003 english dataset @xcite and conll 2002 spanish dataset @xcite ) that consist of news articles. these datasets contain four entity types: loc (location), misc (miscellaneous), org (organization), and per (person). we adopt the standard data splitting and use the micro-averaged f 1 score as the evaluation metric. ## related work deep neural network models have helped peoples released from handcrafted features in a wide range of nlp tasks @xcite @xcite @xcite . lstm-crf @xcite is the most stateof-the-art model for ne recognition. in order to reduce the requirements of training corpus, distant supervised models @xcite @xcite @xcite have been applied to ne recognition. recently, @xcite proposed a knowledge-augmented language model, which trains language models and at the same time compute the probability of the next word being different entity types according to the context given type-specific entity/general vocabularies. unlike these existing approaches, our study focuses on unsupervised ne recognition learning without any extra resources. noisy data is another important factor affecting the neural network models, reinforcement learning has been applied to many tasks, @xcite use reinforcement learning for relation classification from noisy data. @xcite show how to apply reinforcement learning in ne recognition systems by using instance selectors, which can tell high-quality training sentences from noisy data. their work inspires us to use reinforcement leaning after obtaining coarse annotated data from gaussian-hmm. ## conclusion this paper presents an ne recognition model with only pre-trained word embeddings and achieves remarkable results on conll 2003 english and conll 2002 spanish benchmark datasets. the proposed approach yields, to the best of our knowledge, first fully unsupervised ne recognition work on these two benchmark datasets without any annotation data or extra knowledge base.