Deep Neural Model for M anipuri Multiword Named Entity Recognition with Unsupervised Cluster Feature (2020)

Abstract: The recognition task of Multi-Word Named Entities (MNEs) in itself is a challenging task when the language is inflectional and agglutinative. Having breakthrough NLP researches with deep neural network and language modelling techniques, the applicability of such techniques/algorithms for Indian language like Manipuri remains unanswered. In this paper an attempt to recognize Manipuri MNE is performed using a Long Short Term Memory (LSTM) recurrent neural network model in conjunction with Part Of Speech (POS) embeddings. To further improve the classification accuracy, word cluster information using K-means clustering approach is added as a feature embedding. The cluster information is generated using a Skip-gram based words vector that contains the semantic and syntactic information of each word. The model so proposed does not use extensive language morphological features to elevate its accuracy. Finally the model’s performance is compared with the other machine learning based Manipuri MNE models.

## introduction multi word named entity (mne) is a part of multiword expression (mwe) which is an ordered group of words that can exist independently and carries different meaning as opposed to its constituent word @xcite . accurate recognition of such mne plays a vital role in various nlp tasks such as pos tagging @xcite , chunking @xcite , ner classification @xcite . manipuri, being a highly inflectional language where affixes define the nature of the words @xcite , machine recognition of mne presents a challenging task for nlp researchers. in manipuri, the majority of researches on mne classification are done using machine learning approaches such as crf @xcite @xcite and svm @xcite . these researches use extensive morphological features to obtain accurate recognition of the mne. such morphological features include affixes, context words, digit features etc. for an agglutinative and low-resource language, the inflectional nature amounts to the large out of vocabulary (oov) words, thus making any sequence labelling task or morphological feature creation tasks difficult. word embedding is a distributed representation of text in low dimensional real valued vectors and are known to contain semantic or syntactic information and has shown to be an effective feature for many natural language classification tasks of english language @xcite . word embedding has also been extremely useful for chinese language processing @xcite , japanese language processing @xcite and for some indian languages processing @xcite . in manipuri, the effectiveness of word embeddings to any nlp tasks, till this date remains unanswered. the general attempt to work in manipuri nlp task using embedding follows the idea used in english i.e. to learn the embedding of a word from the context. unlike english, manipuri words are a composite of complicated structure with several affixes producing oov words. regardless of its context, affixes to a word plays a major role in defining the semantic meaning @xcite . in this paper, a manipuri mne classification task is attempted using a bi-directional long short term memory (bi-lstm) with skip-gram word embeddings. the usage of extensive morphological features for deep neural network is avoided, as these features would require vector representations which may lead to considerable large input layer to update. to segregate the manipuri mne words based on the semantics, a k-means algorithm based cluster information of each word is added as a feature to the lstm model. ## related works notable amount of manipuri mwe classification research have been done using traditional machine learning approach such as crf (conditional random field) and svm (support vector machine). @xcite reported an improvement of manipuri mwe identification using crf and reduplicated multi-word expressions (rmwe). the mwe identification model uses a dictionary and rule-based approach to first detect the complete, mimic, partial, double and echo rmwes and then prepare a model training set using features such as affix information, pos information word frequency and word length. the crf model attained a performance f-score measure of 72.24%. @xcite conducted a research for identification of rmwe using crf with various features. the features were stem words, number of affixes, stemmed affixes, pos of surrounding words, surrounding words, length of the word, word frequency and digit features. with the features, the crf classification model predicted with recall, precision and an f-score measure of 92.91%, 91.90% and 92.40% respectively. nongmeikapam and bandyopadhyay (2011 ) conducted a research on crf based mwe identification with genetic algorithm (ga) based feature selection method. in genetic algorithm, the mwe features are represented as genes in a chromosome which are binary valued (1 or 0). selection of a feature is done when the gene value is 1. random crossover is performed to select the best possible combination of features. to avoid chromosome repetition, random mutation is performed. as a fitness function, three fold cross validation technique is used. using the technique, the best features were surrounding words, affixes, surrounding pos, word length and word frequency. the crf model with the aforementioned ga based selected features, attained an increase in f-score by 2.91% as compared to baseline crf model without ga based feature selection. overall the model performed with f-@xmath0.74%, @xmath1.84% and re-@xmath2.08%. singh and bandyopadhyay (2010) proposed a web based manipuri corpus for multiword ner and rmwe identification model using svm. for classification purpose, the best features were selected for svm classifier. the features were context word, word affixes, mne and rmwe information, digit features, infrequent words, word length and pos information. the model was trained over 1235 sentences with 28629 words and predicted with an f-score measure of 93.96% for mne and 94.07% for rmwe. ## system design the manipuri mne classification is shown in figure 1 . being a low resource language with limited information on morphological features, extraction of necessary information for a particular word is crucial as these information will help the model in accurate mne classification. in this research, two models have been implemented for manipuri mne classification. they are 1. a baseline model with bi-lstm deep neural networks with word embedding using skip-gram with default pos embedding. the use of word embedding, feature embedding and bi-lstm are described in the following subsections. ## results and evaluation the main aim of this research is to focus on the addition of features and their effects on the final result of mne classification as compared to the base models. we have used the precision, recall and f-score evaluation metrics to measure the performance of the models. ## conclusion finally, a deep neural model has been reported for manipuri mne recognition using bi-lstm and word embeddings in this paper. the training data consists of pos tagged words of 76526 with 12421 number of mne. the bi-lstm model make use of embedding matrix generated using the skip-gram for manipuri words. word clusters information generated using the skip-gram word vectors, is used as an additional feature. the use of k-means cluster information is to create a semantically meaningful mne clusters. it has been reported that the cluster information generated from the skip-gram embedding word vectors carries semantic values and has been able to supplement the mne recognition task resulting in an increase in the average f-score measure by 86% for mne, as compared to the baseline model. the proposed skip-gram model correctly represented the similarity of mne words with about 89% accuracy. overall the model achieved an average f-score measure of 94.98%.