Named Entity Popularity Determination using Ensemble Learning (2020)

Abstract: Determining the popularity of a Named Entity after completion of Named Entity Recognition (NER) task finds many applications. This work studies Named Entities of Music and Movie domains and solves the problem considering relevant 11 features. Decision Trees and Random Forests approaches were applied on the dataset and the latter algorithm resulted in acceptable accuracy.

## introduction over the years the use of electronic devices for various tasks and services have become predominant. these days, services such as e-commerce, video and music streaming over the internet are common. thus, it is crucial that these services provide the users with what they require and also give relevant recommendations. moreover, search engines should know what results to show based not only on the query string but also consider other factors like demographic, geographic location, user's search history, etc. identifying entities in a query string is termed as named entity recognition (ner). due to the vast and diverse collection of data, ner sometimes alone may not be sufficient. in such cases name entity popularity detection can be used to prioritize results which may be more rel-evant to the user based on various factors. this is especially used while designing a conversation smart assistant, to provide the user with the best results. for example, if the user asks the virtual assistant to "play wolves" then it has to decide whether to play the song wolves or play the movie wolves without any given context or sentence associated with. in such a scenario information about the user's search history has conventionally been used to resolve the ambiguity and judge what the user really wanted. this process of using data other than the query string to disambiguate between entities with similar names is known as named entity popularity determination (nepd). our objective is to accomplish this task without considering user history. ## problem statement nepd aims at determining the popularity of the named entities. this approach can be extended for use in conversation smart assistants, helping the application to understand user speech, disambiguate entities and give the best search results. nepd involves primarily ## related work several approaches have been put forward to get better results for ner. some of these methods involve the use of neural architectures in addition to bi-lstm methods. @xcite has developed a method based on transition-based parsing and stack-lstms. building on the approach of ensemble learning for ner, multiple approaches have been combined to obtain a better result @xcite . the results of various classifier algorithms were integrated @xcite , resulting in significant strides in ner performance. in the work of @xcite , information extracted from wikipedia is stored in two databases, and the entities are mapped together. after entity mappings, a disambiguation component is used for named entity disambiguation (ned). to improve the performance of ned, in addition to bi-lstm; other models that use gcn and rnn, along with attention have also been experimented upon (cetoli et al., 2018). combining graphs and popularity ranking into a single model is another approach towards ned @xcite , @xcite , where graphs, knowledge bases, are used for entity generation, entity ranking and ned. joint embeddings have also been used for improving the results for ned @xcite . to enhance the results of the ner models, entity popularity models have been proposed @xcite , @xcite . these models use the personal history of the user, create a ranking of entities, and have resulted in recommendations that are more likely and specifically related to the concerned user. the ranking of entities has been improved by increasing the performance of the language model (lm), during automatic speech recognition @xcite . the model used for predicting entity popularity in this paper resulted in improvements of 20% in word error rate. ## dataset and feature extraction the next step in dealing with the sparseness was building the correlation and covariation matrix.the correlation matrix, shown in figure1 helped decide which features to drop, and which features to retain, to get a better clarity of the data we had collected. the initial dataset of entities and the domains to which they are to be classified into movie/music was generated. considering the domains in the problem statement, a list of applicable features along with the source from which the data for that particular feature can/could be extracted was made. this decision was based on the considerations that a person who is active predominantly in one of the two domains, will have a higher value for the features pertaining to that domain. additionally, people who are active in both the domains will have values for all the features but some comparatively higher than the other. the list of features is given in the problem at hand is finding the popularity of a person in the domain and identification of the domain in which the person is more popular if the person has an existence in both. the idea behind the selection of a feature was based on this problem. the popularity of any person in a domain is based on the works of that person, and the accolades the person has received for their work in the particular field. the feature "run time" was included in par-ticular as some of the artists voice for songs in the movies they have acted in. and hence they might have a considerable number of songs under them. this feature will help us to distinguish between a full-fledged singer (music domain) and actors who just give voice for a song in a movie. the data was collected and a .csv file was formed. most of the entities in the given dataset were disambiguous, and hence the data collected was sparse as the entity was inclined to one of the domains. to overcome this problem of sparseness, the dataset was first divided into two-parts: one file containing the entities that were the names of the people and/or music bands, while the other contained the entities that were the names of the artwork (movie and/or song tracks). dividing the single file into two, helped deal with the sparseness a bit as some of the features that were not applicable to that particular category were removed. there were many challenges in handling the artwork file. many artworks pertaining to the same domain had the same name but different artists. a disambiguation had to be done to select the best one of these. consider the song "bad guy". when we tried to collect data regarding it, the result of the search query included 280 songs of the same name in all languages combined. we took the collection of top 10 searches and the data related to each of those songs which shared the common name. the features "genre", "region" and "release date" were dropped from consideration. the decision to drop them was taken based on the fact that "genre" didn't work well with entities belonging to the movie domain. the "release date" isn't applicable to persons but only to the artwork. hence these features were considered as not needed and were dropped. after dropping the not required features from the dataset, the sparseness that still existed was dealt by filling them up with default values which was determined in such a way so as to not change the dynamics of the dataset (i.e., not change the nature of the data). ## methodology flowchart in figure2 shows the systematic approach taken. the data collected from imdb and musicbrianz was analyzed. we faced the problem of excessive sparseness after building our dataset as some fields were not applicable for a particular domain and hence had to be left blank. features like number of movies acted, number of movies directed, number of movies produced would not be applicable for music related entities. similarly features like release region, release year would not be applicable for movie artists. we realized adding mean/median values to remove the sparseness would corrupt the dataset as the missing values were not applicable for the entity. for entities like number of movies acted, number of movies directed, number of movies produced we replaced null values with zero. for the release year entity, we replaced null values with 2015 and release region with usa to avoid ambiguity as we didn't want these values to influence the classifying decision. we replaced the null values in the rating columns for movie artists to four to maintain uniformity. we made sure not to replace any missing attribute's value with an extreme value so as to avoid influencing the classifying decision. we used label encoding to encode the attribute values as decision trees and random forests do not collecting data from imdb and m sicbrain to build the dataset cleansing our dataset and filling missing alues ith appropriate alues to tackle sparseness performing label encoding on the features training our dataset ith random forest var ing the number of trees and test data si e to obser e accurac steps sho ing the approach taken finall choosing the result hich gi es us the highest accurac fine tuning parameters accept string inputs. we used label encoding over one hot encoding because we observed one hot encoding would result in higher data duplication (multiple columns). initially we used decision trees after performing label encoding on the dataset entities. the intuition behind choosing decision trees was that the algorithm generates rules for classification. the input data was not sequential for us to try machine learning algorithms like rnn or lstm. the decision tree was built using the scikit-learn library of python. the algorithm used to build the decision tree was cart, and hence categorical values weren't supported and encoding had to be done. the decision of splitting the node at a level was based on the gini impurity of the features. to improve the accuracy, we used random forest (set of decision trees) to reduce overfitting and give better results for new test data. random forest algorithm creates decision trees on data samples and then gets the prediction from each of them and finally selects the best solution by means of voting. it is an ensemble method which is better than a single decision tree because it reduces the over-fitting by averaging the result and improves the classification accuracy. ## experimental results on analyzing our dataset, we decided that a machine learning approach would work well for the problem in hand. we initially used decision trees and achieved an accuracy of 86%. figure3 represents the decision tree that was built using the cart algorithm. the split of nodes at each level is depicted along with its gini impurity value. algorithm test data (%) training data (%) accuracy (%) decision tree 20 80 86 random forest 20 80 89 random forest 30 70 93 random forest 40 60 94.3 to increase the accuracy of classification and prevent overfitting we decided to use random forest. using the random forest, we obtained an accuracy of 89%. as our dataset was relatively small, increasing the number of trees for random forest did not yield us with better accuracy, so we increased the test data size to have larger data to classify from 20% to 40%. results are tabulated in ## conclusion the problem of named entity popularity determination was to be solved without any contextual data or user history in the domains of music and movies. in virtual assistants, users usually give only named entity without context or a statement associated with it. we built our customized dataset from imdb and musicbrainz. the issue of excessive sparseness was solved by filling the missing values in the dataset with generic relevant values and made sure it wouldn't influence the classifying decision. with decision trees, an accuracy of 86% was achieved. to improve the accuracy and prevent overfitting random forests was used. we finally achieved an accuracy of 94.3%. the approach used here to disambiguate ambiguous entities can be extended to other related domains like tv shows, podcasts and radios by collecting relevant features. this approach can be modelled to disambiguate ambiguous entities between various related domains without contextual information.