Comparing Named-Entity Recognizers in a Targeted Domain: Handcrafted Rules vs Machine Learning (2016)

Abstract: Comparing Named-Entity Recognizers in a Targeted Domain : Handcrafted Rules vs. Machine Learning Named-Entity Recognition concerns the classification of textual objects in a predefined set of categories such as persons, organizations, and localizations. While Named-Entity Recognition is well studied since 20 years, the application to specialized domains still poses challenges for current systems. We developed a rule-based system and two machine learning approaches to tackle the same task : recognition of product names, brand names, etc., in the domain of Cosmetics, for French. Our systems can thus be compared under ideal conditions. In this paper, we introduce both systems and we compare them.

## introduction the goal of named-entity recognition (ner) is to classify textual objects in a predefined set of categories like for example persons, organizations and localizations. the evaluation of ner tools in campaigns such as muc, @xcite , suggests that ner is a solved problem. however, as highlighted by @xcite , the set of named entity types is very limited in such campaigns (and annotated corpora are also limited) and it is important to highlight the difficulties that ner systems face when dealing with entities in a fine-grained way @xcite . in particular, with the development of e-commerce and personalized marketing, new brands, product names, and other relative entities, appear daily ; being able to detect such entities becomes crucial in several applications. for example, on-line market-places (amazon, ebay) need to extract features about products provided in free text from different sellers. while ner is widely studied, there are few works dedicated to the recognition of name of products and brands @xcite . in this work, we decided to focus on the recognition of fined-grained entities, namely products, name of ranges, brands, divisions, and groups in the domain of cosmetics. a major difficulty comes from the fact that in the cosmetic's domain there is a high level of ambiguity. as indicated by @xcite , in his study about perfume onomastic, ambiguity is even stronger in the cosmetic domain where any word could be considered as a name of perfume. for instance, perfumes can take form of a number (n o 5), a date (1881), an address (24 faubourg), a sentence (la vie est belle), or a pronoun (elle). such a large spectrum for product names that can turn any common noun phrase into a cosmetics entity makes the task of building specific lexical resources very difficult. early, works on ner showed that large coverage specialized lexicons constitute the foundation of any good ner system @xcite . the lack of resources related to cosmetics makes hard to develop systems that can achieve high performances. hence, recent works focused on symbolic approaches based on hand-crafted linguistic rules which alleviate partially the scarcity problem of annotated data @xcite . on the other hand, in the framework of machine learning, ner is casted as a sequence labeling problem and is solved by structured methods. for example, conditional random fields (crfs), are well-known to capture high-level semantics and also can cope with grammatically degraded text, like in short texts, but are subject to the existence of a significant volume of annotated data @xcite . then, which approach should we adopt when dealing with named entity recognition in a specialized domain ? in this work we perform an empirical comparison between our rule-based system and machine learning approaches, tailored to the cosmetics domain for french language. the three systems use the same ensemble of basic linguistic features in order to obtain comparable systems and perform a fair comparison. we study the behavior of the systems in two different types of resources : 1) a well written corpus of journal articles, and 2) a corpus from a blog. in the following section (section 2) we introduce the data used to develop and to evaluate our systems (described in section 3). finally, we compare the systems and discuss the results (section 4). ## data for the evaluation of the different systems two datasets were constructed. the first dataset, called mag, consists of 99 articles (published between 6 of february 2012 and 30 of april 2012) from four french magazines on the domain of cosmetics : beauté infos, féminin pratique, cosmétique hebdo and cosmétique mag. it contains 1,944 sentences. the second dataset, called blog, contains 118 consecutive articles (published between 14 of june 2014 and 19 of november 2014) from the french blog http://www.justsublime.fr . it contains 2,613 sentences. ## methods and setup our objective consists in recognizing 5 types of named entity : products, names of ranges, brands, divisions, and groups in the domain of cosmetics. we compare one rule-based method relying on a syntactic analysis with two state-of-the-art machine learning approaches which we briefly describe in the following sections. ## results regarding mag, it appears that the rule-based system obtains the best performance regardless the type of entities with f1 between 74.8 and 90.7 (standard deviation : 0.07). both machine learning approaches obtain lower results. we observe that the crf has a large variation in its performances : f1 goes from 20.9 to 81.9 (standard deviation : 0.27). similarly, l2s results vary from 45.7 to 84.8 (standard deviation : 0.18). l2s and crf gets a good precision but with a very low recall for products and ranges. both learning methods outperform renco for divisions that are under-represented in our corpus. concerning blog, all the systems obtain lower performances with this dataset certainly due to the lower quality in terms of grammar and structure implying a negative impact on the syntactic analysis. l2s obtains better results than renco and crf regarding brand names and ranges of product : respectively @xmath0.80 and 0.76. however, renco obtains the best results concerning recognition of product names : @xmath1.68. renco, l2s, and crf has a similar standard deviation (0.08) which can be explain by the fact that nor divisions neither groups were quantitatively representative in blog. both l2s and crf obtain better or comparable precision in the case of products with that of renco but at the expense of lower recall. concluding, renco showed a stable behavior in both corpora with lower performance in the blog dataset where l2s and crf exhibit a more robust behavior. note that the development set of renco consists only of journalistic articles. ## conclusion and further work in this work we presented an empirical comparison of a symbolic approach and machine-learning approaches for named entity recognition in the cosmetics domain. the different systems were compared on two datasets (journalistic articles and blog posts). the rule-based system outperformed both learning approaches in the journalistic corpus while the latter exhibited a more robust behavior in the blog corpus which contains noisy text. for future work, we envisage to enhance the generation of features for the machine learning approaches by a) using dense distributed representations @xcite and b) transform the rules of renco to features. we also intend to introduce an approach for learning automatically the weights of the rules inside lrb towards a hybrid system.