TR - SEQ : Named Entity Recognition Dataset for T urkish Search Engine Queries (2021)

Abstract: Recognizing named entities in short search engine queries is a difficult task due to their weaker contextual information compared to long sentences. Standard named entity recognition (NER) systems that are trained on grammatically correct and long sentences fail to perform well on such queries. In this study, we share our efforts towards creating a cleaned and labeled dataset of real Turkish search engine queries (TR-SEQ) and introduce an extended label set to satisfy the search engine needs. A NER system is trained by applying the state-of-the-art deep learning method BERT to the collected data and its high performance on search engine queries is reported. Moreover, we compare our results with the state-of-the-art Turkish NER systems.

## introduction named entity recognition (ner) aims to identify the predetermined entity categories @xcite accurately in a given text. ner studies which is one of the most widely studied subjects in the field of natural language processing (nlp), has been started for english and spread over a range of languages including turkish @xcite . initial ner studies for turkish used small labeled datasets that contain grammatically correct and typo-free sentences. since manual labeling is a time-consuming and a costly process, gathering larger datasets and manually labeling them has not been feasible. with the increase of internet use, correctly classifying named entities has become a crucial need especially for social media entries and search engine (se) queries. in search engines, identifying entities such as famous persons, places or dates in user queries is crucial in order to accurately determine the information to be displayed to the user and response with the right information. for example, if the user is searching for a famous person, se should return the wikipedia information about the person. moreover if this famous person is an artist, the identified named entity assists in conveying the details of his/her movie/album/artwork details. similarly, it is important to determine the location in the query for which we want to know the weather. the state-of-the-art turkish ner tools fail to perform well on search engine queries due to two main reasons. the first reason is that most of the ner tools that have been developed for turkish so far generally carry out categorization studies with three classes (person, organization and location). naturally, a three-class model cannot detect all the named entities in a se query that is required to create a successful se response. for this reason, it has become a necessity to determine the entity types specific to the ses and to make a special development for these types. the second reason is that ner tools are dependent on the texts and domains they are trained on. ner tools, which are usually trained in canonical and grammatically correct long sentences, do not succeed in short se queries that contain abbreviations and spelling errors and mostly consist of two or three words. as the short inputs have poor contextual knowledge compared to long sentences and often include misspellings, existing ner tools are observed to perform poorly on these type of texts and the success rates are far below the desired levels. although the first attempts for turkish ner systems are dominated by rule-based systems @xcite , crf-based models @xcite @xcite came to the fore in the forthcoming studies. in all of these studies, a relatively small dataset consisting of about 30k sentences, clean in terms of spelling and grammar, is used @xcite and models are based on three classes (person, location, organization). recently, models using deep learning methods @xcite , especially bidirectional long short-term memory (blstm), became more popular @xcite . even though the accuracies of these ner studies are high in their test sets, the performance drop is dramatical on social media (e.g., twitter) and se queries @xcite . for this reason, studies are carried out to collect dedicated data for social media texts @xcite . the most recent studies in turkish ner field utilize transformers and bert based models @xcite @xcite . in this paper, we focus on the recognition of a selected group of named entities for turkish search engine queries. in the study, yaani foot_0 search engine entries are collected, the categories for the se needs are determined and a labeling study is carried out using an in-house annotation tool. the data consisting of a total of 100k se queries is labeled by more than one person and a verification study is carried out with the obtained outputs. in addition, a high-performance ner tool is developed by training the transformers-based bert @xcite model, which is one of the most successful approaches and widely used in current nlp studies, with our labeled dataset, and an accuracy of 90.41% is achieved. foot_1 ## dataset creation process for data-driven ner systems to perform well on real world scenarios, it is of crucial importance to have training data that has a similar distribution with the real world data. in order to satisfy this constraint, we carried out an extensive data collection and labeling process for a ner system specific to se queries. we collected a dataset of 100k queries that are submitted to yaani search engine for a specific period of time without having a constraint related to short/long queries. next, we cleaned the malicious content and normalized the spelling errors manually. finally, we removed the duplicates which resulted in a dataset with 97,428 queries. ## experiments and results using this new dataset, we trained a turkish ner system and demonstrated the improved ner performance of using short search queries during training. our turkish ner system is trained by fine-tuning the berturk @xcite its training corpus has a size of 35gb and 44b tokens with a vocabulary size of 128k. for finetuning the berturk model for ne classification, max seq length is set to 128, train batch size and eval batch size are set to 32, learning rate is set to 2e -5 and the model is trained for 5 epochs. in order to evaluate the performance of the ner system, the dataset is split into three: train, validation and test. for the test set, 1% of the data is selected randomly from the whole dataset by preserving the distribution of named entity types. similarly, 1% is randomly selected as validation data and the remaining 98% is used for training. using 1% of the whole dataset for testing purposes resulted about 1000 test queries and all seven classes are included proportionally to their distribution in the dataset. in addition to the bert model, we have also experimented with a pre-trained version of elec-tra @xcite model (also publicly available within the berturk study), which allowed us to compare the performance of the collected dataset on different deep architectures. publicly available turkish ner models and other studies in the literature usually take three named entity types (person, organization and location) into account. the effort towards increasing the number of named entity types is very limited for turkish and to the best of our knowledge there is no other study that focuses on the named entity types discussed in this paper. therefore, comparing the performances of our system with previous studies on the same test set is not immediately clear. for a fair comparison, we removed the remain-ing labels from our test set by keeping three base named entity types (person, organization and location). since our model is based on seven named entity types, we filtered out the output labels that belong to other types (by masking as other) and generated the final results. we compare our results with two recent turkish ner studies: i) the ner model of "turkish-bert-nlp-pipeline" (yıldırım, 2019) that is known for its high performance on well-formed turkish texts and ii) recent neural sequence tagging models discussed in @xcite . as depicted in ## conclusion this study presents a new turkish ner dataset which is created specifically for short word sequences that comprises a large portion of the search engine queries. our aim is to improve the ner performance of the models that are trained using grammatically correct sentences and fail to perform well on search engine queries. the dataset is created by cleaning and labeling 100k search queries and provides a rich resource for turkish ner studies. in addition, our transformer based ner system presents useful baseline accuracies for future studies. our next step is to train a bert-based language model from scratch by using search engine queries and replace the pre-trained bert model provided by berturk. this will allow us to develop a ner system specific to short queries. furthermore, creating a knowledge graph by utilizing the named entity labels will be an important step in enhancing overall search engine performance.