#Turki$h T weets: A Benchmark Dataset for T urkish Text Correction (2020)

Abstract: #Turki$hTweets is a benchmark dataset for the task of correcting the user misspellings, with the purpose of introducing the first public Turkish dataset in this area. #Turki$hTweets provides correct/incorrect word annotations with a detailed misspelling category formulation based on the real user data. We evaluated four state-of-the-art approaches on our dataset to present a preliminary analysis for the sake of reproducibility.

## introduction the extensive use of social media platforms such as twitter, facebook, forums, and blogs has created a new form of written language, which is full of intentional and unintentional misspellings as well as newly-emerged slang words and abbreviations. this new type of language poses significant challenges for various natural language processing (nlp) tasks, mostly requiring properly written textual content for analysis. therefore, text normalization, i.e., transforming non-standard words into their standard forms and spelling correction, i.e., correcting unintentional spelling errors, have become indispensable pre-processing stages. the pre-processing phase is known to boost the model performance for the various nlp applications, including but not limited to pos tagging, sentiment classification and search 1 . although correcting misspelling errors is crucial for nlp applications, it is generally not straightforward and even challenging for morphologically rich languages. there exist many different surface forms of a single word in highly agglutinative languages. specifically for turkish, suffixes should also comply with the vowels and the last letter of the word. this leads to many different variations of a single word, thereby increasing the possibility of misspelling errors. previous studies adapted for english do not fit and thus, there is the need for resources tailored particularly for these languages. currently, there is no publicly available dataset in this area that can be used for model evaluation with reproducible results. in the online platforms, there exist not only intentionally misspelled words but also unintentional spelling errors, both constituting out-of-vocabulary (oov) words in the textual content. intentional user misspellings are quite frequent, particularly in online media and these errors highly vary depending on the intention of use. therefore, a more fine-grained analysis of the error types is required by categorizing the errors and further including them in the evaluation dataset for a proper model evaluation. in this way, a particular model can be assessed whether it provides generalizable results for the corresponding language. based on these, we introduce a new turkish dataset by categorizing, annotating and correcting the distinct misspelling types in text. moreover, we make a fine-grained evaluation of the selected stateof-the-art approaches in literature for reproducibility purposes. to the best of our knowledge, this is the first attempt that introduces a public dataset including the detailed misspelling category formulation with the purpose of providing a reproducible evaluation results on the existing approaches. we hope that the fine-grained analysis of selected models in this work serves as an exemplary usage of the dataset. most similar work to ours is eryigit and @xcite in which authors define rules for correcting the misspelling errors present in social media content. the authors compare the proposed model with the selected stateof-the-art approaches on their dataset which is not publicly available to researchers. hence, we created a benchmark dataset by randomly selecting and annotating turkish tweets from a public dataset of 20m tweets 2 . our contributions in this work are as follows. the paper is structured as follows. in section 2, we give related work. in section 3, we provide the details about data preparation & analysis. in section 4, we show the evaluation results. finally, in section 5 we conclude the paper. ## related work research studies in spelling correction of turkish text date back to 1990s. however, each work has carried out its own evaluation data generation process and none of these datasets are publicly available. in early spelling correction studies, synthetic datasets were used for evaluation. @xcite used words collected from turkish corpus that are perturbed such that the words and their correct forms are 1, 2 or 3 edit distances apart. @xcite , gupta (2020) and büyük (2020) created a synthetic dataset composed of misspelled words with 1 edit distance. büyük (2020) also used a nonpublic dataset manually annotated by @xcite for a better comparison. there are also other works that used real datasets. @xcite evaluated their model on incorrect words in turkish text which are mostly 1 edit distance apart. @xcite manually annotated words from social media text excluding the intentional mistakes such as words without vowels; they separated the task of text normalization and correction of unintentionally misspelled words. @xcite used an open-source morphological analyzer to extract incorrect words from boun corpus @xcite which is composed of newspaper and website textual content. with the rise of social media, new text style has emerged: micro-blogging. those text sources have 2 https://www.kemik.yildiz.edu.tr/data/file/20milyontweet.rar their own jargon including the intentional and unintentional misspellings. torunoglu-selamet and eryigit (2014)'s work focused on text normalization in twitter. they manually aligned 1200 tweets in which some of the tokens are in one-to-many alignment. researchers used this dataset for evaluation of their proposed rule-based model. c ¸olakoglu et al. (2019) used the same dataset, except this, they manually annotated another twitter dataset for model evaluation. nonetheless, the dataset introduced in @xcite is not open to the research community, it can only be obtained upon request under some restricted license constraints. in this work, we propose a new benchmark dataset composing of real turkish tweets with misspelling annotations for different types of oov words. ## data preparation we used a public dataset of 20m turkish tweets written by real users to create the benchmark dataset. first, we applied some pre-processing steps such as cleaning up the lines with metainformation like timestamps, urls, usernames, etc. to provide one sentence per line format. after that, we had 23m sentences. then, out of these 23m sentences, 2000 sentences, including at least one out-of-vocabulary word were randomly selected. to check if there is an oov word in a sentence, we used trmorph (c ¸agrı @xcite , an opensource turkish morphological analyzer. we first tokenized every sentence on each line, using the trmorph's tokenizer and sent each token to the trmorph for morphological analysis. if the tr-morph achieved to provide an analysis for a given token, then the token was assumed to be a correct turkish word, i.e., in-vocabulary (iv) word for the rest of the paper, otherwise incorrect as referred to as oov. in this way, we guaranteed at least one oov word in each sentence obeying one-to-one token alignment. the data statistics and all the details about the error annotation & correction process are provided in section 3.1, 3.2 and 3.3. ## conclusion we introduced a benchmark dataset for turkish text correction by annotating the real users' misspellings in turkish tweets. we categorized the error types and corrected them accordingly. the dataset can be used in various nlp applications, especially for the social media platforms. the error category formulation can also be used for other tasks like query correction in search, which highly affects the search performance. we hope that our work will be a valuable contribution to the turkish research community, being the first work proposing a benchmark dataset with a fine-grained and fair comparative evaluation of some of the state-of-the-art models. as future work, we plan to analyze existing models' deficiencies elaborately and establish new models performing better on our benchmark dataset for distinct error types.