Joint P ersian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT (2020)

Abstract: Words are properly segmented in the Persian writing system; in practice, however, these writing rules are often neglected, resulting in single words being written disjointedly and multiple words written without any white spaces between them. This paper addresses the problems of word segmentation and zero-width non-joiner (ZWNJ) recognition in Persian, which we approach jointly as a sequence labeling problem. We achieved a macro-averaged F1-score of 92.40% on a carefully collected corpus of 500 sentences with a high level of difficulty.

## introduction people who have worked with real-world data in persian natural language processing (nlp) are familiar with the persian writing system's problematic properties, the most important of which are word segmentation and zwnj, both consequences of a phenomenon called "orthographic ligature". in persian, white space character (u+0020) is used to segment words in a sentence, and zwnj (u+200c) is used between free morphemes in compound words (or combining forms), and also between one or several free morphemes and one or several bound morphemes to stop joining letters from connecting into a ligature @xcite . although there is a set of rules regarding the use of white space and zwnj set by the regulatory body of the persian language (academy of persian language and literature, 2015), only a few people follow them in writing formal persian, let alone the informal language. some common errors regarding the use of white space and zwnj are as follows: (1) not segmenting words where the last letter of the first word is a non-joiner character (e.g., va āmār[e] marg|omir behtar az xeyli|hāst 1 "and the mortality rate is better than many"); (2) using zwnj instead of white space which could be categorized as (a) excessive use of zwnj (e.g., for separating first and last names) and (b) zwnj key hit mistakenly as in many cell phone keyboards the space key and the zwnj key are adjacent to each other; (3) using space instead of zwnj (e.g. mi konam "i do"); (4) using neither of them (e.g. mikonam "i do"); and many other spontaneous errors. in comparison to european languages, such as english and german, word segmentation in asian languages like japanese, chinese @xcite , and thai @xcite , is more complicated, because space is not specifically used as an orthographic word boundary delimiter @xcite . for instance, in vietnamese, space is not only used to separate words, but it is also applied to separate syllables (that can be considered as a meaningful word or as a part of multi-syllable words) that make up words @xcite . the problem in persian is not similar to that of the abovementioned languages, but quite identical to urdu @xcite @xcite @xcite , as they both are indo-iranian languages and their writing systems are derived from the arabic script. in persian, words are properly segmented in theory, but these rules are not always followed in practice. these kinds of writing system characteristics result in difficulties in text processing, e.g., sequence labeling. according to the previous research, word segmentation can improve the results of other nlp tasks like information retrieval @xcite , machine translation @xcite , information extraction @xcite , dependency parsing @xcite , etc. in this paper, we address the problems of word segmentation and zwnj correction in persian using sequence labeling models and achieve results that are quite promising and could pave the way for an effective solution for real-world situations. to be sure of this, we gathered a corpus of 500 sentences with a high degree of difficulty regarding the correctness of word segmentation and zwnjs, and evaluated the models' performance on it. after reviewing the related work, we discuss the data used for training, validating, and testing the models in §3, the methodology and the experimental settings in §4, and the results in §5. we then conclude the paper and suggest some ideas for future work. ## related work segmentation techniques could be classified into different categories, such as rule-based methods @xcite , hybrid methods @xcite and @xcite use a hidden markov model with some hand-crafted rules), and machine learning and deep learning methods @xcite and @xcite use conditional random fields and support vector machines, and @xcite use a bidirectional recurrent neural network model). in addition to supervised learning (like @xcite , @xcite , etc.), there are studies on semi-supervised @xcite @xcite and unsupervised word segmentation @xcite @xcite . as for the latest studies, we can mention @xcite al. (2020), nguyen (2018), and yan et al. (2020) in persian, there are few studies conducted on preprocessing toolkits, some of which also include modules for space and zwnj correction. as an early work, we can mention @xcite that designed a persian text preprocessing tool called step-1 to tokenize texts, check spellings, and analyze words morphologically. to design the tokenizer, they use dictionary-based and rule-based methods to recognize the correct places of space and zwnj, and report a performance of 86.6%. as another work, @xcite design a toolkit for persian text processing in four lexical, morphological, syntactic, and semantic levels. space and zwnj positions are checked in the first level using dictionary-based and rule-based methods, achieving a precision of 95%. additionally, hazm foot_0 , an open-source preprocessing library, performs zwnj correction using regular expressions and a list of valid stems. parsivar @xcite , the most powerful persian preprocessing tool, applies rule-based and statistical methods to determine the correct places of zwnj and space, respectively. they trained a naive bayes model on the 10 million word bijankhan corpus @xcite with the iob tagging scheme to find word boundaries, achieving an f 1 -score of 96.5%. as for the latest work, @xcite uses an n-gram language model and a rule-based method to correct space and zwnj between compound words, respectively. they report an f 1 -score of 81.94% for space correction. ## data the 10 million word bijankhan corpus @xcite was used in this research, which is a cleanly tokenized corpus with fine-grained part-of-speech tags (which were not used here). this corpus contains 10,437,194 words or 38,971,131 characters. we chose this corpus since (a) their word segmentation is clean and their approach is practical, (b) the size of the corpus is enormous (∼39m characters), and (c) the corpus comprises many different topics, including news articles, literary prose and poems, informal dialogues, etc. all white spaces in a single token were converted to zwnjs and the characters in the corpus were normalized as well. then, the corpus was split into three parts: the first 10% for testing, the second 10% for validation, and the rest for training. we also collected a test corpus of 500 sentences from twitter, news broadcasting websites, and discussion forums to have real-world data with real white space and zwnj errors. this data was deliberately meant to be a difficult test corpus with numerous extreme cases (and also easy and completely correct ones). this corpus contains 16,574 words or 93,355 characters. ## methodology we see the word segmentation correction and the zwnj recognition tasks as one problem and train a single model to perform these two tasks jointly. we approached the task as a sequence labeling problem, i.e., mapping each character to a tag space of size 3. the tag is 0 when there is no white space or zwnj, 1 when there is white space, and 2 when there is zwnj after the corresponding character. two types of models were trained for this purpose: the input texts were fed into the model in two different settings: we also experimented with parsivar tool, introduced in §2, to rectify white spaces and zwnjs using its normalizer (with the statistical space correction argument set to true) and spellcheck modules. as the latter module might add or remove characters other than white space and zwnj, the differences between the two strings were found using python's difflib library, and placeholder characters were added where needed to make the strings the same length and make their comparison possible. this work's performance measures are precision, recall, and f 1 -score for each class and macro-averaged f 1 -score of all of them. ## results and analysis the results of the abovementioned methods on the test set of bijankhan corpus and the 500 sentence corpus are shown in the errors in bert a and bert b are more or less similar and can be categorized into the following groups: (a) zwnj before an intrusive y between a vowel and an ezafe (e.g., xāney[e] "[the] house of"), which is simply because this writing style is not used in bijankhan corpus (and is also fixable as words with ezafe are labeled as gen in bijankhan corpus); (b) underscores and hash characters in hashtags, which is again because the model has not seen it in the training data; (c) out of vocabulary words, such as tu'iter "twitter" and dāmāš "damash (a soccer team name)"; (d) digits occasionally sticking to the word before or after them; (e) informal words, such as the clitic e "is", which is the short form of ast "is"; (f) words or phrases with different syntactic roles, such as xaridāri|šode "bought (adjective)" and xaridāri šode "bought (verb)"; (g) typos and uncommon spellings. the confusion matrix also reveals that zwnj is mostly mistaken with the 0 class, and not with the 1 class, which results in joiner characters connecting into a ligature and make the text difficult to read. there are also some cases which can not strictly be counted as errors, but different writing styles. unfortunately, our performance measure does not account for these, more or less, common cases. some examples are pul|dār "rich", doruq|gu "liar", and šaffāf|sāzi "clarification". all in all, adding more training samples with the abovementioned features would most probably solve many of the mentioned error groups. ## conclusion and future work in this paper, we experimented with different methods to tackle word segmentation correction and zerowidth non-joiner recognition problems in persian. the results on our collected data show bert outperforming other methods, and the error analysis indicates that it would be relatively easy to increase the performance and pave the way for a practical and effective preprocessing tool. future work could focus on collecting more informal persian data, more diverse topics, and more modern registers of the language, to further improve this work's results. there are also some techniques in the work mentioned in §2, e.g., multi-task learning (say, for the model to learn the difference between joiner and non-joiner characters as an auxiliary task), that could be used in persian as well. covering different writing styles in the training data would also be helpful, e.g., changing some of the ending e[ye] clitics in bijankhan corpus, when the word is tagged as gen, to ey[e], as discussed in §5.