P ersian Ezafe Recognition Using Transformers and Its Role in Part-Of-Speech Tagging (2020)

Abstract: Ezafe is a grammatical particle in some Iranian languages that links two words together. Regardless of the important information it conveys, it is almost always not indicated in Persian script, resulting in mistakes in reading complex sentences and errors in natural language processing tasks. In this paper, we experiment with different machine learning methods to achieve state-of-the-art results in the task of ezafe recognition. Transformer-based methods, BERT and XLMRoBERTa, achieve the best results, the latter achieving 2.68% F1-score more than the previous state-of-the-art. We, moreover, use ezafe information to improve Persian part-of-speech tagging results and show that such information will not be useful to transformer-based methods and explain why that might be the case.

## introduction persian ezafe is an unstressed morpheme that appears on the end of the words, as -e after consonants and as -ye foot_0 after vowels. this syntactic phenomenon links a head noun, head pronoun, head adjective, head preposition, or head adverb to their modifiers in a constituent called 'ezafe construction' @xcite . whether a word in a sentence receives or does not receive ezafe might affect that sentence's semantic and syntactic structures, as demonstrated in examples 1a and 1b in figure 1 . there are some constructions in english that can be translated by ezafe construction in persian. for instance, english 'of' has the same role as persian ezafe to show the part-whole relation, the relationship of possession, or ''s' construction, and possessive pronouns followed by nouns showing genitive cases are mirrored by persian ezafe @xcite . this affix is always pronounced but almost always not written, which results in a high degree of ambiguity in reading and understanding persian texts. it is hence considered as one of the most interesting issues in persian linguistics, and it has been discussed in details from phonological aspects @xcite , morphological aspects @xcite and @xcite , and syntactic aspects @xcite @xcite @xcite . nearly 22% of the persian words have ezafe @xcite , which shows the prevalence of this marker. moreover, this construction also appears in other languages such as hawramani @xcite , zazaki @xcite , kurdish @xcite etc. ezafe construction is also similar to idafa construction in arabic and construct state in hebrew @xcite and zulu @xcite . ezafe recognition is the task of automatically labeling the words ending with ezafe, which is crucial for some tasks such as speech synthesis @xcite , as ezafe is always pronounced, but rarely written. furthermore, as recognizing the positions of this marker in sentences helps determine phrase boundaries, it highly facilitates other natural language processing (nlp) tasks, such as tokenization @xcite , syntactic parsing @xcite , part-of-speech (pos) tagging @xcite , and machine translation @xcite . in this paper, we experiment with different methods to achieve state-of-the-art results in the task of ezafe recognition. we then use the best of these methods to improve the results for the task of pos tagging. after establishing a baseline for this task, we provide the ezafe information to the pos tag-(1) a. darpey-e ettefāqāt-e diruz 'u 'este'fā dād . following-ez. foot_1 events-ez. yesterday he/she resign did . p n adv pro n v delm root nsubj dobj-lvc pobj advmod prep "following the yesterday events, he/she resigned." b. darpey-e ettefāqāt diruz 'u 'este'fā dād . following-ez. events yesterday he/she resign did . p n adv pro n v delm root nsubj dobj-lvc pobj prep advmod after reviewing the previous work of both tasks in section 2, we introduce our methodology in section 3 and data in section 4. we then discuss ezafe recognition and pos tagging tasks and their results in sections 5 and 6, respectively. ## methodology we see both ezafe recognition and pos tagging as sequence labeling problems, i.e., mapping each input word to the corresponding class space of the task. for the ezafe recognition task, the class space size is two, 0 for words without and 1 for words with ezafe. the class space size for pos tagging task is 14, consisting of the coarse-grained pos tags in the 10 million word bijankhan corpus. the results in section 2.1 are unfortunately reported on different, and in most cases irreproducible, test sets, using accuracy as the performance measure (which is insufficient and unsuitable for the task), making the comparison difficult. we hence re-implemented the model that reports the highest accuracy on the largest test set and compare its results with ours. ## data the 10 million word bijankhan @xcite corpus was used in the experiments. we shuffled the corpus, as adjacent sentences might be excerpts from the same texts, with a random seed of 17 using python's random library. this corpus comprises different topics, including news articles, literary texts, scientific textbooks, informal dialogues, etc, making it a suitable corpus for our work. we used the first 10% of the corpus as the test, the next 10% as validation, and the remaining 80% as the train set. ∼22% of the words have ezafe marker and ∼78% of them do not, in each and all of the sets. sentences with more than 512 words were set aside. table accept ezafe @xcite @xcite , there is actually no simple categorization for pos's that accept ezafe and those that do not, which can be seen in ## ezafe recognition for ezafe recognition, we experimented with different sequence labeling techniques and report the performance of them. these techniques include crf 1 , crf 2 , blstm, blstm+cnn, bert, and xlmroberta, as discussed in section 3.1. ## pos tagging for the task of pos tagging, we experimented with crf 1 , crf 2 , blstm+cnn, bert, and xlm-roberta models in the single-task settings, multitask settings with ezafe as the auxiliary task (except for crfs), and also in a single-task setting with ezafe information in the input. for the last one, we added the ezafe output of xlmroberta in section 5 to the input text. in this section, we first explain the role of ezafe information in pos tagging, then we discuss the results of the pos tagging task, and then we analyze it. ## conclusion and future work in this paper, we experimented with different models in the tasks of ezafe recognition and pos tagging and showed that transformer-based models outperform the other models by a wide margin. we also provided ezafe information to the pos tagging models and showed that while crf and rnn-based models benefit from this information, transformer-based models do not. we suggest that this behavior is most probably due to (1) contextual representation, (2) pretrained weights, which means a limited knowledge of syntactic relations between words, (3) the attention heads in these models, and (4) being trained on the pos task, which is related to ezafe recognition. an interesting direction for future work would be to investigate the role of ezafe in transformer-based models in the tasks that such information would be helpful, such as dependency and shallow parsing.