Enhancing T urkish Word Segmentation: A Focus on Borrowed Words and Invalid Morpheme (2024)

Abstract: This study addresses a challenge in morphological segmentation: accurately segmenting words in languages with rich morphology. Current probabilistic methods, such as Morfessor, often produce results that lack consistency with human-segmented words. Our study adds some steps to the Morfessor segmentation process to consider invalid morphemes and borrowed words from other languages to improve morphological segmentation significantly. Comparing our idea to the results obtained from Morfessor demonstrates its efficiency, leading to more accurate morphology segmentation. This is particularly evident in the case of Turkish, highlighting the potential for further advancements in morpheme segmentation for morphologically rich languages.

## introduction morphological analysis refers to the examination of word structure. it involves breaking down words into smaller units called morphemes and studying the different morphological rules that can apply to these units @xcite . these rules include inflectional morphology rules, which generate different forms of a word without altering its core meaning, and derivational morphology rules, which create new words based on existing ones by modifying their meanings @xcite . while both types of rules are important, recent studies have emphasized inflectional morphological analysis more. several methods for performing morphological segmentation include supervised, semi-supervised, and unsupervised approaches. supervised techniques involve using labeled data, such as a collection of previously segmented words, to train a model to recognize similar patterns in new data. however, obtaining large enough datasets to cover all living languages can be prohibitively resourceintensive, requiring significant manual labor and expertise. as a result, many researchers instead opt for unsupervised techniques, which rely solely on raw text data gathered from diverse sources like news articles, movie subtitles, or online content @xcite . another common technique for morphological segmentation is rule-based approaches, which utilize manually crafted rules to identify morpheme boundaries within words @xcite . although effective when implemented correctly, creating and maintaining these rules can be costly and time-consuming, especially for less commonly spoken languages. consequently, only some practical applications employ this method. when applying morphological segmentation, it is crucial to consider how morphemes are positioned within language units, as this can vary widely across different languages. for instance, specific languages may place prefixes before roots, while others might use suffixes after roots. moreover, the complexity of morphological segmentation can pose additional challenges, particularly when creating annotated data, which tends to be expensive and time-consuming. to address this issue, our proposed solution involves developing a comprehensive model capable of handling multiple languages simultaneously. specifically, we will focus on improving morphological segmentation performance for agglutinative languages such as finnish and turkish @xcite , which exhibit high levels of morphological complexity. to accomplish this goal, our project will leverage freely available word lists and perform extensive preprocessing steps to ensure consistency and accuracy. preprocessing will entail converting each dataset into individual lines containing single words, ensuring uniformity in letter casing and font styles, and eliminating extraneous elements such as punctuation marks, numerals, and duplicated entries. once completed, we will apply two distinct algorithms to segment the processed data. our pri-mary algorithm will be morfessor, a probabilistic model explicitly designed for morphological segmentation tasks. we anticipate that evaluating and refining the resulting segmentations through targeted modifications will yield significant improvements in overall performance. ## review of literature the supervised approach of morphological segmentation uses word information such as part-of-speech (pos) tags, morphological rules, morpheme dictionaries, etc. the unsupervised approach exploits morphemes from a raw corpus @xcite . this approach has been studied in different languages. there are some mathematical frameworks for modeling methodologies: map modeling is based on the minimum description length (mdl) principle, which considers accuracy and model complexity. an fsa can specify the various word forms @xcite . the first turkish morphology analyzer is oflazer's model. it is implemented using a pc-kimmo environment @xcite , which is a computational approach for two-level analyzers @xcite and addressing: zemberek is an open-source framework for turkic languages using latin script that includes language structure information and nlp operations. the standard morphological parser identifies the root and potential suffixes @xcite . another morphological analyzer is trmorph, a two-level and rule-based analyzer proposed by @xcite . it uses the stuttgart finite state transducer (sfst) tools and consists of 3 major components: a finite state machine (fsa), a set of two-level rules, and a lexicon that keeps the class of root and some lexical irregularities @xcite . this lexicon is hand-made to be an error-free lexicon created during implementation. based on morpho-phonological considerations, trmorph considers morpheme alternations. morphophonological alternation is dependent on phonological alternation. therefore, turkish vowels and consonants can be dismissed, reproduced, or revised to the closest harmonic letter. for example: spoken turkish follows a two-level rule system for phonetics, while morphotactics is encoded as finite state machines for word categories like verbs and nouns @xcite . furthermore, turkish words have two classes in the morphotactics part: nominal and verbal. all categories except verbs are in nominal class. nominal morphotactic is simple. instead, verbal morphotactics is complicated and has many exceptions. so, both morphotactics should be used simultaneously. this analyzer has been updated and uses the foma fst compiler, a c compiler, and converts the input to the fst, and the lexicon of trmorp is a raw text file precisely the prior version @xcite . in turkish, adding morphemes to a word's root or stem can change it from a noun to a verb or vice versa. these morphemes can also create adverbial constructs. this language has exceptions, and persian, arabic, and other foreign entered words are considered one of them. the morphological analyzer, used by @xcite , is a two-level analyzer with over 49321 entries, arranged under 14 parts of speech. their model uses flag diacritics. flag diacritics' main usage goal is adding a small quantity of memory to the finite state machine during the generation and analysis steps at run-time. if we do not have this, state transitions depend only on the current state and input symbols @xcite . this model is used in itu turkish nlp web service 1 consists of different elements such as tokenizer, normalizer, morphology analyzer, morphology disambiguation, et cetera. in itu, components are categorized under four groups: preprocessing, morphological processing, multiword expression handling, and syntactic processing @xcite . in itu, the morphological layer uses a rule-based analyzer that is proposed by @xcite as well as hfst-helsinki fst proposed by @xcite and a hybrid morphological disambiguator @xcite . besides turkish-specified morphological analyzers, some unsupervised methods can be used for languages without exception. morfessor is used in our project in this way. @xcite proposed the main idea of morfessor in 2002, and they improved their model and named it morfessor. morfessor is an unsupervised generative probabilistic model for predicting morphological segmentation. the first version of it is named morfessor baseline @xcite . then, the idea expanded and introduced other versions, such as morfessor 1.0, morfessor flatcat, allomorfessor, and morfessor 2.0 @xcite @xcite . however, the baseline version is still popular as a morphological analyzer even though other versions have improved the results. morfessor is well suited for rich morphology languages such as finnish, estonian, and german. it uses a corpus of unannotated text as input and produces a segmentation of words observed in the text as its output. unlike most morphological models, morfessor is not restricted to the count of morphemes. morphological analyzers must be built for each language, but morfessor is a general model for unsupervised and semi-supervised morphological segmentation @xcite @xcite @xcite . it has been proved that the weighted function leads to better results. @xcite introduced a semi-supervised model based on the morfessor baseline. the semi-supervised approach is an excellent way for humans to prepare annotated data manually since data is expensive and complicated to obtain. an important question, however, is how many annotated words are required. kohonen et al. have proved that 100 manually segmented words is enough and can improve output quality @xcite . the algorithm processes all combinations of training data in one cycle. for each combination, it checks every possible two-part division and picks the one with the lowest cost. the cost can decrease or stay the same at each step, which means the algorithm will eventually stop working when the cost drops below a certain level. there is also a way to use this algorithm to decode messages, which involves finding the best division for a new message without changing the program's settings @xcite . a newer version, morfessor 2.0, has additional features that allow it to divide messages even when it does not have all the information it needs @xcite . the impact of the smaller size of the annotated data on the cost function is insignificant compared to the likelihood of the unannotated data. therefore, additional weighting parameters should be included in the annotated data to avoid the adverse effects of annotations on the cost function @xcite . the second feature of morfessor 2.0 is an online and batch training mode. so, the model needs to know how much the final size of training data is, and it has access to only one word of training data at a time. morfessor 2.0 can skip analyzed compounds and constructions randomly since the variant compounds can be found in the current text. morfessor 2.0 produces the n-best segmentation of multiple generated segmentations for every compound via the viterbi algorithm. this feature lets the algorithm extract the most conceivable segmentation for a compound @xcite . in other studies, @xcite , and @xcite investigated the effects of different approaches to breaking down words into smaller units for language modeling purposes @xcite . @xcite examined ten languages with diverse structural properties and trained wordlevel language models while adding granular information about the constituent parts of each word throughout the training process. after comparing several segmentation techniques, they concluded that character-based models produced superior outcomes overall. moreover, rule-based approaches tailored to each language's distinct characteristics significantly outperformed alternative partitioning strategies @xcite . meanwhile, haghdoost et al. explored the utility of morfessor-both supervised and unsupervised variants-for generating morphological networks in persian and turkish. specifically, they determined that the supervised approach was preferable for their objectives, enabling them to incorporate newly segmented words into their lexicon @xcite . furthermore, they employed both morfessor versions to establish a turkish mor-phological network, thereby revealing relationships among segmented words through a tree-like framework similar to manual segmentation @xcite . additionally, research on 50 distinct languages revealed disparities in the difficulty levels associated with crafting language models due to fluctuations in grammatical architectures @xcite . prior work demonstrated that morfessor mitigates morphological impacts for numerous languages, and morphologically driven partitions enhance cross-lingual language modeling @xcite . ## our experiment our project aims to accurately segment morphemes using morfessor, as natural language word lists are time-consuming and labor-intensive to create manually. we address the challenge of creating raw and segmented datasets by using the morphochallenge 2010 turkish dataset 2 as input to morfessor. our contributions include gold-segmented words, invalid morphemes (i.e., morphemes not included in the language), and a list of entered foreign words (borrowing words from other languages). preparing datasets is not the only part of our work. we use them in different types of morfessor: supervised, semi-supervised, and unsupervised. then, we observe the behavior of morfessor in the segmentation process and according to this, we add some steps to the segmentation process to enhance the performance of morfessor that will be discussed completely. to put it in a nutshell, our modified process of morfessor does not let the entered foreign words become separated. it also uses an invalid morpheme dataset to prevent their breaking down from the words. also, we produce the second and best probable segmentation by morfessor. all of these experiments are done with the same input dataset-are evaluated and compared with each other. let's dive into the details in the following subsections. ## results and observations curious about morfessor's inner workings, we examined morfessor's output via visual inspection. starting with a random selection of 200,000 words from the word list, we focused on the semi-supervised morfessor's behavior. segmenting 1,600 words from the gold standard file, we allocated 400 words as the evaluation test set. the random words were selected because we only wanted to observe the morfessor in an unbiased and fair way. throughout the segmentation process, morfessor adeptly identified suffixes and partitioned associated suffix groups into one or multiple components. nonetheless, it maintained a reasonable count of word fragments. syllabification played a vital role, enabling efficient segmentation with reduced syllable counts. morfessor categorized words lacking suffixes into two classes: monosyllabic and polysyllabic. monosyllabic words remained undivided, whereas polysyllabic counterparts experienced arbitrary breakage into smaller pieces. segmenting words attached with suffixes, morfessor isolated the base word and either singular or compound suffixes, subject to word length and contextual factors. diminutive suffixes could face division or remain cohesive according to the prevailing situation. overall, morfessor's functional design shed light on its competence in achieving satisfactory segmentation outcomes. based on our observations, we can derive additional insights: as we see in morfessor generates the most likely segmentation as the "1-best". to better understand the impact of other possible segmentations, we choose the "2-best" segmentation. the table shows that precision improves slightly while recall decreases slightly. another option is to select the best of the "5-best" segmentations while accounting for invalid morphemes. if an invalid morpheme is detected, morfessor will choose the subsequent segmentation that does not contain that morpheme. if no segmentations are missing, the algorithm will return the first segmentation as the best. because the results did not improve significantly, we tried another idea. in this case, we used the foreign borrowed words we prepared. we will check to see if morfessor produced any borrowed words. if so, morfessor should combine the segments, as these are prohibited morphemes. the graph shows that considering borrowed words significantly improved morfessor's performance. this suggests that having a dataset of borrowed words can improve morfessor's accuracy and obtain a more detailed segmentation list. ## conclusion in this study, we decided to focus on morfessor 2.0, a powerful probabilistic tool designed for morphological segmentation specifically tailored for the turkish language. our objective was clear: achieve accurate morpheme segmentation in turkish and discover methods to improve morfessor's overall performance. we started by utilizing the morphochallenge 2010 turkish dataset, then expanded our sources by gathering extra data comprising borrowed words and even invalid morphemes. taking a closer look at different methodologies, our research involved three main approaches-supervised, semi-supervised, and unsupervised learning applied to morfessor. when measuring morfessor's effectiveness, we relied on six essential evaluation metrics: boundary recall, boundary precision, morpheme recall, morpheme precision, segmentation recall, and segmentation precision. after thorough analysis, one particular technique stood out among the rest-the semisupervised approach demonstrated superior accuracy compared to the others. while exploring how morfessor functions, we noticed some remarkable abilities; primarily, it excels in recognizing suffixes and breaking them down into smaller sets. crucially, though, it avoids excessive fragmentation during this breakdown process. despite those strengths, however, there were still difficulties faced in segmenting foreign terms. addressing this challenge head-on, we suggested innovative strategies to boost morfessor's capacity to handle foreign vocabulary better. these creative solutions entailed examining alternative segmentations (second-best), discarding faulty morphemes, and incorporating borrowed phrases from various languages. upon merging incorrect morphemes and foreign loans throughout the segmentation procedure, we witnessed notable progress in both precision and recall ratios related to turkish. for interested readers, you can find all relevant experiments and corresponding outputs on our project's github page foot_1 . wrapping up our investigation, we believe our work constitutes a major leap forward in crafting a far-reaching segmentation algorithm equipped to skillfully tackle the labyrinthine nuances found in not only turkish but also other highly inflected languages. as part of our future plans, we intend to spotlight and incorporate even more extensive catalogs of foreign terminology present in turkish, ultimately leading to enhanced precision concerning segmented word directories.