Orthographic vs. Semantic Representations for Unsupervised Morphological Paradigm Clustering (2021)

Abstract: This paper presents two different systems for unsupervised clustering of morphological paradigms, in the context of the SIGMORPHON 2021 Shared Task 2. The goal of this task is to correctly cluster words in a given language by their inflectional paradigm, without any previous knowledge of the language and without supervision from labeled data of any sort. The words in a single morphological paradigm are different inflectional variants of an underlying lemma, meaning that the words share a common core meaning. They also - usually - show a high degree of orthographical similarity. Following these intuitions, we investigate KMeans clustering using two different types of word representations: one focusing on orthographical similarity and the other focusing on semantic similarity. Additionally, we discuss the merits of randomly initialized centroids versus pre-defined centroids for clustering. Pre-defined centroids are identified based on either a standard longest common substring algorithm or a connected graph method built off of longest common substring. For all development languages, the character-based embeddings perform similarly to the baseline, and the semantic embeddings perform well below the baseline. Analysis of the systems’ errors suggests that clustering based on orthographic representations is suitable for a wide range of morphological mechanisms, particularly as part of a larger system.

## introduction one significant barrier to progress in morphological analysis is the lack of available data for most of the world's languages. as a result, there is a dramatic divide between high and low resource languages when it comes to performance on automated morphological analysis (as well as many other language-related tasks). even for languages with resources suitable for computational morphological analysis, there is no guarantee that the available data in fact covers all important aspects of the language, leading to significant error rates on unseen data. this uncertainty regarding training data makes unsupervised learning a natural modeling choice for the field of computational morphology. the unsupervised setting takes away the need for large quantities of labeled text in order to detect linguistic phenomena. the sigmorphon 2021 shared task aims to leverage the unsupervised setting in order to identify morphological paradigms, at the same time including languages with a wide range of morphological properties. for a given language, the morphological paradigms are the models that relate root forms (or lemmas) of words to their surface forms. the task we tackle is to cluster surface word forms into groups that reflect the application of a morphological paradigm to a single lemma. the lemma of the paradigm is typically the dictionary citation form, and the corresponding surface forms are inflected variations of that lemma, conveying grammatical properties such as tense, gender, or plurality. for example, in developing our system, we consider two types of information that could reasonably play a role in unsupervised paradigm induction. first, the words in a single paradigm cluster are different inflectional variants of an underlying lemma, meaning that the words share a common core meaning. they also -usually -show a high degree of orthographical similarity. following these intuitions, we investigate kmeans clustering using two different types of word representations: one focusing on orthographical similarity and the other focusing on semantic similarity. intuitively, we would expect the cluster of forms for walk to be recognizable largely based on orthographic similarity. the partially irregular cluster for bring shows greater orthographical variability in the past-tense form brought and so might be expected to require information beyond orthographic similarity. system overview. the core of our approach is to cluster unlabelled surface word forms using kmeans clustering; a complete architecture diagram can be seen in figure 1 . after reading the input file for a particular language to identify the lexicon and alphabet, we transform each word into two different types of vector representations. to capture semantic information, we train word2vec embeddings from the input data. the orthographybased representations we learn are character embeddings, again trained from the input data. details for both representations appear in section 4.1. for the experiments in this paper, we test each type of representation separately, using randomly initialized centers for the clustering. in later work, we plan to explore the integration of both types of representations. we would also like to explore the use of predefined centers for clustering. these pre-defined centers could be provided using either a longest common subsequence method or a graph-based algorithm such as that described in section 4.3. the final output of the system is a set of clusters, each one representing a morphological paradigm. ## previous work the sigmorphon 2020 shared task set included an open problem calling for unsupervised systems to complete morphological paradigms. for the 2020 task, participants were provided with the set of lemmas available for each language @xcite . in contrast, the 2021 sigmorphon task 2 outlines that submissions are unsupervised systems that cluster input tokens into the appropriate morphological paradigm @xcite . given the novelty of the task, there is a lack of previous work done to cluster morphological paradigms in an unsupervised manner. however, we have identified key methods from previous work in computa-tional morphology and unsupervised learning that could be combined to approach this problem. previous work has identified the benefit of combining rules based on linguistic characteristics with machine learning techniques. @xcite established a baseline for the paradigm discovery problem that clusters the unannotated sentences first by a combination of string similarity and lexical semantics and then uses this clustering as input for a neural transducer. @xcite investigated the benefits of different similarity models as they apply to arabic dialects. their findings demonstrated that word2vec embeddings significantly underperformed in comparison to the levenshtein distance baseline. the highest performing representation was a combination of fast-text and a de-lexicalized morphological analyzer. the fasttext embeddings @xcite have the benefit of including sub-word information by representing words as character n-grams. the de-lexicalized analyzer relies on linguistic expert knowledge of arabic to identify the morphological closeness of two words. in the context of the paper, it is used to prune out word relations that do not conform to arabic morphological rules. the approach mentioned greatly benefits from the use of a morphological analyzer, something that is not readily available for low-resource languages. @xcite focused on the use of morphological transformations as the basis for word representations. their representation can be quite accurate for affix-based morphology. our representations are based entirely off of unlabelled data and do not require linguistic experts to provide morphological transformation rules for the language. additionally, we hoped to create a system that would be robust for languages that include non-affix based morphology. in this work we compare word2vec representations to characterbased representations to represent orthography. we have not yet evaluated additional representations or combinations of the two. ## task overview the 2021 sigmorphon shared task 2 created a call for unsupervised systems that would create morphological paradigm clusters. this was intended to build upon the shared task from 2020 that focused on morphological paradigm completion. participants were provided with tokenized bible data from the jhu bible corpus @xcite and gold standard paradigm clusters for five development languages: maltese, persian, portuguese, russian and swedish. teams could use this data to train their systems and evaluate against the gold standard files as well as a baseline. the baseline provided groups together words that share a substring of length n and then removes any duplicate clusters. the resulting systems were then used to cluster tokenized data from a set of test languages including: basque, bulgarian, english, finnish, german, kannada, navajo, spanish, and turkish. ## system architecture the overall architecture of our system includes several distinct pieces as demonstrated in figure 1 . for a given language, we read the corpus text provided and generate a lexicon of unique words. the lexicon is then fed to an embedding layer and an optional lemma identification layer. the embedding layer generates a vector representation of each word based on either a character level embedding or a word2vec embedding. when used, the lemma identification layer generates a set of predefined lemmas from the lexicon based on either the standard longest common substring or a connected graph formed from the longest common substring. result word embeddings along with the optional set of predefined lemmas are used as input to a kmeans clustering algorithm. in the event predefined lemmas are not provided, the system defaults to using a randomly initialized set of centroids. otherwise, the initial centroids for the clusters are the result of finding the appropriate word embedding for the lemmas identified. once a cluster has been created, the output cluster predictions are formatted into a paradigm dictionary which can be written to a file for evaluation. ## results for test languages, we run clustering only with the better-performing character-based representations. the performance on test languages was evaluated with clusters of size 500, 1000, 1500, and 1900. these results are in ## error analysis we have evaluated the results from the word2vec representations and our character-based embedding and compared them to the gold standard paradigms provided by the task organizers. we have found that, overall, the character-based version is more robust on regular verb forms than the word2vec version, and that neither is effective on irregular forms. additionally, we explore some of the nu-anced errors with the character based embeddings and how they could be addressed for future work.