Unsupervised Deep Language and Dialect Identification for Short Texts (2020)

Abstract: Automatic Language Identification (LI) or Dialect Identification (DI) of short texts of closely related languages or dialects, is one of the primary steps in many natural language processing pipelines. Language identification is considered a solved task in many cases; however, in the case of very closely related languages, or in an unsupervised scenario (where the languages are not known in advance), performance is still poor. In this paper, we propose the Unsupervised Deep Language and Dialect Identification (UDLDI) method, which can simultaneously learn sentence embeddings and cluster assignments from short texts. The UDLDI model understands the sentence constructions of languages by applying attention to character relations which helps to optimize the clustering of languages. We have performed our experiments on three short-text datasets for different language families, each consisting of closely related languages or dialects, with very minimal training sets. Our experimental evaluations on these datasets have shown significant improvement over state-of-the-art unsupervised methods and our model has outperformed state-of-the-art LI and DI systems in supervised settings.

## introduction automatic language identification (li) and dialect identification (di) has become a very crucial part of natural language processing (nlp) pipelines and is part of tasks such as language modelling, categorization and analysis of code-mixed datasets. many researchers have carried out experiments in these domains both in speech and texts starting from @xcite . unsupervised li is an under-explored area, but is highly useful as it can exploit the large amount of unlabelled data and, more importantly, can be employed when the languages to be identified are not known in advance. however, unsupervised li for short texts is a very difficult task, for which performance is still comparatively poor. @xcite have explored unsupervised language identification but this does not work well with closely related short texts. @xcite have worked on the identification of arabic dialects. @xcite studied german dialect identification. these works are mostly supervised works. the task is even harder when it comes to unsupervised di. these previous works have generated some questions: what is the best way to construct sentence embeddings and how to cluster them efficiently for short texts when there is no labelled training data? how to address the hard task of di and li for closely related languages in an unsupervised way without any manual intervention? in this paper, we address these problems by taking inspiration from iterative clustering @xcite and self-attention-based sentence embeddings from @xcite . both of the papers have shown good results on unsupervised clustering and sentence embeddings for a single language dataset whereas our goal is to design a system which can work on different short texts of closely related languages and understand every distinct feature. our system takes different n-gram character features of a sentence and computes attention weights based on their importance in sentence construction. we then fine-tune sentence embeddings with an iterative clustering process with the help of the stochastic gradient descent (sgd) and backpropagation. we call the system unsupervised deep language and dialect identification (udldi). one of the main challenges of the model is to get sentence embeddings. as we do not have any labels here, we cannot use typical loss functions to update character-to-sentence embeddings. further, as the datasets are short text small datasets, there is much less scope to learn features to construct efficient sentence embeddings. to overcome this challenge we propose a loss function which considers the probability distribution of assigning sentences in each cluster and tries to maximize the unique sentence assignments in each cluster. these sentence embeddings define the initial cluster centre assignments, which we then fine-tune with the iterative clustering learning process. our method is tested against three different datasets consisting of under-resourced and closely related languages, the language families and/or varieties in question being south african, dravidian and swiss german. the south african dataset consists of languages from different language families, some of which are similar. we have achieved good results on each dataset and the results are better than other base results. we have also compared our sentence-embedding model to supervised li and di methods which outperformed other state-of-the-art models. the robustness of the model is, it does not require any manual intervention and it works well for short texts in any dataset consisting of closely related languages, for any language family. our contributions are: (1) the mentioned udldi model for language and dialect identification in short texts of closely related languages, (2) efficient sentence embedding construction for short texts, and (3) efficient clustering of closely related languages with a very small dataset. ## related work there are some li and di models which work efficiently for short texts and closely related languages on different datasets @xcite . @xcite have described different li and di models. language identification of closely related languages are explored in malay-indonesian languages @xcite , indian languages @xcite , south slavic languages @xcite @xcite . zampieri and gebre (2014) made a li system which can identify 27 languages. there is some research on li for short texts. @xcite explored an li method based on ngram character sequences for messages that are 5-21 characters long. @xcite have explored li for twitter data. @xcite used conditional random fields (crf) @xcite to build word-level li method. @xcite have investigated li in terms of code-switching. these methods are all supervised and this does not allow them to exploit the large amount of unlabelled data and situations where the languages are not known in advance. unsupervised li is a challenging task. @xcite used fuzzy art nns for clustering of documents written in arabic, persian, and urdu. in fuzzy art, they have shown how to update clusters dynamically. @xcite used a character n-gram representation for text. later k-means algorithm followed by particle-swarm optimization is applied to cluster the languages which results in different small clusters. @xcite worked on unsupervised tweet li with the chinese whispers algorithm of @xcite and graclus clustering. @xcite deal with code-switching identification between modern standard arabic (msa) and dialectal arabic. @xcite used word-level k-means clustering for li and clustered languages based on co-occurring words. @xcite used word2vec word embeddings and k-means clustering to make their li model. @xcite researched short-text tweets for unsupervised language detection which mainly focuses on seven languages. these algorithms work well for some specific languages and long texts. there is some research on dialect identification (di). di has been explored in serbo-croatian dialects @xcite , english varieties @xcite , dutch dialects @xcite , german dialects @xcite , mainland-singaporean-taiwanese chinese @xcite , arabic dialects in social media @xcite , portuguese varieties @xcite . one of the more famous dialect identification systems is an arabic dialect system which was given much emphasis in 2015 @xcite . but most of these works deal with supervised dialect identification. our model bridges the gap between di and unsupervised learning. 3 unsupervised deep language and dialect identification (udldi) unsupervised deep language and dialect identification (udldi) clusters the short texts of closely related languages in an efficient way using character-to-sentence embeddings with self-attention, based on an unsupervised loss function and an iterative clustering method (refer to figure 1 ). the model uses an unsupervised loss function to build the character-to-sentence embeddings. the iterative clustering process fine-tunes the sentence embedding and enhances the cluster assignment. ## results and discussions we have compared our results with various baselines, both for supervised and unsupervised setup, which are provided in ## conclusion in this paper, we propose a novel character attention based unsupervised deep language and dialect identification (udldi) model for short texts of closely related languages. we have performed our experiments on three different language families and outperformed other state-of-the-art models both as a supervised and an unsupervised model. we have also achieved promising results for dialect identification which is considered to be one of the hardest tasks in nlp. our experiments show that udldi is capable of doing language or even dialect identification irrespective of language families. the results in section 5.1 show that the model is able to correctly identify some of the verbs which are distinguishable in different dialects. we have also shown that, based on nmi, our unsupervised model is correctly predicting the number of languages that are present in a dataset, opening up interesting applications to large-scale multilingual dataset processing. our future work will be the improvement of clustering efficiency for closely related languages and dialects. though the model has outperformed other state-of-the-art models, there is a lot of scope to improve the unsupervised learning model. in this model, for example, co-occurrence of words in different contexts has not yet been considered. in the future, we will aim to capture word contexts along with n-grams in sentence embeddings.