Detection of Similar Languages and Dialects Using Deep Supervised Autoencoder (2020)

Abstract: Language detection is considered a difficult task especially for similar languages, varieties, and dialects. With the growing number of online content in different languages, the need for reliable and robust language detection tools also increased. In this work, we use supervised autoencoders with a bayesian optimizer for language detection and highlights its efficiency in detecting similar languages with dialect variance in comparison to other state-of-the-art techniques. We evaluated our approach on multiple datasets (Ling10, Discriminating between Similar Language (DSL), and Indo-Aryan Language Identification (ILI)). Obtained results demonstrate that SAE are higly effective in detecting languages, up to a 100% accuracy in the Ling10. Similarly, we obtain a competitive performance in identifying similar languages, and dialects, 92% and 85% for DSL ans ILI datasets respectively.

## introduction internet content is growing exponentially over time, and as a direct consequence, more languages and dialects need to be processed, as they serve as key components in various natural language processing (nlp) tasks @xcite . language detection is the task of determining the language for a given text. although language detection has significantly improved over the past years, challenges remain. detecting similar languages, detecting languages when multiple language contents exist in a single document, and detecting language in short texts are still active research areas @xcite @xcite . discriminate between very close languages or dialects, for example, german dialect identification, indo-aryan language identification, is considered a difficult task @xcite . although dialect identification is commonly based on the distributions of letters or letter n-grams, these approaches might face serious difficulties when trying to distinguish related dialects that have similar phoneme and grapheme inventories @xcite . in a multilingual country like india, there exist many languages and many of them have multiple dialects (chittaragi and koolagudi, 2019). for example, in the case of the odia language, although the written text is the same, there exist many dialects (e.g. baleswari, ganjami, sambalpuri, desiya. etc.) @xcite . moreover, the automatic identification of dialect in low resource languages suffers from the lack of large training datasets or pre-trained language models. most of the previous research on language identification has focused on using traditional machine learning approaches like naive bayes, support vector machine (svm), in combination with word n-grams, graph-based n-grams, prediction partial matching (ppm) or linear interpolation with postindependent weight optimization and majority voting for combining multiple classifiers @xcite . however, more recently, deep learning techniques have shown substantial results in many nlp tasks including language detection @xcite . for many deep learning tasks, semi-supervised autoencoders have proven to build reliable representations with few annotated data @xcite . to the best of our knowledge, autoencoders (ae) have never been applied for similar language detection. in this paper, we explore the use of supervised autoencoders (sae), hence leveraging labels in the latent space, for language detection. ## proposed method the overall architecture of the proposed method is shown in figure 1 . the following subsections briefly describe the main components of our approach. ## conclusion in this paper, we introduced sae with bo for language detection using n-grams at the character level and illustrated its performance on the discrimination of very close languages or dialects on several well-known corpora. we also presented some advantages of the proposed approach, and discuss some of the future directions for sae-based language detection. 5