Early Guessing for Dialect Identification (2022)

Abstract: This paper deals with the problem of incre-mental dialect identification. Our goal is toreliably determine the dialect before the fullutterance is given as input. The major partof the previous research on dialect identification has been model-centric, focusing on performance. We address a new question: How much input is needed to identify a dialect? Ourapproach is a data-centric analysis that resultsin general criteria for finding the shortest inputneeded to make a plausible guess. Workingwith three sets of language dialects (Swiss German, Indo-Aryan and Arabic languages), weshow that it is possible to generalize across dialects and datasets with two input shorteningcriteria: model confidence and minimal inputlength (adjusted for the input type). The sourcecode for experimental analysis can be found atGithub.

## introduction language identification depends very much on what kind of languages we are discriminating. if languages to be discriminated are distant (e.g., russian vs. chinese), the task is straightforward and a short sequence of words provides enough information to assign the correct class. but if languages are similar and written in the same script (e.g. russian vs. ukrainian), much longer samples are needed to encounter the discriminating features @xcite . the task is even harder when dealing with non-standard orthography, which we find in written dialects and user posts on the internet @xcite . current research is mostly concerned with improving the performance of the task by applying increasingly sophisticated methods, including pretrained models, whose performance varies across different language dialects @xcite . however, many other aspects of the task may play an important role in practical applications. one of such challenges is the possibility to make early guesses on the language or dialect before seeing the whole message. such a feature can be especially useful for more dynamic classification of a continuous stream of messages as in transcribed speech or instant messaging, where sentence boundaries are not clearly defined and the input can be segmented arbitrarily. a reliable early classification can improve further processing (e.g., translation, information extraction). in this paper, we address the problem of early guessing in dialect identification mostly from the data-centric point of view, but also consider some model-centric issues. we search for criteria for shortening the input so that the model performance is the same or similar to the performance obtained with the full input. we perform experimental studies with four datasets representing three sets of dialects with considerably different writing practices (some writings are more standard than others) and find that the same input shortening criteria give the best results in all the cases. ## related work the task of dialect identification and discrimination between similar languages is mostly addressed in the scope of the vardial evaluation campaign @xcite @xcite . the organizers of the tasks released datasets for various dialects and similar languages, such as swiss-german, indo-aryan, uralic, romanian, arabic, chinese, etc. in arabic dialect identifications (adi), other popular datasets include arabic online commentary (aoc) @xcite , nuanced arabic dialect identification (nadi) twitter data set (abdul-mageed et al., 2020) 2 , multi-arabic dialect applications and re-sources (madar) @xcite foot_2 etc. with the advent of transformers-based models, we see a wide use of pre-trained models in dialect classifications @xcite @xcite , but traditional approaches based on n-gram statistics also seems to remain quite popular @xcite @xcite @xcite @xcite . in the standard datasets (data from written sources) of the vardial task, such as indo-aryan and romanian, language specific pre-trained models worked well @xcite , while in the nonstandard datasets (transcribed from speeches), such as swiss german and arabic, pre-trained transformer models didn't boost the performance as expected. regarding other arabic datasets, viz., aoc and nadi, neural (cnn, bilstm, etc.) and transformer models were successfully used @xcite @xcite . in contrast to most previous work, the main focus is not on improving the performance when the task is the classification of the whole utterance but on finding the minimal input on which an acceptable classification performance can be achieved. this aspect of the problem has been so far only minimally addressed in the literature. one such study was on the influence of the length of the utterances in adi on basic traveling expression corpus (btec) @xcite for coarse and fine-grained dialect classification using n-gram features with multinomial naive bayes classifier. they found that with an average length of 7 words, it is possible to classify the dialects with acceptable accuracy @xcite . our study is a more general data-centric exploration of the early classification involving varied datasets in multiple languages. ## data for our study, we select four datasets, three offered by the vardial evaluation campaign (see section 2). we perform three tasks: german dialect identification (gdi) foot_3 , indo-aryan language identifica- the gdi dataset represents four areas of swiss german: basel, bern, lucerne, and zurich. training and the test datasets are obtained from the archimob corpus of spoken swiss german @xcite . gdi datasets are available from the years 2017-2019. we work with the gdi-2018 in the 4-way classification setting. the ili task identifies five closely related languages from the indo-aryan language family, namely, hindi, braj bhasha, awadhi, bhojpuri, and magahi. for each language, 15,000 sentences are extracted mainly from the literature domain. the sources were previously published either on the internet or in print. these languages are often mistakenly considered to be varieties of hindi. the adi var-dial task @xcite focused on five classes, viz., modern standard arabic (msa), egyptian (egy), gulf (glf), levantine (lav), moroccan (mor), and north-african (nor). msa is the modern variety of the language used in news and educational articles. this differs from the actual communication language of native speakers lexically, syntactically and phonetically. the vardial adi dataset is both speech transcribed and transliterated to english. another arabic dialect dataset used for the experimentation is the arabic online commentary (aoc) @xcite dataset. this constitutes a large-scale repository of arabic dialects and covers msa and the dialectal varieties, viz., egyptian (egy), gulf (glf), levantine (lev), and moroccan (mor). ## methods we perform incremental analysis by running the same classifier on varied substrings of the test input. this is to understand the performance at different lengths of the input text. we start with the first word, then repeat the classification with the first two words and so on until we reach the end of the utterances. we refer to all the incremental substrings as fragments. we observe the model's performance at each incremental step and analyze its state (confidence) to determine the earliest point when a plausible guess can be made. we perform extensive analysis on the influence of different factors that directly and indirectly affect the model performance after applying the shortening criteria. ## experiments and results each model was trained for 4 epochs with adam optimizer using a learning rate of 2e-5 on the corresponding training set using 1 tesla k80 gpu. we used the pre-trained models from the huggingface library. 12 bert-base vs. linguistic proximity fragment length the impact of the fragment lengths on the model is analyzed in figure 1 . the solid lines in the plot represent the ratio of the correct predictions at each fragment length n to the total number of test instances (which is constant for a given corpus). the dashed lines represent the ratio of the correct predictions at fragment length n to the number of possible predictions at that length (i.e. sentences of length n or longer). these graphs show that it is possible to make accurate and useful predictions with shorter fragments, and the additional gain with longer fragments is proportionally lower. a first peak for the gdi data is obtained at the length 4, while the peak is on the length 7 for the ili dataset. in adi-aoc, the peak is on 13 we refer to linguistic proximity in context with the language-specific pre-trained models the length 2. the trend in all these cases is the same, modulated by the length of the original utterances (longer in ili). the trends are somewhat different in the adi-vardial dataset: there are no clear peaks, but the best scores are still obtained on shorter segments. input shortening criteria for each set of dialects, we found the minimum input length for best performances by repeated experiments on different fragment lengths. we found that this minimum fragment length was same as the peak values in figure 1 . for adi-vardial, we found that at length 8, the maximum performance is obtained. further, the maximum input length is determined according to two additional criteria: model confidence after temperature scaling and label consistency. we experiment with several criteria defined in terms of these two variables (described in detail in appendix d).we find that the same selection criterion gives the best results in all the data sets: the first decrease in the model confidence after the minimum length threshold. in addition to this, label consistency was helpful in all the data sets except gdi: the results improve when we consider the decrease in the model confidence only when the current and the previous labels are the same. while analyzing the performance on the shortened input, we checked what would happen in an ideal case if we knew where to cut the input utterance in each case. this performance, which is the upper bound on our task is, in fact, better than the performance with the full input (details in appendix b). it provides an empirical justification for the research goal of finding criteria for shortening the input. ## conclusion we have identified general criteria for making early guesses in dialect identification: language spe-cific minimal length of the input and languageindependent change in the model confidence score (the first decrease in the confidence score). while these criteria do not maintain the performance achieved with the full input, they are the starting point for further optimization, which can eventually lead to an overall improvement on the task.