A dual-encoding system for dialect classification (2020)

Abstract: In this paper we present the architecture, processing pipeline and results of the ensemble model developed for Romanian Dialect Identification task. The ensemble model consists of two TF-IDF encoders and a deep learning model aimed together at classifying input samples based on the writing patterns which are specific to each of the two dialects. Although the model performs well on the training set, its performance degrades heavily on the evaluation set. The drop in performance is due to the design decision which makes the model put too much weight on presence/lack of textual marks when determining the sample label.

## introduction the vardial evaluation campaign @xcite @xcite proposes several tasks related to the study of language variations across geographical regions. one of its oldest challenges is the is the identification of some specific dialects which, throughout the years, was tackled using several tasks. the challenge started initially with arabic dialect identification in the third edition of vardial @xcite . the fourth edition @xcite expands the family of dialect identification tasks with german dialect identification. the sixth edition @xcite comes with yet another task named moldavian vs. romanian cross-dialect topic identification. the current edition of the evaluation campaign @xcite introduces the task of romanian dialect identification (rdi). this is a closed task which is aimed at classifying a text sentence as whether it pertains to the moldavian idiom foot_0 or one of the other idioms of romanian language. for this task, our team proposes a simple foot_1 ensemble model which is built on the idea of exploiting the known differences in writing between the two dialects and on emphasizing such differences for classification. this paper describes the architecture of the proposed model, the setup for training experiments, the processing pipeline, and the results obtained for the rdi task. towards the end of the paper we also provide some insights into how we can easily tear-down and recreate the whole environment used for our experiments. the paper is structured as follows: section 2 presents some of the previous approaches in dialect identification tasks which bear a resemblance to our approach, section 3 briefly introduces the dataset used for the model, section 4 describes the architecture of the model, section 5 describes the experiments and the results, and in the end, section 6 presents the conclusions and future work. ## related work the task of discriminating between dialects of the same language has been the subject of various approaches that employ feature engineering, shallow or deep learning, and/or a combination of the afore-mentioned. one such example of using deep learning models for dialect classification is the word-based convolutional neural network @xcite , developed for arabic dialect identification. as authors note, their model "is conceptually similar" to the model described in @xcite . this model uses pretrained and fine-tuned word embeddings as the input to of a convolutional layer with max-pooling, followed by a fully-connected (dense) layer with softmax activation. another approach for dialect classification is to use ensemble models. examples of such models include a meta-classifier @xcite for arabic dialect identification, which leverages the collective knowledge of a cluster of individual classifiers by predicting the dialect based on their individual decisions, and an equally-weighted voting schema between a character-level convolutional network, a long short-term memory network and a string-kernel model for german dialect identification @xcite . at the previous edition of vardial @xcite , the highest-scoring model for the moldavian vs romanian cross-dialect topic identification task, was also an ensemble model @xcite . the model consists of a triplet loss network, a skip-gram network and a svm classifier. the svm classifier receives the concatenated representations learned by each network and based on these makes the prediction of the dialect. unlike the model presented in @xcite where the architecture emphasizes the learning of representation for taking a decision, the model presented herein uses shallow representations and puts more weight learning the distinction between them when deciding which label to predict. ## dataset the data source for romanian dialect identification task is the moroco dataset @xcite which consists of short text sentences (one per line) from both romanian and moldavian dialects of the romanian language. the sentences were extracted from various news articles and besides the already mentioned dialect labels, the dataset also provides additional labels for the topic within each sentence falls: culture, finance, politics, science, sports, and tech. the novelty of the current evaluation campaign is that a supplementary corpus was used for evaluation: moroco tweets @xcite , containing tweets as data samples instead of sentences from news articles. loading data and preprocessing. our approach to working with the data from moroco dataset is to construct an in-memory table, using pandas library foot_2 with four columns: sample id, dialect label, topic label, and the sample itself. after loading the dataset in memory, we perform on the fly, simple preprocessing on it. first, the sample text is converted to lowercase, then the tokenization mechanism splits it into its component words. ## model architecture the model we developed for the rdi task is an ensemble of three parts: two encoders and a deep-learning model consisting of five layers excluding the output layer. the entire architecture of the ensemble model is shown in figure 1 . as stated in section 1, at the core of our model lies the intuition that the peculiarities of each of the two romanian dialects involved in the rdi task can be detected by spotting the differences in writing. one of the most obvious examples of this intuition lies in how each dialect deals with writing words such as atât. the official rules stated by romanian academy say that in the middle of the words the character â should be used whereas the moldavian dialect uses the form î for the same sound, thus the same word -atât -is written in moldavian dialect as atît. our model aims to capture such differences and exploit them in the classification. each input sentence is transformed into two different encodings where each encoding is obtained from the term frequency-inverse document frequency @xcite (tf-idf) representation of the sentence in each of the two dialects. in other words, each sentence gets two tf-idf representations -one for each dialect. after obtaining the tf-idf encodings for an input sentence, the encodings are merged into a single matrix by concatenating columns. although this operation (represented by the merge tf-idf encodings node in the model diagram) is represented by a separate node it is a logical operation in the sense that it is not part of the model architecture. the tensor containing concatenated encodings is passed through a one-dimensional convolutional layer aimed at identifying the prominent features from each of the two encodings of the dialects. the global max-pooling layer that follows the convolutional layer picks the aforementioned features and send them to the following layers. the results of the global max-pooling layer are then passed through a dense layer followed by a dropout @xcite layer with rectified linear unit (nair and hinton, 2010) (relu) activation. then the results pass through yet another group consisting of a dense and dropout with relu activation layers after which, the final dense layer outputs the predictions for each of the rdi labels. a single dropout rate was used for both dropout layers whereas the dense layers are shrinking in size towards the final layer (see lastly, the predictions of the output layer, together with the expected output are fed to a categorical crossentropy loss function and an adam (kingma and ba, 2014) optimizer is used to find the global minimum. ## experiments and results we approach training of the full ensemble model in three separate stages: one training stage per each tf-idf encoder (for romanian and moldavian dialects) and a third and final stage for training the deeplearning classification model. all of these stages are performed in a single run, as part of a training pipeline that follows closely the architecture of the model presented in section 4 (figure 1 ) with the exception that tf-idf encoders were trained one after the other rather than in parallel. we describe each of these stages in the following subsections. in training the whole ensemble model we followed the classical recipe: the model was trained on the train data and validated against the validation set. we report the accuracy on training and validation sets and the score obtained on evaluation set in section 5.4. ## conclusions and future work in this paper we describe the architecture of an ensemble model designed to classify an input sentence as pertaining to romanian or moldavian dialects based primarily on the text markings which make the distinction between those two dialects. alongside the model architecture and performance we describe the processing pipeline used for training the model and the setup of the training environment which allowed us to easily replicate the training environment on the machine where training was performed. to our great disappointment, the results of the evaluation set shown both in even though our model makes use of techniques such as dropout @xcite which are meant to prevent overfitting and boost generalization, the model generalizes poorly and its confidence drops when the input sample lacks the strong textual marks it was trained to look for. in order to improve results -as one might expect -the future work should heavily concentrate on the classification model part of the ensemble presented herein. first and foremost, we need to investigate what is the root cause of long training time and improve the part responsible for it. this will allow us to test multiple variations of the model and perform a proper hyper-parameter tuning. however, if our investigation will conclude that there is no single part responsible for the most of the incurred delay in training, it make sense to consider rewriting the whole model architecture.