Optimizing the Size of Subword Vocabularies in Dialect Classification (2023)

Abstract: Pre-trained models usually come with a pre-defined tokenization and little flexibility as to what subword tokens can be used in downstream tasks. This problem concerns especially multilingual NLP and low-resource languages, which are typically processed using cross-lingual transfer. In this paper, we aim to find out if the right granularity of tokenization is helpful for a text classification task, namely dialect classification. Aiming at generalizations beyond the studied cases, we look for the optimal granularity in four dialect datasets, two with relatively consistent writing (one Arabic and one Indo-Aryan set) and two with considerably inconsistent writing (one Arabic and one Swiss German set). To gain more control over subword tokenization and ensure direct comparability in the experimental settings, we train a CNN classifier from scratch comparing two subword tokenization methods (Unigram model and BPE). For reference, we compare the results obtained in our analysis to the state of the art achieved by fine-tuning pre-trained models. We show that models trained from scratch with an optimal tokenization level perform better than fine-tuned classifiers in the case of highly inconsistent writing. In the case of relatively consistent writing, fine-tuned models remain better regardless of the tokenization level.

## introduction the change from word to subword tokenization opened a large space of tokenization possibilities: any substring of a word (subword) is potentially a good token, but some might be more useful than others. in contrast to this, pre-trained models usually come with a predefined tokenization and little flexibility in input preprocessing. this problem is even more important in a multilingual setting, where, for many languages, only a little data is available, often written in a nonstandard writing (e.g. transcriptions of spoken language, social media posts) with pronounced regional differences. fine-tuning pretrained models (with cross-lingual transfer) has become the primary approach to processing such languages. predefined tokenization, which is part of this research framework, is likely not to be suitable for the level of inconsistency that is typical for target low-resource languages. in this paper, we study the benefits of optimal subword tokenization in one of the basic tasks in multilingual nlp -dialect classification. this task can be seen as a stand-alone task (e.g., for tracing the source of media posts) or a step in other end-user tasks such as machine translation or natural language understanding (nlu). we choose this task as an especially challenging case of text encoding bridging the work on language modelling and text classification. although it is a classification task, it does not rely on an abstract semantic representation of the whole sentence (as in usual text classification) but on surface features of the text, such as distinctive suffixes or prefixes of words, phonetic clusters, and order of tokens, closer to language modelling. these features show up occasionally in the text, which otherwise might look the same in two different dialects @xcite . the right level of tokenization can be expected to help identify these features and thus encode the text better for other purposes too. aiming at generalizations beyond the studied cases, we work with four data sets (two arabic, one indo-aryan, and one swiss german) representing two levels of writing consistency (transcribed speech vs. originally written text) and three different types of languages. we consider three levels of tokenization (character, subword, word) testing two main subword tokenization methods: one example of a probabilistic model (unigram model @xcite ) and one example of a bottom-up compression algorithm (bpe @xcite , also implemented by @xcite ). to gain flexibility with varying the level of tokenization, we train our own classifiers (one bidirectional long short term memory (bilstm) and two convolutional neural networks (cnns)), which we also evaluate against comparable fine-tuned classifiers with bert-based pre-trained models. our findings are expected to generalize to other tasks similar to dialect classification and, to a certain degree, to nlu tasks. ## related work dialect identification replaces language identification whenever a language has many regional variants as in the case of arabic, chinese or (swiss) german. such cases are mostly covered in a series of shared tasks @xcite @xcite @xcite . the solutions submitted to the shared tasks range from traditional machine learning to state-of-the-art deep learning models. traditional machine learning classifiers such as support vector machines (svm), logistic regression (lr), and naive bayes (nb) utilizing character or word level n-gram features were found to perform quite well across different languages and dialects @xcite @xcite @xcite . for instance, a version of a character-level n-gram language model with a domain adaptation technique is the state of the art for identifying swiss german (f-score 0.75) and indo-aryan (f-score 0.96) dialects without acoustic features @xcite . this approach, however, requires numerous model retraining iterations, which is not suitable for larger models. neural networks have been used for this task too including cnn, lstm, and pre-trained transformers models, in which are currently prevailing @xcite @xcite . the performance of these models varies depending on the datasets. ensembles of neural and traditional models are also utilized @xcite @xcite . character-level tokenization proves useful for capturing the relevant features, but previous studies do not address specifically the question of input granularity. outside of dialect identification, @xcite suggest that tokenization could impact neural machine translation (nmt) quality. they compared tokenizers such as moses, sentencepiece, opennmt, stanford, and mecab on japanese, russian, chinese, german and arabic translations to english. they found that moses tokenizer gave the best result for arabic, russian and german; mecab for japanese; and stanford for chinese. uysal and gunal (2014) studied the effect of pre-processing in the english and turkish languages and they observed that using appropriate domain and language dependant pre-processing can improve the performance. @xcite propose a general optimization method for finding subword tokens for machine translation. @xcite find that the surprisal of a language model is minimised cross-linguistically at a particular level of subword segmentation with the resulting size of the input vocabulary being the word-level vocabulary multiplied by 0.4. @xcite find that much smaller vocabularies minimize text redundancy and lead to a converging text entropy across 47 languages. quite a few solutions have been proposed for unsupervised subword segmentation @xcite @xcite @xcite @xcite . the sentencepiece library @xcite implements two very popular methods: bpe, a general data compression algorithm @xcite first applied to text by @xcite and unigram model @xcite , similar to morfessor @xcite in that it considers multiple possible subword splits at the same time. some related works on fine-tuning vocabulary sizes for nlp applications include @xcite @xcite . the work on comparing subword tokenization algorithms reports rather inconsistent outcomes. for instance, @xcite find that bpe gives better results than morfessor on the task of language modeling, but @xcite show that linguistically motivated vocabulary reduction (lmvr), which is an extension of morfessor, gives better results in the context of machine translation. the benefit of using lmvr increases with increased morphological richness. a similar conclusion is reached in a wide-scope multilingual comparison with language modeling as the downstream task is performed by @xcite . a study on english by bostrom and durrett (2020) compares bpe preprocessing with the unigram method by @xcite , again on the task of language modeling, obtaining lower results with bpe tokenization, which also gives a slightly larger vocabulary than the competing method. for our study, we select representative examples of neural models for dialect classification and subword tokenization methods, which are detailed in the next section. ## data and methods four datasets used in the main study have been selected so that they represent different language types and different levels of consistency in writing. the gdi dataset is compiled from the archi-mob corpus of spoken swiss german and covers four areas, namely basel, bern, lucern, and zurich @xcite . we used the gdi-2018 dataset for our experiments and worked in a 4-way classification setting. the ili dataset includes five closely related indo-aryan language dialects: hindi, braj bhasha, awadhi, bhojpuri, and magahi. for each language, 15,000 sentences are extracted, mainly from the literature domain. the adi vardial task @xcite includes five arabic dialects: modern standard arabic (msa), egyptian (egy), gulf (glf), levantine (lav), moroccan (mor), and north-african (nor). msa is the modern variety of language which is used in news and educational articles. it differs lexically, syntactically, and phonetically from the actual communication language of native speakers. the vardial adi dataset is both speech transcribed and transliterated to english from arabic. aoc constitutes a large-scale repository of arabic dialects extracted from reader commentary of three arabic online newspapers. it covers msa and the dialectal varieties, viz., egyptian (egy), gulf (glf), levantine (lev), and moroccan (mor). the languages and dialects represented in these four datasets belong to two language families (indo-european and semitic). two of the data sets (gdi and adi) are created by transcribing spoken language and show a high level of inconsistency in writing. the other two (ili and aoc) are originally written texts with lower level of inconsistency. in addition to these four datasets used in the main study, we perform additional experiments on the data from the nuanced arabic dialect identification (nadi) shared task, which deals with countrylevel and province-level arabic dialect identifications @xcite . nadi 2022 shared task covers 18 country dialects with a training set of ≈ 20k tweets @xcite . ## experimental settings we train and test on the task of dialect classification each of the architectures described in section 3.4 on each version of the data produced with the tokenizers (one version of the data for each vocab_size). the vocabulary size that gave the best performance on the development set is chosen as the optimal vocabulary size. we compare these results to find out if optimizing the input vocabulary improves the classification performance. in addition to this, we compare the performance achieved with the best performing models trained from scratch with the performance achieved by fine-tuning respective pre-trained models. in the remainder of this section, we describe the hyperparameters of the neural models trained from scratch, the vocabulary settings, and the fine-tuning settings, which we consider to be the state of the art. foot_6 ## results and comparisons since the kim_cnn model gave the best results in all the from-scratch settings, we report only its performance in from the f1 score was observed to increase by 4.9 points in gdi, 2.7 points in ili, 3.1 points in aoc, 2.2 points in adi and 0.7 points in nadi dataset. the optimal vocab_sizes are also reported, corresponding to vocab_size that gave the maximum f-scores. the variation with respect to different vocab_sizes in each dataset for the kim_cnn with the unigram model tokens is shown in appendix d, figure d1 . from these results, we conclude that optimized subword-level tokenization gives better dialect classification performance across all data sets (different languages, different levels of consistency) when working with a cnn architecture trained from scratch. similar observations hold for all the nontransformer neural models in table c1 in appendix c. ## conclusion we have shown in this paper that optimizing subword vocabulary size is beneficial to text classification tasks, such as dialect classification, when the datasets contain relatively inconsistent writing (transcribed speech). with an optimized vocabulary as input, a cnn model trained from scratch outperforms fine-tuned models on such datasets. on the other hand, fine-tuning large language-specific pretrained models seems to be the best approach when datasets are relatively consistent (originally written, even if not edited). in this case, vocabulary size does not seem to matter much. regarding the question of which kind of neural architecture is best to use without pretraining, our results point to the cnn architectures, which seem to capture the relevant surface features effectively. established on a relatively diverse sample (three language types from two language families), our findings are especially relevant to multilingual nlp, where datasets tend to be inconsistent and the use of pre-trained models tempting.