Transfer Learning Improves F rench Cross-Domain Dialect Identification: NRC @ V ar D ial 2022 (2022)

Abstract: We describe the systems developed by the National Research Council Canada for the French Cross-Domain Dialect Identification shared task at the 2022 VarDial evaluation campaign. We evaluated two different approaches to this task: SVM and probabilistic classifiers exploiting n-grams as features, and trained from scratch on the data provided; and a pre-trained French language model, CamemBERT, that we fine-tuned on the dialect identification task. The latter method turned out to improve the macro-F1 score on the test set from 0.344 to 0.430 (25% increase), which indicates that transfer learning can be helpful for dialect identification.

## introduction this paper describes the nrc team's submissions to the french cross-domain dialect identification (fdi) task that was organized as part of the evaluation campaign at vardial 2022. for this task, participants had to "train a model on news samples collected from a set of publication sources and evaluate it on news samples collected from a different set of publication sources. not only the sources are different, but also the topics. therefore, participants have to build a model for a cross-domain 4-way classification by dialect task, in which a classification model is required to discriminate between the french (fr), swiss (ch), belgian (be) and canadian (ca) dialects across different news samples." 1 our main motivation to participate in this shared task was that it would allow us to compare finetuning of a pre-trained neural language model to n-gram based methods trained from scratch, which have been successful at discriminating between similar languages (dsl) in the past. this was not possible in many shared tasks on dsl in the 1 https://sites.google.com/view/ vardial-2022/shared-tasks past, at least not since transfer learning became a common approach to various nlp tasks, with the advent of models such as bert @xcite , gpt @xcite , etc. so we took this opportunity to investigate whether dsl is also an area where transfer learning can improve accuracy. we submitted three runs each to the closed and open tracks of the fdi shared task. our closed submissions ended up achieving the highest scores in that track, and we were the only team to submit to the open track. our open submissions outperformed the baselines computed by @xcite as well as our closed submissions, which indicates that transfer learning can be helpful for discriminating between similar languages, at least when a domain shift is present. ## related work thorough surveys of research on language identification are provided by @xcite and @xcite . language identification is one of the few tasks in natural language processing where deep learning methods have yet to provide convincing gains in accuracy, at least in the context of shared tasks. @xcite pointed out that linear svms exploiting character n-grams as features have been highly successful in shared tasks on language identification. the winning submission by the nrc team to the cuneiform language identification task at vardial 2019 @xcite , which involved seven language varieties written in cuneiform script, was the first time a neural system was ranked first on a language identification shared task @xcite . this system was a character-based bert model trained from scratch. however, we also submitted both n-gram models and deep learning models to the uralic language identification (uli) shared task at vardial 2021 @xcite , and in that case, our best n-gram models outperformed our best bert models. these results cast doubt on whether a deep neural network can reliably produce the best results in settings more representative of real-world applications of language identification, as the uli task involved a total of 179 languages, including pairs of very similar languages. they suggested that the simpler, n-gram based approach was still a very strong baseline. note that all our previous shared task participations that involved deep learning were in a closed setting, so no pre-trained models were allowed. this has usually been the case for shared tasks on language identification in our experience. however, transfer learning has been used for language identification outside of shared tasks @xcite . ## data and task definition the fdi task @xcite requires participating systems to predict the french language variety used in a sample of text. the set of four language varieties that the systems must learn to discriminate are the national varieties used in france, belgium, switzerland, and canada. the evaluation metrics for this shared task were not specified, so we chose to focus on macroaveraged f1-score, which is commonly used for language identification and dsl tasks. this shared task featured both open and closed tracks. for the closed track, participants were not allowed to use pre-trained language models or any external data to train their models. this is the usual setting for dsl shared tasks in our experience. for the open track, external resources such as unlabelled corpora, lexicons, and pre-trained language models were allowed, but no additional labelled data could be used. thus, this shared task provided us a unique opportunity to evaluate transfer learning on a dsl shared task. @xcite describe the corpus they developed for this task, which they named frecdo (for french cross-domain [dialect identification]). this corpus contains 413,522 text samples collected from public news websites. the ca class is under-represented in the dataset, as fewer open sources were available. as we will show below, the presence of duplicates makes this class imbalance even greater. efforts were carried out to eliminate potential biases related to factors such as topic and writing style. this was done by using separate sets of publication sources and search keywords to compile the training, validation (aka development), and test sets. the keywords represent general topics that are not specific to any of the four countries involved. the keywords were: "guerre" ("war") and "ukraine" for the training set; "russie" ("russia") and "états-unis" ("united states") for the development set; and "réchauffement climatique" ("global warming") and "covid" for the test set. note that there is likely more topical similarity between the training and development set, than between either and the test set, so the development set may not be a good estimator of test accuracy, which is confirmed by our experiments below. furthermore, named entities were identified using spacy foot_0 and replaced with the special token "@xmath0", again in order to remove biases related to topic or country. the training, development and test sets contain 358,787, 18,002, and 36,733 samples respectively. each text sample is a paragraph containing up to three sentences. @xcite also evaluated three baseline systems on this corpus and concluded that it is a difficult task. their baseline models were able to outperform a naive baseline that always selects the most frequent class, but macro-averaged f1 scores did not exceed 0.4. it turns out that one of those baselines was a finetuned camembert model, which is the model that we used for the open track, although we were not aware of this before submitting our runs. that baseline produced the best results, and the runs we submitted outperformed this baseline by a few points (in terms of macro-f1). this may be due to differences in hyperparameter settings, or to the fact that we used the development set along with the training set to fine-tune the model. whether this was done by @xcite is not specified, so we would tend to assume it was not. they also evaluated svm and xgboost models based on the text encodings produced by a finetuned camembert model, but the best results were achieved by camembert itself. their results were much better on belgian and swiss french than on the other two varieties. @xcite analyzed the most discriminative features of the camembert model, by manually inspecting "a few correctly classified samples [and analysing] the features for which camem-bert has given high scores." they concluded that "there are quite a few noticeable dialectal patterns learned by the model," such as numerals only used in belgian french and a currency only used in switzerland. ## methodology in this section we will explain how we processed the data and trained the models that we used for our submissions to the fdi task. ## results on test set the official scores of our 6 runs on the test set are shown in looking at the confusion matrices of each of our runs, we observed that our open runs did quite a bit better on the ca class, getting up to 157 cases right (run 3), whereas the closed runs all got a single ca case right. the confusion matrix of our best run on the test set is shown in fig. 1 . to get a fuller picture of the results, we investigated various potential sources of errors. first, we looked at the class-wise f1 scores of open run 1 and how they relate to the class fre-quency distribution of the training data, and observed an obvious correlation between the two. another factor that can impact the accuracy of language identification systems is the length of texts. to investigate this, we binned the test examples by length (after removing redundant ne tokens) into 10 bins of approximately equal sizes, and computed the macro-f1 and accuracy for each bin, using the predictions of our best model (open run 1). the results, shown in we also checked whether test cases that were also present in the training data had the same label, and whether our best model (open run 1) got them right. the examples we inspected included the following: in the training data, it appeared in 3 classes: be, ch, and fr. we also inspected the examples where our 6 submissions disagreed the most, and found several examples containing boilerplate such as "vous avez lu 29 des 432 mots de cet article", on which all 4 possible classes were among the predictions of our 6 systems. this boilerplate pattern is also present in a lot of the most likely ca examples in the test set according to our best camembert model, although it generally does not belong to the ca class in the training or test data. we can not provide an explanation for this, but perhaps the lack of diversity of ca examples in the training data is the cause, as well as the frequency of such boilerplate in all classes. one possible reason for the superior performance of camembert is its subword tokenizer. we tokenized the dataset, then trained svm and probcat models on the camembert tokens, using token n-grams (with n between 1 and an upper bound that we raised up to 5) as features. none of the model fared better using camembert tokens, so the superior performance of camembert must be attributable to its pre-trained token embeddings and encoder weights. to explore how camembert's performance might be improved, we checked how many outof-vocabulary tokens, which are represented by "<@xmath1, are produced by camembert's tokenizer on the test set. less than 1% of test examples (342) contain any "<@xmath2, so this is probably not an important source of errors, and expanding the vocabulary of the camembert tokenizer and model seems unlikely to lead to significant gains. on the whole, the analysis presented in this section seems to say more about the properties of the data than it does about the behaviour of our models, and does not point to any obvious means to improve predictive accuracy, as far as we can tell. ## conclusion for the french cross-domain dialect identification shared task at the 2022 vardial evaluation campaign, the nrc team evaluated two different approaches: svm and probabilistic classifiers using n-gram features and trained from scratch on the data provided; and a pre-trained camembert model fine-tuned on that data. the latter increased the macro-averaged f1 score on the test set from 0.344 to 0.430 (25% increase). this indicates that transfer learning can be helpful for dialect identification, and provides clear evidence that neural models can be effective at such tasks, at least when they are pre-trained on large amounts of unlabelled text.