N-gram and Neural Models for U ralic Language Identification: NRC at V ar D ial 2021 (2021)

Abstract: We describe the systems developed by the National Research Council Canada for the Uralic language identification shared task at the 2021 VarDial evaluation campaign. We evaluated two different approaches to this task: a probabilistic classifier exploiting only character 5-grams as features, and a character-based neural network pre-trained through self-supervision, then fine-tuned on the language identification task. The former method turned out to perform better, which casts doubt on the usefulness of deep learning methods for language identification, where they have yet to convincingly and consistently outperform simpler and less costly classification algorithms exploiting n-gram features.

## introduction the goal of the uralic language identification (uli) shared task at vardial 2021 @xcite was to identify and discriminate 29 uralic language varieties, among a total of 178 language varieties from various families, given a short text (typically a sentence). this was a re-run of the uli task at vardial 2020 @xcite . we experimented with two different approaches to the uli task. the first is a probabilistic classifier that exploits only character 5-grams as features. the second is a deep learning approach based on character embeddings and a transformer network @xcite , which is first pretrained through self-supervision, then fine-tuned on the uli task, in a similar fashion to bert @xcite . this second approach is essentially an improved version of the one developed by the nrc team for the first run of the uli task (bernier-colborne and @xcite . by improving the sampling functions used to sample training and evaluation data, and making a few other small changes to the model, we were able to achieve much better results. however, our best results were achieved using the simpler approach, based on character 5-grams. in this paper, we explain these two approaches to language identification and compare the results obtained on the uli task. ## data and task definition the uli data and task are described by @xcite . the goal of the task is to identify the language of a short text, typically a single sentence. if more than one language is used (e.g. code switching), the main language must be identified. this was a closed task, so the only data that could be used was the training set provided. the data contain both relevant and non-relevant languages. the 29 relevant languages are part of the uralic group of languages, spoken mainly in northern eurasia. some are very under-resourced, and some are already extinct. besides these, there are 149 non-relevant languages, belonging to various language families. these include the three largest uralic languages, i.e. estonian, finnish, and hungarian. in total, the data covers 178 language varieties, which is the highest number covered so far in a language identification shared task. the training set contains 646,043 examples for the relevant languages and 63,772,445 examples for the non-relevant ones. so there is about 100 times more data for the non-relevant languages, and there are 5 times more non-relevant languages than relevant ones, therefore the number of examples is about 20 times greater for non-relevant languages, on average. both parts of the training set are highly unbalanced. the class frequencies of relevant languages range from 19 to 214,225, and those of non-relevant ones range from 10,000 to 3,000,000. the task was divided into three tracks, the difference being the way in which the evaluation metric is computed. for tracks 1 and 2, the evaluation metrics are the macro-averaged f1 and micro-averaged f1 respectively, and it only considers examples where either the predicted label or the true label is a relevant language. for track 3, the metric is macro-averaged f1, computed over all examples. some of the challenges of this task were highlighted by bernier-colborne and goutte (2020) and @xcite . these include: the presence of low-resource and closely-related language varieties; the large size of the training set; the large and complex class imbalances in the training set; and the absence of an official development set, which, while a valid design choice for this competition, added a degree of complexity. ## models we evaluated two approaches to this task, which we describe in this section. ## experiments to evaluate and optimize our two approaches, we first had to create a development set, as none was provided for this shared task, as mentioned in section 2. ## official results the scores of our best runs on the official test set are shown in ## related work thorough surveys of research on language identification are provided by @xcite and @xcite . language identification is one of the few tasks in natural language processing where deep learning methods have yet to provide convincing gains in accuracy or robustness. simpler classifiers based on character n-grams continue to provide state-of-the-art results on many benchmarks. the winning submission by the nrc team to the cuneiform language identification task at vardial 2019 was the first time a neural system was ranked first on a language identification shared task @xcite . that task involved 7 language varieties, and less complex class imbalances. the results of this uralic language identification shared task cast doubt on whether a character-based deep neural network can advance the state of the art in settings more representative of real-world applications of language identification, such as crawling the web to automatically compile monolingual corpora for low-resource languages. ## conclusion for the uralic language identification shared task at the 2021 vardial evaluation campaign, the nrc team evaluated two different approaches: a probabilistic classifier exploiting only character 5grams as features, and a character-based neural network pre-trained through self-supervision, then fine-tuned on the language identification task. the former method ended up being ranked first on two of the three tracks, and outperformed our neural approach on all three tracks. we do not exclude the possibility that deep learning approaches could improve accuracy or robustness on this task, but the results we were able to obtain within the limited time constraints of this shared task suggest that the simpler, n-gram based approach is still a very strong baseline.