Challenges in Neural Language Identification: NRC at V ar D ial 2020 (2020)

Abstract: We describe the systems developed by the National Research Council Canada for the Uralic language identification shared task at the 2020 VarDial evaluation campaign. Although our official results were well below the baseline, we show in this paper that this was not due to the neural approach to language identification in general, but to a flaw in the function we used to sample data for training and evaluation purposes. Preliminary experiments conducted after the evaluation period suggest that our neural approach to language identification can achieve state-of-the-art results on this task, although further experimentation is required.

## introduction the goal of the uralic language identification (uli) shared task at vardial 2020 @xcite was to identify and discriminate 29 uralic language varieties. this involves distinguishing the relevant languages from a large set of non-relevant languages and discriminating between these languages. to solve this task, we used an approach similar to the one we developed for the cuneiform language identification shared task at vardial 2019 @xcite . this approach was ranked first in that competition, which was one of the first times a language identification shared task was won by a neural network @xcite . it is a deep learning approach based on character embeddings and a transformer network @xcite trained in a similar fashion to bert @xcite . on this year's uli shared task, we did not have the same level of success using this neural approach to language identification, as our accuracy scores on the held-out test set were significantly lower than the baseline scores. in this paper, we explore the challenges presented by the uli task and explain the reasons for the low accuracy scores achieved by our model in the official evaluation. we then show that neural language identification can produce much better results if trained and tuned correctly. ## task definition and data the goal of this language identification task is simply to identify the language of a given text, which is typically a single sentence. if more than one language is used in the text (e.g. code switching), the main language must be identified. since there are both relevant and non-relevant languages in the test set, the task involves not only discriminating amongst the relevant languages, but also distinguishing them from the non-relevant languages. the shared task was closed, so no data could be used except for the training set provided. the task was subdivided into three tracks. the training and test data is the same for all three tracks, the difference being the way in which evaluation metrics are computed. for tracks 1 and 2, only test cases where either the predicted or true label is a relevant language are evaluated, using macro-averaged and micro-averaged f1 score respectively. for track 3, all test cases are evaluated, using the macro-averaged f1 score. the 29 relevant languages for this task are part of the uralic group of languages, which are spoken mainly in northern eurasia. some of these languages are very under-resourced, and some are already extinct, e.g. kemi saami, which was represented in this task. also included in the test set are 149 other languages considered non-relevant. these include the three largest uralic languages (i.e. estonian, finnish, and hungarian), as well as a wide variety of other languages. in total, 178 languages or language varieties were represented in this task, the highest number covered so far in a language identification shared task. the training set provided contains 646,043 examples for the relevant languages and 63,772,445 examples for the non-relevant ones. thus, there is much less data for the relevant languages, on average. both the relevant and non-relevant portions of the training set are (highly) unbalanced. the class frequencies in the relevant training data range from 19 (kemi saami) to 214,225 (northern saami). the non-relevant portion of the dataset contains anywhere from 10,000 to 3,000,000 examples per class (i.e. language). the training set was compiled from 2 different sources: data for the relevant languages comes from the wanca 2016 corpus @xcite , whereas the non-relevant data was extracted from the leipzig corpus @xcite . the relevant data in the held-out test set comes from the wanca 2017 dataset. for more information on the dataset, the task, and the baseline scores, refer to @xcite . regarding the length of the texts, for relevant languages, 86% of the texts contain 128 characters or less, and only 1.2% are longer than 256 characters. for non-relevant languages, 70% of texts contain 128 characters or less, and 0.02% are longer than 256 characters. note that there are no duplicates in the relevant training data. in the non-relevant training data, less than 0.5% of examples appear more than once (whether in the same language or different languages). ## a neural approach to language identification the neural approach we used for this task is similar to the one we developed for the cuneiform language identification shared task at vardial 2019. the model is a deep neural network which takes sequences of characters as input. the model is trained in two stages: (partially) self-supervised pre-training, and supervised fine-tuning. typically, when training this type of model, the pre-training phase would exploit large amounts of unlabeled text data, then the subsequent fine-tuning phase would exploit a much smaller amount of labeled data. since the uli shared task was a closed one, we used the same training set for both pre-training and fine-tuning. the network is composed of a stack of bidirectional transformer modules @xcite which encode the input sequence. the output of this encoder is fed to one or more output layers, which depend on the tasks used for pre-training and fine-tuning. first, we pre-train the model using 2 pre-training tasks: masked language modeling (mlm) and sentence pair classification (spc). the mlm task is exactly as described by @xcite . the objective is to predict tokens (in this case, characters) chosen at random based on all the other tokens in the text. this produces a model that can predict characters in context for any of the languages represented in the training data (with varying degrees of accuracy), and must therefore have learned the specific patterns of each language (or at least some of them). as for the spc pre-training task, it is similar to the next sentence prediction task proposed by @xcite , but we adapted it for language identification. instead of predicting whether two sequences follow each other in the training corpus, we predict whether two sequences are in the same language. this introduces a supervised signal into the pre-training task (whereas mlm is self-supervised), as we use the labels in the training data in order to label pairs of texts as belonging to the same language or not. learning to predict whether two texts belong to the same language should be helpful to predict the language used in a specific text, when we fine-tune the model on this task later on. thus, the pre-training phase uses two different output layers, one for mlm and one for spc. the first is a simple softmax over the vocabulary, which is composed of characters in this case. the input of this layer is the encoding of a character in the sequence that has been masked beforehand: before feeding the sequence to the encoder, we pick some characters at random and replace them with a special masking token. based on the encoding of a masked token (which encodes information about the other tokens in the sequence), the softmax layer is used to predict the most likely character. for more details on the implementation of masked language modeling, refer to @xcite . the second output layer is a binary classifier which applies average pooling over the final hidden states of the encoder for each of the two input texts (separately), then computes the dot product of the two pooled encodings, then predicts whether the two input texts belong to the same language based on this similarity function. texts whose pooled encodings are similar will therefore be predicted to belong to the same language. note that this bi-encoding approach to spc has a lower computational cost than the cross-encoding approach we used for cuneiform language identification at vardial 2019, where both sequences are concatenated and encoded together. the cost remains quadratic with respect to the length of the sequence, but the sequence is half as long, and we encode two sequences instead of one. once the model is pre-trained, we fine-tune it on the target task, which is to predict the language of each text, using the labeled training data. for this, we use the pre-trained encoder, and add a new output layer for language identification, which is simply a softmax over the 178 languages. the loss function used to learn all tasks is cross-entropy, which is commonly used for multi-class classification problems, averaged across examples in a batch. we did not assign class-specific cost weights, but we did experiment with oversampling of the training data during fine-tuning, which plays a similar role in alleviating class imbalances (see related work in sec. 7). in cases where we used multiple tasks at once for training, losses were summed. the vocabulary (or alphabet) of symbols for which we learn embeddings is composed of every character in the training set whose frequency is greater than some threshold, plus a few reserved symbols (e.g. unknown characters and padding). in the experiments we carried out on our own train/dev/test split, we computed the vocab on the training portion only. using a minimum frequency of 2, this produced a vocab of about 15k characters. input characters that are not part of the vocabulary are mapped to a special symbol for unknown characters. for efficiency, we used a small architecture containing only 8 transformer layers, rather than the 12 layers of the "base" architecture proposed by @xcite . our other hyperparameter settings for pre-training largely follow their recommendations: • optimizer: adam (with β 1 = β 2 = 0.9) note that no pre-processing was applied to the input text (e.g. word or sub-word tokenization, character normalization, etc.). we used a single gpu (nvidia k80 or v100, depending on the experiment) with 12 gb of memory for both pre-training and fine-tuning. note that simply producing the predictions of the model on the held-out test set, containing over 1.5 million examples, took about 8 hours on a v100 (with a batch size of 64 and a maximum sequence length of 256). our code, which exploits the transformers library by huggingface @xcite , is available is at https://www.github.com/gbcolborne/vardial2020 . ## methodology to produce our systems for the uli task, we experimented with various training tasks and hyperparameter settings, and selected the models that seemed to perform best. before we could do this, we first had to create a dev set, as none was provided for this task. a dev set was necessary to tune the hyperparameters of the model and to estimate the accuracy we could expect to achieve on the held-out test set. to get an unbiased estimate of held-out accuracy, we created two dev sets, one used to tune hyperparameters, and the other to evaluate the fully tuned model. we will call the former 'dev' and the latter 'dev-test' when needed to distinguish the two. in the following sections, we explain how we split the training data to obtain these dev sets, then provide an overview of our model selection experiments. ## official results we were the only team who participated in the uli task (perhaps because of some of the challenges we have outlined in this paper), therefore we can only compare our results to the baseline scores computed by the organizers. the baseline scores were computed using the heli method @xcite . our best score on each of the three tracks is shown looking at our precision and recall scores on track 2, we observed that recall was relatively high (0.895), but precision was very low (0.085), indicating that our model was badly over-detecting relevant languages. faced with these results, we set about figuring out what went wrong. was this neural approach simply inadequate for this language identification task or was there some flaw in the way we trained and selected our models? the fact that our dev sets had significantly overestimated our held-out accuracy suggested that the latter explanation was plausible. ## post-evaluation experiments to confirm our suspicion that our dev sets were bad estimators of held-out test accuracy, we evaluated the baseline method, i.e. heli, on our split of the training data. we used a different implementation of heli than the organizers, but our settings were similar to those reported by @xcite . we used the implementation that is publicly available at https://github.com/tosaja/heli , and modified the settings in the heli.java class as follows: we only used lower-case features, all feature count thresholds were increased from the default 1000 to 300,000, the maximum n-gram length was set to 6, and the penalty was set to 7. ## concluding remarks in this paper, we described the systems built by the nrc team for the uralic language identification shared task at the 2020 vardial evaluation campaign. our official results were well below the baseline, and we subsequently identified the reason for this, which turned out to be a flaw in the function we used to sample data for both training and evaluation purposes. the preliminary experiments we were able to conduct after correcting this flaw suggest that a neural network can achieve state-of-the-art results on this language identification task, but further experimentation is required to establish this. we plan on conducting more extensive experiments on our neural approach to language identification, and re-evaluating it on the held-out uli test set if the organizers make it available. we also plan on exploring different classification architectures, in order to improve efficiency and accuracy. future work might also touch on ways to reduce noise in the training data, or its effect on classification accuracy.