A reproduction of Apple’s bi-directional LSTM models for language identification in short strings (2021)

Abstract: Language Identification is the task of identifying a document’s language. For applications like automatic spell checker selection, language identification must use very short strings such as text message fragments. In this work, we reproduce a language identification architecture that Apple briefly sketched in a blog post. We confirm the bi-LSTM model’s performance and find that it outperforms current open-source language identifiers. We further find that its language identification mistakes are due to confusion between related languages.

## introduction automatic language identification is the task of identifying a document's language, an essential task for document classification and machine translation @xcite . generalpurpose, open-source language identification tools like langid.py @xcite and @xcite are the de facto standards for language identification in large documents. during the last two decades, text messaging and social media have generated large amounts of short plain-text documents. language identification on partial and complete short texts presents unique challenges @xcite . successful language identification can support marketing, political, and socioeconomic analyses on large corpora of short texts such as tweets. such analyses can, for example, study hate speech towards immigrants and women @xcite or seek to understand support groups for smoking cessation @xcite . on a smartphone, language identification on short texts can support several features. language identification of incoming text messages can help virtual assistants read incoming text messages, * equal contribution which can be an essential tool for minorities such as visually impaired multilingual speakers. language identification can also help when typing short texts. identifying language from the first few characters typed (a very short string) can allow a smartphone to select the correct spelling and grammar checker automatically. such features motivated a team at apple to study characterlevel language identification using bi-directional lstms @xcite . this paper reproduces the architecture presented in an industry blog post @xcite on language identification on extremely short strings (10 characters or less). the blog post briefly sketches the language identification system used by apple's smartphones and computers. however, due to the use of internal, proprietary corpora, the architecture's performance cannot be compared with the current de facto standards for language identification: the open-source tools langid.py @xcite and fasttext @xcite @xcite . our reproduction confirms the performance described in the original blog post @xcite . we go beyond mere reproduction and (1) compare the bi-lstm model with the current de facto standards for language identification and ( 2 ) analyze performance on related languages. we find that the bi-lstm is more accurate than out-of-the-box fasttext and langid.py, even outperforming the re-trained fasttext. our results suggest that the bi-lstm architecture could be an alternative to fasttext and langid.py for language identification on short strings. 1 2 related work the simplest language identification methods discriminate using elementary distinguishing traits like unique character combinations, frequent or unique words, diacritics, or common n-grams @xcite @xcite . increasing model complexity, some language identification methods model sequences of words, characters, or bytes. some methods focus on modeling the frequency of n-grams, e.g., frequency of character n-grams @xcite . such methods outperform techniques based on unique words. markov model-based approaches estimate the probability of a string based on n-grams of characters or bytes @xcite , as is the case of langid.py @xcite . due to its availability as an open-source library, langid.py is one of the most popular language identifiers. recent language identifiers increasingly use word representations. for example, in a blog post, grave (2017) shows how to identify languages using fasttext vectors @xcite @xcite , which model character n-grams. language identification with fasttext vectors is as performant as langid.py @xcite . similar to langid.py, fasttext language identification models are open-source and, therefore, popular. lanidenn @xcite identifies languages in multilingual documents using a recurrent neural network with a single layer of gated recurrent units (gru). unlike markov-based methods, recurrent neural network architectures do not model character sequences with a fixed window of context. the language identifier that apple briefly sketched in a blog post @xcite uses a recurrent neural network with a two-layer bidirectional lstm to model character sequences. apple's method differs from lanidenn in architecture complexity (two layers, lstm cells instead of the simpler gru cells) and in its focus. lanidenn works with long multilingual documents, whereas apple classify extremely short monolingual strings. in a survey, @xcite present more than the techniques above, discuss challenges, and identify remaining research questions. among the remaining research questions are very short texts (the problem motivating apple) and discrimination of related languages. in this paper, we go beyond reproducing apple's work by analyzing the effect of related languages. ## model architecture figure 1 gives an overview of the two-layer bidirectional lstm architecture powering apple's products, as briefly sketched in a blog post @xcite . the model takes as input strings of characters. in the following, we describe the left-to-right direction of the bi-directional lstm. the right-to-left direction is identical but mirrored. in the first step, vector embeddings replace all characters in the input string. the network uses a single embedding for all languages since the language is unknown at this point. at each time step, the lstm ingests a character's embedding and the hidden layer representation from the previous step. the per-character output from the left-to-right lstm layer is concatenated with that of the right-to-left layer. the concatenated vectors pass to a second lstm layer that is identical to the first but does not share parameters. after the second layer, the concatenated vectors go through a single linear layer, producing a distribution over all supported languages. the linear layer provides character-level language identification. in other words, for each input character, the network generates a probability distribution over the possible languages. with the outputs from the linear layer, apple (2019) state that a max pooling style majority voting decides the dominant language of the string. however, max pooling and majority voting are dif-ferent techniques. a combination of the two is impossible as one cannot perform majority voting over outputs that have been max pooled, and vice versa. instead, we sum over the linear layer's output values at each time step and softmax the summed output to obtain a prediction. we expect this approach to be what the original authors intended. the similarity between our reproduction's performance and what apple report in the original blog post confirms our approach. apple test on 20 languages that use the latin alphabet, but only show results on 9 of the 20 and do not specify the remaining 11 languages. besides the 9 languages in the original blog post, we select 11 languages, some of which are closely related. thus, our experimental setup foot_1 is similar to apple's. including closely related languages increases our data sets' difficulty but supports more interesting and more representative experiments. specifically, it supports performance analysis on related languages, an open research question @xcite . ## experiments and results we use five-fold cross-validation in all experiments. following apple (2019), we evaluate on strings of 10 characters. we test all models on the same strings. we use the adamw optimizer with default parameters in pytorch; we set the character embedding dimension to 150 and the bi-lstm's hidden dimension to 150; we train for 25 epochs using batches of 64 examples and use weighted crossentropy for the loss function. out-of-the-box, fasttext and langid.py can identify more than our set of 20 languages. for fair evaluation, we limit the set of languages that the models output. for langid.py, we use a built-in method that limits the number of languages under consideration. for fasttext, we take the probability distribution over all language predictions, extracting only the relevant 20. we use the large pre-trained fasttext model foot_2 . when re-training fasttext, we use 15 epochs, with a minimum n-gram length of one character and a maximum of six characters. we leave all other parameters at their default. while in one case, accuracy is almost identical (turkish, tr), for most languages, our implementation is either a few points of accuracy below (e.g., french, fr, -2.85 points, and italian, it, -2.62) or above the original model (e.g., dutch, nl, +1.87). for some languages, our implementation considerably underperforms the original (e.g., english, en, -7.4 points, and spanish, es, -16.7). our implementation considerably outperforms the original on german (de +6.91) and swedish (sv +7.54). we attribute the difference in performance to randomness during training and differences in training data. the original blog post does not state the size nor language composition of the data set. ## conclusions we have reproduced the bi-lstm language identification architecture described in a blog post by @xcite . our reproduction experiments confirm the performance claims in the original blog post. we evaluated the bi-lstm against the de facto open-source language identifiers in experiments on two openly available data sets. our evaluation considered dialog and prose, and targeted twenty languages, including some highly similar languages such as danish (da) and norwegian (no) or catalan (ca) and spanish (es). our experiments illustrate the difficulty of identifying the language in very short strings. the reproduced bi-lstm outperformed fasttext and langid.py on all measures, even when training fasttext on the same data. however, we went beyond a straightforward reproduction and considered related languages. our analysis shows that the bi-lstm can easily confuse languages from the same family (e.g., romance, west germanic, or scandinavian) and highly similar languages such as catalan (ca) and spanish (es). we publish our implementation's source code and make a trained model available as a library. in the future, we would like to consider avenues for improving the bi-lstm architecture. for example, we would like to replace the majority voting mechanism in the bi-lstm with a more robust alternative.