V ar D ial in the Wild: Industrial Applications of LID Systems for Closely-Related Language Varieties (2023)

Abstract: This report describes first an industrial use case for identifying closely related languages, e.g.dialects, namely the detection of languages of movie subtitle documents. We then presenta 2-stage architecture that is able to detect macrolanguages in the first stage and languagevariants in the second. Using our architecture, we participated in the DSL-TL Shared Task of the VarDial 2023 workshop. We describe the results of our experiments. In the first experiment we report an accuracy of 97.8% on a set of 460 subtitle files. In our second experimentwe used DSL-TL data and achieve a macroaverage F1 of 76% for the binary task, and 54% for the three-way task in the dev set. In the open track, we augment the data with named entities retrieved from Wikidata and achieve minor increases of about 1% for both tracks.

## introduction in the nlp community the problem of identifying languages of documents is often perceived as being solved @xcite , also due to the good accuracy of this function in tools like google translate. this is especially true for many users as they apply this method in cases where they want to understand text that is not their field of native speaker expertise. however, when applying state of the art language identification tools to applications where an accurate distinction of closely related languages, e.g. dialects is important, it soon becomes clear that these tools often either do not offer variants in their list of covered languages or confuse them regularly. one of these application areas are movie subtitles. as we will see in the next section, although these texts are typically not too small for language identification, they often differ from news domain content, which is the source of the shared task data. section 3 will describe the architecture of our system. using this architecture, we participated in the dsl-tl shared task of the vardial 2023 workshop. we describe this shared task briefly in section 4. our experiments of our system on this shared task and other data can be found in section 5, while section 6 presents a manual oracle experiment that aims at finding out how much an extended ner-like mechanism can reduce errors. finally, section 7 conclude our findings. ## lid for subtitles subtitle files contain the closed captions or subtitles of movies and similar video content. these files come in a variety of different, partially proprietary formats (see the content consists typically of a mix of time stamps, dialogue lines, textual descriptions of visual content, and symbols, e.g. for music (see fig. 1 for an example extract of such content). in order to cope with the diversity of formats, and to extract the textual parts in the target language, a preprocessing stage is needed. afterwards, utf-8-encoded text can be fed into the language identification stage. in our experience the resulting subtitle text documents have a median file size of about 25 kbytes. subtitles and closed captions are meant to be displayed over the video for a certain amount of time (hence the time stamps). the displayed text for this period can contain either single words, parts of sentences, or multiple sentences. these text portions are reflected by line breaks in the document, i.e. line breaks separate different time periods. apart from textual descriptions and symbols, texts transcribe mainly the spoken dialogue. dvd and bluray releases of video content often come with a number of subtitles in different languages and language variants. foot_0 also, releases of these media in different regions or countries come with different sets of subtitle languages. as a result, for a single piece of video content, many different subtitle documents exist. if one imagines that e.g. for a single movie, different versions of that movie are needed even in a single language (realizing different ratings, different cuts, or being trailer versions), the number of subtitles documents over an entire catalogue of movies is very large. ideally, a digital asset management system denotes the language of a single subtitle file. however, in reality, movie studios have a very large catalogue of content that partially predates the widespread availability of low-threshold asset management platforms. this means that there are many subtitle files on many disks in many drawers of which no exact metadata is known. this is where language identification really helps as it avoids the need of re-creating subtitle files for existing movies if they are about to be re-released. also, it helps to verify existing language metadata as these might be incorrect. the user of such a tool will be typically a technician, not a linguist. also, the user will probably have to face a lot of different languages, some of which will be very foreign to the user. finally, it will be very difficult for the user to get a gold language label for a file using standard tools. subtitles are relatively mono-lingual; they might contain a moderate amount of code-switching and the occasional sentence in another language. ## system architecture the requirements for the use case mentioned in section 2 asked for a system that could not only recognize macrolanguages, but also language variants. an earlier internal language identification system implementation based purely on character n-grams and perplexity proved to be especially weak when it comes to closely-related languages and dialects. consulting the literature @xcite we decided in favor of a two-stage system, where the first stage aims at recognizing a "language group". for us, language groups are groups of closely related languages that are difficult to distinguish in the first stage. language groups can be macrolanguages that contain e.g. different language variants, or simply a set of languages that are hard to tell apart for a stage 1 algorithm. if the language in stage 1 is marked in a list as being a member of a certain language group, a stage 2 takes over and aims at determining the concrete member of the language group. using this architecture, not all language group member languages need to be recognized in stage 1. let's now have a closer look at the two stages (see figure 2 ). ## experiments on subtitle files one of the original use cases for our system is subtitle file language identification. ## experiments on dsl-tl data now we will describe our experiments for the dsl-tl shared task. until mentioned otherwise, all stages have been trained on dsl-tl training data. in section 6.2 the system will be tested on the dsl-tl dev set, in section 6.3 on the dsl-tl test set. ## country-informed ner oracle performance we were wondering how much a slightly different approach would fare using an ner-like system that could tell one the country a named entity is typically associated with. one (probably inefficient) way to implement this system would be to google a maximum-length named entity candidate (the english dsl-tl data is very consistently capitalized in terms of named entities) and to take the first country reference that is found in the results (maybe normalizing the result, e.g. from "english" to "uk"). when a knowledge panel appears in the results, this information should first be taken into account. as we cared only about oracle performance, we did this process manually. in order to reduce the english dev set sentences to manually label this way, we only looked at the 92 sentences (from 599) our automatic method from above incorrectly predicted. for each of these sentences, we marked the capitalized proper nouns and then started a google search. we also looked at mentioned currencies. an example of this data can be found in when comparing the labels from the oracle mechanism to the dsl-tl labels, we found four groups: sometimes we could also trace the sentence to a newspaper from a certain country. this opinion is obviously based on our belief of the intention-based criteria for documents of a language variant. therefore, the reader might either count this group as errors or as correct cases. we discussed our suspicion that the labelers are heavily using their knowledge of the country of named entities in order to come to a label. for this group, we basically claim that the labelers did not follow that suspicion. ## conclusion we reported on our use case for language identification, namely movie subtitles. for movie subtitles there is a need to also recognize close languages and variants. we presented the 2-stage architecture of our language identification system that uses a second stage if a language is identified in the first stage that is marked as being a member of a "language group". we reported on the results of our experiments. in the first experiment we reported an accuracy of 97.8% on a set of 460 subtitle files. in our second experiment we used dsl-tl data and achieved for the dev set a macro-averaged f1 of 54% in the three-way classification task, and 76% in the binary classification task, where we see an increase in performance by adding named entities retrieved from a knowledge base. on the dsl-tl test set for the closed task we achieved a macro-averaged f1 of 48% in the three-way classification task, and 76% in the binary classification task. on the open task, we achieved 49% for the three-way, and 77% on the binary task. we reported on a small experiment using a manually executed "country-informed" ner on those sentences of the english dsl-tl dev set that were incorrectly predicted by our system. we did this in order to see how much head room remains in the ner-based approach to identify dsl-tl data as some sort of oracle data. as it turns out, this mechanism can reduce the number of errors by at least a third. as future work we intend to examine the question whether it is better to keep language group languages separated for training stage 1 or whether the data for a language group should be mixed together before training in order to achieve a better stage 1 performance. there are two aspects that are not clear to us when it comes to the dsl-tl dataset, and that might lead to future research. first, how strongly influence named entities the human labellers and is this detrimental to the label accuracy? to quote @xcite [...] named entities [...] can influence [...] also the performance of human annotators. second, again @xcite mentions that a general tendency we observed is that it is easier to identify an instance that is not from the speaker's own language than the opposite. our results indicate that humans are better in telling what is not a text written in their own language or variety than telling what it is. we understand from the description of the new annotation process that a labeller could annotate sentences in his/her own, or another variant. as the ratio of native speakers for the corresponding variants seems not to be mentioned, it is hard to assess the influence of this aspect on the results. maybe it would be worthwhile to trying to evaluate the accuracy of the new labels. this is, of course, not easy as then another, more accurate process would be needed to come to "platinum" labels.