SIGTYP 2021 Shared Task: Robust Spoken Language Identification (2021)

Abstract: While language identification is a fundamental speech and language processing task, for many languages and language families it remains a challenging task. For many low-resource and endangered languages this is in part due to resource availability: where larger datasets exist, they may be single-speaker or have different domains than desired application scenarios, demanding a need for domain and speaker-invariant language identification systems. This year’s shared task on robust spoken language identification sought to investigate just this scenario: systems were to be trained on largely single-speaker speech from one domain, but evaluated on data in other domains recorded from speakers under different recording circumstances, mimicking realistic low-resource scenarios. We see that domain and speaker mismatch proves very challenging for current methods which can perform above 95% accuracy in-domain, which domain adaptation can address to some degree, but that these conditions merit further investigation to make spoken language identification accessible in many scenarios.

## introduction depending on how we count, there are roughly 7000 languages spoken around the world today. the field of linguistic typology is concerned with the study and categorization of the world's languages based on their linguistic structural properties @xcite . while two languages may share structural properties across some typological dimensions, they may vary across others. for example, two languages could have identical speech sounds in their phonetic inventory, yet be perceived as dissimilar because each has its own unique set of phonological rules governing possible sound combinations. this leads to tremendous variation and diversity in speech patterns across the * equal contribution world languages (tucker and wright, 2020), the effects of which are understudied across many downstream applications due in part to lack of available resources. building robust speech technologies which are applicable to any language is crucial to equal access as well as the preservation, documentation, and categorization of the world's languages, especially for endangered languages with a declining speaker community. unfortunately, robust (spoken) language technologies are only available for a small number of languages, mainly for speaker communities with strong economic power. the main hurdle for the development of speech technologies for under-represented languages is the lack of highquality transcribed speech resources (see @xcite for a detailed discussion on linguistic diversity in language technology research). the largest multilingual speech resource in terms of language coverage is the cmu wilderness dataset @xcite , which consists of read speech segments from the bible in ∼700 languages. although this wide-coverage resource provides an opportunity to study many endangered and underrepresented languages, it has a narrow domain and lacks speaker diversity as the vast majority of segments are recorded by low-pitch male speakers. it remains unclear whether such resources can be exploited to build generalizable speech technologies for under-resourced languages. spoken language identification (slid) is an enabling technology for multilingual speech communication with a wide range of applications. earlier slid systems addressed the problem using the phonotactic approach whereby generative models are trained on sequences of phones transduced from the speech signal using an acoustic model @xcite . most current state-of-the-art slid systems are based on deep neural networks which are trained end-to-end from a spectral representation of the acoustic sig-nal (e.g., mfcc feature vectors) without any intermediate symbolic representations @xcite . in addition to their ability to effectively learn to discriminate between closely related language varieties @xcite , it has been shown that neural networks can capture the degree of relatedness and similarity between languages in their emergent representations @xcite . several slid evaluation campaigns have been organized in the past, including the nist language recognition evaluation @xcite , focusing on different aspects of this task including closely related languages, and typically used conversational telephone speech. however, the languages were not sampled according to typologically-aware criteria but rather were geographic or resource-driven choices. therefore, while the nist task languages may represent a diverse subset of the world's languages, there are many languages and language families which have not been observed in past tasks. in this shared task, we aim to address this limitation by broadening the language coverage to a set of typologically diverse languages across seven languages families. we also aim to assess the degree to which single-speaker speech resources from a narrow domain can be utilized to build robust speech language technologies. ## task description while language identification is a fundamental speech and language processing task, it remains a challenging task, especially when going beyond the small set of languages past evaluation has focused on. further, for many low-resource and endangered languages, only single-speaker recordings may be available, demanding a need for domain and speaker-invariant language identification systems. we selected 16 typologically diverse languages, some of which share phonological features, and others where these have been lost or gained due to language contact, to perform what we call robust language identification: systems were to be trained on largely single-speaker speech from one domain, but evaluated on data in other domains recorded from speakers under different recording circumstances, mimicking more realistic low-resource scenarios. ## evaluation the robust language identification shared task allowed two kinds of submissions: first, constrained submissions, for which only the provided training data was used; and second, unconstrained submissions, in which the training data may be extended with any external source of information (e.g. pretrained models, additional data, etc.). ## results and analysis note that most systems submitted are powerful enough to fit the training data: our baseline achieves a macro-averaged f 1 score of .98 (±.01) on the training data, the lipsia system similarly achieves .97 (±.03), the ntr system reaches a score of .99 (±.02). an outlier, the anlirika system reaches only .75 (±.09). on held-out data from cmu wilderness which matches the training data domain, the baseline achieves .96 f1. this suggests an inability to generalize across domains and/or speakers without additional data for adaptation. diving deeper into performance on different languages and families, figure 2 shows confusion matrices for precision and recall, grouped by language family. we can see the superiority of the lipsia 6 each of the "wins" indicated by boldface in iso anlirika baseline lipsia ntr valid. test valid. test valid. test valid. test family: afro-asiatic .329 .214 .181 .235 .670 .453 .102 .082 kab .329 .214 .181 .235 .670 .453 .102 .082 family: austronesian .429 .368 .082 .094 .578 .498 .065 .060 iba .692 .696 .029 .018 .980 .968 .020 .031 ind .350 .108 .033 .105 .700 .338 .096 .074 sun .406 .369 .160 .149 .090 .140 .086 .082 jav .267 .300 .106 .106 .540 .547 .059 .053 family: basque .565 .405 .100 .090 .850 .792 .077 .016 eus .565 .405 .100 .090 .850 .792 .077 .016 family: dravidian .351 .246 .202 .138 .807 .572 .074 .053 tam .342 .272 .348 .204 .800 .609 .172 .046 kan .188 .168 .000 .042 .820 .557 .004 .015 tel .523 .298 .259 .168 .800 .550 .046 .097 family: indo-european .439 .225 .130 .144 .722 .402 .114 .047 hin .458 .378 .091 .099 .780 .635 .021 .011 por .211 .143 .157 .166 .550 .358 .102 .068 rus .630 .034 .014 .014 .900 .065 .050 .049 eng .194 .148 .161 .179 .460 .414 .270 .099 mar .701 .423 .229 .263 .920 .539 .126 .010 family: niger-congo .516 .403 .138 .063 .860 .763 .122 .038 cnh .516 .403 .138 .063 .860 .763 .122 .038 family: tai-kadai .362 .156 .086 .052 .780 .401 .025 .015 tha .362 .156 .086 .052 .780 .401 .025 .015 f1, macro avg. .421 .282 .131 .122 .719 .508 .086 .049 f1, micro avg. .436 .298 .145 .137 .532 .063 accuracy 29.9% 13.7% 53.1% 6.3% interestingly, the other three systems all struggle to tell apart sun and jav, the anlirika and baseline systems classifying both mostly as sun and the lipsia system classifying both mostly as jav. note that the baseline system tends to label many languages' examples as sun (most notably mar, the test data for which contains only female speakers), eus (most notably also rus), and eng (most notably also iba), despite balanced training data. in a similar pattern, the anlirika predicts tam for many languages, in particular ind, the other two dravidian languages kan and tel, por, rus, eng, cnh, and tha. looking more closely at the clearly bestperforming system, the lipsia system, and its performance and confusions, we furthermore find that the biggest divergence from the diagonal after the sun/jav confusion is a tendency to label rus as por, and the second biggest divergence is that mar examples are also sometimes labeled as kan and tel; while the first one is within the same family, in the second case, these are neighbouring languages in ## conclusion this paper describes the sigtyp shared task on robust spoken language identification (slid). this task investigated the ability of current slid models to generalize across speakers and domains. the best system achieved a macro-averaged accuracy of 53% by training on validation data, indicating that even then the task is far from solved. further exploration of few-shot domain and speaker adaptation is necessary for slid systems to be applied outside typical well-matched data scenarios.