On Building Spoken Language Understanding Systems for Low Resourced Languages (2022)

Abstract: Spoken dialog systems are slowly becoming an integral part of the human experience due to their various advantages over textual interfaces. Spoken language understanding (SLU) systems are fundamental building blocks of spoken dialog systems. But creating SLU systems for low resourced languages is still a challenge. In a large number of low resourced language, we don’t have access to enough data to build automatic speech recognition (ASR) technologies, which are fundamental to any SLU system. Also, ASR based SLU systems do not generalize to unwritten languages. In this paper, we present a series of experiments to explore extremely low-resourced settings where we perform intent classification with systems trained on as low as one data-point per intent and with only one speaker in the dataset. We also work in a low-resourced setting where we do not use language specific ASR systems to transcribe input speech, which compounds the challenge of building SLU systems to simulate a true low-resourced setting. We test our system on Belgian Dutch (Flemish) and English and find that using phonetic transcriptions to make intent classification systems in such low-resourced setting performs significantly better than using speech features. Specifically, when using a phonetic transcription based system over a feature based system, we see average improvements of 12.37% and 13.08% for binary and four-class classification problems respectively, when averaged over 49 different experimental settings.

## introduction spoken language understanding (slu) systems form an integral part of any spoken dialog system. a traditional slu pipeline is made up of two modules (figure 1 ) -a speech to text module which converts input audio into textual transcripts, and a natural language understanding (nlu) module which aims to understand the semantic content in the user utterance from the textual transcripts @xcite . the conventional two-module slu pipeline is prone to making speech recognition errors which propagate through the system. to minimize these errors, a lot of recent research has been focused on creating end-to-end spoken language understanding (e2e-slu) systems @xcite . building e2e-slu systems requires an even larger amount of task-specific annotated data when compared to the two-module split slu pipelines @xcite @xcite . while high resourced languages like english are moving towards e2e-slu, the challenges presented by low resourced languages are very different. low resourced languages operate in a regime where we have access to only tens or hundreds of labelled utterances, which are not enough to build robust e2e-slu systems. creating robust automatic speech recognition (asr) systems for low resourced languages is itself a challenge as these require large amounts of manual annotation. for many low resourced languages, we might not even have asr technologies. creating asr technologies for unwritten languages or languages that have only a few hundred or a few thousand speakers alive is not even a viable option. but can we create spoken dialog systems for such languages? in this paper, we present a series of experiments to empirically re-create language-specific and taskspecific low-resourced-ness scenarios and work in the compounded setting where we tackle both challenges at the same time. as we assume language specific low-resourced-ness, we work in a setting where we don't have access to language specific asr systems. one way to tackle this setting is to use an asr system built for a higher resourced language and use the transcriptions generated to perform downstream tasks as used in @xcite . it was later shown in @xcite that using language and speaker independent systems trained on many languages to ex-tract speech features works much better than using asr systems built for a different language, as a different language usually contains a different set of phonemes with a different phone to phoneme set mapping. when this setting is compounded by task-specific low-resourced-ness, we are at an extremely low resourced setting where each data point becomes valuable. to simulate this setting, we pose an i-class intent classification problem (@xmath0, 4) where we have a varying number speakers (s) available for recording training data. each speaker provides only k-utterances per intent for training. in this k-shot setting, we evaluate our system in a granular manner for very small values of s and k. specifically, we evaluate our system for @xmath1, 2, 3, 4, 5, 6, 7 number of speakers, where each speaker records @xmath2, 2, 3, 4, 5, 6, 7 utterances per intent. we evaluate our slu system on robust test sets containing hundreds of utterances collected from multiple speakers which are not present in the training set. we find that using language independent or multilingual speech recognition systems performs significantly better in such low-resourced settings. furthermore, what works even better is to generate a language independent symbolic representation of input speech and create nlu systems for this symbolic representation. this hints that creating slu systems for even extremely low-resourced settings is likely trace conventional slu pipelines where we represent input speech symbolically in the form of text and then build nlu blocks on top of this. the symbolic representation of speech used here is the phonetic transcription. we find that using a phonetic transcription based system is significantly better than using speech features for classification for low-resourced settings. we see average improvements of 12.37% and 13.08% for binary and four-class classification problems respectively, when averaged over 49 different experimental settings, for belgian dutch (flemish) language. apart from english, there are many other spoken dialog datasets available for various languages including french @xcite , @xcite @xcite , chinese mandarin @xcite , sinhala and tamil @xcite , and crosslingual slu datasets exist for english, spanish and thai @xcite . in this paper, we work with two languages -belgian dutch (flemish) @xcite @xcite and english @xcite . one of the major bottlenecks in creating slu systems for low-resourced languages is the creation of asr systems in such low data scenario. this scenario is what we refer to as a language-specific low-resourced setting. previous works have tried to use english-based asr systems for languages like tamil and sinhala. in these sytems, input speech in sinhala/tamil is converted into english script using an english speech recognition system that is then processed by an nlu system @xcite . we use a similar idea as baseline and use wav2vec @xcite to extract speech features for flemish. wav2vec is a self-supervised speech recognition system trained on large amounts of unlabelled speech data which boasts to learn superior language representations for english. in this work, we use wav2vec 2.0 @xcite to extract speech features. a series of recent works @xcite @xcite replace the asr module in the slu pipeline by a universal phone recognition system called allosaurus @xcite . allosaurus is a universal phonetic transcription system that creates language and speaker independent representations of input speech. allosaurus is trained to recognize and transcribe input speech into a series of phones contained in the utterance, providing superior representations of input audio which can also be used for languages linguistically distant from high resourced languages like english. @xcite show that using embeddings generated from allosaurus to encode speech content outperforms previous state-of-the-art methods for sinhala and tamil by large margins, while maintaining high performance on high resourced languages like english (99.08% classification accuracy for a 31-class intent classification problem). but the performance drops as the dataset size decreases and is not optimal for the task-specific low resourced settings that we are dealing with in this paper. to tackle this, we convert input speech into phonetic transcriptions using allosaurus as proposed in @xcite for our compounded low resourced setting. in our paper, we explore a novel and rather unexplored language-specific low-resourced setting compounded with task-specific low-resourced-ness. our aim it to push the limits and demonstrate performance of using existing technologies in extremely low resourced settings, where each data point becomes crucial. ## dataset in our paper, we work with two languages -belgian dutch (flemish) and english. we use two popular slu datasets for our experiments -the fluent speech commands (fsc) dataset @xcite for the english language and the grabo dataset @xcite @xcite for flemish. the primary reason behind the choice of the datasets was that each utterance in the two datasets had clear speaker identities associated with each utterance. our aim is to test true low resourced settings where getting speaker recordings is extremely hard. intent recognition datasets in other languages like french @xcite , 2018), chinese mandarin @xcite , sinhala and tamil @xcite do not maintain speaker identities and hence were not suitable for our work. maintaining a mapping of (anonymized) speaker identities allowed us to create validation and test sets with no speaker overlap with the training set. this allows us to do the most robust evaluation of our systems. moreover, these datasets also allow us to create large test sets such that the results are robust enough to evaluate the system performance and yet have no overlapping speakers with the training set. we choose flemish as our low-resourced language since flemish is not used to train allosaurus or wav2vec 2.0. fsc is a large and well maintained slu dataset for the english language. the dataset contains 19 hours of speech data collected from 97 different speakers. the dataset contains commands suitable for a smart home system. an example command would be asking the system to 'change language to chinese' or to 'turn off the lights in the kitchen'. each utterance has a clear, anonymized speaker identity associated with it. this allows us to create large validation and test sets with no speakers overlap with the training set. the intents chosen for our experiments and the corresponding number of samples in the validation and test sets are shown in the grabo dataset contains 11 speakers and is much smaller than fsc. the dataset consists of commands given to a robot such as 'moving right' or 'drive backwards fast'. we use speaker ids 2-8 to create the training set, speakers 9 and 10 for the validation set, and speakers 11 and 12 for the test set. thus there is no speaker overlap between the training, validation and test sets. the chosen intents and the validation and test set statistics are shown in ## system and model to simulate a language-specific low-resourced setting, we do not use a language specific asr system. we tackle this challenge by exploring two experimental settings. first we use a generic slu pipeline as shown in figure 2 . the first step in this pipeline is to extract speech features. we use wav2vec 2.0 to extract speech features for flemish, which represents using a speech recognition system built for a different language. then, we use the slu system proposed in @xcite as shown in figure 3 . it replaces a language specific asr system with allosaurus @xcite , which is a universal phonetic transcription system. we use allosaurus to convert input speech to phonetic transcriptions. we then build an nlu system from these phonetic transcriptions to perform intent recognition. the model used in this work is very similar to the model used in @xcite which is a character level model built for a sequence of phones generated by allosaurus. the model creates its own embeddings using the annotated task-specific dataset and uses convolutional neural networks (cnn) @xcite to extract contextual information from phonetic input, and a long-short term memory (lstm) @xcite network to make utterance level decision and account for sequential information. this model achieved state-of-the-art intent classification performance for low-resourced languages like tamil and sinhala when used without language specific asr. we keep the model used across experiments constant to identify difference in performance occurring due to difference in feature extraction methods. we reduce the model size to account for the scarcity of data. we use a 256-dimensional embedding layer with just one cnn layer of kernel size 3 and one or two lstm layers of hidden dimension 256 depending on the dataset size. for the case of the generic slu, the embeddings are removed and input feature dimension is dependent on the features extracted. for wav2vec 2.0, the feature dimensions are 768. a detailed description of model architecture is provided in the appendix a. batch normalization (ioffe and szegedy, 2015) layer is removed because there are scenarios where we are working with a training set of as low as 2 samples, which are not enough to learn batch statistics and give unstable performance. ## experiments in this paper, we try to emulate a real world lowresourced data collection scenario. a challenging aspect of building slu systems for low resourced languages is having access to language specific asr systems. to tackle this, we experiment with two alternatives. we first use a speech recognition systems created for a higher resourced language (english) to extract speech features and use those features for intent recognition on flemish data (section 5.1). then, we create an intent recognition system using a phonetic transcription generated by allosaurus (section 5.2). the input audio is converted to language independent phonetic transcriptions, and intent classification is done using the phonetic transcriptions generated. data collection is expensive and difficult, even more so in extremely low resourced languages. for example, canadian indigenious languages like inuktitut or siksika have only a few thousand living speakers. native speakers of such languages are hard to catch hold of for data collection process. this makes every data point collected crucial. this task-specific low-resourced setting compounds the difficulty in making speech technologies for lowresourced languages. we pose two i-class intent classification problems, where @xmath3, 4. the columns of each of the tables 2-9 in the following sections show results for different values of k, where k is the number of utterances recorded by a speaker per intent. this means that if @xmath4, each speaker provided 3 recordings for each intent, which amounts to a total of 3 * i recordings per speaker. in general, each speaker records k * i audios, where k is the number of audios recorded by a speaker per intent, and i is the number of intents. the rows for each of the tables represent the number of speakers (s) involved in creating the dataset. the total training dataset size is s * k * i. all data points in all the following tables represent an average classification accuracy over 3 different random selections of dataset and training the model from scratch on top of it. ## conclusion in this paper, we provide a series of experiments to empirically recreate a real-world, low-resourced, slu system building scenario. we work in the compounded setting of language-specific lowresourced-ness and task-specific low-resourcedness. the challenge posed by a language-specific low-resourced setting is the absence speech recognition technologies. we bypass this in two waysfirstly, we use a speech recognition system built for a different higher resourced language. secondly, we use a universal phone recognition system to convert input speech to phonetic transcriptions. to simulate the task-specific low-resource scenario, we present intent classification results at a granularity where we see the effects of changing the number of speakers and the utterances recorded by each speaker. we simulate these settings for belgian dutch (flemish) and english. we find that using allosaurus, a universal phone recognition system that creates language and speaker independent representations of input speech, performs better than using wav2vec for flemish dataset. when using allosaurus, we convert input speech into phonetic transcriptions and use these transcriptions to build nlu models. we find that using phonetic transcription based model performs better than using wav2vec features. for flemish, we see an average improvement of 12.37% for a binary classification problem and an average improvement of 13.08% for a four-class classification over using wav2vec features, when averaged over 49 different experimental settings. all results are calculated on a large test set containing hundreds of utterances that has no speaker overlap with the training or validation set. also, we find that as the dataset size decreases, phonetic transcription based method consistently outperform wav2vec feature based methods. phonetic transcription based models also need fewer speakers to generalize to a test set with no speaker overlap. finally, we recommend converting input speech into phonetic transcriptions as an intermediate step for creating slu systems in such low resourced settings. doing such conversion allows us to create a task-specific embedding space that uses the small annotated dataset most efficiently. disclaimer. this paper was prepared for informational purposes by the artificial intelligence research group of jpmorgan chase co and its affiliates ("jp morgan"), and is not a product of the research department of jp morgan. jp morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. this document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. © 2022 jpmorgan chase co. all rights reserved.