A Few-Shot Multi-Accented Speech Classification for I ndian Languages using Transformers and LLM ’s Fine-Tuning Approaches (2024)

Abstract: Accented speech classification plays a vital role in the advancement of high-quality automatic speech recognition (ASR) technology. For certain applications, like multi-accented speech classification, it is not always viable to obtain data with accent variation, especially for resource-poor languages. This is one of the major reasons that contributes to the underperformance of the speech classification systems. Therefore, in order to handle speech variability in Indian language speaker accents, we propose a few-shot learning paradigm in this study. It learns generic feature embeddings using an encoder from a pre-trained whisper model and a classification head for classification. The model is refined using LLM’s fine-tuning techniques, such as LoRA and QLoRA, for the six Indian English accents in the Indic Accent Dataset. The experimental findings show that the accuracy of the model is greatly increased by the few-shot learning paradigm’s effectiveness combined with LLM’s fine-tuning techniques. In optimal settings, the model’s accuracy can reach 94% when the trainable parameters are set to 5%.

## introduction in this digital era, speech data has become a valuable resource, alongside text data. in the field of speech processing, recent developments in deep learning have made it possible to create end-toend systems for tasks like speech classification and recognition. much of the ongoing research in speech processing focuses on constructing endto-end devices for automatic speech recognition (asr) capable of handling diverse input data and providing accurate transcriptions. while asr systems have shown remarkable performance in many cases, they face challenges when it comes to generalizing and adapting to resource-poor or resourcelimited languages. even though there are multilingual asr and classification systems that have been trained on different indian languages like tamil, kannada, malayalam, telugu, and others, their effectiveness is still lacking. this is due to the fact that speech is highly influenced by a variety of factors, such as the speaker's accent, gender, age, and more, and the lack of data covering all these variations @xcite . apart from gender, the speaker's accent @xcite ) is recognized as the second most influential speech variation that affects the performance of speech recognition systems. an accent typically refers to a unique way of speaking or pronouncing a non-native language by a native speaker, influenced by the speaker's demographic background or geographical location. accent classification, at its core, involves the categorization of regional or demographic accents within spoken language. the speaker's accent classification is considered a preliminary task for enhancing the capabilities of multilingual asr systems. the primary objective of this research is to tackle the aforementioned accent variations by introducing a data-driven approach to address the challenge of multi-accented speech classification. this paper proposes a few-shot learning method for multiaccented speech classification tasks as a means to handle the diverse array of accents effectively. the few-shot learning paradigm was chosen for this work due to its ability to learn from small amounts of data, which emphasizes how important it is for successfully tackling accent classification problems. the whisper asr model serves as the foundational framework for the task of classifying accented speech. in this work, we primarily use the encoder component of the model, to which we attached a classification head, excluding the decoder component. furthermore, we used two of the most popular large language model (llm) adapters, such as quantized low-rank adaptation (qlora) and low-rank adaptation (lora), to make the training and fine-tuning processes efficient and memoryfriendly. in this work, we utilized the data derived from six indian languages, sourced from the indicaccentdb @xcite and nisp @xcite datasets. both of these datasets are significant for their inclusion of multiaccented speech, featuring conversations in english among native speakers. our main contributions are as follows: the paper contribution is detailed in the sections that follow, which are arranged as follows: section 2 describes the related works, while section 3 completely describes the methodology used in this work. section 4 holds the results and discussion, and we conclude the work in section 5. ## related works previous research has extensively investigated how various components of speech change with accents. notably, spectral features like formant frequencies and temporal features such as intonation and duration's exhibit variation with accent @xcite . to automate accent classification, these features have been combined into different statistical models and machine learning techniques. historically, hidden markov models (hmms) and gaussian mixture models (gmms) have been commonly used in accent classification @xcite @xcite . some studies looked at how the number of gmm components affected the performance of classification @xcite , while others compared hmms to support vector machines (svm) @xcite . further studies explored the impact of gmm component numbers on classification performance @xcite . in @xcite , svm has been used to classify arabic and indian accents using the most popular mel-frequency cepstral coefficients (mfccs) as the speech feature. linear models, such as linear discriminant analysis (lda), have also found applications, as seen in the identification of accents in australian english @xcite . conventional statistical models and machine learning models have played a crucial role in accented speech classification. however, deep learning frameworks like deep neural networks (dnns) and recurrent neural networks (rnns) have been widely used in the latest speech recognition and synthesis systems @xcite @xcite @xcite . notably, for the accent classification task, telugu dialect datasets were created and classified using variants of rnn models like lstm, gru, and bilstm with attention layers @xcite . however, there have been fewer studies evaluating neural networks for accent identification @xcite and @xcite . nevertheless, in related areas like language identification (lid), neural networks have been thoroughly investigated @xcite @xcite . as a breakthrough, convolutional neural networks (cnns) and gated recurrent units (grus) @xcite have been combined to classify accents with approximately 6 hours of speech data for resource-poor languages. moreover, transformers have been a crucial breakthrough in both the natural language processing (nlp) and speech processing domains. most of the transformer-based studies @xcite have been mostly conducted on accented speech recognition and not on the accent classification task. recently, the fewshot paradigm has gained more popularity among researchers because of its ability to learn from a limited amount of data, which resolves the issues with neural net networks, which require a lot of data to train the model @xcite . to the best of our knowledge, the efficacy of few-shot approaches in accent classification is still unknown, despite the fact that these approaches have been used in audio processing @xcite @xcite . based on the existing literature, it is evident that while extensive research has been conducted in the field of speech processing techniques, there remains a significant gap in the development of a system that effectively addresses accent variation and performs classification. in our research, we aim to bridge this gap by introducing a novel approach that utilizes a few-shot learning paradigm for accent classification. to the best of our knowledge, this is the first work that poses multi-accented speech classification as a few-shot learning problem to address the diversity in speech variations caused by speakers' accents. this approach is designed to identify the accents of native speakers from spoken non-native english speech datasets. ## conclusion and future works in this work, we aimed to mitigate the impact of accent variation on speech classification systems. our approach leveraged a data-driven method employing few-shot learning to perform multi-accented speech classification across six indian language speakers' english accents. this was achieved by utilizing the indicaccentdb alongside subsets from the nisp and gujarati digits corpora, utilizing the pre-trained whisper asr model. furthermore, we demonstrated the efficacy of llm fine-tuning techniques such as lora and qlora, which make it possible to fine-tune large language models in a manner that is both memoryefficient and computationally efficient. as can be shown in future endeavors will focus on encompassing other speech variations, such as age group and gender, using the few-shot learning paradigm. this approach will be particularly valuable in scenarios where data availability for these diverse attributes is limited, continuing to enhance the robustness of speech classification systems.