Exploring the impact of noise in low-resource ASR for T amil (2024)

Abstract: The use of deep learning algorithms has resulted in significant progress in automatic speech recognition (ASR). Robust high-accuracy ASR models typically require thousands or tens of thousands of hours of speech data, but even the strongest models tend fail under noisy conditions. Unsurprisingly, the impact of noise on accuracy is more drastic in low-resource settings. In this paper, we investigate the impact of noise on ASR in a low-resource setting. We explore novel methods for developing noise-robust ASR models using a a small dataset for Tamil, a widely-spoken but under-resourced Dravidian languages. We add various noises to the audio data to determine the impact of different kinds of noise (e.g., punctuated vs. constant, man-made vs natural) We also explore the relationship between different data augmentation methods are better suited to handling different types of noise. Our results show that all noises, regardless of the type, had an impact on ASR performance, and that upgrading the architecture alone could not mitigate the impact of noise. SpecAugment, the most common data augmentation method, was not as helpful as raw data augmentation, in which noise is explicitly added to the training data. Raw data augmentation enhances ASR performance on both clean data and noise-mixed data.

## introduction automatic speech recognition (asr) technology is widely used in many modern applications for high-resource languages, such as dictation and personal assistants like amazon alexa and apple's siri @xcite . the success of asr for these applications is due largely to the emergence of deep learning architectures, improvements in computing hardware, and the large amounts of data available for languages like english and mandarin @xcite . the performance of even high-accuracy asr models, however, remains fragile in the presence of external noise. asr accuracy degrades even further in low-resource settings. the motivation for this paper is to research the impact of several types of noise (e.g., continuous vs. punctuated, mechanical vs. natural) on asr performance in a low-resource setting for the dravidian language, tamil. we explore a range of asr architectures, including traditional gmm-hmm @xcite , sgmm, @xcite , and a hybrid dnn model @xcite . additionally, we evaluate whether different data augmentation approaches, such as raw audio augmentation and spectral augmentation (specaugment) @xcite , are particularly well suited to different types of noise. we explore these questions using a low-resource dataset for tamil provided by speechocean.com and microsoft for a low-resource asr challenge in interspeech 2018 @xcite . we observe that all types of noises, regardless of the acoustic model architecture, degrade asr performance. although not entirely surprising, we also find that raw audio augmentation outperforms the popular specaugment @xcite data augmentation method on clean data as well as noise-mixed data. ## previous work there is some prior work on using noise-mixed data to make asr more robust to noise and other external conditions, but most of this work focuses on high-resource languages. @xcite provided a comparative study on various acoustic and deep learning models, creating robust models in a noisy environment. the models were trained on noise-augmented training data and tested on both clean and noisy data. @xcite proposed a noise-robust speech recognition system called interactive feature fusion fetwork (iff-net) to learn the missing latent information by combin-ing the enhanced features and the original noisy features into a fused representation. this system achieved better results on the rats channel-a corpus. @xcite investigated challenges due to changes in environmental conditions and speaker characteristics and proposed a method to increase the robustness of the asr systems using speech enhancement techniques like spectral normalization and spectral subtraction. @xcite explained how energy normalization and speech re-synthesis can improve the performance of asr systems by recognizing speech signals in high-noisy conditions (negative snr). kinoshita et al. (2020) investigated whether the usage of single-channel time-domain neural networks can help in the reduction of noise and thereby improve the performance. gupta et al. (2016) proposed a back-propagation artificial neural network with feature compression using mfccs yielding improved performance with low signal-to-noise ratios. ## data the data used for the approach is a lowresource tamil language dataset, provided by spee-chocean.com and microsoft for a low-resource asr challenge for indian languages at interspeech 2018 @xcite . the dataset consists of read speech and conversations that have been split into utterances and transcribed. the dataset contains a total of 50 hours of recorded speech data in a clean noise-free environment. the dataset is partitioned into 40 hours of speech data for training, 5 hours for testing, and 5 hours for development. we do not use the dev set in our work. all the audio files are 16-bit mono audio sampled at 16khz. a total of 1900 speakers are included in the dataset. utterances range in length from 3000ms to 10000ms. there are a total of 42,212 unique utterances in the dataset. ## methodology we explore three different asr architectures available with the kaldi speech recognition toolkit @xcite . we note that none of these can be considered state-of-the-art. however, prior work has shown that the kaldi hybrid dnn is competitive with fine-tuning from multilingual end-to-end models using wav2vec xlsr @xcite and whisper @xcite in low-resource settings @xcite . ## results the baseline model was trained using the 40 hours of unaugmented training data and tested using the five hours of test data with no modification or augmentation of the test or train data. in the clean wer column in we see that, not unsurprisingly, the dnn architecture produces the best (lowest) wer, with the gmm-hmm acoustic model having the weakest performance. the results on test data mixed with the eight noises show that adding noises degrades asr performance regardless of architecture and sound type, sometimes very dramatically. the cat meowing has the smallest impact of the 8 noises, while party chatter has the largest. ## conclusion the goal of this research is to investigate the impact of noise on automatic speech recognition models using a low-resource tamil language dataset. we investigated the impact of noise by mixing several kinds of noises into the testing data. we evaluated the performance on the baseline model trained exclusively on the original unaugmented data. we discovered that all noises, regardless of whether they were mechanical or natural, continuous or punctuated, degraded asr performance, and that upgrading the architecture alone was unable to fully mitigate the impact of noise. to reduce the impact of noise in asr models, we augmented the training data by superimposing noises from each of the four categories onto the training data. we also employed the popular spectral augmentation technique, specaugment to create another augmented training dataset. we discovered that raw data augmentation improves wer, regardless of the combination of noises in the training and test sets. we also found that targeted raw augmentation improves asr performance: adding noise to the training data that shares one or more characteristics with the noise in the test data yields larger improvements. our methods outperform specaugment, although we recognize that specaugment is designed for end-to-end neural asr architectures rather than statistical or hybrid architectures like those explored here. in our future work, we plan to investigate these augmentation methods with an endto-end architectures, with a particular focus on approaches that involve fine-tuning multilingual models to low resource datasets, including wav2vec xlsr @xcite and whisper @xcite .