Automatic Transcription for E stonian Children’s Speech (2023)

Abstract: We evaluate the impact of recent improvements in Automatic Speech Recognition (ASR) on transcribing Estonian children’s speech. Our research focuses on fine-tuning large ASR models with a 10-hour Estonian children’s speech dataset to create accurate transcriptions. Our results show that large pre-trained models hold great potential when fine-tuned first with a more substantial Estonian adult speech corpus and then further trained with children’s speech.

## introduction automatic speech recognition (asr) continues to face challenges in accurately transcribing children's speech. research efforts are underway to adapt adult asr models to better handle the unique pronunciation variations and limited vocabulary that are characteristic of children's speech @xcite . these adaptations are necessary due to the limitations of current asr systems, which often lack adequate representation of children's speech and struggle to generalize to new examples. recent advancements in asr technology, including the use of large transformer-based models and unsupervised pre-training techniques, have resulted in improved performance for adult speech recognition, with the ability to train on a diverse range of data without human annotations @xcite @xcite . these models demonstrate greater robustness and generalization compared to previous systems. however, the effectiveness of these advanced asr models for children's speech, especially in low-resource languages like estonian, remains untested. in this paper, we are investigating two multilingual speech models -facebook's wav2vec2-xls-r @xcite ) and openai's whisper @xcite -as potential starting points for building an asr system transcribing estonian children's speech. our objective is to determine the potential of these models in creating low-effort asr systems for children speaking a low-resource language like estonian, for which there are no asr systems for children's speech. to accomplish this, we fine-tune the xls-r and whisper models from scratch using children's speech data. we also fine-tune pre-existing models for the estonian language with additional children's speech recordings. furthermore, we compare the quality of the asr system by evaluating a pre-made estonian asr system provided by microsoft azure and exploring its fine-tuning capabilities. our research indicates that xls-r models and whisper models can serve as effective starting points for building an asr system using only 10 hours of children's speech. however, for optimal performance, these models should first be finetuned with estonian adult speech. we achieve the best word error rate of around 15 using an xls-r model that was fine-tuned with estonian asr datasets and further trained with children's speech. furthermore, our results show that the azure speech-to-text model performs similarly to the estonian xls-r and whisper models but not as well as the fine-tuned public models. two models that achieved the lowest wer scores are available in huggingface 12 . in the next sections, we describe which data we used for evaluation and training, which models we used and how we fine-tuned these and last but not least we present and analyse the results. ## dataset and evaluation the children asr dataset used in this work consists of speech recordings from 53 children aged 6 to 13. the data was collected by the children's clinic of tartu university hospital and contains a mix of both boys and girls speaking about various topics such as answering questions, describing pictures, talking about their family and friends, and more. the dataset is divided into three subsets -test, dev, and train -with no overlap in speakers or texts. the test set contains all age and gender groups and has a total recording duration of 278 minutes (approximately 4.6 hours). the development set is missing some speakers and has a total recording duration of 182 minutes (approximately 3 hours). the training set is also missing some speakers and has a total recording duration of 613 minutes (approximately 10 hours). a breakdown of the total recording duration for the test set by age and gender of the speakers is shown in the children in the dataset speak about a wide range of topics, covering everything from answering questions and describing pictures to discussing their family and friends. they also include recordings of children reading fairytales, reciting poems, and saying specific sentences. the utterances in the dataset vary in their level of spontaneity -some are unscripted expressions of thoughts, while others feature children reading. we evaluate the performance of our speech recognition models using the standard measure of word error rate (wer). this involves converting all text to lowercase and removing punctua-tion but not standardizing different spelling variations. our reference transcriptions reflect the pronunciation of children, including any errors they may make. however, the line between correct and incorrect pronunciation is often blurry and some children's speech can be difficult to comprehend. we do not consider the ambiguity in human transcriptions and simply compare the models' output to our reference transcription, which could lead to increased wers. ## models and training we are using both public large speech models and private black box speech service. in the case of public models, we also searched for models already fine-tuned with estonian speech data. we fine-tune the selection of these models with the children's speech dataset mentioned in the last section. for public models, we use two multilingual ones: facebook's xls-r and openai's whisper @xcite . xls-r model is trained with speech modelling objective, not asr but it can be fine-tuned to asr with connectionist temporal classification (ctc) @xcite algorithm. the whisper on the other hand is a multipurpose model that contains both transformer encoder and decoder blocks and has been trained on several speech-processing tasks, like multilingual speech recognition, speech translation and voice activity detection @xcite . the available xls-r models have 300 million, 1 billion and 2 billion parameters, we are using the two smaller ones in this work. the whisper model comes in six different sizes; we are using medium and large-v2 since the estonian error rates for other ones are relatively high. there is one estonian-specific fine-tuned model available for the 300 million parameter version, trained with over 700 hours of estonian speech data @xcite . there are several estonian whisper models available in huggingface but these are trained with fewer data examples. we are using the best available medium and large-v2 ones. 34 . following the submission of this paper, a new estonian whisper model was released 5 , which is trained using a larger dataset. in the scope of this work, we evaluate the model but do not fine-tune it using children's speech. we use standard fine-tuning procedures. for training xls-r-based asr models from scratch, we use the learning rate of 3e-4, a 400-step warmup and train the models for 60 epochs with children's speech dataset, which is less than 4000 steps. when further fine-tuning the estonian xls-r model with children's speech, we use the learning rate of 2e-5 and 200 warmup steps. we finetune all the whisper models with warmup 10% of the steps and learning rate 1e-05. when finetuning the out-of-the-box whisper models, we train these for 5000 steps or atound 40 epochs and when fine-tuned models already trained with estonian adult speech, we train the large model for 2000 steps or over 16 epochs and medium model for 1000 steps or eight epochs. for the private model, we use microsoft azure speech service's speech-to-text foot_5 , which requires an azure subscription and a speech resource. the transcription services can be accessed by making rest requests. microsoft azure offers the option to fine-tune the model with custom datasets. this process involves uploading data to train the models, followed by deploying the trained models. since audio-based fine-tuning is not available for estonian, we use text-based tuning for our work with the texts from the children's speech dataset. ## results in this section, we describe the results of all the models based on facebook's xls-r, openai's whisper and microsoft azure speech-to-text. ## discussion our experiments show that children's speech recognition continues to be a tricky problem but big speech models are looking promising. it is possible to build an asr system for estonian children's speech without any bells and whistles using only 10 hours of data and get output that is decent and might be good enough for use in chatbots. however, when it comes to six-year-olds, whose speech is difficult to understand even for the human ear, the system is still struggling. we evaluate different models and it appears that both openai's whisper and facebook's xls-r are viable options for developing a speech recognition model for estonian children's speech. the current best word error rate is around 15 with xls-r. however, it remains unclear if this pretrained model is optimal for children's speech or if a lower error rate could be achieved with whisper after fine-tuning with a similar amount of estonian adult speech. additionally, we do not obtain comparable results with the azure service, as it does not permit fine-tuning with audio data. our findings suggest that the results could be improved by using a larger xls-r model as the base or by fine-tuning whisper models with more data. additionally, we do not use a separate language model, which is possible with both whisper and xls-r models and could potentially enhance the performance of these models. ## conclusion we test the performance of two speech recognition models, xls-r and whisper, on transcribing estonian children's speech. we fine-tune the models with children's speech data and compared them to an off-the-shelf system from microsoft azure. both models fine-tuned with children's speech, outperform microsoft azure, which does not allow fine-tuning with audio for estonian, and are promising for children's asr system.