Using ASR -Generated Text for Spoken Language Modeling (2022)

Abstract: This papers aims at improving spoken language modeling (LM) using very large amount of automatically transcribed speech. We leverage the INA (French National Audiovisual Institute) collection and obtain 19GB of text after applying ASR on 350,000 hours of diverse TV shows. From this, spoken language models are trained either by fine-tuning an existing LM (FlauBERT) or through training a LM from scratch. The new models (FlauBERT-Oral) will be shared with the community and are evaluated not only in terms of word prediction accuracy but also for two downstream tasks : classification of TV shows and syntactic parsing of speech. Experimental results show that FlauBERT-Oral is better than its initial FlauBERT version demonstrating that, despite its inherent noisy nature, ASR-Generated text can be useful to improve spoken language modeling.

## introduction large language models are trained with massive texts which do not reflect well the specific aspects of spoken language. hence, modeling spoken language is challenging as crawling 'oralstyle' transcripts is a difficult task. to overcome this, our pilot study investigates the use of massive automatic speech recognition (asr) generated text for spoken language modeling. we believe that this methodology could bring diversity (oral/spontaneous style, different topics) to the language modeling data. this might be also useful for languages with fewer text resources but potential high availability of speech recordings. we also see long-term benefits to using asr generated text as speech recordings convey potentially useful metadata (ex: male/female speech) that could be leveraged for building lms from more balanced data. finally, as speech transcripts are naturally grounded with other modalities (if extracted from videos for instance), asr could help building large scale multimodal language understanding corpora. the contributions of this paper are the following: ## related works we mention here related works to better position our approach: learning lms from spoken transcripts, multimodal models and using lms to rescore asr. learning lms from spoken transcripts. @xcite probes bert based language models (bert, roberta) trained on spoken transcripts to investigate their ability to encode properties of spoken language. their empirical results show that lm is surprisingly good at capturing conversational properties such as pause prediction and overtalk detection from lexical tokens. but their lms evaluated are mostly trained on clean (non asr) spoken transcripts except one called asroberta which is trained on 2000h of transcribed speech only (1k librispeech + 1k proprietary dataset). as a comparison with this study, we train our models on 175x more asr data. multimodal models. while our approach uses asr to build text-based spoken language models, @xcite proposed an audio-andtext jointly learned speechbert model for spoken question answering task. they show their model is able to extract information out of audio data that is complementary to (noisy) asr output text. the architecture proposed by @xcite is different in the sense that it learns a joint language model with phoneme sequence and asr transcript to learn phonetic-aware representations that are robust to asr errors (not exactly a multimodal model). while speech or multimodal unsupervised representation learning is an interesting direction, this is out of the scope of this paper which focuses on language modeling from text transcripts only. bert for asr re-ranking. we also mention here lms to rescore asr as this could be an interesting application of our proposed spoken language models. chiu and chen (2021) used bert models for reranking of n-best hypotheses produced by automatic speech recognition (asr). their experiments on the ami benchmark demonstrate the effectiveness of the approach in comparison to rnnbased re-ranking. a similar idea is introduced by @xcite where bert features are added to the neural re-ranker used to rescore asr hypotheses. even more recently, @xcite showed how to train a bert-based rescoring model to incorporate a discriminative loss into the finetuning step of deep bidirectional pretrained models for asr. ## word prediction experiments the first step in evaluating our models is to look at their behaviour for the word prediction task. in addition to the performance on the trained models, we also want to have an idea of the performance on texts of different nature (written style or oral style). we therefore assembled several datasets to measure the word prediction performance of the models we trained. we make sure that these datasets are not included in the training data of the default flaubert model nor in our own. we have a first corpus (afp2021) of afp dispatches from the year 2021, i.e. after the period of our training data collected from the online press. this will allow us to have a measure of performance on written text. secondly, we want to evaluate our models on oral texts. we use the transcripts of the french national assembly sessions. 6 we are using the 13th (under sarkozy parl_13) and 15th (currently under macron parl_15) mandates. these texts are a manual transcription of what is said in the hemicycle, which are prepared speeches with some degree of spontaneous style as well. a second corpus is constituted with, once again, the manual transcriptions made for educational videos foot_6 and interviews foot_7 that ina makes available via its web studio (studio_manual). these transcriptions are of very good quality. we also transcribed these videos from the studio with our asr system (studio_asr) in order to be able to compare the performance on both types of data. we report in the graphs the accuracy obtained on the different datasets for a word prediction task after a word has been masked. the masking parameters are the same as those used during training with mlm loss. flaubert_base_uncased model (fbu). for the finetuned flaubert-o-base_uncased model, we notice a slight improvement in performance for afp and studio datasets, obtained from the first epoch, which means that adding asr generated text improves word prediction task on these datasets. we observe that globally, whatever the model, the datasets of the parliamentary sessions are those for which the best performances are obtained on the word prediction task, even exceeding that of the training dataset for the flaubert-o-base_uncased and flaubert-o-mixed models. these models are trained on written and spoken texts and it is not surprising that the performance is good since the very nature of the parliamentary data is a mix-ture of prepared and spontaneous speech. there is no significant difference between parl_13 and parl_15. on these parlementary speeches, there is no significant performance difference between the 3 models that have seen written text during their training (fbu, ft and mix). as we observed also that our flaubert-o models improve also on written text (afp2021), we explain this by the fact that those texts are strongly related to news events, so they are in a similar context to our asr data which is focused on news slot transcripts. for the last corpus, from the ina web studio, we have educational videos or interviews of personalities which are more distant from news data. there is a great disparity in performance depending on whether we consider manual (studio_manual) or automatic (stu-dio_asr) transcription. we believe that the different sentence segmentation algorithms have a very clear impact on this corpus. finally, we notice that the oral_nb model performs slightly worse than the oral model. the bpe tokenizer obviously has an impact on the overall performance of the lms and it seems, from this result, that using bpe units extracted from clean data (and not noisy asr data) is beneficial even if the training material is itself asr generated text. ## downstream task 2: syntactic analysis of spoken conversations this section is about the downstream task of jointly predicting part of speech tags (pos) and building a labelled dependency tree. the models performing these tasks typically rely on word representations, that are often pretrained, especially when the data is scarce. we will use our different spoken language models to obtain contextual word representations of a syntactically annotated and manually transcribed oral french corpus. for each of these representations, a model will be trained to perform the joint prediction of pos tags and labelled dependencies. we also use as baseline a model trained using non-contextual representations obtained with fasttext, foot_9 and a model learning its own representations without any pretraining. ## conclusion and future work we investigated spoken language modeling using asr generated text (350,000 hours of diverse tv shows). the new models for french (flaubert-o) are shared with the community. experimental results show that flaubert-o is generally better than its initial flaubert version for the downstream speech tasks we experimented with. however we should also check its performance on text downstream tasks (such as @xcite ) and on more downstream speech tasks (slu or asr re-scoring). in this work, all our texts were uncased as our asr only generates lowercased transcripts. we believe that applying massively re-capitalisation (and restoring punctuation as well) might be beneficial to train stronger spoken lms. we also plan to analyze more the specificities of our asr-generated texts (do they contain more oral features such as word repetitions, more interjections?). finally, some of the results obtained lead us to believe that it is important to further evaluate the impact of bpe units for spoken language modeling.