WER - BERT : Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigm (2021)

Abstract: Automatic Speech Recognition (ASR) systems are evaluated using Word Error Rate (WER), which is calculated by comparing the number of errors between the ground truth and the transcription of the ASR system. This calculation, however, requires manual transcription of the speech signal to obtain the ground truth. Since transcribing audio signals is a costly process, Automatic WER Evaluation (e-WER) methods have been developed to automatically predict the WER of a speech system by only relying on the transcription and the speech signal features. While WER is a continuous variable, previous works have shown that positing e-WER as a classification problem is more effective than regression. However, while converting to a classification setting, these approaches suffer from heavy class imbalance. In this paper, we propose a new balanced paradigm for e-WER in a classification setting. Within this paradigm, we also propose WER-BERT, a BERT based architecture with speech features for e-WER. Furthermore, we introduce a distance loss function to tackle the ordinal nature of e-WER classification. The proposed approach and paradigm are evaluated on the Librispeech dataset and a commercial (black box) ASR system, Google Cloud’s Speech-to-Text API. The results and experiments demonstrate that WER-BERT establishes a new state-of-the-art in automatic WER estimation.

## introduction asr systems are ubiquitous now. they are available across applications such as voice assistants, assisted living or hands free device usage. however, with the widespread usage of asr systems, there comes a heavy need for asr evaluation as well -to select, compare or improve alternate asr *the authors contributed equally to the work. systems. wer is widely considered as the standard metric for asr evaluation. a higher wer means a higher percentage of errors between the ground truth and the transcription from the system. wer is calculated by aligning the two text segments using string alignment in a dynamic programming setting. the formula is as follows: the overall contributions of our paper can be summarized follows: ## related work while the importance of an automatic wer prediction system is immense, there have not been many works directly addressing it. related works such as exploring the word-level confidence in asr prediction are abundant @xcite . there have also been works predicting the errors or error estimates as well in some form @xcite @xcite . these approaches either predict some of the errors described in wer prediction or alternate metrics to rate asr systems such as accuracy or error type classification. however, they lack calculation of the complete wer score. transcrater @xcite was one of the first works which aim at predicting wer directly. they propose a neural network in a regression setting trained on various features such as parts of speech, language model, lexicon, and signal features. however, more recent approaches @xcite @xcite phrase wer prediction as a classification approach. @xcite propose two types of models based on the input available -the glassbox model which uses internal features of the target asr system such as its confidence in transcribing the audio clip; and the black box model which only uses the transcripts and other features generated from the transcript such as the word and the grapheme count. they propose a bag-of-words model along with additional transcription features such as duration for e-wer. the black box setting is a harder task since asr model features such as the average log likelihood and the transcription confidence can give a good indication on how many errors may have occurred during the automatic transcription. however, the black box approach is not specific to the architec-tural design of an asr system and can be used with any asr system without access to its internal metrics such as the aforementioned confidence. thus our proposed approach is trained in a black box setting. @xcite build a cnn based model for wer classification. we built models based on them as baselines to evaluate wer-bert's performance. they are further explained in the sections 4 and 6. asr errors often make a transcription ungrammatical or semantically unsound. identifying such constructs is also reflected in the dataset of corpus of linguistic acceptability(cola) @xcite . cola is a dataset intended to gauge at the linguistic competence of models by making them judge the grammatical acceptability of a sentence. cola is also part of the popular glue benchmark datasets for natural language understanding @xcite . bert @xcite is known for outperforming previous glue state-of-the-art models, including the cola dataset. ## dataset for our experiments, we have used the librispeech dataset @xcite which is a diverse collection of audio book data along with the ground text. it has around 1000 hours of audio recordings with different levels of complexity. we pass these audio clips through an asr system to get its transcripts and the wer is calculated by comparing it with the ground text. this paper reports findings in the experiments run with google cloud's speech-to-text api. we chose this commercial asr system, rather than reporting results on an internal asr system, since it's easily accessible through the google-api and the results are reproducible. for our experiments, we have used the 10 and 100 hour datasets and made a 60:20:20 split into train, dev and test sets for each dataset. as can be seen in as shown in equation 1, the wer of an utterance is the fraction obtained by the division of 2 integers -errors per sentence (err), which is the total number of insertions, deletions and substitutions needed to convert an asr's transcript to the ground text, and the word count of the ground text (n). since the wer of a sentence is a continuous variable between 0 and 1 (mostly), a common way to model this is through a regression model. @xcite instead present a way to turn this into a classification problem for e-wer. they experiment with various combinations of text and audio signal inputs and show that a classification approach outperforms its corresponding regression approach trained on the same inputs. elloumi et al. ( 2018 )'s approach estimates wer directly with a 6 class classification model (with classes corresponding to 0%, 25%, 50%, 75%, 100% and 150%). once the model is trained, the predictions are calculated as follows: ## wer-bert in this section we explain our proposed architecture wer-bert, which is primarily made of four subnetworks. our architecture is shown with details in figure 3 . signal sub-network: elloumi et al. (2018) use the raw signal of the audio clip to generate features such as mfcc and mel spectrogram. they are features commonly used in the design of asr systems, particularly systems which use an acoustic model and furthermore these features aid their model performance. these signal features are passed through the m18 architecture @xcite . m18 is a popular deep convolutional neural network (cnn) used for classification with audio signal features. this cnn model has 17 convolutional+max pooling layers which is followed by global average pooling. l2 regularization of 1e -4 and batch normalization are added after each of the convolutional layers. numerical features sub-network: ali and renals (2018) black box models had two major components -text input and numerical features. these numerical features are important to the model as they contain information regarding the number of errors. for instance, in asr systems, there are errors if a user speaks too fast or too slow and this is directly reflected in the duration and word count features. the numerical features we have used are word count, grapheme count and duration. these features are concatenated and passed through a simple feed forward network which is used to upscale the numerical features fed into the model (from 3 to 32). bert: bi-directional encoder representations (bert) @xcite is a pre-trained unsupervised natural language processing model. it is a masked language model which has been trained on a large corpus including the entire wikipedia corpus. the transcription samples from the asr system, are passed through the associated tokenizer which gives a contextual representation for each word. it also adds 2 special tokens -the [cls] token at the beginning and the [sep] token at the end of the sentence. we have used the bert-large uncased variant. the large variant has 24 stacked transformer @xcite encoders. it gives an output of the shape (sequence length x 1024) of which only the 1024 shaped output corresponding to the [cls] token is used. in wer-bert, bert weights are fine tuned with rest of architecture during training. feed forward sub-network: this subnetwork is a deep fully connected network which is used to concatenate and process the features generated by the sub-networks predating it (bert, numerical sub-network and the signal sub-network). it has 4 hidden layers (512, 256, 128 and 64 neurons) followed by the output softmax layer. dropout regularization is added to prevent overfitting considering the large amount of parameters. to account for outputs from the eclectic sub-networks with disparate distributions, we further add layer normalization (ba et al., 2016) before concatenation. normalization is important to lessen the impact of any bias the network may learn towards one or the other representations. distance loss for ordinal classification: typical classification problems deal with classes which are mutually exclusive and independent such as sentiment prediction or whether an image is a cat or a dog. in such a setting, classification accuracy is the most important metric and there is no relation or relative ordering between the classes. however, e-wer in a classification setting is an ordinal classification problem @xcite . previous approaches which propose wer estimation as classification tasks ignore this idea @xcite @xcite . while the classification accuracy is important, it is more important that given a sample is misclassified, the predicted label is close to the true label. for instance, if the true label corresponds to the wer class of 0.1, a prediction of 0.2 and a prediction of 0.7 are treated the same in the current classification scenario. since we want the prediction to be as close as possible, if not exactly the same, we introduce a "distance" loss which is ground truth google cloud's speech-to-text true predicted transcription wer wer one historian says that an event was produced by napoleon's power another that it was produced by alexander's when is dorian says that an event was produced by napoleon's power another that it was produced by alexander's 16.7 16.5 rynch watched dispassionately before he caught the needler jerking it away from the prisoner the man eyed him steadily and his expression did not alter even when rynch swung the off world weapon to center its sights on the late owner wrench watch dispassionately before he caught a kneeler jerking it away from the prisoner the man i can steadily and his expression did not alter even when wrench swampy off world weapon to center its sights on the late owner 21.9 22.1 of acting a father's part to augustine until he was fairly launched in life he had a child of his own acting a father's part 2 augustine until he was fairly launched in life 42.8 42.7 supported by an honorable name how could she extricate herself from this labyrinth to whom would she apply to help her out of this painful situation debray to whom she had run with the first instinct of a woman towards the man she loves and who yet betrays her supported by an honorable name how could you extricate herself in this labyrinth to whom would she apply to help her out of this painful situation dubray to whom should run the first instinct of a woman towards the man she loves and who yep betrays her 14.3 14.4 seventeen twenty four 1724 100.0 30.7 saint james's seven st james 7 100.0 32.1 mamma says i am never within mama says i am never with him 50.0 13. 44 ## experiments and baselines for each of the experiments below, the training is repeated for 10 runs and we report the average performance of all the runs on the test set. for all refer to section a.1 of the appendix for tuning of the distance loss hyperparameter α the experiments, we use crossentropy as the loss function and m ae of w er as the evaluation metric. ## results and discussion comparing figure 4 and 5, we see that the wer-bert models much better in the lower and mid regions compared to the cnn balanced model. ## conclusion we propose wer-bert for automatic wer estimation. while bert is an effective model, addition of speech signal features boosts the performance. phrasing wer classification as a ordinal classification problem by training using a custom distance loss encodes the information regarding relative ordering of the wer classes into the training. finally, we propose a balanced paradigm for training wer estimation systems. training in a balanced setting allows proposed model to predict wer adequately even in regions where samples are scarce. furthermore, this balanced paradigm is independent of wer prediction model, asr system or the speech dataset, making it efficient and scalable. su-youn yoon, lei chen, and klaus zechner. 2010. predicting word accuracy for the automatic speech recognition of non-native speech. in eleventh annual conference of the international speech communication association.