Attentively Embracing Noise for Robust Latent Representation in BERT (2020)

Abstract: Modern digital personal assistants interact with users through voice. Therefore, they heavily rely on automatic speech recognition (ASR) in order to convert speech to text and perform further tasks. We introduce EBERT, which stands for EmbraceBERT, with the goal of extracting more robust latent representations for the task of noisy ASR text classification. Conventionally, BERT is fine-tuned for downstream classification tasks using only the [CLS] starter token, with the remaining tokens being discarded. We propose using all encoded transformer tokens and further encode them using a novel attentive embracement layer and multi-head attention layer. This approach uses the otherwise discarded tokens as a source of additional information and the multi-head attention in conjunction with the attentive embracement layer to select important features from clean data during training. This allows for the extraction of a robust latent vector resulting in improved classification performance during testing when presented with noisy inputs. We show the impact of our model on both the Chatbot and Snips corpora for intent classification with ASR error. Results, in terms of F1-score and mean between 10 runs, show that our model significantly outperforms the baseline model.

## introduction in recent year, machine learning methods have significantly contributed to the development of automatic speech recognition (asr) @xcite @xcite . the fast advances in the asr field can be explained by the urgent need of efficiently bridging the gap between humans and the ever evolving technologies surrounding us. as technology becomes smaller and more portable, it becomes clear that the only feasible way of interaction is through a digital personal assistant with voice interface @xcite . however, high error rates still prevail and pose a tremendous hurdle in the widespread adoption of speech technology by users worldwide @xcite . researchers in the computational linguistics community analyze noisy text @xcite obtained through informal settings, such as social networks, or through processing techniques, such as optical character recognition @xcite , asr @xcite @xcite , or machine translation models @xcite . we focus on noisy asr text classification. researchers in this area have simulated noise introduced through asr systems and they have investigated the performances of naive bayes and support vector machine (svm) classifiers in noisy text classification @xcite . @xcite have also investigated the impact in performance of using text embedded with speech data with neural networks and logistic regression. the aforementioned researches, however, focus mainly on investigating the impact that noise has in the performance of shallow machine learning models, such as svm or vanilla neural network, rather than focusing on extracting more robust input representation for improved performance. more recently, the transformer @xcite based model bert @xcite has achieved state of the art in various natural language processing tasks. this inspired @xcite to suggest the use of a transformer encoder-decoder architecture for the task of asr correction and @xcite to improve readability also using transformer-based sequence-to-sequence architectures for the same task. the pre-trained model bert is fine-tuned for downstream tasks by extracting an encoded transformer token for each corresponding input token. in bert's classic text classification approach, only the sentence starter @xcite token is used for classification. however, we hypothesize that the discarded tokens can be used as a source of additional information, and should therefore also be used. we also search for inspiration outside of the noisy text analytics domain into the multimodal data domain, with concepts from embracenet @xcite , which is proposed to improve robustness in the absence of data. the authors achieve that with an embracement layer that randomly selects important features from each modality vector and meshes them into one robust vector called embraced vector. we propose using the otherwise discarded tokens as a source of additional information and further encode them with a novel attentive embracement and multi-head attention layer. the combination of these layers allows the model to select important features from clean data during training, so that when presented with noisy inputs during testing, it is able to extract a robust latent vector for improved classification performance. we call our proposed model embracebert (ebert). in summary, our contributions are three-fold: the model is evaluated on three settings: (1) trained and tested with clean data, (2) trained with clean and tested with noisy data, and (3) trained and tested with noisy data. ## proposed model in this section, we propose the novel ebert model for the task of noisy asr text classification. as the name suggests, our model is based on bert, and as such, its first layers are transformer layers that encode the input embeddings e into tokens t . conventionally, when fine-tuning bert for downstream classification tasks, only the [cls] starter token is used, with the remaining t i∀@xmath0,...,n } tokens being discarded. although we concede that the [cls] token might be a good single representation of the input for clean data, it lacks the same representation efficiency in noisy data. we tackle this issue by proposing the use of all encoded tokens as a source of additional information to improve the model's robustness and performance. now, instead of a single vector that can directly be used for classification, we are faced with multiple ones. our solution, shown in fig. 2 , is to introduce a multi-head attention layer with q, k, v obtained from t to extract additional important information d, followed by a novel attentive embracement layer to obtain the embraced vector e. note that we consider each token t , which can be missing or incorrect, as a different modality in order to use the embracement layer. the embraced vector is then given to a projection layer together with the [cls] token, resulting in a single robust representation vector t c that can be used for classification. the projection layer is as in eq. ( 3 ): figure 2: proposed ebert model: hierarchical structure of transformer layers and multi-head attention followed by a novel attentive embracement layer to obtain the embraced vector e from tokens t . the embraced vector is then given to a projection layer together with the [cls] token, resulting in a single robust representation vector t c that can be used for classification. in the conventional embracement layer (fig. 3(a )), the embraced vector @xmath1, e (2) , . . . , e (l) is obtained by first drawing a vector r (i) from a multinomial distribution, as in eq. ( 5 ): with n p @xmath2. now, assume that each vector has length l and the sampling vector r @xmath3, r finally, the embraced vector is obtained as follows, in eq. ( 8 ): in the proposed attentive embracement layer (fig. 3(b )), the embraced vector @xmath4, e ( foot_0 ) , . . . , e (l) is obtained in a similar manner, except the probability p of each token being chosen to compose e (i) . in the conventional method, the probability of each token being selected is the same, and by all means random. this approach has two limitations. first, in order to know the probability of each token, the user must indicate which data are missing and adjust p accordingly. second, not all features in the data have similar importance, and this needs to be addressed for a more robust account of the data. we tackle both issues by introducing an attention layer to obtain the importance of each token t @xmath5,...,n } , compared to the t [cls] token, to be used in the selection process, as in eq. ( 9 ): ## dataset the dataset used to evaluate the model's performance is the chatbot nlu evaluation corpus for intent classification, introduced by braun et al. @xcite to test nlu services. it is a publicly available 2 benchmark and is composed of sentences obtained from a german telegram chatbot used to answer questions about public transport connections. the dataset has two intents, namely departure time and find connection with 100 train and 106 test samples foot_1 . even though english is the main language of the benchmark, this dataset contains a few german station and street names. the original dataset contains clean data, so in order to include asr noise, we apply a text-to-speech (tts) followed by a speech-to-text (stt) module to that data. this process is shown in fig. 4 , and it's 2020. 7. 2. original text (complete) obtained text (incomplete) tts intermediate audio stt original text (clean) obtained text (noisy) tts intermediate audio stt original text (clean) obtained text (noisy) tts intermediate audio stt figure 4: diagram of 2-step process to obtain text with asr error from clean data. efficient due to tts and stt modules available being imperfect, resulting in sentences with reasonable levels of asr noise. the tts module is called macsay foot_2 , named after the terminal command say used to convert text to speech in the mac os platform. we obtain two datasets with asr noise by applying two distinct stt modules after the tts module: witai foot_3 , freely available and maintained by wit.ai, and sphinx foot_4 , open-source python functionality in the cmu sphinx speech recognition engine. the mentioned tts and stt modules are chosen according to code availability and whether it's freely available or has high daily usage limitations. the noise level in the sentences is measured quantitatively with the word error rate (wer) metric, a common metric used to evaluate asr systems where lower scores mean lower noise levels foot_5 .