M ulti QT : Multimodal learning for real-time question tracking in speech (2020)

Abstract: We address a challenging and practical task of labeling questions in speech in real time during telephone calls to emergency medical services in English, which embeds within a broader decision support system for emergency call-takers. We propose a novel multimodal approach to real-time sequence labeling in speech. Our model treats speech and its own textual representation as two separate modalities or views, as it jointly learns from streamed audio and its noisy transcription into text via automatic speech recognition. Our results show significant gains of jointly learning from the two modalities when compared to text or audio only, under adverse noise and limited volume of training data. The results generalize to medical symptoms detection where we observe a similar pattern of improvements with multimodal learning.

## introduction our paper addresses the challenge of learning to discover and label questions in telephone calls to emergency medical services in english. the task is demanding in two key aspects: ## multimodal speech labeling we define the multimodal speech labeler multiqt as a combination of three neural networks that we apply to a number of temporal input modalities. in our case, we consider speech and associated machine transcripts as the separate modalities or views. the model is illustrated in figure 2 . to obtain temporal alignment between speech and text, we propose a simple approach that uses the output of an asr system as the textual representation. here, we take the asr to be a neural network trained with the connectionist temporal classification (ctc) loss function @xcite . given audio, it produces a temporal softmax of length t s with a feature dimension defined as a categorical distribution, typically over characters, words or subword units, per timestep. for @xmath0..t m ], where r a,l , r a,r , r s,l and r s,r are the left and right half receptive fields of f a and f s , respectively. for f a , r a,@xmath1,@xmath2. for @xmath3 a and x (i) s by zero padding, effectively padding with half the receptive field on the left and right sides of the input. this then implies that t @xmath4. this lets us do convolutions without padding the internal representations for each layer in the neural networks, which in turn allows for online streaming. to form a joint multimodal representation from z (t) a and z (t) s we join the representations along the feature dimension. in the multimodal learning litterature such an operation is sometimes called fusion @xcite . we denote the combined multimodal representation by z (t) m and obtain it in a time-binded manner such that for a certain timestep the latter is similar to the fusion introduced by @xcite , but we do not collapse the time dimension since our model predicts sequential labels. ## data our dataset consists of 525 phone calls to an english-speaking medical emergency service. the call audio is mono-channel, pcm-encoded and sampled at 8000 hz. the duration of the calls has the mean of 166 s (st. dev. 65 s, iqr 52 s). all calls are manually annotated for questions by trained native english speakers. each question is annotated with its start and stop time and assigned with one of 13 predefined question labels or an additional label for any question that falls outside of the 13 categories. figure 1 illustrates these annotations. we observe an initial inter-annotator agreement of α = 0.8 @xcite . each call has been additionally corrected at least once by a different annotator to improve the quality of the data. on average it took roughly 30 minutes to annotate a single call. for our experiments, we choose the five most frequent questions classes, which are explained in the utterances that we track are complex phrases with a high variance: there are many different ways to express a question or a medical symptom in conversation. this linguistic complexity sets our research apart from most work in speech labeling which is much closer to exact pattern matching @xcite . the textual modality is formed by application of an asr system to the audio modality. in all reported experiments, only asr outputs are used and never human transcriptions, both in training and evaluation. the audio input to the asr is encoded in the same way as described above. the asr available to us has a purely convolutional architecture similar to the one in @xcite with an overall stride of 2. for multiqt, this means that t @xmath5. the asr is trained on 600 hours of phone calls to medical emergency services in english from the same emergency service provider as the question and symptoms tracking datasets. both of these are contained in the asr test set. the asr is trained using the connectionist temporal classification (ctc) loss function @xcite and has a character error rate of 14 % and a word error rate of 31 %. its feature dimension is 29 which corresponds to the english alphabet including apostrophe, space and a blank token for the ctc loss. systems. the basic version of multiqt uses a single softmax cross-entropy loss function and forms a time-bound multimodal representation by concatenating the unimodal representations. we then augment this model in three ways: baselines. multiqt can easily be adapted to a single modality by excluding the respective convolutional transformation f a or f s . for example, multiqt can be trained unimodally on audio by removing f s and then defining z (t) a instead of concatenation or tensor fusion. we baseline the multimodal multiqt models against versions trained unimodally on audio and text. we also compare multiqt to two distinct baseline models: hyperparameters. we use 1d convolutions for f a and f s . for f a we use three layers with kernel sizes of 10, 20 and 40, filters of 64, 128 and 128 units and strides of 2, 2 and 2 in the first, second and third layer, respectively. for f s we use two layers with kernel sizes of 20 and 40, filters of 128 and 128 units and strides of 2 and 2. before each nonlinear transformation in both f a and f s we use batch normalization @xcite with momentum 0.99 and trainable scale and bias, and we apply dropout @xcite with a dropout rate of 0.2. for g we use three fully connected layers of 256 units each and before each nonlinear transformation we use batch normalization as above and apply dropout with a dropout rate of 0.4. we l 2 regularize all learnable parameters with a weighting of 0.1. the fnn model uses the same classifier as is used for g in multiqt with a dropout rate of 0.3 and an l 2 regularization factor of 0.05. all neural models are trained with the adam optimizer (kingma and ba, 2015) using a learning rate of 1 × 10 -4 , β 1 = 0.9 and β 2 = 0.999 and batch size 6 except for those with tensor fusion which use a batch size of 1 due to memory constraints. larger batch sizes were prohibitive since we use entire calls as single examples but results were generally consistent across different batch sizes. all hyperparameters were tuned manually and heuristically. it takes approximately one hour to train the base multiqt model on one nvidia geforce gtx 1080 ti gpu card. evaluation. for each model we report two f1 scores with respective precisions and recalls macroaveraged over the classes. the baseline models are evaluated per timestep by labeling segments from the test set in a sliding window fashion. the size of the window varies from 3 to 9 seconds to encompass all possible lengths of a question with the stride set to one word. defining the stride in terms of words is possible because the asr produces timestamps for the resulting transcript per word. ## related work the broader context of our work is to track the dialogue state in calls to emergency medical services, where conversations are typically formed as sequences of questions and answers that pertain to various medical symptoms. the predominant approach to dialogue state tracking (dst) in speech is to first transcribe the speech by using asr @xcite @xcite . in our specific context, to entirely rely on asr is prohibitive because of significantly higher wer in comparison to standard datasets. to exemplify, while wer is normally distributed with a mean of 37.6% in our data, the noisiest dst challenge datasets rarely involve with wer above 30% @xcite while standard asr benchmarks offer even lower wer @xcite . none of the standard asr scenarios thus directly apply to a real-life asr noise scenario. from another viewpoint, work in audio recognition mainly involves with detecting simple singleword commands or keyword spotting @xcite , recognizing acoustic events such as environmental or urban sounds @xcite @xcite or music patterns, or document-level classification of entire audio sequences @xcite . @xcite provide a more extensive overview. while approaches in this line of work relate to ours, e.g. in the use of convolutional networks over audio @xcite , our challenge features questions as linguistic units of significantly greater complexity. finally, research into multimodal or multi-view deep learning @xcite offers insights to effectively combine multiple data modalities or views on the same learning problem. however, most work does not directly apply to our problem: i) the audio-text modality is significantly under-represented, ii) the models are typically not required to work online, and iii) most tasks are cast as document-level classification and not sequence labeling @xcite . ## conclusions we proposed a novel approach to speech sequence labeling by learning a multimodal representation from the temporal binding of the audio signal and its automatic transcription. this way we learn a model to identify questions in real time with a high accuracy while trained on a small annotated dataset. we show the multimodal representation to be more accurate and more robust to noise than the unimodal approaches. our findings generalize to a medical symptoms labeling task, suggesting that our model is applicable as a general-purpose speech tagger wherever the speech modality is coupled in real time to asr output.