Digital Voicing of Silent Speech (2020)

Abstract: In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While prior work has focused on training speech synthesis models from EMG collected during vocalized speech, we are the first to train from EMG collected during silently articulated speech. We introduce a method of training on silent EMG by transferring audio targets from vocalized to silent signals. Our method greatly improves intelligibility of audio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4% in one data condition and 88% to 68% in another. To spur further development on this task, we share our new dataset of silent and vocalized facial EMG measurements.

## introduction in this paper, we are interested in in enabling speech-like communication without requiring sound to be produced. by using muscular sensor measurements of speech articulator movement, we aim to capture silent speech -utterances that have been articulated without producing sound. in particular, we focus on the task which we call digital voicing, or generating synthetic speech to be transmitted or played back. digitally voicing silent speech has a wide array of potential applications. for example, it could be used to create a device analogous to a bluetooth headset that allows people to carry on phone conversations without disrupting those around them. such a device could also be useful in settings where the environment is too loud to capture audible speech or where maintaining silence is important. alternatively, the technology could be used by some people who are no longer able to produce audible speech, such as individuals whose larynx has been removed due to trauma or disease @xcite . in addition to these direct uses of digital voicing for silent speech, it may also be useful as a component technology for creating silent speechto-text systems @xcite , making silent speech accessible to our devices and digital assistants by leveraging existing high-quality audio-based speech-to-text systems. to capture information about articulator movement, we make use of surface electromyography (emg). surface emg uses electrodes placed on top of the skin to measure electrical potentials caused by nearby muscle activity. by placing electrodes around the face and neck, we are able to capture signals from muscles in the speech articulators. figure 1 shows the emg electrodes used to capture signals, and figure 2 shows an example of emg signals captured. we collect emg measurements during both vocalized speech (normal speech production that has voicing, frication, and other speech sounds) and silent speech (speech-like articulations which do not produce sound). we denote these emg signals e v and e s , respectively. during the vocalized speech we can also record audio a v , but during silent speech there is no meaningful audio to record. a substantial body of prior work has explored the use of facial emg for silent speech-to-text interfaces @xcite @xcite . several initial attempts have also been made to convert emg signals to speech, similar to the task we approach in this paper @xcite @xcite . however, these works have focused on the artificial task of recovering audio from emg that was recorded during vocalized speech, rather than the end-goal task of generating from silent speech. in terms of signals in figure 2 , prior work learned a model for producing audio a v from vocalized emg e v and tested primarily on other vocalized emg signals. while one might hope that a model trained in this way could directly transfer to silent emg e s , @xcite show that such a transfer causes a substantial degradation in quality, which we confirm in section 4. this direct transfer from vocalized models fails to account for differences between features of the two speaking modes, such as a lack of voicing in the vocal folds and other changes in articulation to suppress sound. in this paper, we extend digital voicing to train on silent emg e s rather than only vocalized emg e v . training with silent emg is more challenging than with vocalized emg, because when training on vocalized emg data we have both emg inputs and time-aligned speech targets, but for silent emg any recorded audio will be silent. our solution is to adopt a target-transfer approach, where audio output targets are transferred from vocalized recordings to silent recordings of the same utterances. we align the emg features of the instance pairs with dynamic time warping @xcite , then make refinements to the alignments using canonical correlation analysis @xcite and audio feature outputs from a partially trained model. the alignments can then be used to associate speech outputs with the silent emg signals e s , and these speech outputs are used as targets for training a recurrent neural transduction model. we validate our method using both human and automatic metrics, and find that a model trained with our target transfer approach greatly outperforms a model trained on vocalized emg alone. on a closed-vocabulary domain (date and time expressions §2.1), transcription word error rate (wer) from a human evaluation improves from 64% to just 4%. on a more challenging open vocabulary domain (reading from books §2.2) intelligibility measurements improve by 20% -from 88% to 68% with automatic transcription or 95% to 75% with human transcription. we release our dataset of emg signals collected during both silent and vocalized speech. the dataset contains nearly 20 hours of facial emg signals from a single speaker. to our knowledge, the largest public emg-speech dataset previously available contains just 2 hours of data @xcite , and many papers continue to use private datasets. we hope that this public release will encourage development on the task and allow for fair comparisons between methods. ## data collection we collect a dataset of emg signals and timealigned audio from a single speaker during both silent and vocalized speech. figure 2 shows an example from the data collected. the primary portion of the dataset consists of parallel silent / vocalized data, where the same utterances are recorded using both speaking modes. these examples can be viewed as tuples (e s , e v , a v ) of silent emg, vocalized emg, and vocalized audio, where e v and a v are time-aligned. both speaking modes of an utterance were collected within a single session to ensure that electrode placement is consistent between them. for some utterances, we record only the vocalized speaking mode. we refer to these instances as non-parallel data, and represent them with the tuple (e v , a v ). examples are segmented at the utterance level. the text that was read is included with each instance in the dataset, and is used as a reference when evaluating intelligibility in section 4. for comparison, we record data from two domains: a closed vocabulary and open vocabulary condition, which are described in sections 2.1 and 2.2 below. section 2.3 then provides additional details about the recording setup. ## method our method is built around a recurrent neural transduction model from emg features to time-aligned speech features (section 3.1). we will denote the featurized version of the signals used by the transduction model e s/v and a v for emg and audio respectively. when training solely on vocal- a core contribution of our work is a method of training the transducer model on silent emg signals, which no longer have time-aligned audio to use as training targets. we briefly describe our method here, then refer to section section 3.2 for more details. using a set of utterances recorded in both silent and vocalized speaking modes, we find alignments between the two recordings and use them to associate speech features from the vocalized instance (a v ) with the silent emg e s . the alignment is initially found using dynamic time warping between emg signals and then is refined using canonical correlation analysis (cca) and predicted audio from a partially trained model. finally, to generate audio from predicted speech features, we use a wavenet decoder, as described in section 3.3. ## experiments in this section, we run experiments to measure intelligibility of audio generated by our model from silent emg signals e s . since prior work has trained only on vocalized emg signals e v , we compare our method to a direct transfer baseline which trains a transducer model only on vocalized emg e v before testing on the silent emg e s . 4 the baseline transducer and wavenet models have identical architecture to those used by our method, but are not trained with silent emg using our target transfer approach. since one may hypothesize that most of the differences between silent and vocalized emg will take place near the vocal folds, we also test a variant of this baseline where the electrode placed on the neck is ignored. we first test on the closed vocabulary data described in section 2.1, then on the open vocabulary data from section 2.2. on the open vocabulary data, we also run ablations to evaluate different alignment refinements with cca and predicted audio (see sections 3.2.1 and 3.2.2). ## conclusion our results show that digital voicing of silent speech, while still challenging in open domain settings, shows promise as an achievable technology. we show that it is important to account for differences in emg signals between silent and vocalized speaking modes and demonstrate an effective method of doing so. on silent emg recordings from closed vocabulary data our speech outputs achieve high intelligibility, with a 3.6% transcription word error rate and relative error reduction of 95% from our baseline. we also significantly improve intelligibility in an open vocabulary condition, with a relative error reduction over 20%. we hope that our public release of data will encourage others to further improve models for this task. 6