An Improved Model for Voicing Silent Speech (2021)

Abstract: In this paper, we present an improved model for voicing silent speech, where audio is synthesized from facial electromyography (EMG) signals. To give our model greater flexibility to learn its own input features, we directly use EMG signals as input in the place of hand-designed features used by prior work. Our model uses convolutional layers to extract features from the signals and Transformer layers to propagate information across longer distances. To provide better signal for learning, we also introduce an auxiliary task of predicting phoneme labels in addition to predicting speech audio features. On an open vocabulary intelligibility evaluation, our model improves the state of the art for this task by an absolute 25.8%.

## introduction emg-based voicing of silent speech is a task that aims to synthesize vocal audio from muscular signals captured by electrodes on the face while words are silently mouthed @xcite . while recent work has demonstrated a high intelligibility of generated audio when restricted to a narrow vocabulary @xcite , in a more challenging open vocabulary setting the intelligibility remained low (68% wer). in this work, we introduce an new model for voicing silent speech that greatly improves intelligibility. we achieve our improvements by modifying several different components of the model. first, we improve the input representation. while prior work on emg speech processing uses hand-designed features @xcite @xcite which may throw away some information from the raw signals, our model learns directly from the complete signals with minimal pre-processing by using a set of convolutional neural network layers as feature ex-tractors. this modification follows recent work in speech processing from raw waveforms @xcite and gives our model the ability to learn its own features for emg. second, we improve the neural architecture of the model. while other silent speech models have been based around recurrent layers such as lstms @xcite , we use the self-attention-based transformer architecture @xcite , which has been shown to be a more powerful replacement across a range of tasks. finally, we improve the signal used for learning. since the relatively small data sizes for this task creates a challenging learning problem, we introduce an auxiliary task of predicting phoneme labels to provide additional guidance. this auxiliary loss is inspired by prior work on the related problem of generating speech from ecog sensors on the brain, which greatly benefited from intermediate prediction of phonemic information @xcite . we evaluate intelligibility of audio synthesized by our model on the single-speaker data from @xcite in the most challenging open-vocabulary setting. our results reflect an absolute improvement in error rate of 25.8% over the state of the art, from 68.0% to 42.2%, as measured by automatic transcription. evaluation by human transcription gives an even lower error rate of 32%. ## model at a high level, our system works by predicting a sequence of speech features from emg signals and using a wavenet vocoder @xcite to synthesize audio from those predicted features, as was done in @xcite . the first component, dubbed the transduction model, takes in emg signals from eight electrodes around the face and outputs a sequence of speech features represented as mel-frequency cepstral coefficients (mfccs). the final step of vocoding audio from mfccs is unchanged in our work, so we defer to @xcite for the details of the wavenet model. the neural architecture for our transduction model is made up of a set of residual convolution blocks followed by a transformer with relative position embeddings, as shown in figure 1 . we describe these two components in sections 2.1 and 2.2 below. next, in section 2.3 we describe our training procedure, which aligns each silent utterance to a corresponding vocalized utterance as in @xcite but with some minor modifications. finally, in section 2.4 we describe the auxiliary phoneme-prediction loss that provides additional signal to our model during training. foot_0 ## results we train our model on the open-vocabulary data from @xcite . this data contains 19 hours of facial emg data recordings from a single english speaker during silent and vocalized speech. our primary evaluation uses the automatic metric from that work, which transcribes outputs with an automatic speech recognizer foot_1 and compares to a reference with a word error rate (wer) metric. we also evaluate human intelligibility in section 3.1 below. 3 the results of the automatic evaluation are shown in ## phoneme error analysis one additional advantage to using an auxiliary phoneme prediction task is that it provides a more easily interpretable view of model predictions. although the phoneme predictions are not directly part of the audio synthesis process, we have observed that mistakes in audio and phoneme prediction are often correlated. therefore, to better understand the errors that our model makes, we analyze the errors of our model's phoneme predictions. to analyze the phoneme predictions, we align predictions on a silent utterance to phoneme labels of a vocalized utterance using the procedure described above in sections 2.3 and 2.4, then evaluate the phonemes using the measures described in sections 4.1 and 4.2 below. ## conclusion by improving several model components for voicing silent speech, our work has achieved a 38% relative error reduction on this task. although the problem is still far from solved, we believe the large rate of improvement is a promising sign for continued progress.