S pike V oice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network (2024)

Abstract: Brain-inspired Spiking Neural Network (SNN) has demonstrated its effectiveness and efficiency in vision, natural language, and speech understanding tasks, indicating their capacity to “see”, “listen”, and “read”. In this paper, we design SpikeVoice, which performs high-quality Text-To-Speech (TTS) via SNN, to explore the potential of SNN to “speak”. A major obstacle to using SNN for such generative tasks lies in the demand for models to grasp long-term dependencies. The serial nature of spiking neurons, however, leads to the invisibility of information at future spiking time steps, limiting SNN models to capture sequence dependencies solely within the same time step. We term this phenomenon “partial-time dependency”. To address this issue, we introduce Spiking Temporal-Sequential Attention (STSA) in the SpikeVoice. To the best of our knowledge, SpikeVoice is the first TTS work in the SNN field. We perform experiments using four well-established datasets that cover both Chinese and English languages, encompassing scenarios with both single-speaker and multi-speaker configurations. The results demonstrate that SpikeVoice can achieve results comparable to Artificial Neural Networks (ANN) with only 10.5% energy consumption of ANN. Both our demo and code are available as supplementary material.

## introduction since the advent of artificial neural networks (ann), remarkable achievements have been made in the field of image @xcite @xcite , natural language @xcite @xcite , and speech @xcite . in recent years, with the success of large language models @xcite @xcite @xcite , there has been a notable upward trend in energy consumption. at the same time, spiking neural network (snn), inspired by the biological nervous system and recognized as the third generation of neural networks @xcite , employs spiking neurons @xcite @xcite with charge-fire-reset temporal dynamic. the temporal dynamic makes snn to exhibit the event-driven feature of sparse firing and the binary spike communication feature between neurons using 0s and 1s, providing a distinct advantage in energy efficiency @xcite . recently, snn has achieved remarkable progress on several tasks, such as object detection and image classification @xcite @xcite , speech recognition @xcite , and text classification tasks @xcite . it is the success of these tasks that have led us to believe that snn has preliminarily acquired the abilities of "seeing", "listening", and "reading". however, applying snn to generative tasks encounters some obstacles, particularly in addressing the challenge of snn capturing long-term dependencies. as mentioned above, spiking neurons have a temporal dynamic of charge-fire-reset. such a serial process hinders the capture of information from future time steps in the spiking temporal dimension. existing snn models performing attention operations in the spiking sequential dimension can only establish sequence dependencies within the same time step or, in other words, among partial binary embedding @xcite , hindering the establishment of long-term dependencies. we term this phenomenon as "partial-time dependency". in this paper, we introduce spikevoice, a high-quality text-to-speech (tts) model with a transformer-based snn framework @xcite solving the "partial-time dependency" problem, and successfully explore the potential of snn to "speak". to address the issue of "partial-time dependency", we propose spiking temporal-sequential attention (stsa) in spikevoice. stsa performs temporal-mixing in the spiking temporal dimension to capture information from future time steps, enabling access to the global information of binary embedding at each spiking time step. after time-mixing, stsa performs sequential-mixing in the spiking sequential dimension to integrate contextual information. furthermore, we implement spikevoice in a spike-driven manner with the leaky integrate-and-fire (lif) @xcite neurons, fully harnessing the energy efficiency of snn. spike-driven denotes the concurrent existence of both the binary spike communication feature and the event-driven feature. to the best of our knowledge, spikevoice is the first tts model within the snn framework, which not only promotes the development of snn in generative tasks but also expands the scope of the snn model in practical applications. the main contributions are summarized as follows: transformers in snn: training in snn is primarily categorized into two methods: ann-to-snn conversion (ann2snn) @xcite @xcite and surrogate training @xcite @xcite . leveraging ann2snn, @xcite integrates the transformer architecture into snn. nevertheless, this approach demands dozens or even hundreds of time steps to attain satisfactory performance. spikeformer @xcite conducts direct training of the transformer within the snn framework and achieves state-of-the-art performance on imagenet with just four time steps. however, it doesn't fully harness the energy-efficient advantages of snn due to the presence of multiplyand-accumulate (mac) operations. spike-driven transformer @xcite incorporates the spike-driven paradigm into transformer architecture and introduces the spike-driven self-attenton (sdsa) @xcite . sdsa utilizes sparse additive operations as a replacement for multiplication operations in attention mechanisms, effectively addressing the issues present in spikeformer related to mac operations. spikegpt @xcite is the first to introduce text generation tasks into the snn framework. however, it still does not make full of the energy-efficient capabilities of snn. transformers in tts: tactron2 @xcite employs rnn @xcite for speech synthesis which results in low training efficiency and struggles to establish long-term dependencies. to address these issues, transformer-tts @xcite introduces an autoregressive tts model that combines tactron2 with the transformer, enhancing training efficiency while capturing long-term dependencies. however, autoregressive tts models often suffer from slow synthesis speed and less robust speech synthesis. fastspeech @xcite , on the other hand, utilizes knowledge distillation during training to build a non-autoregressive tts model, yet the training process can be complicated. fastspeech2 @xcite simplifies the training process by removing knowledge distillation from the fastspeech training pipeline and adopting the end-to-end training approach, effectively addressing the issue of the extended training duration associated with fast-speech. ## method in this study, we propose spikevoice, the first spikedriven tts model. the overall model architecture is illustrated in fig.1. the spiking phoneme encoder (spe) performs binary embedding on the input phoneme embedding sequence and generates high-level spiking phoneme representations. the spiking variance adaptor (sva) enhances the spiking phoneme representations by incorporating variance information related to duration, pitch, and energy. finally, the spiking mel decoder (smd) and spiking postnet generate mel-spectrograms in a non-autoregressive manner. in the following sections, we will first introduce the lif neurons, and then introduce the components of spikevoice. ## experiments we conducted experiments with spikevoice on single-speaker and multi-speaker datasets, encompassing both english and chinese. the singlespeaker datasets include ljspeech @xcite and baker 1 , while the multi-speaker datasets comprise libritts @xcite and aishell3 @xcite . in the following subsections, we present results on subjective and objective metrics for ground truth denoted as 'gt', ann baseline denoted as 'fastspeech2', spikevoice signified as 'spikevoice-stsa', and snn baselines: spikevoice with attention in spikeformer replacing the stsa, which is denoted as 'spikevoice-attn' and spikevoice with only spiking sequential attention, which denoted as 'spikevoice-sdsa'. additionally, in section 4.5, we perform visual analysis, and in section 4.6, we discuss the balance between spikevoice's energy consumption and the quality of synthesized speech. ## conclusion in this paper, we introduce spikevoice. to the best of our knowledge, it is the first tts model that achieves high-quality speech synthesis within the snn framework and for the first time endows snn with the ability to "speak". additionally, spikevoice is a spike-driven model with highly energy-efficient. in spikevoice, we propose stsa, which performs temporal-mixing in the spiking temporal dimension to address the issue of information invisibility at future time steps on the spiking temporal dimension caused by the serial nature of spiking neurons and thereby address the issue of "partial-time dependency". we conducted experiments on both singlespeaker and multi-speaker datasets in both chinese and english. the results demonstrate that spikevoice achieves performance comparable to ann models while consuming only 10.5% of the energy required by ann. our successful practice proves the feasibility of tts tasks within the snn framework and offers an energy-saving solution for tts tasks.