Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech (2025)

Abstract: Autoregressive (AR) Transformer-based sequence models are known to have difficulty generalizing to sequences longer than those seen during training. When applied to text-to-speech (TTS), these models tend to drop or repeat words or produce erratic output, especially for longer utterances. In this paper, we introduce enhancements aimed at AR Transformer-based encoder-decoder TTS systems that address these robustness and length generalization issues. Our approach uses an alignment mechanism to provide cross-attention operations with relative location information. The associated alignment position is learned as a latent property of the model via backpropagation and requires no external alignment information during training. While the approach is tailored to the monotonic nature of TTS input-output alignment, it is still able to benefit from the flexible modeling power of interleaved multi-head self- and cross-attention operations. A system incorporating these improvements, which we call Very Attentive Tacotron, matches the naturalness and expressiveness of a baseline T5-based TTS system, while eliminating problems with repeated or dropped words and enabling generalization to any practical utterance length.

## introduction autoregressive (ar) transformer-based sequence models @xcite are used today in a majority of state-of-the-art systems across text, vision, and audio domains. though swiftly adopted for sequence-to-sequence text-output tasks like machine translation, video captioning @xcite , and speech recognition @xcite , adoption for audio-output tasks like text-to-speech (tts) is notably more recent. inherent difficulties in reliably evaluating tts systems coupled with the low quality of public tts datasets likely played a role in the delayed adoption. this was exacerbated by the failure of ar models of continuous sequences (e.g., spectrograms) to match the performance of their discrete counterparts in the text domain. it was only after the arrival of audio discretization techniques suitable for speech generation tasks @xcite that the tts community began to direct more effort toward ar transformer-based models. a key challenge faced by such models, however, stems from their extensive reliance on attention operations, which tend to cause robustness issues that make them unfit for use in a production tts environment. early attention-based tts models, such as tacotron @xcite , often exhibited problems like word omission or repetition, and struggled to generalize beyond the training lengths. attempts to mitigate these issues using monotonic alignment mechanisms yielded some success @xcite @xcite , but many ultimately turned to non-autoregressive, duration-based models, which were more robust and efficient during synthesis @xcite @xcite . while significant research has focused on improving robustness and length generalization for transformers in general, these issues still persist in the latest ar tts systems @xcite . in this paper we introduce a system called very attentive tacotron (vat), a discrete ar transformer-based encoder-decoder model designed for robust speech synthesis. vat augments a baseline t5-based @xcite tts architecture with an alignment mechanism that exploits the monotonic nature of the text-to-speech task, while preserving the powerful modeling capabilities of stacked self-and cross-attention layers. this leads to virtually limitless length generalization, while matching the naturalness and expressiveness of the t5 baseline system and eliminating issues with repeated or dropped words. our work was inspired by audiolm @xcite , which showed impressive speech generation results using a decoder-only ar transformer trained to model discrete targets produced by a self-supervised speech representation and a neural audio codec. this was extended with text inputs for tts in the decoder-only case by vall-e @xcite , and in the encoder-decoder case by spear-tts @xcite and mq-tts @xcite . however, @xcite found that vall-e was highly non-robust (in ways we note are reminiscent of early attention-based tts) and proposed ella-v, which essentially performs duration modeling by learning to predict "end-ofphoneme" markers via forced alignments of training data. with similar motivations, vall-t @xcite extends vall-e by replacing the objective with a transducer mechanism @xcite that marginalizes over hard alignments during training and interacts with alignment-shifted text-side relative position embeddings. unlike standard ar decoding, this scheme requires a costly sequence-wide inference pass each time the alignment shifts, and it is unclear whether it can generalize significantly beyond the training lengths. finally, to prevent major transcript deviations, mq-tts had to limit cross-attention to a single layer and head which is applied over a very narrow (3-6 input tokens) monotonically advancing window. in summary, current work has not resolved robustness issues without also sacrificing the power and flexibility of the underlying ar transformer. in this work, we focus on encoder-decoder models with cross-attention which we adapt for robustness and length generalization. the cross-attention operations are informed by a single monotonic alignment position that produces stability without degrading the modeling power of the repeated multi-head cross-attention layers present in the original transformer architecture. the scalar alignment position is learned directly and doesn't rely on dynamic programming or forced alignments during training. ## results to aid the reader, audio examples from the naturalness evaluation, length generalization assessment, and repeated words stress test are available online. 2 ## discussion we have shown that the proposed attention enhancements are able to eliminate robustness issues typically observed in transformer-based tts systems while matching the synthesis quality of a contemporary t5-based system. the vat model that incorporates these enhancements is able to reliably produce all words in the input text out to seemingly unbounded lengths, and the alignment-informed, multi-layer, multi-head cross attention it uses is inherently more powerful than the single-phoneme alignment mechanisms used in other robustnessoriented tts models (see appendix e). this approach can be directly applied to any encoder-decoder model that uses cross-attention layers in the decoder. because the alignment position is constrained to be monotonic, it is best suited for tasks that exhibit broad monotonic alignment between input and output (e.g., tts and asr). however, the flexibility afforded by query-key comparisons within wide irpb windows should allow the model to adapt well to off-monotonic alignments if needed (e.g., when using unverbalized text). due to the scalability of transformer-based models, future work should test vat on significantly larger datasets, potentially with encoder or decoder pre-training on unpaired text or audio, respectively. additionally, using fancier, more powerful approaches to discretization and waveform generation is likely to yield low-level audio quality improvements especially if paired with cleaner, higher-quality datasets.