  Typical high quality text-to-speech (TTS) systems today use a two-stage
architecture, with a spectrum model stage that generates spectral frames and a
vocoder stage that generates the actual audio. High-quality spectrum models
usually incorporate the encoder-decoder architecture with self-attention or
bi-directional long short-term (BLSTM) units. While these models can produce
high quality speech, they often incur O($L$) increase in both latency and
real-time factor (RTF) with respect to input length $L$. In other words, longer
inputs leads to longer delay and slower synthesis speed, limiting its use in
real-time applications. In this paper, we propose a multi-rate attention
architecture that breaks the latency and RTF bottlenecks by computing a compact
representation during encoding and recurrently generating the attention vector
in a streaming manner during decoding. The proposed architecture achieves high
audio quality (MOS of 4.31 compared to groundtruth 4.48), low latency, and low
RTF at the same time. Meanwhile, both latency and RTF of the proposed system
stay constant regardless of input lengths, making it ideal for real-time
applications.
