The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence (2025)

Abstract: Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a simple solution is not enough in the case of speech-to-text (S2T) trainings, where evolved and more complex variants of the Transformer architecture – e.g., Conformer or Branchformer – are used in light of their better performance. As a workaround, OWSM designed a double linear warmup of the LR, increasing it to a very small value in the first phase before updating it to a higher value in the second phase. While this solution worked well in practice, it was not compared with alternative solutions, nor was the impact on the final performance of different LR warmup schedules studied. This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.

## introduction following the success of large language models (llm) @xcite , large-scale speech-totext (s2t) trainings have gained increased interest with the goal of building large speech models (lsm) or speech foundation models (sfm) with similar abilities for the speech modality @xcite @xcite . scaling the size of the training data and trained models with respect to traditional small-scale speech trainings has posed many challenges beyond engineering efforts and demanding hardware requirements. among them, a significant challenge was ensuring the convergence of large models, * equal contribution. in this paper, we fill these gaps by studying which factors lead to a more difficult convergence of large-scale models and what is the impact of different lr warmup policies on the final performance. to this aim, we train large-scale s2t conformer @xcite models on more than 150k hours of speech data, exploring alternative warmup methods -specifically an exponential and a polynomial policy -operating between the double linear warmup by owsm and the traditional linear warmup phase of the inverse square root lr scheduler. our experiments demonstrate that: to ease future research on the topic, foster reproducibility of our work, and in accordance with the open science principles @xcite , we release the code, logs, and intermediate checkpoints under the open-source apache 2.0 license at https://github.com/hlt-mt/fbk-fairseq . ## learning rate schedulers this section describes the lr schedulers analyzed in this work, starting from the widely adopted inverse square root with linear warmup ( §2.1) and piecewise-linear warmup ( §2.2), to the alternative sub-linear warmup policies, namely polynomial ( §2.3) and exponential ( §2.4), designed to be as close as possible to the traditional inverse square root lr. all lr schedulers are shown in figure 1 . ## experimental settings to ensure that divergence issues are not due to a particularly challenging setting, we avoided multi-task trainings, resorting to training s2t models on the automatic speech recognition (asr) task for two languages (english and italian). as training data, we use ∼150k hours of publicly available speech datasets, which are described in appendix a. for validation, we use the english (en) and italian (it) dev sets of commonvoice @xcite . our encoder-decoder models have a transformer decoder and a conformer encoder preceded by two 1d convolutional layers that downsample the sequence length by a factor of 4. for the conformer encoder, we use the implementation by @xcite that fixes issues in padding handling. given the results of preliminary experiments ( §4.1), we set 24 encoder layers and 12 decoder layers for the experiments in §4.2. the embeddings have 1024 features, with an ffn hidden dimension of 4096 and 16 attention heads. in total, our models have 878m parameters. further details are provided in appendix a. ## conclusions in this study, we analyzed one of the key challenges -beyond engineering, data curation, and hardware efforts -associated with training largescale s2t models i.e., the role of the lr scheduler and, in particular, of its warmup strategy in model convergence and final performance. to this aim, we compared the standard linear warmup and the piecewise-linear warmup strategies with two policies -polynomial and exponential -aimed at finding the highest possible lr in the warmup phase that does not lead to convergence issues. through experiments on large-scale asr trainings of a ∼900m parameters conformer model, we demonstrated that while the lr warmup phase is crucial for stabilizing convergence, it has a minimal impact on final model performance and that the lr warmup phase should follow an exponential or sub-exponential rise to ensure model convergence.