Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks (2024)

Abstract: Hyperparameter tuning, the process of searching for suitable hyperparameters, becomes more difficult as the computing resources required to train neural networks continue to grow. This topic continues to receive little attention and discussion—much of it hearsay—despite its obvious importance. We attempt to formalize hyperparameter sensitivity using two metrics: similarity-based sensitivity and performance-based sensitivity. We then use these metrics to quantify two such claims: (1) transformers are more sensitive to hyperparameter choices than LSTMs and (2) transformers are particularly sensitive to batch size. We conduct experiments on two different character-level sequence-to-sequence tasks and find that, indeed, the transformer is slightly more sensitive to hyperparameters according to both of our metrics. However, we do not find that it is more sensitive to batch size in particular.

## introduction neural networks are famously hard to interpret and slightly mysterious to researchers and practitioners alike. hyperparameter tuning, typically an important part of developing a neural model, is often perceived as black magic-based on trial and error or inherited recipes. this has resulted in urban legends within the speech and nlp communities regarding deep learning models and their hyperparameters. at the same time, actual hyperparameter tuning is not commonly performed. larger models, along with the increasing multilinguality of nlp models and tasks, make hyperparameter tuning more expensive, and the prevalence of the pretraining-fine-tuning paradigm has many convinced that hyperparameter tuning is unimportant. we consider two specific claims regarding hyperparameters. first, we test whether transformers @xcite are more sensitive to specific hyperparameter choices than lstms @xcite . this can be found in ". . . (transformer) neural networks are very sensitive to architecture and hyperparameter settings." @xcite "unlike the lstm-based model. . . the transformer-based architecture was found to be sensitive to such changes." @xcite "the transformer models were even tuned longer, since they were more sensitive to small hyperparameter changes." @xcite "it is acknowledged that transformer @xcite model is extremely sensitive to the hyper-parameters. . . " @xcite "we. . . find (i) framing the task. . . and (ii) several additional techniques. . . can mitigate the (pretrained transformer)'s extreme sensitivity to hyperparameters." @xcite we first motivate and describe task-and architecture-agnostic metrics for quantifying hyperparameter sensitivity. then, we run a study where we apply our metrics to two character transduction tasks (i.e., character-level sequence-to-sequence tasks that typically have very short sequences; @xcite : morphological inflection and grapheme-to-phoneme conversion. we perform an extensive hyperparameter search for both lstm and transformer models for the two tasks, together covering ten languages. we find that transformers are typically more sensitive to hyperparameters according to both of our metrics. while, for batch size in particular, the transformer is more sensitive, we do not find that this sensitivity is any greater than all parameters together, which contradicts the claims of previous work. in addition, we also define a new state of the art for both tasks and each language as a side effect of our extensive tuning: surprisingly, we show that, in contrast to the results of @xcite , transformers do not outperform lstms with attention on either task with equivalent random tuning budget. finally, we analyze the most successful hyperparameter ranges for each architecture, and find that typical ranges used previously for lstms may be sub-optimal. ## related work hyperparameter sensitivity is frequently discussed in the nlp literature beyond the claims in ## hyperparameter sensitivity we define hyperparameter sensitivity as the extent to which an architecture changes in performance due to changes in hyperparameters. we distinguish between two types of sensitivity: performancebased sensitivity describes how likely high performance is across a random hyperparameter search-a common technique often resulting in good hyperparameters @xcite . in contrast, similarity-based sensitivity refers to how close performance is for similar hyperparameter configurations, a metric more relevant for structured hyperparameter optimization (e.g., @xcite . both types of sensitivity are quantified in terms of changes in model accuracy. we present multiple metrics for each type of hyperparameter sensitivity. figure 1 provides a visual intuition of the two types of hyperparameter sensitivity and the relationship to accuracy that the metrics measures. in case 1, we have an architecture that is fully robust to any changes in hyperparameters. this is not sensitive at all according to the performance-based metrics. in case 2, we have an architecture that linearly increases in performance as we make small changes to hyperparameters. although this may be a sensitive model according to performance-based metrics, it is not sensitive according to similarity-based metrics. finally, case 3 is more sensitive in terms of both types of metrics due to its inconsistent changes in performance, though still less sensitive than a completely random distribution of accuracy. given the large hyperparameter space that we study, we expect our outcomes to be closest to case 3. ## case study: character transduction we perform a case study to explore the hyperparameter sensitivity of transformers and lstms for character transduction tasks. we choose those tasks as the vocabularies and sequence lengths are typically quite short, and models train faster than for many other tasks, which makes it possible to run a large set of experiments within a few weeks. furthermore, there are several high-quality benchmarks available. we also believe that hyperparameter sensitivity is of particular interest to the community working on these problems because most character transduction datasets are highly multilingual-this is true of transliteration (e.g., @xcite ) and text normalization (e.g., bollmann, 2019) in addition to the tasks we report results on-and typically one model is trained for each language in the task. while our findings about the relative hyperparameter sensitivity of lstms vs. transformers may not generalize to the many tasks which lack these properties, the metrics we introduce can be applied to virtually any machine learning system. we submit that even for large pretrained models, where hyperparameter tuning is difficult and prohibitively expensive, sensitivity in the fine-tuning stage should be considered. ## results all results are discussed with respect to the average over all languages. we report per-language values in the appendix, in ## conclusion we have presented metrics for quantifying hyperparameter sensitivity according to two different definitions. in applying these metrics, we confirm that the lstm is less sensitive to hyperparameters than the transformer for character transduction. like in previous work, transformers also appear to be more sensitive to batch size than lstms. however, we find that they are no more sensitive to batch size than to all hyperparameters together. lastly, we find that lstms outperform transformers for these tasks and languages with few exceptions, which contradicts previous findings. this is likely because we have sampled hyperparameters outside of the typical range for lstms in character transduction. we believe that a careful measurement of the relationship between architectures and hyperparameter search is important for testing claims about hyperparameter sensitivity. we hope that our metrics will be used to explore hyperparameter sensitivity in other tasks.