Show Your Work with Confidence: Confidence Bands for Tuning Curves (2024)

Abstract: The choice of hyperparameters greatly impacts performance in natural language processing. Often, it is hard to tell if a method is better than another or just better tuned. *Tuning curves* fix this ambiguity by accounting for tuning effort. Specifically, they plot validation performance as a function of the number of hyperparameter choices tried so far. While several estimators exist for these curves, it is common to use point estimates, which we show fail silently and give contradictory results when given too little data.Beyond point estimates, *confidence bands* are necessary to rigorously establish the relationship between different approaches. We present the first method to construct valid confidence bands for tuning curves. The bands are exact, simultaneous, and distribution-free, thus they provide a robust basis for comparing methods.Empirical analysis shows that while bootstrap confidence bands, which serve as a baseline, fail to approximate their target confidence, ours achieve it exactly. We validate our design with ablations, analyze the effect of sample size, and provide guidance on comparing models with our method. To promote confident comparisons in future work, we release opda: an easy-to-use library that you can install with pip. https://github.com/nicholaslourie/opda

## introduction accounting for hyperparameters when comparing models is an important, open problem in nlp and deep learning research. this problem is particularly relevant right now, as the rush to scale up has left us with large, costly language models and little understanding of how different designs compare. indeed, many earlier models are now outperformed by others an order of magnitude smaller-the main difference: better hyperparameters. even worse, the challenge of managing hyperparameters during research has produced false scientific conclusions, such as the belief that model size should scale faster and lstm text classifiers on reuters @xcite , based on data from @xcite . the tuning curve plots the f1 score of the best model after each round of random search. the left compares point estimates for the mean based on u and v-statistics; the right compares 50% confidence bands for the median tuning curve. the top and the bottom run the same analysis on different samples of 25 search iterations. the point estimates give contradictory conclusions without warning on different samples, disagreeing whether the lstm ever beats the mlp. the differences between estimators are small in comparison to this sample variation. confidence bands, in contrast, directly show the variation due to sampling. to tackle this issue, researchers have developed the tuning curve @xcite @xcite . the tuning curve plots the current best validation score as a function of tuning effort, for example the search iterations or total compute used in hyperparameter tuning (see figure 1 ). by comparing tuning curves, you can determine which method is best for a given budget or tell if the hyperparameters are fully tuned. in theory, tuning curves fix the hyperparameter comparison problem; in practice, these curves must be estimated from data. prior work has developed efficient estimators for tuning curves @xcite ; however, these estimators lack corresponding methods to quantify their uncertainty. generic techniques, such as bootstrap resampling @xcite , break down for tuning curve estimates and thus fail to meaningfully capture their uncertainty @xcite . as a result, it is hard to know whether a conclusion is trustworthy or if more data is required ( §5.1). the absence of confidence bands has hindered tuning curves' more widespread adoption. it is this gap in the literature we seek to fill. to address these shortcomings, we present the first confidence bands for tuning curves. where the bootstrap breaks down, these confidence bands achieve meaningful coverage-both empirically and provably. beyond this basic necessity, they also have desirable theoretical properties; namely, they are exact, simultaneous, and distribution-free. being exact, the bands contain the true tuning curve with precisely the prescribed probabilityeven in finite samples. thus, at 95% confidence, they have 95% coverage, or contain the curve 95% of the time. in contrast, other confidence bands are often conservative, containing the true curve more often than claimed, or asymptotic, attaining correct coverage only as the sample size goes to infinity. being simultaneous, the bands contain the entire tuning curve at once. this guarantee is markedly stronger than pointwise coverage, or covering each point separately with the desired probability. for example, pointwise 95% confidence bands would often fail to cover a part of the true curve, while simultaneous bands will rarely fail to do so. and being distribution-free, the bands are free from restrictive parametric assumptions. as long as the validation score has a continuous distribution, the bands are exact; and even if the distribution is not continuous, the bands will still be conservative. as a result, the bands are guaranteed under general assumptions and thus widely applicable. thus, the confidence bands provide a scientific basis for evaluating models. the tuning curve captures tuning effort, while the confidence bands quantify uncertainty in the conclusion. being exact, that quantification is precise; being simultaneous, we can assess the model across all tuning budgets; and being distribution-free, the results are reliable and robust. our key insight is: take simultaneous confidence bands for the test scores' cumulative distribution function (cdf), then translate them to the tuning curve via an algebraic relationship ( §3.2). namely, we start with nonparametric bounds for the cdf, translate those to bound the cdf of the best score for a given tuning budget, then leverage that to bound a summary of the best score's distribution (e.g., the median or the mean). figure 3 exhibits the end result: given competing models (deberta and debertav3), we account for tuning effort by comparing tuning curves, and for sample variation by reporting them with confidence bands. in this way, the confidence bands empower researchers and practitioners to confidently identify the cost regimes where one method outperforms the other. complementing theoretical analysis, we study the confidence bands empirically. even when point estimates agree, different samples give different results due to undetected sample variation-which the bootstrap fails to capture ( §5.1). in contrast, our confidence bands attain exact coverage, both in theory and practice, as we confirm in experiments ( §5.2). beyond its core strategy, our approach also incorporates several more subtle design decisions. we show via ablation studies how each tightens the confidence bands ( §5.3). generalizing prior work, we then consider median as well as mean tuning curves, and find the median provides a more useful, interpretable, and tractable point of comparison ( §5.4). last, while confidence bands reveal sample variation, they do not eliminate it-so, we study the effect of sample size to guide how much data is necessary to estimate the tuning curve well ( §5.5). before diving into details, let us exhibit the end result. the next section demonstrates how to use the confidence bands in a practical scenario drawn from the recent literature: evaluating debertav3 @xcite against its baseline, deberta @xcite . bringing it all together, this tutorial shows how tuning curves promote reliable comparisons by accounting for tuning effort, while confidence bands reveal when a method is truly better versus when more data is required to reach a conclusion. because our confidence bands are distribution-free with exact coverage, they provide a rigorous, statistical basis for comparing methods that involve hyperparameters, sampling, or random initialization. to promote reliable comparisons and more reproducible research, we release an easy-touse library implementing our confidence bands at https://github.com/nicholaslourie/opda . ## tutorial: evaluating debertav3 let's walk through a case study fine-tuning a pretrained model, debertav3 @xcite , to see how our analysis works in practice. first, we design the search distribution. just like grid search, for each hyperparameter we choose a linear or log scale, pick upper and lower bounds, then set a (log) uniform distribution between them. for debertav3, we use a log scale for the learning rate (1e-6, 1e-3), and a linear scale for number of epochs (1, 4), batch size (16, 64), proportion of the first epoch for learning rate warmup (0, 0.6), and dropout (0, 0.3). with this distribution, we then run 48 rounds of random search. from these samples, we construct the tuning curve and confidence bands in figure 2 . this plot quantifies the model's performance as a function of tuning budget. from it, practitioners can tell if the model is cost effective, and researchers can judge whether it is fully tuned. we see that debertav3 has high accuracy after only 2-6 rounds of search. beyond absolute judgments, we need relative comparisons. figure 3 plots debertav3 against its baseline, deberta. 1 it is tempting to require that the bands have no overlap before deciding one model is better than the other; however, this rule is too conservative. inspired by minka (2002), we (tentatively) suggest a heuristic: evidence is weak if one band excludes the other's point estimate, fair if each band excludes the other's point estimate, and strong if the bands have no overlap, for a nontrivial portion of the curve. thus, there is strong evidence debertav3 beats deberta for all budgets. after establishing debertav3 beats deberta, we might ask: what makes the model work so well? we could test if a hyperparameter is important to tune, or ablate components to see what improves the score the most. since ablations work like model comparisons, let us analyze a hyperparameter. hyperparameter importance can be defined in many ways @xcite @xcite . @xcite give an intuitive definition as tuning risk: the difference in performance between tuning all hyperparameters and leaving one at its default. tuning risk depends on tuning effort; let's measure it with tuning curves. figure 4 compares tuning all hyperparameters to fixing epochs at its default. as epochs impacts the training time, we multiply the search iterations by an appropriate measure of cost (average epochs per iteration). adjusted for compute, we find weak evidence that the default wins over tuning epochs. ## theory we derive confidence bands for tuning curves that are simultaneous and distribution-free. these bands are conservative for the mean tuning curve and exact for the median tuning curve when the score distribution is continuous. see §a for proofs. ## experimental setup our experiments address two kinds of questions: comparisons to prior work. we adapt data foot_2 from @xcite , who identified many of the challenges inspiring our work. the data consists of 145 and 152 rounds of random search for mlp and lstm classifiers' f1 scores on the reuters document classification dataset @xcite . the hyperparameter search included learning rate, batch size, and dropout among others. like tang et al. (2020), we measure the empirical coverage of bootstrap confidence bands in simulation. we fit kernel density estimates (kde) to mlp and lstm hyperparameter tuning data, to simulate random search while also being able to compute the true tuning curve in this simulation. we improve @xcite protocol by using a kde method that includes a boundary correction, to account for f1 being bounded by 0 and 1. see §b.1 for details. analysis in a realistic scenario in addition to the experiments above, we demonstrate our tool on a practical problem from the recent literature: comparing debertav3 @xcite to its baseline, deberta @xcite . using the original implementation, foot_3 we trained deberta and debertav3 on multinli @xcite evaluated with accuracy over 1,024 rounds of random search. we tuned the dropout, learning rate, batch size, number of epochs, and warmup proportion of the first epoch for both models. last of all, we fit a kde model to this data for our exact coverage analysis ( §5.2). see §b.2 for more details. ## analysis we validate our theory with empirical results. ## related work language models' unprecedented success creates the need for more realistic, comprehensive, and challenging evaluations @xcite . as scaling reliably improves performance @xcite @xcite , performance comparisons without consideration for cost have become inadequate (ethayarajh and jurafsky, 2020): there is always a bigger model with better performance. thus, many researchers have brought attention to modern models' considerable costs, and proposed frameworks to account for them @xcite @xcite . due to hyperparameter search's heavy compute requirements, evaluations should identify not just the best model, but the best model for the tuning budget. many efficient tuning algorithms have been proposed @xcite @xcite ; however, these find the best hyperparameters for a given method, rather than evaluate the method for a given budget. beyond finding the best hyperparameters for deployment, researchers and practitioners need systematic tools to manage them during development-taking the guesswork out of evaluation. to meet this need, @xcite proposed the first estimator for the expected tuning curve, or mean performance as a function of hyperparameter search iterations. to compare models fairly, we must fix a tuning algorithm. random search is easy to implement, a strong baseline, and the basis of several state of the art techniques @xcite ; thus, it offers a great choice for standardizing tuning curves. insightfully, @xcite leveraged the independent and identically distributed nature of random search to estimate tuning curves using @xcite . v-statistics have many desirable properties; however, they can be biased, so @xcite introduced a complementary, unbiased estimator based on u-statistics @xcite . follow up work showed that although these estimators can disagree, neither is uniformly more correct @xcite . what is more, bootstrapping these estimators fails to create valid confidence bands @xcite . consequently, it is difficult to know when an estimate is reliable, versus when more data is necessary. we resolve this issue by providing the first valid confidence bands for tuning curves. ## conclusion we began with a tutorial on how to tell if a method is cost-effective, well-tuned, or actually better than a baseline, using our confidence bands ( §2). then, we derived our exact, simultaneous, distributionfree confidence bands for tuning curves ( §3). to complement theory, we designed empirical studies ( §4). these studies probed the shortcomings of existing solutions ( §5.1), and confirmed our bands achieve the exact coverage necessary to address them ( §5.2). beyond these main results, ablations revealed how several key ideas tighten the bands ( §5.3). lastly, we investigated the effect of sample size to show the benefits of the median over the mean tuning curve ( §5.4) and illustrate an empirical relationship that informs how many samples are needed to produce useful confidence bands ( §5.5). hyperparameters complicate evaluation. luckily, tuning curves let us compare methods while accounting for tuning effort. still, point estimates leave open the question of whether or not the data is sufficient to support a given conclusion. to solve this issue, we present the first valid confidence bands for tuning curves. our confidence bands are simultaneous, distribution-free, and achieve exact coverage in finite samples. using these confidence bands, researchers and practitioners can compare methods reliably and reproducibly. to analyze your own experiments with confidence, try our library at https://github.com/nicholaslourie/opda .