  Penalized logistic regression methods are frequently used to investigate the
relationship between a binary outcome and a set of explanatory variables. The
model performance can be assessed by measures such as the concordance statistic
(c-statistic), the discrimination slope and the Brier score. Often, data
resampling techniques, e.g. crossvalidation, are employed to correct for
optimism in these model performance criteria. Especially with small samples or
a rare binary outcome variable, leave-one-out crossvalidation is a popular
choice. Using simulations and a real data example, we compared the effect of
different resampling techniques on the estimation of c-statistics,
discrimination slopes and Brier scores for three estimators of logistic
regression models, including the maximum likelihood and two maximum
penalized-likelihood estimators. Our simulation study confirms earlier studies
reporting that leave-one-out crossvalidated c-statistics can be strongly biased
towards zero. In addition, our study reveals that this bias is more pronounced
for estimators shrinking predicted probabilities towards the observed event
rate, such as ridge regression. Leave-one-out crossvalidation also provided
pessimistic estimates of the discrimination slope but nearly unbiased estimates
of the Brier score. We recommend to use leave-pair-out crossvalidation,
five-fold crossvalidation with repetition, the enhanced or the .632+ bootstrap
to estimate c-statistics and leave-pair-out or five-fold crossvalidation to
estimate discrimination slopes.
